# Formance Documentation ## Other - [Build with Formance](https://docs.formance.com/): Learn how to get started with Formance. ## Changelog - [Stack v3.2](https://docs.formance.com/changelog/v3.2) - [Stack Changelog](https://docs.formance.com/changelog) - [Stack v3.1](https://docs.formance.com/changelog/v3.1) - [Stack v3.0](https://docs.formance.com/changelog/v3.0) ## Cloud - [Core Concepts](https://docs.formance.com/cloud/banking-bridge/core-concepts): Crash course into the world of banking. Start here for a glossary of terms. - [Banking Bridge](https://docs.formance.com/cloud/banking-bridge) - [Initiate Bank Transfers in Bulk](https://docs.formance.com/cloud/banking-bridge/payment-initiation-instructions): Automate bulk payouts from your bank account with a simple REST API - [Never Lose Track of Your Bank Account](https://docs.formance.com/cloud/banking-bridge/account-statement-ingestion): Make bank statements a native part of your reconciliation process - [Connect Directly to Your Bank](https://docs.formance.com/cloud/banking-bridge/direct-to-bank-transfer): Formance manages the EBICS connection to your bank so your systems only ever talk to a simple REST API. ## Cookbook - [Omnibus Account Management](https://docs.formance.com/examples/advanced/omnibus) - [RideShare Tutorial](https://docs.formance.com/examples/getting-started/rideshare-tutorial) - [Introduction](https://docs.formance.com/examples/introduction) - [Payment Card Acceptance Processing](https://docs.formance.com/examples/advanced/payment-acceptance) - [Card Issuing & Financial Host](https://docs.formance.com/examples/advanced/issuing-financial-host) - [Account Hierarchy Examples](https://docs.formance.com/examples/account-hierarchies) - [Stablecoin On-Ramp & Off-Ramp Operations](https://docs.formance.com/examples/advanced/stablecoin-operations) - [Stablecoin Issuer](https://docs.formance.com/examples/advanced/stablecoin-issuer) - [Neobank (FBO)](https://docs.formance.com/examples/advanced/neobank-fbo) - [Crypto Custody](https://docs.formance.com/examples/advanced/crypto-custody) - [BNPL & Lending](https://docs.formance.com/examples/advanced/bnpl-lending) - [Marketplace Payouts](https://docs.formance.com/examples/advanced/marketplace-payouts) ## Deploy - [Overview](https://docs.formance.com/deploy/cloud/overview) - [Deployment Overview](https://docs.formance.com/deploy/overview) - [All-in-one Demo Chart](https://docs.formance.com/deploy/self-hosted/demo/quick-start) - [Operator Setup](https://docs.formance.com/deploy/self-hosted/installation/operator-setup) - [Control Plane Setup](https://docs.formance.com/deploy/self-hosted/installation/control-plane) - [Requirements](https://docs.formance.com/deploy/self-hosted/requirements) - [PostgreSQL](https://docs.formance.com/deploy/self-hosted/infrastructure/postgresql) - [Message Broker](https://docs.formance.com/deploy/self-hosted/infrastructure/message-broker) - [Temporal](https://docs.formance.com/deploy/self-hosted/infrastructure/temporal) - [Stack Configuration](https://docs.formance.com/deploy/self-hosted/modules/stack) - [Gateway Module](https://docs.formance.com/deploy/self-hosted/modules/gateway) - [Ledger Module](https://docs.formance.com/deploy/self-hosted/modules/ledger) - [Payments Module](https://docs.formance.com/deploy/self-hosted/modules/payments) - [Auth Module](https://docs.formance.com/deploy/self-hosted/modules/auth) - [Orchestration Module](https://docs.formance.com/deploy/self-hosted/modules/orchestration) - [Reconciliation Module](https://docs.formance.com/deploy/self-hosted/modules/reconciliation) - [Wallets Module](https://docs.formance.com/deploy/self-hosted/modules/wallets) - [Webhooks Module](https://docs.formance.com/deploy/self-hosted/modules/webhooks) - [Operator settings reference](https://docs.formance.com/deploy/self-hosted/config/settings) - [API Reference](https://docs.formance.com/deploy/self-hosted/config/api-reference) - [Configure OpenTelemetry](https://docs.formance.com/manage/observability/opentelemetry) - [Backup management](https://docs.formance.com/deploy/self-hosted/backups) - [Traces](https://docs.formance.com/manage/observability/traces) - [Metrics](https://docs.formance.com/manage/observability/metrics) - [Support Telemetry](https://docs.formance.com/manage/observability/support-telemetry) - [Upgrade from the operator](https://docs.formance.com/manage/upgrade/from-operator) - [Database update](https://docs.formance.com/manage/upgrade/database) - [Troubleshooting](https://docs.formance.com/deploy/self-hosted/troubleshooting) ## Get Started - [Try the Sandbox](https://docs.formance.com/getting-started/sandbox) - [Platform Quick Start](https://docs.formance.com/getting-started/quickstart) - [Connect Your App](https://docs.formance.com/getting-started/connect-app) - [Core Concepts](https://docs.formance.com/getting-started/concepts) - [How the Modules Fit Together](https://docs.formance.com/getting-started/modules-fit-together) - [Getting Help](https://docs.formance.com/getting-started/help) ## Modules - [Architecting for scale](https://docs.formance.com/modules/ledger/advanced/architecting-for-scale) - [Ledger](https://docs.formance.com/modules/ledger/core-concepts/ledger) - [Configure an Event Sink](https://docs.formance.com/modules/ledger/events/configure-sink): Send supported Ledger events to a broker, webhook, or analytical store. - [Overview](https://docs.formance.com/modules/ledger/get-started/overview): Understand the Ledger accounting model and choose your next task. - [Plan the Migration](https://docs.formance.com/modules/ledger/migration/overview): Define the migration path, responsibilities, evidence, and rollback boundary. - [Design a Chart of Accounts](https://docs.formance.com/modules/ledger/modeling/chart-of-accounts): Choose ledger boundaries and stable account addresses for your business model. - [Prepare a Ledger Deployment](https://docs.formance.com/modules/ledger/operations/deploy): Evaluate Ledger directly, then prepare the requirements and acceptance checks for a Stack v4 deployment. - [Read Ledger Data](https://docs.formance.com/modules/ledger/querying/read-ledger-data): Retrieve accounts and transactions, then trace requests through logs and audit entries. - [Create Transactions with Postings](https://docs.formance.com/modules/ledger/transactions/create-with-postings): Record a known set of account movements and verify the resulting volumes. - [Ledger](https://docs.formance.com/modules/ledger) - [Getting Started](https://docs.formance.com/modules/payments/connectors/generic/getting-started) - [Quick Start](https://docs.formance.com/modules/ledger/quick-start): Create a ledger, record your first transactions, and verify the resulting balances. - [Hands-on walkthrough](https://docs.formance.com/modules/ledger/hands-on): Run one Ledger 3.0 session end to end: amounts and volumes, colors, metadata, reads, and aggregation. - [Accounts](https://docs.formance.com/modules/ledger/core-concepts/accounts) - [Consume, Monitor, and Recover Events](https://docs.formance.com/modules/ledger/events/consume-and-recover): Process at-least-once events without duplicate business effects and recover a stalled sink. - [Prepare the Source](https://docs.formance.com/modules/ledger/migration/prepare-source): Inventory Ledger v2 and prove that the target can read its complete log. - [Assets, Amounts, Balances, and Volumes](https://docs.formance.com/modules/ledger/modeling/assets-amounts-and-balances): Encode monetary values as integers and interpret account and aggregate volumes. - [Plan Topology and Manage Cluster Lifecycle](https://docs.formance.com/modules/ledger/operations/cluster-lifecycle): Choose a voting topology, then scale and replace members without losing quorum. - [Filter and Paginate Results](https://docs.formance.com/modules/ledger/querying/filters-and-pagination): Build target-aware filters and resume large result sets safely. - [Write Transactions with Numscript](https://docs.formance.com/modules/ledger/transactions/numscript): Resolve balance-aware and multi-leg transaction logic inside Ledger. - [Assets](https://docs.formance.com/modules/ledger/core-concepts/assets) - [Core Concepts](https://docs.formance.com/modules/ledger/core-concepts) - [Update Ledger Clients](https://docs.formance.com/modules/ledger/migration/update-clients): Adapt each Ledger v2 application workflow to the current API contract. - [Metadata and Typed Values](https://docs.formance.com/modules/ledger/modeling/metadata): Store application context and declare metadata types for reliable queries. - [Authentication and Authorization](https://docs.formance.com/modules/ledger/operations/authentication): Authenticate Ledger clients with OIDC or Ed25519 JWTs and grant only the scopes they need. - [Choose Read Consistency and Use Checkpoints](https://docs.formance.com/modules/ledger/querying/consistency-and-checkpoints): Control live-read freshness and freeze a ledger view for multi-query reconciliations. - [Reuse and Version Numscript Programs](https://docs.formance.com/modules/ledger/transactions/numscript-library): Publish immutable Numscript versions and execute them by reference. - [Transactions & Postings](https://docs.formance.com/modules/ledger/core-concepts/transactions) - [Create a Mirror](https://docs.formance.com/modules/ledger/migration/create-mirror): Create a read-only target that follows Ledger v2 over HTTP or PostgreSQL. - [Account Types and Enforcement](https://docs.formance.com/modules/ledger/modeling/account-types): Validate account-address patterns and choose how each account's volumes are stored. - [TLS and Network Security](https://docs.formance.com/modules/ledger/operations/tls-and-network-security): Encrypt Ledger gRPC traffic, preserve cluster trust during rollout, and restrict network access. - [Create and Manage Indexes](https://docs.formance.com/modules/ledger/querying/indexes): Declare query indexes, verify per-replica readiness, inspect metadata, and remove unused indexes. - [Retry Writes Safely](https://docs.formance.com/modules/ledger/transactions/idempotency-and-references): Use idempotency keys and transaction references to prevent duplicate financial operations. - [Double-entry Bookkeeping](https://docs.formance.com/modules/ledger/core-concepts/double-entry) - [Transform Mirrored Data](https://docs.formance.com/modules/ledger/migration/transformations): Create deterministic rules for intentional differences between source and target. - [Track Fund Origin with Colors](https://docs.formance.com/modules/ledger/modeling/color-of-money): Keep separate balances for funds that share an asset but have different origins or restrictions. - [Request and Response Signing](https://docs.formance.com/modules/ledger/operations/signing): Sign admission batches, manage signing keys, and understand response-verification limits. - [Create Prepared Queries](https://docs.formance.com/modules/ledger/querying/prepared-queries): Store, parameterize, execute, evolve, and remove shared query definitions. - [Revert Transactions](https://docs.formance.com/modules/ledger/transactions/revert): Create an auditable compensating transaction without deleting the original record. - [Capabilities](https://docs.formance.com/modules/payments/capabilities) - [Generic Connector](https://docs.formance.com/modules/payments/connectors/generic) - [Chart of Accounts](https://docs.formance.com/modules/ledger/core-concepts/chart-of-accounts) - [Monitor the Mirror](https://docs.formance.com/modules/ledger/migration/monitor-mirror): Measure catch-up, diagnose a blocked log, and decide whether validation can proceed. - [Monitor Ledger](https://docs.formance.com/modules/ledger/operations/monitoring): Connect Ledger telemetry and act on signals that threaten availability or correctness. - [Analyze and Aggregate Ledger Data](https://docs.formance.com/modules/ledger/querying/analyze-and-aggregate): Discover data patterns, interpret ledger counters, and calculate filtered volume totals. - [Process Operations in Bulk](https://docs.formance.com/modules/ledger/transactions/bulk): Submit ordered REST operations with explicit failure and retry behavior. - [Validate the Migration](https://docs.formance.com/modules/ledger/migration/validation): Collect repeatable technical and business evidence before promotion. - [Plan Capacity and Tune Performance](https://docs.formance.com/modules/ledger/operations/capacity-and-performance): Size every Ledger voter for the workload, then tune measured constraints safely. - [Cut Over and Promote](https://docs.formance.com/modules/ledger/migration/cut-over-and-promote): Freeze Ledger v2, ingest the final tail, promote once, and open target traffic. - [Use Maintenance Mode](https://docs.formance.com/modules/ledger/operations/maintenance-mode): Create a stable write boundary for a planned Ledger operation, then reopen safely. - [Back Up a Cluster](https://docs.formance.com/modules/ledger/operations/backup): Create full and incremental backups, then retain the evidence needed for recovery. - [Restore a Cluster](https://docs.formance.com/modules/ledger/operations/restore): Restore a backup into fresh storage, verify its recovery point, and reopen traffic deliberately. - [Assets & currency conversion](https://docs.formance.com/modules/ledger/working-with/currency-conversion) - [PSP Connectors](https://docs.formance.com/modules/payments/connectors/psp) - [Operations](https://docs.formance.com/modules/payments/operations) - [Verify Ledger Integrity](https://docs.formance.com/modules/ledger/operations/integrity): Run integrity checks, understand their boundaries, and respond without modifying evidence. - [Bi-temporality](https://docs.formance.com/modules/ledger/working-with/bi-temporality) - [Upgrade Ledger](https://docs.formance.com/modules/ledger/operations/upgrade): Roll Stack, Ledger, and ledgerctl versions with explicit compatibility and rollback checks. - [Reverting Transactions](https://docs.formance.com/modules/ledger/working-with/reverting-transactions) - [Troubleshoot Ledger](https://docs.formance.com/modules/ledger/operations/troubleshooting): Start from a user-visible symptom, collect safe evidence, and reach the owning runbook. - [Bulk processing](https://docs.formance.com/modules/ledger/working-with/bulk-processing) - [Manage Ledger Configuration as Code](https://docs.formance.com/modules/ledger/operations/configuration-as-code): Export, review, preview, apply, and verify editable ledger configuration. - [Filtering queries](https://docs.formance.com/modules/ledger/working-with/filtering-queries) - [Idempotency](https://docs.formance.com/modules/ledger/working-with/idempotency) - [Accounts](https://docs.formance.com/modules/payments/accounts) - [Exchange Connectors](https://docs.formance.com/modules/payments/connectors/exchange) - [Data isolation with buckets](https://docs.formance.com/modules/ledger/working-with/data-isolation-buckets) - [Streaming to analytics systems](https://docs.formance.com/modules/ledger/working-with/data-streaming) - [Ledger Schema](https://docs.formance.com/modules/ledger/working-with/ledger-schema) - [Open Banking](https://docs.formance.com/modules/payments/connectors/open-banking) - [Payments](https://docs.formance.com/modules/payments/payments) - [Orders](https://docs.formance.com/modules/payments/orders) - [Conversions](https://docs.formance.com/modules/payments/conversions) - [Events Publishers](https://docs.formance.com/modules/ledger/advanced/events-publishers) - [Build a connector](https://docs.formance.com/modules/payments/connectors/build-connector) - [Performance model](https://docs.formance.com/modules/ledger/advanced/performance-model) - [Verifying integrity](https://docs.formance.com/modules/ledger/advanced/verifying-integrity): How to detect and recover from tampering in a tamper-evident ledger, and what each HASH_LOGS mode gives you to verify against. - [Numscript](https://docs.formance.com/modules/numscript) - [Program Structure](https://docs.formance.com/modules/numscript/program-structure) - [Selecting an Interpreter](https://docs.formance.com/modules/numscript/interpreter) - [Send](https://docs.formance.com/modules/numscript/reference/send) - [Payment Initiation](https://docs.formance.com/modules/payments/transfer-initiation) - [Sources](https://docs.formance.com/modules/numscript/reference/sources) - [Destinations](https://docs.formance.com/modules/numscript/reference/destinations) - [Rounding](https://docs.formance.com/modules/numscript/reference/rounding) - [Save](https://docs.formance.com/modules/numscript/reference/save) - [Overdraft](https://docs.formance.com/modules/numscript/reference/overdraft) - [Account Pools](https://docs.formance.com/modules/payments/cash-pools) - [Variables](https://docs.formance.com/modules/numscript/reference/variables) - [Metadata](https://docs.formance.com/modules/numscript/reference/metadata) - [Unambiguous Monetary Notation](https://docs.formance.com/modules/numscript/monetary-notation) - [CLI](https://docs.formance.com/modules/numscript/cli) - [Functions](https://docs.formance.com/modules/numscript/reference/functions) - [Numscript specs format](https://docs.formance.com/modules/numscript/specs) - [Payment Service Users](https://docs.formance.com/modules/payments/payment-service-users) - [oneof](https://docs.formance.com/modules/numscript/reference/oneof) - [Payments](https://docs.formance.com/modules/payments) - [Account Interpolation](https://docs.formance.com/modules/numscript/reference/account-interpolation) - [get_asset](https://docs.formance.com/modules/numscript/reference/get-asset) - [get_amount](https://docs.formance.com/modules/numscript/reference/get-amount) - [Mid-script Function Calls](https://docs.formance.com/modules/numscript/reference/mid-script-functions) - [Asset Colors](https://docs.formance.com/modules/numscript/reference/asset-colors) - [Getting Started with Open Banking](https://docs.formance.com/modules/payments/connectors/open-banking/getting-started) - [Plaid](https://docs.formance.com/modules/payments/connectors/open-banking/plaid): Connect Plaid to Formance Payments to let end users link their bank accounts and sync the resulting accounts, balances, and transactions. - [Tink](https://docs.formance.com/modules/payments/connectors/open-banking/tink): Connect Tink to Formance Payments to let end users link their European bank accounts and sync the resulting accounts, balances, and transactions. - [Reference](https://docs.formance.com/modules/numscript/reference) - [Powens](https://docs.formance.com/modules/payments/connectors/open-banking/powens): Connect Powens to Formance Payments to let end users link their European bank accounts via Powens Webview and sync the resulting accounts, balances, and transactions. - [Connectors](https://docs.formance.com/modules/payments/connectors) - [How it Works](https://docs.formance.com/modules/payments/connectors/generic/how-it-works) - [Adyen](https://docs.formance.com/modules/payments/connectors/psp/adyen): Connect an Adyen merchant account to Formance Payments to sync the live account list and ingest webhook events. - [Atlar](https://docs.formance.com/modules/payments/connectors/psp/atlar): Connect an Atlar workspace to Formance Payments to sync accounts, external counterparties, and transactions across SEPA, BACS, and ACH rails. - [Banking Circle](https://docs.formance.com/modules/payments/connectors/psp/bankingcircle): Connect a Banking Circle account to Formance Payments to sync accounts, balances, and transactions, and to initiate transfers, payouts, and bank-account creation. - [Coinbase Prime](https://docs.formance.com/modules/payments/connectors/exchange/coinbaseprime): Connect a Coinbase Prime portfolio to Formance Payments to sync wallets, balances, payments, orders, and conversions. - [Column](https://docs.formance.com/modules/payments/connectors/psp/column): Connect a Column bank-as-a-service tenant to Formance Payments to sync accounts, balances, and payments across ACH, wire, and book-transfer rails, with full outbound initiation and webhook ingest. - [Currencycloud](https://docs.formance.com/modules/payments/connectors/psp/currencycloud): Connect a Currencycloud account to Formance Payments to sync FX accounts, multi-currency balances, beneficiaries, and conversions, with outbound transfer and payout initiation. - [Fireblocks](https://docs.formance.com/modules/payments/connectors/exchange/fireblocks): Connect a Fireblocks workspace to Formance Payments to sync vault accounts, balances, and transactions. - [Increase](https://docs.formance.com/modules/payments/connectors/psp/increase): Connect an Increase account to Formance Payments to sync US bank accounts, balances, payments, and counterparties across ACH, wire, and check rails, with full outbound initiation and webhook ingest. - [Mangopay](https://docs.formance.com/modules/payments/connectors/psp/mangopay): Connect a Mangopay account to Formance Payments to sync e-wallets, balances, payouts, transfers, and beneficiaries, with full outbound initiation and webhook ingest. - [Modulr](https://docs.formance.com/modules/payments/connectors/psp/modulr): Connect a Modulr account to Formance Payments to sync UK/EU bank accounts, balances, and payments across Faster Payments and SEPA, with outbound transfer and payout initiation. - [Moneycorp](https://docs.formance.com/modules/payments/connectors/psp/moneycorp): Connect a Moneycorp account to Formance Payments to sync multi-currency accounts, balances, beneficiaries, and transactions, with outbound transfer and payout initiation. - [Bitstamp](https://docs.formance.com/modules/payments/connectors/exchange/bitstamp): Connect a Bitstamp account to Formance Payments to sync wallets, balances, payments, orders, and conversions. - [Qonto](https://docs.formance.com/modules/payments/connectors/psp/qonto): Connect a Qonto organization to Formance Payments to sync read-only business accounts, balances, beneficiaries, and transactions. - [Stripe](https://docs.formance.com/modules/payments/connectors/psp/stripe): Connect a Stripe account to Formance Payments to sync balances, payments, payouts, and external accounts, and to initiate transfers and payouts from the platform. - [Kraken Pro](https://docs.formance.com/modules/payments/connectors/exchange/krakenpro): Connect a Kraken Pro account to Formance Payments to sync wallets, balances, payments, orders, and conversions. - [Wise](https://docs.formance.com/modules/payments/connectors/psp/wise): Connect a Wise Business profile to Formance Payments to sync multi-currency balances, recipients, and transactions, with outbound transfer and payout initiation across the Wise rail network. - [Wallets](https://docs.formance.com/modules/wallets) - [Prerequisites](https://docs.formance.com/modules/wallets/prerequisites) - [Basic wallet operations](https://docs.formance.com/modules/wallets/basic-operations) - [Managing holds](https://docs.formance.com/modules/wallets/managing-holds) - [Creating Wallets](https://docs.formance.com/modules/wallets/creating) - [Adding funds](https://docs.formance.com/modules/wallets/adding-funds) - [Spending funds](https://docs.formance.com/modules/wallets/spending-funds) - [Holding and confirming](https://docs.formance.com/modules/wallets/holding-confirm) - [Flows](https://docs.formance.com/modules/flows) - [Workflows definition](https://docs.formance.com/modules/flows/definition) - [Workflows execution](https://docs.formance.com/modules/flows/execution) - [Triggers](https://docs.formance.com/modules/flows/triggers) - [Banking Bridge](https://docs.formance.com/modules/payments/connectors/psp/bankingbridge): Connect a Banking Bridge workspace to Formance Payments to sync accounts, balances, and payments with full PSP-specific reference enrichment under com.formance.connectors.bankingbridge.* - [Send Statement](https://docs.formance.com/modules/flows/stages/send) - [Waiting for events](https://docs.formance.com/modules/flows/stages/wait-event) - [Waiting for a delay](https://docs.formance.com/modules/flows/stages/delay) - [Ledger to Ledger](https://docs.formance.com/modules/flows/examples/ledger-to-ledger) - [Payment to Wallet](https://docs.formance.com/modules/flows/examples/payment-to-wallet) - [Routable](https://docs.formance.com/modules/payments/connectors/psp/routable): Connect a Routable workspace to Formance Payments to sync settings accounts, companies, payables and receivables, and initiate ACH / wire / check / international payouts and transfers. - [Ledger to Stripe Payout](https://docs.formance.com/modules/flows/examples/ledger-to-payout) - [Reconciliation](https://docs.formance.com/modules/reconciliation) - [Concepts](https://docs.formance.com/modules/reconciliation/concepts) - [Getting Started](https://docs.formance.com/modules/reconciliation/getting-started) - [Control Templates](https://docs.formance.com/modules/reconciliation/controls): Choose and configure the financial control that Reconciliation evaluates. - [Webhooks](https://docs.formance.com/modules/webhooks) - [Alerts and Evidence](https://docs.formance.com/modules/reconciliation/alerts): Investigate discrepancies and close them with an auditable decision. - [Receiving Webhooks](https://docs.formance.com/modules/webhooks/receiving) - [Delivery Lifecycle and Guarantees](https://docs.formance.com/modules/webhooks/deliveries) - [Event Reference](https://docs.formance.com/modules/webhooks/events) ## Platform - [Architecture](https://docs.formance.com/manage/architecture) - [Single sign-on (SSO) for organizations](https://docs.formance.com/manage/identity/sso) - [Access Control](https://docs.formance.com/manage/identity/rbac) - [Invite users](https://docs.formance.com/manage/identity/inviting-team) - [Audit Logs](https://docs.formance.com/manage/identity/audit-logs) - [Event Streaming](https://docs.formance.com/manage/events/streaming) - [SDKs](https://docs.formance.com/manage/sdks) - [Release Policy](https://docs.formance.com/release-policy) - [MCP](https://docs.formance.com/mcp) ## Reference - [Membership API](https://docs.formance.com/membership-api) - [Stack API Reference](https://docs.formance.com/stack-api-reference) --- ## Build with Formance Source: https://docs.formance.com/

Explore the modules

--- ## Stack v3.2 Source: https://docs.formance.com/changelog/v3.2 **March 16, 2026** | Component | Version | |----------------|---------| | ledger | v2.4 | | payments | v3.2 | | wallets | v2.1 | | orchestration | v2.1 | | reconciliation | v2.1 | ## Ledger ### Ledger Schema You can now define a [Ledger Schema](/modules/ledger/working-with/ledger-schema) to validate account addresses, enforce naming conventions, and auto-assign metadata at account creation. Schemas also support reusable transaction templates for common patterns. Every transaction records which schema version validated it for auditability. ### Bucket deletion and restoration You can now [delete buckets](/modules/ledger/working-with/data-isolation-buckets#deleting-and-restoring-buckets) via the API. Deleted buckets are preserved for a configurable retention period (default 30 days) and can be restored before expiry. ### `$in` filter operator A new [`$in` operator](/modules/ledger/working-with/filtering-queries#using-the-in-operator) matches resources against a list of values in a single filter, replacing verbose `$or` + `$match` chains. ### Log type filtering You can now [filter logs](/modules/ledger/working-with/filtering-queries#logs) by type (`SET_METADATA`, `NEW_TRANSACTION`, `REVERTED_TRANSACTION`, `DELETE_METADATA`). ### `reverted_at` filtering Filter transactions by revert timestamp using the `reverted_at` field with `$lt`, `$lte`, `$gt`, `$gte`, `$match`, or `$exists` operators. ### Idempotency-Hit response header A new `Idempotency-Hit` header in API responses indicates when a request matched an existing [idempotency](/modules/ledger/working-with/idempotency) key, distinguishing cache hits from new operations. ### Experimental Numscript interpreter The new standalone [Numscript](https://github.com/formancehq/numscript) interpreter is now available in experimental mode, offering improved parsing and execution. ### Metadata on bulk reverts You can now attach custom metadata when reverting transactions via the [bulk endpoint](/modules/ledger/working-with/bulk-processing). ### Logical replication support Added replica identity on tables without primary keys, enabling PostgreSQL logical replication for CDC pipelines. ### Exporter config updates Exporter configurations can now be updated without recreating them, simplifying maintenance of [streaming pipelines](/modules/ledger/working-with/data-streaming). ### Observability improvements - New request-level API metrics for monitoring - OpenTelemetry histogram units and metric names updated to match semantic conventions ### Performance and bug fixes - Fixed slow metadata updates caused by a missing index - Point-in-time now respected in `/volumes` account metadata filters - Query filters now use `big.Int` for large numeric values - Fixed `startTime`/`endTime` database column mapping for time range parameters - Faster migrations when upgrading from earlier versions - Fixed `$or` handling of multiple filter values for the same field - Fixed interaction between Numscript `save` and overdraft, plus world variable error - Relaxed asset regex to support a wider range of characters (e.g., hyphens) - Address filtering performance improved by pushing filters into LATERAL joins for GIN index usage ## Payments ### New connectors Eight new connectors are available: - **[Column](https://column.com/)** — Banking as a Service platform for businesses - **[Increase](https://increase.com/)** — Modern banking API for developers and businesses - **[Qonto](https://qonto.com/)** — Business banking and financial management platform - **[Fireblocks](https://www.fireblocks.com/)** — Enterprise-grade digital asset custody, transfer, and settlement platform - **[Coinbase Prime](https://prime.coinbase.com/)** — Institutional crypto custody and trading platform by Coinbase - **[Tink](https://tink.com/)** — Pan-European banking connectivity with enriched transaction and payment data - **[Powens](https://www.powens.com/)** — Open banking access to European financial data across banks, savings, and more - **[Plaid](https://plaid.com/)** — Secure access to U.S. bank accounts for data, identity, and payment verification The Tink, Powens, and Plaid connectors support new [open banking](/modules/payments/connectors/open-banking/getting-started) workflows: - Create a PSU (Payment Service User) to represent an end-user and forward them to the provider - Retrieve a URL to authenticate to the chosen bank - Renew a connection ### Dynamic pools Pools can now be defined with dynamic membership based on queries, in addition to static account lists. Balance computation is now performed inside PostgreSQL for better performance. ### Generic connector payout capability The generic connector now supports payout operations, expanding its use for custom payment integrations. ### Stripe `balance.available` webhooks The Stripe connector now processes `balance.available` webhook events for real-time balance tracking. ### Polling changes Page size is no longer user-configurable — a fixed value is now defined per provider for optimal data ingestion speed. Minimum and default polling periods have been adjusted to ensure continuous and reliable service. These values can be overridden via environment variables. ### Performance and bug fixes - Reduced Temporal overhead by removing unnecessary sub-workflows and optimizing event processing - Fixed Stripe handling of unsupported adjustment types and refund-without-source scenarios - Connector updates now propagate correctly across multiple pods - Fixed connector uninstall timeouts via batched deletion - Server/worker now starts even when a plugin is invalid - Fixed hyphen handling in asset identifiers --- ## Stack Changelog Source: https://docs.formance.com/changelog Release notes for Formance Stack v3.x. Each entry lists the component versions pinned in the release, the features it introduces, and the bug fixes it carries. Ledger Schema, bucket deletion, new Numscript interpreter, 8 new payment connectors, dynamic pools Process separation, async log hashing, log exporter system, JSON bulk streams, pagination improvements Stateless ledger, zero-downtime upgrades, new Numscript interpreter, enhanced bulk endpoint, Payments V3 API --- ## Stack v3.1 Source: https://docs.formance.com/changelog/v3.1 **September 30, 2025** | Component | Version | |----------------|---------| | ledger | v2.3 | | payments | v3.0 | | wallets | v2.1 | | orchestration | v2.1 | | reconciliation | v2.1 | ## Ledger ### Ledger Service Process Separation The ledger service has been split into two separate processes for improved performance. The existing `ledger serve` command now handles only HTTP requests, while a new `ledger worker` command manages all stateful operations like block hashing and log exports. ### First Usage Filtering Added a new `first-usage` filter to the `/volumes` endpoint, allowing you to filter volumes based on when they were first used. ### Optional Asynchronous Log Hashing Added configurable async hash computation to improve ledger throughput. You can now choose between synchronous hashing (maximum tamper-proof security) and asynchronous block-based hashing (better performance with reduced attack window). This addresses the sequential bottleneck in hash chain computation while maintaining audit trail integrity. ### JSON transactions support for bulk streams Bulk streams now support JSON transactions in addition to the existing Numscript format. The endpoint now accepts `application/json` content type for bulk operations. ### Pagination improvements Introduced a new `sort` query parameter to multiple list endpoints in the Ledger, allowing clients to specify sorting by field and order. It also adds support for paginating and sorting on additional fields, such as `first_usage` for accounts and `date`/`id` for logs and transactions. The account schema is updated to expose a new `firstUsage` timestamp field. ### Additional metadata support for transaction reverts The `POST /api/ledger/v2/{ledgerID}/transactions/{transactionID}/revert` endpoint now accepts optional user-provided metadata in the request. This allows for enhanced traceability and custom business logic while preserving the automatically-added `com.formance.spec/state/reverts` metadata that identifies the original transaction being reverted. ### Big Integer String Serialization Added a new rendering layer that allows big integers to be serialized as strings in API responses when requested via an HTTP header, providing better compatibility with clients that cannot handle large numeric values. ### Log exporting system for real-time streaming Introduced a comprehensive log exporter system for real-time streaming of ledger transaction logs to external systems. The system includes configurable exporters (Elasticsearch, HTTP, ClickHouse, stdout), pipelines that connect ledgers to exporters with reliable state tracking, and APIs for pipeline management via `/v2/{ledger}/pipelines` and direct export via `/v2/{ledger}/logs/export`. Features include automatic retry logic, configurable batching, and persistent state to prevent data loss. ### Removed experimental features for data isolation Experimental data isolation features `INDEX_ADDRESS_SEGMENTS` and `INDEX_TRANSACTION_ACCOUNTS` are no longer supported and have been removed. --- ## Stack v3.0 Source: https://docs.formance.com/changelog/v3.0 **February 10, 2025** | Component | Version | |----------------|---------| | ledger | v2.2 | | payments | v3.0 | | wallets | v2.1 | | orchestration | v2.1 | | reconciliation | v2.1 | ## Ledger ### Stateless Ledger Service The Ledger service is now fully stateless and can operate with multiple replicas without requiring nodes synchronization. ### Zero-Downtime Upgrades Upgrades between versions now occur with zero downtime, ensuring a seamless transition and uninterrupted service. ### New Numscript Interpreter The new numscript interpreter is now available in experimental mode. Numscript used to be bundled in the ledger, but is now a separate, standalone library: [github.com/formancehq/numscript](https://github.com/formancehq/numscript). ### Configurable Feature Disabling at the Bucket Level You can now disable specific features on a per-ledger basis at the bucket level, allowing for better optimization based on your usage patterns. ### Enhanced Write Performance Significant improvements have been made to write operations, resulting in a higher throughput of transactions per second. ### Improved Read Performance Most endpoints have been optimized for faster read operations, leading to a more responsive system overall. ### Enhanced `_bulk` Endpoint The `_bulk` endpoint now supports streaming, parallel processing, and atomic execution, providing greater flexibility and control over bulk processing, optimizing for performance, consistency, and reliability. For details, refer to [Bulk processing](/modules/ledger/working-with/bulk-processing). ## Payments ### New dependency: Temporal Temporal is now required to deploy the Payments service. ### New V3 API improvements **New task endpoint** Added an endpoint to fetch a task, which represents a background user operation. Once completed, the task will have a status of either "failed" or "successful." If successful, the ID of any created object can be retrieved from the task. For connector uninstallation, payment initiation, and bank account creation, HTTP code 202 and a taskID are returned. The operations run in the background, and you can monitor their status and retrieve relevant IDs (e.g., payment or account ID) upon success by polling the task. --- ## Core Concepts Source: https://docs.formance.com/cloud/banking-bridge/core-concepts ## ISO 20022: the global standard Since 2025, [ISO 20022](https://www.iso20022.org/) is the global standard for financial messaging — every major payment system, from SWIFT to domestic clearing, has migrated to it. Banking Bridge speaks ISO 20022 natively, so payout instructions and account statements move between you and your bank in a common, structured format. ## PAIN.001 PAIN.001 (Customer Credit Transfer Initiation) is the message you send to initiate an outbound transfer. It's the format behind SEPA Credit Transfer in Europe. ## PAIN.002 PAIN.002 (Customer Payment Status Report) is the message your bank sends back in response to a PAIN.001 submission, reporting the status of each payment in the batch. Banking Bridge can analyze it alongside CAMT.053 so you can tell which of your payments actually cleared, rather than trying to sift through statements manually. ## PAIN.008 PAIN.008 (Customer Direct Debit Initiation) is the message you send to collect a payment via SEPA Direct Debit — pulling funds from a payer's account under a prior mandate. ## CAMT.053 CAMT.053 (Bank to Customer Statement) is the account statement message your bank sends back: daily balances and every booked transaction. It isn't tied to a single payment scheme — any bank that has adopted ISO 20022 reporting can issue one, which is why Banking Bridge uses it as the universal format to reconcile activity across every connection. ## CAMT.052 CAMT.052 (Bank to Customer Account Report) is the intraday counterpart to CAMT.053 — your bank issues it multiple times a day rather than once at close of business, reporting the transactions and balance seen so far that day. It's a report, not a statement: intraday entries can still be reversed before the end-of-day CAMT.053 finalizes them. Banking Bridge can use CAMT.052 to give a running view of account activity between statements, so you're not waiting until the next day's close to see funds move. ## EBICS EBICS (Electronic Banking Internet Communication Standard) is a certificate-based protocol for exchanging files between corporates and banks over HTTPS. Banks in Germany, France, Austria, and Switzerland offer EBICS access, and Banking Bridge establishes and maintains that connection on your behalf — handling onboarding and certificate management so you don't have to. Banking Bridge is currently in closed beta. Reach out to us for more information. --- ## Banking Bridge Source: https://docs.formance.com/cloud/banking-bridge Bridge the gap between your bank account and your accounting processes. Issue payout instructions directly to your bank cutting through the complexity of implementing and maintaining tricky banking protocols yourself. Monitor all the money movements on your bank account and reconcile your ledger with automated statement reporting accessible via a simple REST API. Crash course into the world of banking. Start here for a glossary of terms. Automate bulk payouts from your bank account with a simple REST API. Make bank statements a native part of your reconciliation process. Automate the bank connection over EBICS in Germany, France, Austria, and Switzerland. ## Why Banking Bridge Bank connectivity is notoriously hard to build in-house: obtuse bank-specific protocols used nowhere else in the industry, XML message generation and parsing, bespoke per-country quirks that make international coverage complicated, and archaic file-based reconciliation processes that fall way out of the norm of what modern tech companies are used to dealing with. Banking Bridge is a fully managed SaaS offering that takes all of that off your plate, so your team integrates against one simple REST API. No need for your team to learn to "speak bank" or deal with bank files at all. Banking Bridge is a SaaS offering: Formance operates the infrastructure, certificates, and bank relationships so you don't have to. In 2025 the global banking community converged on ISO 20022. Being able to seamlessly handle these files is key to integrating with banks. Talk to one or more banks through a single REST API. No need for your systems to handle bank file formats at all. Banking Bridge is currently in closed beta. Reach out to us for more information. --- ## Initiate Bank Transfers in Bulk Source: https://docs.formance.com/cloud/banking-bridge/payment-initiation-instructions ## Bulk payouts; one API call Payout instructions are generated as **PAIN.001** (Payment Initiation) messages, the ISO 20022 standard for initiating credit transfers. A single bulk API call accepts a batch of transfers, and Banking Bridge takes care of PAIN.001 generation and delivery to the bank — no XML generation to build yourself. ## Use cases Bulk payment initiation fits any flow where you need to pay many recipients from a single funding account: - **Employee salary payouts** — run payroll as a single batch instead of one transfer per employee. - **Supplier and vendor payments** — settle invoices across your supplier base in one submission. - **Marketplace and gig payouts** — pay out sellers, drivers, or freelancers in bulk on your payout schedule. ## Reconciling payment initiation requests Banks often lump bulk payments together in their reporting, and the data that comes back can hide the details you'd normally use to identify a payment (like the creditor name) behind a single bulk reference. That makes it hard to tell which of the individual payments in your batch actually went through. Banking Bridge can analyze both **PAIN.002** (the bank's payment status report) and **CAMT.053** (the account statement) to help you track which of your payments actually cleared, and match each one back to the money movement it produced on your bank account. Banking Bridge is currently in closed beta. Reach out to us for more information. --- ## Never Lose Track of Your Bank Account Source: https://docs.formance.com/cloud/banking-bridge/account-statement-ingestion ## Bank statements, structured Banking Bridge ingests **CAMT.053** (Bank to Customer Statement) files from your bank and turns them into structured, queryable data — no parsing bank statement files yourself. ## What you get - **Daily reporting** — statements are ingested as your bank issues them, so your reconciliation process runs on the bank's own daily cadence instead of a manual pull. - **End of day balances** — every statement carries the closing balance for the account, so you always know your position at close of business. - **Detailed transaction reporting** — every booked transaction comes through exactly the way it looks to your bank: booking date, value date, references, and counterparty details intact. Banking Bridge is currently in closed beta. Reach out to us for more information. --- ## Connect Directly to Your Bank Source: https://docs.formance.com/cloud/banking-bridge/direct-to-bank-transfer ## Bank connectivity is hard Banks are notorious for relying on legacy, certificate-heavy connectivity protocols wrapped in strict regulatory requirements. Onboarding a single bank connection can mean weeks of paperwork, dedicated infrastructure, and protocol-specific integration work — a cost that repeats for every additional bank and country you add. ## As many connections as you need; one API Banking Bridge abstracts away the technicalities of bank file exchange. Once a connection is established, your applications only need OAuth2 client credentials to reap the benefits of a bank integration — no certificates, file formats, or protocol quirks for your systems to manage. ## Built on EBICS For banks operating in Germany, France, Austria, and Switzerland, Banking Bridge sets up an **EBICS** (Electronic Banking Internet Communication Standard) pipeline: a certificate-based, direct connection to the bank. EBICS is the mandated bank-connectivity protocol in Germany and France, and the de facto standard in Austria and Switzerland. Banking Bridge handles the EBICS onboarding and certificate lifecycle on your behalf, so it disappears behind a single OAuth2-secured API. ### Supported protocol versions - **EBICS 3.0** (H005) — the default protocol version used for new connections. - **EBICS 2.5** (H004) — still widely deployed by banks that haven't completed their 3.0 migration. - **EBICS 2.4** (H003) — supported on request, for banks that haven't upgraded past the legacy protocol version. ## Your bank doesn't speak EBICS? EBICS is where Banking Bridge started but the platform is expanding to work with open banking protocols too. If your bank or use case doesn't fit EBICS, talk to us about what you need; we're actively building out coverage beyond it. Banking Bridge is currently in closed beta. Reach out to us for more information. --- ## Omnibus Account Management Source: https://docs.formance.com/examples/advanced/omnibus This example shows how to implement omnibus account patterns in Formance using a declarative ledger schema. An omnibus account is a pooled account held at a financial institution that aggregates assets belonging to multiple end-users — the assets represent **liabilities** to your clients. **Common use cases:** - **Banking services** holding pooled client funds in a single settlement account - **Financial markets** managing custodial accounts on behalf of investors - **Crypto platforms** with fiat reserve management across banking partners This is an illustrative example. Adapt the schema to your specific business requirements, regulatory obligations, and financial practices. ## The Complete Schema This is the full ledger schema for an omnibus account system. The sections below explain each part. ## Chart of Accounts The `chart` section defines three account groups using standard correspondent banking terminology. **Nostro** ("ours") accounts represent assets you hold at partner institutions. **Vostro** ("yours") accounts represent liabilities — funds clients hold with you. ### Banks (Nostro Accounts) These are **normal debit accounts** — they represent your assets held at partner banks. The `$bank_id` is typically an IBAN (`FR7630004028379876543210943`) or routing:account format (`021000089:123456789`). The `payout` sub-accounts isolate each withdrawal as a separate staging area, so you can track the lifecycle of each payout independently. ### Clients (Vostro Accounts) **Normal credit accounts** — they represent your liabilities to clients. The credit balance shows funds you owe them. ### Platform **Mixed nature accounts** — operational accounts for suspense handling, revenue, and costs. Suspense accounts are normal debit (assets awaiting attribution); revenue and cost accounts follow standard income statement conventions. ## Transaction Patterns ### Client Deposit The `CLIENT_DEPOSIT` transaction records an identified deposit. The `allowing unbounded overdraft` clause on the bank account permits it to go negative — this accommodates the common case where ledger entries are recorded before bank statement reconciliation. This pattern works for any currency. Pass `EUR/2`, `USD/2`, or any asset in [Universal Monetary Notation](/modules/numscript/monetary-notation). ### Unidentified Deposit When funds arrive but you can't identify the client (missing reference, intermediary payment, etc.), `UNIDENTIFIED_DEPOSIT` parks the funds in a suspense account. You cannot refuse incoming funds to an omnibus account — always book immediately. Once the client is identified, `SUSPENSE_RESOLUTION` moves the funds from suspense to the correct client account. Monitor your suspense accounts closely. Funds should not remain unresolved for extended periods — most regulatory frameworks require timely resolution. ### Client Withdrawal (Payout) Payouts are a two-step process: `PAYOUT_RESERVE` moves funds from the client account to a payout staging account tied to a specific reference. This ensures the client can't spend funds that are being withdrawn. Once the bank confirms the transfer, `PAYOUT_SETTLEMENT` moves funds from the staging account to the bank's main account, completing the cycle. If a payout fails, you reverse the reservation by sending from the staging account back to the client. ## Why Not Use `@world`? The `@world` account is Formance's infinite source/sink. While it simplifies examples, **omnibus accounting requires explicit tracking** of where funds actually are: - **Bank accounts go negative** (overdraft) to represent "we received funds but haven't reconciled yet" — this is intentional and meaningful - **Client accounts are liabilities** — their balance represents your obligation - **Suspense accounts** enable the "book now, attribute later" pattern that omnibus operations require Using `@world` would obscure these distinctions and make reconciliation impossible. ## Queries The `queries` section defines reusable lookups: - `CLIENT_BALANCE` — get a specific client's position - `PENDING_SUSPENSE` — find all unresolved deposits (operational monitoring) - `INFLIGHT_PAYOUTS` — track reserved but unsettled withdrawals (risk management) These leverage the hierarchical account structure — filtering on `:suspense:payin` matches across all platforms, and `:payout:` matches all staging accounts. --- ## RideShare Tutorial Source: https://docs.formance.com/examples/getting-started/rideshare-tutorial In this tutorial, we will set up and use Formance Ledger for a ride-sharing platform called "RideShare." We will design a Chart of Accounts, create a schema to enforce it, use Numscript to execute transactions, and check account balances. ## The Complete Schema Here is the full ledger schema for the RideShare tutorial. It defines the account hierarchy and all four transaction patterns we'll walk through below. ## Design the Chart of Accounts The [Chart of Accounts](/modules/ledger/advanced/architecting-for-scale#chart-of-accounts) should reflect all account types relevant to your flow of funds and reporting needs. We recommend starting with the key questions you need the ledger to answer. For RideShare: - How much did we pay the drivers? - How much did we collect in service fees? - How much did an individual rider pay? - How much did an individual driver earn? And key events the ledger must support: - Rider books a ride - RideShare confirms the ride - Driver finishes the ride - RideShare completes ride verification - Driver requests earnings withdrawal The schema defines four account groups to support these questions and events: ### World The `@world` account represents the external world — funds entering and exiting the ledger. ### Rider Each rider gets a per-ride payment account (`rider:$rider_id:ride:$ride_id:payment`) that records the estimated charge for that ride. ### Ride Each ride has a `main` account (total charged) and a `fees` account (RideShare's service fee). ### Driver Each driver has per-ride earning accounts and a `main` account that accumulates total earnings across rides. The structured [naming convention](/modules/ledger/core-concepts/accounts#naming-accounts) helps organize accounts into segments, making it easier to manage and query them. ## Create the Schema Now let's create the [schema](/modules/ledger/working-with/ledger-schema) to enforce the Chart of Accounts. First, get your sandbox API endpoint: ```bash fctl stack show ``` Then create the schema using the API. You can copy the YAML from the schema viewer above, or use the JSON equivalent: By default, the ledger runs in `audit` mode—transactions are validated against your schema but allowed through. For strict enforcement, see [enforcement modes](/modules/ledger/working-with/ledger-schema#enforcement-modes). ## Introduce Money into the Ledger When a rider books a ride, we record the estimated payment from `@world` into the rider's ride payment account. Execute this transaction: {"fctl ledger transactions num intro.num"} This adds $20 to the `@rider:xx:ride:yy:payment` account. ## Run Transactions on the Ledger ### Ride Confirmation When RideShare confirms the ride, transfer the payment from the rider's account to the ride's main account. {"fctl ledger transactions num payment.num"} ### Ride Completion After the ride is completed, split the payment between the driver's earnings and RideShare's service fees. {"fctl ledger transactions num payment_split.num"} ### Driver Payout After some days, the driver requests their earnings. Transfer all accumulated earnings back to `@world` to represent funds leaving the ledger. The wildcard `*` sends the entire balance. {"fctl ledger transactions num driver_payment.num"} ## Check Account Balances You can check the balances of the accounts using the [Formance Console](https://portal.formance.cloud). --- ## Introduction Source: https://docs.formance.com/examples/introduction This guide helps you apply Formance Ledger patterns to real-world financial operations. The recipes that follow are illustrative — they show how Numscript can serve as an intent layer where business, finance, and engineering teams agree on financial logic in a clear, executable form. Your own regulatory, legal, and accounting obligations will shape the final implementation; treat these as a starting vocabulary, not a turnkey solution. ## Intent-Based Finance At the heart of the Formance approach is the concept of **Numscript as an intent layer**. This means: ### Separation of Concerns Your business and finance teams identify all parties involved in your operations: customers, banks, payment processors, internal accounts, liability accounts, etc. Each actor is represented by accounts in the ledger with clear naming conventions and metadata. Together, teams enumerate all the financial events that can occur in your business: deposits, withdrawals, authorizations, settlements, refunds, chargebacks, minting, burning, etc. Each event translates to specific accounting movements between accounts. Once actors, accounts, and events are defined, Numscripts encode the business logic. These scripts become the single source of truth that both business and technical teams can read and validate. Numscripts are intentionally readable. Your finance team should be able to review a script and confirm it matches their understanding of the business logic. When events occur in your system, you simply "play" the appropriate Numscript with the correct variables. The ledger handles the rest: validating constraints, recording transactions, maintaining balances. ### Benefits of This Approach - **Business Clarity**: Finance and operations teams can understand and validate the logic without deep technical knowledge. - **Rapid Iteration**: Changes to business logic require updating Numscripts, not redeploying entire systems. - **Consistency**: The same Numscript produces the same results, eliminating implementation drift across services. ## Leveraging Bi-Temporality A critical feature of the Formance Ledger is **[bi-temporality](/modules/ledger/working-with/bi-temporality)**, which means every transaction tracks two timestamps: - **Request Time**: When the transaction was submitted to the ledger (machine clock time) - **Transaction Time**: When the transaction is considered to have occurred (business effective time) ### Why This Matters for Financial Operations In real-world financial operations, you frequently encounter timing mismatches: - **Settlement Delays**: Payment authorizations happen in real-time, but bank settlements occur T+1 to T+3 days later. Your ledger should reflect the true effective dates of these movements, not just when you learned about them. - **Batch Processing**: Bank statements arrive the next day. You need to record yesterday's transactions with their correct effective timestamps. - **Corrections and Adjustments**: When errors are discovered or chargebacks occur, you may need to backdate transactions to reflect when they truly should have occurred. - **Multi-Timezone Operations**: Events happening in different timezones need consistent temporal representation. ### Implementing Bi-Temporality in Your Numscripts When executing Numscripts for real-world events, **always provide the appropriate transaction timestamp**, if reporting for point in time balances of accounts is important: ```json Request with Transaction Time { "script": { "plain": "send [USD/2 100.00] (\n source = @bank:account\n destination = @user:123:main\n)", "vars": { "amount": "100.00", "userId": "123" } }, "timestamp": "2024-10-20T14:30:00Z" // Business effective time, not current time } ``` If you don't specify a timestamp, the ledger uses the current machine time. **Backdating Considerations** When inserting backdated transactions (transaction time in the past), the ledger validates that the entire timeline remains consistent. This means a backdated transaction could be rejected if it would cause account balances to become invalid at any point in the future timeline. Plan your account structures and constraints carefully to accommodate late-arriving information, or consider allowing overdraft. ## Key Capabilities Demonstrated Throughout these recipes, you'll see how the Formance Ledger handles: ### Multi-Asset Operations Accounts are multi-asset by default. The same account structure and Numscripts work across USD, EUR, GBP, stablecoins, or any other asset you define. ```numscript vars { monetary $payment } send $payment ( source = @bank:nostro destination = @user:main ) ``` The same script runs against any currency — call it with `$payment = "USD/2 100"`, `"EUR/2 50"`, `"BTC/8 1000000"`, or any asset you've defined. A `monetary` is the asset code and amount together (e.g. `USD/2 100`), sent bare with `send $payment (…)`. To pass them separately use `asset $a` and `number $n` declarations, then write `send [$a $n] (…)`. ### Rich Metadata Attach business context to accounts and transactions directly within your Numscripts for powerful querying, reporting, and integration. ```numscript Card Authorization with Metadata vars { account $user monetary $amount string $auth_id string $card_scheme } send $amount ( source = { @user:main @user:credit_line } destination = @hold:authorization ) set_tx_meta("payment_method", "card") set_tx_meta("card_scheme", $card_scheme) set_tx_meta("authorization_id", $auth_id) ``` Metadata can be any type: strings, numbers, monetary values, portions, or even account references. See the [Numscript metadata reference](/modules/numscript/reference/metadata) for complete details. ### Constraint Enforcement The ledger automatically enforces business rules: - Prevent overdrafts on customer accounts - Ensure sufficient reserves for liabilities - Validate transaction amounts and parties --- These are **starting points**, not finished solutions. Your business is unique, and your ledger implementation should reflect that uniqueness while maintaining the clarity and auditability that Formance enables. --- ## Payment Card Acceptance Processing Source: https://docs.formance.com/examples/advanced/payment-acceptance This example shows how to implement card payment acceptance in Formance using a declarative ledger schema. Payment acceptance covers the full lifecycle of card transactions — from the moment a customer authorizes a payment, through acquirer settlement, to handling refunds and chargebacks. The schema models authorizations as assets (binding promises to pay) so that funds are available to end-users immediately, before settlement completes. This is an illustrative example. Adapt the schema to your specific acquirer relationships, fee structures, and regulatory requirements. ## The Complete Schema This is the full ledger schema for a card payment acceptance system. The sections below explain each part. ## Chart of Accounts The `chart` section defines four account groups: ### Acquirers **Normal debit accounts** representing your relationship with payment acquirers (Stripe, Adyen, etc.). The balance on an acquirer account reflects **unsettled authorizations** — money promised but not yet received. Acquirer accounts go negative when authorizations are recorded before settlement, which is expected behavior. ### Banks (Nostro Accounts) **Normal debit accounts** representing your assets held at partner banks. These are credited when the acquirer settles funds to your bank account. ### Clients **Normal credit accounts** representing your liabilities to clients. When a card authorization tops up a wallet, the client balance increases immediately. ### Platform **Normal credit accounts** used as operational accounts that track fees, revenue, and chargeback costs. These make the economics of each transaction explicit and auditable. ### Balances & Settlement Timing Card payment acceptance creates a fundamental tension between real-time user experience and asynchronous settlement. Clients see funds credited to their wallets immediately upon authorization, but the acquirer does not actually settle those funds to your bank account until T+1 to T+3. The acquirer's negative balance represents exactly this time gap — the sum of all authorized-but-unsettled transactions. ## Transaction Patterns ### Card Authorization (Gross Top-Up) `CARD_AUTHORIZATION_GROSS_TOPUP` records a card authorization and immediately credits the client's wallet with the full gross amount. The acquirer account goes negative — this is intentional. **Authorization as an asset**: A card authorization is a binding promise from the issuing bank to pay. This schema treats that promise as an asset, allowing you to credit the client immediately rather than waiting days for settlement. The acquirer's negative balance represents unsettled promises. ### Acquirer Settlement `ACQUIRER_SETTLEMENT` records the settlement from the acquirer to your bank account. Acquirers settle the **net amount** (gross minus fees), so this transaction has two movements: 1. The net amount flows from the bank to the acquirer account (reducing the acquirer's negative balance) 2. The fee amount flows from the platform fee account to the acquirer (zeroing out the remainder) After settlement, the acquirer balance for those transactions should return to zero. ### Card Refund `CARD_REFUND` reverses a previous authorization by debiting the client wallet and crediting the acquirer account. The acquirer then processes the refund back to the cardholder's issuing bank. ### Chargeback `CHARGEBACK` handles a dispute initiated by the cardholder through their issuing bank. It debits the client wallet for the disputed amount and separately records the chargeback fee charged by the acquirer. Chargebacks use `allowing unbounded overdraft` on the client account because a chargeback can occur even if the client has already spent the funds. The resulting negative balance represents a debt owed by the client to your platform. ## Why Model Authorizations as Assets? In traditional accounting, you would wait for settlement before recording funds. This schema takes a different approach: - **Immediate availability** — clients can use funds as soon as the authorization succeeds, which is critical for wallet top-ups and marketplace payouts - **Explicit fee tracking** — settlement records the net amount and fees separately, giving you a clear picture of acquirer costs - **Clean reconciliation** — each acquirer account should trend toward zero after settlement; a persistent balance signals missing settlements ### Key Differences from Omnibus Accounts If you have worked through the [omnibus wallet example](/examples/advanced/omnibus), the payment acceptance model will feel similar but differs in key ways: - **Source of funds**: In an omnibus model, funds are already at the bank (deposits received); in payment acceptance, funds are promised by the acquirer but not yet received. - **Negative balances**: Omnibus accounts use suspense accounts for unidentified deposits; payment acceptance expects acquirer accounts to go negative (overdraft) as a normal part of the authorization-before-settlement flow. - **Settlement direction**: Omnibus payouts are typically two-step (reserve then settle); payment acceptance settlement is acquirer-driven — the acquirer pushes net funds to your bank on its own schedule. - **Fee tracking**: Payment acceptance makes fees explicit at every stage — acquirer processing fees on settlement and chargeback fees on disputes — whereas omnibus models typically handle fees separately. ## Queries The `queries` section defines reusable lookups: - `CLIENT_BALANCE` — get a specific client's wallet balance - `ACQUIRER_BALANCE` — get a specific acquirer's unsettled balance (useful for reconciliation) - `UNSETTLED_AUTHORIZATIONS` — all acquirer accounts with outstanding balances (risk monitoring) - `PLATFORM_FEES` — fee volumes across all platforms - `PLATFORM_CHARGEBACK_FEES` — chargeback fee volumes (dispute cost tracking) These leverage the hierarchical account structure — filtering on `acquirers::main` matches all acquirer main accounts, and `platform::fees` matches all platform fee accounts. --- ## Card Issuing & Financial Host Source: https://docs.formance.com/examples/advanced/issuing-financial-host A financial host (or issuer processor) is the system that sits between a card network and a cardholder's funds. It receives real-time authorization requests, manages hold accounts that ring-fence funds during the authorization-to-settlement lifecycle, and ultimately settles against scheme liability accounts. This example models the full card issuing flow in a single Formance ledger schema — authorizations, presentments, refunds, chargebacks, and stand-in processing. This is an illustrative example. Adapt the schema to your specific card program, BIN sponsor requirements, and scheme rules. ## The Complete Schema This is the full ledger schema for a card issuing financial host. The sections below explain each part. ## System Actors A card issuing financial host involves five distinct actors, each representing a different side of every card transaction. Understanding their roles — and how their account balances behave — is essential before reading the ledger schema. - **Cardholders**: The end users whose funds you manage. Their accounts are **normal credit accounts** (liabilities) — the balance represents funds you owe them. - **Banks**: Your banking partners. **Normal debit accounts** — represent actual currency reserves held at partner institutions. - **Schemes**: Card networks (Visa, Mastercard, etc.). **Normal credit accounts** — track settlement liabilities between you and the network. - **Program Manager**: BIN sponsors or third-party card program managers. Tracks reverse liability obligations. - **Platform**: Your operational accounts for fees, revenue, and dispute costs. The Chart of Accounts below shows how these actors map to the ledger's account hierarchy. ## Chart of Accounts The `chart` section defines five account groups that mirror the participants in a card transaction. ### Cardholders Each cardholder has a **main** account (their spendable balance), plus dynamic sub-accounts created per authorization hold and per pending refund. The `main` balance minus total holds gives the **available balance** the cardholder can spend. ### Schemes Scheme accounts track what the network owes you (or you owe the network) across the settlement cycle. The `chargeback` sub-account isolates dispute flows from normal settlement. ### Banks, Program Manager, Platform - **`banks:$bank_id:main`** — **Normal debit accounts.** Nostro accounts representing actual funds at partner banks. - **`program_manager:$pm_id:liability`** — **Normal credit accounts.** Tracks reverse liability obligations to BIN sponsors or program managers. - **`platform:$platform_name`** — **Normal debit accounts.** Operational accounts for fees, revenue, and chargeback fees. ## Transaction Patterns ### Authorization Lifecycle The authorization flow manages the time between a card swipe and final settlement. Note: A declined authorization creates **no ledger posting**. The ledger only records approved transactions. Declined attempts should be logged separately for fraud monitoring and analytics. **`CARD_AUTHORIZATION_APPROVED`** moves funds from the cardholder's main account to a hold account keyed by authorization ID. An optional overdraft limit can be specified per cardholder. **`CARD_AUTHORIZATION_PARTIAL`** uses a wildcard amount (`[$asset *]`) with a `max` clause — it approves only what is available up to the requested amount. This is common at fuel dispensers or split-tender scenarios. **`CARD_AUTHORIZATION_INCREMENTAL`** adds funds to an existing hold account. Used for hotels, car rentals, and other scenarios where the final amount is not known upfront. **`AUTHORIZATION_REVERSAL`** releases a specific amount from a hold back to main. Supports both full and partial reversals. **`HOLD_REVERSAL_WILDCARD`** releases whatever balance remains in a hold account using a wildcard amount. Used after partial presentments, expired authorizations, or cancellations. Authorization requests are latency-critical — card networks typically require responses within 100-200ms. Design your integration to minimize round-trips to the ledger. ### Presentment (Settlement) Presentment transactions settle authorized (or offline) transactions against the scheme. They typically arrive in batch files from the network, hours or days after the original authorization. **`PRESENTMENT`** moves funds from the hold account to the scheme's main liability account. The presentment amount may differ from the authorization amount (e.g., due to currency conversion or partial fulfillment). After presentment, any remaining hold balance should be released via `HOLD_REVERSAL_WILDCARD`. **`PRESENTMENT_WITH_TIP`** handles cases where the settled amount exceeds the authorization (e.g., restaurant tips). It drains the hold for the authorized amount and debits the cardholder's main account for the additional amount — two sends in a single atomic transaction. **`OFFLINE_PRESENTMENT`** processes transactions where the card's chip approved the transaction without an online authorization (e.g., transit, in-flight purchases). Uses `allowing unbounded overdraft` because the issuer is obligated to honor the chip's decision. Offline presentments arrive without a prior authorization. They should be flagged for risk review since the cardholder's balance was not checked at the time of purchase. ### Refunds Refunds follow a two-step authorization-then-posting pattern, mirroring how refund messages flow through card networks. `REFUND_AUTHORIZATION` moves funds from the scheme to a pending refund account. The cardholder cannot spend these funds yet. `REFUND_POSTING` releases the pending funds to the cardholder's main account, making them available. ### Chargebacks Chargebacks involve a multi-step dispute process between cardholder, issuer, and merchant. The schema uses a separate `chargeback` sub-account on schemes to keep dispute flows isolated from normal settlement. **`CHARGEBACK_ACCEPTANCE`** credits the cardholder from the scheme's dedicated chargeback account (with unbounded overdraft, since the scheme owes you). This is the provisional credit to the cardholder while the dispute is in progress. **`CHARGEBACK_CONFIRMATION`** moves funds from the scheme's main settlement account to cover the chargeback account, aligning with the actual settlement deduction from the network. If the merchant wins the dispute, **`SECOND_PRESENTMENT`** reverses the chargeback — debits the cardholder and re-establishes the scheme liability. Uses unbounded overdraft because the reversal is mandatory regardless of the cardholder's current balance. ### Stand-In Processing (STIP) **`STIP_ADVICE`** handles transactions that were approved by the network's stand-in processor when the issuer was unavailable. These are mandatory debits with unbounded overdraft — the transaction was already approved and the cardholder has already received the goods or services. STIP transactions bypass your authorization logic entirely. Monitor these closely and ensure your system can process the resulting advice messages promptly to keep balances accurate. ## Available Balance vs. Actual Balance A cardholder's **actual balance** is the balance on their `main` account. Their **available balance** is the actual balance minus the sum of all active holds. When deciding whether to approve an authorization, use available balance — not actual balance. The schema achieves this naturally: approving an authorization moves funds out of `main` into a `hold` sub-account, so the `main` balance always reflects what is truly available. ## Overdraft Policies The schema supports three overdraft modes depending on the transaction type: - **Bounded overdraft** (`allowing overdraft up to [$asset $overdraft]`) — Used for standard authorizations where you control the credit limit per cardholder. - **Unbounded overdraft** (`allowing unbounded overdraft`) — Used for mandatory debits like offline presentments, STIP advice, and second presentments where the transaction has already been approved. - **No overdraft** — The default. Presentments from hold accounts and refund postings should never need overdraft since funds are already ring-fenced. ## Queries The schema defines reusable queries for operational monitoring and reconciliation. These leverage the hierarchical account structure and metadata to filter across the ledger. **Authorization monitoring:** - **`ALL_HOLDS`** / **`CARDHOLDER_HOLDS`** / **`SPECIFIC_HOLD`** — Inspect authorization holds at various levels of granularity. - **`ACTIVE_HOLDS_FOR_CARD`** — Find active holds for a specific payment instrument (card), filtered by metadata and positive balance. **Settlement and reconciliation:** - **`SCHEME_LIABILITY`** / **`ALL_SCHEME_LIABILITIES`** — Track what each network owes you (or vice versa) for settlement reconciliation. - **`ALL_CARDHOLDER_BALANCES`** — Overview of all cardholder positions. **Disputes and refunds:** - **`PENDING_REFUNDS`** — Find all authorized but unposted refunds across the system. - **`SCHEME_CHARGEBACKS`** / **`ALL_CHARGEBACKS`** — Monitor dispute-related liabilities. **Risk and audit:** - **`OFFLINE_TRANSACTIONS`** — Audit trail of all offline presentments, useful for risk monitoring. --- ## Account Hierarchy Examples Source: https://docs.formance.com/examples/account-hierarchies These examples show how to structure account hierarchies for common fintech use cases. For the principles behind the naming and structure choices, see [Chart of Accounts](/modules/ledger/core-concepts/chart-of-accounts). ## E-commerce platform Key design decisions in this hierarchy: - **Customers** have wallet accounts with a wallet sub-account, keeping balance state isolated per user. - **Merchants** separate earnings (accrued revenue) from settlements (disbursed funds), making payout reconciliation straightforward. - **Orders** have lifecycle accounts (`pending`, `captured`) so each order's state is traceable without relying on metadata. - **Platform** centralizes fees, VAT, and refund pools as named accounts, giving finance a clean query surface for revenue reporting. ## Lending platform Key design decisions in this hierarchy: - **Borrowers** and **lenders** are separated into their own hierarchies, keeping lender capital distinct from borrower liabilities. - **Loan accounts** (`outstanding`, `payments`) track state at the individual loan level; each loan's outstanding balance and payment history are isolated, making arrears management and early repayment calculation straightforward. ## Multi-currency wallet Key design decisions in this hierarchy: - **Per-user, per-currency wallet accounts** (`users:{id}:wallet:USD`, `users:{id}:wallet:EUR`) allow clean balance isolation; an account's USD balance and EUR balance never interfere. - **Exchange liquidity pools** hold pre-funded reserves for funded conversions. Debiting from `exchange:liquidity:USD` and crediting `users:{id}:wallet:USD` keeps BTC or EUR creation strictly bounded by what was pre-funded. - **Exchange fees** are collected in a single platform account, making fee revenue reporting a simple balance query. --- ## Stablecoin On-Ramp & Off-Ramp Operations Source: https://docs.formance.com/examples/advanced/stablecoin-operations This example shows how to implement stablecoin on-ramp (fiat-to-crypto) and off-ramp (crypto-to-fiat) operations in Formance using a declarative ledger schema. The system bridges traditional banking with blockchain minting and burning, tracking every step of the conversion lifecycle so that fiat reserves always back circulating stablecoins 1:1. This is an illustrative example. Adapt the schema to your specific business requirements, regulatory obligations, and financial practices. ## Key Concepts - **Payment Authorization as Promise** — A real-time PSP confirmation (card auth, instant payment acknowledgment) is treated as a binding asset, allowing the platform to credit the user immediately before bank settlement completes. - **1:1 Peg** — Every stablecoin in circulation must be backed by exactly one unit of fiat held in reserve. The ledger enforces this invariant across all minting and burning operations. - **Multi-Stage Settlement** — Three concurrent timelines run in parallel: payment authorization (instant), blockchain confirmation (seconds to minutes), and bank settlement (T+1 to T+3). The schema tracks each independently. - **In-Flight Tracking** — Dedicated accounts (`mint_in_flight`, `burn_in_flight`, pending withdrawal reserves) track assets that are between systems, giving precise visibility into what is pending at any moment. - **Operational Costs** — Gas fees and payment processing fees are typically absorbed by the platform and tracked in dedicated expense accounts, keeping client balances clean. ## The Complete Schema This is the full ledger schema for stablecoin on-ramp and off-ramp operations. The sections below explain each part. ## Chart of Accounts The `chart` section defines five account groups: ### PSP (Payment Service Providers) Accounts representing your payment processors (card acquirers, instant payment providers). These are **normal debit accounts** — a debit balance represents a promise of incoming fiat that the PSP owes you. The `$psp_id` segment identifies each provider. ### Banks (Nostro Accounts) Your bank accounts that hold actual fiat reserves. The `withdrawal` sub-accounts isolate each outbound transfer by reference, so you can track the lifecycle of every fiat payout independently. ### Blockchain (On-Chain Supply Tracking) These are **liability accounts** that track the on-chain state of your stablecoin across networks — credit balances represent supply you are responsible for: - **`circulating`** — total supply currently in circulation on a given network - **`mint_in_flight`** — mints that have been submitted but not yet confirmed on-chain - **`burn_in_flight`** — burns that have been submitted but not yet confirmed on-chain Blockchain confirmations are asynchronous and can take seconds to minutes depending on the network. The in-flight accounts let you track this latency window precisely. ### Clients Each client has a `stablecoin` account representing their token balance. These are **normal credit accounts** (liabilities to users) — the credit balance shows how many stablecoins you owe them. ### Platform Operational accounts for the platform itself: - **`pivot:stablecoin_issuance`** — the conversion pivot that bridges fiat and stablecoin asset types (see below) - **`expenses`** — payment processing fees and blockchain gas fees - **`revenue`** — transaction fee collection - **`reserves`** — fiat backing reserves and pending withdrawal staging ## On-Ramp Flow (Fiat to Crypto) The on-ramp converts a user's fiat payment into stablecoins through four steps. The key design principle: **credit the user immediately** (good UX), then settle the blockchain and banking sides asynchronously. `ONRAMP_STEP1_PAYMENT_AUTH_CREDIT` — The user initiates a fiat payment (card, instant payment). Fiat flows from the PSP through the pivot account, which immediately converts it into stablecoins credited to the user's balance. The user sees tokens right away. `ONRAMP_STEP2_MINT_INSTRUCTION` — The platform submits a mint transaction to the blockchain. The stablecoin obligation moves from the pivot to a mint in-flight tracking account while waiting for on-chain confirmation. `ONRAMP_STEP3_MINT_CONFIRMATION` — The blockchain confirms the mint. The in-flight account resolves as the circulating supply increases. Tokens are now officially on-chain. `ONRAMP_STEP4_PSP_SETTLEMENT` — The PSP settles fiat to your bank account, net of processing fees. This resolves the fiat side of the pivot, completing the full backing cycle. Every stablecoin issued must be backed 1:1 by fiat reserves. The four-step on-ramp ensures both the blockchain mint and the bank settlement complete before the cycle is considered closed. ## Off-Ramp Flow (Crypto to Fiat) The off-ramp converts a user's stablecoins back into fiat through three steps: `OFFRAMP_STEP1_BURN_INSTRUCTION` — The user requests a fiat withdrawal. Their stablecoins are debited and moved to a burn in-flight account while the platform submits a burn transaction on-chain. `OFFRAMP_STEP2_BURN_CONFIRMATION` — The blockchain confirms the burn. Tokens are permanently removed from circulation, and the corresponding fiat amount moves from the pivot to pending withdrawal reserves. `OFFRAMP_STEP3_FIAT_WITHDRAWAL` — The platform initiates a bank transfer. Fiat moves from pending withdrawal reserves to an in-flight withdrawal account tied to a specific transfer reference. ## The Pivot Account The `platform:pivot:stablecoin_issuance` account is the central mechanism that bridges two different asset types (fiat and stablecoin) within the ledger. It acts as a conversion point: - On the **fiat side**, it receives funds from the PSP and releases them when burns are confirmed - On the **stablecoin side**, it issues tokens to users and reclaims them when mints are submitted The pivot account's balance should trend toward zero over time. A non-zero balance indicates unsettled conversions — either mints pending confirmation or PSP settlements not yet received. Monitor this account as a key health indicator. ## Queries The `queries` section defines reusable lookups: - **`CLIENT_STABLECOIN_BALANCE`** — get a specific client's token position - **`CIRCULATING_SUPPLY`** — total on-chain supply across all networks - **`INFLIGHT_MINTS`** / **`INFLIGHT_BURNS`** — pending blockchain operations (operational monitoring) - **`PENDING_WITHDRAWALS`** — fiat awaiting transfer to clients - **`INFLIGHT_FIAT_WITHDRAWALS`** — bank transfers in progress - **`PIVOT_BALANCE`** — the pivot account's current position (should trend to zero) These leverage the hierarchical account structure — filtering on `blockchain::circulating` matches across all networks, and `banks::withdrawal:` matches all in-flight payouts. ## Key Differences from Traditional Operations - **Triple asynchrony** — Unlike traditional payments where you manage a single settlement timeline, stablecoin operations juggle three concurrent ones: payment authorization, blockchain confirmation, and bank settlement, each with different latency profiles. - **Asset creation vs. movement** — Minting creates new assets and burning destroys them. This is fundamentally different from traditional transfers that move existing funds between parties. - **24/7 vs. banking hours** — Blockchain networks operate continuously while bank settlements follow business day schedules, meaning the fiat and crypto sides of a transaction may resolve days apart. - **Precision differences** — Fiat currencies use 2 decimal places while stablecoins may use 6 to 18, requiring careful handling of decimal precision in the ledger to avoid rounding mismatches. --- ## Stablecoin Issuer Source: https://docs.formance.com/examples/advanced/stablecoin-issuer This example shows how to model a reserve-backed stablecoin issuer in Formance using a declarative ledger schema. Unlike a pure on-ramp/off-ramp gateway, an issuer holds the fiat reserves itself: every token in circulation is a liability backed by segregated cash at one or more reserve banks. The schema tracks the full mint and redemption lifecycle, interbank reserve rebalancing, and reserve yield — and exposes a single load-bearing query that proves circulating supply equals total backing 1:1. **Common use cases:** - **Regulated stablecoin issuers** holding segregated fiat reserves across multiple banking partners - **E-money and tokenized-deposit programs** that must demonstrate full backing to auditors and regulators - **Treasury operations** that earn and account for yield on idle reserves This is an illustrative example. Adapt the schema to your specific business requirements, regulatory obligations, and financial practices. ## Key Concepts - **1:1 Parity Invariant** — The central guarantee: total circulating token supply must equal total settled fiat reserve, plus backing in motion between banks, minus redemptions that are burned-but-not-yet-paid (their backing is owed out). The `parity_invariant_supply_vs_total_backing` query proves this daily. - **Reserve Accounts** — Each reserve bank holds a `reserve` sub-account of settled fiat backing. This is the real cash that stands behind the tokens. Reserves can move between banks via rebalancing without ever breaking parity. - **In-Transit Accounts** — Dedicated per-operation accounts isolate fiat that is mid-flight: `mints:$mint_id:inTransit` (wired but not settled, and therefore not yet backing), `redemptions:$redemption_id:settling` (backing owed out), and `reserves:rebalance:$rebalance_id:inTransit` (backing moving between banks). Each drains to zero at settlement or return. - **On-Chain Supply as an External Sink** — The per-network `external:networks:$network_id:supply` boundary runs **negative** as tokens are issued into circulation; its absolute value is the on-chain circulating supply. It provides an independent, second measure of supply to cross-check against the holder-side total. - **Two-Phase Mint and Redemption** — Both lifecycles split into an initiate/request step and a settle step, so the ledger reflects bank settlement timing precisely and supports clean returns at either stage. - **Yield Accrual and Sweep** — Interest earned on reserves accrues into a segregated `yield:accrued` sub-account, then sweeps to operational revenue on a schedule, keeping earned-but-unswept yield distinct from booked revenue. - **Redemption Fee** — Redemptions carry a fee (10 bps in this schema) skimmed at request time into a dedicated fee account, leaving the net payout obligation behind. ## The Complete Schema This is the full ledger schema for a reserve-backed stablecoin issuer. The sections below explain each part. ## Chart of Accounts The `chart` section defines four account groups: the issuer's internal `platform` accounts, the `holders` who own tokens, the reserve-bank `counterparties`, and the `external` world boundaries for fiat and on-chain supply. ### Platform These are the issuer's operational accounts: - **`banks:$bank_id:reserve`** — settled fiat backing held at each reserve bank. A **normal debit** account: the debit balance is real cash you hold, and the sum across banks is the settled-backing figure in the parity proof. - **`banks:$bank_id:yield:accrued`** — a segregated sub-account holding interest credited by that bank but not yet swept to revenue. - **`mints:$mint_id:inTransit`** — per-mint staging for fiat that has been wired but not yet settled. Excluded from backing, because the tokens are not yet minted. - **`redemptions:$redemption_id:settling`** and **`redemptions:$redemption_id:payable`** — per-redemption staging: `settling` holds the gross obligation while the bank settles, and `payable` holds the net fiat owed to the holder. - **`reserves:rebalance:$rebalance_id:inTransit`** — per-rebalance staging for reserve cash moving between banks; this is backing in motion and counts toward parity. - **`fees:redemption`** — accrued redemption-fee balance, a revenue account. - **`revenue:yield`** — the running total of reserve yield booked as operational revenue. The `$bank_id`, `$mint_id`, `$redemption_id`, and `$rebalance_id` placeholders are substituted at posting time with the concrete identifier for each bank or operation, so every mint, redemption, and rebalance gets its own isolated lifecycle. ### Holders Each holder's `$holder_id` account carries their circulating token balance. These are **normal credit accounts** — the credit balance is the issuer's liability to that holder. The reserved `.self` account is available for issuer-held positions. The sum across all holders is the platform-wide circulating liability and the holder-side input to the parity proof. ### Counterparties The `banks:$bank_id` accounts represent the reserve banks as external counterparties. They are the source of credited interest in the yield-accrual flow, kept distinct from the `platform:banks` accounts that hold your actual reserves and accrued yield. ### External The boundary accounts where value enters and leaves the ledger: - **`fiat:wires`** — incoming mint wires from holders. - **`fiat:payouts`** — outgoing redemption payouts to holders. - **`networks:$network_id:supply`** — the per-network on-chain boundary. Tokens are issued from here into circulation, so the account runs **negative**; its absolute value is the on-chain circulating supply for that network. The wildcard `external:networks::supply` rolls up the whole fleet. ## Mint Flow Minting is a two-phase process tied to bank settlement: the holder's fiat wire is recorded in transit first, and the token is only issued once the reserve bank confirms settlement. A wire reversed before settlement is cleanly returned with no token ever minted. `MINT_INITIATE` — A holder's fiat wire is acknowledged. The cash is recorded in the per-mint `inTransit` account (sourced from `external:fiat:wires`), but no token is credited yet. This fiat is deliberately excluded from backing until the mint settles. `MINT_SETTLE` — The reserve bank confirms settlement. The in-transit fiat becomes settled `reserve`, and in the same transaction the token is minted to the holder by sourcing it from the on-chain `supply` boundary (driving it further negative). Backing and circulating supply increase together, preserving parity. `MINT_RETURN` — If the wire is reversed before settlement, the in-transit fiat is returned to `external:fiat:wires` and no token is minted. The transaction is tagged as an adjustment referencing the original posting. ## Transfer Flow Holders can move tokens between each other without touching reserves — a peer-to-peer transfer is purely a reallocation of the issuer's circulating liability, so total supply and total backing are both unchanged. `TRANSFER` — Tokens move directly from one holder to another. No reserve, in-transit, or supply account is touched. `TRANSFER_REVERSE` — Reverses a prior transfer, returning the tokens to the original sender. Tagged as an adjustment referencing the original posting. ## Redemption Flow Redemption is the mirror of minting and also two-phase. The holder burns tokens at request time and a fee is skimmed, but the reserve is untouched until the bank actually settles the payout. A settled payout that later fails can be fully unwound. `REDEEM_REQUEST` — The holder burns tokens back to the on-chain `supply` boundary (reducing circulating supply), and the gross fiat obligation is booked: a 10 bps `fee` is split off to `platform:fees:redemption`, and the remaining net amount lands in the per-redemption `payable` account. The reserve is deliberately left untouched until settlement. `REDEEM_SETTLE` — The reserve bank settles the payout. The gross amount drains from the bank's `reserve` into the redemption `settling` account, and the net fiat leaves to `external:fiat:payouts`. Backing and circulating supply fall together. `REDEEM_RETURN` — If a settled payout is returned, the redemption is cancelled end to end: net fiat and fee flow back into the bank's `reserve`, and the burned tokens are re-issued to the holder from the on-chain `supply` boundary. Tagged as an adjustment referencing the original posting. ## Reserve Rebalancing Reserves can be moved between banking partners — to manage concentration risk, fund a payout at the right bank, or optimize yield — without affecting circulating supply. The cash sits in a per-rebalance in-transit account while it moves, where it still counts as backing in motion. `REBALANCE_INITIATE` — Reserve cash leaves one bank's `reserve` into the per-rebalance `inTransit` account, where it is held until the interbank transfer settles. `REBALANCE_SETTLE` — The in-transit cash settles into the destination bank's `reserve`. Total settled-plus-in-motion backing is unchanged throughout. `REBALANCE_RETURN` — A failed interbank transfer returns the in-transit cash to the origin bank's `reserve`. Tagged as an adjustment referencing the original posting. ## Yield Flow Idle reserves earn interest. The schema keeps earned-but-unswept yield segregated from booked revenue, so the reserve balance that backs tokens is never inflated by accrued interest that has not yet been recognized. `YIELD_ACCRUE` — A reserve bank credits interest, which is recorded from the bank `counterparty` into the segregated `platform:banks:$bank_id:yield:accrued` sub-account, tagged with the accrual period. This keeps accrued yield out of the `reserve` account that drives the parity proof. `YIELD_SWEEP` — On a monthly cadence, accrued yield sweeps from `yield:accrued` into `platform:revenue:yield`, recognizing it as operational revenue. ## The Parity Invariant The load-bearing guarantee of the whole schema is the 1:1 parity proof, expressed by `parity_invariant_supply_vs_total_backing` and run daily. Stated in prose: > Total circulating token supply (the sum of all `holders:` balances) must equal total settled fiat reserve (`platform:banks::reserve`) **plus** reserve backing in motion (`platform:reserves:rebalance::inTransit`), **minus** redemptions that are burned-but-not-yet-paid (`platform:redemptions::settling`), whose backing is already owed out. The two-phase design is what makes this hold exactly: - **Mint cash in transit is excluded.** During `MINT_INITIATE` the fiat sits in `mints::inTransit` but no token exists yet, so that cash is not yet backing. Only at `MINT_SETTLE` do reserve and token appear together. - **Burned-but-unpaid redemptions are subtracted.** At `REDEEM_REQUEST` the token is already burned (supply down) but the reserve has not yet drained; the `settling` balance represents backing owed out and is removed from the backing side until `REDEEM_SETTLE` completes. - **Rebalances stay in the count.** Cash in `reserves:rebalance::inTransit` is still backing, just between banks, so it is added back in. - **Accrued yield is excluded from backing.** It lives in a separate `yield:accrued` sub-account, never in `reserve`, so earning interest never overstates what backs the tokens. The `cross_check_holder_supply_vs_network_supply` query provides an independent second proof: the holder-side total must equal the absolute value of the on-chain `external:networks::supply` boundaries. A drift between the two means a mint or burn touched one side without the other. ## Queries The `queries` section defines reusable lookups. The two parity proofs come first, followed by reconciliation, operational dashboards, aging checks, and drill-downs: - **`parity_invariant_supply_vs_total_backing`** — the load-bearing 1:1 proof; holder supply versus total backing, compared daily. - **`cross_check_holder_supply_vs_network_supply`** — independent supply check against the on-chain network boundaries. - **`per_bank_reserve_balance`** — one bank's settled reserve, for reconciliation against that bank's statement (substitute the bank id). - **`total_circulating_supply_holder_side`** — the platform-wide circulating liability summed across all holders. - **`per_holder_circulating_balance`** — one holder's token balance (substitute the holder id). - **`per_network_circulating_supply`** — circulating supply on one network, from its on-chain boundary (absolute value). - **`total_settled_reserve`** — settled reserve summed across both banks; the settled-backing figure in the parity proof. - **`in_flight_backing_dashboard`** — every fiat amount in motion or owed: mints in transit, redemptions settling, and rebalances in transit, at a glance. - **`accrued_yield_awaiting_sweep`** — interest credited but not yet swept, held in the segregated yield-accrued sub-accounts. - **`daily_redemption_fee_revenue`** — the running redemption-fee balance. - **`swept_yield_revenue`** — the running total of reserve yield booked as revenue. - **`daily_redemption_fee_revenue_flow`** — fee revenue earned in a day, read as volume into the fee account. - **`reserve_settlement_throughput`** — fiat settled into and out of one reserve bank over a period (substitute bank id and window). - **`per_holder_token_throughput`** — token in and out of one holder over a period (substitute holder id and window). - **`aging_mints_in_transit`** — per-mint in-transit accounts with a non-zero balance: wired but neither settled nor returned. - **`aging_redemptions_settling`** — per-redemption settling accounts still outstanding: token burned but payout not yet settled. - **`aging_rebalances_in_transit`** — per-rebalance in-transit accounts that have not settled or returned. - **`all_transactions_touching_one_holder`** — every posting that hit a holder, in either direction, for drill-down (substitute the holder id). - **`trace_one_redemption_end_to_end`** — every transaction tagged with a redemption id, across request, settlement, and any return (substitute the redemption id). --- ## Neobank (FBO) Source: https://docs.formance.com/examples/advanced/neobank-fbo This example shows how to model a neobank built on a single FBO ("For Benefit Of") bank account in Formance using a declarative ledger schema. The neobank holds all customer money commingled in one omnibus account at a sponsor bank, while the ledger tracks each customer's slice as a separate liability balance. Every customer movement — deposits, card spend, peer-to-peer transfers, withdrawals, and early-access advances — flows through the same pooled cash position, and a daily FBO reconciliation proves the ledger's view of pooled cash against the sponsor's reported statement. **Common use cases:** - **Neobanks and BaaS programs** operating customer accounts on top of a sponsor-bank FBO - **Earned-wage / early-access products** that front advances against an expected deposit - **Card programs** managing authorization holds, captures, and refunds against pooled cash This is an illustrative example. Adapt the schema to your specific business requirements, regulatory obligations, and financial practices. ## Key Concepts - **FBO Pooling** — All customer funds sit commingled in one bank account (`platform:banks:sponsor:fbo:settled`). The ledger is the system of record for who owns what inside that pool; the bank only ever sees the aggregate. - **Per-Customer Liability Balances** — Each customer has an `available` balance plus subtrees for authorization `holds`, in-flight `withdrawals`, and `advances`. These are **liabilities** — the credit balance is what the platform owes that customer. - **Daily FBO Reconciliation** — The single asset-side account (`platform:banks:sponsor:fbo:settled`) is reconciled every day against the sponsor's reported FBO statement. It runs negative; its absolute value is the ledger's view of settled cash in the pool, and total positive claims against it form an internal solvency check. - **In-Flight Isolation** — Authorization holds, pending ACH withdrawals, and outstanding advances each get their own per-reference account, so funds that are earmarked, settling, or fronted are never confused with spendable balance. - **Platform Float Separation** — The platform's own money (the FBO `buffer` and the corporate operating account) lives outside the `customers` root, giving structural proof that platform funds are distinct from customer funds. ## The Complete Schema This is the full ledger schema for a neobank FBO system. The sections below explain each part. ## Chart of Accounts The `chart` section defines two top-level groups: `customers` (the per-customer liability subtree) and `platform` (banks, revenue, and expense accounts the platform controls). ### Customers **Normal credit accounts** — these represent your liabilities to customers. Each `$customer_id` has an `available` balance (spendable funds), a `holds` subtree (one account per card authorization, by `$auth_id`), a `withdrawals` subtree (one `pending` account per outbound ACH, by `$withdrawal_id`), and an `advances` subtree (one `outstanding` receivable per advance, by `$advance_id`). The advance `outstanding` accounts run negative while open — they are receivables the platform fronts — and drain to zero when the expected deposit settles. ### Platform **Mixed nature accounts** controlled by the platform. The sponsor FBO `settled` account is the asset/nostro position: it represents the pooled cash held at the sponsor bank and runs negative as the ledger books credits to customers ahead of bank settlement. The FBO `buffer` and corporate `operating`/`settled` accounts hold platform float. `revenue:interest` captures FBO interest recognized as platform revenue, and `expense:advanceLoss` absorbs unrecoverable advances. ## Deposit Flow Inbound funds settle into the FBO and are credited to the customer's available balance. The `allowing unbounded overdraft` clause on the FBO settled account lets it go negative — this is intentional, reflecting that the ledger books the customer credit before bank reconciliation. `ACH_DIRECT_DEPOSIT` settles an inbound ACH direct deposit into the FBO and credits the customer's available balance. `INCOMING_WIRE` does the same for an inbound wire — funds settle into the FBO and credit the customer. `ACH_DEPOSIT_RETURN` reverses a settled ACH deposit returned by the originating bank, debiting the customer's available balance back to the FBO. ## Card Flow Card spend follows the authorization-then-capture lifecycle. An authorization places a hold on available balance; the hold is later captured, reversed, or expired. `CARD_AUTH` places a hold on the customer's available balance, moving funds into a per-authorization `holds:$auth_id` account. `CARD_CAPTURE` settles against an authorization: the captured amount leaves the FBO and the remainder of the hold is restored to available. `CARD_AUTH_REVERSE` is a merchant-initiated reversal of an authorization — the hold returns to available. `CARD_AUTH_EXPIRE` releases an unused authorization: the remaining hold balance returns to available. A merchant refund of a prior capture is credited back through `CARD_REFUND`, restoring funds from the FBO to the customer. ## Peer-to-Peer Transfer `P2P_TRANSFER` moves funds instantly between two customers' available balances. Because both customers share the same FBO pool, the transfer is purely in-ledger and never touches the bank. ## Withdrawal Flow Outbound ACH withdrawals are a two-step process so that funds being withdrawn can't be spent while in flight. `ACH_WITHDRAWAL_RESERVE` earmarks funds from the customer's available balance into a per-withdrawal `withdrawals:$withdrawal_id:pending` account. `ACH_WITHDRAWAL_SETTLE` completes the cycle once the outbound ACH settles — pending funds leave the FBO. If the outbound ACH fails or is returned, `ACH_WITHDRAWAL_RETURN` restores the pending funds to the customer's available balance. ## Advance Flow Early-access advances let the platform front money against an expected deposit, booked as a receivable on a per-advance `advances:$advance_id:outstanding` account. `ADVANCE_ORIGINATION` fronts the advance: the outstanding receivable goes negative and the customer's available balance is credited. `ADVANCE_SETTLEMENT` applies the expected deposit when it settles, repaying the advance first and crediting any remainder to available. `ADVANCE_WRITEOFF` recognizes an unrecoverable advance as a loss against `platform:expense:advanceLoss` and funds the FBO from the corporate bank. ## Platform Operations Two recurring platform movements run on a machine interpreter. `FBO_INTEREST_SWEEP` recognizes monthly sponsor interest on the FBO as platform revenue. `OPERATING_MOVEMENT_TO_CORPORATE` and `OPERATING_MOVEMENT_TO_FBO` sweep platform float between the FBO buffer and the corporate bank, keeping operating cash topped up in either direction. ## The Daily FBO Reconciliation The FBO settled account is the heart of the model. Because the ledger books customer credits before the bank confirms settlement, `platform:banks:sponsor:fbo:settled` runs negative — its absolute value is the ledger's view of settled cash in the pool. Reconciliation compares two figures every day: - The **asset side** — the magnitude of the FBO settled account, matched against the sponsor's reported FBO statement balance. - The **claims side** — the sum of every positive claim against that cash: all customer available, held, and pending-withdrawal balances, plus the platform's FBO float and accrued interest. The claims against the FBO, plus outstanding advances, should equal the FBO backing. A mismatch signals an unreconciled movement or a missing posting — making the FBO settled account a key health indicator to monitor daily. ## Queries The `queries` section defines reusable lookups for reconciliation, per-customer reads, and operational monitoring: **Reconciliation and solvency** - `daily_fbo_reconciliation_invariant_1` — the FBO settled account to reconcile against the sponsor statement - `total_claims_against_the_fbo_internal_solvency_check` — every positive claim against FBO cash, for the internal solvency check **Per-customer reads** - `customer_available_balance` — one customer's spendable balance right now - `customer_total_holds` — all current authorization holds for one customer - `customer_full_position` — every account in one customer's subtree in a single read **Platform liability and float** - `total_customer_spendable_and_held_liability` — the headline platform liability across all customers - `total_platform_float` — the platform's own money, provably separate from customer funds **Advances** - `outstanding_advances` — every advance still carrying a balance - `aging_advances` — advances still open after the expected-deposit window - `total_advance_exposure` — the platform's total early-access advance exposure **In-flight and volumes** - `pending_withdrawals_in_flight` — outbound ACH earmarked but not yet settled or returned - `card_capture_volume` — capture cash leaving the FBO over a window - `p2p_transfer_volume` — peer-to-peer volume through customer available balances over a window - `monthly_interest_revenue` — FBO interest recognized to revenue over the month - `advance_loss_to_date` — realized loss on unrecovered advances **Audit** - `customer_transaction_audit` — every transaction touching a customer's subtree, for drill-down - `authorization_lifecycle_audit` — every transaction for one card authorization These leverage the hierarchical account structure — filtering on `customers::available` matches every customer's spendable balance, and `customers::advances::outstanding` matches every advance receivable. --- ## Crypto Custody Source: https://docs.formance.com/examples/advanced/crypto-custody This example shows how to implement multi-asset crypto custody in Formance using a declarative ledger schema. Each customer holds entitlements across several assets — USD held at FBO banks alongside crypto held in pooled custody — while the platform reconciles those entitlements against the wallet and on-chain balances that back them. Every deposit, withdrawal, and trade is tracked through its confirmation lifecycle so that what customers are owed never drifts from what the platform actually holds. **Common use cases:** - **Custodial exchanges** holding customer fiat and crypto across multiple assets and networks - **Brokerage platforms** offering buy/sell against an OTC desk with a spread - **Wallet providers** managing pooled custody with on-chain reconciliation This is an illustrative example. Adapt the schema to your specific business requirements, regulatory obligations, and financial practices. ## Key Concepts - **Multi-Asset Entitlements** — Each customer holds balances across several assets (USD, BTC, ETH, multiple USDC variants) under a single `customers:$customer_id` namespace. Cash and crypto live in separate sub-trees, each split between funds that are spendable and funds still in flight. - **Omnibus Custody vs. Per-Customer Entitlements** — Customer crypto balances are *entitlements*: a record of what the platform owes each customer. The actual coins sit pooled in `platform:custody` accounts — per-network hot wallets and per-custodian omnibus wallets — shared across all customers. The chart deliberately separates the two so a customer's spendable balance is never confused with where the asset physically lives. - **On-Chain Reconciliation** — Because entitlements and custody are tracked separately, the platform can continuously reconcile the sum of customer crypto entitlements against the sum of hot-wallet and omnibus backing, per asset. A divergence means an unbooked deposit, an untracked withdrawal, or a custody discrepancy. - **Deposit & Withdrawal Confirmation Latency** — On-chain deposits are credited as `confirming` the moment they are observed, then promoted to `available` once they reach confirmation depth. Fiat deposits move through a `pending` cash window while the ACH or wire clears. Withdrawals are reserved into per-withdrawal accounts before they settle to chain. - **In-Flight Tracking** — Dedicated accounts isolate every asset that is between systems: `cash:pending`, `crypto:confirming`, per-withdrawal `pending` reserves, in-transit FBO balances, and per-trade conversion accounts. Each gives precise visibility into what is committed but not yet final. ## The Complete Schema This is the full ledger schema for a multi-asset crypto custody system. The sections below explain each part. ## Chart of Accounts The `chart` section defines the customer-facing entitlements, the platform-side custody and operational accounts, and the external endpoints money flows to and from. ### Customers **Normal credit accounts** — they represent the platform's liabilities to customers. Each customer namespace carries a `cash` sub-tree (`pending` while a fiat deposit clears, `available` once spendable) and a `crypto` sub-tree (`confirming` while an on-chain deposit awaits confirmation depth, `available` once spendable). The `withdrawals:$withdrawal_id:pending` accounts isolate each outbound crypto withdrawal as its own reserve, so reserved funds can't be spent twice and each withdrawal's lifecycle is tracked independently. ### FBO **Normal debit accounts** — the for-benefit-of bank accounts that hold the platform's real fiat. The `settled` balance is reconciled against the bank's statement; `inTransit` represents the ACH/wire clearing window before funds settle. ### Platform **Mixed nature accounts** — the operational backbone. `custody:hot:$network` holds per-network hot-wallet backing and `custody:$custodian:omnibus` holds pooled custodian backing; together they back the crypto entitlements customers hold. `treasury:gas:$network` tracks gas balances per network, `revenue:spread` accrues the spread earned on trades, `expense:networkFees` absorbs on-chain fees, and `suspense:deposits` parks incoming funds not yet attributed to a customer. ### Counterparties The `otcDesk` account represents the OTC desk that fills buy and sell orders — it goes negative under `allowing unbounded overdraft` while a trade is mid-flight and nets back as the legs settle. ### Exchanges Per-trade conversion accounts under `conv:$conversion_id` hold the in-progress legs of a buy or sell while the OTC desk settles. Account metadata records the trade side, the customer, and the trade status. A non-zero balance after settlement is a stranded leg that needs a compensating entry. ### External The external endpoints money flows to and from: `ach` and `wire` for fiat rails, and `$network:$address` for on-chain destinations. ## Fiat Deposits & Withdrawals Customer fiat moves through a `pending` window while the bank rail clears. `FIAT_DEPOSIT_INITIATE` — Customer USD arrives by ACH or wire and is credited as pending cash. The FBO `inTransit` account is allowed to overdraft, since the ledger entry is recorded before the bank statement confirms the funds. `FIAT_DEPOSIT_SETTLE` — The ACH or wire clears at the FBO bank. The settled backing is recognized and the customer's pending cash becomes available. Outbound fiat reserves the customer's available cash, then settles or returns: `FIAT_WITHDRAWAL_INITIATE` — Reserve customer USD for an outbound ACH or wire by moving it into the FBO `inTransit` account. `FIAT_WITHDRAWAL_SETTLE` — The outbound transfer clears the FBO bank; in-transit backing moves to settled. If the transfer is rejected, `FIAT_WITHDRAWAL_RETURN` sends the reserved funds back to the customer's available cash and flags the entry as an adjustment. ## Crypto Deposits On-chain deposits are credited as soon as they are observed, then promoted once confirmed. `CRYPTO_DEPOSIT_DETECTED` — An on-chain deposit is observed but has not yet reached confirmation depth. The custodian omnibus backing is drawn down (overdraft permitted) and the customer is credited as `confirming`. `CRYPTO_DEPOSIT_CONFIRMED` — The deposit reaches confirmation depth. The customer's confirming balance becomes available and spendable. Block confirmations are asynchronous and can take seconds to minutes depending on the network. Crediting deposits as `confirming` gives customers immediate visibility while keeping unconfirmed funds out of the spendable balance. ## Trading (Buy & Sell via OTC Desk) Trades route through a per-trade conversion account so each leg is isolated until the OTC desk settles. The platform earns a spread on every fill. `BUY_TRADE_INITIATE` — Lock the customer's USD (gross of spread) into a per-trade conversion account. Metadata records the side, customer, and a pending status. `BUY_TRADE_SETTLE` — The OTC desk delivers crypto. The spread is booked to `revenue:spread`, the remaining USD goes to the desk, the customer is credited their crypto, and the custodian omnibus backing is replenished from the desk. The sell flow mirrors the buy flow, exchanging crypto for USD: `SELL_TRADE_INITIATE` — Lock the customer's crypto into a per-trade conversion account. `SELL_TRADE_SETTLE` — The OTC desk delivers USD. The crypto goes to the desk and replenishes omnibus backing, the spread is booked, and the remaining USD is credited to the customer's available cash. If a conversion leg is left stranded — a settlement never completes — `CONVERSION_COMPENSATE` reverses the held balance back to the customer and flags the entry as an adjustment. ## Crypto Withdrawals Outbound crypto is reserved into a per-withdrawal account before it is sent on-chain. `CRYPTO_WITHDRAWAL_INITIATE` — Reserve the customer's crypto for an outbound on-chain withdrawal, moving it from available into a per-withdrawal pending reserve. `CRYPTO_WITHDRAWAL_SETTLE` — Send the crypto out of the per-network hot wallet to chain. The platform absorbs the network fee, booking it from `expense:networkFees` into the network's gas balance. If a reserved withdrawal is abandoned before it settles, `CRYPTO_WITHDRAWAL_CANCEL` releases the reserve back to the customer's available crypto and flags the entry as an adjustment. To keep the hot wallet funded for outbound withdrawals, `CUSTODIAN_REFILL` moves crypto from a custodian into the hot wallet on the same network. ## The Reconciliation Invariant Because customer entitlements and platform custody are tracked in separate account trees, the system maintains a continuous invariant: for every asset, the sum of all customer crypto entitlements must equal the sum of all platform backing (hot wallets plus custodian omnibus wallets). Entitlements (`customers::crypto:available`) are what the platform owes customers; backing (`platform:custody::omnibus` and `platform:custody:hot:`) is what the platform actually holds. These two totals should match per asset at all times. A divergence points to an unbooked deposit, an untracked withdrawal, or a custody discrepancy — surface it before it compounds. ## Queries The `queries` section defines reusable lookups for monitoring, reconciliation, and audit: - **`custody_vs_entitlement_parity`** — total customer crypto entitlement per asset, the entitlement side of the reconciliation check - **`hot_wallet_backing`** — backing held across all per-network hot wallets - **`per_customer_multi_asset_position`** — every account under one customer with its per-asset balance: the customer's full statement - **`total_customer_entitlement_per_asset`** — every customer's spendable cash and crypto summed per asset: the platform's total liability to customers - **`total_custody_backing_per_asset`** — all platform crypto backing across every custodian and hot wallet, summed per asset - **`crypto_reserved_for_pending_withdrawals`** — per-withdrawal reserves still holding a balance: crypto reserved but not yet settled or cancelled - **`funds_in_flight_pending_and_confirming`** — customer fiat in the clearing window and crypto awaiting confirmation: credited but not yet spendable - **`stuck_conversion_aging`** — per-trade conversion accounts with a non-zero balance: stranded legs needing a compensating entry - **`unattributed_deposit_suspense`** — incoming funds received but not yet attributed to a customer - **`daily_spread_revenue`** — volume into the spread revenue account over the daily window - **`daily_absorbed_network_fees`** — on-chain fees the platform absorbed over the daily window, read per native asset - **`per_customer_crypto_throughput`** — volume in and out of one customer's spendable crypto balance over a period - **`per_customer_transaction_audit`** — every transaction touching any account under one customer, for audit and dispute investigation - **`fbo_bank_statement_reconciliation`** — settled and in-transit USD backing at the FBO bank, matched against the bank's daily statement These leverage the hierarchical account structure — filtering on `customers::crypto:available` matches across all customers, and `platform:custody:hot:` matches every per-network hot wallet. --- ## BNPL & Lending Source: https://docs.formance.com/examples/advanced/bnpl-lending This example shows how to model Buy-Now-Pay-Later and installment lending in Formance using a declarative ledger schema. The platform pays merchants upfront while the shopper repays over a fixed installment schedule, so the ledger has to track per-installment principal receivables, accrued interest and late fees, merchant payables, PSP collection float, and the eventual write-off of defaulted balances — all at once. **Common use cases:** - **BNPL providers** financing point-of-sale purchases across a merchant network - **Consumer lenders** servicing fixed-term installment loans - **Embedded finance platforms** offering pay-over-time at checkout This is an illustrative example. Adapt the schema to your specific business requirements, regulatory obligations, and financial practices. ## Key Concepts - **The receivable is an asset** — When a purchase is financed, the principal the shopper owes is booked per installment under `borrowers:` as outstanding principal. These are **normal debit accounts**: their balance is what the shopper still owes, the live receivable on the platform's book. - **Per-installment schedule** — Each plan is split into numbered installments (`installments:1` through `installments:4`), and each installment tracks its own `principal`, `interest`, and `fees` components independently. This gives a precise amortization and repayment picture rather than a single rolled-up balance. - **Merchant settled upfront** — The merchant is owed the purchase price net of a merchant-discount fee the moment the plan originates, recorded as a payable under `counterparties:merchants:`. The platform settles that payable on its own weekly cadence while the shopper repays over time — the platform carries the financing risk in between. - **Earned-on-collection** — Interest and late fees are accrued onto the receivable and a parallel `earnedNotCollected` memo, but no revenue is recognized until the cash is actually collected. Recognized revenue therefore never includes accrued-but-uncollected interest. - **Write-off and recovery** — When an installment defaults, its outstanding principal is reclassified to `writtenOff` and the never-collected interest and fee receivables are cancelled against their memos with no P&L impact. Later recoveries post back against the charged-off balance, so the written-off total always reads net of recoveries. ## The Complete Schema This is the full ledger schema for a BNPL and lending system. The sections below explain each part. ## Chart of Accounts The `chart` section defines three account groups: the shopper receivables (`borrowers`), the external parties the platform owes or collects through (`counterparties`), and the platform's own cash and revenue (`platform`). ### Borrowers Each borrower holds one or more `plans`, and each plan is broken into numbered `installments`. Every installment tracks three components — `principal`, `interest`, and `fees` — each with its own set of sub-accounts: - **`principal:outstanding`** — the receivable still owed. **Normal debit account** (an asset); its balance is live principal on the book. - **`principal:paid`** / **`principal:writtenOff`** — repaid principal, and principal charged off as a loss (recoveries post back here). - **`interest:accrued`** / **`fees:accrued`** — interest and late fees that have been assessed onto the receivable. - **`interest:earnedNotCollected`** / **`fees:earnedNotCollected`** — the memo that holds accrued income before it is recognized; under the earned-on-collection model it must not be treated as revenue until collected. - **`interest:paid`** / **`fees:paid`** — the collected portion of each component. ### Counterparties The external parties the platform transacts with. `merchants:$merchant_id:payable` is what the platform owes each merchant — a **liability** accrued at origination and drained to zero by the weekly settlement run. `psp:$psp_id:collections:pending` is the PSP collection float: cash the payment processor has collected from shoppers on the platform's behalf but not yet remitted to the operating bank. ### Platform The platform's own accounts. `banks:$bank_id:operating` holds the cash position at the partner bank. The `revenue` group breaks recognized income out by stream: `revenue:interest`, `revenue:fees:late`, and `revenue:fees:merchantDiscount` — these follow standard income-statement conventions and only ever receive cash that has actually been collected. ## Flows ### Origination & Merchant Settlement `MERCHANT_PURCHASE_FINANCED` books the principal receivable across the four installments, accrues the merchant payable net of the merchant-discount fee, and retains that discount fee as revenue. The shopper now owes principal per installment; the merchant is owed the purchase price upfront. `MERCHANT_WEEKLY_SETTLEMENT` pays the merchant's accrued payable by ACH from the operating bank on the platform's weekly cadence, draining the payable to zero. This is independent of how far along the shopper is in repayment — the platform fronts the cash. If a merchant ACH payout is returned, `MERCHANT_SETTLEMENT_RETURN` restores the merchant payable and reverses the operating-bank movement. ### Interest & Fee Accrual Over the life of the plan, income is assessed onto the receivable without recognizing revenue. `INTEREST_ACCRUAL` accrues interest on one installment into both `interest:accrued` (the receivable) and `interest:earnedNotCollected` (the memo). `LATE_FEE_ASSESSED` does the same for a late fee, pushing it onto `fees:accrued` and `fees:earnedNotCollected`. Neither accrual touches a revenue account. Under the earned-on-collection model, interest and fees only become revenue at the moment the cash is collected — until then they live in the `earnedNotCollected` memo. ### Installment Repayment `INSTALLMENT_COLLECTION` collects an installment through the PSP card autopay and applies a fee-interest-principal waterfall: incoming cash clears outstanding fees first, then interest, then principal. The collected fee and interest portions are recognized as revenue into `revenue:fees:late` and `revenue:interest`. `INSTALLMENT_COLLECTION_RETURN` reverses a returned or charged-back collection: it restores the receivables, removes the PSP-float cash, and reverses the interest and fee revenue that had been recognized. `PSP_REMITTANCE` settles collected card payments from the PSP into the operating bank, clearing the PSP collection float and growing the operating-bank backing. ### Default, Write-Off & Recovery When an installment hits 90 days past due, `PLAN_WRITE_OFF` charges it off: outstanding principal is reclassified to `principal:writtenOff`, and the never-collected interest and fee receivables are cancelled against their `earnedNotCollected` memos. Because no interest or fees were ever recognized as revenue, the write-off has no P&L impact beyond the principal loss. If the platform later recovers part of a charged-off balance, `RECOVERY` posts the recovered cash (via the PSP float) back against `principal:writtenOff`, reducing the charged-off balance toward zero. ## Why Accrue Separately From Recognizing Revenue Splitting `accrued` from `earnedNotCollected`, and recognizing revenue only on collection, keeps the income statement honest. Interest and late fees sit on the receivable as something the shopper owes, but they do not inflate revenue while they remain uncollected. If a plan defaults, the write-off simply cancels those uncollected receivables against their memos — there is no revenue to reverse, because none was ever recognized. The only loss that hits the book is the principal that was actually advanced, which is exactly the charged-off principal (net of any recovery posted back). ## Queries The `queries` section defines reusable lookups across servicing, accounting, and reconciliation: **Per-plan and per-shopper servicing** - `outstanding_principal_for_one_plan` — each installment's outstanding principal for one plan: what principal is still owed. - `full_schedule_for_one_plan` — every installment-component account for one plan (principal, interest, fees across all states): the complete amortization and repayment picture in one read. - `everything_for_one_shopper` — every account and balance for one shopper across all of their plans and installments. - `charged_off_principal_for_one_plan` — the charged-off principal for one plan, already net of any recovery posted back. **Book-wide accounting totals** - `total_outstanding_principal` — live principal receivable across the entire book. - `interest_accrued_not_yet_collected` — interest accrued but not yet recognized as revenue. - `late_fees_accrued_not_yet_collected` — late fees accrued but not yet recognized as revenue. - `total_charged_off_principal_net_of_recoveries` — the single platform loan-loss figure, net of recoveries. - `recognized_interest_revenue` — interest actually collected (never accrued-but-uncollected interest). - `recognized_late_fee_revenue` — late fees actually collected. - `revenue_by_stream` — every platform revenue account broken out by stream (interest, late fees, merchant discount). **Counterparty and cash positions** - `merchant_payable_for_one_merchant` — what the platform currently owes one merchant, the amount the next weekly ACH will drain. - `total_merchant_payable` — total merchant liability the weekly settlement run discharges. - `psp_collection_float` — cash the PSP has collected but not yet remitted. - `operating_bank_cash_position` — cash the platform holds at the partner bank. **Operational worklists** - `open_installment_receivables` — every outstanding-principal account that still carries a balance, the set of installments not yet fully collected. - `charged_off_balances_outstanding` — charged-off installments that still carry a balance, the collections follow-up worklist. **Daily reconciliation volumes** - `psp_collections_received_today` — volume into the PSP collection float over the day, matched against the PSP's daily collection report. - `merchant_settlement_outflow_today` — volume out of the merchant payable over the day, matched against the day's ACH file. --- ## Marketplace Payouts Source: https://docs.formance.com/examples/advanced/marketplace-payouts This example shows how to model a marketplace in Formance using a declarative ledger schema. When a buyer pays for an order, the gross amount is split between the platform's commission, the PSP's processing fee, and the seller's share — which is held in escrow until delivery is confirmed, released to the seller's balance, and eventually paid out. The schema also tracks how refunds and chargebacks claw funds back from sellers, with the platform absorbing any shortfall. **Common use cases:** - **E-commerce marketplaces** splitting each order between platform commission and seller proceeds - **Gig and services platforms** holding funds in escrow until a job is confirmed complete - **Multi-vendor platforms** running periodic payout batches and managing seller liabilities This is an illustrative example. Adapt the schema to your specific business requirements, regulatory obligations, and financial practices. ## Key Concepts - **Order Split** — A single buyer payment is split atomically at capture into three parts: the platform's `commission`, the PSP's `feesPayable`, and the remainder held for the seller. The split happens in one transaction, so the books always balance against the gross captured amount. - **Escrow Until Delivery** — The seller's share is not credited to the seller immediately. It sits in a per-order escrow `held` account until delivery is confirmed, at which point it is released to the seller's `payable` balance. This keeps undelivered orders separate from funds the seller can actually be paid. - **Seller Liability Balances** — Each seller's `payable` is a liability the platform owes them. Payouts move through a staged `payout:pending` account so reserved funds can't be double-spent and failed transfers can be returned cleanly. - **Chargeback Absorption** — When a buyer disputes a charge, the loss is charged against the seller's `payable` first, and the platform absorbs whatever the seller can't cover into its own `chargebacks` expense account. The same first-seller-then-platform waterfall applies to refunds. ## The Complete Schema This is the full ledger schema for a marketplace payouts system. The sections below explain each part. ## Chart of Accounts The `chart` section defines the account groups for the marketplace: sellers, per-order escrow, the platform's own books, the PSPs and exchanges acting as counterparties. ### Sellers **Normal credit accounts** — each seller's `payable` represents a liability the platform owes them. The `payout:pending` sub-account stages funds reserved for an in-flight payout, isolating them from the available `payable` balance so a transfer in progress can't be spent or paid out twice. A seller is keyed by `$seller_id`. ### Escrow A per-order `held` account, keyed by `$order_id`. It receives the seller's share of an order at capture and drains to zero when the order is released to the seller or refunded to the buyer. A lingering non-zero balance means the order is still awaiting delivery confirmation. ### Platform **Mixed nature accounts** — the platform's own books. `revenue:commission` and `revenue:fx` record earned income; `expense:refunds` and `expense:chargebacks` capture the costs the platform absorbs. The `banks` group holds each settlement bank's `operating` and `payout` accounts (keyed by `$bank_id`), `treasury:fx` carries the open FX book, and `suspense:reconciliation:differences` parks reconciliation deltas pending investigation. ### Counterparties The PSPs the platform settles with, keyed by `$psp_id`. The `settlement` account tracks the receivable owed by the PSP from captured payments (it runs negative until the PSP sweeps cash to the bank), while `feesPayable` accrues the processing fees the PSP retains. ### Exchanges Per-conversion crossing accounts keyed by `$conversion_id`, used as the pivot when a seller's payable is converted from one currency to another. Each conversion account is tagged with its execution rate and settles to zero once the crossing completes. ## Order Capture & Split When a buyer's card payment is captured at the PSP, the gross amount is split in a single transaction. `ORDER_PAYMENT_CAPTURED` records the captured payment. The gross is sourced from the PSP settlement receivable and split three ways: up to `$commission` to the platform's commission revenue, up to `$psp_fee` to the PSP's fees payable, and the remainder held in the order's escrow account for the seller. `PSP_SETTLEMENT` records the daily cash sweep: the platform's operating bank account funds the PSP settlement receivable, and the PSP returns the fees it retained. This clears the receivable that capture left running negative. ## Seller Funds Release & Payout Once delivery is confirmed, the seller's escrowed funds are released and become eligible for payout. `SELLER_FUNDS_RELEASE` moves the held order funds out of escrow and into the seller's `payable` balance once delivery is confirmed. The funds are now part of what the platform owes the seller. `SELLER_PAYOUT_INITIATE` runs as part of the weekly payout batch: it reserves an amount from the seller's `payable` into the `payout:pending` staging account, so it can no longer be spent while the transfer is in flight. `SELLER_PAYOUT_SETTLE` completes the cycle once the bank transfer is confirmed: the reserved amount moves from `payout:pending` to the platform's `banks` payout account. If a reserved payout fails before settling, `SELLER_PAYOUT_RETURN` reverses the reservation by sending the in-flight amount from `payout:pending` back to the seller's `payable`. ### Multi-Currency Payout When the seller is paid out in a different currency than they earned, `MULTI_CURRENCY_PAYOUT` crosses the payable through a per-conversion exchange account at the treasury rate. The platform's `revenue:fx` captures the spread, and the converted remainder lands in the seller's `payout:pending`. ## Refunds & Chargebacks Refunds and chargebacks both claw funds back toward the PSP, but they differ in how the loss is allocated. `REFUND` returns a past order to the buyer. The gross is sourced through a waterfall: first the seller's net share from `payable`, then the platform's `commission` is given back, and any remaining shortfall is absorbed into the platform's `expense:refunds`. The funds flow to the PSP settlement account. `CHARGEBACK` handles a buyer dispute. The full gross is charged against the seller's `payable` first, with the platform absorbing any shortfall into `expense:chargebacks`. The funds flow to the PSP settlement account. ## Who Absorbs the Loss Both `REFUND` and `CHARGEBACK` use a source waterfall that charges the seller first and the platform second. This encodes the marketplace's risk policy directly in the ledger: - The seller's `payable` is debited up to the amount they hold. If the seller still has a balance, the loss comes out of their funds. - Whatever the seller cannot cover falls through to a platform expense account — `expense:refunds` for the absorbed PSP fee on a refund, or `expense:chargebacks` for a dispute shortfall. Because the seller's `payable` is allowed to go negative on a refund, a dispute that lands after the seller has already been paid out leaves the seller with a negative balance — a debt the platform must recover on the next payout. The `sellers_with_a_negative_balance` query surfaces exactly these cases. ## Queries The `queries` section defines reusable lookups across the marketplace: - `one_seller_s_outstanding_position` — every account and balance under one seller: released payable plus any in-flight payout. - `held_versus_available_for_one_seller` — the seller's available payable balance only, excluding per-order escrow. - `sellers_with_a_negative_balance` — seller payables gone negative after a refund landed post-payout; the amounts the platform must recover. - `total_seller_liability` — the sum of every seller's released payable, per asset. - `total_held_in_delivery_confirmation` — the sum of all per-order escrow holds awaiting delivery, per asset. - `platform_revenue_by_stream` — each platform revenue stream broken out (commission and FX spread), per asset. - `treasury_fx_position` — the platform's open FX book position from currency crossings, per asset. - `platform_absorbed_refund_and_chargeback_cost` — the platform's expense accounts for absorbed PSP fees and chargeback shortfalls, per asset. - `per_order_fee_and_commission_split` — how much flowed into commission, the PSP fee, and the held escrow for one order. - `revenue_earned_over_a_period` — volume into the platform revenue accounts over a reporting window. - `payout_throughput_for_one_seller` — how much money moved out to one seller over a period. - `psp_net_position` — the platform's full net position with one PSP: settlement receivable plus fees payable. - `psp_settlement_reconciliation` — the ledger side of the daily PSP settlement reconciliation. - `payout_bank_file_reconciliation` — the ledger side of the weekly payout reconciliation against the payout bank file. - `aging_escrow_holds` — per-order escrow holds with a lingering non-zero balance past the expected hold window. - `unresolved_reconciliation_differences` — the suspense differences account; non-zero means reconciliation deltas booked but not yet resolved. - `audit_trail_for_one_order` — every transaction tagged with one order id, in any direction: capture, release, refund, chargeback. These leverage the hierarchical account structure — filtering on `sellers::payable` matches across all sellers, and `escrow:orders::held` matches every per-order escrow account. --- ## Overview Source: https://docs.formance.com/deploy/cloud/overview Formance Cloud is a fully hosted and managed deployment environment which can be used for your production applications. Information regarding the available plans can be found on the [Formance Cloud Pricing page](https://www.formance.com/pricing). ## Available Regions Formance Cloud is available in the following regions: ## Getting Started ### Prerequisites Before creating a stack, ensure you have: - [fctl installed](/getting-started/quickstart) - A Formance Cloud account ### Creating a Stack Once you have set up a billing account (by reaching out to us), your available production deployments quota will be increased. You can then create a production deployment: ```shell fctl stack create my-stack ``` To specify a region: ```shell fctl stack create my-stack --region aws-eu-west-1-hosting ``` ### Listing Regions To list all available regions: ```shell fctl cloud regions ls ``` To get detailed information about a specific region, including available modules: ```shell fctl cloud regions show aws-eu-west-1-hosting ``` Example output: ``` # Information ID | c9abcd0df-480b-42c4-b5e7-dfght8e7c852 Name | example Base URL | https://example.com Active | Yes Public | Yes Version | latest Last ping | 2024-05-15T14:09:02Z # Capabilities MODULE_LIST | auth, gateway, ledger, orchestration, payments, reconciliation, stargate, wallets, webhooks ``` ``` # Information ID | c9abcd0df-480b-42c4-b5e7-dfght8e7c852 Name | example Base URL | https://example.com Active | Yes Public | Yes Version | latest Last ping | 2024-05-15T14:09:02Z # Capabilities MODULE_LIST | auth, gateway, ledger, stargate ``` ## Managing Modules By default, a stack includes the following modules: - **Ledger** - Double-entry accounting - **Gateway** - API gateway - **Payments** - Payment provider integrations - **Auth** - Authentication - **Stargate** - Connectivity layer - **Ledger** - Double-entry accounting - **Gateway** - API gateway - **Auth** - Authentication - **Stargate** - Connectivity layer ### Enable/Disable Modules You can enable or disable additional modules using fctl: ```shell # Enable a module fctl stack modules enable orchestration # Disable a module fctl stack modules disable orchestration ``` ```shell # Enable a module, using one of the names below fctl stack modules enable ledger # Disable a module fctl stack modules disable ledger ``` ### Available Modules The following modules can be enabled on your stack: | Module | Description | | --- | --- | | `ledger` | Double-entry accounting system | | `payments` | Payment provider integrations | | `wallets` | Digital wallet management | | `orchestration` | Workflow orchestration (Flows) | | `reconciliation` | Payment reconciliation | | `webhooks` | Event notifications | | `auth` | Authentication | | `gateway` | API gateway | | `stargate` | Connectivity layer | | Module | Description | | --- | --- | | `ledger` | Double-entry accounting system | | `auth` | Authentication | | `gateway` | API gateway | | `stargate` | Connectivity layer | ## Verifying Your Stack After creating your stack, verify it's working properly: ```shell # Check stack status fctl stack show my-stack # Query ledger server info fctl ledger server-infos --stack=my-stack ``` ## Managing Your Stack ### List Stacks ```shell fctl stack ls ``` ### Delete a Stack ```shell fctl stack delete my-stack ``` ## Information and Requests For more information and any requests regarding the Formance Cloud offering, please contact us at [support@formance.com](mailto:support@formance.com). As a SOC-2 Type II certified company, we take security and privacy seriously — and will be happy to answer any questions you may have, as well as provide you with a copy of our SOC-2 report upon request. --- ## Deployment Overview Source: https://docs.formance.com/deploy/overview With a modular and cloud-native architecture, the Formance Platform is designed to be deployed in a variety of ways depending on your requirements, ranging from fully managed to fully self-hosted. More details on the platform architecture can be found in the [Architecture](/manage/architecture) section. ## Hosting Options | | Formance Cloud | Self-Hosting | |------------------------|----------------|------------------------| | Control plane | Formance | Self-hosted | | Data plane | Formance | Self-hosted | | Available editions | N/A | Community / Enterprise | | Support | Formance | Community / Formance | ### Formance Cloud Formance Cloud is a fully managed deployment of the Formance Platform, which is hosted and managed by Formance. It is the easiest way to get started, and the recommended deployment option for companies looking to minimize the infrastructure overhead of running the platform. This deployment option is available to all companies, as part of the Formance Cloud starter plan. Learn more about Formance Cloud and available regions ### Self-Hosting Self-hosting allows you to deploy Formance on your own Kubernetes infrastructure with full control over your data and environment. Two editions are available: #### Community Edition The Community Edition is open-source and free to use. It includes core modules (Ledger, Payments, Gateway) and is supported by the community. The Community Edition is open-source and free to use. It includes core modules (Ledger, Gateway) and is supported by the community. Open-source, community-supported deployment #### Enterprise Edition The Enterprise Edition includes everything in Community Edition plus: - **Auth** - Authentication and authorization - **Wallets** - Digital wallet management - **Flows** - Workflow orchestration - **Reconciliation** - Payment reconciliation - **Webhooks** - Event notifications - **Web Console** - Graphical interface for managing your stack - **SSO/OIDC** - Single Sign-On integration - **RBAC** - Role-Based Access Control - **Audit Logs** - Complete audit trail - **Formance Support** - Direct support from Formance - **Auth** - Authentication and authorization - **Web Console** - Graphical interface for managing your stack - **SSO/OIDC** - Single Sign-On integration - **RBAC** - Role-Based Access Control - **Audit Logs** - Complete audit trail - **Formance Support** - Direct support from Formance Commercial license with Console, SSO, RBAC, and support ## Choosing the Right Option | Use Case | Recommended Option | |----------|-------------------| | Getting started quickly | Formance Cloud | | Minimal infrastructure overhead | Formance Cloud | | Full control over data | Self-Hosting | | Open-source / community support | Self-Hosting (Community) | | Enterprise features (Console, SSO, RBAC) | Self-Hosting (Enterprise) | | Compliance requirements | Self-Hosting (Enterprise) | ## Quick Start Fully managed, get started in minutes Self-hosted, open-source Self-hosted with enterprise features --- ## All-in-one Demo Chart Source: https://docs.formance.com/deploy/self-hosted/demo/quick-start This guide shows you how to deploy a fully working Formance stack for testing using the unified `formance` Helm chart with the CE demo profile. ## What Gets Deployed The CE demo profile layers on top of `ce-minimal.yaml` and adds: - **Bundled PostgreSQL** — single-node Bitnami database for all modules - **A `demo-dev` stack** — pre-configured via `regions.stacks` with Gateway and Ledger modules (version `v3.1`) - **Settings** — PostgreSQL URI, resource requests, and replica count configured automatically via `regions.settings` - Each module runs with **2 replicas** and minimal resources (50m CPU, 500Mi memory) **Default host:** `formance.localhost` ## Install the Demo The `ce-demo.yaml` profile [currently injects an unknown field](https://github.com/formancehq/helm/blob/formance-{{infra.helm-chart-version}}/charts/regions/templates/stack.yaml) into the `Stack` resource that fails CRD validation. Until the fix lands, install with only `ce-minimal.yaml` and apply the demo stack manually — that's what the steps below do. Install the chart with the minimal profile: ```bash helm upgrade --install formance {{infra.helm-chart-url}} \ --version {{infra.helm-chart-version}} \ --namespace formance-system \ --create-namespace \ -f https://raw.githubusercontent.com/formancehq/helm/formance-{{infra.helm-chart-version}}/charts/formance/profiles/ce-minimal.yaml ``` Apply the demo Stack and modules: ```yaml # demo.yaml apiVersion: formance.com/v1beta1 kind: Stack metadata: name: demo-dev spec: debug: true versionsFromFile: v3.1 --- apiVersion: formance.com/v1beta1 kind: Gateway metadata: name: demo-dev spec: stack: demo-dev --- apiVersion: formance.com/v1beta1 kind: Ledger metadata: name: demo-dev spec: stack: demo-dev ``` ```bash kubectl apply -f demo.yaml ``` The demo uses a bundled PostgreSQL without persistent storage — not for production. ## Verify the Installation Check that the operator and PostgreSQL pods are running: ```bash kubectl get pods -n formance-system ``` Check that the stack has been created: ```bash kubectl get stacks ``` You should see a `demo-dev` stack. Wait for all stack modules to become ready: ```bash kubectl get pods --all-namespaces -l formance.com/stack ``` ## Access the Demo Port-forward the Gateway to access the stack API locally: ```bash kubectl port-forward -n demo-dev svc/gateway 8080:8080 ``` Verify the stack is responding: ```bash curl http://localhost:8080/versions ``` ## What's Inside the CE Demo Profile The `ce-demo.yaml` profile configures the following Helm values: ```yaml global: serviceHost: "formance.localhost" regions: settings: postgres-uri: key: "postgres.*.uri" value: "postgresql://formance:formance@postgresql.formance-system.svc:5432?disableSSLMode=true" resource-requests: key: "deployments.*.containers.*.resource-requirements.requests" value: "cpu=50m,memory=500Mi" replicas: key: "deployments.*.replicas" value: "2" stacks: demo-dev: debug: true versionsFromFile: "v3.1" modules: gateway: {} ledger: {} ``` This creates: - A `Stack` CRD named `demo-dev` with the `v3.1` module versions - `Settings` CRDs for PostgreSQL URI, resource requests, and replica count ## Create Your Own Demo Stack Instead of using the profile, define a custom demo stack directly in your values: ```yaml # custom-demo.yaml regions: settings: postgres-uri: key: "postgres.*.uri" value: "postgresql://formance:formance@postgresql.formance-system.svc:5432?disableSSLMode=true" stacks: my-test-stack: versionsFromFile: "v3.1" modules: gateway: {} ledger: {} payments: {} ``` ```bash helm upgrade --install formance {{infra.helm-chart-url}} \ --version {{infra.helm-chart-version}} \ --namespace formance-system \ --create-namespace \ -f custom-demo.yaml ``` ## Clean Up To remove the demo deployment: ```bash helm uninstall formance -n formance-system ``` --- ## Operator Setup Source: https://docs.formance.com/deploy/self-hosted/installation/operator-setup This guide walks you through installing Formance on your Kubernetes cluster using the unified Formance Helm chart. Enterprise Edition requires a valid license. Contact [sales@formance.com](mailto:sales@formance.com) to obtain one. ## Prerequisites Before you begin, ensure you have: - Kubernetes + cluster - Helm installed - `kubectl` configured to access your cluster - An Ingress Controller (nginx, traefik, etc.) - SSL certificates for your domain - **Enterprise Edition license** (Cluster ID and License Token) For detailed requirements, see the [Requirements](/deploy/self-hosted/requirements) page. ## Quick Start Install Formance with a single command: ```bash helm upgrade --install formance {{infra.helm-chart-url}} \ --version {{infra.helm-chart-version}} \ --namespace formance-system \ --create-namespace ``` This installs the Formance Operator, CRDs, and a bundled PostgreSQL instance. Community Edition is enabled by default. ### Set Environment Variables ```bash ``` ### Install ```bash helm upgrade --install formance {{infra.helm-chart-url}} \ --version {{infra.helm-chart-version}} \ --namespace formance-system \ --create-namespace \ --set tags.EntrepriseEdition=true \ --set global.licence.clusterID=$CLUSTER_ID \ --set global.licence.token=$LICENCE_TOKEN \ --set global.serviceHost=$BASE_DOMAIN ``` ### Configure Service Hosts Enterprise Edition requires service hosts for the control plane components: | Component | Value Key | |-----------|-----------| | Global | global.serviceHost | | Membership | membership.serviceHost | | Dex (OIDC) | dex.serviceHost | | Portal | portal.serviceHost | | Console V3 | console.serviceHost | ## Using Minimal Profile For a minimal installation with reduced resource requirements: ```bash helm upgrade --install formance {{infra.helm-chart-url}} \ --version {{infra.helm-chart-version}} \ --namespace formance-system \ --create-namespace \ -f https://raw.githubusercontent.com/formancehq/helm/main/charts/formance/profiles/ce-minimal.yaml ``` For a minimal Enterprise installation: ```bash helm upgrade --install formance {{infra.helm-chart-url}} \ --version {{infra.helm-chart-version}} \ --namespace formance-system \ --create-namespace \ --set tags.EntrepriseEdition=true \ --set global.licence.clusterID=$CLUSTER_ID \ --set global.licence.token=$LICENCE_TOKEN \ -f https://raw.githubusercontent.com/formancehq/helm/main/charts/formance/profiles/ee-minimal.yaml ``` ## Custom Configuration ### Using a Values File Create a custom `values.yaml` to configure the platform: ```yaml # values.yaml postgresql: enabled: false # Disable bundled PostgreSQL global: postgresql: host: "my-postgres.example.com" additionalArgs: "sslmode=require" auth: username: "formance" password: "your-secure-password" database: "formance" regions: operator: enabled: true versions: create: true settings: postgres-uri: key: "postgres.*.uri" value: "postgresql://formance:your-secure-password@my-postgres.example.com:5432?disableSSLMode=true" ``` For a production deployment with a custom domain and TLS: The chart contains predictable OAuth, session, Dex, and PostgreSQL credentials for local evaluation. Before running this command in production, create a values file that replaces the defaults with Kubernetes Secret references. See [Replace Default Secrets Before Production](/deploy/self-hosted/installation/control-plane#replace-default-secrets-before-production). ```bash helm upgrade --install formance {{infra.helm-chart-url}} \ --version {{infra.helm-chart-version}} \ --namespace formance-system \ --create-namespace \ --set tags.EntrepriseEdition=true \ --set global.licence.clusterID=$CLUSTER_ID \ --set global.licence.token=$LICENCE_TOKEN \ --set global.serviceHost=$BASE_DOMAIN ``` ### View All Configuration Options ```bash helm show values {{infra.helm-chart-url}} --version {{infra.helm-chart-version}} ``` ## Verify Installation Check that all components are running: ```bash kubectl get pods -n formance-system ``` You should see the operator pod in `Running` state: ``` NAME READY STATUS RESTARTS AGE formance-operator-xxxxx-xxxxx 1/1 Running 0 1m ``` You should see: ``` NAME READY STATUS RESTARTS AGE formance-operator-xxxxx-xxxxx 1/1 Running 0 2m membership-xxxxx-xxxxx 1/1 Running 0 2m console-v3-xxxxx-xxxxx 1/1 Running 0 2m membership-dex-xxxxx-xxxxx 1/1 Running 0 2m postgresql-0 1/1 Running 0 2m ``` ## License Validation The chart enforces license validation when `tags.EntrepriseEdition` is `true`: - Installation will **fail** if `global.licence.clusterID` is missing - Installation will **fail** if `global.licence.token` is missing Get your Cluster ID with: `kubectl get ns kube-system -o jsonpath='\{.metadata.uid\}'` ## Troubleshooting ### `no matches for kind "Versions" in version "formance.com/v1beta1"` If the install fails with `no matches for kind "Versions" in version "formance.com/v1beta1"` (often alongside `ensure CRDs are installed first`), the Formance operator's Custom Resource Definitions (CRDs) aren't registered in your cluster yet. The chart is trying to create a `Versions` custom resource (`kind: Versions`, `formance.com/v1beta1`) before the CRD that defines it exists. **Cause.** The CRD that backs `kind: Versions` — `versions.formance.com` — is provided by the operator's bundled `operator-crds` subchart, and that subchart is **not installed by default**: `regions.operator.operator-crds.create` defaults to `false` (and the bundled subchart is deprecated in favor of a standalone `operator-crds` chart). With nothing to register `versions.formance.com`, the `Versions` resource the chart applies has no matching CRD. The CRDs must be installed **before** the operator and stack. **Fix — install the operator CRDs first, then install the operator and stack.** Use one of the options below. #### Option 1 — install the `operator-crds` chart as its own release (recommended) Install the standalone `operator-crds` chart from the Formance OCI registry before installing Formance. It follows the operator's `3.x` version line (independent of the unified chart version), so pin it to the operator version bundled by your Formance chart release — the version below matches Formance chart : ```bash helm upgrade --install operator-crds oci://ghcr.io/formancehq/helm/operator-crds \ --version 3.9.2 \ --namespace formance-system \ --create-namespace ``` #### Option 2 — enable the bundled CRDs via values Alternatively, let the unified chart create the CRDs by enabling the bundled subchart on the install. Helm orders the CRDs ahead of the `Versions` resource within the same release: ```bash helm upgrade --install formance {{infra.helm-chart-url}} \ --version {{infra.helm-chart-version}} \ --namespace formance-system \ --create-namespace \ --set regions.operator.operator-crds.create=true ``` The values-file equivalent is: ```yaml regions: operator: operator-crds: create: true ``` #### Confirm the CRDs, then install the operator and stack After installing the CRDs, confirm the `Versions` CRD is established before continuing: ```bash kubectl wait --for condition=established --timeout=60s crd/versions.formance.com ``` Then install (or re-run) the operator and stack using the install command for your edition (Community or Enterprise) documented above under [Quick Start](#quick-start). If you used Option 2, the single install command already created the CRDs alongside the rest of the release. You can also hit this error by bypassing Helm's normal install flow — for example rendering the chart offline and piping it to `kubectl` (`helm template ... | kubectl apply -f -`), or passing `--skip-crds`. The fix is the same: install the CRDs (via either option above) and wait for `versions.formance.com` to be established before applying anything that references `kind: Versions`. ## Next Steps After installing the operator: 1. Configure your [Infrastructure Services](/deploy/self-hosted/infrastructure/postgresql) (PostgreSQL, Message Broker) 2. Deploy your first [Stack](/deploy/self-hosted/modules/stack) 3. Enable the modules you need 4. [Configure the Control Plane](/deploy/self-hosted/installation/control-plane) — Set up Dex, create organizations, and configure SSO --- ## Control Plane Setup Source: https://docs.formance.com/deploy/self-hosted/installation/control-plane After installing Formance Enterprise Edition, you need to configure the control plane components: Dex (OIDC provider), Membership (management API), and Console (web interface). The Membership service is versioned independently from the stack. The `admin wizard` bootstrap command landed in `v2.0.0`, so the instructions on this page depend on which Membership release the Helm chart you're installing ships. Pick yours below. ## Components Overview | Component | Description | Default URL | |-----------|-------------|-------------| | **Dex** | OIDC/OAuth2 provider for authentication | `dex.` | | **Membership** | Management API for organizations, users, and stacks | `membership.` | | **Console** | Web interface for managing your Formance stack | `console.` | ## Replace Default Secrets Before Production The Helm chart includes predictable credentials for local evaluation. Do not use them in a production or shared environment. Treat the installation as incomplete until you replace every value in the table below and remove the default Dex user. | Credential | Default | Purpose | |------------|---------|---------| | Portal OAuth client secret | `changeMe1` | Authenticates Portal to Membership | | Console OAuth client secret | `changeMe2` | Authenticates Console to Membership | | Membership OIDC client secret | `changeMe` | Authenticates Membership to Dex | | Portal cookie secret | `changeMe2` | Protects Portal sessions | | Console cookie encryption key | `changeMe00` | Protects Console sessions | | Dex static user | `admin@formance.com` / `password` | Provides a local test login | | PostgreSQL credentials | `formance` / `formance` | Authenticates services to the bundled database | Generate an independent, cryptographically random value for each credential. Store the values in your secret manager and synchronize them to a Kubernetes Secret in the Formance namespace. Do not commit the secret values to your Helm values file. The following example assumes a Secret named `formance-control-plane-secrets` already exists in the release namespace: ```yaml # production-values.yaml global: platform: portal: oauth: client: existingSecret: formance-control-plane-secrets secretKeys: secret: portal-oauth-client-secret consoleV3: oauth: client: existingSecret: formance-control-plane-secrets secretKeys: secret: console-oauth-client-secret cloudprem: membership: config: oidc: existingSecret: formance-control-plane-secrets secretKeys: secret: membership-oidc-client-secret dex: envVars: - name: MEMBERSHIP_CLIENT_SECRET valueFrom: secretKeyRef: name: formance-control-plane-secrets key: membership-oidc-client-secret configOverrides: enablePasswordDB: false staticPasswords: [] portal: config: cookie: existingSecret: formance-control-plane-secrets secretKeys: secret: portal-cookie-secret console-v3: config: cookie: existingSecret: formance-control-plane-secrets secretKeys: encryptionKey: console-cookie-encryption-key ``` `existingSecret` changes how the workloads read credentials; it does not create the Secret. Create it before the Helm release, or manage it with a controller such as External Secrets. Membership and Dex must read the same `membership-oidc-client-secret` value. For PostgreSQL, use a managed database in production and reference its credentials through `global.postgresql.auth.existingSecret`. See [PostgreSQL configuration](/deploy/self-hosted/infrastructure/postgresql). ## Dex Configuration [Dex](https://dexidp.io/) is deployed as the default OIDC provider. You can configure it to: - Create local users for testing - Connect to external identity providers (SAML, LDAP, OAuth2) - Integrate with your existing SSO infrastructure ### Default Configuration The default Dex configuration creates a simple setup for testing with a static admin user. For production, configure your own identity provider. ### Connecting to External Identity Providers Dex configuration is set through `cloudprem.membership.dex.configOverrides`, which merges with the default Dex configuration generated by the Membership chart. ```yaml # values.yaml cloudprem: membership: dex: configOverrides: connectors: - type: saml id: saml name: SAML config: ssoURL: https://your-idp.example.com/sso ca: /path/to/ca.pem redirectURI: https://dex.your-domain.com/callback usernameAttr: name emailAttr: email ``` ```yaml # values.yaml cloudprem: membership: dex: configOverrides: connectors: - type: ldap id: ldap name: LDAP config: host: ldap.example.com:636 insecureNoSSL: false bindDN: cn=admin,dc=example,dc=com bindPW: admin-password userSearch: baseDN: ou=users,dc=example,dc=com filter: "(objectClass=person)" username: uid idAttr: uid emailAttr: mail nameAttr: cn ``` ```yaml # values.yaml cloudprem: membership: dex: configOverrides: connectors: - type: oidc id: okta name: Okta config: issuer: https://your-org.okta.com clientID: your-client-id clientSecret: your-client-secret redirectURI: https://dex.your-domain.com/callback scopes: - openid - profile - email ``` ### Using Your Own OIDC Provider If you already have an OIDC provider (e.g., Keycloak), you can bypass Dex entirely: ```yaml # values.yaml global: platform: membership: relyingParty: scheme: "https" host: "keycloak.example.com" path: "/realms/master" cloudprem: membership: dex: enabled: false config: oidc: clientId: "membership" clientSecret: "your-client-secret" scopes: - openid - email - federated:id ``` The `global.platform.membership.relyingParty` defines the OIDC issuer URL. Use the `path` field when your provider uses a sub-path (e.g., Keycloak realms). ### Membership Configuration You can customize authentication behavior and stack defaults: ```yaml cloudprem: membership: config: oidc: clientId: "membership" clientSecret: "changeMe" scopes: - openid - email - federated:id auth: loginWithSSO: false # Enable email-based SSO selector tokenValidity: accessToken: "5m" refreshToken: "72h" stack: minimalStackModules: - Auth - Ledger - Gateway additionalModules: - Payments - Stargate fctl: true # Enable fctl CLI support ``` ## Service Ports | Service | Port | Protocol | |---------|------|----------| | Membership | 8080 | HTTP | | Membership | 8082 | gRPC | | Dex | 5556 | HTTP | | Portal | 3000 | HTTP | | Console V3 | 3000 | HTTP | ## Initial Setup with Membership CLI The Membership CLI provides commands to configure your self-hosted deployment. You run these commands directly inside the Membership container. ### Accessing the Membership CLI Connect to the Membership container: ```bash kubectl exec -it -n formance-system deployment/membership -- sh ``` ### Bootstrap with the wizard Membership ships a single `admin wizard` command that creates the resources you need to get a fresh deployment running — users, organizations, private regions, and stacks. It runs either interactively or against a YAML config you supply, and the chart embeds the same config path so the install job can run it automatically on first boot. #### Interactive ```bash membership admin wizard ``` The wizard prompts you through a menu — create user, manage organizations, manage private regions, manage stacks. Each step asks for the inputs it needs (email, name, base URL, modules). Use this when you want a one-off setup with no values file changes. #### From a config file (idempotent) For repeatable installs, declare the resources in a YAML file and pass it with `--config`: ```bash membership admin wizard --config /path/to/wizard.yaml ``` ```yaml # wizard.yaml wizard: users: - id: "admin-uuid" email: "admin@your-company.com" role: "ADMIN" organizations: - id: "my-org" ownerId: "admin-uuid" name: "My Company" # Built-in policy ID applied to users who join the org without an explicit # policy assignment. 10 = OrganizationAdminStackAdmin, 8 = OrganizationAdmin, # 4 = OrganizationGuest, etc. See "Built-in policies" in /manage/identity/rbac. defaultPolicyId: 10 regions: - id: "region-1" name: "default" baseUrl: "https://formance.your-domain.com" organizationId: "my-org" stacks: - id: "production" name: "production" organizationId: "my-org" regionId: "region-1" version: "v3.1" modules: ["Auth", "Ledger", "Gateway"] ``` The wizard reconciles each section idempotently: existing resources are checked against the config (and the wizard errors out if there's a mismatch), missing resources are created. Re-running with the same config is safe. User-level `role` is the **system role** (whether the user is a platform admin), distinct from the org-level RBAC policy assigned via `defaultPolicyId`: - `ADMIN` — full system access - `USER` — standard access The chart embeds this same YAML directly under `cloudprem.membership.config.wizard.setup` in your `values.yaml` and runs the wizard job once on first install — see [Wizard Bootstrap (Alternative to CLI)](#wizard-bootstrap-alternative-to-cli) below for the values-file variant used by the `ee-demo.yaml` profile. ### Available commands The full membership CLI surface: ``` membership ├── serve # Run the membership HTTP service ├── version # Print version info ├── storage │ └── migrations # Database migration commands ├── admin │ └── wizard # Interactive or config-driven bootstrap ├── stargate # Stargate relay service ├── stream # Internal event-stream commands ├── cycle # Stack-cycle worker ├── gc # Garbage collection └── report # Reporting tools ``` Run `membership --help` to see flags for any subcommand. ### Bootstrap on Membership v1.x Membership v1.x predates the `admin wizard` subcommand. There's no CLI for it — bootstrapping is a two-step process: pre-seed an admin user in the database, then call the membership HTTP API to create the organization, private region, and stack. v1.x is no longer the recommended path. If you can upgrade the Helm chart to a release that ships Membership `v2.0.0` or later (released February 2026), the steps below collapse into a single `admin wizard` invocation. Treat this section as bootstrap guidance for installations stuck on v1.x while you plan the upgrade. #### 1. Login via fctl After deploying the chart and configuring Dex, the very first `fctl login` against your Membership URL creates the initial user + organization in one step: ```bash fctl login --membership-uri "https://membership./api" ``` A browser opens for the Dex sign-in. Once you complete it, Membership writes a `users` row for that email and an `organizations` row owned by that user. No SQL needed for those two. #### 2. (Optional) Configure domain auto-join If you want every subsequent sign-in from `your-company.com` to land directly in the freshly-created organization with default access, set the org's domain: ```bash fctl cloud organizations list fctl cloud organizations update "" \ --domain=your-company.com \ --default-organization-role=GUEST \ --default-stack-role=GUEST ``` Allowed role values are `GUEST` and `ADMIN`. #### 3. Create a region and a stack via SQL v1.x has no API or CLI for region / stack creation — those rows go directly into the Membership database. Open a `psql` session against the PostgreSQL the chart provisioned (or your external one), then run: ```sql -- Create a private region pointing at the URL where the regional agent -- will reach Membership. The subquery picks the org auto-created during -- step 1 (`fctl login`); adjust the LIMIT 1 filters if you've already -- created multiple orgs. INSERT INTO membership.regions (id, base_url, name, organization_id, creator_id, active) VALUES ( gen_random_uuid(), 'https://', 'default', (SELECT id FROM membership.organizations LIMIT 1), (SELECT owner_id FROM membership.organizations LIMIT 1), true ); -- Create the stack against the region just inserted. INSERT INTO membership.stacks (name, organization_id, id, region_id, client_secret) VALUES ( 'default', (SELECT id FROM membership.organizations LIMIT 1), 'demo', -- stack id; update if needed (SELECT id FROM membership.regions LIMIT 1), gen_random_uuid()::text ); ``` `created_at` / `updated_at` populate from `DEFAULT now()`. `state` / `status` / `expected_status` populate from their enum defaults (`'ACTIVE'` / `'UNKNOWN'` / `'READY'`). Replace `` with the domain the regional agent will reach Membership at. Save the organization id and stack id — both are needed when you deploy the Operator and the Stack in the next steps. The membership database schema evolves across v1.x point releases — columns like `stargate_enabled` (dropped in v1.5+), `secret_hash` on regions (dropped at v1.11), and `state` / `status` (renamed from integer to enum in late 2023) have all moved over time. This recipe uses the minimum stable column set; if `psql` reports an unknown column, drop it from the INSERT. The Formance docs-v1 archive at chart v3.0 has a longer variant with `stargate_enabled` and the explicit `'ACTIVE' / 'READY' / 'READY'` state values, which only works against membership v1.3-era schemas. The shorter form above relies on defaults and works across the full v1.x range we still observe in production. ## Configure Organization with fctl After the initial setup, you can use `fctl` for additional configuration. ### Login to Your Cluster ```bash fctl login --membership-uri "https://membership./api" ``` ### Configure Auto-Login for Your Domain ```bash fctl cloud organizations update "" \ --domain=your-company.com \ --default-organization-role=GUEST \ --default-stack-role=GUEST ``` ## Wizard Bootstrap (Alternative to CLI) Instead of using the Membership CLI, you can auto-create users, organizations, regions, and stacks on first install using the wizard in your values file: ```yaml cloudprem: membership: dex: configOverrides: staticPasswords: - email: "admin@example.com" hash: "$2a$10$..." # bcrypt hash of password username: "admin" userID: "unique-uuid" role: ADMIN config: wizard: setup: | users: - id: "unique-uuid" email: "admin@example.com" role: "ADMIN" organizations: - id: "my-org" ownerId: "unique-uuid" name: "My Organization" regions: - id: "region-1" name: "production" organizationId: "my-org" baseUrl: "https://formance.example.com" stacks: - id: "main" name: "main" organizationId: "my-org" regionId: "region-1" version: "v3.1" ``` The wizard job runs once on first install and creates all the configured resources. This is the approach used by the `ee-demo.yaml` profile. ## Accessing the Console After setup, access the Console at: ``` https://console. ``` Log in with your configured identity provider credentials. ## Verifying the Setup ### Check Component Health ```bash # Check all pods are running kubectl get pods -n formance-system # Check services kubectl get svc -n formance-system # Check ingresses kubectl get ingress -n formance-system ``` ### Test Authentication ```bash # Test Dex curl -I "https://dex./.well-known/openid-configuration" # Test Membership API curl -I "https://membership./api/_info" # Test Console curl -I "https://console." ``` ## Troubleshooting 1. Verify ingress is configured: ```bash kubectl get ingress -n formance-system ``` 2. Check TLS certificate: ```bash kubectl describe certificate -n formance-system ``` 3. Check Console logs: ```bash kubectl logs -n formance-system -l app=console ``` 1. Check Dex configuration: ```bash kubectl get secret -n formance-system membership-dex-config -o jsonpath='{.data.config\.yaml}' | base64 -d ``` 2. Verify Dex is accessible: ```bash curl "https://dex./.well-known/openid-configuration" ``` 3. Check Dex logs: ```bash kubectl logs -n formance-system -l app.kubernetes.io/name=dex ``` 1. Check Membership logs: ```bash kubectl logs -n formance-system -l app=membership ``` 2. Verify the migration state from inside the container: ```bash kubectl exec -it -n formance-system deployment/membership -- membership storage migrations version ``` 3. Verify PostgreSQL is running: ```bash kubectl get pods -n formance-system -l app.kubernetes.io/name=postgresql ``` --- ## Requirements Source: https://docs.formance.com/deploy/self-hosted/requirements Before deploying the Formance Platform, make sure your cluster meets the following requirements. ## Kubernetes Cluster - Kubernetes **+** - Helm **** - `kubectl` configured to access your cluster ## Ingress Controller (optional) If you want to expose your stacks outside of your Kubernetes cluster, you'll need an Ingress Controller. At Formance, we use Traefik, but any controller that supports standard `Ingress` objects works. SSL certificate management can be done either on the LoadBalancer in front of the Ingress Controller or directly by your Ingress Controller. The Formance Operator creates standard `Ingress` objects. Alternative ingress controllers work but might require additional configuration not covered in this guide. ## Stateful Dependencies The Formance modules require certain stateful dependencies. The unified `formance` Helm chart can bundle some of these for you, or you can bring your own. ### PostgreSQL **Version 14 or higher is required.** The Formance chart includes a bundled [Bitnami PostgreSQL](https://github.com/bitnami/charts/tree/main/bitnami/postgresql) instance, enabled by default (`postgresql.enabled: true`). This is convenient for development and testing but **not recommended for production**. For production, use a managed PostgreSQL service (e.g., AWS RDS, Google Cloud SQL, Azure Database) and disable the bundled instance. See the [PostgreSQL setup guide](/deploy/self-hosted/infrastructure/postgresql) for configuration details. ### NATS (optional) NATS provides event streaming capabilities. It's disabled by default and can be enabled with `global.nats.enabled: true`. When enabled, configure the NATS URL and authentication: ```yaml global: nats: enabled: true url: "nats://nats.example.com:4222" ``` ## Dependencies by Edition | Dependency | Community Edition | Enterprise Edition | Bundled in Chart | |------------|:-----------------:|:------------------:|:----------------:| | PostgreSQL 14+ | Required | Required | Yes (`postgresql.enabled: true`) | | NATS | Optional | Optional | No (external only) | | Ingress Controller | Optional | Recommended | No | | TLS Certificates | Optional | Recommended | No | | License Key | — | Required | — | The Enterprise Edition deploys additional control plane components (Membership, Console, Dex) that all require PostgreSQL. The bundled PostgreSQL instance serves both the data plane (stacks) and the control plane. --- ## PostgreSQL Source: https://docs.formance.com/deploy/self-hosted/infrastructure/postgresql *Version 14 or higher is required.* ## Bundled PostgreSQL The unified `formance` Helm chart includes a [Bitnami PostgreSQL](https://github.com/bitnami/charts/tree/main/bitnami/postgresql) instance enabled by default: ```yaml postgresql: enabled: true # default ``` This deploys a single-node PostgreSQL server in your `formance-system` namespace. It's useful for development and testing, but **not recommended for production** workloads. The bundled instance uses the credentials defined in `global.postgresql.auth`: ```yaml global: postgresql: host: "postgresql" auth: username: formance password: formance database: formance ``` The bundled PostgreSQL does not include persistent storage by default. Do not rely on it for production data. ## External PostgreSQL For production, use a managed PostgreSQL service and disable the bundled instance: ```yaml postgresql: enabled: false global: postgresql: host: "my-postgres.example.com" additionalArgs: "sslmode=require" auth: username: "formance" password: "your-secure-password" database: "formance" existingSecret: "" # or reference a Kubernetes secret secretKeys: adminPasswordKey: "" userPasswordKey: "" service: ports: postgresql: 5432 ``` Use `global.postgresql.auth.existingSecret` to reference a Kubernetes Secret instead of storing passwords in your values file. ## Configure PostgreSQL URI via Helm Values The Formance Operator uses `Settings` CRDs to configure database connections for each stack. You can create these Settings declaratively through Helm values using `regions.settings`: ```yaml regions: settings: postgres-uri: key: "postgres.*.uri" value: "postgresql://formance:formance@my-postgres.example.com:5432?disableSSLMode=true" ``` This creates a `Settings` resource that applies the PostgreSQL URI to **all stacks** and **all modules** (the `*` wildcard). ### Scope to Specific Stacks To target a specific stack, add a `stacks` field: ```yaml regions: settings: postgres-uri: key: "postgres.*.uri" stacks: - "my-stack" value: "postgresql://formance:formance@my-postgres.example.com:5432?disableSSLMode=true" ``` ### Per-Module Configuration Use the module name instead of `*` to configure different PostgreSQL servers per module: ```yaml regions: settings: postgres-ledger: key: "postgres.ledger.uri" value: "postgresql://formance:formance@pg-ledger.example.com:5432?disableSSLMode=true" postgres-payments: key: "postgres.payments.uri" value: "postgresql://formance:formance@pg-payments.example.com:5432?disableSSLMode=true" ``` ## Create Database Settings Manually If you prefer to manage Settings CRDs directly with `kubectl`, you can create them manually. ### Option 1: Use the same server for all modules Set up a PostgreSQL cluster for all modules of the `formance-dev` stack. Each module gets its own database, named `{stackName}-{module}`. The database name uses the format `{stackName}-{module}`. ```yaml apiVersion: formance.com/v1beta1 kind: Settings metadata: name: formance-dev-postgres-uri spec: key: postgres.*.uri stacks: - 'formance-dev' value: postgresql://formance:formance@postgresql.formance-system.svc:5432?disableSSLMode=true ``` ### Option 2: Use different servers for each module ```yaml --- apiVersion: formance.com/v1beta1 kind: Settings metadata: name: formance-dev-ledger-postgres-uri spec: key: postgres.ledger.uri stacks: - 'formance-dev' value: postgresql://formance:formance@postgresql-ledger.formance-system.svc:5432?disableSSLMode=true --- apiVersion: formance.com/v1beta1 kind: Settings metadata: name: formance-dev-payments-postgres-uri spec: key: postgres.payments.uri stacks: - 'formance-dev' value: postgresql://formance:formance@postgresql-payments.formance-system.svc:5432?disableSSLMode=true ``` ### Option 3: Use PostgreSQL on AWS RDS with an IAM role ```yaml apiVersion: v1 kind: ServiceAccount metadata: name: aws-rds-access-role namespace: formance-system labels: formance.com/stack: any annotations: eks.amazonaws.com/role-arn: arn:aws:iam::AWS_ACCOUNT_ID:role/AWS_ROLE_NAME --- apiVersion: formance.com/v1beta1 kind: Settings metadata: name: formance-dev-aws-service-account spec: stacks: - formance-dev key: aws.service-account value: aws-rds-access-role --- apiVersion: formance.com/v1beta1 kind: Settings metadata: name: formance-dev-postgres-uri spec: key: postgres.*.uri stacks: - 'formance-dev' value: postgresql://formance@postgresql.formance-system.svc:5432 ``` --- ## Message Broker Source: https://docs.formance.com/deploy/self-hosted/infrastructure/message-broker The broker is the messaging system that the Formance stack uses to communicate between its modules. The Formance stack supports both [NATS](https://nats.io/) and [Kafka](https://kafka.apache.org/) as brokers. The broker sends messages between the different modules of the Formance stack. The producers are Ledger, Gateway, and Payments. The consumers are Benthos, Orchestration and Webhooks. Stream names use the format `{stackName}-{module}`. ## Option 1: NATS *Version 2.6 or higher with Jetstream is required.* The recommended way to spin up a NATS deployment is through the official NATS helm [chart](https://artifacthub.io/packages/helm/nats/nats). Depending on your setup, you may need to activate Jetstream mode on your NATS deployment manually. Jetstream is required for the resources deployed by the Formance Operator to function properly. ### Create the NATS settings In this example, you'll set up a configuration for the Broker of the `formance-dev` stack. This configuration will apply to all the modules of this stack. ```yaml apiVersion: formance.com/v1beta1 kind: Settings metadata: name: formance-dev-nats spec: key: broker.dsn stacks: - 'formance-dev' value: nats://nats.formance-system.svc:4222?replicas=3 ``` ## Option 2: Kafka The Formance stack also supports Kafka as a broker. To use Kafka, you need to set up a Kafka cluster and configure the Formance Operator to use it. ### Create the Kafka settings In this example, you'll set up a configuration for the Broker of the `formance-dev` stack. This configuration will apply to all the modules of this stack. ```yaml apiVersion: formance.com/v1beta1 kind: Settings metadata: name: formance-dev-nats spec: key: broker.dsn stacks: - 'formance-dev' value: kafka://kafka.formance-system.svc:9092 ``` --- ## Temporal Source: https://docs.formance.com/deploy/self-hosted/infrastructure/temporal Temporal is required when running the Payments or Flows modules — both ship a worker that registers workflows and activities against a Temporal namespace. If your stack uses neither today, you can skip it and add it later when you enable one of them. The recommended way to spin up a Temporal deployment is through Temporal Cloud, or by using the official Temporal helm [chart](https://github.com/temporalio/helm-charts). ## Create the Temporal settings In this example, you'll set up a configuration for the Orchestration module of the `formance-dev` stack. This configuration will apply to all the modules of this stack. ```yaml apiVersion: formance.com/v1beta1 kind: Settings metadata: name: formance-dev-temporal-dsn spec: key: temporal.dsn stacks: - 'formance-dev' value: temporal://dev-eu-west-1.fsdfsdf.tmprl.cloud:7233/dev-eu-west-1.fsdfsdf? --- apiVersion: formance.com/v1beta1 kind: Settings metadata: name: formance-dev-temporal-tls-crt spec: key: temporal.tls.crt stacks: - 'formance-dev' value: | -----BEGIN CERTIFICATE----- CERTIFICATE_CONTENT -----END CERTIFICATE----- --- apiVersion: formance.com/v1beta1 kind: Settings metadata: name: formance-dev-temporal-tls-key spec: key: temporal.tls.key stacks: - 'formance-dev' value: | -----BEGIN PRIVATE KEY----- PRIVATE_KEY_CONTENT -----END PRIVATE KEY----- ``` --- ## Stack Configuration Source: https://docs.formance.com/deploy/self-hosted/modules/stack ## Overview A Stack represents a set of modules that are deployed together. It is used as a way to group modules and to deploy them together with a consistent set of versions and configurations. When you deploy a Formance module, such as Ledger or Payments, you deploy them within a Stack. This allows you to ensure that they are all deployed with the same versions and configurations. ## Stack Object The first object that needs to be created is the Stack object. This allows linking the different modules and deploying them together. You can find all the available parameters in [the comprehensive CRD documentation](/stack-api-reference/introduction). ```yaml apiVersion: formance.com/v1beta1 kind: Stack metadata: name: formance-dev spec: versionsFromFile: v2.0 ``` During the deployment of the Operator and for its future upgrades, the Versions files are automatically updated according to semver. It is possible to specify a specific version by using the `versionFromFile` field in the Stack file. --- ## Gateway Module Source: https://docs.formance.com/deploy/self-hosted/modules/gateway The Gateway's role is to centralize all incoming connections to your stack. To expose your stack externally, you must route all traffic through it. ## Gateway Object You can find all the available parameters in [the comprehensive CRD documentation](/stack-api-reference/introduction). ```yaml apiVersion: formance.com/v1beta1 kind: Gateway metadata: name: formance-dev spec: stack: formance-dev ``` ### Expose the Gateway with Ingress To expose the Gateway externally, we will use an Ingress object. The Ingress will be the entry point to your stack. ```yaml apiVersion: formance.com/v1beta1 kind: Gateway metadata: name: formance-dev spec: stack: formance-dev ingress: host: YOUR_DOMAIN scheme: http|https ``` ### Serving on Multiple Hosts You can serve a stack on multiple hostnames by adding the `hosts` field alongside `host`. The ingress will contain a rule for each host, and TLS will cover all of them. ```yaml apiVersion: formance.com/v1beta1 kind: Gateway metadata: name: formance-dev spec: stack: formance-dev ingress: host: app.example.com hosts: - app.example.org - app.example.net scheme: https ``` Additional hosts can also be provided via a [Settings](/deploy/self-hosted/config/settings) resource using the `gateway.ingress.hosts` key. Hosts from the setting are merged with those defined on the Gateway CRD. The `{stack}` placeholder is replaced with the stack name. ```yaml apiVersion: formance.com/v1beta1 kind: Settings metadata: name: gateway-extra-hosts spec: key: gateway.ingress.hosts stacks: - '*' value: "{stack}.example.com, {stack}.example.org" ``` --- ## Ledger Module Source: https://docs.formance.com/deploy/self-hosted/modules/ledger Formance Ledger is a real-time money tracking microservice that lets you model and record complex financial transactions. It offers atomic, multi-posting transactions and is programmable using Numscript, a dedicated DSL (Domain Specific Language) to model and templatize such transactions. ## Requirements Formance Ledger requires: - **PostgreSQL**: See configuration guide [here](/deploy/self-hosted/infrastructure/postgresql). - (Optional) **Broker**: See configuration guide [here](/deploy/self-hosted/infrastructure/message-broker). ## Ledger Object You can find all the available parameters in [the comprehensive CRD documentation](/stack-api-reference/introduction). ```yaml apiVersion: formance.com/v1beta1 kind: Ledger metadata: name: formance-dev spec: stack: formance-dev ``` ## Worker Settings (v2.3+) Starting with Ledger v2.3, a separate worker process is deployed alongside the main Ledger API. Configure the worker using the Settings CRD. ### Async Block Hasher Configure the async block hasher behavior: ```yaml apiVersion: formance.com/v1beta1 kind: Settings metadata: name: ledger-worker-async-block-hasher spec: stacks: ["*"] key: ledger.worker.async-block-hasher value: max-block-size=500, schedule="0 */5 * * * *" ``` Available fields: - `max-block-size`: Maximum block size for the async block hasher - `schedule`: Cron schedule for the async block hasher ### Pipelines Configure the worker pipelines behavior: ```yaml apiVersion: formance.com/v1beta1 kind: Settings metadata: name: ledger-worker-pipelines spec: stacks: ["*"] key: ledger.worker.pipelines value: pull-interval=5s, push-retry-period=10s, sync-period=1m, logs-page-size=100 ``` Available fields: - `pull-interval`: Interval between pipeline pulls - `push-retry-period`: Retry period for failed pushes - `sync-period`: Synchronization period - `logs-page-size`: Number of logs per page --- ## Payments Module Source: https://docs.formance.com/deploy/self-hosted/modules/payments Formance Payments is a unified API that abstracts over multiple different payment providers like Stripe and Wise, simplifying the process of building money flows that stitch multiple payment providers together. ## Requirements Formance Payments requires: - **PostgreSQL**: See configuration guide [here](/deploy/self-hosted/infrastructure/postgresql). - **Temporal**: See configuration guide [here](/deploy/self-hosted/infrastructure/temporal). - (Optional) **Broker**: See configuration guide [here](/deploy/self-hosted/infrastructure/message-broker). ## Setting up Temporal Temporal is a required dependency for Formance Payments. The easiest way to set up Temporal with Payments is to use the docker-compose configuration provided in the [payments repository](https://github.com/formancehq/payments), which includes all necessary services: - `temporal` and `temporal-worker` - `postgres` - `payments-migrate` - `payments-worker` - `payments` The docker-compose file in the main stack repository does not include the complete Temporal setup. Make sure to use the configuration from the payments repository instead. ### Alternative: Payments without Temporal If you don't need the latest Payments features and prefer not to manage Temporal infrastructure, you can use an earlier version of Payments that doesn't require Temporal while providing similar core functionality. ## Payments Object You can find all the available parameters in [the comprehensive CRD documentation](/stack-api-reference/introduction). ```yaml apiVersion: formance.com/v1beta1 kind: Payments metadata: name: formance-dev spec: stack: formance-dev ``` --- ## Auth Module Source: https://docs.formance.com/deploy/self-hosted/modules/auth ## Requirements Formance Auth requires: - **PostgreSQL**: See configuration guide [here](/deploy/self-hosted/infrastructure/postgresql). ## Auth Object You can find all the available parameters in [the comprehensive CRD documentation](/deploy/self-hosted/config/api-reference). ```yaml apiVersion: formance.com/v1beta1 kind: Auth metadata: name: formance-dev spec: stack: formance-dev ``` ### Define oAuth2 clients You can define oAuth2 clients to secure your stack. The Auth will then be able to authenticate the incoming requests and forward them to the right service. Here, you can replace `YOUR_ID` and `YOUR_SECRET` with your own arbitrary string values. ```yaml apiVersion: formance.com/v1beta1 kind: AuthClient metadata: name: formance-dev-clients spec: id: YOUR_ID secret: YOUR_SECRET scopes: - ledger:read - ledger:write - payments:read - payments:write stack: formance-dev ``` --- ## Orchestration Module Source: https://docs.formance.com/deploy/self-hosted/modules/orchestration Formance Flows is a handy service that lets you quickly set up end-to-end money flows, without the headache of piecing together APIs and untangling complex systems. With a clever compatibility model, you can easily move value between different ledgers, wallets, and payment processors. Plus, Formance Flows takes care of translating and interpreting the transactions for you. On top of that, Formance Flows comes with flexible workflow capabilities, so you can create complex flows that account for delays or external events, as well as retry and fallback options. All of this helps make your financial management smoother and stress-free. ## Requirements Formance Flows requires: - **PostgreSQL**: See configuration guide [here](/deploy/self-hosted/infrastructure/postgresql). - **Temporal**: See configuration guide [here](/deploy/self-hosted/infrastructure/temporal). - (Optional) **Broker**: See configuration guide [here](/deploy/self-hosted/infrastructure/message-broker). ## Orchestration Object You can find all the available parameters in [the comprehensive CRD documentation](/deploy/self-hosted/config/api-reference). ```yaml apiVersion: formance.com/v1beta1 kind: Orchestration metadata: name: formance-dev spec: stack: formance-dev ``` --- ## Reconciliation Module Source: https://docs.formance.com/deploy/self-hosted/modules/reconciliation ## Requirements Formance Reconciliation requires: - **PostgreSQL**: See configuration guide [here](/deploy/self-hosted/infrastructure/postgresql). - **Ledger**: See configuration guide [here](/deploy/self-hosted/modules/ledger). - **Payments**: See configuration guide [here](/deploy/self-hosted/modules/payments). Reconciliation 2.4.0 requires Ledger 2.4.11 or later for correct point-in-time reads with metadata-filtered account sets. Scheduled rules are executed by the separate Reconciliation worker. The Formance Operator deploys the worker alongside the stateless API service. PostgreSQL coordinates schedules and evaluation jobs, so rolling updates and multiple worker replicas do not duplicate committed evaluation results. ## Reconciliation Object You can find all the available parameters in [the comprehensive CRD documentation](/deploy/self-hosted/config/api-reference). ```yaml apiVersion: formance.com/v1beta1 kind: Reconciliation metadata: name: formance-dev spec: stack: formance-dev ``` --- ## Wallets Module Source: https://docs.formance.com/deploy/self-hosted/modules/wallets Wallets is a fully managed, white-label wallet service to materialize and spend users' funds. It comes with built-in support for multi-currency balances and temporary holds capabilities (and upcoming support for reserved funds and expirable fungibles). It is built on top of the Formance Ledger service and is designed to provide an easy way to add wallet capabilities to your application without having to worry about the underlying transaction structure, providing an opinionated model implementation. ## Requirements Formance Wallets requires: - **Ledger**: See configuration guide [here](/deploy/self-hosted/modules/ledger). ## Wallets Object You can find all the available parameters in [the comprehensive CRD documentation](/deploy/self-hosted/config/api-reference). ```yaml apiVersion: formance.com/v1beta1 kind: Wallets metadata: name: formance-dev spec: stack: formance-dev ``` --- ## Webhooks Module Source: https://docs.formance.com/deploy/self-hosted/modules/webhooks ## Requirements Formance Webhooks requires: - **PostgreSQL**: See configuration guide [here](/deploy/self-hosted/infrastructure/postgresql). - **Broker**: See configuration guide [here](/deploy/self-hosted/infrastructure/message-broker). ## Webhooks Object You can find all the available parameters in [the comprehensive CRD documentation](/deploy/self-hosted/config/api-reference). ```yaml apiVersion: formance.com/v1beta1 kind: Webhooks metadata: name: formance-dev spec: stack: formance-dev ``` --- ## Operator settings reference Source: https://docs.formance.com/deploy/self-hosted/config/settings Settings are Kubernetes custom resources that configure how the Formance Operator runs your stack. Each setting controls a specific behavior — from database connections to deployment replicas to gateway routing. Settings are encoded as strings, but under the hood, each setting can be unmarshalled to a specific type. The basic types are string, number, and bool. There are also two complex structures: * **Maps**: one-level dictionaries with string values. Repeat the `=` pattern for each entry, separated by commas. * **URIs**: used to address external resources (Postgres, Kafka, etc.). See [URI format reference](#uri-format-reference) for details. ## How to apply settings Create a Settings resource and apply it to your cluster. The Operator picks it up automatically. ```yaml apiVersion: formance.com/v1beta1 kind: Settings metadata: name: spec: stacks: - '' key: value: ``` The `spec` fields are: - **stacks**: which stacks use this setting. Use `*` to apply to all stacks. - **key**: the setting key from the tables below. Use `*` as a wildcard to match all modules or services (e.g., `postgres.*.uri`). - **value**: the setting value. ## Available settings ### Infrastructure | Key | Type | Example | Description | |---|---|---|---| | aws.service-account | string | formance-role | AWS IAM role name for service account authentication | | postgres.``.uri | URI | postgresql://user:pass@host:5432 | Postgres database configuration. See [URI format](#postgres-uri-format) | | temporal.dsn | URI | temporal://temporal:7233/default | Temporal server URI. See [URI format](#temporal-uri-format) | | temporal.tls.crt | string | | Temporal TLS certificate (PEM-encoded or Kubernetes secret reference) | | temporal.tls.key | string | | Temporal TLS certificate key (PEM-encoded or Kubernetes secret reference) | | broker.dsn | URI | nats://nats:4222 | Message broker URI. See [URI format](#broker-uri-format) | | opentelemetry.traces.dsn | URI | grpc://otel-collector:4317 | OpenTelemetry collector URI | | opentelemetry.traces.resource-attributes | Map | key1=value1,key2=value2 | OpenTelemetry additional resource attributes | | clear-database | bool | true | **Destructive.** Remove databases when a stack is deleted | | logging.json | bool | true | Configure services to output structured JSON logs | | modules.``.database.connection-pool | Map | max-idle=10, max-idle-time=10s, max-open=10, max-lifetime=5m | Configure database connection pool for each module. See [Golang documentation](https://go.dev/doc/database/manage-connections) | | modules.``.grace-period | string | 5s | Set the grace period before application shutdown | ### Ledger | Key | Type | Example | Description | |---|---|---|---| | ledger.deployment-strategy | string | single | Set the Ledger deployment type | | ledger.logs.max-batch-size | Int | 1024 | Set the max batch size for Ledger log writes | | ledger.api.bulk-max-size | Int | 100 | Set the max number of operations per bulk request | | ledger.api.default-page-size | Int | 15 | Set the default page size for Ledger list endpoints | | ledger.api.max-page-size | Int | 100 | Set the maximum allowed page size for Ledger list endpoints | | ledger.experimental-features | Bool | true | Enable experimental features | | ledger.experimental-numscript | Bool | true | Enable new numscript interpreter | | ledger.experimental-numscript-flags | Array | experimental-overdraft-function experimental-get-asset-function | Enable numscript interpreter flags | | ledger.experimental-exporters | Bool | true | Enable new exporters feature | | ledger.worker.async-block-hasher | Map | max-block-size=1000, schedule="0 * * * * *" | Configure async block hasher for the Ledger worker (v2.3+). Fields: `max-block-size`, `schedule` | | ledger.worker.pipelines | Map | pull-interval=5s, push-retry-period=10s, sync-period=1m, logs-page-size=100 | Configure pipelines for the Ledger worker (v2.3+). Fields: `pull-interval`, `push-retry-period`, `sync-period`, `logs-page-size` | | transactionplane.worker-enabled | bool | false | Enable the embedded worker inside the transactionplane server to run a single service | ### Payments | Key | Type | Example | Description | |---|---|---|---| | payments.encryption-key | string | | Set the encryption key for Payments data | | payments.worker.temporal-max-concurrent-workflow-task-pollers | Int | | Set max concurrent workflow task pollers | | payments.worker.temporal-max-concurrent-activity-task-pollers | Int | | Set max concurrent activity task pollers | | payments.worker.temporal-max-slots-per-poller | Int | | Set max slots per poller | | payments.worker.temporal-max-local-activity-slots | Int | | Set max local activity slots | ### Orchestration | Key | Type | Example | Description | |---|---|---|---| | orchestration.max-parallel-activities | Int | 10 | Set max parallel Temporal activities on orchestration workers | ### Deployments and jobs | Key | Type | Example | Description | |---|---|---|---| | deployments.``.replicas | string | 2 | Set the number of pod replicas for the deployment | | deployments.``.semconv-metrics-names | Bool | true | Enable semantic convention metrics names by setting SEMCONV_METRICS_NAME env var to true in all containers | | deployments.``.spec.template.annotations | Map | firstannotation=X, anotherannotation=X | Custom annotations for deployment pod templates | | deployments.``.spec.template.spec.termination-grace-period-seconds | Int | 30 | Set the termination grace period for the deployment | | deployments.``.init-containers.``.resource-requirements | Map | cpu=X, mem=X | Resource requirements (CPU and memory) for init containers | | deployments.``.containers.``.resource-requirements | Map | cpu=X, mem=X | Resource requirements (CPU and memory) for containers | | deployments.``.init-containers.``.run-as | Map | user=X, group=X | Security context (user and group IDs) for init containers | | deployments.``.containers.``.run-as | Map | user=X, group=X | Security context (user and group IDs) for containers | | deployments.``.topology-spread-constraints | Bool | true | Enable topology spread constraints to maximize high availability | | caddy.image | string | | Set the Caddy reverse proxy image | | jobs.``.spec.template.annotations | Map | firstannotation=X, anotherannotation=Y | Custom annotations for job pod templates | | jobs.``.init-containers.``.run-as | Map | user=X, group=X | Security context (user and group IDs) for job init containers | | jobs.``.containers.``.run-as | Map | user=X, group=X | Security context (user and group IDs) for job containers | ### Registries and services | Key | Type | Example | Description | |---|---|---|---| | registries.``.endpoint | string | example.com?pullSecret=foo | Custom endpoint for a specific docker repository | | registries.``.images.``.rewrite | string | formancehq/example | Override the image path for a specific registry | | services.``.annotations | Map | key1=value1,key2=value2 | Custom annotations for Kubernetes services | | services.``.traffic-distribution | string | PreferSameZone | Configure traffic distribution for Kubernetes services (requires Kubernetes 1.34+) | ### Gateway | Key | Type | Example | Description | |---|---|---|---| | gateway.ingress.annotations | Map | cert-manager.io/cluster-issuer=letsencrypt | Custom annotations for the gateway ingress | | gateway.ingress.hosts | string | \{stack\}.example.com,\{stack\}.example.org | Comma-separated list of additional hosts for the gateway ingress. Supports `\{stack\}` placeholder | | gateway.ingress.labels | Map | env=production,team=platform | Custom labels for the gateway ingress | | gateway.ingress.tls.enabled | bool | true | Enable TLS if not enabled at Gateway CRD level | | gateway.caddyfile.trusted-proxies | string | 10.0.0.0/8,192.168.0.0/16 | Comma-separated list of trusted proxy CIDRs. Use `private_ranges` for all private IPv4/IPv6 ranges | | gateway.caddyfile.trusted-proxies-strict | bool | false | Enable strict (right-to-left) parsing of X-Forwarded-For header. Recommended with upstream proxies (HAProxy, Cloudflare, ALB) | | gateway.config.idle-timeout | string | 10m | Idle timeout for client connections (default: 5m). Use Go duration format (e.g., 30s, 5m, 1h) | | gateway.dns.private.enabled | bool | false | Enable generation of private DNS endpoints for the gateway | | gateway.dns.private.dns-names | string | \{stack\}.internal.example.com | DNS name pattern(s) for private DNS endpoints. Comma-separated. Supports `\{stack\}` placeholder | | gateway.dns.private.targets | string | internal-lb.example.com | Target(s) for private DNS records. Comma-separated | | gateway.dns.private.record-type | string | CNAME | DNS record type (e.g., CNAME, A, AAAA) | | gateway.dns.private.provider-specific | Map | alias=true,aws/target-hosted-zone=same-zone | Provider-specific DNS settings for private endpoints | | gateway.dns.private.annotations | Map | external-dns.alpha.kubernetes.io/ttl=300 | Annotations for the private DNSEndpoint resource | | gateway.dns.public.enabled | bool | false | Enable generation of public DNS endpoints for the gateway | | gateway.dns.public.dns-names | string | \{stack\}.example.com | DNS name pattern(s) for public DNS endpoints. Comma-separated. Supports `\{stack\}` placeholder | | gateway.dns.public.targets | string | public-lb.example.com | Target(s) for public DNS records. Comma-separated | | gateway.dns.public.record-type | string | CNAME | DNS record type (e.g., CNAME, A, AAAA) | | gateway.dns.public.provider-specific | Map | alias=true,aws/target-hosted-zone=same-zone | Provider-specific DNS settings for public endpoints | | gateway.dns.public.annotations | Map | external-dns.alpha.kubernetes.io/ttl=300 | Annotations for the public DNSEndpoint resource | | networkpolicies.enabled | bool | true | Enable network micro-segmentation within a Stack namespace. Only the Gateway can reach other services | ### Namespace | Key | Type | Example | Description | |---|---|---|---| | namespace.labels | Map | somelabel=somevalue,anotherlabel=anothervalue | Add static labels to namespace | | namespace.annotations | Map | someannotation=somevalue,anotherannotation=anothervalue | Add static annotations to namespace | ## URI format reference ### Postgres URI format Scheme: postgresql Query params: | Name | Type | Default | Description | | -------------- | ------ | ------- | ---------------------------------------------- | | secret | string | | Specify a secret where credentials are defined | | disableSSLMode | bool | false | Disable SSL on Postgres connection | ### Temporal URI format Scheme: temporal Path: Match the temporal namespace Query params: | Name | Type | Default | Description | |----------------------|--------|---------|-----------------------------------------------------------| | secret | string | | Specify a secret where temporal certificates are defined | | encryptionKeySecret | string | | Specify a secret where temporal encryption key is defined | | initSearchAttributes | string | false | Initialize search attributes on temporal namespace | ### Broker URI format Scheme: nats | kafka #### Broker URI format (nats) Scheme: nats Query params: | Name | Type | Default | Description | | -------- | ------ | ------- | ------------------------------------------------------------------------- | | replicas | number | 1 | Specify the number of replicas to configure on newly created nats streams | #### Broker URI format (kafka) Scheme: kafka Query params: | Name | Type | Default | Description | | ---------------- | ------ | ------- | ---------------------------------------------- | | saslEnabled | bool | false | Specify if SASL authentication must be enabled | | saslUsername | string | | Username for SASL authentication | | saslPassword | string | | Password for SASL authentication | | saslMechanism | string | | Mechanism for SASL authentication | | saslSCRAMSHASize | string | | SCRAM SHA size for SASL authentication | | tls | bool | false | Whether to enable SSL | ## Examples ### Define PostgreSQL clusters Configure a PostgreSQL cluster for the `formance-dev` stack, applying to all modules. Each module is isolated in its own database. Database names use the format `{stackName}-{module}`. {`apiVersion: formance.com/v1beta1 kind: Settings metadata: name: formance-dev-postgres-uri spec: key: postgres.*.uri stacks: - 'formance-dev' value: postgresql://formance:formance@postgresql.formance-system.svc:5432?disableSSLMode=true`} ### Use AWS IAM Role Use an AWS IAM role to connect to the database instead of static credentials. {`apiVersion: v1 kind: ServiceAccount metadata: name: aws-rds-access-role namespace: formance-system labels: formance.com/stack: any annotations: eks.amazonaws.com/role-arn: arn:aws:iam::AWS_ACCOUNT_ID:role/AWS_ROLE_NAME --- apiVersion: formance.com/v1beta1 kind: Settings metadata: name: formance-dev-postgres-uri spec: key: postgres.*.uri stacks: - 'formance-dev' value: postgresql://formance@postgresql.formance-system.svc:5432`} ### Define module resource requests Set resource requests for all containers across all modules in the stack. {`apiVersion: formance.com/v1beta1 kind: Settings metadata: name: formance-dev-resource-requests spec: key: deployments.*.containers.*.resource-requirements.requests stacks: - 'formance-dev' value: cpu=10m,memory=100Mi`} ### Configure database job security context Set the user and group IDs for database create and drop job containers. ```yaml apiVersion: formance.com/v1beta1 kind: Settings metadata: name: formance-dev-database-create-run-as spec: key: jobs.database.containers.create-database.run-as stacks: - 'formance-dev' value: user=1234,group=1234 --- apiVersion: formance.com/v1beta1 kind: Settings metadata: name: formance-dev-database-drop-run-as spec: key: jobs.database.containers.drop-database.run-as stacks: - 'formance-dev' value: user=1234,group=1234 ``` ### Configure DNS Endpoints The operator can generate `externaldns.k8s.io/v1alpha1` DNSEndpoint resources for your Gateway components, managed by the external-dns operator. You can configure private and public DNS endpoints independently. The external-dns operator must be installed in your cluster. The Operator creates DNSEndpoint resources, but external-dns is responsible for creating the actual DNS records. #### Private DNS Endpoint Configure a private DNS endpoint for the `formance-dev` stack. ```yaml apiVersion: formance.com/v1beta1 kind: Settings metadata: name: formance-dev-dns-private-enabled spec: key: gateway.dns.private.enabled stacks: - 'formance-dev' value: "true" --- apiVersion: formance.com/v1beta1 kind: Settings metadata: name: formance-dev-dns-private-dns-names spec: key: gateway.dns.private.dns-names stacks: - 'formance-dev' value: "\{stack\}-eks-euw1-01.dev.acme.frmnc.net,\{stack\}.dev.acme.frmnc.net" --- apiVersion: formance.com/v1beta1 kind: Settings metadata: name: formance-dev-dns-private-targets spec: key: gateway.dns.private.targets stacks: - 'formance-dev' value: "rp-01-eks-euw1-01.dev.acme.frmnc.net" --- apiVersion: formance.com/v1beta1 kind: Settings metadata: name: formance-dev-dns-private-provider-specific spec: key: gateway.dns.private.provider-specific stacks: - 'formance-dev' value: "alias=true,aws/target-hosted-zone=same-zone" --- apiVersion: formance.com/v1beta1 kind: Settings metadata: name: formance-dev-dns-private-annotations spec: key: gateway.dns.private.annotations stacks: - 'formance-dev' value: "service.beta.kubernetes.io/aws-load-balancer-internal=true" ``` #### Public DNS Endpoint Configure a public DNS endpoint for the `formance-dev` stack. ```yaml apiVersion: formance.com/v1beta1 kind: Settings metadata: name: formance-dev-dns-public-enabled spec: key: gateway.dns.public.enabled stacks: - 'formance-dev' value: "true" --- apiVersion: formance.com/v1beta1 kind: Settings metadata: name: formance-dev-dns-public-dns-names spec: key: gateway.dns.public.dns-names stacks: - 'formance-dev' value: "\{stack\}.acme.frmnc.net" --- apiVersion: formance.com/v1beta1 kind: Settings metadata: name: formance-dev-dns-public-targets spec: key: gateway.dns.public.targets stacks: - 'formance-dev' value: "rp-01-eks-euw1-01.dev.acme.frmnc.net" --- apiVersion: formance.com/v1beta1 kind: Settings metadata: name: formance-dev-dns-public-provider-specific spec: key: gateway.dns.public.provider-specific stacks: - 'formance-dev' value: "alias=true,aws/target-hosted-zone=same-zone" ``` #### DNS Settings Details - **DNS Names**: The `dns-names` setting supports the `\{stack\}` placeholder which will be replaced with the actual stack name. You can specify multiple DNS names by separating them with commas. Each DNS name will create a separate endpoint in the DNSEndpoint resource. - **Targets**: Multiple targets can be specified by separating them with commas. All targets will be added to each DNS record endpoint. - **Record Type**: Defaults to `CNAME` if not specified. Common values include `CNAME`, `A`, `AAAA`, `TXT`, etc. - **Provider-Specific Settings**: These are provider-specific DNS configurations. For AWS Route53, common settings include: - `alias=true`: Enable alias records - `aws/target-hosted-zone=same-zone`: Use the same hosted zone for the target - **Annotations**: Annotations added to the DNSEndpoint resource. Useful for provider-specific configurations or metadata. ### Configure Network Policies The operator can create Kubernetes NetworkPolicies to enforce network micro-segmentation within a Stack namespace. When enabled: - All ingress traffic is denied by default - The Gateway is accessible by everyone (entry point) - All other services are only accessible from the Gateway Egress traffic is not restricted. A CNI plugin that supports NetworkPolicies (Calico, Cilium, etc.) must be installed in your cluster. The default CNI in some environments (e.g., Flannel) does not enforce NetworkPolicies. ```yaml apiVersion: formance.com/v1beta1 kind: Settings metadata: name: enable-networkpolicies spec: key: networkpolicies.enabled stacks: - '*' value: "true" ``` When enabled, 3 NetworkPolicies are created in the Stack namespace: | Name | Effect | |------|--------| | `default-deny-ingress` | Denies all ingress traffic to all pods | | `allow-gateway-ingress` | Allows all ingress traffic to pods labeled `app.kubernetes.io/name: gateway` | | `allow-from-gateway` | Allows ingress traffic from gateway pods to all other pods | Since Kubernetes NetworkPolicies are additive, the Gateway receives both `deny-all` and `allow-all`, making it fully accessible. Other services receive `deny-all` and `allow-from-gateway`, restricting access to Gateway only. If no Gateway module is deployed, the policies are still created. The deny-all policy protects all services, and the allow-from-gateway rule has no matching source — resulting in all ingress being blocked, which is the safest default. --- ## API Reference Source: https://docs.formance.com/deploy/self-hosted/config/api-reference ## Packages - [formance.com/v1beta1](#formancecomv1beta1) ## formance.com/v1beta1 Package v1beta1 contains API Schema definitions for the formance v1beta1 API group. It allow to configure a Formance stack. A stack is composed of a [Stack](#stack) resource and some [modules](#modules). Each module can create multiple resources following its needs. See [Other resources](#other-resources). Various parts of the stack can be configured either using the CRD properties or using some [Settings](#settings). Modules : - [Analytics](#analytics) - [Auth](#auth) - [Gateway](#gateway) - [Ledger](#ledger) - [Orchestration](#orchestration) - [Payments](#payments) - [Reconciliation](#reconciliation) - [Stargate](#stargate) - [Wallets](#wallets) - [Webhooks](#webhooks) Other resources : - [AuthClient](#authclient) - [Benthos](#benthos) - [BenthosStream](#benthosstream) - [Broker](#broker) - [BrokerConsumer](#brokerconsumer) - [BrokerTopic](#brokertopic) - [Database](#database) - [GatewayHTTPAPI](#gatewayhttpapi) - [ResourceReference](#resourcereference) - [Versions](#versions) ### Main resources #### Stack Stack represents a formance stack. A Stack is basically a container. It holds some global properties and creates a namespace if not already existing. To do more, you need to create some [modules](#modules). The Stack resource allow to specify the version of the stack. It can be specified using either the field `.spec.version` or the `.spec.versionsFromFile` field (Refer to the documentation of [Versions](#versions) resource. The `version` field will have priority over `versionFromFile`. If `versions` and `versionsFromFile` are not specified, "latest" will be used. | Field | Description | Default | Validation | | --- | --- | --- | --- | | `apiVersion` _string_ | `formance.com/v1beta1` | | | | `kind` _string_ | `Stack` | | | | `metadata` _[ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#objectmeta-v1-meta)_ | Refer to Kubernetes API documentation for fields of `metadata`. | | | | `spec` _[StackSpec](#stackspec)_ | | | | | `status` _[StackStatus](#stackstatus)_ | | | | ##### StackSpec | Field | Description | Default | Validation | | --- | --- | --- | --- | | `debug` _boolean_ | Allow to enable debug mode on the module | false | | | `dev` _boolean_ | Allow to enable dev mode on the moduleDev mode is used to allow some application to do custom setup in development mode (allow insecure certificates for example) | false | | | `version` _string_ | Version allow to specify the version of the componentsMust be a valid docker tag | | | | `versionsFromFile` _string_ | VersionsFromFile allow to specify a formance.com/Versions object which contains individual versionsfor each component.Must reference a valid formance.com/Versions object | | | | `enableAudit` _boolean_ | EnableAudit enable audit at the stack level.Actually, it enables audit on [Gateway](#gateway) | false | | | `disabled` _boolean_ | Disabled indicate the stack is disabled.A disabled stack disable everythingIt just keeps the namespace and the [Database](#database) resources. | false | | ##### StackStatus | Field | Description | Default | Validation | | --- | --- | --- | --- | | `ready` _boolean_ | Ready indicates if the resource is seen as completely reconciled | | | | `info` _string_ | Info can contain any additional like reconciliation errors | | | | `modules` _string array_ | Modules register detected modules | | | #### Settings Settings represents a configurable piece of the stacks. The purpose of this resource is to be able to configure some common settings between a set of stacks. Example : ```yaml apiVersion: formance.com/v1beta1 kind: Settings metadata: name: postgres-uri spec: key: postgres.ledger.uri stacks: - stack0 value: postgresql://postgresql.formance.svc.cluster.local:5432 ``` This example create a setting named `postgres-uri` targeting the stack named `stack0` and the service `ledger` (see the key `postgres.ledger.uri`). Therefore, a [Database](#database) created for the stack `stack0` and the service named 'ledger' will use the uri `postgresql://postgresql.formance.svc.cluster.local:5432`. Settings allow to use wildcards in keys and in stacks list. For example, if you want to use the same database server for all the modules of a specific stack, you can write : ```yaml apiVersion: formance.com/v1beta1 kind: Settings metadata: name: postgres-uri spec: key: postgres.*.uri # There, we use a wildcard to indicate we want to use that setting of all services of the stack `stack0` stacks: - stack0 value: postgresql://postgresql.formance.svc.cluster.local:5432 ``` Also, we could use that setting for all of our stacks using : ```yaml apiVersion: formance.com/v1beta1 kind: Settings metadata: name: postgres-uri spec: key: postgres.*.uri # There, we use a wildcard to indicate we want to use that setting for all services of all stacks stacks: - * # There we select all the stacks value: postgresql://postgresql.formance.svc.cluster.local:5432 ``` Some settings are really global, while some are used by specific module. Refer to the documentation of each module and resource to discover available Settings. ##### Global settings ###### AWS account A stack can use an AWS account for authentication. It can be used to connect to any AWS service we could use. It includes RDS and MSK. To do so, you can create the following setting: ```yaml apiVersion: formance.com/v1beta1 kind: Settings metadata: name: aws-service-account spec: key: aws.service-account stacks: - '*' value: aws-access ``` This setting instruct the operator than there is somewhere on the cluster a service account named `aws-access`. So, each time a service has the capability to use AWS, the operator will use this service account. The service account could look like that : ```yaml apiVersion: v1 kind: ServiceAccount metadata: annotations: eks.amazonaws.com/role-arn: arn:aws:iam::************:role/staging-eu-west-1-hosting-stack-access labels: formance.com/stack: any name: aws-access ``` You can note two things : 1. We have an annotation indicating the role arn used to connect to AWS. Refer to the AWS documentation to create this role 2. We have a label `formance.com/stack=any` indicating we are targeting all stacks. Refer to the documentation of [ResourceReference](#resourcereference) for further information. ###### JSON logging You can use the setting `logging.json` with the value `true` to configure eligible service to log as json. Example: ```yaml apiVersion: formance.com/v1beta1 kind: Settings metadata: name: json-logging spec: key: logging.json stacks: - '*' value: "true" ``` | Field | Description | Default | Validation | | --- | --- | --- | --- | | `apiVersion` _string_ | `formance.com/v1beta1` | | | | `kind` _string_ | `Settings` | | | | `metadata` _[ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#objectmeta-v1-meta)_ | Refer to Kubernetes API documentation for fields of `metadata`. | | | | `spec` _[SettingsSpec](#settingsspec)_ | | | | ##### SettingsSpec | Field | Description | Default | Validation | | --- | --- | --- | --- | | `stacks` _string array_ | Stacks on which the setting is applied. Can contain `*` to indicate a wildcard. | | | | `key` _string_ | The setting Key. See the documentation of each module or [global settings](#global-settings) to discover them. | | | | `value` _string_ | The value. It must have a specific format following the Key. | | | ### Modules #### Analytics Analytics is the Schema for the analytics API | Field | Description | Default | Validation | | --- | --- | --- | --- | | `apiVersion` _string_ | `formance.com/v1beta1` | | | | `kind` _string_ | `Analytics` | | | | `metadata` _[ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#objectmeta-v1-meta)_ | Refer to Kubernetes API documentation for fields of `metadata`. | | | | `spec` _[AnalyticsSpec](#analyticsspec)_ | | | | | `status` _[AnalyticsStatus](#analyticsstatus)_ | | | | ##### AnalyticsSpec AnalyticsSpec defines the desired state of Analytics | Field | Description | Default | Validation | | --- | --- | --- | --- | | `debug` _boolean_ | Allow to enable debug mode on the module | false | | | `dev` _boolean_ | Allow to enable dev mode on the moduleDev mode is used to allow some application to do custom setup in development mode (allow insecure certificates for example) | false | | | `version` _string_ | Version allow to override global version defined at stack level for a specific module | | | | `stack` _string_ | Stack indicates the stack on which the module is installed | | | ##### AnalyticsStatus AnalyticsStatus defines the observed state of Analytics | Field | Description | Default | Validation | | --- | --- | --- | --- | | `ready` _boolean_ | Ready indicates if the resource is seen as completely reconciled | | | | `info` _string_ | Info can contain any additional like reconciliation errors | | | #### Auth Auth represent the authentication module of a stack. It is an OIDC compliant server. Creating it for a stack automatically add authentication on all supported modules. The auth service is basically a proxy to another OIDC compliant server. | Field | Description | Default | Validation | | --- | --- | --- | --- | | `apiVersion` _string_ | `formance.com/v1beta1` | | | | `kind` _string_ | `Auth` | | | | `metadata` _[ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#objectmeta-v1-meta)_ | Refer to Kubernetes API documentation for fields of `metadata`. | | | | `spec` _[AuthSpec](#authspec)_ | | | | | `status` _[AuthStatus](#authstatus)_ | | | | ##### AuthSpec | Field | Description | Default | Validation | | --- | --- | --- | --- | | `debug` _boolean_ | Allow to enable debug mode on the module | false | | | `dev` _boolean_ | Allow to enable dev mode on the moduleDev mode is used to allow some application to do custom setup in development mode (allow insecure certificates for example) | false | | | `version` _string_ | Version allow to override global version defined at stack level for a specific module | | | | `stack` _string_ | Stack indicates the stack on which the module is installed | | | | `delegatedOIDCServer` _[DelegatedOIDCServerConfiguration](#delegatedoidcserverconfiguration)_ | Contains information about a delegated authentication server to use to delegate authentication | | | | `signingKey` _string_ | Allow to override the default signing key used to sign JWT tokens. | | | | `signingKeyFromSecret` _[SecretKeySelector](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#secretkeyselector-v1-core)_ | Allow to override the default signing key used to sign JWT tokens using a k8s secret | | | | `enableScopes` _boolean_ | Allow to enable scopes usage on authentication.If not enabled, each service will check the authentication but will not restrict access following scopes.in this case, if authenticated, it is ok. | false | | ###### DelegatedOIDCServerConfiguration | Field | Description | Default | Validation | | --- | --- | --- | --- | | `issuer` _string_ | Issuer is the url of the delegated oidc server | | | | `clientID` _string_ | ClientID is the client id to use for authentication | | | | `clientSecret` _string_ | ClientSecret is the client secret to use for authentication | | | | `clientSecretFromSecret` _[SecretKeySelector](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#secretkeyselector-v1-core)_ | ClientSecretFromSecret is the client secret to use for authentication | | | ##### AuthStatus | Field | Description | Default | Validation | | --- | --- | --- | --- | | `ready` _boolean_ | Ready indicates if the resource is seen as completely reconciled | | | | `info` _string_ | Info can contain any additional like reconciliation errors | | | | `clients` _string array_ | Clients contains the list of clients created using [AuthClient](#authclient) | | | #### Gateway Gateway is the Schema for the gateways API | Field | Description | Default | Validation | | --- | --- | --- | --- | | `apiVersion` _string_ | `formance.com/v1beta1` | | | | `kind` _string_ | `Gateway` | | | | `metadata` _[ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#objectmeta-v1-meta)_ | Refer to Kubernetes API documentation for fields of `metadata`. | | | | `spec` _[GatewaySpec](#gatewayspec)_ | | | | | `status` _[GatewayStatus](#gatewaystatus)_ | | | | ##### GatewaySpec | Field | Description | Default | Validation | | --- | --- | --- | --- | | `stack` _string_ | Stack indicates the stack on which the module is installed | | | | `debug` _boolean_ | Allow to enable debug mode on the module | false | | | `dev` _boolean_ | Allow to enable dev mode on the moduleDev mode is used to allow some application to do custom setup in development mode (allow insecure certificates for example) | false | | | `version` _string_ | Version allow to override global version defined at stack level for a specific module | | | | `ingress` _[GatewayIngress](#gatewayingress)_ | Allow to customize the generated ingress | | | ###### GatewayIngress | Field | Description | Default | Validation | | --- | --- | --- | --- | | `host` _string_ | Indicates the hostname on which the stack will be served.Example : `formance.example.com` | | | | `scheme` _string_ | Indicate the scheme.Actually, It should be `https` unless you know what you are doing. | https | | | `ingressClassName` _string_ | Ingress class to use | | | | `annotations` _object (keys:string, values:string)_ | Custom annotations to add on the ingress | | | | `tls` _[GatewayIngressTLS](#gatewayingresstls)_ | Allow to customize the tls part of the ingress | | | ###### GatewayIngressTLS | Field | Description | Default | Validation | | --- | --- | --- | --- | | `secretName` _string_ | Specify the secret name used for the tls configuration on the ingress | | | ##### GatewayStatus | Field | Description | Default | Validation | | --- | --- | --- | --- | | `ready` _boolean_ | Ready indicates if the resource is seen as completely reconciled | | | | `info` _string_ | Info can contain any additional like reconciliation errors | | | | `syncHTTPAPIs` _string array_ | Detected http apis. See [GatewayHTTPAPI](#gatewayhttpapi) | | | #### Ledger Ledger is the module allowing to install a ledger instance. The ledger is actually a stateful application on the writer part. So we cannot scale the ledger as we want without prior configuration. So, the ledger can run in two modes : * single instance: Only one instance will be deployed. We cannot scale in that mode. * single writer / multiple reader: In this mode, we will have a single writer and multiple readers if needed. Use setting `ledger.deployment-strategy` with either the value : - single : For the single instance mode. - single-writer: For the single writer / multiple reader mode. Under the hood, the operator create two deployments and force the scaling of the writer to stay at 1. Then you can scale the deployment of the reader to the value you want. | Field | Description | Default | Validation | | --- | --- | --- | --- | | `apiVersion` _string_ | `formance.com/v1beta1` | | | | `kind` _string_ | `Ledger` | | | | `metadata` _[ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#objectmeta-v1-meta)_ | Refer to Kubernetes API documentation for fields of `metadata`. | | | | `spec` _[LedgerSpec](#ledgerspec)_ | | | | | `status` _[LedgerStatus](#ledgerstatus)_ | | | | ##### LedgerSpec | Field | Description | Default | Validation | | --- | --- | --- | --- | | `debug` _boolean_ | Allow to enable debug mode on the module | false | | | `dev` _boolean_ | Allow to enable dev mode on the moduleDev mode is used to allow some application to do custom setup in development mode (allow insecure certificates for example) | false | | | `version` _string_ | Version allow to override global version defined at stack level for a specific module | | | | `stack` _string_ | Stack indicates the stack on which the module is installed | | | | `deploymentStrategy` _[DeploymentStrategy](#deploymentstrategy)_ | Deprecated. | single | | | `locking` _[LockingStrategy](#lockingstrategy)_ | Locking is intended for ledger v1 only | | | ###### DeploymentStrategy _Underlying type:_ _string_ ###### LockingStrategy | Field | Description | Default | Validation | | --- | --- | --- | --- | | `strategy` _string_ | | memory | | | `redis` _[LockingStrategyRedisConfig](#lockingstrategyredisconfig)_ | | | | ###### LockingStrategyRedisConfig | Field | Description | Default | Validation | | --- | --- | --- | --- | | `uri` _string_ | | | | | `tls` _boolean_ | | false | | | `insecure` _boolean_ | | false | | | `duration` _string_ | | | | | `retry` _string_ | | | | ##### LedgerStatus | Field | Description | Default | Validation | | --- | --- | --- | --- | | `ready` _boolean_ | Ready indicates if the resource is seen as completely reconciled | | | | `info` _string_ | Info can contain any additional like reconciliation errors | | | #### Orchestration Orchestration is the Schema for the orchestrations API | Field | Description | Default | Validation | | --- | --- | --- | --- | | `apiVersion` _string_ | `formance.com/v1beta1` | | | | `kind` _string_ | `Orchestration` | | | | `metadata` _[ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#objectmeta-v1-meta)_ | Refer to Kubernetes API documentation for fields of `metadata`. | | | | `spec` _[OrchestrationSpec](#orchestrationspec)_ | | | | | `status` _[OrchestrationStatus](#orchestrationstatus)_ | | | | ##### OrchestrationSpec | Field | Description | Default | Validation | | --- | --- | --- | --- | | `stack` _string_ | Stack indicates the stack on which the module is installed | | | | `debug` _boolean_ | Allow to enable debug mode on the module | false | | | `dev` _boolean_ | Allow to enable dev mode on the moduleDev mode is used to allow some application to do custom setup in development mode (allow insecure certificates for example) | false | | | `version` _string_ | Version allow to override global version defined at stack level for a specific module | | | ##### OrchestrationStatus | Field | Description | Default | Validation | | --- | --- | --- | --- | | `ready` _boolean_ | Ready indicates if the resource is seen as completely reconciled | | | | `info` _string_ | Info can contain any additional like reconciliation errors | | | | `temporalURI` _string_ | | | Type: string | #### Payments Payments is the Schema for the payments API | Field | Description | Default | Validation | | --- | --- | --- | --- | | `apiVersion` _string_ | `formance.com/v1beta1` | | | | `kind` _string_ | `Payments` | | | | `metadata` _[ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#objectmeta-v1-meta)_ | Refer to Kubernetes API documentation for fields of `metadata`. | | | | `spec` _[PaymentsSpec](#paymentsspec)_ | | | | | `status` _[PaymentsStatus](#paymentsstatus)_ | | | | ##### PaymentsSpec | Field | Description | Default | Validation | | --- | --- | --- | --- | | `stack` _string_ | Stack indicates the stack on which the module is installed | | | | `debug` _boolean_ | Allow to enable debug mode on the module | false | | | `dev` _boolean_ | Allow to enable dev mode on the moduleDev mode is used to allow some application to do custom setup in development mode (allow insecure certificates for example) | false | | | `version` _string_ | Version allow to override global version defined at stack level for a specific module | | | | `encryptionKey` _string_ | | | | ##### PaymentsStatus | Field | Description | Default | Validation | | --- | --- | --- | --- | | `ready` _boolean_ | Ready indicates if the resource is seen as completely reconciled | | | | `info` _string_ | Info can contain any additional like reconciliation errors | | | #### Reconciliation Reconciliation is the Schema for the reconciliations API | Field | Description | Default | Validation | | --- | --- | --- | --- | | `apiVersion` _string_ | `formance.com/v1beta1` | | | | `kind` _string_ | `Reconciliation` | | | | `metadata` _[ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#objectmeta-v1-meta)_ | Refer to Kubernetes API documentation for fields of `metadata`. | | | | `spec` _[ReconciliationSpec](#reconciliationspec)_ | | | | | `status` _[ReconciliationStatus](#reconciliationstatus)_ | | | | ##### ReconciliationSpec | Field | Description | Default | Validation | | --- | --- | --- | --- | | `stack` _string_ | Stack indicates the stack on which the module is installed | | | | `debug` _boolean_ | Allow to enable debug mode on the module | false | | | `dev` _boolean_ | Allow to enable dev mode on the moduleDev mode is used to allow some application to do custom setup in development mode (allow insecure certificates for example) | false | | | `version` _string_ | Version allow to override global version defined at stack level for a specific module | | | ##### ReconciliationStatus | Field | Description | Default | Validation | | --- | --- | --- | --- | | `ready` _boolean_ | Ready indicates if the resource is seen as completely reconciled | | | | `info` _string_ | Info can contain any additional like reconciliation errors | | | #### Stargate Stargate is the Schema for the stargates API | Field | Description | Default | Validation | | --- | --- | --- | --- | | `apiVersion` _string_ | `formance.com/v1beta1` | | | | `kind` _string_ | `Stargate` | | | | `metadata` _[ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#objectmeta-v1-meta)_ | Refer to Kubernetes API documentation for fields of `metadata`. | | | | `spec` _[StargateSpec](#stargatespec)_ | | | | | `status` _[StargateStatus](#stargatestatus)_ | | | | ##### StargateSpec | Field | Description | Default | Validation | | --- | --- | --- | --- | | `debug` _boolean_ | Allow to enable debug mode on the module | false | | | `dev` _boolean_ | Allow to enable dev mode on the moduleDev mode is used to allow some application to do custom setup in development mode (allow insecure certificates for example) | false | | | `version` _string_ | Version allow to override global version defined at stack level for a specific module | | | | `stack` _string_ | Stack indicates the stack on which the module is installed | | | | `serverURL` _string_ | | | | | `organizationID` _string_ | | | | | `stackID` _string_ | | | | | `auth` _[StargateAuthSpec](#stargateauthspec)_ | | | | | `tls` _[StargateTLSConfig](#stargatetlsconfig)_ | | | | ###### StargateAuthSpec | Field | Description | Default | Validation | | --- | --- | --- | --- | | `clientID` _string_ | | | | | `clientSecret` _string_ | | | | | `issuer` _string_ | | | | ###### StargateTLSConfig | Field | Description | Default | Validation | | --- | --- | --- | --- | | `disable` _boolean_ | Disable TLS protocol -- use at your own risks, the transmission will be in clear. | | | ##### StargateStatus StargateStatus defines the observed state of Stargate | Field | Description | Default | Validation | | --- | --- | --- | --- | | `ready` _boolean_ | Ready indicates if the resource is seen as completely reconciled | | | | `info` _string_ | Info can contain any additional like reconciliation errors | | | #### Wallets Wallets is the Schema for the wallets API | Field | Description | Default | Validation | | --- | --- | --- | --- | | `apiVersion` _string_ | `formance.com/v1beta1` | | | | `kind` _string_ | `Wallets` | | | | `metadata` _[ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#objectmeta-v1-meta)_ | Refer to Kubernetes API documentation for fields of `metadata`. | | | | `spec` _[WalletsSpec](#walletsspec)_ | | | | | `status` _[WalletsStatus](#walletsstatus)_ | | | | ##### WalletsSpec | Field | Description | Default | Validation | | --- | --- | --- | --- | | `debug` _boolean_ | Allow to enable debug mode on the module | false | | | `dev` _boolean_ | Allow to enable dev mode on the moduleDev mode is used to allow some application to do custom setup in development mode (allow insecure certificates for example) | false | | | `version` _string_ | Version allow to override global version defined at stack level for a specific module | | | | `stack` _string_ | Stack indicates the stack on which the module is installed | | | ##### WalletsStatus WalletsStatus defines the observed state of Wallets | Field | Description | Default | Validation | | --- | --- | --- | --- | | `ready` _boolean_ | Ready indicates if the resource is seen as completely reconciled | | | | `info` _string_ | Info can contain any additional like reconciliation errors | | | #### Webhooks Webhooks is the Schema for the webhooks API | Field | Description | Default | Validation | | --- | --- | --- | --- | | `apiVersion` _string_ | `formance.com/v1beta1` | | | | `kind` _string_ | `Webhooks` | | | | `metadata` _[ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#objectmeta-v1-meta)_ | Refer to Kubernetes API documentation for fields of `metadata`. | | | | `spec` _[WebhooksSpec](#webhooksspec)_ | | | | | `status` _[WebhooksStatus](#webhooksstatus)_ | | | | ##### WebhooksSpec | Field | Description | Default | Validation | | --- | --- | --- | --- | | `stack` _string_ | Stack indicates the stack on which the module is installed | | | | `debug` _boolean_ | Allow to enable debug mode on the module | false | | | `dev` _boolean_ | Allow to enable dev mode on the moduleDev mode is used to allow some application to do custom setup in development mode (allow insecure certificates for example) | false | | | `version` _string_ | Version allow to override global version defined at stack level for a specific module | | | ##### WebhooksStatus | Field | Description | Default | Validation | | --- | --- | --- | --- | | `ready` _boolean_ | Ready indicates if the resource is seen as completely reconciled | | | | `info` _string_ | Info can contain any additional like reconciliation errors | | | ### Other resources #### AuthClient AuthClient allow to create OAuth2/OIDC clients on the auth server (see [Auth](#auth)) | Field | Description | Default | Validation | | --- | --- | --- | --- | | `apiVersion` _string_ | `formance.com/v1beta1` | | | | `kind` _string_ | `AuthClient` | | | | `metadata` _[ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#objectmeta-v1-meta)_ | Refer to Kubernetes API documentation for fields of `metadata`. | | | | `spec` _[AuthClientSpec](#authclientspec)_ | | | | | `status` _[AuthClientStatus](#authclientstatus)_ | | | | ##### AuthClientSpec | Field | Description | Default | Validation | | --- | --- | --- | --- | | `stack` _string_ | Stack indicates the stack on which the module is installed | | | | `id` _string_ | ID indicates the client idIt must be used with oauth2 `client_id` parameter | | | | `public` _boolean_ | Public indicate whether a client is confidential or not.Confidential clients are clients which the secret can be kept secret...As opposed to public clients which cannot have a secret (application single page for example) | false | | | `description` _string_ | Description represents an optional description of the client | | | | `redirectUris` _string array_ | RedirectUris allow to list allowed redirect uris for the client | | | | `postLogoutRedirectUris` _string array_ | RedirectUris allow to list allowed post logout redirect uris for the client | | | | `scopes` _string array_ | Scopes allow to five some scope to the client | | | | `secret` _string_ | Secret allow to configure a secret for the client.It is not required as some client could use some oauth2 flows which does not requires a client secret | | | | `secretFromSecret` _[SecretKeySelector](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#secretkeyselector-v1-core)_ | | | | ##### AuthClientStatus | Field | Description | Default | Validation | | --- | --- | --- | --- | | `ready` _boolean_ | Ready indicates if the resource is seen as completely reconciled | | | | `info` _string_ | Info can contain any additional like reconciliation errors | | | | `hash` _string_ | | | | #### Benthos Benthos is the Schema for the benthos API | Field | Description | Default | Validation | | --- | --- | --- | --- | | `apiVersion` _string_ | `formance.com/v1beta1` | | | | `kind` _string_ | `Benthos` | | | | `metadata` _[ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#objectmeta-v1-meta)_ | Refer to Kubernetes API documentation for fields of `metadata`. | | | | `spec` _[BenthosSpec](#benthosspec)_ | | | | | `status` _[BenthosStatus](#benthosstatus)_ | | | | ##### BenthosSpec | Field | Description | Default | Validation | | --- | --- | --- | --- | | `stack` _string_ | Stack indicates the stack on which the module is installed | | | | `debug` _boolean_ | Allow to enable debug mode on the module | false | | | `dev` _boolean_ | Allow to enable dev mode on the moduleDev mode is used to allow some application to do custom setup in development mode (allow insecure certificates for example) | false | | | `resourceRequirements` _[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#resourcerequirements-v1-core)_ | | | | | `batching` _[Batching](#batching)_ | | | | | `initContainers` _[Container](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#container-v1-core) array_ | | | | | `imagePullSecrets` _[LocalObjectReference](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#localobjectreference-v1-core) array_ | | | | ##### BenthosStatus | Field | Description | Default | Validation | | --- | --- | --- | --- | | `ready` _boolean_ | Ready indicates if the resource is seen as completely reconciled | | | | `info` _string_ | Info can contain any additional like reconciliation errors | | | #### BenthosStream BenthosStream is the Schema for the benthosstreams API | Field | Description | Default | Validation | | --- | --- | --- | --- | | `apiVersion` _string_ | `formance.com/v1beta1` | | | | `kind` _string_ | `BenthosStream` | | | | `metadata` _[ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#objectmeta-v1-meta)_ | Refer to Kubernetes API documentation for fields of `metadata`. | | | | `spec` _[BenthosStreamSpec](#benthosstreamspec)_ | | | | | `status` _[BenthosStreamStatus](#benthosstreamstatus)_ | | | | ##### BenthosStreamSpec | Field | Description | Default | Validation | | --- | --- | --- | --- | | `stack` _string_ | Stack indicates the stack on which the module is installed | | | | `data` _string_ | | | | | `name` _string_ | | | | ##### BenthosStreamStatus | Field | Description | Default | Validation | | --- | --- | --- | --- | | `ready` _boolean_ | Ready indicates if the resource is seen as completely reconciled | | | | `info` _string_ | Info can contain any additional like reconciliation errors | | | #### Broker Broker is the Schema for the brokers API | Field | Description | Default | Validation | | --- | --- | --- | --- | | `apiVersion` _string_ | `formance.com/v1beta1` | | | | `kind` _string_ | `Broker` | | | | `metadata` _[ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#objectmeta-v1-meta)_ | Refer to Kubernetes API documentation for fields of `metadata`. | | | | `spec` _[BrokerSpec](#brokerspec)_ | | | | | `status` _[BrokerStatus](#brokerstatus)_ | | | | ##### BrokerSpec | Field | Description | Default | Validation | | --- | --- | --- | --- | | `stack` _string_ | Stack indicates the stack on which the module is installed | | | ##### BrokerStatus | Field | Description | Default | Validation | | --- | --- | --- | --- | | `ready` _boolean_ | Ready indicates if the resource is seen as completely reconciled | | | | `info` _string_ | Info can contain any additional like reconciliation errors | | | | `uri` _string_ | | | Type: string | | `mode` _[Mode](#mode)_ | Mode indicating the configuration of the nats streamsTwo modes are defined :* ModeOneStreamByService: In this case, each service will have a dedicated stream created* ModeOneStreamByStack: In this case, a stream will be created for the stack and each service will use a specific subject inside this stream | | Enum: [OneStreamByService OneStreamByStack] | | `streams` _string array_ | Streams list streams created when Mode == ModeOneStreamByService | | | ###### Mode _Underlying type:_ _string_ Mode defined how streams are created on the broker (mainly nats) #### BrokerConsumer BrokerConsumer is the Schema for the brokerconsumers API | Field | Description | Default | Validation | | --- | --- | --- | --- | | `apiVersion` _string_ | `formance.com/v1beta1` | | | | `kind` _string_ | `BrokerConsumer` | | | | `metadata` _[ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#objectmeta-v1-meta)_ | Refer to Kubernetes API documentation for fields of `metadata`. | | | | `spec` _[BrokerConsumerSpec](#brokerconsumerspec)_ | | | | | `status` _[BrokerConsumerStatus](#brokerconsumerstatus)_ | | | | ##### BrokerConsumerSpec | Field | Description | Default | Validation | | --- | --- | --- | --- | | `stack` _string_ | Stack indicates the stack on which the module is installed | | | | `services` _string array_ | | | | | `queriedBy` _string_ | | | | | `name` _string_ | As the name is optional, if not provided, the name will be the QueriedBy propertyThis is only applied when using one stream by stack see Mode | | | ##### BrokerConsumerStatus | Field | Description | Default | Validation | | --- | --- | --- | --- | | `ready` _boolean_ | Ready indicates if the resource is seen as completely reconciled | | | | `info` _string_ | Info can contain any additional like reconciliation errors | | | #### BrokerTopic BrokerTopic is the Schema for the brokertopics API | Field | Description | Default | Validation | | --- | --- | --- | --- | | `apiVersion` _string_ | `formance.com/v1beta1` | | | | `kind` _string_ | `BrokerTopic` | | | | `metadata` _[ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#objectmeta-v1-meta)_ | Refer to Kubernetes API documentation for fields of `metadata`. | | | | `spec` _[BrokerTopicSpec](#brokertopicspec)_ | | | | | `status` _[BrokerTopicStatus](#brokertopicstatus)_ | | | | ##### BrokerTopicSpec | Field | Description | Default | Validation | | --- | --- | --- | --- | | `stack` _string_ | Stack indicates the stack on which the module is installed | | | | `service` _string_ | | | | ##### BrokerTopicStatus | Field | Description | Default | Validation | | --- | --- | --- | --- | | `ready` _boolean_ | Ready indicates if the resource is seen as completely reconciled | | | | `info` _string_ | Info can contain any additional like reconciliation errors | | | #### Database Database represent a concrete database on a PostgreSQL server, it is created by modules requiring a database ([Ledger](#ledger) for example). It uses the settings `postgres..uri` which must have the following uri format: `postgresql://[@]@/` Additionally, the uri can define a query param `secret` indicating a k8s secret, than must be used to retrieve database credentials. On creation, the reconciler behind the Database object will create the database on the postgresql server using a k8s job. On Deletion, by default, the reconciler will let the database untouched. You can allow the reconciler to drop the database on the server by using the [Settings](#settings) `clear-database` with the value `true`. If you use that setting, the reconciler will use another job to drop the database. Be careful, no backup are performed! Database resource honors `aws.service-account` setting, so, you can create databases on an AWS server if you need. See [AWS accounts](#aws-account) Once a database is fully configured, it retains the postgres uri used. If the setting indicating the server uri changed, the Database object will set the field `.status.outOfSync` to true and will not change anything. Therefore, to switch to a new server, you must change the setting value, then drop the Database object. It will be recreated with correct uri. | Field | Description | Default | Validation | | --- | --- | --- | --- | | `apiVersion` _string_ | `formance.com/v1beta1` | | | | `kind` _string_ | `Database` | | | | `metadata` _[ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#objectmeta-v1-meta)_ | Refer to Kubernetes API documentation for fields of `metadata`. | | | | `spec` _[DatabaseSpec](#databasespec)_ | | | | | `status` _[DatabaseStatus](#databasestatus)_ | | | | ##### DatabaseSpec | Field | Description | Default | Validation | | --- | --- | --- | --- | | `stack` _string_ | Stack indicates the stack on which the module is installed | | | | `service` _string_ | Service is a discriminator for the created database.Actually, it will be the module name (ledger, payments...).Therefore, the created database will be named `` | | | | `debug` _boolean_ | | false | | ##### DatabaseStatus | Field | Description | Default | Validation | | --- | --- | --- | --- | | `ready` _boolean_ | Ready indicates if the resource is seen as completely reconciled | | | | `info` _string_ | Info can contain any additional like reconciliation errors | | | | `uri` _string_ | | | Type: string | | `database` _string_ | The generated database name | | | | `outOfSync` _boolean_ | OutOfSync indicates than a settings changed the uri of the postgres serverThe Database object need to be removed to be recreated | | | #### GatewayHTTPAPI GatewayHTTPAPI is the Schema for the HTTPAPIs API | Field | Description | Default | Validation | | --- | --- | --- | --- | | `apiVersion` _string_ | `formance.com/v1beta1` | | | | `kind` _string_ | `GatewayHTTPAPI` | | | | `metadata` _[ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#objectmeta-v1-meta)_ | Refer to Kubernetes API documentation for fields of `metadata`. | | | | `spec` _[GatewayHTTPAPISpec](#gatewayhttpapispec)_ | | | | | `status` _[GatewayHTTPAPIStatus](#gatewayhttpapistatus)_ | | | | ##### GatewayHTTPAPISpec | Field | Description | Default | Validation | | --- | --- | --- | --- | | `stack` _string_ | Stack indicates the stack on which the module is installed | | | | `name` _string_ | Name indicates prefix api | | | | `rules` _[GatewayHTTPAPIRule](#gatewayhttpapirule) array_ | Rules | | | | `healthCheckEndpoint` _string_ | Health check endpoint | | | ###### GatewayHTTPAPIRule | Field | Description | Default | Validation | | --- | --- | --- | --- | | `path` _string_ | | | | | `methods` _string array_ | | | | | `secured` _boolean_ | | false | | ##### GatewayHTTPAPIStatus | Field | Description | Default | Validation | | --- | --- | --- | --- | | `ready` _boolean_ | Ready indicates if the resource is seen as completely reconciled | | | | `info` _string_ | Info can contain any additional like reconciliation errors | | | | `ready` _boolean_ | | | | #### ResourceReference ResourceReference is a special resources used to refer to externally created resources. It includes k8s service accounts and secrets. Why? Because the operator create a namespace by stack, so, a stack does not have access to secrets and service accounts created externally. A ResourceReference is created by other resource who need to use a specific secret or service account. For example, if you want to use a secret for your database connection (see [Database](#database), you will create a setting indicating a secret name. You will need to create this secret yourself, and you will put this secret inside the namespace you want (`default` maybe). The Database reconciler will create a ResourceReference looking like that : ``` apiVersion: formance.com/v1beta1 kind: ResourceReference metadata: name: jqkuffjxcezj-qlii-auth-postgres ownerReferences: - apiVersion: formance.com/v1beta1 blockOwnerDeletion: true controller: true kind: Database name: jqkuffjxcezj-qlii-auth uid: 2cc4b788-3ffb-4e3d-8a30-07ed3941c8d2 spec: gvk: group: "" kind: Secret version: v1 name: postgres stack: jqkuffjxcezj-qlii status: ... ``` This reconciler behind this ResourceReference will search, in all namespaces, for a secret named "postgres". The secret must have a label `formance.com/stack` with the value matching either a specific stack or `any` to target any stack. Once the reconciler has found the secret, it will copy it inside the stack namespace, allowing the ResourceReconciler owner to use it. | Field | Description | Default | Validation | | --- | --- | --- | --- | | `apiVersion` _string_ | `formance.com/v1beta1` | | | | `kind` _string_ | `ResourceReference` | | | | `metadata` _[ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#objectmeta-v1-meta)_ | Refer to Kubernetes API documentation for fields of `metadata`. | | | | `spec` _[ResourceReferenceSpec](#resourcereferencespec)_ | | | | | `status` _[ResourceReferenceStatus](#resourcereferencestatus)_ | | | | ##### ResourceReferenceSpec | Field | Description | Default | Validation | | --- | --- | --- | --- | | `stack` _string_ | Stack indicates the stack on which the module is installed | | | | `gvk` _[GroupVersionKind](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#groupversionkind-v1-meta)_ | | | | | `name` _string_ | | | | ##### ResourceReferenceStatus | Field | Description | Default | Validation | | --- | --- | --- | --- | | `ready` _boolean_ | Ready indicates if the resource is seen as completely reconciled | | | | `info` _string_ | Info can contain any additional like reconciliation errors | | | | `syncedResource` _string_ | | | | | `hash` _string_ | | | | #### Versions Versions is the Schema for the versions API | Field | Description | Default | Validation | | --- | --- | --- | --- | | `apiVersion` _string_ | `formance.com/v1beta1` | | | | `kind` _string_ | `Versions` | | | | `metadata` _[ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#objectmeta-v1-meta)_ | Refer to Kubernetes API documentation for fields of `metadata`. | | | | `spec` _object (keys:string, values:string)_ | | | | --- ## Configure OpenTelemetry Source: https://docs.formance.com/manage/observability/opentelemetry ## Overview Formance uses OpenTelemetry to collect and send telemetry data to your monitoring system. [OpenTelemetry](https://opentelemetry.io/) is an open-source observability framework for cloud-native software. It provides a single set of APIs, libraries, agents, and instrumentation to capture distributed traces and metrics from your applications. ## Configuration - An **OpenTelemetry** collector running and accessible from your cluster ### Configuration In this example, you'll set up an OpenTelemetry configuration for the `formance-dev` stack. This configuration will apply to all the modules of this stack. ```yaml apiVersion: formance.com/v1beta1 kind: Settings metadata: name: stacks-otel-collector spec: key: opentelemetry.*.dsn stacks: - "formance-dev" value: grpc://opentelemetry-collector.formance-system.svc:4317?insecure=true ``` --- ## Backup management Source: https://docs.formance.com/deploy/self-hosted/backups When deploying the Formance Platform in production on your own infrastructure, you are responsible for the deployment of a proper backup mechanism. This mechanism will typically be based on the specifics of the cloud provider you are using. However, let's go through some general principles worth following. PostgreSQL databases must be backed up first. If you're using Flows, you also need to back up Temporal data correctly (or use their Cloud version). ## Snapshots We recommend using snapshots. This will enable you to restore all your data at once and have a consistent backup. Most cloud providers support snapshot backups on their databases. ## Replication It is recommended to replicate your data in multiple locations. This will ensure that you can recover from a disaster in one location. For example, you could replicate your data in multiple regions of the same cloud provider, or even in multiple cloud providers. Based on your cloud provider, you might have access to specific replication mechanisms. For example, AWS offers [Cross-Region Replication](https://docs.aws.amazon.com/aws-backup/latest/devguide/cross-region-backup.html). ## Testing and validation As obvious as it might sound, prior to deploying the Formance Platform in production, you should have tested and validated your backup mechanism. This includes testing the backup procedure itself, but also the restoration procedure. --- ## Traces Source: https://docs.formance.com/manage/observability/traces ## Overview All Formance Platform services emit distributed traces using the [OpenTelemetry](https://opentelemetry.io/) standard. Traces provide end-to-end visibility into request flows across services, helping you debug latency issues, understand service dependencies, and monitor platform behavior. Traces are sent to any backend that supports the OpenTelemetry protocol (OTLP), such as Jaeger, Grafana Tempo, Datadog, or any compatible collector. ## Configuration Traces are configured through the OpenTelemetry settings at the stack level. See [Configure OpenTelemetry](/manage/observability/opentelemetry) for the general setup. ## Stack identification Every trace and span emitted by the platform includes a stack identifier in the format: ``` organizationID-stackID ``` This allows you to filter and correlate traces across services for a specific stack in your observability backend. ## Instrumented services All data plane services emit traces: --- ## Metrics Source: https://docs.formance.com/manage/observability/metrics ## Overview The Formance platform relies on the OpenTelemetry standard to collect and send telemetry data to your monitoring system. OpenTelemetry is an open-source observability framework for cloud-native software. It provides a single set of APIs, libraries, agents, and instrumentation to capture distributed traces and metrics from your applications. ## Metrics --- ## Support Telemetry Source: https://docs.formance.com/manage/observability/support-telemetry ## Overview Support Telemetry forwards traces and metrics from your cluster to Formance. When enabled, Formance support engineers can inspect the runtime behavior of your Formance components, which helps reduce investigation time. Enabling support telemetry does not affect your existing monitoring setup. Your own observability backends continue to receive data as before. Support telemetry requires operator chart version **>= 3.12.0**. ## Telemetry data inventory Traces and metrics are exported unfiltered from the OpenTelemetry Collector. No redaction or sampling is applied. The following data is sent to Formance when support telemetry is enabled: **Always included:** - Span names and trace structure - HTTP method, route, status code, and request duration - HTTP request and response bodies — forwarded as-is, without redaction - gRPC method and status code - Ledger account names and resource identifiers - Transaction, payment, and workflow IDs - Error messages and stack traces - Resource attributes (`service.name`, `service.version`, stack identifier) - Custom resource attributes defined in the `OtelExporterEndpoint` CRD **Conditionally included:** - SQL query text and results when a service runs in debug mode - Secrets, tokens, or credentials if a service instruments them into trace spans Enabling support telemetry exports potentially sensitive operational data outside your cluster. Review the inventory above and ensure this is compatible with your data governance requirements before enabling. ## Enable Support Telemetry Set `global.monitoring.support.enabled` to `true` in your chart values: ```yaml global: monitoring: support: enabled: true ``` Authentication reuses the `global.licence.token` you provided during [installation](/deploy/self-hosted/installation/operator-setup). ## Verify After enabling Support Telemetry, verify that the support exporter is ready: ```bash kubectl get otelexporterendpoints formance-support ``` You should see `READY: true`. ## Disable Support Telemetry Set `enabled` back to `false`: ```yaml global: monitoring: support: enabled: false ``` The operator cleans up all managed collector resources. No telemetry is exported to Formance after disabling. ## Coexistence with Settings Support telemetry works alongside your existing `opentelemetry.traces.dsn` Settings configuration. No migration is required. ## Egress Requirements If your cluster restricts outbound traffic, allow HTTPS (port 443) to `otel-support.internal.frmnc.net`. --- ## Upgrade from the operator Source: https://docs.formance.com/manage/upgrade/from-operator It is good practice to keep the operator up to date with the latest [releases](https://github.com/formancehq/operator/releases) to anticipate any future component upgrades. As there is no immediate impact on the deployed stacks when upgrading the operator, it is advised to simply ensure the operator is running the latest version most of the time. You can download the latest version of Formance directly from [our github repo](https://github.com/formancehq/operator/releases). ```bash helm upgrade regions oci://ghcr.io/formancehq/helm/regions \ --version LATEST_RELEASE \ --namespace formance-system ``` Once the upgrade is complete, you can verify the operator is running the latest version with the usual `kubectl` commands: ```bash kubectl -n formance-system describe deployments operator ``` ## Components upgrade The upgrade process is managed by the operator, which upgrades the components one by one, as specified in the `versions` CRD. Any migration that needs to be carried out will also be managed by the operator. When the Operator is upgraded to the latest version, the patch versions will be automatically applied to all the stacks managed by the Operator. If you wish to change the Minor or Major version, you need to modify your Stack object and specify the new version. The version is represented by the `versionFromFile` object. ```bash kubectl edit stack stack1 ``` Do not update the versions CRD unless you are ready to upgrade your deployment. Once updated, the operator will start the upgrade process. You can follow the process in the `status` section of the `stack` CRD. The status will be updated in real time, and will indicate the health status of the various services, such as whether they are `healthy` or not. For each component, a [rolling upgrade](https://kubernetes.io/docs/tutorials/kubernetes-basics/update/update-intro/) will be performed, meaning that the component will be upgraded one by one, and the service will be kept running during the upgrade process. Note that the rollout process is managed by Kubernetes, and as such, the operator has no control over it. As a result, the upgrade process may be interrupted by Kubernetes, and a service may be unavailable for a short period of time. ## Updating from Operator v1 to Operator v2 If you wish to update the operator from version 1 to version 2, we have managed the major steps of the migration for you. Thus, the Configuration CRD from v1 will be migrated to the Settings CRD of v2, and the same will happen for the Versions CRD. All the Stack CRDs will also be split and migrated to the different Module CRDs of v2. However, it is important to note that the Stack CRDs from v1 will not be deleted, and the Operator will replicate each change to the v2 objects. Therefore, after the migration, you will have both v1 and v2 objects in your cluster. We recommend deleting the v1 objects once you are sure that the migration went smoothly. To migrate the operator from version 1 to version 2, you simply need to update the operator with the following command: ```bash helm upgrade regions oci://ghcr.io/formancehq/helm/regions \ --version LATEST_RELEASE \ --namespace formance-system ``` As soon as Operator v2 starts, it will begin migrating v1 objects to v2 objects, then create the new associated resources. All steps have been designed to be carried out without any service interruption. However, we recommend performing this update in Staging before doing it in Production. If you encounter any issues, do not hesitate to contact us. We will be happy to help you with the migration process. --- ## Database update Source: https://docs.formance.com/manage/upgrade/database This page describes how the operator updates the database schema when deploying a new version of a module. ## Database updates management The database migrations are handled automatically by the Formance Kubernetes Operator. This applies to every service the operator deploys. The services come with the guarantee that the database schema will be updated in a backward-compatible way. This means that the database schema will be updated in a way that the old version of the service will still be able to work with the new schema. ## Database update process The process starts when the operator detects that a new version of a module is set to be deployed. First, the operator starts a Kubernetes Job that runs the database migration. The job is responsible for updating the database schema to the new version. Most of the time, the database migration is done using the `migrate` command embedded in the service. If the database migration fails, the operator will stop the deployment of the new version of the module and will keep the old version running. The database migration is applied as a transaction, so if it fails, the database is not updated and the old version of the module is still able to work with the old schema. When the database migration is complete, the operator will start the deployment of the new version of the module. It ensures that the new version is running properly before sending traffic to it. Once the new version is running, the operator will eventually stop the old version of the module and clean up the resources that are not needed anymore. --- ## Troubleshooting Source: https://docs.formance.com/deploy/self-hosted/troubleshooting There are two places where you can encounter issues with your deployment. ## Option 1: Operator failures If the operator itself encounters an error, it is very likely that there is either a bug in the operator or an issue with your Kubernetes cluster. In this case, you should contact the Formance support team. ## Option 2: Stack failures If the operator is running fine, but the stack is failing, you can troubleshoot the stack by turning on OpenTelemetry export, as described in the [observability section](/manage/observability/opentelemetry). --- ## Try the Sandbox Source: https://docs.formance.com/getting-started/sandbox A sandbox is a free, fully-managed Formance Cloud stack you can use for design, testing, and learning. Same APIs, same modules, same data model as a production stack — with a few production-only enforcement differences called out below. ## What you get - **Every module, ready to use.** Ledger, Payments, Auth, Gateway, and Stargate are enabled by default. The EE modules — Wallets, Flows, Reconciliation, Webhooks, Search — are free to enable on your sandbox with `fctl stack modules enable ` (and `disable` to turn them back off). - **A real HTTPS endpoint.** Each stack gets its own URL on `*..sandbox.formance.cloud`. Call it from anywhere: your laptop, CI, a deployed app. - **The Portal** at [portal.formance.cloud](https://portal.formance.cloud) to manage organizations, stacks, members, and OAuth clients. - **The Console** at [console.formance.cloud](https://console.formance.cloud) to browse ledgers, accounts, transactions, and connector payments visually. The API surface, modules, Numscript semantics, and data model match a production stack: your Numscript programs, transaction validation, and webhooks behave the way they will in production, so you design and test against the real thing rather than a mock. Parity stops at production-only enforcement. Sandbox is more relaxed on some security checks, which makes authentication and authorization the one area where code that passes on sandbox can still fail in production — the OAuth scope difference below is the case that bites in practice. The auth service on sandbox doesn't enforce OAuth2 scopes — a `client_credentials` token request that omits `scope` still gets you a working token. **Production enforces scopes per [RFC 6749 §3.3](https://datatracker.ietf.org/doc/html/rfc6749#section-3.3)**, so the same code that works on sandbox returns `401` against a production stack. If you're using the [Formance SDK](/getting-started/connect-app), this is handled for you. If you're integrating raw OAuth, include `scope` in the token request: ```bash curl --request POST \ --url $STACK_URL/api/auth/oauth/token \ --header 'Content-Type: application/x-www-form-urlencoded' \ --data grant_type=client_credentials \ --data client_id=$CLIENT_ID \ --data client_secret=$CLIENT_SECRET \ --data scope=ledger:read ``` Scopes are `:read` and `:write` — reads accept either, writes require `:write`. For more than one, pass the space-delimited value through `--data-urlencode` so the space is encoded instead of splitting into a second argument: `--data-urlencode 'scope=ledger:write payments:read'`. ## Get a sandbox Sign in (or create an account — it's free) on the Portal: Once you have an account, the [Platform Quick Start](/getting-started/quickstart) walks you through installing `fctl`, authenticating, and creating your first sandbox in a few commands. Sandboxes are sized for design and test, not production. Data inside is correct and the modules behave the same way functionally — but we don't run backups, the stack sleeps after prolonged inactivity (the first request after a sleep takes a little longer to warm up), and it auto-disables after 15 days then deletes after 30. Need yours kept past that window? Ask support to restore it. Production stacks run with the operational discipline you'd expect from core financial infrastructure: backups, no sleep, SOC 2 Type II and ISO 27001 certified, DORA-compliant, and regular third-party penetration testing. → [**Formance Trust Center**](https://trust.formance.com/) — audit reports, sub-processor list, certifications, and the full security posture. ## Next steps --- ## Platform Quick Start Source: https://docs.formance.com/getting-started/quickstart This guide gets you up and running with Formance as fast as possible. By the end, you'll have moved money in a Formance Cloud sandbox using the command line. Install `fctl`, the Formance command-line tool: ```bash brew install formancehq/tap/fctl ``` ```bash # Add the Formance repository echo "deb [trusted=yes] https://apt.fury.io/formance/ /" | sudo tee /etc/apt/sources.list.d/formance.list # Install fctl sudo apt update && sudo apt install fctl ``` Download the latest release from [GitHub Releases](https://github.com/formancehq/fctl/releases) and add the `fctl.exe` binary to your `PATH`. Select your operating system above. Verify the installation: ```bash fctl version ``` Authenticate with Formance Cloud (this opens a browser): ```bash fctl login ``` List the regions available to your account: ```bash fctl cloud regions list ``` Pick a region ending in `-sandbox` for a free development environment, then create the stack: ```bash fctl stack create playground --region="" ``` If you omit `--region`, fctl prompts you interactively from the same list. The sandbox typically provisions in a few minutes — exact time varies with cloud capacity. It includes Ledger, Payments, Auth, and Gateway modules ready to use. Verify your sandbox is ready and get your stack ID: ```bash fctl stack show --name=playground ``` Look for the **ID** field in the output: ``` # Information ID | ecwj | ← This is your stack ID Name | playground | Region | eu-sandbox | Status | ACTIVE | ``` You'll need this stack ID for the next commands. Send $1.00 from `world` to `alice`: ```bash fctl ledger send world alice 100 USD/2 --ledger=quickstart --stack="" ``` `world` is a built-in account that can go negative. This is useful for creating funds out of thin air to get started quickly. In production, you'd fund accounts through actual payment flows instead. `USD/2` means USD with 2 decimal places, so `100` represents $1.00. This is how Formance handles monetary precision without floating-point errors. Check the balances: ```bash fctl ledger volumes list --ledger=quickstart --stack="" ``` You should see `world` with a negative balance (-100) and `alice` with a positive balance (100). That's double-entry bookkeeping in action! **Congratulations!** 🎉 You just executed your first transaction on a programmable ledger. When you're done experimenting, delete your sandbox to free up resources. Each account is limited to one active sandbox. ```bash fctl stack delete --name=playground ``` Sandboxes are automatically disabled after 15 days and deleted after 30 days. Data persistence is not guaranteed. --- ## Connect Your App Source: https://docs.formance.com/getting-started/connect-app This guide shows you how to send transactions programmatically using the Formance SDK. By the end, you'll have a working script that creates a transaction via the API. **Prerequisite:** Complete the [Platform Quick Start](/getting-started/quickstart) first. You need a running sandbox. Create OAuth client credentials to authenticate your application. First, get your stack ID: ```bash fctl stack show --name=playground ``` Look for the **ID** field in the output: ``` # Information ID | ecwj | ← This is your stack ID Name | playground | Region | eu-sandbox | Status | ACTIVE | ``` Create the client using your stack ID: ```bash fctl auth clients create my-app --stack "" ``` You'll see output like: ``` ID | 6a936dfe-xxxx-yyyy-zzzz-9019a1e9b9e3 Name | my-app ``` Copy the **ID** — this is your `Client ID`. Now create a secret for this client: ```bash fctl auth clients secrets create "" app-secret --stack "" ``` Replace `` with the ID from above. You'll see: ``` ID | 3bddd5f6-xxxx-yyyy-zzzz-e8cd839c9d79 Name | app-secret Clear | 20bd58c4-xxxx-yyyy-zzzz-dc05258bc959 ``` Copy the **Clear** value immediately — this is your `Client Secret` and it's only shown once! Finally, get your API endpoint: ```bash fctl stack show --name=playground ``` Look for the URL pattern: `https://xxxxxxxxxx-xxxx.sandbox.formance.cloud` **You now have everything you need:** - `Client ID` — from the first command - `Client Secret` — the "Clear" value from the second command - `API URL` — from `fctl stack show` **Create a new project:** ```bash mkdir formance-demo && cd formance-demo npm init -y npm install @formance/formance-sdk typescript tsx @types/node --save ``` **Open `package.json` and add `"type": "module"` after the `"name"` line:** ```json { "name": "formance-demo", "type": "module", "version": "1.0.0", "main": "index.js", "scripts": { "test": "echo \"Error: no test specified\" && exit 1" }, "keywords": [], "author": "", "license": "ISC", "description": "", "dependencies": { "@formance/formance-sdk": "...", "@types/node": "...", "tsx": "...", "typescript": "..." } } ``` **Create `index.ts`:** ```typescript // Replace with your actual values from Step 1 const formance = new Formance({ serverURL: "https://xxxxxxxxxx-xxxx.sandbox.formance.cloud", // Your stack URL security: { clientID: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", // From: fctl auth client create clientSecret: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", // From: fctl auth client secrets create }, }); async function main() { // Create a transaction: send $1.00 from 'world' to 'alice' const result = await formance.ledger.v2.createTransaction({ ledger: "quickstart", v2PostTransaction: { postings: [ { amount: BigInt(100), // 100 = $1.00 (USD/2 has 2 decimal places) asset: "USD/2", source: "world", destination: "alice", }, ], metadata: { order_id: "ORD-12345", description: "First SDK transaction", }, }, }); console.log("✅ Transaction created!"); console.log("Transaction ID:", result.v2CreateTransactionResponse?.data.id); } main().catch(console.error); ``` **Run it:** ```bash npx tsx index.ts ``` **Getting an authentication error?** Check these common issues: - Make sure your `serverURL` has **no trailing slash** - Verify you created the client with the correct `--stack` flag - Confirm your sandbox is still active with `fctl stack show --name=playground` **Create a new project:** ```bash mkdir formance-demo && cd formance-demo go mod init formance-demo go get github.com/formancehq/formance-sdk-go/v3 ``` **Create `main.go`:** ```go package main "context" "fmt" "log" "math/big" formancesdkgo "github.com/formancehq/formance-sdk-go/v3" "github.com/formancehq/formance-sdk-go/v3/pkg/models/operations" "github.com/formancehq/formance-sdk-go/v3/pkg/models/shared" ) func main() { // Replace with your credentials from Step 1 client := formancesdkgo.New( formancesdkgo.WithServerURL("https://xxxxxxxxxx-xxxx.sandbox.formance.cloud"), formancesdkgo.WithSecurity(shared.Security{ ClientID: formancesdkgo.String("your-client-id"), ClientSecret: formancesdkgo.String("your-client-secret"), }), ) ctx := context.Background() // Create a transaction: send $1.00 from 'world' to 'alice' res, err := client.Ledger.V2.CreateTransaction(ctx, operations.V2CreateTransactionRequest{ Ledger: "quickstart", V2PostTransaction: shared.V2PostTransaction{ Postings: []shared.V2Posting{ { Amount: big.NewInt(100), // 100 = $1.00 (USD/2 has 2 decimal places) Asset: "USD/2", Source: "world", Destination: "alice", }, }, Metadata: map[string]string{ "order_id": "ORD-12345", "description": "First SDK transaction", }, }, }) if err != nil { log.Fatal(err) } fmt.Println("✅ Transaction created!") fmt.Println("Transaction ID:", res.V2CreateTransactionResponse.Data.ID) } ``` **Run it:** ```bash go run main.go ``` **Install the SDK:** ```bash pip3 install formance-sdk-python ``` **Create a new directory and file:** ```bash mkdir formance-demo && cd formance-demo ``` **Create `main.py`:** ```python from formance_sdk_python import SDK from formance_sdk_python.models import shared, operations # Replace with your credentials from Step 1 sdk = SDK( server_url="https://xxxxxxxxxx-xxxx.sandbox.formance.cloud", security=shared.Security( client_id="your-client-id", client_secret="your-client-secret", ), ) # Create a transaction: send $1.00 from 'world' to 'alice' result = sdk.ledger.v2.create_transaction( request=operations.V2CreateTransactionRequest( ledger="quickstart", v2_post_transaction=shared.V2PostTransaction( postings=[ shared.V2Posting( amount=100, # 100 = $1.00 (USD/2 has 2 decimal places) asset="USD/2", source="world", destination="alice", ) ], metadata={ "order_id": "ORD-12345", "description": "First SDK transaction", }, ), ) ) print("✅ Transaction created!") print(f"Transaction ID: {result.v2_create_transaction_response.data.id}") ``` **Run it:** ```bash python main.py ``` **Create a new project with Gradle:** ```bash mkdir formance-demo && cd formance-demo gradle init --type java-application ``` **Add the dependency to `build.gradle`:** ```groovy dependencies { implementation 'com.formance:formance-sdk:+' } ``` **Create `src/main/java/App.java`:** ```java package formance.demo; public class App { public static void main(String[] args) throws Exception { // Replace with your credentials from Step 1 SDK sdk = SDK.builder() .serverURL("https://xxxxxxxxxx-xxxx.sandbox.formance.cloud") .security(Security.builder() .clientID("your-client-id") .clientSecret("your-client-secret") .build()) .build(); // Create a transaction: send $1.00 from 'world' to 'alice' var response = sdk.ledger().v2().createTransaction() .request(V2CreateTransactionRequest.builder() .ledger("quickstart") .v2PostTransaction(V2PostTransaction.builder() .postings(List.of( V2Posting.builder() .amount(BigInteger.valueOf(100)) // 100 = $1.00 .asset("USD/2") .source("world") .destination("alice") .build() )) .metadata(Map.of( "order_id", "ORD-12345", "description", "First SDK transaction" )) .build()) .build()) .call(); System.out.println("✅ Transaction created!"); System.out.println("Transaction ID: " + response.v2CreateTransactionResponse().get().data().id()); } } ``` **Run it:** ```bash gradle run ``` **Create a new project:** ```bash mkdir formance-demo && cd formance-demo dotnet new console dotnet add package FormanceSDK ``` **Replace `Program.cs`:** ```csharp using System; using System.Numerics; using System.Threading.Tasks; using System.Collections.Generic; using FormanceSDK; using FormanceSDK.Models.Components; using FormanceSDK.Models.Requests; class Program { static async Task Main(string[] args) { // Replace with your credentials from Step 1 var sdk = new Formance( serverUrl: "https://xxxxxxxxxx-xxxx.sandbox.formance.cloud", security: new Security() { ClientID = "your-client-id", ClientSecret = "your-client-secret" } ); // Create a transaction: send $1.00 from 'world' to 'alice' var response = await sdk.Ledger.V2.CreateTransactionAsync( new V2CreateTransactionRequest() { Ledger = "quickstart", V2PostTransaction = new V2PostTransaction() { Postings = new List { new V2Posting() { Amount = new BigInteger(100), // 100 = $1.00 Asset = "USD/2", Source = "world", Destination = "alice" } }, Metadata = new Dictionary { { "order_id", "ORD-12345" }, { "description", "First SDK transaction" } } } } ); Console.WriteLine("✅ Transaction created!"); Console.WriteLine($"Transaction ID: {response.V2CreateTransactionResponse?.Data.Id}"); } } ``` **Run it:** ```bash dotnet run ``` **Create a new project:** ```bash mkdir formance-demo && cd formance-demo composer init --no-interaction composer require formance/formance-sdk ``` **Create `index.php`:** ```php setServerURL('https://xxxxxxxxxx-xxxx.sandbox.formance.cloud') ->setSecurity(new Shared\Security( clientID: 'your-client-id', clientSecret: 'your-client-secret', )) ->build(); // Create a transaction: send $1.00 from 'world' to 'alice' $response = $sdk->ledger->v2->createTransaction( new Operations\V2CreateTransactionRequest( ledger: 'quickstart', v2PostTransaction: new Shared\V2PostTransaction( postings: [ new Shared\V2Posting( amount: 100, // 100 = $1.00 (USD/2 has 2 decimal places) asset: 'USD/2', source: 'world', destination: 'alice', ), ], metadata: [ 'order_id' => 'ORD-12345', 'description' => 'First SDK transaction', ], ), ) ); echo "✅ Transaction created!\n"; echo "Transaction ID: " . $response->v2CreateTransactionResponse->data->id . "\n"; ``` **Run it:** ```bash php index.php ``` **Expected output:** ``` ✅ Transaction created! Transaction ID: 2 ``` ## Verify Your Transaction Check that the transaction was recorded using the CLI: ```bash fctl ledger transactions list --ledger=quickstart ``` You should see both your CLI transaction from the Quick Start and your new SDK transaction. **Security Note:** The `client_credentials` grant shown here is for server-to-server communication. Never expose these credentials in client-side code. For web/mobile apps, use the [authorization code flow](https://oauth.net/2/grant-types/authorization-code/). The Formance SDK requests the right scopes for you. If you're calling `/api/auth/oauth/token` directly, you **must** include `scope` in the request — production enforces it per [RFC 6749 §3.3](https://datatracker.ietf.org/doc/html/rfc6749#section-3.3) and tokens without a scope are rejected with `401` on subsequent API calls. ```bash curl --request POST \ --url $STACK_URL/api/auth/oauth/token \ --header 'Content-Type: application/x-www-form-urlencoded' \ --data grant_type=client_credentials \ --data client_id=$CLIENT_ID \ --data client_secret=$CLIENT_SECRET \ --data scope=ledger:write # what the example above needs ``` Scopes are `:read` and `:write`. Reads accept either, writes require `:write` — the create-transaction call above is a `POST`, so a token scoped `ledger:read` gets rejected. `ledger:write` also covers the `GET`s that follow it. Need several modules? Space-delimit them and let curl encode the value: `--data-urlencode 'scope=ledger:write payments:read'`. Sandbox is more permissive (scope enforcement is off there) so this is the most common "works on sandbox, breaks on prod" case — also called out on the [sandbox page](/getting-started/sandbox). --- ## Core Concepts Source: https://docs.formance.com/getting-started/concepts This page introduces Formance modules. ## The Big Picture Formance helps you track and move money. At its core: 1. **Ledger** records where assets are and where they move 2. **Numscript** expresses the intent for how assets should move 3. **Payments** provides a normalized view of external providers like Stripe or bank accounts 4. **Reconciliation** verifies your internal records match external reality 5. **Flows** automates multi-step operations across modules Let's look at each one. --- ## Ledger The Ledger is your source of truth for money. It tracks: - **Accounts**: Named addresses that maintain a balance of assets received or sent (e.g., `users:alice:wallet`, `platform:fees`) - **Transactions**: Records of assets moving between accounts - **Balances**: The net result of all assets received and sent for each account ### How it works Every time money moves, the Ledger creates a transaction with one or more **postings**. A posting is simply: "Move X amount from account A to account B." ```json { "source": "users:alice:wallet", "destination": "merchants:coffee-shop", "amount": 450, "asset": "USD/2" } ``` This moves $4.50 from Alice's wallet to the coffee shop. ### Key principles - **Double-entry**: Every transaction balances. Assets leaving one account equal assets entering another. - **Immutable**: Transactions cannot be edited or deleted. To cancel the effect of a transaction, you create a new transaction that offsets it. This can be a reversal or a more intentional corrective booking depending on your audit requirements. - **Atomic**: Multi-posting transactions succeed or fail together. No partial transfers. ### Account naming Accounts are identified by addresses like `users:alice:wallet` or `orders:12345:pending`. The colons create segments you can use to organize and filter accounts. You don't need to create accounts before using them. The first transaction involving an account creates it automatically. ### Where your ledger lives You post transactions against a **named ledger**, which sits inside a hierarchy: a **stack** is an isolated Formance Platform instance with its own API URL and data; the stack runs the **Ledger service**; the service groups ledgers into **buckets** (one PostgreSQL schema each); and a bucket holds one or more **named ledgers**. Two named ledgers never share accounts, balances, or transaction numbering — the same account name can exist in both without interference. Storage separation is the bucket's job, and write throughput is scoped per named ledger. [Where a ledger lives →](/modules/ledger/core-concepts/ledger#where-a-ledger-lives) [Learn more about Ledger →](/modules/ledger) --- ## Numscript Numscript is a domain-specific language for expressing the **intent** of money movements. Instead of writing code to calculate splits and handle edge cases, you describe what should happen and Numscript produces **deterministic postings**. ### Why use it? Consider paying out a rideshare driver: the rider pays $10, the platform takes 20%, and the driver gets the rest. In Numscript: ```numscript send [USD/2 1000] ( source = @rider:wallet destination = { 20% to @platform:fees remaining to @driver:wallet } ) ``` Numscript captures your intent, ensures the transaction balances, and produces predictable, auditable postings every time. ### Key features - **Percentage splits**: Divide payments automatically (`50% to @a, remaining to @b`) - **Multiple sources**: Pull from several accounts to fund a payment - **Overdraft control**: Express precisely how overdrafts should be handled. Accounts can be bounded (prevent negative balances), allowed a specific limit, or unbounded. - **Metadata**: Attach information to transactions for tracking The `@` symbol denotes an account. `@world` is a special account that is unbounded by default, useful for creating initial balances. [Learn more about Numscript →](/modules/numscript) --- ## Payments The Payments module is a **separate system from the Ledger**. It does not automatically write to your Ledger or affect balances there. Instead, it provides a **normalized data store** of your connections to external payment providers like Stripe, Wise, Adyen, and banks. Think of it as a unified view of: - **Accounts** across all your providers - **Balances** for each external account - **Transactions** as they happen in those systems (via periodic polling) The data in Payments and the data in your Ledger are independent. You use Flows (or your own application logic) to create Ledger transactions based on events from the Payments module when needed. Some providers also support **write operations** like initiating transfers or payouts. Check the [Connector Capabilities](/modules/payments/capabilities) table to see what each provider supports. ### Account types - **Internal accounts**: Accounts you control (e.g., your Stripe balance). You can see balances and potentially initiate transfers from these. - **External accounts**: Destination-only accounts (e.g., a customer's bank account for payouts). You can send money to these but can't see their balance. ### Payment types | Type | Description | |------|-------------| | PAY-IN | Money coming in (e.g., customer payment) | | PAYOUT | Money going out (e.g., paying a vendor) | | TRANSFER | Moving money between your own accounts | ### Cash pools A cash pool groups accounts from different providers together. For example, you might pool your Stripe, PayPal, and bank accounts to see your total available funds across all of them. This is especially useful for reconciliation. [Learn more about Payments →](/modules/payments) --- ## Reconciliation Reconciliation answers a critical question: **Does the money in my ledger match the money in my actual accounts?** Your Ledger tracks what *should* be true. Your payment providers track what *is* true. Reconciliation compares them and flags any differences. ### How it works 1. **Create a policy**: Define which ledger accounts to compare against which cash pool 2. **Run reconciliation**: The system compares balances 3. **Review results**: See if balances match or if there's "drift" (a discrepancy) ### Why it matters - Catch errors before they become problems - Prove to auditors that your records are accurate - Identify missing transactions or duplicate entries Reconciliation is part of Formance Enterprise Edition. [Learn more about Reconciliation →](/modules/reconciliation) --- ## Flows (Orchestration) Flows lets you build automated, multi-step money operations. Instead of writing code to handle each step (and all the error cases), you define a workflow and Flows executes it. ### When to use Flows Use Flows when you need to: - **Chain multiple operations**: Move money in the ledger, then trigger a payout - **Wait for events**: Hold a transaction until a webhook confirms payment - **Add delays**: Release funds 7 days after purchase - **Handle failures**: Retry failed steps or run fallback logic ### Example: Marketplace payout A typical marketplace payout might: 1. Wait for a "delivery confirmed" event 2. Move funds from escrow to the seller's account in the ledger 3. Initiate a payout to the seller's bank account 4. Send a notification With Flows, you define this once and the system handles timing, retries, and error cases. Flows is part of Formance Enterprise Edition. [Learn more about Flows →](/modules/flows) --- ## How They Work Together Here's an example of how the modules connect in a typical e-commerce marketplace: 1. **Customer pays via Stripe** → Stripe processes the card payment 2. **Payments syncs the transaction** → The Payments module polls Stripe's API and picks up the new payment 3. **Flows reacts to the event** → A workflow triggers when the payment is detected 4. **Ledger records the funds** → The workflow creates a transaction moving funds from `world` to `orders:123:pending` 5. **Order ships** → The workflow waits for the shipping confirmation event 6. **Funds released** → Numscript splits the payment between seller (80%) and platform (20%) 7. **Payout initiated** → Flows triggers a payout to the seller's bank (for providers that support it) 8. **Reconciliation runs** → Verifies your ledger balances match actual provider balances ### The role of each module - **Payments** is a **normalized data store** of your external providers. It syncs accounts, balances, and transactions from providers like Stripe, Wise, and banks. It is separate from the Ledger and does not automatically create Ledger entries. Some providers also support initiating transfers and payouts. See the [Connector Capabilities](/modules/payments/capabilities) table for details. - **Ledger** is your **internal source of truth**: it tracks what assets exist and where they move within your system. - **Flows** is for **orchestration**: it reacts to events and coordinates multi-step operations across modules, including creating Ledger transactions based on Payments events. - **Reconciliation** is for **verification**: it compares your Ledger balances against your Payments data to catch discrepancies. --- --- ## How the Modules Fit Together Source: https://docs.formance.com/getting-started/modules-fit-together Three parts of the platform work together, and it helps to think of them as three distinct jobs. - **Ledger is your source of truth.** It records what your business believes to be true about every balance and movement of money, in double-entry form. When there is a disagreement about "how much do we have," the Ledger holds your answer. - **Payments is your window onto the outside world.** It reports what your external payment providers say is happening in the accounts they hold on your behalf. - **Reconciliation is your quality assurance.** It compares the two and reports when they disagree, so you can find the cause before it becomes a problem. A typical money flow touches all three. A payment moves through a provider, and the Payments module reports it in a standard shape. Your application records the matching entries in the Ledger. On your schedule, Reconciliation compares the two and confirms they still agree, or shows you exactly where they do not. Each part has one clear job, and keeping them separate is what keeps each one reliable as the system grows. ## Payments Every payment provider has its own API, its own data formats, and its own vocabulary for the same ideas. One provider's "transfer" is another's "payout" is another's "settlement." Payments sits in front of all of them and presents a single, consistent model of accounts, balances, and payments. You integrate once and read one shape of data, regardless of how many providers sit behind it. Formance builds and maintains native connectors for a growing list of providers (Stripe, Wise, Adyen, and others). The value grows as you add providers. Orchestrating a flow that collects on one provider and pays out through another becomes one piece of work, instead of several separate integrations to build and maintain. ### The Generic Connector When a provider you need is not yet on the native list, the Generic Connector lets you add it yourself rather than wait for Formance to ship one. You map that provider's data to the standard Payments model, and from that point it behaves like any native connector. That gives you: - The same synchronization machinery native connectors use: polling, pagination, and incremental sync. - The provider's accounts, balances, and payments behind the same API as everything else. - The provider's data inside the shared model, so unified reporting and Reconciliation work against it with no special handling. A missing connector never blocks you, and the provider you add joins the same integration surface as the rest rather than living in a silo you maintain alone. ## Reconciliation Reconciliation is a managed service, reached through the API rather than run by hand. You define a **rule**: a control template, the balance sources it compares, and when it runs. A rule can compare Ledger balances against a **cash pool** (a grouping of external payment accounts), check that two groups of Ledger accounts stay in balance with each other, watch an account group against a threshold, or compare any two supported sources, including pool to pool. Each run produces a persisted **evaluation** that passes, fails, or errors. A failure opens an alert per affected asset, keeping the values observed at the time, so your team can investigate, correct, or formally accept the discrepancy. A later passing run closes it. Each balance source is read at an explicit point in time, and the two sides can be read at different instants to absorb the settlement lag between your Ledger and a provider. So you can check the state as of a chosen moment rather than only the present one. Rules, scheduled evaluations, and managed alerts arrived in Reconciliation 2.4.0. Before that, the module exposes a simpler synchronous model: a **policy** selects Ledger accounts and a cash pool, and running it returns a report of the drift between them. ## The Other Modules Two other modules build on this foundation. Wallets is a managed layer over the Ledger for holding user funds. Flows is a workflow engine for multi-step money operations: when a payment arrives in the Payments module, Flows can record the matching Ledger entries, wait for a fulfillment event, release funds after a delay, or trigger a payout — handling retries and failures so your application does not have to. --- ## Getting Help Source: https://docs.formance.com/getting-started/help There are two ways to get help with Formance, depending on what you need. --- ## Architecting for scale Source: https://docs.formance.com/modules/ledger/advanced/architecting-for-scale ## Scaling for writes ### Writes contention The Formance Ledger uses a multi-ledger, single-writer, sequential writes architecture to create an auditable, easy to reason about trail of transactions. The latest stable version of the ledger is optimized for 1K writes per second on an underlying commodity storage instance. As a result, write-heavy applications above this threshold are advised to leverage the Formance Ledger's segmentation capabilities to scale horizontally. This will allow you to create multiple ledgers and to write to them in parallel. ### PostgreSQL Row Locks When posting transactions, you may experience decreased throughput if all transactions use the same source account (e.g., the default `world` account). Even if the account allows overdraft (no balance enforcement), **using the same source account repeatedly creates a bottleneck at the PostgreSQL level**. Every transaction involves writing changes to the `accounts_volumes` table. PostgreSQL applies row-level locks when updating balances, causing concurrent transactions targeting the same account to queue. Locks are applied per (account, asset) pair. High concurrency on a single source account will cause transactions to serialize, significantly reducing throughput. ### Solutions to Increase Throughput #### Option 1: Spread Load Across Multiple Source Accounts Distribute writes across multiple accounts instead of just one: - Use dynamic account identifiers like `@world:` where the suffix is randomly generated - Or use dedicated overdraft source accounts meaningful to your use-case, such as `@provider::payment:` - Set a reasonable pool size (e.g., 20) to balance between performance and account count This reduces contention on a single row and leverages PostgreSQL's ability to parallelize updates across multiple rows. #### Option 2: Disable or Async Log Hashing Log hashing uses a **PostgreSQL advisory lock** on the entire ledger, which can limit write throughput. To disable or make it asynchronous, create a ledger with the appropriate feature configuration: ```bash # Disable log hashing entirely curl -X POST http://localhost:3068/v2/myledger -d '{"features": {"HASH_LOGS": "DISABLED"}}' # Or make it asynchronous curl -X POST http://localhost:3068/v2/myledger -d '{"features": {"HASH_LOGS": "ASYNC"}}' ``` **Benefits**: No advisory lock is taken, resulting in significantly higher write throughput. **Tradeoff**: You lose the ability to cryptographically prove that the logs have not been altered or injected manually. ### Performance Summary | Concern | Description | |---------|-------------| | **PostgreSQL locking bottleneck** | Happens on balance updates; causes queuing if same row (source account) is updated concurrently | | **Mitigation** | Use many source accounts (`@world:`) | | **Log Hashing** | Ensures log immutability, but adds overhead. Can be safely disabled if not needed | | **Consistency** | Guaranteed by PostgreSQL transactional semantics (COMMIT/ROLLBACK) | | **Assets as parallelization tool** | Locks are per (account, asset) → spreading across assets avoids contention | ### Transaction commit cost breakdown As outlined in the [performance model](/modules/ledger/advanced/performance-model), we consider the cost of a write to be `O(N) + W`. Let's consider the following example:
```numscript send [USD/2 9900] ( source = { @users:1234:main @users:1234:vouchers @users:1234:creditcard @world } destination = @payments:1234 ) ```
In this transaction, we are attempting to increase the balance of `@payments:1234` by $99.00, sourcing the funds from three user accounts before defaulting `@world` as the source of last resort. Let's break down the operations that will be performed: * Read the balance of `@users:1234:main` * If the balance of `@users:1234:main` was less than $99.00, read the balance of `@users:1234:vouchers` * If the balance of the previous account was less than $99.00, read the balance of `@users:1234:creditcard` * Model the transaction * Write the new balance of `@users:1234:main` As we can see here, the exact cost of this transaction will depend on the number of accounts that are being sourced from, but it will be at least `O(1) + W` and at most `O(3) + W`. This is because the transaction will need to read the balance of at least one account, and at most three accounts, and then commit the computed transaction to the ledger. When optimizing for write throughput, it is important to keep in mind this cost breakdown and to design your transactions accordingly. ## Scaling for reads ### Chart of accounts While usually not a concern for the scaling of writes, designing a proper chart of accounts is key to the performance of your reads. You'll especially want to make sure that the addresses of your accounts are using properly separated segments, as this will allow you to use wildcards to query your accounts. An example of an inefficient account address would be `users:1234_main`. Implemented instead as `users:1234:main`, the latter will enable the ledger to efficiently query all accounts of a given user by using the `users:1234:*` wildcard or all accounts of all users of a given type by using the `users:*:main` wildcard. Once you've designed your chart of accounts, you can enforce it using a [Ledger Schema](/modules/ledger/working-with/ledger-schema). Schemas ensure transactions only use valid account structures, support pattern validation for dynamic segments, and record which schema version was used for each transaction for auditability. ### Ledgers sizing As a consequence of their append-only nature, ledgers will only grow in size over time indefinitely. Optimal mitigation of this nature is one of the Formance Ledger's design goals. As a result of these optimization efforts, not all queries are negatively impacted by the size of the ledger, as explained in the [performance model](/modules/ledger/advanced/performance-model). It is however to be noted that some queries will be impacted by the size of the ledger, relative to the underlying storage database capacity. In order to equip you with primitives to mitigate this, the Formance Ledger comes with the ability to segment your ledgers, allowing you to create multiple ledgers and to query them in parallel. ### Segmentation strategies Let's consider an example financial application where users can deposit funds into their account and subsequently use them for international transfers. This application is poised to grow in popularity and will need to be able to handle a large number of users. We can devise a segmentation strategy that will allow us to scale our ledgering system horizontally and allow ourselves to handle millions of users without constraining ourselves with impacts on the performance of our queries. A first approach would be to segment our ledgers by users, and club together a certain number of users into a single ledger. Another approach would be to segment our ledgers by time, creating for example a ledger for each year and then for each month as we continue to grow the number of transactions committed per day. The proper segmentation strategy will eventually be a decision that you will have to make based on the nature of your application and the expected growth. --- ## Ledger Source: https://docs.formance.com/modules/ledger/core-concepts/ledger A **ledger** is a named, isolated accounting database. It holds accounts, records transactions, and derives balances from an immutable log. Everything in Formance (accounts, transactions, assets) exists within a ledger. ## What it's for A ledger is not a general-ledger ERP tool. It is a programmable, developer-facing accounting primitive. Use it to track the movement of assets between accounts with immutability, concurrency safety, and an audit trail built in. ## Product Ledger vs General Ledger | Ledger Type | Purpose | |-------------|---------| | **General Ledger (GL)** | Organizes financial events to produce clarity on the financial position of a business. Typically lives in an ERP (Sage) or GL tool (QuickBooks). | | **Product Ledger** | Technological foundation for automated flow of funds. Focuses on scarcity, concurrency, auditability, immutability, and performance. | Formance Ledger enforces double-entry by design: every transaction involves two or more accounts in compensating directions, and balances are derived from an immutable log. It is flexible regarding accounting business rules (account classification, debit/credit conventions), letting you choose how much accounting logic to handle in real-time versus delegating to a GL. Use Formance as a product ledger. Add basic account classification in your [Chart of Accounts](/modules/ledger/core-concepts/chart-of-accounts) to lay a foundation for subsequent data mapping to your GL. ## Where a ledger lives When you post a transaction, you post it against a named ledger — `POST /v2/{ledger}/transactions`. That name is the last step in a hierarchy that determines where the data is stored, what it is isolated from, and what it contends with for writes. - A **stack** is a single, isolated instance of the Formance Platform. It is the top-level tenancy boundary: its own data, its own set of enabled modules, its own API URL. On Formance Cloud a stack is provisioned in a **region**, the cloud location that hosts its infrastructure (see [available regions](/deploy/cloud/overview#available-regions)). Self-hosted stacks have no region — you choose where the infrastructure runs. - The **Ledger service** is one module running inside a stack, alongside any others you enable such as Payments or Reconciliation. - A **bucket** is a storage boundary within the Ledger service, mapped one-to-one onto a PostgreSQL schema. Every ledger belongs to exactly one bucket, and defaults to `_default`. - A **named ledger** is a self-contained book: its own accounts, transactions, and immutable log, with its own transaction numbering starting at 1. This is the resource you address in the API. So: a stack runs the Ledger service, the service organizes ledgers into buckets, and a bucket holds one or more named ledgers. On Formance Cloud, a region hosts the stack; self-hosted, that layer is wherever you run the infrastructure. ## Data Model | Resource | Description | |----------|-------------| | **Accounts** | Containers for assets. See [Accounts](/modules/ledger/core-concepts/accounts). | | **Transactions** | Movements of assets between accounts. See [Transactions & Postings](/modules/ledger/core-concepts/transactions). | | **Logs** | Immutable log entries (`NEW_TRANSACTION`, `SET_METADATA`), the primary source of truth. | ## What the ledger boundary isolates Two named ledgers never interfere with each other's bookkeeping, even in the same bucket. Every row is scoped by ledger name — the uniqueness constraint on accounts is `(ledger, address)` — so `users:alice:wallet` in `ledger-a` and `users:alice:wallet` in `ledger-b` are different accounts with independent balances. There is no cross-ledger transaction, and no cross-ledger balance query. Physical isolation is a separate question, and it is the bucket's job, not the ledger's: | Boundary | Ledgers share… | | --- | --- | | Same bucket | the same PostgreSQL schema and the same tables, scoped by ledger name | | Different buckets in one stack | the same database server, but no tables | | Different stacks | nothing | If you need ledgers to be separable at the storage layer — per-tenant data, independent retention, or dropping one client's data without touching another's — put them in different buckets. See [Data isolation with buckets](/modules/ledger/working-with/data-isolation-buckets). ## Data immutability Each log entry produces a hash from its data combined with the previous entry's hash, creating a tamper-evident chain. This ensures the ledger remains a permanent, indelible history of transactions. Log hashing is controlled by the `HASH_LOGS` feature, which accepts `SYNC` (hash on write), `ASYNC` (hash in the background), or `DISABLED` (skip hashing for performance). See [Data isolation with buckets](/modules/ledger/working-with/data-isolation-buckets#features) for configuration. ## Single vs multi-ledger Formance Ledger is multi-ledger: you can operate multiple independent ledgers in a single instance. The choice depends on your application: - **Multi-ledger**: better horizontal scaling plus full logical separation, good for multi-tenant apps or high write volume with data segregation - **Single ledger**: simpler, with no need to manage cross-ledger consistency or check which ledger an account belongs to The Ledger uses a single-writer, sequential-writes design, and the write serialization is scoped to the **named ledger**, not to the service or the bucket. With the default `HASH_LOGS: SYNC`, appending a log takes a PostgreSQL advisory lock keyed on the ledger, so logs within one ledger are written one at a time. The practical consequence: adding named ledgers is what buys you write parallelism, because two ledgers take two different locks. Adding accounts inside one ledger does not — though spreading writes across source accounts does relieve a separate bottleneck, row-level locks on `accounts_volumes`. Both are covered in [Architecting for scale](/modules/ledger/advanced/architecting-for-scale#writes-contention). One ledger is the right default. Reach for more when you need storage separation, when a single ledger's write throughput becomes the constraint, or when a ledger grows large enough that you want to segment it. That decision is your segmentation strategy — see [Ledgers sizing](/modules/ledger/advanced/architecting-for-scale#ledgers-sizing) and [Segmentation strategies](/modules/ledger/advanced/architecting-for-scale#segmentation-strategies). Two properties are fixed at creation and cannot be changed afterwards: the ledger's bucket, and its [feature set](/modules/ledger/working-with/data-isolation-buckets#features) — including `HASH_LOGS`. Decide both before you start writing. ## Do / Don't - **Do:** Use one ledger per logical product domain (e.g. one for your wallet product, one for your lending product). - **Do:** Use multi-ledger for multi-tenant SaaS where each tenant needs its own book; pair it with separate buckets when tenants also need storage separation. - **Don't:** Create separate ledgers to represent different account types; use account naming conventions instead. - **Don't:** Use ledger boundaries as a substitute for access control; isolation is for data separation, not security. Next: [Accounts](/modules/ledger/core-concepts/accounts) --- ## Configure an Event Sink Source: https://docs.formance.com/modules/ledger/events/configure-sink Add an event sink when another system needs to react to Ledger activity: keeping a downstream service in sync, feeding an analytics warehouse, or notifying a customer, without polling the Ledger API. One sink reads supported changes from the global log and sends them to one external destination. Each named sink keeps its own cursor and error state, so one slow destination never holds back another sink or a Ledger write. ## Prerequisites This guide uses `ledgerctl`. Prepare the following: - a configured `ledgerctl` profile for the target cluster; - a principal with `ledger:OpsWrite` to add or remove a sink, and `ledger:OpsRead` to list sinks; - `ledgerctl` or a gRPC client for the add and remove operations, because the HTTP API can list sinks but exposes no route to create or delete one; - network access from every cluster member to the destination, because any member can become the leader; - a destination ready to acknowledge writes; - an idempotent consumer or deduplication policy; - enough destination capacity for the retained history that a new sink will replay. ### Supported event types The event system publishes these event types: | Event type | Published when | | --- | --- | | `CREATED_LEDGER` | A ledger is created. | | `DELETED_LEDGER` | A ledger is deleted. | | `COMMITTED_TRANSACTION` | A transaction is committed. | | `REVERTED_TRANSACTION` | A transaction is reverted. | | `SAVED_METADATA` | Account or transaction metadata is saved. | | `DELETED_METADATA` | Account or transaction metadata is deleted. | | `SKIPPED_ORDER` | A bulk order is skipped for an allowed business error. | Ledger does not currently publish an event for every configuration change. For example, do not expect events for indexes, prepared queries, Numscript library entries, account types, metadata schema changes, or signing configuration. NATS, Kafka, and HTTP events contain the complete committed log, including transaction postings and metadata. ClickHouse and Databricks store a flattened form of the same business data. Apply destination access controls and retention rules as carefully as you do for the Ledger API. ## Choose a destination | Destination | Use it for | Delivery contract | | --- | --- | --- | | NATS JetStream | Durable event-driven applications | Ledger publishes one message per event to `{topic}.{ledger}.{lowercase_event_type}` and waits for the JetStream acknowledgement. Configure a stream that captures the resulting subjects. | | Kafka | Partitioned stream processing | Ledger writes to one topic, uses the ledger name as the record key, adds an `event_type` header containing the lowercase form of the JSON body's uppercase event type, and waits for every record in the batch. | | HTTP | Webhooks and direct service integrations | Ledger sends one `POST` per event. Any `2xx` response acknowledges it; another status or a 30-second timeout causes a retry. Optional HMAC signing is available, but arbitrary authorization headers are not. | | ClickHouse | Operational analytics | Ledger creates the target table when needed and inserts batches. The table uses `ReplacingMergeTree`; duplicate removal happens during background merges. | | Databricks | Delta Lake analytics through a SQL Warehouse | Ledger creates a Delta table and inserts batches. The table does not remove duplicate deliveries for you. | The current NATS configuration exposes only a URL and subject prefix. Kafka exposes a TLS switch and SASL `PLAIN`, `SCRAM-SHA-256`, or `SCRAM-SHA-512`; it does not expose custom CA or client-certificate fields. Confirm that these authentication options match your environment before creating the sink. ## Choose delivery settings All sink types share these settings: | Flag | Effective default | Meaning | | --- | --- | --- | | `--format` | `json` | `json` or `protobuf` for NATS, Kafka, and HTTP. ClickHouse and Databricks use their analytical row format and ignore this setting. | | `--batch-size` | `64` | Maximum number of events acknowledged before the cursor advances. The current maximum is `100000`. | | `--batch-delay-ms` | `10` | Maximum wait before Ledger flushes a non-full batch. | | `--event-types` | All supported types | Comma-separated allow-list of event types. | Keep the defaults for webhooks and other latency-sensitive consumers. Raise `--batch-size` for analytical destinations, where throughput matters more than per-event latency; the server rejects anything above `100000`. A new sink name starts without a cursor. It replays every supported event still in the global log, then follows new commits. History is permanent in the primary store, so size the destination for a catch-up over the whole retained log rather than a recent window. The cursor also advances past excluded and unsupported log entries. If you widen `--event-types` later, Ledger does not republish the events that the old filter excluded. The cursor has already advanced past them. ## Protect sink credentials Populate the environment variables in the following examples from your secret manager. The current CLI passes destination secrets as command arguments. Environment substitution keeps the literal out of the command text, but the expanded value is still visible to other processes on the host while the command runs, so run these commands from a restricted administration environment and disable shell tracing around each one. Sink configurations are also replicated with cluster state and included in backups. `ledgerctl events list` masks destination secrets before displaying them, but the raw gRPC and HTTP read interfaces currently return the stored configuration to any caller with `ledger:OpsRead`. Restrict that scope, avoid logging raw responses, and treat backups as secret-bearing. ## Add the sink Choose exactly one destination. Keep the default JSON format unless your consumer already decodes the `events.Event` protobuf message. ```bash ledgerctl events add-sink \ --name application-events \ --nats-url "$NATS_URL" \ --nats-topic ledger.events \ --event-types COMMITTED_TRANSACTION,REVERTED_TRANSACTION ``` With this configuration, a committed transaction for the `orders` ledger is published on `ledger.events.orders.committed_transaction`. ```bash ledgerctl events add-sink \ --name application-events \ --kafka-brokers broker-1.example.com:9093,broker-2.example.com:9093 \ --kafka-topic ledger-events \ --kafka-tls \ --kafka-sasl-mechanism SCRAM-SHA-256 \ --kafka-sasl-username "$KAFKA_SASL_USERNAME" \ --kafka-sasl-password "$KAFKA_SASL_PASSWORD" ``` Kafka preserves order within a partition. Because Ledger keys records by ledger name, do not assume that a consumer observes one global order across all partitions. ```bash ledgerctl events add-sink \ --name application-webhook \ --http-endpoint https://events.example.com/ledger \ --http-secret "$LEDGER_WEBHOOK_SECRET" \ --batch-size 64 ``` Ledger signs the raw request body as `sha256=` in `X-Webhook-Signature`. See [Consume, Monitor, and Recover Events](/modules/ledger/events/consume-and-recover) for receiver code. ```bash ledgerctl events add-sink \ --name ledger-analytics \ --clickhouse-dsn "$CLICKHOUSE_DSN" \ --clickhouse-table ledger_events \ --batch-size 256 ``` The configured ClickHouse user needs permission to connect, create the table if it does not exist, and insert rows. The sink requires the experimental JSON and Variant types used by the generated table. ```bash ledgerctl events add-sink \ --name ledger-analytics \ --databricks-host "$DATABRICKS_HOST" \ --databricks-http-path "$DATABRICKS_HTTP_PATH" \ --databricks-catalog main \ --databricks-schema finance \ --databricks-table ledger_events \ --databricks-client-id "$DATABRICKS_CLIENT_ID" \ --databricks-client-secret "$DATABRICKS_CLIENT_SECRET" ``` Use either OAuth M2M as shown above or `--databricks-token "$DATABRICKS_TOKEN"`, never both. The identity needs permission to use the SQL Warehouse and create or insert into the target table. Only the HTTP sink is always compiled in. NATS, Kafka, ClickHouse, and Databricks are each behind a Go build tag, and a sink whose tag is absent fails to start with `unsupported events sink type: (not compiled in this build)`. Official Ledger releases are built with all four. An image built from the repository Dockerfile is not: it enables no optional sink unless the build passes the tag. Confirm the build tags of the exact image you deploy before treating one of these integrations as production-supported. ## Verify delivery Run this acceptance test in one Bash session. It creates a disposable normal ledger, commits one referenced transaction, finds that transaction's global `logSequence`, and waits for the chosen sink to acknowledge at least that sequence. Choose the mutation branch that matches the cluster's request-signing policy, then use the common delivery checks that follow it. You need `curl`, `jq`, `ledgerctl`, and: - an HTTP bearer token with the `ledger:LedgerRead`, `ledger:LedgerWrite`, `ledger:TransactionWrite`, and `ledger:OpsRead` granular scopes, or the default `ledger:write` and `ledger:read` scope bundles; - a `ledgerctl` profile that reaches the same Ledger cluster over gRPC; - the exact name of a sink whose event filter is empty or includes `COMMITTED_TRANSACTION`. The HTTP ledger create, transaction create, and ledger delete routes used below always submit unsigned mutations. They fail when `require_signatures=true`. Do not lower the policy to run this test. Use the signed `ledgerctl` branch below, or a version-pinned gRPC client that signs the exact `ApplyBatch`, and keep the HTTP calls only for read checks. A global signing flag protects `ledgerctl` commands only when they use the signed `Apply` path; it does not make an unsigned HTTP or unary gRPC mutation compatible. See [Request and Response Signing](/modules/ledger/operations/signing#check-mutation-compatibility). Set the HTTP endpoint, profile, and sink name, and take the bearer token from the environment rather than the script. `LEDGER_URL` is the root immediately before Ledger's `/v3` routes. Change `SINK_NAME` if you used `application-webhook` or `ledger-analytics` in the examples above. ```bash set -euo pipefail # Export LEDGER_TOKEN from your secret manager before running this block, so the # bearer token never reaches shell history or CI logs. This assertion fails fast # if it is unset. : "${LEDGER_TOKEN:?export the bearer token from your secret manager first}" ``` First, assert that the named sink exists and will not filter out the test transaction: ```bash ledgerctl events list --json > "/tmp/events-sinks-$RUN_ID.json" jq -e --arg sink "$SINK_NAME" ' [.sinks[] | select(.name == $sink)] as $matches | (($matches | length) == 1) and ( ($matches[0].eventTypes | length) == 0 or ($matches[0].eventTypes | index("COMMITTED_TRANSACTION") != null) ) ' "/tmp/events-sinks-$RUN_ID.json" > /dev/null ``` Choose one of the following mutation branches. Both create the same ledger and transaction, so the common log and destination checks work without modification. ### Branch A: unsigned HTTP mutations Use this branch only while `require_signatures=false`. Create a dedicated ledger so its test log is easy to identify: ```bash curl --fail-with-body --silent --show-error \ -X POST "$LEDGER_URL/v3/$LEDGER_NAME" \ -H "Authorization: Bearer $LEDGER_TOKEN" \ -H 'Content-Type: application/json' \ -H "Idempotency-Key: events-smoke:create:$RUN_ID" \ --data '{"mode":"NORMAL"}' \ | jq -e --arg ledger "$LEDGER_NAME" \ '.data.name == $ledger' > /dev/null ``` Build one request body, commit it, and retry the exact same request. The second response must identify the same transaction; this proves that a lost HTTP response can be retried without emitting a second committed-transaction event. ```bash jq -nc --arg reference "$EVENT_REFERENCE" '{ reference: $reference, postings: [{ source: "world", destination: "events:verification", asset: "USD/2", amount: 100 }] }' )" curl --fail-with-body --silent --show-error \ -X POST "$LEDGER_URL/v3/$LEDGER_NAME/transactions" \ -H "Authorization: Bearer $LEDGER_TOKEN" \ -H 'Content-Type: application/json' \ -H "Idempotency-Key: $EVENT_IDEMPOTENCY_KEY" \ --data "$EVENT_BODY" \ > "/tmp/events-transaction-first-$RUN_ID.json" curl --fail-with-body --silent --show-error \ -X POST "$LEDGER_URL/v3/$LEDGER_NAME/transactions" \ -H "Authorization: Bearer $LEDGER_TOKEN" \ -H 'Content-Type: application/json' \ -H "Idempotency-Key: $EVENT_IDEMPOTENCY_KEY" \ --data "$EVENT_BODY" \ > "/tmp/events-transaction-retry-$RUN_ID.json" jq -e \ --arg reference "$EVENT_REFERENCE" \ --slurpfile retry "/tmp/events-transaction-retry-$RUN_ID.json" ' (.data.transaction.reference == $reference) and (.data.transaction.id == $retry[0].data.transaction.id) ' "/tmp/events-transaction-first-$RUN_ID.json" > /dev/null ``` ### Branch B: mandatory request signatures Use this branch when `require_signatures=true`. Configure the registered signing key through your secret manager, then create the ledger and transaction through `ledgerctl` commands that use the signed gRPC `Apply` path: ```bash ledgerctl \ --signing-key "$LEDGER_SIGNING_KEY_FILE" \ --signing-key-id "$LEDGER_SIGNING_KEY_ID" \ ledgers create --name "$LEDGER_NAME" ledgerctl \ --signing-key "$LEDGER_SIGNING_KEY_FILE" \ --signing-key-id "$LEDGER_SIGNING_KEY_ID" \ transactions create \ --ledger "$LEDGER_NAME" \ --posting "world,events:verification,100,USD/2" \ --reference "$EVENT_REFERENCE" \ --json \ > "/tmp/events-transaction-signed-$RUN_ID.json" ``` These `ledgerctl` commands currently submit an empty batch idempotency key. Do not retry either command after an ambiguous timeout until a read or the global log proves whether it committed. If this acceptance test must also prove retry idempotency, use a signed gRPC client that persists the original idempotency key and resends the exact serialized `ApplyBatch`; rebuilding and re-signing a batch creates a different operation. ### Confirm the log sequence and sink cursor Run these checks after either branch. The transaction response does not expose its global log sequence. Poll the ledger log until the asynchronous read index exposes the entry, then extract the sequence from the matching reference: ```bash deadline=$((SECONDS + 30)) while [[ -z "$LOG_SEQUENCE" ]]; do ledgerctl logs list \ --ledger "$LEDGER_NAME" \ --page-size 10 \ --json \ > "/tmp/events-logs-$RUN_ID.json" LOG_SEQUENCE="$( jq -er --arg reference "$EVENT_REFERENCE" ' [ .[] | select( .payload.apply.log.data.createdTransaction.transaction.reference == $reference ) | .sequence ] | if length == 1 then .[0] else empty end ' "/tmp/events-logs-$RUN_ID.json" || true )" if [[ -z "$LOG_SEQUENCE" && $SECONDS -ge $deadline ]]; then echo "Timed out while waiting for the transaction log" >&2 exit 1 fi [[ -n "$LOG_SEQUENCE" ]] || sleep 1 done jq -en --argjson sequence "$LOG_SEQUENCE" '$sequence > 0' > /dev/null ``` Finally, poll the sink status for up to 60 seconds. The assertion passes only when that sink's cursor is at or beyond the test transaction and its error field is empty: ```bash deadline=$((SECONDS + 60)) while true; do ledgerctl events list --json > "/tmp/events-sink-status-$RUN_ID.json" if jq -e \ --arg sink "$SINK_NAME" \ --argjson sequence "$LOG_SEQUENCE" ' .sinkStatuses[] | select(.sinkName == $sink) | ((.cursor | tonumber) >= $sequence) and ((.error == null) or ((.error.message // "") == "")) ' "/tmp/events-sink-status-$RUN_ID.json" > /dev/null; then break fi if ((SECONDS >= deadline)); then echo "Sink did not acknowledge logSequence $LOG_SEQUENCE within 60 seconds" >&2 jq --arg sink "$SINK_NAME" \ '.sinkStatuses[] | select(.sinkName == $sink)' \ "/tmp/events-sink-status-$RUN_ID.json" >&2 exit 1 fi sleep 2 done jq -e \ --arg sink "$SINK_NAME" \ --argjson sequence "$LOG_SEQUENCE" ' .sinkStatuses[] | select(.sinkName == $sink) | { sinkName, cursor: (.cursor | tonumber), error } | select(.cursor >= $sequence) | select((.error == null) or ((.error.message // "") == "")) ' "/tmp/events-sink-status-$RUN_ID.json" ``` The final object is the Ledger-side acceptance result. A cursor at or above `LOG_SEQUENCE` means the sink acknowledged the complete batch containing the event. It does not establish exactly-once consumption: if Ledger published a batch but could not persist its new cursor before a failure or leader change, that batch can be delivered again. Deduplicate at the consumer with the stable cluster source you assign and the event's `logSequence`. If the command times out, inspect the printed `error` and the Ledger server logs for `SINK_NAME`. Ledger keeps the cursor before a failed batch and retries publishing with exponential backoff. Correct destination connectivity, credentials, or acknowledgement behavior, then run the polling block again; there is no separate retry API to call. ### Confirm the destination and tear down After the Ledger-side assertion passes, confirm that the destination contains the event with `reference == EVENT_REFERENCE` and `logSequence == LOG_SEQUENCE`: | Destination | What to verify | | --- | --- | | NATS | The stream captures the configured subject prefix and the message contains the expected `reference` and `logSequence`. | | Kafka | The record is on the configured topic, its key is `LEDGER_NAME`, the body contains the expected `reference` and `logSequence`, and the decoded `event_type` header equals the lowercase form of the body's `type`. | | HTTP | The receiver durably stores the expected `reference` and `logSequence`, verifies the signature, and returns `2xx` only after that acceptance. | | ClickHouse | `SELECT log_sequence, type, ledger, data.transaction.reference FROM ledger_events FINAL WHERE log_sequence = ` returns the expected ledger and reference. | | Databricks | `SELECT log_sequence, type, ledger, data:transaction.reference FROM main.finance.ledger_events WHERE log_sequence = ` returns the expected ledger and reference. | For the Kafka event produced by this test, assert this exact relationship after decoding the header bytes as UTF-8: ```text body.type == "COMMITTED_TRANSACTION" headers["event_type"] == "committed_transaction" headers["event_type"] == ascii_lowercase(body.type) ``` The header and the JSON enum are intentionally not byte-equal. After confirming the destination, delete the test ledger with the same mutation branch you used to create it. The block runs under `set -euo pipefail`, so an unexpected response aborts the session and leaves the disposable ledger in place; re-run the teardown commands until the read check returns `404`. For Branch A, reuse the derived idempotency key if the HTTP DELETE response is lost, then verify independently that the ledger is gone: ```bash delete_status="$( curl --silent --show-error \ -X DELETE "$LEDGER_URL/v3/$LEDGER_NAME" \ -H "Authorization: Bearer $LEDGER_TOKEN" \ -H "Idempotency-Key: events-smoke:delete:$RUN_ID" \ --output /dev/null \ --write-out '%{http_code}' )" [[ "$delete_status" == "204" ]] get_status="$( curl --silent --show-error \ -H "Authorization: Bearer $LEDGER_TOKEN" \ "$LEDGER_URL/v3/$LEDGER_NAME" \ --output /dev/null \ --write-out '%{http_code}' )" [[ "$get_status" == "404" ]] ``` For Branch B, delete the ledger through the signed `Apply` path. This command does not carry an idempotency key, so reconcile an ambiguous result with the read check before considering another delete: ```bash ledgerctl \ --signing-key "$LEDGER_SIGNING_KEY_FILE" \ --signing-key-id "$LEDGER_SIGNING_KEY_ID" \ ledgers delete --name "$LEDGER_NAME" --yes get_status="$( curl --silent --show-error \ -H "Authorization: Bearer $LEDGER_TOKEN" \ "$LEDGER_URL/v3/$LEDGER_NAME" \ --output /dev/null \ --write-out '%{http_code}' )" [[ "$get_status" == "404" ]] ``` A sink that `ledgerctl events list` reports as `healthy` is not necessarily delivering: that status only means no publish error is recorded, and an initial connection or configuration failure is not recorded there. A referenced test event, a destination receipt, and the asserted cursor advance are the acceptance criteria. See [Monitor each sink](/modules/ledger/events/consume-and-recover#monitor-each-sink). ## Replace or remove a sink The current server rejects `add-sink` when the name already exists, even though some CLI help text describes the operation as an upsert. To replace a configuration safely: 1. Add the corrected configuration under a new name. 2. Allow for replay from the retained global log. 3. Verify a known event and cursor advancement on the new sink. 4. Confirm that the consumer deduplicates replayed events. 5. Remove the old sink. ```bash ledgerctl events remove-sink --name application-events-old ``` Removing the final sink stops event delivery. There is no public cursor reset or replay command. Do not rely on removing and re-adding the same name to reset its cursor; use a new name and account for retained-history replay instead. Continue with [Consume, Monitor, and Recover Events](/modules/ledger/events/consume-and-recover) before sending production traffic. --- ## Overview Source: https://docs.formance.com/modules/ledger/get-started/overview Formance Ledger is a programmable product ledger. Applications use it to record financial movements, keep balances consistent under concurrent activity, and retain the history needed to investigate and reconcile those movements. ## The four accounting primitives | Primitive | What it represents | Example | | --- | --- | --- | | **Ledger** | An isolated accounting namespace. | The balances and transactions for one product or tenant. | | **Account** | A named holder of one or more asset balances. | `wallets:alice` or `platform:fees`. | | **Transaction** | A group of postings that succeeds or fails as one operation. | A customer payment split between a merchant and a fee account. | | **Posting** | One integer amount and asset moving from a source to a destination. | `USD/2 1250` from `wallets:alice` to `merchants:coffee-shop`. | Accounts do not need to be created before their first transaction. Ledger creates an account when a posting first refers to its address. ## A financial movement in Ledger The special `world` account represents value entering or leaving the boundary modeled by your ledger. This example brings USD 100.00 into Alice's wallet, then records a USD 12.50 purchase. ```mermaid flowchart LR World["Outside the modeled system"] -->|"USD/2 10000"| Alice["wallets:alice"] Alice -->|"USD/2 1250"| Merchant["merchants:coffee-shop"] ``` `USD/2` identifies the asset and its two-decimal display precision. Ledger records `10000` and `1250` as integers; your application formats them as USD 100.00 and USD 12.50 for display. ## Guarantees you can build on - **Balanced movement:** every posting records the same asset and amount leaving one source and entering one destination. - **Atomic transactions:** Ledger records every posting in a transaction or records none of them. - **Funds validation:** Ledger rejects a debit that exceeds the available balance by default. - **Integer amounts:** Ledger never depends on floating-point arithmetic for financial values. - **Traceable corrections:** recorded transactions remain in history; a correction creates an additional transaction instead of editing the original. - **Logical isolation:** accounts and transactions belong to one ledger and do not leak into another ledger's namespace. These guarantees cover the accounting record. Your application still owns authorization decisions, the meaning of each account, and the business event that authorizes a transaction. ## Connect from an application or an operator tool Applications commonly use the HTTP API. Operational tooling can use the public gRPC API directly, and `ledgerctl` provides a command-line client for that gRPC surface. A feature can exist on one surface before an equivalent workflow exists on another, so follow the guide for the interface you use. ## Choose your next task | I want to... | Continue with... | | --- | --- | | Record and verify a first transaction | [Quick Start](/modules/ledger/quick-start) | | Design account names, assets, and metadata | [Model Your Ledger](/modules/ledger/modeling) | | Record postings or execute Numscript | [Write Transactions](/modules/ledger/transactions) | | Read balances, history, and query results | [Read and Query Data](/modules/ledger/querying) | | Send ledger changes to downstream systems | [Integrate with Events](/modules/ledger/events) | | Secure, monitor, back up, and restore Ledger | [Operate Ledger](/modules/ledger/operations) | Complete the shortest working path from connection to an idempotent retry. Turn your product's financial boundaries into stable accounts, assets, and rules. --- ## Plan the Migration Source: https://docs.formance.com/modules/ledger/migration/overview Ledger does not upgrade a Ledger v2 database in place. Run the source and target side by side, keep Ledger v2 authoritative, and move production traffic only after you have validated the target. ```mermaid flowchart LR Prepare["Prepare Ledger v2"] --> Clients["Update and test clients"] Clients --> Mirror["Create a read-only mirror"] Mirror --> Validate["Validate continuously"] Validate --> Freeze["Freeze Ledger v2 writes"] Freeze --> Final["Ingest and validate the final tail"] Final --> Promote["Promote once"] Promote --> Open["Open target traffic"] ``` ## Choose the source transport The mirror reads the Ledger v2 log through one of two transports. | Transport | Choose it when | Access required | Main tradeoff | | --- | --- | --- | --- | | HTTP | The target can reach the Ledger v2 API and the API can sustain the replay workload. This is the preferred boundary. | `GET /v2/{ledger}/logs`, with OAuth2 client credentials when the source requires them. | Preserves the Ledger v2 service as the database boundary, but adds API and authorization overhead. | | PostgreSQL | The API is unavailable for this workload or direct database access is an approved operational choice. | Read access to `_system.ledgers` and the source bucket's `logs` table. | Removes API overhead, but grants the target direct access to the Ledger v2 database. | Do not choose PostgreSQL only for convenience. It expands the network, credential, and database-permission scope of the migration. ## Understand the authority and rollback boundary Before promotion: - applications continue writing to Ledger v2; - the target remains read-only to application writes; - you can delete the target, correct its source or rewrite configuration, and replay from the beginning; - validation can run repeatedly without changing the source. Promotion stops source replication and changes the target to normal, writable mode. Ledger does not support demoting that ledger back into mirror mode. Promotion is irreversible for the mirror. After the first target-only production write, routing traffic back to Ledger v2 would also lose that new state because Ledger v2 never received it. ## Assign responsibilities Record a named owner for each responsibility before creating the production mirror. | Responsibility | Required decision | | --- | --- | | Migration lead | Owns the runbook, go/no-go decision, and migration record. | | Ledger v2 owner | Grants read access, monitors source load, and freezes source writes. | | Target administrator | Creates, monitors, recreates, and promotes the mirror. | | Application owner | Updates API clients, authentication, retries, and event consumers. | | Validation owner | Defines material invariants and signs the final comparison. | | Incident owner | Decides when to stop the cutover and coordinates recovery. | The person who can promote the mirror should not be the only person who approves the validation evidence. ## Define the phases and exit conditions | Phase | Work | Exit condition | | --- | --- | --- | | Assess | Inventory ledgers, clients, data patterns, dependencies, and source access. | Every ledger and workflow has an owner and disposition. | | Rehearse | Create a disposable mirror with production-like data and configuration. | The complete procedure has run without unexplained differences. | | Follow | Run the production mirror while Ledger v2 remains active. | The cursor advances and repeatedly reaches the observed source head. | | Validate | Compare technical state, business invariants, clients, and consumers. | Every material check passes or has an approved explanation. | | Cut over | Freeze Ledger v2, ingest the final tail, and rerun critical checks. | The target cursor equals the recorded final source log ID and approval is recorded. | | Promote | Stop replication, switch clients, and send one controlled write. | The write, read, and downstream event all match the expected result. | ## Produce a migration record Keep one record per source ledger. At minimum, include: - source and target ledger names; - source transport and credential owner; - initial and final Ledger v2 log IDs; - target cursor and observed source head; - exact rewrite rules, including their order; - expected transformed or dropped entries; - client and event-consumer versions; - validation results and evidence locations; - rehearsal duration and production interruption budget; - go/no-go approvers and the promotion timestamp. ## Stop before creating the mirror when - the target environment cannot read the full source log; - the client contract or authentication path has not been tested; - a required data difference has no deterministic transformation rule; - no one owns the balance and business-invariant comparison; - the team has no way to freeze all Ledger v2 writers; - the cutover plan assumes that promotion can be undone. Continue with [Prepare the Source](/modules/ledger/migration/prepare-source). --- ## Design a Chart of Accounts Source: https://docs.formance.com/modules/ledger/modeling/chart-of-accounts A chart of accounts maps business responsibilities to stable account addresses. A reader should be able to look at `merchants:acme:settlement` and understand who owns the value and why the account exists. Ledger creates an account when a transaction or metadata operation first refers to its address. There is no separate account-creation step, so the address itself is part of your application contract. ## Choose the ledger boundary first A ledger is a logical namespace for accounts, transactions, metadata, and sequence numbers. Multiple ledgers in the same cluster share the cluster's ordering, storage estate, availability, and capacity. Use another ledger when you need a distinct accounting book but can accept the same operational boundary. Use another cluster when the workload needs independent capacity, maintenance, backup, security, or failure boundaries. | Requirement | Model it as | | --- | --- | | Another customer, merchant, wallet, or internal pool in the same book | Another account address | | A logically separate book with its own transactions and sequences | Another ledger | | Independent availability, capacity, or operational ownership | Another cluster | | Searchable context that does not change the balance identity | Metadata | Do not create one ledger per user only to separate balances. That also separates transaction history and makes every new user part of cluster-level ledger management. ## Build addresses from stable segments Account addresses consist of non-empty segments separated by colons. Use broad business ownership first and more specific purpose later: ```text customers:{customerId}:wallets:main customers:{customerId}:wallets:rewards merchants:{merchantId}:earnings merchants:{merchantId}:settlement platform:fees platform:clearing:card ``` The resulting hierarchy keeps related accounts visible together and gives prefix queries a predictable boundary: ```mermaid flowchart TD Ledger["marketplace ledger"] --> Customers["customers"] Ledger --> Merchants["merchants"] Ledger --> Platform["platform"] Customers --> Alice["alice"] Alice --> Main["wallets:main"] Alice --> Rewards["wallets:rewards"] Merchants --> Acme["acme"] Acme --> Earnings["earnings"] Acme --> Settlement["settlement"] Platform --> Fees["fees"] Platform --> Clearing["clearing:card"] ``` Ledger accepts ASCII letters, digits, `_`, and `-` inside an address segment. An address can be at most 256 bytes. Leading, trailing, and repeated colons create empty segments and are rejected. Prefix matching operates on the raw address string, not on parsed segments. The prefix `customers:1` also matches `customers:10`. Include a trailing colon when you intend to select a hierarchy, for example `customers:1:`. ## Put identity in the address and attributes in metadata An address should contain the fields that define which balance bucket a posting changes. Metadata should contain descriptive or queryable context that can change without creating another balance. | Question | Address segment | Metadata | | --- | --- | --- | | Would changing the value move future postings to a different balance? | Yes | No | | Must the value be visible in every transaction posting? | Yes | Usually no | | Can the value change while the account keeps the same identity? | Usually no | Yes | | Example | `merchants:acme:settlement` | `region: "eu"` | Avoid embedding labels such as a customer's display name in the address. Prefer an immutable application identifier in the address and keep the display name in metadata. ## Decide what `world` means `world` is the built-in boundary between the value modeled inside the ledger and value outside it. It can source funds without a balance check and can receive funds leaving the modeled system. For a product ledger that tracks customer and merchant claims, a deposit can enter from `world`: ```text world -> customers:alice:wallets:main ``` If the ledger also represents treasury positions, name the internal asset-side account instead: ```text treasury:bank:eu -> customers:alice:wallets:main ``` The second model carries more reconciliation detail, but it couples product transactions to treasury topology. Choose the smallest boundary that still supports the reconciliations you must perform. ## Model temporary business states explicitly Temporary does not mean unimportant. Pending payments, settlement positions, fee accruals, and clearing balances often need their own accounts because they answer different operational questions. ```text payments:{paymentId}:authorization payments:{paymentId}:capture settlements:{settlementId}:pending settlements:{settlementId}:completed ``` Use an account only when you need a distinct balance. Use transaction metadata for state that describes a movement without owning a balance. ## Test the model with real flows Before you enforce the chart, write down the accounts touched by each critical flow: | Flow | Accounts that should move | | --- | --- | | Customer funding | `world` or treasury account -> customer wallet | | Purchase | customer wallet -> merchant earnings and platform fees | | Merchant payout | merchant earnings -> settlement or `world` | | Refund | merchant or platform liability -> customer wallet | | Reconciliation adjustment | named clearing account -> target account | For every flow, verify: 1. the address identifies the owner and purpose without external lookup; 2. the same business entity always produces the same address; 3. a prefix selects the intended group without selecting siblings accidentally; 4. the asset and precision are canonical; 5. every external boundary movement has a reconciliation owner. ## Evolve the chart without rewriting history Recorded postings retain their original account addresses. If you introduce a new naming convention, old and new addresses will coexist in reads and reports. Plan an evolution as an application rollout: 1. add the new [account types](/modules/ledger/modeling/account-types) in `AUDIT` mode; 2. update writers to emit the new addresses; 3. update readers and reconciliations to cover both conventions; 4. move balances with explicit transactions when the business requires it; 5. keep historical address mappings documented. Do not rename an address by changing metadata. Metadata does not move its volumes or rewrite its postings. ## Common failure modes | Symptom | Likely modeling issue | Correction | | --- | --- | --- | | Reports require many metadata joins to identify a balance | The address is too generic | Put stable ownership and purpose in address segments. | | A customer rename requires moving balances | A mutable label is part of the address | Use an immutable ID and store the label as metadata. | | Prefix totals include unrelated accounts | The prefix ends mid-segment | End hierarchical prefixes with `:`. | | Product code changes whenever a bank account changes | Treasury details are embedded too deeply | Use `world` or a stable treasury pool boundary when appropriate. | | Old refunds or reverts fail after enforcement | New account types do not cover historical addresses | Keep compatible types until historical operations no longer need them. | Next, define the [assets, amounts, balances, and volumes](/modules/ledger/modeling/assets-amounts-and-balances) carried by these accounts. --- ## Prepare a Ledger Deployment Source: https://docs.formance.com/modules/ledger/operations/deploy Deploy Ledger through Stack v4 and the Formance Stack Operator. A production deployment needs more than running pods: it needs a voting topology, durable node identities, enough storage and memory, protected network paths, and telemetry that proves the cluster can commit writes. The Stack v4 production resource schema is not available in this preview. Use the direct-server workflow below only for evaluation. Wait for the release-specific Stack Operator reference before applying a production Ledger resource. ## Evaluate Ledger with a direct server This workflow starts one Ledger process directly for evaluation and integration development. It does not provide production scheduling, failure-domain placement, secret distribution, rolling convergence, or automated recovery. Use Stack v4 and its released Operator workflow for production. The direct server opens three listeners by default: | Port | Surface | Exposure for this evaluation | | --- | --- | --- | | `7777` | Raft transport and cluster join | `--bind-addr` defaults to `0.0.0.0:7777`. Pass `--bind-addr 127.0.0.1:7777` as the examples below do, and do not expose the port to application clients. | | `8888` | Public gRPC service used by `ledgerctl` | The direct server listens on the configured port on all interfaces. Keep the host firewalled and connect through `localhost` with `--insecure` only in this plaintext evaluation. | | `9000` | HTTP API and health probes | The direct server listens on the configured port on all interfaces. Keep the host firewalled and use `localhost` for the Quick Start. | ### Bootstrap the first start once Create distinct paths for the Raft WAL and Ledger data: ```bash mkdir -p .ledger-evaluation/node-1/wal .ledger-evaluation/node-1/data ``` Start the first node in a terminal: ```bash ledger run \ --node-id 1 \ --cluster-id local-evaluation \ --bind-addr 127.0.0.1:7777 \ --grpc-port 8888 \ --http-port 9000 \ --wal-dir ./.ledger-evaluation/node-1/wal \ --data-dir ./.ledger-evaluation/node-1/data \ --bootstrap ``` `--bootstrap` creates a new single-voter cluster. Use it only for this node's first start. A fresh node that has neither `--bootstrap` nor `--join` stops with `first start requires --bootstrap or --join`. ### Wait for the process and the cluster Run the checks from another terminal: ```bash curl --fail --silent --show-error http://localhost:9000/livez curl --fail --silent --show-error http://localhost:9000/readyz until curl --fail --silent --show-error http://localhost:9000/clusterz; do sleep 1 done ledgerctl \ --server localhost:8888 \ --insecure \ cluster status ``` Each HTTP probe returns: ```json {"data":{"status":"ok"}} ``` The probes answer different questions: | Probe | Meaning | | --- | --- | | `/livez` | The HTTP process is running. | | `/readyz` | The local Raft loop started. It can return `200` before the cluster elects a leader, so use it for process orchestration rather than traffic admission. | | `/clusterz` | The local node is connected to a healthy Raft state and sees an elected leader. Wait for this before a client sends cluster-dependent traffic. | Disk and clock protection can still reject writes after `/clusterz` succeeds. Monitor those gates separately. `cluster status` must show node `1` as a voter and the current leader in this single-node evaluation. ### Restart without bootstrap Stop the process cleanly, then restart it with the same identity and storage paths and without `--bootstrap`: ```bash ledger run \ --node-id 1 \ --cluster-id local-evaluation \ --bind-addr 127.0.0.1:7777 \ --grpc-port 8888 \ --http-port 9000 \ --wal-dir ./.ledger-evaluation/node-1/wal \ --data-dir ./.ledger-evaluation/node-1/data ``` Wait for `/clusterz` again, then run `ledgerctl cluster status`. The same node must return as voter and leader. Ledger persists the node ID and cluster ID with its data and keeps the consensus identity and history in the WAL directory. Changing either identity or attaching another node's directory causes startup validation to fail; do not delete either path to work around that failure. ### Understand the direct multi-node boundary A fresh additional process joins through the Raft address of an existing member with `--join `. Give every member a unique non-zero node ID, advertised endpoint, WAL directory, and data directory. Processes sharing one host also need distinct Raft, gRPC, and HTTP ports. Use `--join` only for that member's first start. After the cluster accepts it, restart the same member with its original paths and without `--join` or `--bootstrap`. Direct multi-node operation also requires TLS and one shared cluster secret before the Raft or gRPC listeners cross a trusted loopback boundary. It remains an advanced development path, not a substitute for the Stack v4 production workflow. Follow [Plan Topology and Manage Cluster Lifecycle](/modules/ledger/operations/cluster-lifecycle) for membership checks and stop conditions. ## Before you deploy Confirm these prerequisites with the platform and application owners: | Area | Required decision | | --- | --- | | Release | Pin one Stack v4 release whose version manifest resolves Ledger to `3.0.0` and identifies compatible Operator and CLI versions. | | Topology | Use three voting replicas as the normal production starting point. Choose five only when the additional failure tolerance justifies the consensus and operating cost. | | Placement | Spread voters across independent Kubernetes nodes and, where available, failure zones. Prevent routine disruption from removing quorum. | | Storage | Give every member its own retained persistent storage for the Raft WAL and Ledger data. Confirm expansion support, IOPS, encryption, and the retention policy before creating the cluster. | | Capacity | Size every voter for the complete dataset and workload, including compaction, index builds, backup staging, and catch-up. | | Security | Prepare client authentication, internal TLS, network policy, and secrets without placing credentials in the Stack resource or Git. | | Observability | Route metrics, traces, and logs to an owned backend. Preserve cluster and node labels so operators can distinguish a cluster-wide failure from one slow replica. | | Recovery | Define RPO and RTO, choose object storage, and schedule an isolated restore test. | Review [Plan Capacity and Tune Performance](/modules/ledger/operations/capacity-and-performance) before choosing resource requests or volume sizes. Review [Plan Topology and Manage Cluster Lifecycle](/modules/ledger/operations/cluster-lifecycle) before choosing replica count and disruption settings. ## Verify Stack v4 supports your deployment Before production, confirm that the Stack Operator reference shipped with your selected release explains how to declare and observe all of the following: - the Stack v4 version selection and the resolved Ledger image; - voting replicas, failure-domain placement, and disruption controls; - distinct WAL and data persistence, storage classes, sizes, and retention; - CPU, memory, `GOMEMLIMIT`, and supported Ledger tuning settings; - public HTTP and gRPC exposure, internal peer traffic, and TLS; - authentication and secret references; - OpenTelemetry export and stable cluster/node resource attributes; - backup credentials and schedules when managed by the Operator; - reconciliation conditions for bootstrap, learner catch-up, voter promotion, readiness, and failure; - the supported pause, rollback, and deletion behavior. If the released reference omits one of these controls, resolve it before production rather than guessing a resource field. ## Deploy the cluster ### 1. Run the precheck 1. Install the Operator and its CRDs using the [self-hosted Operator setup](/deploy/self-hosted/installation/operator-setup). 2. Confirm that the installed CRD and controller versions match the selected Stack v4 release. 3. Confirm that all target failure zones can provision the selected persistent volumes. 4. Confirm that the node pool can schedule all voters at their requested CPU and memory at the same time. 5. Test access from the Ledger namespace to the telemetry, identity, and backup services it will use. 6. Record the expected Stack, Ledger, and Operator versions in the change request. Do not continue while an existing resource uses the intended cluster identity or while retained volumes from another member could be attached to the new cluster. ### 2. Apply the supported Stack v4 resources Use the exact resource shape from the Stack Operator reference shipped with the selected release. Keep the declaration in version control, keep secrets in the configured secret manager, and review the rendered diff before applying it. The first member bootstraps the cluster. Additional members must join as learners, catch up, and become voters. Do not open application traffic while reconciliation still reports a learner, a restoring member, or an unresolved version. ### 3. Observe reconciliation Follow the Stack and Ledger status until: - the Operator reports the selected Ledger version on every member; - the expected members exist with distinct identities and persistent volumes; - one member is the leader; - every intended replica is a voter rather than a learner; - no member is still restoring, synchronizing, or restarting; - the HTTP and gRPC services expose only the intended network paths; - telemetry contains one series of cluster-level signals and distinct node labels. Operator readiness is necessary, but it does not replace the Ledger checks below. ## Verify the deployment Configure `ledgerctl` with the production endpoint and credentials, then inspect the cluster: ```bash ledgerctl cluster status ledgerctl ledgers list --json ``` `cluster status` must show a leader, the expected member count, the expected voter roles, and no unexplained gap between the committed and applied positions. Query a specific member when the cluster view suggests that one replica is behind: ```bash set -euo pipefail ledgerctl cluster status --json > cluster-status.json MEMBER_ID="$( jq -er ' . as $cluster | [ .nodes[] | select( .id != $cluster.leader and ( .progress.match != $cluster.raftStatus.commit or .progress.recentActive != true or ((.syncProgress.status // "normal") != "normal") ) ) ] | sort_by(.id) | .[0].id ' cluster-status.json )" ledgerctl cluster status --node-id "$MEMBER_ID" ``` The extraction selects the lowest-ID follower whose replication position, activity, or sync state needs investigation. It exits without choosing an arbitrary member when the cluster view contains no such follower. Review `cluster-status.json` first when several members are affected. Complete these independent checks before opening traffic: 1. Send a controlled, idempotent write to a non-production verification ledger and read it back with current-read consistency. 2. Repeat the read and write path through every externally reachable endpoint so a follower-forwarding or trust error cannot hide behind the leader. 3. Confirm that an unauthorized read or write returns the expected `401` or `403` result. 4. Confirm that the no-leader, disk, queue, process-memory, and write-stall alerts are loaded and routed to an owner. 5. Build every index required by the launch workload and wait for readiness on every replica. 6. Produce a backup, restore it into an isolated cluster, and run the integrity and business checks. The deployment result is a cluster that can commit a controlled write, serve the corresponding current read, lose one voter within its designed tolerance, and produce actionable telemetry. ## Roll back or stop safely Before application traffic or production data exists, revert the Stack declaration through the Operator and follow the documented persistent-volume retention policy. After production data exists: - stop the rollout when a member cannot catch up or the cluster loses its expected failure tolerance; - do not delete persistent volumes, change node identities, or bootstrap another member over existing state; - do not downgrade the Ledger image unless the release compatibility matrix confirms that the previous binary can read the current persisted format; - preserve the cluster state and follow [Upgrade Ledger](/modules/ledger/operations/upgrade) or the relevant recovery runbook. If deployment validation fails, keep traffic closed. Capture the Stack conditions, resolved versions, `ledgerctl cluster status`, affected node logs, and volume state, then start with [Troubleshoot Ledger](/modules/ledger/operations/troubleshooting). --- ## Read Ledger Data Source: https://docs.formance.com/modules/ledger/querying/read-ledger-data Ledger exposes several records for different questions. Read an account for its current volumes, read a transaction for an accepted movement, use logs for committed changes, and use the audit trail to investigate both accepted and rejected proposals (a proposal is a write batch submitted to Ledger). ## Prerequisites These reads target a ledger that already holds the records you want to inspect: ```bash ``` Grant each application only the scopes needed by the routes it calls. See [Authentication and Authorization](/modules/ledger/operations/authentication). | HTTP read | Required scope | | --- | --- | | Accounts and volumes | `ledger:AccountRead` | | Transactions | `ledger:TransactionRead` | | Per-ledger logs and statistics | `ledger:LedgerRead` | | Bucket-wide audit entries | `ledger:AuditRead` | The gRPC `ListLogs` method used by `ledgerctl logs list` requires the same `ledger:LedgerRead` scope as the HTTP log route. ## Choose the record | Question | Record to read | Data boundary | | --- | --- | --- | | What does this account hold now? | Account | One ledger | | Which postings and metadata were committed? | Transaction | One ledger | | Which successful mutations changed this ledger? | Ledger log | One ledger | | Why was a proposal accepted or rejected? | Audit entry | The whole bucket | | How many transactions, postings, logs, and related records exist? | Ledger statistics | One ledger | Logs and audit entries are related, but they are not interchangeable. Each fresh proposal that reaches Ledger's state machine produces one audit entry, whether it succeeds or fails. A successful proposal can produce several logs or none. An idempotent replay produces neither another log nor another audit entry, and a request rejected before application is not part of this audit trail. ## Read an account Retrieve one account by its full address: ```bash curl --fail-with-body --silent --show-error \ "${CURL_AUTH[@]}" \ "$LEDGER_URL/v3/$LEDGER_NAME/accounts/users:alice" ``` The response contains the address, typed metadata, and volumes. Volumes are separated by asset and color. Add `?collapseColors=true` only when the caller intentionally wants one total per asset and does not need the color boundary. List accounts when the application does not know the complete address: ```bash curl --fail-with-body --silent --show-error --get \ "${CURL_AUTH[@]}" \ --data-urlencode 'filter=address ^= "users:"' \ --data-urlencode 'pageSize=100' \ "$LEDGER_URL/v3/$LEDGER_NAME/accounts" ``` Account lists are ordered alphabetically. See [Filter and Paginate Results](/modules/ledger/querying/filters-and-pagination) before using a list route for an unbounded export. ## Read a transaction Retrieve a transaction by its numeric ID: ```bash : "${TRANSACTION_ID:?set this to the ID returned for a committed transaction}" curl --fail-with-body --silent --show-error \ "${CURL_AUTH[@]}" \ "$LEDGER_URL/v3/$LEDGER_NAME/transactions/$TRANSACTION_ID" ``` The response contains the committed transaction. List transactions to search by reference, account, metadata, or date: ```bash curl --fail-with-body --silent --show-error --get \ "${CURL_AUTH[@]}" \ --data-urlencode 'filter={"$match":{"reference":"order-2026-0042"}}' \ --data-urlencode 'pageSize=100' \ "$LEDGER_URL/v3/$LEDGER_NAME/transactions" ``` The default transaction order is newest first. Pass `reverse=true` for oldest first. Reference and date filters require their corresponding indexes to be ready. The structured reference match is exact. The textual filter grammar does not expose a reference condition. ## Trace a successful transaction Use this sequence when an application needs to explain what happened to a successful write. It correlates a business reference with the transaction, the global log sequence, and the bucket-wide audit entry, in three hops: 1. `reference` to the transaction `id`, from the transaction list; 2. transaction `id` to the global `sequence`, from the ledger logs, at `payload.apply.log.data.createdTransaction.transaction.id`; 3. global `sequence` to the audit entry, with `filter=log_seq == `. Start with the business reference and idempotency key recorded by the caller, then wait for one exact transaction. A filtered HTTP list can briefly lag behind a fresh write, so an empty result is not yet a negative result: ```bash if ! [[ "$TRANSACTION_WAIT_SECONDS" =~ ^[1-9][0-9]*$ ]]; then echo "TRANSACTION_WAIT_SECONDS must be a positive integer" >&2 exit 2 fi REFERENCE_FILTER="$( jq -nc --arg reference "$REFERENCE" \ '{"$match":{"reference":$reference}}' )" TRANSACTION_ID="" TRANSACTION_DEADLINE=$((SECONDS + TRANSACTION_WAIT_SECONDS)) while (( SECONDS < TRANSACTION_DEADLINE )); do TRANSACTION_RESULTS="$(curl --fail-with-body --silent --show-error --get \ "${CURL_AUTH[@]}" \ --data-urlencode "filter=$REFERENCE_FILTER" \ --data-urlencode 'pageSize=2' \ "$LEDGER_URL/v3/$LEDGER_NAME/transactions")" TRANSACTION_MATCH_COUNT="$(jq --arg reference "$REFERENCE" \ '[.data[] | select(.reference == $reference)] | length' \ <<<"$TRANSACTION_RESULTS")" case "$TRANSACTION_MATCH_COUNT" in 0) sleep 1 ;; 1) TRANSACTION_ID="$(jq -er --arg reference "$REFERENCE" ' [.data[] | select(.reference == $reference)][0].id | tostring ' <<<"$TRANSACTION_RESULTS")" break ;; *) echo "More than one transaction matched reference $REFERENCE" >&2 exit 1 ;; esac done if [ -z "$TRANSACTION_ID" ]; then echo "Transaction index did not expose reference $REFERENCE within ${TRANSACTION_WAIT_SECONDS}s; correlation is still unknown" >&2 exit 1 fi ``` The loop waits for a positive match and reports an **unknown** correlation after its deadline. `pageSize=2` is intentional: references are unique, and a second exact match would reveal a violated assumption instead of silently choosing one. Next, traverse the ledger logs until the created-transaction payload contains that transaction ID. The HTTP log list can also briefly lag behind a fresh write. Keep the cursor at the last log returned and poll from there until the target arrives or the wait expires. The `after` value comes from the ledger-local log ID, while `LOG_SEQUENCE` comes from the enclosing global log: ```bash if ! [[ "$LOG_WAIT_SECONDS" =~ ^[1-9][0-9]*$ ]]; then echo "LOG_WAIT_SECONDS must be a positive integer" >&2 exit 2 fi LOG_PAGE_SIZE=1000 LOG_AFTER="" LOG_SEQUENCE="" LOG_DEADLINE=$((SECONDS + LOG_WAIT_SECONDS)) while (( SECONDS < LOG_DEADLINE )); do log_curl_args=( curl --fail-with-body --silent --show-error --get "${CURL_AUTH[@]}" --data-urlencode "pageSize=$LOG_PAGE_SIZE" ) if [ -n "$LOG_AFTER" ]; then log_curl_args+=(--data-urlencode "after=$LOG_AFTER") fi log_curl_args+=("$LEDGER_URL/v3/$LEDGER_NAME/logs") LOG_PAGE="$("${log_curl_args[@]}")" LOG_SEQUENCE="$(jq -r --arg transactionId "$TRANSACTION_ID" ' [.data[] | select( (.payload.apply.log.data.createdTransaction.transaction.id | tostring) == $transactionId ) | (.sequence | tostring)] | if length == 0 then "" elif length == 1 then .[0] else error("found more than one creation log for this transaction") end ' <<<"$LOG_PAGE")" if [ -n "$LOG_SEQUENCE" ]; then break fi LOG_ROW_COUNT="$(jq '.data | length' <<<"$LOG_PAGE")" if (( LOG_ROW_COUNT > 0 )); then LOG_AFTER="$( jq -er '.data[-1].payload.apply.log.id | tostring' <<<"$LOG_PAGE" )" fi if (( LOG_ROW_COUNT < LOG_PAGE_SIZE )); then sleep 1 fi done if [ -z "$LOG_SEQUENCE" ]; then echo "Log index did not expose the creation log for transaction $TRANSACTION_ID within ${LOG_WAIT_SECONDS}s; correlation is still unknown" >&2 exit 1 fi ``` Advancing `LOG_AFTER` is safe because ledger logs become visible in ledger-local ID order, without gaps. A short or empty page means “no more logs are visible yet,” not “the creation log does not exist.” The loop waits there for more logs and never turns the timeout into an absence claim. Finally, use the global log sequence to locate and retrieve the audit entry. The HTTP audit filter reads an asynchronous secondary index, so wait for that index instead of treating an empty first response as proof that the audit entry is missing: ```bash if ! [[ "$AUDIT_WAIT_SECONDS" =~ ^[1-9][0-9]*$ ]]; then echo "AUDIT_WAIT_SECONDS must be a positive integer" >&2 exit 2 fi AUDIT_SEQUENCE="" AUDIT_DEADLINE=$((SECONDS + AUDIT_WAIT_SECONDS)) while (( SECONDS < AUDIT_DEADLINE )); do AUDIT_RESULTS="$(curl --fail-with-body --silent --show-error --get \ "${CURL_AUTH[@]}" \ --data-urlencode "filter=log_seq == $LOG_SEQUENCE" \ --data-urlencode 'pageSize=100' \ "$LEDGER_URL/v3/_/audit-entries")" AUDIT_MATCH_COUNT="$(jq '.data | length' <<<"$AUDIT_RESULTS")" case "$AUDIT_MATCH_COUNT" in 0) sleep 1 ;; 1) AUDIT_SEQUENCE="$(jq -er '.data[0].sequence | tostring' \ <<<"$AUDIT_RESULTS")" break ;; *) echo "More than one audit entry matched log sequence $LOG_SEQUENCE" >&2 exit 1 ;; esac done if [ -z "$AUDIT_SEQUENCE" ]; then echo "Audit index did not expose log sequence $LOG_SEQUENCE within ${AUDIT_WAIT_SECONDS}s; correlation is still unknown" >&2 exit 1 fi AUDIT_ENTRY="$(curl --fail-with-body --silent --show-error \ "${CURL_AUTH[@]}" \ "$LEDGER_URL/v3/_/audit-entries/$AUDIT_SEQUENCE")" jq -e \ --arg ledger "$LEDGER_NAME" \ --arg logSequence "$LOG_SEQUENCE" \ --arg idempotencyKey "$EXPECTED_IDEMPOTENCY_KEY" ' .data.success != null and ((.data.ledgers | index($ledger)) != null) and any(.data.items[]; (.logSequence | tostring) == $logSequence) and .data.idempotency.key == $idempotencyKey ' <<<"$AUDIT_ENTRY" jq \ --arg transactionId "$TRANSACTION_ID" \ --arg logSequence "$LOG_SEQUENCE" '{ transactionId: $transactionId, logSequence: $logSequence, auditSequence: (.data.sequence | tostring), idempotencyKey: (.data.idempotency.key // null), callerSubject: (.data.callerSnapshot.identity.subject // null) }' <<<"$AUDIT_ENTRY" ``` The loop waits for a positive match and fails with an **unknown** result after its deadline; it never turns temporary audit-index lag into a missing-record conclusion. The first `jq` prints `true` only when all three records agree on the ledger, global log sequence, and expected idempotency key. The second prints the correlation summary. The audit entry's `sequence` is its own identifier; it is not the global log sequence. The caller snapshot is the identity and permissions Ledger recorded when it admitted the proposal. The audit API cannot filter by idempotency key. If the key is the only available clue, narrow an audit list by ledger, caller, time, and outcome, then retrieve candidate details and compare `.data.idempotency.key` in the client. ## Investigate failed proposals A failed proposal has no transaction or committed log to follow. Query the audit trail directly by outcome, ledger, caller, proposal, or time. A date filter on the ledger log route additionally requires the opt-in log-date index; see [Create and Manage Indexes](/modules/ledger/querying/indexes). For a recently failed proposal, use its idempotency key as the final correlation check. The filter narrows the candidate set, but the HTTP audit index can lag, so retry for a bounded period and retrieve each compact candidate before deciding: ```bash : "${EXPECTED_FAILED_IDEMPOTENCY_KEY:?set this to the idempotency key of the failed request}" if ! [[ "$FAILED_AUDIT_WAIT_SECONDS" =~ ^[1-9][0-9]*$ ]]; then echo "FAILED_AUDIT_WAIT_SECONDS must be a positive integer" >&2 exit 2 fi FAILED_AUDIT_SEQUENCE="" FAILED_AUDIT_ENTRY="" FAILED_AUDIT_DEADLINE=$((SECONDS + FAILED_AUDIT_WAIT_SECONDS)) while (( SECONDS < FAILED_AUDIT_DEADLINE )); do FAILED_AUDIT_RESULTS="$(curl --fail-with-body --silent --show-error --get \ "${CURL_AUTH[@]}" \ --data-urlencode "filter=ledger == \"$LEDGER_NAME\" and outcome == failure" \ --data-urlencode 'pageSize=100' \ --data-urlencode 'reverse=true' \ "$LEDGER_URL/v3/_/audit-entries")" while IFS= read -r FAILED_CANDIDATE_SEQUENCE; do [ -n "$FAILED_CANDIDATE_SEQUENCE" ] || continue FAILED_CANDIDATE_ENTRY="$(curl --fail-with-body --silent --show-error \ "${CURL_AUTH[@]}" \ "$LEDGER_URL/v3/_/audit-entries/$FAILED_CANDIDATE_SEQUENCE")" if jq -e --arg key "$EXPECTED_FAILED_IDEMPOTENCY_KEY" \ '.data.idempotency.key == $key' \ <<<"$FAILED_CANDIDATE_ENTRY" >/dev/null; then FAILED_AUDIT_SEQUENCE="$FAILED_CANDIDATE_SEQUENCE" FAILED_AUDIT_ENTRY="$FAILED_CANDIDATE_ENTRY" break fi done < <(jq -r '.data[].sequence | tostring' <<<"$FAILED_AUDIT_RESULTS") [ -z "$FAILED_AUDIT_SEQUENCE" ] || break sleep 1 done if [ -z "$FAILED_AUDIT_SEQUENCE" ]; then echo "No current audit match after ${FAILED_AUDIT_WAIT_SECONDS}s; the failed proposal remains uncorrelated, not proven absent" >&2 exit 1 fi jq -e \ --arg ledger "$LEDGER_NAME" \ --arg key "$EXPECTED_FAILED_IDEMPOTENCY_KEY" ' .data.failure != null and ((.data.ledgers | index($ledger)) != null) and (.data.idempotency.key == $key) ' <<<"$FAILED_AUDIT_ENTRY" jq '.data | { sequence: (.sequence | tostring), idempotencyKey: .idempotency.key, failure, items }' <<<"$FAILED_AUDIT_ENTRY" ``` An audit item has `logSequence: 0` when its order produced no log. If the candidate volume can exceed 100 entries during the wait window, narrow the filter further with the known caller, proposal, or time range, or page through all candidates. A timeout is an observability result, not proof that Ledger never admitted the failed proposal. Audit routes are bucket-scoped because one proposal can contain orders for more than one ledger. Select a ledger with the audit `filter`; do not add a ledger name to the URL. ## Use ledgerctl over gRPC `ledgerctl` exposes the gRPC read surface and can print machine-readable output: ```bash ledgerctl accounts get users:alice --ledger "$LEDGER_NAME" --json ledgerctl transactions get "$TRANSACTION_ID" --ledger "$LEDGER_NAME" --json ledgerctl logs list --ledger "$LEDGER_NAME" --page-size 100 --json ledgerctl audit list --filter "ledger == \"$LEDGER_NAME\" and outcome == failure" --json ledgerctl ledgers stats --ledger "$LEDGER_NAME" --json ``` Use gRPC or `ledgerctl` when the read must wait for a minimum log sequence or target a query checkpoint. HTTP reads use the default linearizable consistency for each request, but the direct list routes do not expose those additional controls. A linearizable barrier confirms committed cluster state; it does not by itself make an asynchronous secondary index current. See [Choose Read Consistency and Use Checkpoints](/modules/ledger/querying/consistency-and-checkpoints). ## Verify the result - A direct lookup returns `404` when the record does not exist in the selected ledger or bucket. - A list returning no rows is not proof that a recent write failed. First check the selected ledger, filter, index readiness, and read-consistency choice. - A transaction proves that Ledger committed the movement. An audit failure explains why Ledger rejected a proposal before it became a transaction. - Statistics are not an instant write receipt. `transactionCount` and `logCount` advance synchronously, while the [seven usage counters](/modules/ledger/querying/analyze-and-aggregate#read-ledger-statistics) can briefly lag on the replica that serves the request. See [Verify Ledger Integrity](/modules/ledger/operations/integrity) for an integrity assessment of stored state; query results are not an integrity proof. Continue with [Filter and Paginate Results](/modules/ledger/querying/filters-and-pagination) when one read must return a subset larger than a single response. --- ## Create Transactions with Postings Source: https://docs.formance.com/modules/ledger/transactions/create-with-postings Use explicit postings when your application has already decided every source, destination, asset, and amount. Ledger commits the postings as one transaction or rejects the request without applying a partial transaction. Use [Numscript](/modules/ledger/transactions/numscript) instead when the result depends on balances, fallback accounts, or percentage splits that Ledger should resolve at execution time. ## Prerequisites You need: - an existing ledger in normal mode; - a token with `ledger:write` for the transaction and `ledger:read` for the account checks; - the base URL of the Ledger HTTP service, without a trailing slash; - amounts converted to integer minor units. If the ledger enforces account types, replace the demo addresses with values that match its configured patterns. The examples use these environment variables: ```bash set -eo pipefail ``` Ledger stores amounts as integers. The `/2` precision in `USD/2` means that `10000` represents USD 100.00. Do not send a floating-point amount. ## Create a multi-posting transaction This transaction receives USD 100.00 into a clearing account, sends USD 97.00 to a merchant, and records a USD 3.00 fee. The clearing account ends at zero. ```bash HTTP_STATUS=$(curl --silent --show-error \ --output /tmp/ledger-transaction.json \ --write-out "%{http_code}" \ --request POST "$LEDGER_URL/v3/$LEDGER_NAME/transactions" \ --header "Authorization: Bearer $LEDGER_TOKEN" \ --header "Content-Type: application/json" \ --header "Idempotency-Key: $PAYMENT_REFERENCE:create" \ --data @- < The HTTP and gRPC create surfaces, and `ledgerctl transactions create --force`, support forced writes. `force: true` bypasses balance checks and can create negative balances. Reserve it for a controlled correction or import, record the approval context in metadata, and verify every affected account afterward. ## Handle errors | Result | Meaning | Action | | --- | --- | --- | | `400` | Invalid body, more than one content source, malformed posting, or a business validation failure | Read `errorCode`, correct the request, and use a new idempotency key if the logical operation changes. | | `401` / `403` | Missing credentials or insufficient scope | Refresh the token or grant `ledger:write`. Do not retry unchanged credentials. | | `404` | The ledger does not exist | Check `LEDGER_NAME` and the target environment. | | `409` | The reference is already claimed or the idempotency key conflicts | Determine whether the existing transaction is the intended result before creating another operation. | | `503` | Ledger cannot currently accept the write, or concurrent state invalidated proposal inputs | Honor `Retry-After` when present; otherwise use bounded exponential backoff. Retry with the same key and identical body while the outcome is uncertain. | ## Retry or correct the operation Keep the same `Idempotency-Key` and the same request body when a timeout or `503` leaves the outcome uncertain. Ledger returns the original outcome instead of creating a second transaction while the key remains in its retention window. A definitive balance or account-rule rejection that Ledger recorded as the operation's outcome is also retained with its key. If you fund an account, change `force`, or otherwise correct the logical operation, send the corrected content with a new idempotency key. The HTTP status alone does not prove whether a rejection was retained; see [Retry Writes Safely](/modules/ledger/transactions/idempotency-and-references) for the boundary. A committed transaction is immutable. Do not attempt to delete or edit its postings. Use [Revert Transactions](/modules/ledger/transactions/revert) to record a compensating movement, or create a separate correcting transaction when a full reversal does not match the business event. Continue with [Retry Writes Safely](/modules/ledger/transactions/idempotency-and-references) before adding this call to a production client. --- ## Ledger Source: https://docs.formance.com/modules/ledger Formance Ledger records financial movements as balanced transactions. It gives applications a programmable accounting core for holding balances, enforcing funds availability, and preserving a complete history of every change. ## Choose your next task Your first transactions in 5 minutes: fund accounts, split payments, check balances. The scripting language for modeling financial operations: splits, holds, overdrafts, and more. The data model: Ledgers, Accounts, Assets, Transactions, and the double-entry principle. Study complete models for issuing, payment acceptance, and omnibus accounts. Create a ledger, record a posting, read the resulting balance, and retry the write safely. Learn the four accounting primitives and decide what to read next. Design accounts, assets, metadata, and validation rules for your domain. Record postings, run Numscript, retry writes, and correct transactions. Read balances and history with deliberate filtering and consistency choices. Publish ledger changes to downstream systems and recover consumers safely. Deploy, secure, monitor, back up, and restore a production ledger. Express multi-leg financial rules as deterministic transactions. ## Product ledger and general ledger | Ledger type | Primary responsibility | | --- | --- | | **Product ledger** | Controls balances and financial movements inside an application, where concurrency, scarcity, and an exact history matter at transaction time. | | **General ledger** | Classifies financial events to report the financial position of a business, commonly in an ERP or accounting system. | Formance Ledger is a product ledger. You define the account structure and business rules your application needs, then map the resulting financial events into your general ledger where required. Ledger enforces the double-entry transaction model by design: every transaction involves two or more accounts in compensating directions, and balances are derived from an immutable log. It stays flexible on accounting business rules — account classification, debit/credit conventions — so you choose how much accounting logic to handle in real time versus delegating to a GL. Start with account names that describe the business boundary they represent. A deliberate [chart of accounts](/modules/ledger/core-concepts/chart-of-accounts) makes later reconciliation and reporting work easier. For details on data immutability, see [Ledger](/modules/ledger/core-concepts/ledger#data-immutability). Start with account names that describe the business boundary they represent. A deliberate [chart of accounts](/modules/ledger/modeling/chart-of-accounts) makes later reconciliation and reporting work easier. ## Core model | Resource | Meaning | | --- | --- | | **Ledger** | An isolated accounting namespace for accounts and transactions. | | **Account** | An address that holds balances for one or more assets. | | **Transaction** | An atomic group of postings recorded together. | | **Posting** | One movement of an integer amount and asset from a source account to a destination account. | Every posting has one source and one destination. Ledger applies all postings in a transaction together or rejects the transaction without a partial result. It derives balances from the recorded movements instead of accepting balance updates directly. ## History and corrections Ledger keeps an append-only history of financial operations. You do not edit a recorded transaction in place. When a business event needs correction, record a new operation that preserves both the original event and the correction. ## One ledger or several Use one ledger when all accounts share the same business and operational boundary. Use several ledgers when tenants or domains require separate accounting namespaces. Accounts and transactions never cross ledger boundaries by accident. If workloads require independent availability, capacity, security, or recovery policies, place them in separate Ledger deployments rather than relying only on logical ledger separation. For details on single vs multi-ledger architecture, see [Ledger](/modules/ledger/core-concepts/ledger#single-vs-multi-ledger). --- ## Getting Started Source: https://docs.formance.com/modules/payments/connectors/generic/getting-started The Generic Connector for Formance Payments provides a way to connect your Formance Stack with Payment Service Providers that are not natively supported by Formance. In this tutorial, you will connect your Formance Stack to a **mock PSP server** that simulates a real Payment Service Provider. You will generate test data through the mock's admin panel and watch it flow into Formance through the Generic Connector. The Generic Connector interacts with a Payment Service Provider by sending requests formatted according to the [Generic Connector contract](/modules/payments/connectors/generic/how-it-works) to an endpoint that you develop and provide. This endpoint acts as a middleware between Formance and the PSP. ```mermaid graph LR Payments["Payments Core"] connector["Generic Connector"] service["Integration Service"] psp["Payment Service Provider"] subgraph Formance Payments Payments <--> connector end subgraph Your infrastructure connector <-- HTTP --> service end service <-- PSP specific protocol --> psp ``` ## Setup For this tutorial, you will use the **Generic Connector Mock** — a multi-tenant PSP simulator hosted at `https://formance-generic-connector-mock.fly.dev`. The mock implements the full Generic Connector contract (accounts, beneficiaries, transactions, payouts, transfers) and ships with a browser admin panel for seeding test data. The mock server is multi-tenant: each API key creates an isolated dataset. Choose a unique key that will be yours for this tutorial (e.g. `my-company-test`, your email, or a UUID). Anyone using the same API key sees and modifies the same data. Pick something unique to avoid collisions. Open the [mock admin panel](https://formance-generic-connector-mock.fly.dev/admin.html) in your browser and enter your API key in the header field. From there you can: - Create **internal accounts** and **beneficiaries** using the **New** button. - Generate **transactions** (payins, payouts, transfers) using the **Generate** button. - Simulate payment lifecycle transitions (disputes, refunds, captures) using the action buttons on each row. Create a few accounts and beneficiaries, then generate some transactions so the Generic Connector has data to poll. Create a `generic-connector.json` file with your API key and the mock endpoint: ```json title="generic-connector.json" { "name": "generic-tutorial", "apiKey": "YOUR_UNIQUE_API_KEY", "endpoint": "https://formance-generic-connector-mock.fly.dev", "pollingPeriod": "20m" } ``` Replace `YOUR_UNIQUE_API_KEY` with the same key you used in the admin panel. Then install the connector on your stack: ```bash fctl payments connectors install generic generic-connector.json ``` You should see output similar to: ``` You are about to install connector 'generic'. Do you want to continue [Y/n]y SUCCESS generic: connector 'eyJQcm92aWRlciI6IkdFTkVSSUMiLCJSZWZlcmVuY2UiOiI2NzI5YjI5NS00MGZmLTQzOWQtOTU1NS1mNTYwN2M2MzAzODUifQ' installed! ``` ## Explore the data Open the Formance console: ```bash fctl ui ``` Navigate to the Payments page. You should see the accounts, beneficiaries, and transactions you created in the mock admin panel flowing into Formance. If you don't see any data yet, wait for the next polling cycle (based on the `pollingPeriod` you configured) or click the **Refresh status** button. ## Generate more data You can return to the [mock admin panel](https://formance-generic-connector-mock.fly.dev/admin.html) at any time to: - Generate additional transactions to test incremental polling. - Transition transaction statuses (e.g. move a payment from `PENDING` to `SUCCEEDED`, trigger a `DISPUTE`, issue a `REFUND`) to observe how status changes propagate into Formance. - Create new accounts and beneficiaries. The Generic Connector picks up any new or updated data on the next polling cycle. Use a short `pollingPeriod` (e.g. `20m`, the connector's minimum) during testing so changes appear quickly in Formance. For production, a longer period like `40m` or `2h` reduces load on your integration service. --- ## Quick Start Source: https://docs.formance.com/modules/ledger/quick-start Get from zero to your first transactions. By the end you have funded an account, split a payment, and read the resulting balances, all using Numscript.

Estimated effort: 5 minutes

## Record your first transaction This creates a ledger named `main`. Every account and transaction in this guide lives here. Send $100 from `@world` (an infinite funding source) to a user account: ```numscript send [USD/2 10000] ( source = @world destination = @user:alice ) ``` ```bash fctl ledger transactions num - --confirm --ledger main <<< 'send [USD/2 10000] ( source = @world destination = @user:alice )' ``` `USD/2` means US dollars with two decimal places, so `10000` represents $100.00. Ledger creates an account on first use. Alice buys something for $30. The merchant gets 90%, the platform takes 10% as a fee: ```numscript send [USD/2 3000] ( source = @user:alice destination = { 90% to @merchant:bob remaining to @platform:fees } ) ``` {"fctl ledger transactions num - --confirm --ledger main <<< 'send [USD/2 3000] ( source = @user:alice destination = { 90% to @merchant:bob remaining to @platform:fees } )'"} The ledger enforces that Alice has sufficient funds. If she does not, Ledger rejects the whole transaction: no partial execution, no overdraft. Ledger returns these balances: | Account | Balance (USD/2) | Meaning | |---------|---------------:|---------| | `user:alice` | 7000 | \$70.00 remaining | | `merchant:bob` | 2700 | \$27.00 (90% of \$30) | | `platform:fees` | 300 | \$3.00 (10% of \$30) | Every cent is accounted for. The sum of all non-world balances equals the total funded. ## What just happened - **Double-entry**: every `send` creates balanced movements — what leaves one account enters another - **`@world`**: a special infinite source/sink for money entering and exiting the ledger - **`USD/2`**: [Universal Monetary Notation](/modules/numscript/monetary-notation) — the asset type and precision, no floating-point - **Splits**: Numscript natively handles percentage splits with `remaining` catching rounding - **Scarcity**: the ledger rejects transactions when the source account has insufficient funds ## Overview In this guide, you will: - create a disposable ledger; - record USD 100.00 into an account; - read the result back and confirm the balance; - retry the same write and prove it is not recorded twice.

Estimated effort: 15 minutes

Every step uses the HTTP API, because it accepts an `Idempotency-Key` header on each write — that is what lets the last step retry a transaction safely and show that Ledger records it only once. `ledgerctl` is the usual tool for day-to-day interactive work over gRPC; it is not used here because the retry behavior is easier to see when each request is explicit. ## Prerequisites Create a run identifier. The disposable ledger name, transaction reference, idempotency keys, and temporary files are derived from it: ```bash mkdir -p "$QUICKSTART_DIR" ``` If you repeat this tutorial, make sure to generate a new `RUN_ID`. Idempotency keys identify a write across the complete Ledger service until their configured retention period expires, even after the disposable ledger is deleted. ## Record and verify the transaction Check cluster availability before sending a write: ```bash curl --fail --silent --show-error \ --output "$QUICKSTART_DIR/clusterz.json" \ "$LEDGER_URL/clusterz" jq -e '.data.status == "ok"' "$QUICKSTART_DIR/clusterz.json" ``` The response must report an `ok` status: ```json { "data": { "status": "ok" } } ``` Then prove that your credentials can read Ledger data: ```bash curl --fail-with-body --silent --show-error \ "${CURL_AUTH[@]}" \ --output "$QUICKSTART_DIR/ledgers.json" \ "$LEDGER_URL/v3/" jq -e '.data | type == "array"' "$QUICKSTART_DIR/ledgers.json" ``` The `data` field contains an array. An empty array is a valid result when the service does not contain a ledger yet. Create a normal ledger explicitly. The idempotency key makes this creation safe to retry if the response is lost. ```bash curl --fail-with-body --silent --show-error \ --request POST "$LEDGER_URL/v3/$LEDGER_NAME" \ "${CURL_AUTH[@]}" \ --header 'Content-Type: application/json' \ --header "Idempotency-Key: $CREATE_LEDGER_KEY" \ --data '{"mode":"NORMAL"}' \ --output "$QUICKSTART_DIR/create-ledger.json" ``` Verify the ledger independently instead of relying only on the write response: ```bash curl --fail-with-body --silent --show-error \ "${CURL_AUTH[@]}" \ --output "$QUICKSTART_DIR/ledger.json" \ "$LEDGER_URL/v3/$LEDGER_NAME" jq -e --arg ledger "$LEDGER_NAME" ' .data | (.name == $ledger and ((.mode // "NORMAL") == "NORMAL")) ' "$QUICKSTART_DIR/ledger.json" jq '.data | {name, mode: (.mode // "NORMAL")}' \ "$QUICKSTART_DIR/ledger.json" ``` `NORMAL` is the default mode. The current server can omit `mode` for a normal ledger, so the display command normalizes an absent value to `NORMAL`. Expected result: ```json { "name": "quickstart-", "mode": "NORMAL" } ``` Record USD 100.00 entering the modeled system and arriving in Alice's wallet. The HTTP API uses `world` as the source account address. Numscript represents the same special account as `@world`. Build the request once so a retry can reuse byte-identical content: ```bash jq -n --arg reference "$TRANSACTION_REFERENCE" '{ reference: $reference, postings: [ { source: "world", destination: "wallets:alice", asset: "USD/2", amount: 10000 } ] }' > "$QUICKSTART_DIR/transaction-request.json" curl --fail-with-body --silent --show-error \ --request POST "$LEDGER_URL/v3/$LEDGER_NAME/transactions" \ "${CURL_AUTH[@]}" \ --header 'Content-Type: application/json' \ --header "Idempotency-Key: $TRANSACTION_KEY" \ --data-binary @"$QUICKSTART_DIR/transaction-request.json" \ --output "$QUICKSTART_DIR/transaction-first.json" ``` `USD/2` has two decimal places, so the integer `10000` represents USD 100.00. Store the transaction ID for the next read: ```bash .data.transaction.id | select(. != null) ' "$QUICKSTART_DIR/transaction-first.json")" test -n "$TRANSACTION_ID" ``` Read the transaction by its server-assigned ID: ```bash curl --fail-with-body --silent --show-error \ "${CURL_AUTH[@]}" \ --output "$QUICKSTART_DIR/transaction-read.json" \ "$LEDGER_URL/v3/$LEDGER_NAME/transactions/$TRANSACTION_ID" jq -e \ --arg id "$TRANSACTION_ID" \ --arg reference "$TRANSACTION_REFERENCE" ' .data.transaction | ((.id | tostring) == $id) and (.reference == $reference) and (.postings | length == 1) ' "$QUICKSTART_DIR/transaction-read.json" ``` Then read Alice's account: ```bash curl --fail-with-body --silent --show-error \ "${CURL_AUTH[@]}" \ --output "$QUICKSTART_DIR/account-first.json" \ "$LEDGER_URL/v3/$LEDGER_NAME/accounts/wallets:alice" jq -e ' .data.address == "wallets:alice" and ([ .data.volumes[] | select(.asset == "USD/2" and .color == "") | .volumes ] == [{input: "10000", output: "0", balance: "10000"}]) ' "$QUICKSTART_DIR/account-first.json" jq '.data | {address, volumes}' "$QUICKSTART_DIR/account-first.json" ``` The account contains one uncolored USD volume. Its `input` and `balance` are both the string `"10000"`, and its `output` is `"0"`. ```json { "address": "wallets:alice", "volumes": [ { "asset": "USD/2", "color": "", "volumes": { "input": "10000", "output": "0", "balance": "10000" } } ] } ``` Send the exact same body with the exact same idempotency key. This models a client retry after a timeout or lost response. ```bash curl --fail-with-body --silent --show-error \ --request POST "$LEDGER_URL/v3/$LEDGER_NAME/transactions" \ "${CURL_AUTH[@]}" \ --header 'Content-Type: application/json' \ --header "Idempotency-Key: $TRANSACTION_KEY" \ --data-binary @"$QUICKSTART_DIR/transaction-request.json" \ --output "$QUICKSTART_DIR/transaction-retry.json" ``` Compare the two transaction IDs: ```bash jq -e --arg first "$TRANSACTION_ID" ' (.data.transaction.id | tostring) == $first ' "$QUICKSTART_DIR/transaction-retry.json" jq --arg first "$TRANSACTION_ID" ' { sameTransaction: ((.data.transaction.id | tostring) == $first), first: $first, retry: (.data.transaction.id | tostring) } ' "$QUICKSTART_DIR/transaction-retry.json" ``` `sameTransaction` must be `true`. Verify the accounting result as a separate check: ```bash curl --fail-with-body --silent --show-error \ "${CURL_AUTH[@]}" \ --output "$QUICKSTART_DIR/account-after-retry.json" \ "$LEDGER_URL/v3/$LEDGER_NAME/accounts/wallets:alice" balance="$(jq -er ' .data.volumes[] | select(.asset == "USD/2" and .color == "") | .volumes.balance ' "$QUICKSTART_DIR/account-after-retry.json")" test "$balance" = "10000" ``` The balance must remain `10000`, not `20000`. Reuse an idempotency key only for an identical request. Ledger rejects the same key with different content because it cannot safely decide which write you intended. If the first request times out, the connection closes, or a `5xx` arrives after the body was sent, the transaction may already be committed. Retry the saved `transaction-request.json` with the same `TRANSACTION_KEY` until you receive a definitive response. Do not generate a new key while the outcome is unknown. Delete only the disposable ledger created for this guide: ```bash status="$(curl --fail-with-body --silent --show-error \ --request DELETE "$LEDGER_URL/v3/$LEDGER_NAME" \ "${CURL_AUTH[@]}" \ --header "Idempotency-Key: $DELETE_LEDGER_KEY" \ --output "$QUICKSTART_DIR/delete-ledger.json" \ --write-out '%{http_code}')" test "$status" = "204" read_status="$(curl --silent --show-error \ "${CURL_AUTH[@]}" \ --output "$QUICKSTART_DIR/ledger-after-delete.json" \ --write-out '%{http_code}' \ "$LEDGER_URL/v3/$LEDGER_NAME")" test "$read_status" = "404" rm -rf "$QUICKSTART_DIR" ``` If the delete response is lost, repeat the same `DELETE` with `DELETE_LEDGER_KEY`, then perform the `404` read check again. Ledger deletion is not a way to correct production history. Use transaction correction workflows for recorded business events. ## Troubleshoot the workflow | Result | What it means | What to check | | --- | --- | --- | | `/clusterz` returns `503` | The endpoint cannot currently serve cluster-dependent traffic. | Wait for the service to become available or ask the operator to check cluster health. Do not send the write yet. | | A request returns `401` | Ledger did not accept the bearer token. | Check the token issuer, audience, expiration, and whether the `Authorization` header reached Ledger. | | A request returns `403` | The token is valid but lacks the required permission. | Use `ledger:read` for reads and `ledger:write` for writes. | | A write returns `409` | The request conflicts with existing state. | Check whether the ledger or transaction reference already exists, or whether an idempotency key was reused with different content. | | A write times out, loses the connection, or returns `5xx` after sending the body | The outcome is unknown; Ledger may already have committed the write. | Retry the byte-identical request with the same idempotency key. Never switch keys until a definitive response or an independent read establishes the outcome. | | A read returns `404` | The ledger, transaction, or account was not found. | Check `LEDGER_NAME`, `TRANSACTION_ID`, and the account address before retrying. | ## Continue from here Design stable account addresses, assets, metadata, and validation rules. Move beyond one posting with Numscript, references, reversals, and bulk operations. Read larger histories with filters, pagination, indexes, and deliberate consistency. --- ## Hands-on walkthrough Source: https://docs.formance.com/modules/ledger/hands-on Ledger 3.0 has three behaviors that decide how you model a product: cumulative volumes that a reversal never rewinds, colors that segregate balances inside a single asset, and typed metadata that makes a key queryable. This page proves all three against a live server, so you can see the semantics before committing to a design. Every command runs against one disposable ledger, in one shell session, in order. Each part proves one behavior and links to the page that explains it. If you have not recorded a transaction yet, start with the [Quick Start](/modules/ledger/quick-start) first — it creates a ledger, records one transaction, and proves that retrying the write does not record it twice. This page assumes that ground. ## Prerequisites Create one run identifier, then the disposable ledger every part below uses. Deriving names from `RUN_ID` makes the walkthrough safe to repeat: ```bash mkdir -p "$WORK_DIR" curl --fail-with-body --silent --show-error \ --request POST "$LEDGER_URL/v3/$LEDGER_NAME" \ "${CURL_AUTH[@]}" \ --header "Content-Type: application/json" \ --header "Idempotency-Key: handson:$RUN_ID:create-ledger" \ --data '{}' ``` Generate a new `RUN_ID` when you repeat this page. Idempotency keys identify a write across the complete Ledger service until their configured retention period expires, even after the disposable ledger is deleted. ## Amounts, balances, and volumes Record three movements in `EUR/2`, then read the cumulative volumes they produce. `world` sends `10000` to Alice, Alice sends `2500` to a merchant, and the merchant sends `500` to platform fees. ```bash create_movement() { local reference="$1" local source="$2" local destination="$3" local amount="$4" local output="$5" curl --fail-with-body --silent --show-error \ --request POST "$LEDGER_URL/v3/$LEDGER_NAME/transactions" \ "${CURL_AUTH[@]}" \ --header "Content-Type: application/json" \ --header "Idempotency-Key: $RUN_ID:$reference:create" \ --data "$(jq -n \ --arg reference "$reference" \ --arg source "$source" \ --arg destination "$destination" \ --argjson amount "$amount" \ '{ reference: $reference, postings: [{ source: $source, destination: $destination, asset: "EUR/2", amount: $amount }] }')" \ > "$output" } create_movement \ "amounts:seed-alice" \ "world" \ "customers:alice:wallets:main" \ 10000 \ "$WORK_DIR/amounts-seed-alice.json" create_movement \ "amounts:alice-to-acme" \ "customers:alice:wallets:main" \ "merchants:acme:earnings" \ 2500 \ "$WORK_DIR/amounts-alice-to-acme.json" create_movement \ "amounts:acme-fee" \ "merchants:acme:earnings" \ "platform:fees" \ 500 \ "$WORK_DIR/amounts-acme-fee.json" ``` Read all three accounts and assert the expected state, rather than relying only on the write responses: ```bash read_account() { curl --fail-with-body --silent --show-error \ "${CURL_AUTH[@]}" \ "$LEDGER_URL/v3/$LEDGER_NAME/accounts/$1" } read_account "customers:alice:wallets:main" > "$WORK_DIR/amounts-alice.json" read_account "merchants:acme:earnings" > "$WORK_DIR/amounts-acme.json" read_account "platform:fees" > "$WORK_DIR/amounts-fees.json" jq -e ' any(.data.volumes[]; .asset == "EUR/2" and .color == "" and .volumes == {input:"10000", output:"2500", balance:"7500"} ) ' "$WORK_DIR/amounts-alice.json" jq -e ' any(.data.volumes[]; .asset == "EUR/2" and .color == "" and .volumes == {input:"2500", output:"500", balance:"2000"} ) ' "$WORK_DIR/amounts-acme.json" jq -e ' any(.data.volumes[]; .asset == "EUR/2" and .color == "" and .volumes == {input:"500", output:"0", balance:"500"} ) ' "$WORK_DIR/amounts-fees.json" ``` Now revert the fee transaction and read the two affected accounts again. The balances recover, but `input` and `output` keep both movements — a reversal adds compensating postings rather than subtracting from cumulative volumes: ```bash FEE_TRANSACTION_ID="$(jq -er '.data.transaction.id' "$WORK_DIR/amounts-acme-fee.json")" REVERT_STATUS="$(curl --silent --show-error \ --output "$WORK_DIR/amounts-acme-fee-revert.json" \ --write-out '%{http_code}' \ --request POST \ "${CURL_AUTH[@]}" \ --header "Content-Type: application/json" \ --header "Idempotency-Key: $RUN_ID:amounts:acme-fee:revert" \ --data '{"metadata":{"reason":"handson_check"}}' \ "$LEDGER_URL/v3/$LEDGER_NAME/transactions/$FEE_TRANSACTION_ID/revert")" test "$REVERT_STATUS" = "201" read_account "merchants:acme:earnings" > "$WORK_DIR/amounts-acme-after-revert.json" read_account "platform:fees" > "$WORK_DIR/amounts-fees-after-revert.json" jq -e ' any(.data.volumes[]; .asset == "EUR/2" and .color == "" and .volumes == {input:"3000", output:"500", balance:"2500"} ) ' "$WORK_DIR/amounts-acme-after-revert.json" jq -e ' any(.data.volumes[]; .asset == "EUR/2" and .color == "" and .volumes == {input:"500", output:"500", balance:"0"} ) ' "$WORK_DIR/amounts-fees-after-revert.json" ``` Finally, total a set of accounts with the aggregate endpoint. `filter` is the only account selector it accepts — an unrecognized parameter is ignored rather than rejected, so a misspelled selector returns a successful total over every account in the ledger, `world` included: ```bash curl --fail-with-body --silent --show-error \ "${CURL_AUTH[@]}" \ --get --data-urlencode 'filter=address ^= "customers:"' \ "$LEDGER_URL/v3/$LEDGER_NAME/volumes" \ > "$WORK_DIR/customer-volumes.json" curl --fail-with-body --silent --show-error \ "${CURL_AUTH[@]}" \ "$LEDGER_URL/v3/$LEDGER_NAME/volumes" \ > "$WORK_DIR/all-volumes.json" jq -e '.data.volumes | length > 0' "$WORK_DIR/customer-volumes.json" >/dev/null ``` The response sums per asset and color across the selected accounts; it carries no addresses, so nothing in it says which accounts were selected. Prove the filter took effect by comparing it with the unfiltered aggregate, using the asset this section created. `world` sent `10000`, Alice sent `2500`, the fee movement sent `500`, and reverting that fee added a compensating `500` — so the whole ledger holds `13500` of output while the `customers:` selection reports Alice's `2500`: ```bash sum_output() { jq -r '[.data.volumes[] | select(.asset == "EUR/2") | .output | tonumber] | add // 0' "$1" } SCOPED_OUTPUT="$(sum_output "$WORK_DIR/customer-volumes.json")" TOTAL_OUTPUT="$(sum_output "$WORK_DIR/all-volumes.json")" test "$SCOPED_OUTPUT" -lt "$TOTAL_OUTPUT" ``` If those two are equal, the selector was ignored and the "scoped" total is the whole ledger. How address patterns are validated, what `AUDIT` and `STRICT` enforcement do, and how a persistence mode changes what an account read returns. This walkthrough does not create account types; that rollout needs its own ledger. Why amounts are unsigned integers, how precision is encoded in the asset, what `input`, `output`, and `balance` each answer, and the aggregate query parameters. ## Colors: isolated balances within one asset Create three balances on one account in a single transaction — one uncolored, two colored. Color follows the value from source to destination, so a posting cannot debit one color and credit another. ```bash curl --fail-with-body --silent --show-error \ --request POST "$LEDGER_URL/v3/$LEDGER_NAME/transactions" \ "${CURL_AUTH[@]}" \ --header "Content-Type: application/json" \ --header "Idempotency-Key: $RUN_ID:color:dataset" \ --data '{ "reference": "color-dataset-001", "postings": [ { "source": "world", "destination": "customers:alice:wallets:main", "asset": "USD/2", "amount": 10000 }, { "source": "world", "destination": "customers:alice:wallets:main", "asset": "USD/2", "amount": 5000, "color": "GRANTS" }, { "source": "world", "destination": "customers:alice:wallets:main", "asset": "USD/2", "amount": 2500, "color": "OPS" } ] }' ``` Read the account without a collapse option. Ledger returns one volume row per `(asset, color)`, including `color: ""` for the uncolored bucket: ```bash curl --fail-with-body --silent --show-error \ "${CURL_AUTH[@]}" \ "$LEDGER_URL/v3/$LEDGER_NAME/accounts/customers:alice:wallets:main" ``` Now prove the buckets are isolated. Attempt to spend `5001` from `GRANTS`, which holds `5000`. Capture both accounts before the request so you can show the rejection changed neither: ```bash account_state() { local address="$1" local body local status body="$(mktemp)" status="$(curl --silent --show-error \ --output "$body" \ --write-out '%{http_code}' \ "${CURL_AUTH[@]}" \ "$LEDGER_URL/v3/$LEDGER_NAME/accounts/$address")" case "$status" in 200) jq -cS '{ status: 200, volumes: (.data.volumes | sort_by([.asset, .color])) }' "$body" ;; 404) printf '{"status":404}\n' ;; *) rm -f "$body" return 1 ;; esac rm -f "$body" } SOURCE_ACCOUNT="customers:alice:wallets:main" DESTINATION_ACCOUNT="merchants:qa:overspend" SOURCE_BEFORE="$(account_state "$SOURCE_ACCOUNT")" DESTINATION_BEFORE="$(account_state "$DESTINATION_ACCOUNT")" OVERSPEND_STATUS="$(curl --silent --show-error \ --output "$WORK_DIR/color-overspend.json" \ --write-out '%{http_code}' \ --request POST "$LEDGER_URL/v3/$LEDGER_NAME/transactions" \ "${CURL_AUTH[@]}" \ --header "Content-Type: application/json" \ --header "Idempotency-Key: $RUN_ID:color:overspend" \ --data '{ "reference": "color:overspend:expected-failure", "postings": [{ "source": "customers:alice:wallets:main", "destination": "merchants:qa:overspend", "asset": "USD/2", "amount": 5001, "color": "GRANTS" }] }')" test "$OVERSPEND_STATUS" = "400" jq -e '.errorCode == "INSUFFICIENT_FUNDS"' "$WORK_DIR/color-overspend.json" SOURCE_AFTER="$(account_state "$SOURCE_ACCOUNT")" DESTINATION_AFTER="$(account_state "$DESTINATION_ACCOUNT")" test "$SOURCE_AFTER" = "$SOURCE_BEFORE" test "$DESTINATION_AFTER" = "$DESTINATION_BEFORE" ``` The status code proves which rule rejected the request. The two independent reads prove the rejection was atomic. Collapse the colored rows for a reporting view. This sums the buckets and returns the result with `color: ""`; it does not merge or recolor stored balances: ```bash curl --fail-with-body --silent --show-error \ "${CURL_AUTH[@]}" \ "$LEDGER_URL/v3/$LEDGER_NAME/accounts/customers:alice:wallets:main?collapseColors=true" curl --fail-with-body --silent --show-error \ "${CURL_AUTH[@]}" \ --get --data-urlencode 'filter=address ^= "customers:"' \ "$LEDGER_URL/v3/$LEDGER_NAME/volumes?collapseColors=true" ``` Never authorize a spend from a collapsed balance when the spend must respect color restrictions. A collapsed empty color is a summary across buckets; an uncollapsed empty color is the real uncolored bucket. ### Reclassify value between colors There is no in-place recoloring primitive. Move value with one transaction holding two equal postings — retire the old color to `world`, issue the new color from `world`. Both postings must be in the same transaction so Ledger commits both or neither. Read the starting balances first: ```bash color_balance() { local color="$1" jq -er --arg color "$color" ' [ .data.volumes[] | select(.asset == "USD/2" and .color == $color) | .volumes.balance | tonumber ] | if length == 0 then 0 elif length == 1 then .[0] else error("duplicate color bucket") end ' } usd_total() { jq -er ' [.data.volumes[] | select(.asset == "USD/2") | .volumes.balance | tonumber] | add // 0 ' } BEFORE_RECLASSIFICATION="$(read_account customers:alice:wallets:main)" GRANTS_BEFORE="$(color_balance GRANTS <<<"$BEFORE_RECLASSIFICATION")" OPS_BEFORE="$(color_balance OPS <<<"$BEFORE_RECLASSIFICATION")" TOTAL_BEFORE="$(usd_total <<<"$BEFORE_RECLASSIFICATION")" test "$GRANTS_BEFORE" -ge 1000 ``` Submit both postings in one request: ```bash RECLASSIFICATION_STATUS="$(curl --silent --show-error \ --output "$WORK_DIR/color-reclassification.json" \ --write-out '%{http_code}' \ --request POST "$LEDGER_URL/v3/$LEDGER_NAME/transactions" \ "${CURL_AUTH[@]}" \ --header "Content-Type: application/json" \ --header "Idempotency-Key: $RUN_ID:color:reclassify" \ --data '{ "reference": "color:reclassify:grants-to-ops:001", "metadata": { "reason": "approved_budget_reclassification", "approvalId": "approval-001" }, "postings": [ { "source": "customers:alice:wallets:main", "destination": "world", "asset": "USD/2", "amount": 1000, "color": "GRANTS" }, { "source": "world", "destination": "customers:alice:wallets:main", "asset": "USD/2", "amount": 1000, "color": "OPS" } ] }')" test "$RECLASSIFICATION_STATUS" = "201" ``` Read the account again and assert both bucket changes and the invariant total: ```bash AFTER_RECLASSIFICATION="$(read_account customers:alice:wallets:main)" GRANTS_AFTER="$(color_balance GRANTS <<<"$AFTER_RECLASSIFICATION")" OPS_AFTER="$(color_balance OPS <<<"$AFTER_RECLASSIFICATION")" TOTAL_AFTER="$(usd_total <<<"$AFTER_RECLASSIFICATION")" test "$GRANTS_AFTER" -eq "$((GRANTS_BEFORE - 1000))" test "$OPS_AFTER" -eq "$((OPS_BEFORE + 1000))" test "$TOTAL_AFTER" -eq "$TOTAL_BEFORE" ``` These arithmetic assertions rely on the small values in this dataset. Production clients must parse Ledger's balance strings with arbitrary-precision integer arithmetic. When to reach for colors, why conservation holds per bucket, how Numscript spends a color, what reversals preserve, and the current limits. ## Metadata and typed values Save account metadata with its own idempotency key. Note `externalSequence`: a bare JSON integer above `2^53 - 1` loses precision in the HTTP parser, so the exact value goes over the wire as a string. ```bash curl --fail-with-body --silent --show-error \ --request POST \ "${CURL_AUTH[@]}" \ --header "Content-Type: application/json" \ --header "Idempotency-Key: $RUN_ID:account-alice-metadata" \ --data '{ "customerId": "cus_01J2Y8M9D8", "country": "FR", "riskTier": 2, "active": true, "externalSequence": "9007199254740993", "reviewNote": "remove-after-verification" }' \ "$LEDGER_URL/v3/$LEDGER_NAME/accounts/customers:alice:wallets:main/metadata" ``` Read the account back through a separate request and assert what was stored: ```bash curl --fail-with-body --silent --show-error \ "${CURL_AUTH[@]}" \ "$LEDGER_URL/v3/$LEDGER_NAME/accounts/customers:alice:wallets:main" \ > "$WORK_DIR/metadata-account-after-save.json" jq -e ' .data.metadata.customerId == "cus_01J2Y8M9D8" and .data.metadata.country == "FR" and .data.metadata.riskTier == 2 and .data.metadata.active == true and .data.metadata.externalSequence == "9007199254740993" and .data.metadata.reviewNote == "remove-after-verification" ' "$WORK_DIR/metadata-account-after-save.json" ``` A save merges keys; it never removes ones you omitted. Delete a key explicitly, then prove the others survived: ```bash curl --fail-with-body --silent --show-error \ --request DELETE \ "${CURL_AUTH[@]}" \ --header "Idempotency-Key: $RUN_ID:account-alice-review-note-delete" \ "$LEDGER_URL/v3/$LEDGER_NAME/accounts/customers:alice:wallets:main/metadata/reviewNote" curl --fail-with-body --silent --show-error \ "${CURL_AUTH[@]}" \ "$LEDGER_URL/v3/$LEDGER_NAME/accounts/customers:alice:wallets:main" \ > "$WORK_DIR/metadata-account-after-delete.json" jq -e ' (.data.metadata | has("reviewNote") | not) and .data.metadata.customerId == "cus_01J2Y8M9D8" and .data.metadata.riskTier == 2 and .data.metadata.externalSequence == "9007199254740993" ' "$WORK_DIR/metadata-account-after-delete.json" ``` Do not rely on `null` in a save request to delete a key — the HTTP parser skips null entries. Use the `DELETE .../metadata/{key}` route so the deletion is explicit and auditable. ### Declare types for indexed queries Declaring a type tells Ledger how to interpret a key when it builds and queries a metadata index. Declare `riskTier` as a bounded integer, the HTTP-safe `externalSequence` string as `uint64`, and a transaction timestamp for range queries: ```bash curl --fail-with-body --silent --show-error \ --request PUT "${CURL_AUTH[@]}" \ --header "Content-Type: application/json" \ --data '{"type":"uint8"}' \ "$LEDGER_URL/v3/$LEDGER_NAME/metadata-schema/account/riskTier" curl --fail-with-body --silent --show-error \ --request PUT "${CURL_AUTH[@]}" \ --header "Content-Type: application/json" \ --data '{"type":"uint64"}' \ "$LEDGER_URL/v3/$LEDGER_NAME/metadata-schema/account/externalSequence" curl --fail-with-body --silent --show-error \ --request PUT "${CURL_AUTH[@]}" \ --header "Content-Type: application/json" \ --data '{"type":"datetime"}' \ "$LEDGER_URL/v3/$LEDGER_NAME/metadata-schema/transaction/settledAt" ``` Retrieve the declarations currently stored on the ledger: ```bash curl --fail-with-body --silent --show-error \ "${CURL_AUTH[@]}" \ "$LEDGER_URL/v3/$LEDGER_NAME/metadata-schema" ``` Remove a declaration when the key no longer has one query type. Stored values are untouched, but the typed interpretation and its index are dropped, so update dependent queries first: ```bash curl --fail-with-body --silent --show-error \ --request DELETE "${CURL_AUTH[@]}" \ "$LEDGER_URL/v3/$LEDGER_NAME/metadata-schema/account/riskTier" ``` A declaration does not create an index, and an index is not ready the moment it is registered. Building one and gating on every read replica needs direct per-pod URLs, so it stays with the operational guidance in [Metadata and Typed Values](/modules/ledger/modeling/metadata#build-the-index-and-verify-every-read-replica). Which target to use, how JSON maps to stored types, the large-integer precision trap, what a declaration changes, and how to validate historical values before a type change. ## Clean up Delete the disposable ledger, then the working directory. Everything this page created derives from `RUN_ID`, so nothing outside this run is touched: ```bash status="$(curl --fail-with-body --silent --show-error \ --request DELETE "$LEDGER_URL/v3/$LEDGER_NAME" \ "${CURL_AUTH[@]}" \ --header "Idempotency-Key: handson:$RUN_ID:delete-ledger" \ --output "$WORK_DIR/delete-ledger.json" \ --write-out '%{http_code}')" test "$status" = "204" read_status="$(curl --silent --show-error \ "${CURL_AUTH[@]}" \ --output "$WORK_DIR/ledger-after-delete.json" \ --write-out '%{http_code}' \ "$LEDGER_URL/v3/$LEDGER_NAME")" test "$read_status" = "404" rm -rf "$WORK_DIR" ``` If the delete response is lost, repeat the same `DELETE` with `handson:$RUN_ID:delete-ledger`, then run the `404` read check again. Deleting a ledger is not a way to correct history. To undo a recorded business event, use [Revert a Transaction](/modules/ledger/transactions/revert). --- ## Accounts Source: https://docs.formance.com/modules/ledger/core-concepts/accounts An account is a named container for asset balances inside a ledger. Its primary function is to group assets together based on criteria that best suit the business's needs. In a food delivery app, accounts organize financial activities efficiently. Each restaurant has an account to manage their earnings from orders. Riders have accounts to track their delivery payments. Customers have accounts for their wallets, where they can add and spend money for orders. The platform maintains separate accounts for service fees and taxes. ## Creating accounts The number of accounts in a ledger is unlimited. Accounts do not need to be created prior to being used; submitting a transaction involving an account automatically creates it in the ledger. ## Naming accounts Account addresses are **permanent identifiers**. Once an account is used in a transaction, its address is fixed and cannot be renamed. Choose names that will remain accurate for the lifetime of the account. ### What belongs in the address vs in metadata The address should encode structural identity: the account's type and a stable external ID. It should not contain mutable data, status flags, or PII that might change. | Put in the address | Put in metadata | |--------------------|-----------------| | Account type (`users`, `merchants`, `orders`) | Display name, email address | | Stable external ID (`usr-123`, `mch-abc`) | Account status (`pending`, `active`) | | Sub-account purpose (`wallet`, `earnings`) | Business attributes, tags | Good examples: ``` users:usr-123:wallet payments:stripe:ch_abc456 merchants:acme-corp:earnings ``` Bad examples: ``` users:john@example.com ← email could change orders:pending:ord-789 ← "pending" is mutable state merchants:Acme Corp:main ← spaces aren't allowed, name could change ``` ### Hierarchy enables prefix filtering The colon-separated segments are not just for readability; they enable prefix-based queries. You can retrieve every account under `users:usr-123:` to see all of a user's accounts, or every account under `payments:stripe:` to see all Stripe payment records. Design the hierarchy around how you'll query, not just how things look. To enforce specific naming conventions beyond the basic pattern, use a [Ledger Schema](/modules/ledger/working-with/ledger-schema). Schemas let you define which account structures are valid and reject transactions that use undefined accounts. ## Using Metadata Accounts can carry metadata: arbitrary key-value pairs for storing external references, labels, or any contextual information. Metadata is additive: each update merges new keys with existing ones. | Operation | Behavior | |-----------|----------| | Set a new key | Key is created with the provided value | | Set an existing key | Previous value is overwritten | | Remove a key | Supported via the [delete account metadata](/stack-api-reference/introduction) endpoint | Account metadata can also be used in Numscript transactions, see [metadata in Numscript](/modules/numscript/reference/metadata) for more information. Transactions also support metadata. See [Transactions & Postings](/modules/ledger/core-concepts/transactions#metadata) for details. --- ## Consume, Monitor, and Recover Events Source: https://docs.formance.com/modules/ledger/events/consume-and-recover This page covers the consumer side: processing Ledger events without applying a business effect twice, verifying webhook authenticity, watching each sink's progress, and getting a stalled sink moving again. Ledger advances a sink cursor only after the destination acknowledges a batch. If delivery succeeds but Ledger cannot persist the new cursor before a restart or leader change, it sends that batch again. Consumers must therefore accept duplicate deliveries without applying the same business effect twice. ## Read the event contract NATS, Kafka, and HTTP receive the complete event envelope: | Field | Meaning | | --- | --- | | `app` | Stable producer identifier. Always `ledger`. | | `version` | Event envelope generation. Always `v3`. | | `type` | One of the [supported event types](/modules/ledger/events/configure-sink#supported-event-types). | | `ledger` | Ledger name associated with the event. | | `date` | Timestamp assigned to the committed change. | | `logSequence` | Monotonically increasing position in the cluster's global log. | | `log` | Complete committed log and its typed payload. | In JSON, `logSequence` is an unsigned 64-bit number. Use an integer-safe JSON parser and an integer or decimal database type. JavaScript's `number` type cannot represent every 64-bit value exactly. ClickHouse and Databricks use a different storage shape. They write `log_sequence`, `type`, `ledger`, and `date` columns, then store a flattened event payload in `data`. Do not write one decoder and assume that all five destinations receive the same representation. ## Deduplicate before applying side effects Use the pair `(source, logSequence)` as the event identity. `logSequence` is unique inside one Ledger cluster, but the envelope does not include a cluster identifier. Assign a stable `source` value in your integration when several clusters feed the same consumer. The analytical row shape has no place to record that `source`: ClickHouse and Databricks tables hold only `log_sequence`, `type`, `ledger`, `date`, and `data`. Point each cluster at its own table. If two clusters write to one ClickHouse table, `ReplacingMergeTree` sorts by `(ledger, log_sequence)` and collapses two distinct events that happen to share that pair; on Databricks, deduplicating by `log_sequence` in a query discards them the same way. A receipt table can enforce this rule: ```sql CREATE TABLE ledger_event_receipts ( source text NOT NULL, log_sequence numeric(20, 0) NOT NULL, received_at timestamptz NOT NULL DEFAULT now(), PRIMARY KEY (source, log_sequence) ); ``` For every event, insert the receipt and apply the business change in one database transaction: ```sql BEGIN; INSERT INTO ledger_event_receipts (source, log_sequence) VALUES (:source, :log_sequence) ON CONFLICT DO NOTHING RETURNING log_sequence; -- Apply the business change only when the INSERT returned a row. COMMIT; ``` If the business change and receipt cannot share a transaction, use an inbox/outbox workflow that records the receipt durably before an asynchronous worker runs the side effect. Do not deduplicate with only `logSequence <= lastProcessedSequence`. Kafka partitions, NATS subjects, and concurrent webhook workers can expose events out of order. A single high-water mark could then discard a valid event that arrives later. Sequence gaps are also valid. Filters advance the sink cursor past excluded event types, and Ledger does not publish events for every internal log entry. Use `logSequence` to identify and compare events, not to require a contiguous series. ## Acknowledge only durable work | Destination | Consumer acknowledgement rule | | --- | --- | | NATS JetStream | Acknowledge the message after the receipt and business change commit. Configure consumer redelivery limits and route terminal failures from your application. | | Kafka | Commit the consumer offset after the receipt and business change commit. The ledger name is the record key, so Kafka preserves per-ledger order within a partition. | | HTTP | Return `2xx` only after durable acceptance. Ledger retries network errors, timeouts, and non-`2xx` responses. | | ClickHouse | Use the table as an analytical projection. Query with `FINAL` or `GROUP BY` when exact counts matter because `ReplacingMergeTree` removes duplicate rows asynchronously. | | Databricks | Deduplicate `log_sequence` in your query or downstream table. The generated Delta table uses ordinary inserts and does not collapse re-delivered rows. | The event producer scans the global log in order, but the destination can partition or process messages concurrently. If a workflow requires ordering, define the scope that matters—usually one ledger or one business entity—and serialize that scope in the consumer. ## Verify HTTP webhook signatures When an HTTP sink has a secret, Ledger sends these headers: | Header | Value | | --- | --- | | `Content-Type` | `application/json` or `application/protobuf` | | `X-Event-Type` | Lowercase event type | | `X-Ledger` | Ledger name | | `X-Log-Sequence` | Global log sequence | | `X-Webhook-Signature` | `sha256=` when a secret is configured | Calculate the HMAC over the raw request body before parsing or modifying it: ```typescript rawBody: Buffer, suppliedSignature: string | undefined, secrets: readonly string[], ): boolean { if (!suppliedSignature) { return false } const supplied = Buffer.from(suppliedSignature, "utf8") let accepted = false for (const secret of secrets) { const expected = Buffer.from( `sha256=${createHmac("sha256", secret).update(rawBody).digest("hex")}`, "utf8", ) if (supplied.length === expected.length && timingSafeEqual(supplied, expected)) { accepted = true } } return accepted } ``` Reject an invalid signature without running the consumer. The verifier accepts a set of secrets rather than one because a sink configuration cannot be updated in place: rotating a webhook secret means adding a second sink that carries the new secret, then removing the old sink once you have confirmed the new one. Both sinks deliver during that window, each signing with its own secret, so keep every valid secret in `secrets` until the old sink is gone. The replacement sink also starts without a cursor and replays the retained global log, so expect the receiver to absorb that catch-up as duplicate deliveries against its receipt table. ## Monitor each sink Read the cluster-wide status with a principal that has `ledger:OpsRead`: ```bash ledgerctl events list --json \ | jq '.sinkStatuses[] | {sinkName, cursor, error}' ``` The CLI masks secrets in its structured output. The fields have these operational meanings: | Field | Meaning | | --- | --- | | `sinkName` | Stable identity of the configuration, cursor, and error state. | | `cursor` | Highest global-log position the sink has processed. It can include filtered entries that produced no destination message. | | `error.message` | Latest publish failure for a sink that started successfully. | | `error.occurredAt` | Time at which Ledger most recently recorded that failure. | Poll this status and run a periodic synthetic delivery check. The current release branch does not expose a stable event-specific metrics contract, and status alone cannot prove end-to-end delivery. | Signal | Likely meaning | Action | | --- | --- | --- | | Cursor advances and the test event is present | The sink is reading and the destination is acknowledging events. | Continue monitoring receipt latency and consumer failures. | | Error is present and cursor stops | A running sink cannot publish. | Restore the destination or credentials, then wait for the error to clear and cursor to advance. | | Cursor stays at `0` with no error | There may be no supported event, or sink startup may have failed before error reporting began. | Commit a test event and inspect Ledger server logs for the sink name. | | Cursor passes an enabled test event but the consumer sees nothing | Ledger processed the event, but filtering, routing, or consumer configuration may be wrong. | Check the sink event filter, NATS subject, Kafka topic and key, webhook logs, or analytical table. | | One sink stalls while another advances | Each sink progresses independently. | Repair only the stalled destination; Ledger writes and other sinks continue. | | Duplicate `logSequence` values appear | Delivery succeeded before the cursor update completed, or a batch was partially acknowledged. | Keep one receipt and suppress the repeated side effect. | | Sequence values contain gaps | Filters or unsupported internal logs were skipped. | Do not treat gaps alone as data loss. Verify a known event instead. | A sink shown without an error is not necessarily delivering. Connection failures during initial startup are written to Ledger server logs but are not currently reflected in `SinkStatus`. Always require a destination receipt and cursor advancement for readiness. ## Recover a running sink after a destination outage Once a sink has started, publish failures leave its cursor unchanged and record an error. Ledger retries with exponential backoff, starting around one second and capping around one minute. A successful batch advances the cursor and clears the error. 1. Leave the sink configured while you restore the destination if its address and credentials remain valid. 2. Confirm that Ledger writes still succeed; event delivery is asynchronous and must not be part of transaction availability. 3. Restore network access, broker availability, webhook responses, or analytical-store capacity. 4. Poll `ledgerctl events list --json` until the error clears and the cursor advances. 5. Commit a uniquely referenced test transaction. 6. Confirm its `logSequence` at the destination and in your receipt table. 7. Confirm that a repeated delivery would not repeat the business side effect. Do not manually move the cursor forward to silence an error. Ledger exposes no public cursor mutation, replay, or sink-level dead-letter API. ## Recover an initial startup failure An invalid address, missing build feature, or failed connection can prevent NATS, Kafka, ClickHouse, or Databricks from starting before publish retries exist. In that case, the status can remain error-free and the sink does not automatically reconnect through the publish backoff. 1. Inspect the Ledger server logs for the sink name and the connection error. 2. Confirm that the deployed binary includes the destination's optional sink feature. 3. Correct networking, privileges, and credentials. 4. Add the corrected configuration under a new sink name. 5. Generate a test event and verify its receipt and cursor. 6. Remove the failed configuration only after the new sink works. A new name replays every supported event in the global log from the beginning. Plan destination capacity and rely on consumer deduplication during that catch-up. ## Run a failure drill Complete this drill before depending on a sink in production: 1. Deliver a baseline test event and record its `logSequence` and the sink cursor. 2. Make the destination unavailable without stopping Ledger. 3. Commit another uniquely referenced transaction and confirm that the Ledger write succeeds. 4. For a sink that had already started, confirm that its cursor stops and an error appears. 5. Restore the destination. 6. Confirm that the error clears and the cursor passes the test event's sequence. 7. Confirm that the receipt table contains one row and the business effect occurred once, even if the destination received the event more than once. See [Configure an Event Sink](/modules/ledger/events/configure-sink) when you need to change destination settings or event filters. --- ## Prepare the Source Source: https://docs.formance.com/modules/ledger/migration/prepare-source Prepare each Ledger v2 ledger as an independent migration unit. The output of this page is a source worksheet and a tested, least-privilege read path from the target environment. These checks do not change Ledger v2. ## Inventory the ledger Record the following before granting access: | Area | Record | | --- | --- | | Identity | Ledger name, bucket, environment, owner, and expected target name. | | History | Current highest log ID, transaction count, oldest retained log, and known gaps or deleted data. | | Accounting | Assets, material account prefixes, balance invariants, backdated transactions, and reversions. | | Metadata | Account and transaction keys, values with inconsistent formats, and deleted keys. | | Uniqueness | Transaction references and application-generated idempotency keys. | | Clients | Every writer, reader, scheduled job, reconciliation, SDK, and API path. | | Events | Exporters, pipelines, consumers, ordering assumptions, and deduplication keys. | | Queries | Filters, pagination, reports, and indexes required on the target. | | Operations | Peak write rate, maintenance window, source-load budget, and escalation owner. | Select representative fixtures now: high-value accounts, zero-balance accounts, each asset, boundary amounts, reverted transactions, backdated transactions, and metadata edge cases. You will use the same fixtures during rehearsal and production validation. ## Preflight an HTTP source Run the request from the same network boundary as the target cluster. Use a client credential that can read source logs and nothing more. ```bash set -euo pipefail : "${V2_URL:?set V2_URL}" : "${SOURCE_LEDGER:?set SOURCE_LEDGER}" : "${V2_TOKEN:?set V2_TOKEN}" command -v curl >/dev/null curl --fail --silent --show-error \ --header "Authorization: Bearer $V2_TOKEN" \ "$V2_URL/v2/$SOURCE_LEDGER/logs?pageSize=1" ``` The response must contain either an empty `cursor.data` array or the newest source log. Record the returned log ID as the initial source head. When Ledger v2 uses OAuth2 client credentials, test both the token endpoint and the log request. Provision the scope required by your Ledger v2 authorization policy; `ledger:read` is an example, not a universal scope name. ```bash set -euo pipefail : "${V2_TOKEN_ENDPOINT:?set V2_TOKEN_ENDPOINT}" : "${V2_CLIENT_ID:?set V2_CLIENT_ID}" : "${V2_CLIENT_SECRET:?set V2_CLIENT_SECRET}" : "${V2_SCOPES:?set V2_SCOPES}" command -v curl >/dev/null curl --fail --silent --show-error \ --user "$V2_CLIENT_ID:$V2_CLIENT_SECRET" \ --data "grant_type=client_credentials" \ --data-urlencode "scope=$V2_SCOPES" \ "$V2_TOKEN_ENDPOINT" ``` Do not save the token or client secret in the migration record. Save the secret-manager entry name, field, owner, and rotation date. ## Preflight a PostgreSQL source Run the following queries through the same hostname, TLS policy, database user, and network path that the target will use. ```sql \set source_ledger 'main-v2' SELECT name, bucket AS source_bucket FROM _system.ledgers WHERE name = :'source_ledger' \gset ``` The query must return exactly one row. `\gset` stores the returned bucket as `source_bucket`; psql quotes it as an identifier in the next query: ```sql SELECT COALESCE(MAX(id), 0) AS source_log_id FROM :"source_bucket".logs WHERE ledger = :'source_ledger'; ``` The PostgreSQL identity needs `CONNECT` on the database, `USAGE` on `_system` and the selected bucket schema, `SELECT` on `_system.ledgers`, and `SELECT` on the selected bucket's `logs` table. Grant access only to the bucket returned for this ledger. The mirror does not need permission to create or modify Ledger v2 tables or schemas. For PostgreSQL password authentication: - store the password in the target deployment's secret manager; - require TLS according to your database policy; - test certificate and hostname verification when using `verify-ca` or `verify-full`. For Amazon RDS IAM authentication: - enable IAM database authentication for the source; - grant the database user `rds_iam` as required by your RDS setup; - grant the target Ledger runtime identity `rds-db:connect` for the exact database-user ARN; - set the libpq DSN parameter to `sslmode=require`, `sslmode=verify-ca`, or `sslmode=verify-full`; - if you assume a role, test `sts:AssumeRole` from the target runtime identity first. - for rehearsal and cutover reads, generate a fresh IAM token for every psql process and pass it through that process's `PGPASSWORD`; do not store it in the DSN, command arguments, or evidence. ## Measure source capacity The mirror reads logs in ascending order and continues polling after catch-up. A production rehearsal should establish: - how many logs the mirror processes per minute; - the added API or database load; - how long a full replay takes; - whether the mirror catches up after peak write periods; - the target disk headroom during replay and index building. Keep the default batch size for the first rehearsal. Increase it only after measuring source latency, target apply latency, and operational headroom. ## Define stop conditions Stop the rehearsal or production replay when: - source request latency or database load crosses the agreed limit; - source credentials expose more access than approved; - the target cannot read the earliest retained source log; - a malformed source entry blocks the cursor; - a transformation produces an unexplained balance or metadata difference; - target disk pressure or write stalls prevent steady progress. The mirror does not repair missing source history and does not compare source and target balances. Source access is a prerequisite, not proof that the migration is correct. ## Source-ready checklist - [ ] Every source ledger has a worksheet and owner. - [ ] The target network can reach the selected source transport. - [ ] The credential has the minimum required read access. - [ ] TLS and hostname validation match the production policy. - [ ] The initial source log ID is recorded. - [ ] Representative validation fixtures are recorded. - [ ] Source load and stop thresholds are approved. - [ ] All Ledger v2 writers can be identified and frozen during cutover. Continue with [Update Ledger Clients](/modules/ledger/migration/update-clients). --- ## Assets, Amounts, Balances, and Volumes Source: https://docs.formance.com/modules/ledger/modeling/assets-amounts-and-balances Ledger records amounts as unsigned integers. The asset identifier carries the unit and, when needed, its decimal precision. This keeps monetary arithmetic deterministic and avoids floating-point rounding inside the ledger. To run these behaviors against a live ledger, follow [Amounts, balances, and volumes](/modules/ledger/hands-on#amounts-balances-and-volumes) in the hands-on walkthrough. It sets up a disposable ledger once and reproduces every result shown below. For the complete route contracts, see [Create Transactions with Postings](/modules/ledger/transactions/create-with-postings) and [Revert Transactions](/modules/ledger/transactions/revert). ## Define one asset identifier An asset uses an uppercase base followed by an optional decimal precision: ```text EUR/2 USD/2 TOKEN/18 POINTS ``` The base must match `[A-Z][A-Z0-9]{0,16}`. The optional precision is an integer from 1 through 255 without a leading zero. `USD`, `USD/2`, and `USD/4` are distinct stored assets. Choose one representation for each business asset and use it in every writer. Inconsistent precision creates separate balances that only look related to a human reader. ## Convert display values to integers The precision states how many decimal places your application applies when it displays an amount: | Display value | Asset | Ledger amount | | --- | --- | ---: | | EUR 12.34 | `EUR/2` | `1234` | | USD 0.01 | `USD/2` | `1` | | 1 token | `TOKEN/18` | `1000000000000000000` | | 25 points | `POINTS` | `25` | Use the same rule in both directions: ```text ledger amount = display amount * 10^precision display amount = ledger amount / 10^precision ``` Perform this conversion with decimal or integer arithmetic. Do not pass an IEEE-754 floating-point result into a transaction request. Posting amounts and cumulative volumes can use the full unsigned 256-bit range. JavaScript `Number` cannot represent all of these integers. Use `BigInt`, a decimal library, or a serializer that preserves the original integer text. The HTTP transaction request represents a posting amount as a JSON integer: ```json { "postings": [ { "source": "world", "destination": "customers:alice:wallets:main", "asset": "EUR/2", "amount": 1234 } ] } ``` Ledger rejects malformed asset identifiers and any operation that makes a cumulative input or output volume exceed `2^256 - 1`. ## Read input, output, and balance For each `(account, asset, color)` bucket, Ledger maintains two cumulative volumes: - `input` is the total received by the account; - `output` is the total sent by the account; - `balance` is `input - output`. Consider three movements in `EUR/2`: 1. `world` sends `10000` to Alice; 2. Alice sends `2500` to a merchant; 3. the merchant sends `500` to platform fees. | Account | Input | Output | Balance | | --- | ---: | ---: | ---: | | `customers:alice:wallets:main` | 10000 | 2500 | 7500 | | `merchants:acme:earnings` | 2500 | 500 | 2000 | | `platform:fees` | 500 | 0 | 500 | Reading an account returns those volumes as decimal strings: ```json { "data": { "address": "customers:alice:wallets:main", "volumes": [ { "asset": "EUR/2", "color": "", "volumes": { "input": "10000", "output": "2500", "balance": "7500" } } ] } } ``` Parse these response values as arbitrary-precision integers. A balance can be negative when a transaction explicitly permits an overdraft, even though `input` and `output` remain unsigned. ## Volumes preserve movement history Input and output answer different questions from balance. Two accounts can have the same balance but very different activity. A reversal records new postings in the opposite direction. It does not subtract from the original cumulative volumes. After a transfer and its reversal, the balance can return to its previous value while both input and output have increased. For `NORMAL` account types, these volumes remain available after the balance returns to zero. An `EPHEMERAL` type can remove a zero-balance volume row, and a `TRANSIENT` type does not store a volume row after the transaction. Do not use either mode when reports require lifetime input and output from direct account reads. Use: - `balance` for the current net position; - `input` and `output` for cumulative flow and turnover; - transactions and logs for the exact history that produced those values. ## Aggregate volumes across accounts Use the `GET /v3/{ledger}/volumes` endpoint when you need totals for a set of accounts. It returns decimal-string `input`, `output`, and `balance` values, and supports: | Parameter | Purpose | | --- | --- | | `filter` | Select the accounts to total. `address ^= "customers:"` matches an address prefix. This is the only account selector the route accepts. | | `groupByPrefixes` | Return separate totals for a comma-separated set of prefixes. | | `useMaxPrecision=true` | Rescale assets with the same base to the highest precision found in the result. | | `collapseColors=true` | Sum color buckets for reporting. Stored balances remain separate. | Ledger 3.0 removed the `prefix` parameter this route used to accept. An unrecognized parameter is ignored rather than rejected, so sending `prefix=` still returns `200` — with a total across every account in the ledger, `world` included, that looks exactly like a scoped one. The response carries no addresses, so nothing in it reveals the difference. `useMaxPrecision` is a reporting aid, not a correction for inconsistent writers. Standardize the asset at write time and use precision merging only when you deliberately need a combined view. ## Verify an amount integration Run the following checks before production traffic: 1. convert a display amount to an integer and back without loss; 2. send the smallest supported unit, such as `1` for `EUR/2`; 3. read the destination and confirm its `input` and `balance` increased by exactly that integer; 4. move part of the balance out and confirm only `output` increased; 5. revert the movement and confirm the balance recovers while cumulative volumes continue to record activity; 6. test values above your language's native safe-integer range. ## Common failure modes | Symptom | Cause | Correction | | --- | --- | --- | | EUR 12.34 is recorded as `12` or `1233` | Floating-point conversion or wrong scale | Convert with decimal arithmetic to `1234 EUR/2`. | | One currency appears in several balance rows | Writers use `USD`, `USD/2`, and `USD/4` inconsistently | Publish one official asset identifier. | | A client rounds large amounts | It parses JSON values through a fixed-width or floating-point number | Preserve arbitrary-precision integers end to end. | | Balance returns to zero but activity totals do not | A reversal added compensating postings | Use balance for net position and volumes for cumulative flow. | | Prefix aggregation includes `customers:10` when requesting `customers:1` | Prefix matching is byte-based | Include the intended segment delimiter, such as `customers:1:`. | Use [colors](/modules/ledger/modeling/color-of-money) only when two balances with the same asset must remain isolated. Do not encode provenance by creating ad hoc asset names. --- ## Plan Topology and Manage Cluster Lifecycle Source: https://docs.formance.com/modules/ledger/operations/cluster-lifecycle Every Ledger member stores a copy of the cluster state and participates in one Raft group. Plan changes around voters and quorum rather than around Kubernetes pod count. ## Choose the topology | Voting replicas | Majority required | Voter failures tolerated | | --- | --- | --- | | 1 | 1 | 0 | | 3 | 2 | 1 | | 5 | 3 | 2 | Use one voter only for local development or a non-critical environment. Three voters are the normal production starting point. Five voters improve failure tolerance, but every committed write crosses a larger consensus group and every voter needs the full storage and memory budget. Even replica counts do not increase failure tolerance. For example, both three and four voters tolerate one voter failure, while four voters require three members for a majority. Spread voters across independent nodes and, where possible, zones. Keep a disruption budget that preserves a majority during node drains and upgrades. Each member must own its node identity, WAL, and data volume for its whole lifetime. ```mermaid flowchart LR Desired["Operator adds a member"] --> Learner["Member joins as a learner"] Learner --> CatchUp["State catches up"] CatchUp --> Voter["Member becomes a voter"] Voter --> Tolerance["Member counts toward failure tolerance"] ``` Do not count a pod, a ready container, or a learner toward failure tolerance. Count the member only after the Ledger cluster view reports it as a voter and its applied position has caught up. ## Before any membership change Run the same precheck for scaling, replacement, and rolling changes: ```bash ledgerctl cluster status ``` Record: - the current leader, voters, and learners; - each member's committed and applied positions; - the current alert state and recent request error rate; - WAL and data headroom on every member; - the persistent-volume retention policy; - the intended voter set after the change. Do not begin if the cluster has no leader, already lacks its expected failure tolerance, has an out-of-sync member, or is blocking writes because of disk or clock health. ## Add capacity or failure tolerance Use the Stack Operator's supported replica field and reconciliation workflow. Do not combine an Operator-managed change with manual membership commands. 1. Confirm that the target failure domain can schedule the member and provision both persistent stores. 2. Apply one desired-state change. 3. Wait for the new member to appear as a learner. 4. Keep watching until its state and log position catch up. 5. Wait for the member to become a voter. 6. Send a controlled write and current read. 7. Confirm the new voter appears in telemetry and the alert baseline returns to normal. ```bash ledgerctl cluster watch --interval 2s ``` The operation succeeds when the expected odd voter count is present, every voter has caught up, and the cluster can still commit after one voter becomes unavailable within the designed tolerance. ### Roll back a failed addition If the learner cannot catch up, stop the reconciliation and investigate network, storage, identity, TLS, and resource pressure. Remove the failed learner through the supported Operator workflow only after confirming that it never became a required voter. Keep the original voter set and its volumes unchanged. ## Replace a member Prefer add-then-remove when capacity and the Operator support it: 1. Run the membership precheck. 2. Add the replacement with a new node identity and new or explicitly restored storage. 3. Wait for the replacement to become a caught-up voter. 4. Transfer leadership away from the member being removed when necessary. 5. Remove the old member through the Operator workflow. 6. Confirm that the old identity no longer appears in the Ledger membership view. 7. Apply the approved retention or deletion policy to its volume. 8. Verify a controlled write, a current read, and the expected failure tolerance. When leadership must be transferred as part of an approved workflow, the Ledger command is: ```bash set -euo pipefail ledgerctl cluster status --json > cluster-before.json PREVIOUS_LEADER="$(jq -er '.leader | select(. != 0)' cluster-before.json)" TARGET_NODE_ID="$( jq -er ' . as $cluster | [ .nodes[] | select( .suffrage == "Voter" and .id != $cluster.leader and .progress.isLearner == false and .progress.recentActive == true and .progress.state == "Replicate" and .progress.msgAppFlowPaused == false and .progress.isPaused == false and .progress.match == $cluster.raftStatus.commit and .syncProgress.status == "normal" ) ] | sort_by(.id) | .[0].id ' cluster-before.json )" ledgerctl cluster transfer-leader "$TARGET_NODE_ID" ledgerctl cluster status --json > cluster-after.json jq -e \ --argjson previous "$PREVIOUS_LEADER" \ --argjson target "$TARGET_NODE_ID" \ '.leader == $target and .leader != $previous' \ cluster-after.json ``` The selection accepts only an active, unpaused follower that is a voter, reports normal sync, and has replicated through the leader's current commit index. It fails instead of transferring when no member meets those conditions. Review `cluster-before.json` before running the transfer when a change requires a specific failure domain or excludes another member. The final `jq` command verifies that the cluster view reports the selected member as the new leader. Never attach an old member's volume to a different node identity, reuse a removed identity for a replacement, or bootstrap a new cluster over retained member data. These shortcuts can create divergent membership and state. ### Roll back a replacement Before removing the old voter, roll back by removing the new learner or voter through the same Operator workflow and returning to the original desired count. After removing the old voter, do not reintroduce it from stale state. Recover the replacement or add another fresh member instead. ## Remove a member or scale down Scaling down changes quorum and failure tolerance. Prefer an odd target count and remove one member at a time. 1. Confirm that the remaining voters form a majority and meet the target service objective. 2. Transfer leadership if the target is the leader. 3. Apply the supported Operator change. 4. Wait for the target to disappear from cluster membership, not only from Kubernetes. 5. Stop and retain or delete its storage according to the approved policy. 6. Verify the final voter set and send a controlled write and read. A three-to-two change leaves both remaining voters required for writes. Treat it as a temporary transition, not a production steady state. `ledgerctl cluster remove-node --force` bypasses Raft consensus. Use it only as an approved break-glass action for a permanently unreachable member, on the current leader, with a recovery plan that guarantees the removed node and its stale state can never rejoin. Do not use it to speed up a normal scale-down. ## Run a rolling change For an image or supported node-configuration change, follow [Upgrade Ledger](/modules/ledger/operations/upgrade). That page owns the authoritative sequence, including which upgrade class the release permits. The lifecycle invariant it enforces is always the same: 1. change one follower; 2. wait for it to rejoin and catch up; 3. verify the cluster and a controlled request; 4. change the next follower; 5. transfer leadership to an updated voter; 6. change the former leader; 7. run the final integrity and service checks. Stop when a member cannot catch up, when a second member would be unavailable, or when the current voter set no longer meets the service objective. Changes to internal TLS need a compatibility transition across mixed members. Follow [Secure Cluster Traffic with TLS](/modules/ledger/operations/tls-and-network-security) instead of treating certificate changes as an ordinary restart. ## Completion record Attach the following evidence to the change: - cluster status before and after; - the identity and storage disposition of every added or removed member; - learner-to-voter and catch-up confirmation; - any leadership transfer; - controlled write and current-read results; - alert and integrity results; - the rollback decision and any manual intervention. A Kubernetes rollout completing is not evidence that Ledger membership converged. The Ledger cluster view is the source for voter state. --- ## Filter and Paginate Results Source: https://docs.formance.com/modules/ledger/querying/filters-and-pagination Apply filters on the server when an application needs a subset of accounts, transactions, logs, or audit entries. Each filter is validated against its target, so a valid transaction field can still be rejected by an account query. The HTTP examples use the `LEDGER_URL`, `LEDGER_NAME`, and `CURL_AUTH` variables defined in [Read Ledger Data](/modules/ledger/querying/read-ledger-data). They also use `jq` for executable pagination checks. The token needs the read scope for the selected target; see [Authentication and Authorization](/modules/ledger/operations/authentication). ## Choose a filter representation Account, transaction, and log list routes accept textual or structured JSON filters. URL-encode the complete `filter` value in both cases. The two formats overlap, but they are not interchangeable for every condition. Use the textual form for commands and filters that people review: ```bash curl --fail-with-body --silent --show-error --get \ "${CURL_AUTH[@]}" \ --data-urlencode 'filter=metadata[status] == "active" and address ^= "users:"' \ "$LEDGER_URL/v3/$LEDGER_NAME/accounts" ``` Use the structured form when an application already builds a JSON expression tree: ```bash curl --fail-with-body --silent --show-error --get \ "${CURL_AUTH[@]}" \ --data-urlencode 'filter={"$and":[{"$match":{"metadata[status]":"active"}},{"$match":{"address":"users:"}}]}' \ "$LEDGER_URL/v3/$LEDGER_NAME/accounts" ``` These two specific examples select active accounts whose addresses start with `users:`. In structured JSON, a trailing colon in an address value marks a prefix match. Do not translate filters mechanically between the two formats. A structured transaction `reference` condition is an exact match, not a prefix or substring one. The main representation differences are: | Capability | Textual form | Structured JSON | | --- | --- | --- | | Audit fields | Supported | Not supported | | Transaction `reference`, `reverted`, `id`, `insertedAt`, and `revertedAt` | Not supported | Supported | | Log `logId` | Not supported | Supported | | `in` | Supported | Not supported; `$in` is rejected | | Address prefix | Explicit `^=` operator | An address value ending in `:` | | Numeric metadata equality | `== 10` | Combine `$gte: 10` and `$lte: 10` | ## Build expressions | Intent | Textual form | Structured form | | --- | --- | --- | | Equal | `metadata[status] == "active"` | `{"$match":{"metadata[status]":"active"}}` | | Prefix | `address ^= "users:"` | `{"$match":{"address":"users:"}}` | | Greater than or equal | `metadata[score] >= 10` | `{"$gte":{"metadata[score]":10}}` | | Closed range | `metadata[score] between 10 and 20` | `{"$and":[{"$gte":{"metadata[score]":10}},{"$lte":{"metadata[score]":20}}]}` | | Metadata exists | `metadata[customerId] exists` | `{"$exists":{"metadata":"customerId"}}` | | Account has held an asset | `has asset USD/2` | `{"$exists":{"asset":"USD/2"}}` | | All conditions | `a and b` | `{"$and":[a,b]}` | | Any condition | `a or b` | `{"$or":[a,b]}` | | Negation | `not a` | `{"$not":a}` | An `$and` or `$or` array must contain at least one expression. Each structured node must contain exactly one `$` operator. ## Match the filter to its target | Target | Common fields | Index requirement | | --- | --- | --- | | Accounts | `address` | No opt-in index for address matching | | Accounts | `metadata[key]` | Declared metadata type and ready metadata index | | Accounts | `has asset BASE/PRECISION` | Ready account-asset index | | Transactions | `id`, `reverted` | No opt-in index | | Transactions | `reference` | Structured `$match` exact match and a ready reference index | | Transactions | `timestamp`, `insertedAt`, `revertedAt` | Matching date index | | Transactions | `address`, `source`, `destination` | Matching transaction-address index | | Transactions | `metadata[key]` | Declared metadata type and ready metadata index | | Logs | `logId` | No opt-in index | | Logs | `date` | Ready log-date index | | Audit | `seq`, `proposal_id`, `log_seq`, `timestamp`, `outcome`, `ledger`, `caller_subject`, `order_type` | Audit indexes maintained by Ledger | Built-in transaction, log, and audit date fields accept RFC 3339 timestamps or Unix microseconds. Dates before `1970-01-01T00:00:00Z` are rejected because those built-in indexes use an unsigned representation. A metadata key declared as `datetime` uses signed Unix microseconds in filter bounds, including dates before the epoch. Numeric built-in fields such as transaction `id` and the date fields are range-only in structured JSON. Use `$gt`, `$gte`, `$lt`, or `$lte` rather than `$match`. Send unsigned 64-bit values as decimal strings when they can exceed JavaScript's safe integer range. Audit conditions support `and` and `or` but not `not`, and use bare field names such as `outcome` and `ledger` rather than an `audit[...]` prefix. The account asset condition means that the account has held the asset cell at least once. It does not assert that the current balance is non-zero. Create only the indexes required by known queries, then wait for them to become available on every serving replica. See [Create and Manage Indexes](/modules/ledger/querying/indexes). ## Paginate HTTP reads The HTTP list routes use `pageSize` and an exclusive `after` position: | Route | `after` value | Default order | Reverse order | | --- | --- | --- | --- | | Accounts | Last account address | Alphabetical | `reverse=true` returns reverse alphabetical order | | Transactions | Last transaction ID | Newest first | `reverse=true` returns oldest first | | Ledger logs | Last ledger-local log ID | Oldest first | Not exposed over HTTP | | Audit entries | Last global audit sequence | Oldest first | `reverse=true` returns newest first | When `pageSize` is omitted or set to `0`, the server uses 100 rows. The maximum is 1,000 rows per request. For example, resume a transaction list after a transaction ID returned by the preceding page: ```bash : "${AFTER:?set this to the last transaction ID from the preceding page}" curl --fail-with-body --silent --show-error --get \ "${CURL_AUTH[@]}" \ --data-urlencode 'pageSize=100' \ --data-urlencode "after=$AFTER" \ "$LEDGER_URL/v3/$LEDGER_NAME/transactions" ``` The composed setup does not set `-u`, so without that guard an unset `AFTER` sends `after=` and silently returns the first page instead of resuming. The following loop performs a complete oldest-first traversal of transactions involving an account below `users:`. It keeps the filter, order, and page size unchanged and derives `after` from the last row of each response. Create the transaction-address index before using this filter. ```bash FILTER='address ^= "users:"' REVERSE=true PAGE_SIZE=100 AFTER="" while true; do curl_args=( curl --fail-with-body --silent --show-error --get "${CURL_AUTH[@]}" --data-urlencode "filter=$FILTER" --data-urlencode "reverse=$REVERSE" --data-urlencode "pageSize=$PAGE_SIZE" ) if [ -n "$AFTER" ]; then curl_args+=(--data-urlencode "after=$AFTER") fi curl_args+=("$LEDGER_URL/v3/$LEDGER_NAME/transactions") PAGE="$("${curl_args[@]}")" jq -c '.data[]' <<<"$PAGE" ROW_COUNT="$(jq '.data | length' <<<"$PAGE")" if (( ROW_COUNT < PAGE_SIZE )); then break fi AFTER="$(jq -er '.data[-1].id | tostring' <<<"$PAGE")" done ``` HTTP does not return `hasMore` or a next cursor. A page shorter than `PAGE_SIZE` ends the traversal; when a page contains exactly `PAGE_SIZE` rows, request the next page even if it might be empty. HTTP `after` values are resource positions, not interchangeable cursors. An account address cannot resume a transaction list, and a ledger-local log ID cannot resume the bucket-wide audit trail. ## Paginate gRPC reads The gRPC list surface returns an opaque next cursor in the `x-next-cursor` trailer. `ledgerctl` prints a resume hint when another page exists: ```bash ledgerctl accounts list \ --ledger "$LEDGER_NAME" \ --filter 'address ^= "users:"' \ --reverse \ --page-size 100 \ --json ledgerctl accounts list \ --ledger "$LEDGER_NAME" \ --filter 'address ^= "users:"' \ --reverse \ --page-size 100 \ --cursor '' \ --json ``` Treat a gRPC cursor as opaque. Store and return it unchanged; do not parse it or build a new one from the last row. Repeat the same filter, `--reverse` choice, and page size on every call. With structured output, `ledgerctl` writes the next-cursor hint to standard error so standard output remains valid JSON or YAML. `ledgerctl accounts list --all` and `ledgerctl transactions list --all` follow every cursor. The log and audit commands do not provide `--all`; consume their emitted cursor explicitly. Omitting `--page-size` selects the default page size and does not request every page. ## Keep a multi-page reconciliation stable A live traversal can overlap new writes. Each page reads a new snapshot: a new row inserted before the current cursor can be omitted, while other changes can produce duplicates. Deduplication handles duplicates but cannot recover an omitted row. Use live pagination only for best-effort interfaces, or give an incremental worker its own tested watermark-and-overlap protocol. It is not sufficient when the traversal must be complete or when several pages and queries must describe one fixed ledger state. For a reconciliation: 1. create a query checkpoint; 2. retry a checkpoint-scoped probe until it succeeds on the node serving the reads; 3. pass the same checkpoint ID to every gRPC read; 4. finish every page before deleting the checkpoint. The direct HTTP list routes do not expose checkpoint reads. Follow [Choose Read Consistency and Use Checkpoints](/modules/ledger/querying/consistency-and-checkpoints) and use gRPC or `ledgerctl` for this workflow. ## Diagnose rejected or empty queries | Result | Check | | --- | --- | | `400` invalid filter | Confirm the syntax, target, field name, and value type | | Index not found | Declare the metadata type or create the required built-in index | | HTTP `503` with `INDEX_BUILDING` | Wait for the local index version to become ready; retry with bounded backoff rather than a tight loop | | Empty page after a recent write | Check the read freshness, ledger name, cursor direction, and filter | | Unexpected account prefix matches | Remember that prefixes compare raw bytes: `accounts:1` also matches `accounts:10` | Continue with [Choose Read Consistency and Use Checkpoints](/modules/ledger/querying/consistency-and-checkpoints) when a paginated read must span one fixed view of the ledger. --- ## Write Transactions with Numscript Source: https://docs.formance.com/modules/ledger/transactions/numscript Numscript describes the result you want while Ledger resolves the postings against current balances. Use it for ordered fallback sources, percentage splits, bounded overdrafts, or metadata changes that belong to the same transaction. Use [explicit postings](/modules/ledger/transactions/create-with-postings) when your application already knows every movement. That keeps the request smaller and the application remains the source of the routing decision. ## Prerequisites You need an existing normal ledger and a token with `ledger:write` for the transactions plus `ledger:read` for the account checks. If the ledger enforces account types, use addresses that match its configured patterns. ```bash set -eo pipefail ``` The example pays USD 80.00 from two balances. Fund the wallet with USD 60.00 and the bank account with USD 50.00 first: ```bash jq --null-input \ --arg reference "$FUNDING_REFERENCE" \ --arg wallet "$WALLET_ACCOUNT" \ --arg bank "$BANK_ACCOUNT" \ '{ reference: $reference, postings: [ {source: "world", destination: $wallet, amount: 6000, asset: "USD/2"}, {source: "world", destination: $bank, amount: 5000, asset: "USD/2"} ] }' \ | curl --fail-with-body --silent --show-error \ --request POST "$LEDGER_URL/v3/$LEDGER_NAME/transactions" \ --header "Authorization: Bearer $LEDGER_TOKEN" \ --header "Content-Type: application/json" \ --header "Idempotency-Key: $FUNDING_REFERENCE" \ --data @- \ > /dev/null ``` ## Declare the program and its variables The program lists sources in priority order. Ledger takes the available amount from the wallet before using the bank account. ```numscript vars { account $wallet account $bank account $destination monetary $amount string $order_id } set_tx_meta("order_id", $order_id) set_tx_meta("kind", "purchase") send $amount ( source = { $wallet $bank } destination = $destination ) ``` Put the same source in a shell variable for the request below: ```bash SCRIPT=$(cat <<'NUMSCRIPT' vars { account $wallet account $bank account $destination monetary $amount string $order_id } set_tx_meta("order_id", $order_id) set_tx_meta("kind", "purchase") send $amount ( source = { $wallet $bank } destination = $destination ) NUMSCRIPT ) ``` Pass values separately from the source. This avoids constructing executable code from application input. | Numscript type | Variable value example | | --- | --- | | `account` | `users:alice:wallet` | | `monetary` | `USD/2 8000` | | `asset` | `USD/2` | | `string` | `ord-2026-0043` | | `number` | `42` | | `portion` | `2.5%` or `1/4` | ## Execute the inline program Build the JSON body with `jq` so newlines and quotes in the program are encoded correctly: ```bash jq --null-input \ --arg plain "$SCRIPT" \ --arg wallet "$WALLET_ACCOUNT" \ --arg bank "$BANK_ACCOUNT" \ --arg destination "$DESTINATION_ACCOUNT" \ --arg reference "$PAYMENT_REFERENCE" \ --arg orderId "$RUN_ID" \ '{ reference: $reference, script: { plain: $plain, vars: { wallet: $wallet, bank: $bank, destination: $destination, amount: "USD/2 8000", order_id: $orderId } } }' \ | curl --fail-with-body --silent --show-error \ --request POST "$LEDGER_URL/v3/$LEDGER_NAME/transactions" \ --header "Authorization: Bearer $LEDGER_TOKEN" \ --header "Content-Type: application/json" \ --header "Idempotency-Key: $PAYMENT_REFERENCE:create" \ --data @- \ > /tmp/numscript-transaction.json ``` Ledger returns `201 Created`. Inspect the resolved postings rather than duplicating the resolution logic in your application: ```bash jq '.data.transaction | { id, reference, metadata, postings }' /tmp/numscript-transaction.json ``` The transaction contains two postings to `$DESTINATION_ACCOUNT`: USD 60.00 from `$WALLET_ACCOUNT` and USD 20.00 from `$BANK_ACCOUNT`. It also contains the run ID and `kind: "purchase"` in its metadata. ## Verify the resolved state Check the destination and both sources. Each command fails if the expected balance is absent: ```bash get_usd_balance() { local account=$1 curl --fail-with-body --silent --show-error \ "$LEDGER_URL/v3/$LEDGER_NAME/accounts/$account" \ --header "Authorization: Bearer $LEDGER_TOKEN" \ | jq -er ' .data.volumes[] | select(.asset == "USD/2" and .color == "") | .volumes.balance' } test "$(get_usd_balance "$WALLET_ACCOUNT")" = "0" && test "$(get_usd_balance "$BANK_ACCOUNT")" = "3000" && test "$(get_usd_balance "$DESTINATION_ACCOUNT")" = "8000" ``` After this example: - the wallet balance is `"0"`; - the bank balance is `"3000"`; - the merchant balance is `"8000"`. ## Choose source behavior deliberately | Construct | Behavior | Use it when | | --- | --- | --- | | `@world` | Creates or destroys an amount at the ledger boundary | The business event represents issuance, deposit, withdrawal, or another explicit boundary movement. | | `{ $wallet $bank }` | Draws from sources in order until the amount is satisfied | A payment has a documented funding priority. | | `allowing overdraft up to $limit` | Permits a negative balance up to a monetary bound | The account has a controlled credit limit in the same asset. | | `allowing unbounded overdraft` | Removes the balance floor for that source | The account is explicitly modeled as an unlimited liability or technical clearing account. | | `set_tx_meta` / `set_account_meta` | Writes metadata in the same committed transaction | The metadata describes the accounting event or an involved account at that point in time. | An unbounded overdraft can hide a funding error by allowing any negative balance. Prefer a bounded overdraft when the domain has a credit limit, and alert on negative balances independently. ## Handle errors | Failure | What to inspect | Action | | --- | --- | --- | | Numscript parse error | The parser details in `errorCode` and `errorMessage` | Fix the program. Do not retry the same invalid source. | | Invalid variable | The declared type and the string in `script.vars` | Encode accounts without `@`, monetary values as `ASSET amount`, and portions in a supported format. | | Insufficient funds | The asset, required amount, and available balance in the structured error | Fund another source, lower the amount, or change the documented funding policy. The current Numscript error does not identify which resolved account or color ran short. | | Invalid account address or account-type rejection | The account variables and the ledger's account-type rules | Correct the variable or use an address accepted by the chart of accounts. A new valid destination account is created by the transaction and does not need a separate setup step. | | `409` idempotency conflict | The key was already used with different request content | Restore the original body for a retry, or use a new key for a genuinely new operation. | | `503` | Inspect `errorCode`; the status alone does not identify the cause | Retry `STALE_INPUTS_RESOLUTION` with the same body and key. Retry `PRELOAD_UNAVAILABLE` only a bounded number of times, then validate the program and variables. Treat `NO_LEADER` or a failing health check as a cluster-availability incident. | ## Retry or change the program Some interpreter and variable errors are reported as `PRELOAD_UNAVAILABLE`. If the same response persists after bounded backoff, stop the workflow and correct the input or investigate the cluster. Do not retry a `503` indefinitely. Changing the source, a variable, or the program changes the logical operation. Use a new idempotency key and, when appropriate, a new business reference. A committed Numscript transaction is immutable; correct it with a revert or a new compensating transaction. Continue with [Reuse and Version Numscript Programs](/modules/ledger/transactions/numscript-library) when several callers must share the same program. --- ## Assets Source: https://docs.formance.com/modules/ledger/core-concepts/assets An asset is anything that can be held in an account and moved between accounts. In Formance, an asset is identified by a string: a name, optionally followed by a decimal precision. ## What it's for Assets let the ledger track fundamentally different things in the same system (currencies, loyalty points, in-app tokens, and anything else) without mixing them. Each account holds balances per asset independently. ## Asset notation An asset is written as `NAME` or `NAME/PRECISION`, where `NAME` is an uppercase identifier and `PRECISION`, when present, is the number of decimal places. Whole-unit assets such as `JPY` and `COIN` carry no suffix. All amounts in the ledger are stored as integers, and the precision suffix is a naming convention for *your* benefit: it tells your application where to put the decimal point. The ledger does not read it. | Asset | Meaning | Example amount | Represents | |-------|---------|----------------|------------| | `USD/2` | US dollars, 2 decimal places | `10000` | $100.00 | | `EUR/2` | Euros, 2 decimal places | `4999` | €49.99 | | `BTC/8` | Bitcoin, 8 decimal places | `100000000` | 1 BTC | | `JPY` | Japanese yen, whole units | `5000` | ¥5,000 | There is no currency registry, and the ledger does not interpret the precision. It checks that the asset matches the pattern `[A-Z][A-Z0-9]{0,16}(_[A-Z]{1,16})?(/\d{1,6})?` and then treats the whole string as an opaque key for a balance. It never divides by 10^precision, and it does not relate `USD/2` to `USD/3` — to the ledger those are two unrelated assets. Scaling and formatting are entirely your application's responsibility. The one place on the platform that splits an asset into `/` is Numscript's asset-scaling feature, which moves value between differently-scaled balances of the same asset. It is opt-in: it requires the experimental interpreter and the `experimental-asset-scaling` feature flag. Everywhere else — and for any name without a `/`, such as `JPY` or `COIN` — the asset is just a string. For a detailed specification of how assets and amounts are represented in Numscript, see [Unambiguous Monetary Notation](/modules/numscript/monetary-notation). ## Multi-asset accounts The Formance Ledger is a multi-asset ledger. Multiple assets can be held within a single account natively. An account's balance for each asset is tracked independently. Sending `USD/2` from one account does not affect its `EUR/2` balance. ```json { "users:alice:wallet": { "USD/2": 10000, "EUR/2": 4500, "COIN": 200 } } ``` Moving between different assets requires an explicit conversion transaction. See [Currency Conversion](/modules/ledger/working-with/currency-conversion) for strategies on funded and non-funded exchanges. ## Do / Don't - **Do:** Always specify precision consistently; if you use `USD/2` in one transaction, use it everywhere for that currency. - **Do:** Use whole-unit asset names like `COIN` or `JPY` for integer-only assets like loyalty points, votes, or zero-decimal currencies. - **Don't:** Mix precisions for the same real-world currency (e.g. using both `USD/2` and `USD/3`); they are treated as different assets. - **Do:** Divide the raw integer amount by 10^precision before displaying it to users (e.g. `10000` tagged `USD/2` → divide by 100 → display as `$100.00`). Omit the precision suffix from the rendered asset label (`USD`, not `USD/2`). - **Don't:** Display the raw integer directly or simply strip the `/2`; that would render `$10000` instead of `$100.00`. --- ## Core Concepts Source: https://docs.formance.com/modules/ledger/core-concepts This section covers the core concepts of the Formance Ledger: what a ledger is, how accounts and assets are structured, how transactions move value between accounts, and how the double-entry accounting model enforces consistency. --- ## Update Ledger Clients Source: https://docs.formance.com/modules/ledger/migration/update-clients Shared accounting concepts do not make Ledger v2 clients wire-compatible with the target. The target serves business endpoints under `/v3` and does not expose a Ledger v2 compatibility API. Update and test clients before the production cutover. During mirroring, point non-production readers at the target; keep production writes on Ledger v2. ## Build a workflow inventory Review behavior, not only endpoint names. For every workflow, record: | Contract area | What to verify | | --- | --- | | Endpoint | Method, `/v3` path, ledger name, and protocol. | | Authentication | Issuer, audience, bearer token, and required read/write scope. | | Request | Field names, amount encoding, metadata types, and mutually exclusive inputs. | | Response | Wrapper, field names, post-commit volume snapshots, and absent fields. | | Pagination | Cursor direction, page size, continuation value, and termination condition. | | Filtering | Filter syntax, type conversion, and required indexes. | | Errors | HTTP or gRPC status, domain reason, retryability, and conflict handling. | | Idempotency | Key construction, scope, payload reuse, and retention assumptions. | | SDK | Package major, generated method, numeric type, and timestamp type. | | Observability | Correlation, audit lookup, metrics, and alert ownership. | Cover create, revert, account reads, transaction reads, metadata, filters, pagination, bulk, Numscript, and events when your application uses them. ## Update the base endpoint and authentication HTTP calls move from Ledger v2 paths to `/v3/{ledgerName}/...`. Send the target access token in the standard header: ```http Authorization: Bearer ``` Do not reuse a Ledger v2 token without verifying its issuer, audience, and scopes against the target. Test from every application environment, including workers and scheduled jobs. Use [Authentication](/modules/ledger/operations/authentication) to map target identities and effective scopes before changing an application endpoint. HTTP and gRPC do not expose every operational capability symmetrically. Select the protocol per workflow and keep cluster operations out of normal application clients. ## Update transaction writes A transaction request declares exactly one content source: - explicit `postings`; - an inline Numscript `script`; - a `scriptReference` to a saved program. Use the `Idempotency-Key` HTTP header for request idempotency. Do not add a body-level idempotency key. Ledger v2 transaction response fields for pre-commit and effective volumes are not part of the target contract. The target computes and returns `postCommitVolumes` unconditionally; there is no volume-expansion parameter to opt into. Read it at `data.transaction.postCommitVolumes` on create and at `data.revertTransaction.postCommitVolumes` on revert. `postCommitVolumes` is an immutable snapshot taken at that transaction's sequence. It is never recomputed from later state, so it is not a current balance. If an application needs durable current balances, read the affected account or run the required aggregate query instead of reusing a write response. ## Redesign idempotency keys Idempotency covers more target operations and uses a cluster-level namespace. A key that was treated as independent per Ledger v2 ledger can conflict when reused on the target. The observable failure is specific: when a key is reused anywhere in the cluster for different request content, the target rejects the request with HTTP `409` and `errorCode: IDEMPOTENCY_KEY_CONFLICT`. Assert on that pair in test cases 2 and 5 below. A key is at most 256 bytes. Keys expire after the cluster's `--idempotency-ttl`, which defaults to 24 hours and never expires when set to `0`; a retry sent after that window re-executes the request instead of replaying the first outcome. Construct keys from stable business identity, for example: ```text create-transaction:tenant-42:main:payment-018f1 ``` Test all of these cases: 1. same key and same payload; 2. same key and a different payload; 3. a retry after a client timeout; 4. the same business reference with a new idempotency key; 5. the same key accidentally reused for another ledger or operation. See [Idempotency and References](/modules/ledger/transactions/idempotency-and-references) for the target request and conflict behavior. ## Replace Ledger Schema workflows Ledger v2 combined several concerns behind one schema API. The target models them independently: | Ledger v2 responsibility | Target resource | | --- | --- | | Validate account addresses | Account types and enforcement mode | | Define metadata fields | Typed metadata schema | | Reuse transaction templates | Versioned Numscript library | | Reuse query templates | Prepared queries | There is no single schema version that selects all four resources for a transaction. Update deployment and application ownership for each resource you use. ## Update reads and filters The target introduces typed filters, explicit indexes, prepared queries, and query checkpoints. Verify: - the fields and types accepted by each query target; - which filter requires an index; - that every required index is ready before switching readers; - that the client follows the target pagination contract rather than deserializing a Ledger v2 cursor shape; - whether the workflow needs an HTTP live read or a gRPC consistency option. Do not compare complete source and target JSON payloads byte for byte. Normalize each workflow to the business fields it depends on. Use [Filters and Pagination](/modules/ledger/querying/filters-and-pagination) and [Indexes](/modules/ledger/querying/indexes) to build and verify each production read path. ## Update bulk handling The HTTP bulk route is `/v3/{ledgerName}/bulk`; Ledger v2 `_bulk` paths do not carry over. The target returns per-element results and distinguishes business failures from infrastructure failures. Choose the failure model explicitly: - the default sequential mode commits successful elements in order and stops after the first failure; - `continueOnFailure=true` continues with later independent elements after a business failure; - `atomic=true` submits the complete array as one Apply batch, so every element commits or none commits. In atomic mode, send one batch-level `Idempotency-Key` header and preserve it with the exact array on an ambiguous retry. Per-element `ik` values identify requests only in sequential mode. Do not combine `atomic=true` with `continueOnFailure=true`: an atomic batch has no successful prefix to continue after. The `release/v3.0` HTTP handler supports `atomic=true` even though the current OpenAPI parameter description still says that it is not supported. Pin the application to the released server contract, inspect every per-element result, and test sequential failure, continuation, atomic rollback, and ambiguous whole-batch retries before enabling production bulk traffic. See [Bulk Transactions](/modules/ledger/transactions/bulk) for the target request and result model. ## Replace event integrations Ledger v2 exporters and pipelines are not target client APIs. However, mirroring does not hide imported history from the target event system. Each supported Ledger v2 change becomes an ordinary target log and produces the same standard event type as a target-native change, such as `COMMITTED_TRANSACTION`, `REVERTED_TRANSACTION`, `SAVED_METADATA`, or `DELETED_METADATA`. A newly created target sink starts at cursor `0` and replays every supported event still present in the target global log, including the history imported from Ledger v2. Pointed at a live consumer, it re-fires the payouts, webhooks, and notifications that Ledger v2 already delivered. The sink configuration exposes no start cursor that skips this replay. Do not point a new target sink at a consumer that can immediately repeat production side effects. Stage the destination or consumer in quarantine first: accept and inspect replayed messages, but suppress notifications, payouts, webhooks, projections, and any other externally visible effect until the approved cutover boundary. Inventory each consumer and define: - the target event sink and event types; - a target-delivery identity, normally the target cluster identity plus `logSequence`, for at-least-once redelivery; - a business-effect identity that is stable across the Ledger v2 pipeline and the target replay; - the durable sink and consumer checkpoints used to prove that historical replay has drained; - at-least-once delivery handling; - the exact boundary after which target events may produce business effects; - the source consumer stop point and the target consumer activation point. The target `logSequence` identifies delivery inside the target cluster, but it does not identify an effect that the Ledger v2 consumer already applied. Build the cross-version key from business data that survives mirroring—for example, a unique transaction reference when your domain guarantees one—and specify a key for every event type that causes an effect. If no stable cross-version identity exists for a workflow, keep its target effects disabled until you have reconciled and checkpointed the replay explicitly. During rehearsal, let the target sink replay into quarantine, verify the standard target event shapes, and test the same mirrored business change against the Ledger v2 receipt history. Record the point through which every replayed event is durably accepted without repeating an effect. This is a consumer-side migration checkpoint, not a sink start-cursor setting. Data parity does not prove event-consumer compatibility. A successful rehearsal must prove both full replay handling and safe activation for the first post-boundary target event. Use [Configure an Event Sink](/modules/ledger/events/configure-sink) and [Consume and Recover](/modules/ledger/events/consume-and-recover) to define the target delivery and recovery path. ## Integration test gate Run a non-production client against a disposable normal ledger for writes and against the mirror for reads. - [ ] Create a transaction with the target request shape. - [ ] Retry it with the same idempotency key. - [ ] Read it by ID and business reference. - [ ] Read affected accounts and volumes. - [ ] Revert a transaction and verify the compensating state. - [ ] Save and delete account and transaction metadata. - [ ] Traverse more than one page for each list workflow. - [ ] Exercise every production filter after indexes are ready. - [ ] Test expected 4xx conflicts and retryable infrastructure failures. - [ ] Authenticate every application identity. - [ ] Replay target events into quarantine without producing business effects. - [ ] Deduplicate an imported event against the corresponding Ledger v2 business effect. - [ ] Record the replay checkpoint and prove that only post-boundary events can activate effects. Do not generate a production client from a branch snapshot. Pin the SDK or generated client to the same released contract as the target deployed by Stack v4. Continue with [Create a Mirror](/modules/ledger/migration/create-mirror). --- ## Metadata and Typed Values Source: https://docs.formance.com/modules/ledger/modeling/metadata Metadata attaches application context to a ledger, account, or transaction without changing its accounting identity. Use it for identifiers, classifications, workflow state, and reconciliation references. Keep monetary value in postings and volumes. ## Choose the right target | Target | Use it for | Example keys | | --- | --- | --- | | Ledger | Configuration or ownership shared by the whole book | `environment`, `region`, `team` | | Account | Context that follows one balance identity | `customerId`, `country`, `riskTier` | | Transaction | Context for one business movement | `paymentId`, `orderId`, `channel` | Put stable balance identity in the [account address](/modules/ledger/modeling/chart-of-accounts). Put context that can change without moving a balance in account metadata. To write metadata, declare types, and inspect the schema against a live ledger, follow [Metadata and typed values](/modules/ledger/hands-on#metadata-and-typed-values) in the hands-on walkthrough. ## Write metadata The HTTP API infers metadata types from JSON: | JSON value | Stored value | | --- | --- | | `"text"` | String | | `42` | Unsigned integer | | `-42` | Signed integer | | `true` or `false` | Boolean | | Decimal number | Rejected | | Object or array | Rejected | The Ledger 3.0 HTTP parser decodes a JSON number through `float64` before it creates a signed or unsigned metadata integer. A bare integer above `2^53 - 1` can therefore lose precision before Ledger stores it. Reject such values in the HTTP client or send their exact decimal form as a string. This form preserves an exact large identifier over HTTP: ```json { "externalSequence": "9007199254740993" } ``` If queries must compare that field as a number, declare it as `uint64` and create its metadata index. Direct reads still return the stored string, while the index compares its decimal value as `uint64`. Do not use a large bare JSON number as a shortcut. The same operation exists for ledger and transaction metadata: ```text POST /v3/{ledgerName}/metadata POST /v3/{ledgerName}/transactions/{transactionId}/metadata ``` A save merges the supplied keys with existing metadata. Supplying a key again replaces its value. It does not remove keys omitted from the request. Do not rely on `null` in a save request to delete an existing key. The HTTP parser skips null entries. Use the target's `DELETE .../metadata/{key}` endpoint so the deletion is explicit and auditable. ## Declare types for indexed queries A per-ledger metadata schema declares how a key should be interpreted when Ledger builds and queries a metadata index. Declarations are scoped independently to account, transaction, and ledger metadata. Supported declarations are: - `string`; - `bool`; - `int8`, `int16`, `int32`, and `int64`; - `uint8`, `uint16`, `uint32`, and `uint64`; - `datetime`. Write datetime values as RFC 3339 strings, for example `2026-07-17T09:30:00Z`. Ledger's index representation uses signed microseconds since the Unix epoch, so it can order values before and after 1970. Fractions beyond microsecond precision are truncated for indexing. ## Understand what a declaration changes In Ledger 3.0, a metadata type declaration changes how an index interprets one key: - reads return the value in the JSON type originally written by the client; - the declaration tells the metadata index how to convert and order that key; - declaring a new type does not rewrite stored account or transaction metadata; - changing a declaration schedules a new encoding for an existing index; - removing a declaration also removes an index attached to that field. This means a raw account read can still return `"2"` as a string even when the account schema declares `riskTier` as `uint8`. A typed query interprets the indexed representation as `uint8`. Setting a type does not start a historical metadata conversion, and there is no conversion state to poll. `GET /metadata-schema` confirms the declaration only. Validate historical values yourself and monitor any related metadata index before a query depends on the new type. Declare types before accepting production values and before creating their indexes. This avoids mixed representations and reduces the work required to validate historical data. ## Inspect the schema Retrieving the stored declarations groups fields by target: ```json { "data": { "accountFields": { "riskTier": { "declaredType": "uint8" }, "externalSequence": { "declaredType": "uint64" } }, "transactionFields": { "settledAt": { "declaredType": "datetime" } }, "ledgerFields": {} } } ``` The schema response confirms the declaration. Check the related metadata index separately before you depend on it for a production query. ## Build the index and verify every read replica This gate targets a deployed ledger with more than one replica, so set its name explicitly: ```bash ``` A type declaration does not create an index. Register the exact target and key, then keep the canonical ID returned by Ledger: ```bash REQUESTED_INDEX_ID="metadata:TARGET_TYPE_ACCOUNT:riskTier" INDEX_RESPONSE="$(curl --fail-with-body --silent --show-error \ --request POST \ "${CURL_AUTH[@]}" \ --header "Content-Type: application/json" \ --header "Idempotency-Key: create-account-risk-tier-index" \ --data "$(jq -n --arg id "$REQUESTED_INDEX_ID" '{id: $id}')" \ "$LEDGER_URL/v3/$LEDGER_NAME/indexes")" INDEX_ID="$(jq -er '.data.id' <<<"$INDEX_RESPONSE")" test "$INDEX_ID" = "$REQUESTED_INDEX_ID" ``` Index creation and type changes start asynchronous work on each replica. Configure one direct URL for every Ledger pod that can serve reads. A load-balanced URL cannot prove that every replica is ready: ```bash ``` Poll the canonical status route on each replica. This gate succeeds only when the contacted replica serves the latest encoding and has no newer version in progress: ```bash for endpoint in $LEDGER_REPLICA_URLS; do READY=false for attempt in $(seq 1 120); do STATUS="$(curl --fail-with-body --silent --show-error \ "${CURL_AUTH[@]}" \ "${endpoint%/}/v3/$LEDGER_NAME/indexes/$INDEX_ID/status")" jq -r --arg endpoint "$endpoint" ' .data | [ $endpoint, (.currentVersion // 0), (.pendingVersion // 0), (.index.forwardEncodingVersion // 0) ] | @tsv ' <<<"$STATUS" if jq -e ' (.data.currentVersion // 0) > 0 and (.data.pendingVersion // 0) == 0 and .data.currentVersion == .data.index.forwardEncodingVersion ' <<<"$STATUS" >/dev/null; then READY=true break fi sleep 2 done test "$READY" = true done ``` Run the same gate after changing a declared type on an indexed key. The declaration becomes visible before every replica finishes rebuilding its local index, so `GET /metadata-schema` alone is not a readiness check. See [Wait for readiness](/modules/ledger/querying/indexes#wait-for-readiness) for the index lifecycle and failure modes. ## Validate historical values before a type change Type conversion for an index follows explicit rules. Common conversions include decimal strings to integers, RFC 3339 strings to datetime, and `true`/`false` or `1`/`0` strings to booleans. Values outside a bounded integer range or strings with an invalid format cannot produce the requested typed index value. Use this change sequence: 1. inventory existing values for the target and key; 2. test every known representation against the new type; 3. stop writers from producing incompatible values; 4. set the declaration; 5. monitor the associated index rebuild; 6. run representative equality and range queries; 7. compare query results with direct entity reads. Do not infer successful historical conversion from the schema declaration alone. The declaration records intent; your validation proves that the existing values satisfy it. ## Remove a declaration Removing a field declaration leaves the stored metadata values unchanged. It removes the typed interpretation and drops the metadata index for that field, so update dependent queries first. ## Common failure modes | Symptom | Cause | Correction | | --- | --- | --- | | A decimal metadata value is rejected | Metadata does not accept floating-point JSON values | Store a non-monetary scaled integer or string; keep money in postings. | | An HTTP integer above `2^53 - 1` changes by one or more units | The JSON number passed through `float64` before Ledger stored it | Send the exact decimal value as a string, then declare a numeric index type if queries need numeric comparisons. | | A range query omits some historical entities | Old values cannot be converted to the declared type | Inventory and normalize source values before changing the declaration. | | A raw read still returns a string after declaring `uint8` | Declarations type index values, not stored response values | Treat raw metadata and indexed interpretation as separate contracts. | | A query stops working after removing a declaration | Removing the declaration also drops its index | Update dependent queries before removal. | | Sending `null` leaves a key unchanged | Null entries are skipped by the save parser | Call the explicit delete endpoint. | | A timestamp sorts unexpectedly | Writers used inconsistent RFC 3339 values | Standardize UTC RFC 3339 strings and test boundary values. | When address shape itself must be validated, use [account types](/modules/ledger/modeling/account-types) rather than metadata. --- ## Authentication and Authorization Source: https://docs.formance.com/modules/ledger/operations/authentication Enable authentication before exposing a Ledger API outside a trusted development environment. Ledger accepts JWT bearer tokens from an OIDC issuer or from registered Ed25519 machine credentials, then checks the scopes required by each operation. This page covers client identity and permissions. Configure [TLS](/modules/ledger/operations/tls-and-network-security) first: a node started with `--auth-enabled` refuses to boot unless `--tls-mode` is `required`. [Request signing](/modules/ledger/operations/signing) is a separate, optional control on top of both. ## Choose a credential source | Credential source | Use it for | How Ledger validates the token | | --- | --- | --- | | OIDC | People, services, and environments that already use an identity provider | Ledger discovers the issuer's JWKS and validates the signature, issuer, and expiration | | Ed25519 JWT | Workloads that need a Ledger-specific machine identity | Ledger verifies an EdDSA JWT against a static public key and its scope allowlist | Both sources can be active at the same time. Prefer OIDC when your organization already manages workload identities and rotation. Use Ed25519 credentials for bounded automation such as a CI job or an in-cluster agent. ## Understand scopes The default virtual scopes provide three useful permission bundles. Their prefix comes from the configured auth service name (`--auth-service`, default `ledger`), so `--auth-service=myapp` makes them `myapp:read`, `myapp:write`, and `myapp:admin`. Granular scope names are unaffected: they always begin with the literal `ledger:`, whatever the service name is. | Virtual scope | Grants | | --- | --- | | `ledger:read` | Read ledgers, transactions, accounts, audit records, operational state, and queries | | `ledger:write` | Write ledgers, transactions, metadata, audit state, operational state, and queries | | `ledger:admin` | Read and change cluster state | Use granular scopes when a client needs less than a complete bundle: ```text ledger:LedgerRead ledger:LedgerWrite ledger:TransactionRead ledger:TransactionWrite ledger:AccountRead ledger:MetadataWrite ledger:AuditRead ledger:AuditWrite ledger:OpsRead ledger:OpsWrite ledger:QueryRead ledger:QueryWrite ledger:ClusterRead ledger:ClusterWrite ``` For example, an application that creates and reads transactions but cannot create ledgers can receive `ledger:TransactionRead`, `ledger:AccountRead`, and `ledger:TransactionWrite`. ## Prepare OIDC for Stack v4 Before you begin, confirm that every Ledger node can resolve and reach the issuer URL. OIDC discovery runs when a node starts; an unreachable issuer prevents an auth-enabled node from becoming ready. Operator-managed authentication is not available in this Stack v4 preview. Use the direct-server workflow below only for evaluation, and wait for your release's Stack Operator reference before configuring production authentication. Before production, verify that your Stack release lets you: - enable authentication explicitly; - set the OIDC issuer and the scope-service prefix; - reference credentials through Secrets without placing private key material in the Stack resource; - observe whether every Ledger replica loaded the intended issuer and key set; - rotate a key without removing the last credential that clients can use. The issuer must expose its OpenID configuration at: ```text https://identity.example.com/.well-known/openid-configuration ``` Configure the identity provider to put one or more Ledger scopes in the token's scope claim. A read-only service token can use: ```text ledger:read ``` To verify Ledger authentication in a disposable direct deployment, start the first node with the same identity and storage paths described in [Deploy Ledger](/modules/ledger/operations/deploy), plus the authentication flags: ```bash ledger run \ --node-id 1 \ --cluster-id auth-evaluation \ --bind-addr 127.0.0.1:7777 \ --grpc-port 8888 \ --http-port 9000 \ --wal-dir ./auth-evaluation/wal/node-1 \ --data-dir ./auth-evaluation/data/node-1 \ --bootstrap \ --tls-mode required \ --tls-cert-file ./tls.crt \ --tls-key-file ./tls.key \ --auth-enabled \ --auth-issuer https://identity.example.com \ --auth-service ledger ``` Use `--bootstrap` only for this node's first start. On restart, keep the same node ID, cluster ID, WAL directory, data directory, and authentication flags, but remove `--bootstrap`. If `--tls-mode` is `disabled` or `optional`, then a node started with `--auth-enabled` refuses to start — bearer tokens would otherwise be exposed to plaintext interception, and `optional` still accepts plaintext connections. This is a startup precondition on every topology, single-node included. Configure [TLS](/modules/ledger/operations/tls-and-network-security) first. On a multi-node cluster it also carries the cluster secret: followers forward writes to the leader with the original caller identity, so every node must share one protected secret over TLS. ### Send a token HTTP clients send the JWT in the standard header: ```http Authorization: Bearer ``` For `ledgerctl`, use a short-lived environment variable in automation: ```bash ledgerctl \ --server ledger-payments.formance.svc.cluster.local:8888 \ --tls-ca-cert ./ca.crt \ ledgers list ``` An application using gRPC sends the same value in request metadata: ```go md := metadata.Pairs("authorization", "Bearer "+token) ctx := metadata.NewOutgoingContext(ctx, md) ``` ## Create an Ed25519 machine credential Operator-managed Ed25519 credentials are not available in this Stack v4 preview. In production, require Secret-backed private seeds, distribute only public keys to Ledger, cap the scopes each key may issue, and verify that every replica loaded the intended key set. If a key entry in `auth-keys.json` sets `"god": true`, then tokens signed by that key receive every granular scope and the entry's `scopes` allowlist is ignored entirely. Capping scopes per key has no effect on such a key. Never set it on an application credential, and audit for it when reviewing a key set. ### Verify the direct server behavior For a disposable direct deployment, generate a keypair and write its exact key ID into the server configuration: ```bash set -euo pipefail for file in ./payment-service-keys/seed.hex ./payment-service-keys/pubkey.hex; do test ! -e "$file" || { printf 'refusing to overwrite existing credential: %s\n' "$file" >&2 exit 1 } done KEY_OUTPUT="$(ledgerctl auth generate-key ./payment-service-keys)" printf '%s\n' "$KEY_OUTPUT" KEY_ID="$(awk '/^[[:space:]]*Key ID:/ { print $3; exit }' <<<"$KEY_OUTPUT")" : "${KEY_ID:?ledgerctl did not return a key ID}" jq -n \ --arg keyId "$KEY_ID" \ '{ keys: [{ keyId: $keyId, publicKeyFile: "./payment-service-keys/pubkey.hex", scopes: [ "ledger:read", "ledger:TransactionWrite", "ledger:MetadataWrite" ] }] }' > auth-keys.json ``` Start Ledger with the public-key file in the first terminal. This command remains in the foreground: ```bash mkdir -p ./payments/wal/node-1 ./payments/data/node-1 ledger run \ --bootstrap \ --node-id 1 \ --cluster-id payments \ --bind-addr 127.0.0.1:7777 \ --grpc-port 8888 \ --http-port 9000 \ --wal-dir ./payments/wal/node-1 \ --data-dir ./payments/data/node-1 \ --tls-mode required \ --tls-cert-file ./tls.crt \ --tls-key-file ./tls.key \ --auth-enabled \ --auth-ed25519-keys ./auth-keys.json ``` In another terminal, read the same key ID from the configuration, mint a short-lived token, and call Ledger: ```bash set -euo pipefail KEY_ID="$(jq -er '.keys[0].keyId' auth-keys.json)" TOKEN=$(ledgerctl auth generate-token \ --signing-key ./payment-service-keys/seed.hex \ --key-id "$KEY_ID" \ --subject payment-service \ --scopes ledger:read,ledger:TransactionWrite,ledger:MetadataWrite \ --expiration 1h) LEDGERCTL_AUTH_TOKEN="$TOKEN" ledgerctl \ --server localhost:8888 \ --tls-ca-cert ./ca.crt \ --tls-server-name \ ledgers list ``` `--insecure` cannot reach this server, and it is mutually exclusive with both `--tls-ca-cert` and `--tls-server-name` in any case. The list command must succeed. The scopes in the token must be a subset of the scopes attached to that key in `auth-keys.json`; Ledger rejects a token that claims additional permissions. Protect `seed.hex` as a private key. Do not print it in CI logs or store it in source control. The public `pubkey.hex` can be distributed to Ledger replicas. ## Allow anonymous reads deliberately Strict authentication is the default: a request without a token has no scopes. To test reads without a token while keeping writes authenticated in a direct deployment, add this server flag to the authentication configuration: ```bash --auth-anonymous-scopes '*:read' ``` Operator-managed anonymous scopes are not available in this Stack v4 preview. Do not translate this direct-server flag into an undocumented Stack resource field. This fallback applies only when the request has no bearer token. | Request | Expected result | | --- | --- | | Read with no token | Allowed by `*:read` | | Read with an invalid or expired token | `401 Unauthorized` | | Read with a valid write-only token | `403 Forbidden`; anonymous scopes are not merged into a valid identity | | Write with no token | `401 Unauthorized` | | Write with a valid token missing the write scope | `403 Forbidden` | Use explicit granular scopes instead of `*:read` when only part of the read API should be public. For a direct server, pass them as a comma-separated value: ```bash --auth-anonymous-scopes 'ledger:LedgerRead,ledger:TransactionRead' ``` The HTTP endpoints `/health`, `/livez`, `/readyz`, and `/_info` bypass authentication entirely. `/clusterz` carries no scope requirement, so it also answers a request that sends no token — but, unlike those four, it returns `401` if the request presents a malformed or expired one. The gRPC health service, `BucketService.Discovery`, and gRPC reflection are also unauthenticated. ## Verify the policy Run these checks against the public HTTP endpoint before routing application traffic. Strict mode should produce `401`, `401`, `200`, and `403` in that order: ```bash curl --silent --output /dev/null --write-out '%{http_code}\n' \ "$LEDGER_URL/v3/" curl --silent --output /dev/null --write-out '%{http_code}\n' \ --header 'Authorization: Bearer not-a-valid-token' \ "$LEDGER_URL/v3/" curl --silent --output /dev/null --write-out '%{http_code}\n' \ --header "Authorization: Bearer $READ_TOKEN" \ "$LEDGER_URL/v3/" curl --silent --output /dev/null --write-out '%{http_code}\n' \ --request POST \ --header "Authorization: Bearer $READ_TOKEN" \ --header 'Content-Type: application/json' \ --data '{not-valid-json' \ "$LEDGER_URL/v3/authz-probe/transactions" ``` The final request targets a real transaction-write route but deliberately sends malformed JSON. Authorization runs before body parsing, so the read-only token must receive `403`. Even if the token unexpectedly has write access, body parsing returns `400` before Ledger can submit a mutation; this probe cannot create a ledger or transaction. If anonymous reads are enabled, only the first expected status changes from `401` to `200`. The invalid-token request must still return `401`. ## Diagnose authorization failures | Symptom | Likely cause | Check | | --- | --- | --- | | Node does not become ready after enabling OIDC | Discovery URL or JWKS cannot be reached, or the issuer is invalid | Fetch the issuer's OpenID configuration from the Ledger network path and inspect node startup logs | | `401 Unauthorized` | Token is absent, malformed, expired, signed by an unknown key, or issued by another issuer | Decode the JWT header and claims; verify `kid`, `iss`, and `exp` without logging the token | | `403 Forbidden` | The token is valid but lacks the operation's required scope | Compare the requested operation with the token scopes and the configured scope mapping | | Ed25519 token always returns `403` | The token uses `read` or `write` instead of a configured virtual or granular scope | Use `ledger:read`, `ledger:write`, or an exact `ledger:` value | | One replica rejects a token after credential rotation | Its mounted public-key configuration has not converged | Check the deployment rollout and test the same token through every reachable replica | | Follower receives a write but the leader rejects it | Internal TLS or the cluster secret is missing or inconsistent | Follow the forwarded-write checks in [TLS and Network Security](/modules/ledger/operations/tls-and-network-security) | ## Rotate an Ed25519 credential Do not replace the only accepted key in one step. Use the release-specific Stack v4 workflow to register the replacement public key alongside the old one, with narrower or equal scopes. Wait until every replica accepts the new key, move clients to tokens signed by the new seed, and remove the old credential only after those tokens succeed through every reachable replica. OIDC signing-key rotation remains owned by the identity provider. Keep the old key available in its JWKS until issued tokens have expired, and verify a token with the new `kid` before removing the old key. --- ## Choose Read Consistency and Use Checkpoints Source: https://docs.formance.com/modules/ledger/querying/consistency-and-checkpoints Choose read consistency from the business consequence of stale or changing data. A user-facing balance usually needs a fresh live read. A dashboard can often accept bounded lag. A reconciliation that spans several requests needs a query checkpoint. A query checkpoint freezes one committed sequence of the ledger so that many reads — several queries, and every page of each — all describe the same unchanging state, even while writes continue. The commands assume an authenticated `ledgerctl` profile, `jq`, and `export LEDGER_NAME=main`. Entity reads require their normal read scopes. Creating or deleting a query checkpoint, and setting or deleting its schedule, requires `ledger:ClusterWrite`; listing, inspecting, or reading the schedule requires `ledger:ClusterRead`. See [Authentication and Authorization](/modules/ledger/operations/authentication). ## Choose a live-read mode The gRPC API accepts a consistency level through the `x-consistency` metadata field. `ledgerctl` exposes it as a global `--consistency` flag. A read barrier is a round-trip to a majority of nodes that confirms the reading node has seen every committed write; it costs latency and needs that majority reachable. | Mode | Guarantee | Cost and availability | Use it for | | --- | --- | --- | --- | | `linearizable` | Confirms the latest committed state with the cluster before reading | Uses a Raft read barrier, requires a working quorum, and can wait while the node catches up | Balances, transaction confirmation, decisions based on current state | | `leader` | Routes the read to the leader. A remote leader applies its own read barrier; a node that is already the leader serves the read locally without one | Depends on leader reachability | Fresh reads when a contacted follower may lag | | `stale` | Reads the contacted node's local state without a read barrier | Can return older committed data | Dashboards, diagnostics, and other lag-tolerant views | `linearizable` is the default for both gRPC and HTTP reads: ```bash # Default linearizable read ledgerctl accounts get users:alice --ledger "$LEDGER_NAME" # Route the read to the leader ledgerctl --consistency leader accounts get users:alice --ledger "$LEDGER_NAME" # Read local state without waiting for a barrier ledgerctl --consistency stale accounts list --ledger "$LEDGER_NAME" --page-size 100 ``` Do not use `stale` for a check that authorizes a withdrawal, confirms a write, or decides whether money can move. The response can predate a write that another caller has already observed. The direct HTTP routes do not expose the `stale` or `leader` selector, so each request uses the default linearizable read. The direct HTTP list routes also do not expose checkpoint or minimum-sequence options. Each response therefore reflects committed cluster state, but two requests can observe different commits while writes continue. A linearizable barrier also does not guarantee that asynchronous query indexes have processed that state. Use gRPC or `ledgerctl` for a fixed checkpoint or a minimum-sequence control. ## Wait for the read index A successful write can return before the node serving a later query has indexed that write. The read index is the replica's local view optimized for queries. When a follow-up query must include a write, capture the highest global log sequence from the full gRPC `ApplyResponse` and pass that value as `minLogSequence` or `--min-log-sequence`. The following example assumes the application saved the complete response as `apply-response.json`: ```bash MIN_LOG_SEQUENCE="$( jq -er ' .logs[-1].sequence // error("the response contains no new log sequence") | tostring ' apply-response.json )" TRANSACTION_ID="$( jq -er ' [.logs[] | .payload.apply.log.data.createdTransaction.transaction.id // empty] | if length > 0 then .[0] | tostring else error("the response contains no created transaction") end ' apply-response.json )" TRANSACTION_FILTER="$( jq -nc --arg id "$TRANSACTION_ID" ' {"$and":[{"$gte":{"id":$id}},{"$lte":{"id":$id}}]} ' )" ledgerctl transactions list \ --ledger "$LEDGER_NAME" \ --filter "$TRANSACTION_FILTER" \ --min-log-sequence "$MIN_LOG_SEQUENCE" \ --page-size 2 \ --json \ | jq -e --arg id "$TRANSACTION_ID" \ 'any(.[]; (.id | tostring) == $id)' ``` `ApplyResponse.logs` is returned in commit order, so the last entry carries the highest sequence produced by that proposal. A response with no fresh log produced no new sequence to wait for, so the filter raises an error rather than yielding the string `null`, which `--min-log-sequence` would otherwise accept. The final `jq` prints `true` and exits with status `0` only when the read contains the transaction returned by the write. Direct gRPC list and aggregation calls wait until the contacted node has indexed `MIN_LOG_SEQUENCE`, or until the request times out. Do not estimate this value from a transaction ID or a timestamp. `ledgerctl transactions create --json` prints the created transaction, not the full `ApplyResponse` wrapper. An application that needs a sequence bound must retain `logs[].sequence` from its gRPC `Apply` call, or obtain the correlated global sequence through the [transaction-to-audit trace](/modules/ledger/querying/read-ledger-data#trace-a-successful-transaction). `minLogSequence` guarantees general read-index progress. It does not prove that an index backfill or metadata-index version transition has switched to its new version. Verify index readiness separately on every serving replica. The REST prepared-query execution route is the one HTTP query surface that accepts `minLogSequence` in its request body. Prepared-query execution uses it as a freshness gate rather than waiting inside the request. On gRPC, prepared-query execution returns `READ_INDEX_NOT_CAUGHT_UP` when the local index is behind, which a client can retry with bounded backoff; use that surface for reason-specific retries in v3.0. This one-shot gate is specific to prepared-query execution: the direct gRPC list and aggregation calls above wait instead of returning that reason. The direct account, transaction, log, and audit list routes do not expose the field. Support differs by operation. Every row below describes the gRPC and `ledgerctl` surfaces, not the direct HTTP list routes: | Operation (gRPC and `ledgerctl`) | `minLogSequence` | Query checkpoint | | --- | --- | --- | | Account, transaction, or log list | Yes | Yes | | Audit list | Yes | Yes, with the filtered-read warning below | | Audit get or `--expand` | No | No | | Account volume aggregation | Yes | Yes | | Prepared-query execution | Yes, as a freshness gate | No | | Ledger statistics | No | Yes, with limited counter semantics | | Account or transaction analysis | No | No | ## Freeze a view with a query checkpoint A checkpoint captures both the ledger data and its derived query data at that sequence, so a checkpoint-scoped read never mixes a frozen entity with a live index. ```mermaid flowchart LR Writes["Live writes continue"] --> Live["Current ledger state"] Live -->|"create checkpoint"| Frozen["Frozen main store and read index"] Frozen --> A["Account pages"] Frozen --> T["Transaction pages"] Frozen --> V["Volume aggregation"] A --> Reconciliation["One consistent reconciliation"] T --> Reconciliation V --> Reconciliation ``` Create the checkpoint and derive both returned values from the command output: ```bash CHECKPOINT_JSON="$(ledgerctl query-checkpoint create --json)" CHECKPOINT_ID="$(jq -er '.checkpointId | tostring' <<<"$CHECKPOINT_JSON")" cleanup_checkpoint() { local exit_status=$? trap - EXIT INT TERM if [ -n "${CHECKPOINT_ID:-}" ]; then if ! ledgerctl query-checkpoint delete "$CHECKPOINT_ID"; then printf '%s\n' \ "Checkpoint cleanup failed for ID: $CHECKPOINT_ID" \ "Ledger will not delete this checkpoint automatically." \ "Delete it manually after resolving the error:" \ " ledgerctl query-checkpoint delete $(printf '%q' "$CHECKPOINT_ID")" \ >&2 if [ "$exit_status" -eq 0 ]; then exit_status=1 fi fi fi exit "$exit_status" } trap cleanup_checkpoint EXIT trap 'exit 130' INT trap 'exit 143' TERM # Parsed after the trap is armed: a failure here must still delete the # checkpoint the create call already made durable. CHECKPOINT_MAX_SEQUENCE="$(jq -er '.maxSequence | tostring' <<<"$CHECKPOINT_JSON")" CHECKPOINT_REGISTERED=false EXPECTED_NOT_FOUND="query checkpoint $CHECKPOINT_ID not found" for attempt in {1..30}; do if CHECKPOINT_INFO="$( ledgerctl query-checkpoint info "$CHECKPOINT_ID" --json \ 2>checkpoint-info.err )" then if ! jq -e \ --arg id "$CHECKPOINT_ID" \ --arg max "$CHECKPOINT_MAX_SEQUENCE" ' (.checkpointId | tostring) == $id and (.maxSequence | tostring) == $max ' <<<"$CHECKPOINT_INFO" then exit 1 fi CHECKPOINT_REGISTERED=true break fi if ! grep -Fq "$EXPECTED_NOT_FOUND" checkpoint-info.err; then cat checkpoint-info.err >&2 exit 1 fi sleep 2 done if [ "$CHECKPOINT_REGISTERED" != true ]; then cat checkpoint-info.err >&2 exit 1 fi ``` The `tostring` conversions are required, not defensive noise: `query-checkpoint create --json` reports `checkpointId` and `maxSequence` as JSON numbers, while `query-checkpoint info --json` reports the same two fields as JSON strings, so the comparison must be made on a common type. This check prints `true` only when one contacted replica has registered the exact checkpoint that was just created. A replica that has not yet applied the creation can temporarily return `NotFound`, so the loop retries only the exact captured ID and stops on every other error. Keep `CHECKPOINT_ID` in the same job that performs and later deletes the reconciliation. The `EXIT` trap is a last-resort cleanup guard, not a retention policy. An interrupt exits with status `130`, a termination signal exits with status `143`, and the cleanup preserves any other existing failure status. If its deletion fails, it prints the captured checkpoint ID and the exact manual command instead of hiding the failure. Ledger does not expire the checkpoint automatically. There is no separate `READY` field in checkpoint information. A successful checkpoint-scoped read is the readiness criterion for the replica that served it. Probe with bounded retries: ```bash CHECKPOINT_READY=false EXPECTED_NOT_READY="query checkpoint $CHECKPOINT_ID read index is still building" for attempt in {1..30}; do if ledgerctl accounts list \ --ledger "$LEDGER_NAME" \ --checkpoint-id "$CHECKPOINT_ID" \ --page-size 1 \ --json \ >/dev/null 2>checkpoint-read.err then CHECKPOINT_READY=true break fi if ! grep -Fq "$EXPECTED_NOT_READY" checkpoint-read.err; then cat checkpoint-read.err >&2 exit 1 fi sleep 2 done if [ "$CHECKPOINT_READY" != true ]; then cat checkpoint-read.err >&2 exit 1 fi ``` The gRPC error carries `CHECKPOINT_NOT_READY` in `google.rpc.ErrorInfo.reason`. Application clients should retry only that reason. Ledger v3.0's `ledgerctl` output omits the structured reason, so the shell loop matches the exact server message and checkpoint ID that represent it; every other failure stops immediately. The probe proves readiness only for the replica that served it. Keep subsequent reconciliation reads on that ready route, or repeat the probe against every pod-direct endpoint that can receive them. Use the same captured ID for every read in the reconciliation: ```bash ledgerctl accounts list \ --ledger "$LEDGER_NAME" \ --checkpoint-id "$CHECKPOINT_ID" \ --page-size 500 \ --json ledgerctl transactions list \ --ledger "$LEDGER_NAME" \ --checkpoint-id "$CHECKPOINT_ID" \ --page-size 500 \ --json ledgerctl accounts aggregate-volumes \ --ledger "$LEDGER_NAME" \ --checkpoint-id "$CHECKPOINT_ID" \ --json ``` Follow each emitted cursor with the same checkpoint ID until the traversal finishes. Do not switch back to a live read between pages. Do not combine an audit filter with a checkpoint when completeness is required. In Ledger v3.0, an audit entry that the asynchronous audit index had not processed when the checkpoint was created can remain absent from a filtered checkpoint read. Read the checkpoint's audit stream without a filter, then filter it in the client. Checkpoint reads ignore `minLogSequence`: the checkpoint already identifies one past sequence. After every consumer finishes, delete the exact captured checkpoint and verify that it no longer appears in the registry: ```bash if ! ledgerctl query-checkpoint delete "$CHECKPOINT_ID"; then exit 1 fi CHECKPOINT_DELETED=false for attempt in {1..30}; do if ! ledgerctl query-checkpoint list --json \ >checkpoint-registry.json 2>checkpoint-registry.err then cat checkpoint-registry.err >&2 exit 1 fi if jq -e --arg id "$CHECKPOINT_ID" \ 'all(.[]; (.checkpointId | tostring) != $id)' \ checkpoint-registry.json >/dev/null then CHECKPOINT_DELETED=true break fi sleep 2 done if [ "$CHECKPOINT_DELETED" != true ]; then printf '%s\n' \ "Checkpoint $CHECKPOINT_ID still present in the registry after the delete." \ "Confirm it against each replica before deleting it again." >&2 exit 1 fi CHECKPOINT_ID="" trap - EXIT INT TERM ``` The loop retries because `list` is served by one replica. Deletion propagates the same way creation does, so a list routed to a replica that has not applied it yet still reports the checkpoint. Without the retry a successful cleanup can exit non-zero, which leaves `CHECKPOINT_ID` set and makes the `EXIT` trap issue a second delete for a checkpoint that is already gone. Clearing `CHECKPOINT_ID` before disarming the trap is what makes the confirmed path skip that second delete. Absence from the registry does not prove that every replica has reclaimed the checkpoint's disk space. Do not derive a deletion candidate from the most recent list entry; another job may have created a checkpoint in the meantime. ## Handle checkpoint readiness Checkpoint materialization happens independently on each replica. If a read is routed to a replica that has not yet materialized the checkpoint, it returns `CHECKPOINT_NOT_READY`. Creation waits only for the replica that handled the create command. Treat the outcomes differently: | Outcome | Meaning | Client action | | --- | --- | --- | | `CHECKPOINT_NOT_READY` / gRPC `Unavailable` | The checkpoint exists, but this replica has not materialized it | Retry with bounded backoff or route subsequent reads to the replica that is ready | | `NotFound` | The checkpoint does not exist or was deleted | Stop retrying and confirm the checkpoint ID | Ledger never returns a partially built checkpoint. If one replica remains `CHECKPOINT_NOT_READY` beyond your operational timeout, route the reconciliation to a ready replica or recreate the checkpoint. Repeated retries cannot repair a checkpoint that the replica can no longer materialize. ## Schedule checkpoints Schedule creation when a recurring reconciliation needs predictable cut-off points: ```bash # Every day at midnight ledgerctl query-checkpoint set-schedule '0 0 * * *' ledgerctl query-checkpoint get-schedule ``` The scheduler accepts a standard five-field cron expression or a six-field expression with leading seconds. Only the leader creates the checkpoint when the schedule fires. Ledger does not delete old query checkpoints automatically. Define a retention policy and use the captured-ID deletion workflow above after every consumer has finished. ```bash # Disable future scheduled creation ledgerctl query-checkpoint delete-schedule ``` ## Request-signing limitations In the current 3.0 preview, `ledgerctl query-checkpoint create` and `ledgerctl query-checkpoint delete` call unary gRPC methods that submit unsigned mutations. They fail while mandatory request signing is enabled. A configured checkpoint schedule also submits each scheduled creation unsigned, so the schedule can remain visible without producing checkpoints. Before enabling mandatory signatures, finish every checkpoint-based reconciliation, delete its captured checkpoint, remove any checkpoint schedule, and verify the checkpoint registry. Existing checkpoint reads remain available after the policy change, but the current CLI cannot delete those checkpoints. Do not depend on checkpoint creation or cleanup again until the deployed release provides a tested signed path for both operations. ## Keep checkpoints distinct from backups | Mechanism | Purpose | Query interface | Retention | | --- | --- | --- | --- | | Query checkpoint | Repeatable, point-in-time application reads | Ledger read APIs with a checkpoint ID | Explicit deletion | | Raft snapshot | Restore consensus state for cluster operation | Not an application-query surface | Managed by Ledger | | Full backup | Recover a complete durable state | Read after restore | Backup policy | | Incremental backup | Add committed changes since the preceding backup boundary | Read after a compatible restore chain | Backup policy | Ledger keeps log and audit history permanently in the primary store, so no history ages out from under a read or from behind a checkpoint. Checkpoint creation takes no sequence argument: a new checkpoint freezes the state current at creation time, and reaching an earlier point means having created a checkpoint then. A query checkpoint is still a read tool, not a disaster-recovery artifact: do not use it as the only copy of ledger data. Continue with [Create and Manage Indexes](/modules/ledger/querying/indexes) when a filtered read needs an opt-in index before it can serve traffic. --- ## Reuse and Version Numscript Programs Source: https://docs.formance.com/modules/ledger/transactions/numscript-library Store a Numscript program in a ledger when several callers must execute the same reviewed transaction logic. Callers send a name, version, and variables instead of embedding the source in every request. For production traffic, publish an immutable semantic version and request that exact version. This makes a deployment reproducible and gives the application an explicit rollback target. ## Prerequisites You need an existing normal ledger and a token with `ledger:write` to publish and execute programs — the save needs `ledger:LedgerWrite`, the execution needs `ledger:TransactionWrite` — plus `ledger:read` for the verification requests. The virtual scopes below assume the default scope mapping. If the ledger enforces account types, replace the demo destination with an address that matches its configured patterns. ```bash set -eo pipefail ``` ## Publish an immutable version Define a reusable transfer program: ```numscript vars { account $source account $destination monetary $amount } send $amount ( source = $source destination = $destination ) ``` Put the same source in a shell variable for the request below: ```bash TRANSFER_V1=$(cat <<'NUMSCRIPT' vars { account $source account $destination monetary $amount } send $amount ( source = $source destination = $destination ) NUMSCRIPT ) ``` Save it under the run-specific name in `$NUMSCRIPT_NAME` as version `1.0.0`. The `Idempotency-Key` makes the save safe to retry after a lost response: ```bash jq --null-input \ --arg content "$TRANSFER_V1" \ '{content: $content, version: "1.0.0"}' \ | curl --fail-with-body --silent --show-error \ --request PUT "$LEDGER_URL/v3/$LEDGER_NAME/numscripts/$NUMSCRIPT_NAME" \ --header "Authorization: Bearer $LEDGER_TOKEN" \ --header "Content-Type: application/json" \ --header "Idempotency-Key: numscript:$NUMSCRIPT_NAME:1.0.0" \ --data @- \ > /tmp/numscript-transfer-1.0.0.json ``` Ledger parses the program before saving it. A successful save returns `201 Created` with the stored program info. Verify the stored source before pointing callers at the new version. Verify the stored source through an exact-version lookup: ```bash curl --fail-with-body --silent --show-error \ "$LEDGER_URL/v3/$LEDGER_NAME/numscripts/$NUMSCRIPT_NAME?version=1.0.0" \ --header "Authorization: Bearer $LEDGER_TOKEN" \ | jq -e --arg name "$NUMSCRIPT_NAME" --arg content "$TRANSFER_V1" ' .data | select(.name == $name and .version == "1.0.0" and .content == $content) | {name, version, content, createdAt}' ``` The command exits successfully only when the response reports the requested name, exact version, and submitted source. ## Execute the saved program Reference the exact version in a transaction request: ```bash jq --null-input \ --arg reference "$TRANSFER_REFERENCE" \ --arg name "$NUMSCRIPT_NAME" \ --arg destination "$DESTINATION_ACCOUNT" \ '{ reference: $reference, scriptReference: { name: $name, version: "1.0.0", vars: { source: "world", destination: $destination, amount: "USD/2 2500" } } }' \ | curl --fail-with-body --silent --show-error \ --request POST "$LEDGER_URL/v3/$LEDGER_NAME/transactions" \ --header "Authorization: Bearer $LEDGER_TOKEN" \ --header "Content-Type: application/json" \ --header "Idempotency-Key: $TRANSFER_REFERENCE:create" \ --data @- \ > /tmp/referenced-numscript-transaction.json ``` Expected result: Ledger returns `201 Created`, and `.data.transaction.postings` contains the movement produced by the exact `1.0.0` version. This example uses `world` as an explicit funding boundary. For an internal transfer, pass a funded account instead. Verify both the program choice and the accounting result: ```bash jq '.data.transaction | {id, reference, postings}' \ /tmp/referenced-numscript-transaction.json curl --fail-with-body --silent --show-error \ "$LEDGER_URL/v3/$LEDGER_NAME/accounts/$DESTINATION_ACCOUNT" \ --header "Authorization: Bearer $LEDGER_TOKEN" \ | jq -e ' .data.volumes[] | select(.asset == "USD/2" and .color == "") | .volumes.input == "2500" and .volumes.output == "0" and .volumes.balance == "2500"' ``` The destination balance increases by `2500` minor units. ## Publish a compatible change Semantic versions are immutable. Publish changed behavior under a new version instead of overwriting `1.0.0`. Version `1.1.0` adds a metadata tag: ```numscript vars { account $source account $destination monetary $amount } set_tx_meta("program", "transfer@1.1.0") send $amount ( source = $source destination = $destination ) ``` Put that source in a shell variable and save it as `1.1.0`: ```bash TRANSFER_V1_1=$(cat <<'NUMSCRIPT' vars { account $source account $destination monetary $amount } set_tx_meta("program", "transfer@1.1.0") send $amount ( source = $source destination = $destination ) NUMSCRIPT ) jq --null-input \ --arg content "$TRANSFER_V1_1" \ '{content: $content, version: "1.1.0"}' \ | curl --fail-with-body --silent --show-error \ --request PUT "$LEDGER_URL/v3/$LEDGER_NAME/numscripts/$NUMSCRIPT_NAME" \ --header "Authorization: Bearer $LEDGER_TOKEN" \ --header "Content-Type: application/json" \ --header "Idempotency-Key: numscript:$NUMSCRIPT_NAME:1.1.0" \ --data @- \ > /dev/null ``` Confirm that both versions remain available: ```bash ( set -euo pipefail for VERSION in 1.0.0 1.1.0; do curl --fail-with-body --silent --show-error \ "$LEDGER_URL/v3/$LEDGER_NAME/numscripts/$NUMSCRIPT_NAME?version=$VERSION" \ --header "Authorization: Bearer $LEDGER_TOKEN" \ | jq -er --arg name "$NUMSCRIPT_NAME" --arg version "$VERSION" ' .data | select(.name == $name and .version == $version) | .name + "@" + .version' done ) ``` Expected output, with the run-specific name substituted for ``: ```text @1.0.0 @1.1.0 ``` Move callers to `1.1.0` only after testing representative balances, metadata, and failure cases. Roll back the application by changing its `scriptReference.version` to `1.0.0`; no library mutation is required. ## Understand version resolution A selector means different things on the read path and the write path. Reads are permissive; execution and publication are not. Accepted on `GET /v3/{ledgerName}/numscripts/{name}?version=`: | Requested version | Resolves to | | --- | --- | | `1.0.0` | That exact immutable version. | | `1.0` | The greatest stored `1.0.x` version. | | `1` | The greatest stored `1.x.y` version. | | `latest` | The greatest stored semantic version. | | Omitted | The greatest stored semantic version. `latest` and an omitted value are the same selector. | Accepted in `scriptReference.version` on a transaction: | Requested version | Behavior | Production guidance | | --- | --- | --- | | `1.0.0` | Executes that exact immutable version. | Recommended for a stable deployment. | | `latest` | Re-resolves to the greatest stored version on every execution. | Use for development or an explicitly mutable workflow. | | `1`, `1.0`, or omitted | Rejected with `400` and an invalid-version reason. Partial and omitted selectors are read-only. | Not available. Pin an exact version. | `PUT .../numscripts/{name}` accepts an exact `major.minor.patch` only. `latest`, a partial selector, and an omitted version are all rejected with `400` and an invalid-version reason. Ledger accepts only numeric selectors with one to three components (`1`, `1.2`, or `1.2.3`), the literal `latest`, or an omitted value. Version ranges, prerelease labels, and build metadata are not supported. Saving a version advances the program's pointer only when the new version is the greatest stored: the pointer is `max(current, new)`, so adding an older version out of order does not move it. There is no separate mutable `latest` slot — `latest` is a selector that re-resolves through that pointer on every request. `latest` re-resolves on every execution, so an operation without a retained idempotency outcome can run different source after another caller publishes a greater version. A retained request that submitted `latest` replays its first outcome: Ledger audits the selector as submitted and never rewrites it to a concrete version, so an unchanged body cannot become an idempotency conflict because the resolution moved. Exact semantic versions avoid the ambiguity altogether. ## Check whether a program name is still used The usage counter answers one question: is anything still calling this program name? It aggregates invocations across every version stored under that name: ```bash ( USAGE_TIMEOUT_SECONDS=${USAGE_TIMEOUT_SECONDS:-30} USAGE_DEADLINE=$((SECONDS + USAGE_TIMEOUT_SECONDS)) USAGE_READY=false while ((SECONDS < USAGE_DEADLINE)); do if ! curl --fail-with-body --silent --show-error \ "$LEDGER_URL/v3/$LEDGER_NAME/numscripts/$NUMSCRIPT_NAME/usage" \ --header "Authorization: Bearer $LEDGER_TOKEN" \ > /tmp/numscript-usage.json; then echo "Usage request failed; inspect /tmp/numscript-usage.json before retrying." >&2 exit 1 fi USAGE_COUNT=$(jq -er '.data.count | select(type == "number")' \ /tmp/numscript-usage.json) || { echo "Usage response did not contain a numeric data.count." >&2 exit 1 } if ((USAGE_COUNT >= 1)); then USAGE_READY=true break fi sleep 2 done if [[ "$USAGE_READY" != "true" ]]; then echo "Usage did not converge within ${USAGE_TIMEOUT_SECONDS}s; the transaction result is unchanged. Retry this read later." >&2 exit 1 fi jq '.data | {count, lastUsed}' /tmp/numscript-usage.json ) ``` After the example transaction, the result eventually reports a positive count and the greatest effective transaction timestamp seen for this name: ```json { "count": 1, "lastUsed": "2026-07-17T10:30:00Z" } ``` The HTTP route requires the `ledger:read` virtual scope; the direct gRPC method checks the granular query-read permission. It does not accept a checkpoint or minimum log sequence. The counter normally follows committed writes after a short background interval, but catch-up after startup or restore can take longer; increase `USAGE_TIMEOUT_SECONDS` or retry the read later in those cases. It counts only committed transactions that used `scriptReference`: failed or skipped attempts, inline scripts, and idempotent replays do not add another invocation. An unknown program name on an existing ledger returns `count: 0`; an unknown ledger returns `404`. Treat usage as an operational adoption signal, not a billing counter. The counter is eventually consistent, so a `0` response can mean the program has never run or that the counter has not caught up yet. `lastUsed` is an effective transaction timestamp, not the time the counter observed the invocation. Entries without a saved-program identity, including records replayed from another ledger, do not add to this count. The count is per program name, not per semantic version. ## List the programs in a ledger List the current entry for each program in the ledger: ```bash curl --fail-with-body --silent --show-error \ "$LEDGER_URL/v3/$LEDGER_NAME/numscripts" \ --header "Authorization: Bearer $LEDGER_TOKEN" \ | jq '.data[] | {name, version, createdAt}' ``` The Ledger 3.0 numscript surface has no delete operation: the routes are `GET` for the listing, an individual program, its usage, and its versions, plus `PUT` to save. A published semantic version is immutable and permanent, so retiring a program means moving its callers off it rather than removing it from the ledger. Combine the usage signal with a search of deployed callers and recent transaction evidence before you treat a program name as unused; a low or zero count alone is not enough. ## Handle errors and retries | Result | Meaning | Action | | --- | --- | --- | | `400` with a parse reason | The Numscript source is invalid | Fix and test the source before publishing a version. | | `400` with an invalid-version reason | A save version is not an exact `major.minor.patch`, or an executable `scriptReference.version` is neither an exact version nor `latest` | Publish with `major.minor.patch`, and execute with an exact version. | | `404` | The program or requested version does not exist | Check the ledger, name, and version constraint. | | `409` | The semantic version already exists | Do not overwrite it. Compare the stored content, then select the existing version or publish a new one. | | `503` | The write path is temporarily unavailable | Read the exact target version before retrying, because the save may have committed. | `PUT .../numscripts/{name}` honors the `Idempotency-Key` header, so send one and retry the identical body under the same key when a network failure makes a save uncertain. If you did not send a key, fetch the exact semantic version and compare its content instead. If it exists and matches, treat the publication as successful. If it differs, preserve the stored version and publish the intended change under a new version. Continue with [Retry Writes Safely](/modules/ledger/transactions/idempotency-and-references) and apply it to every transaction that executes a saved program. --- ## Transactions & Postings Source: https://docs.formance.com/modules/ledger/core-concepts/transactions The Formance Ledger represents financial activity using two primitives: **postings** and **transactions**. Postings are the basic unit of movement; transactions are timestamped containers that group one or more postings together. A committed transaction's financial content is immutable: its postings, amounts, assets, and timestamp can never change. To correct an error, see [Reverting Transactions](/modules/ledger/working-with/reverting-transactions). Metadata is the exception — it can be added or removed at any time after commit, as described in [Metadata](#metadata) below. ## Postings A posting records the movement of an amount of an asset from a source account to a destination account. ```json { "source": "alice", "destination": "bob", "asset": "COIN", "amount": 100 } ``` ![A posting moving 100 COIN from alice to bob](/assets/images/ledger/core-concepts/posting.svg) Each posting has exactly one source and one destination. The Formance Ledger records the full history of postings; account balances are derived from that history rather than stored directly. The `asset` field must match the ledger's asset pattern — an uppercase identifier, optionally with an `_SUFFIX` and an optional `/precision` — but beyond that shape the ledger attaches no meaning to it: there is no currency registry. By convention the name follows the ISO 4217 three-letter currency code (e.g. `USD`, `EUR`, `GBP`), suffixed with a precision indicator where the asset has decimals (`USD/2` means amounts are expressed in cents). Custom asset names like `COIN` or `REWARD_POINT` are equally valid. See [Assets](/modules/ledger/core-concepts/assets) for the exact pattern. ## Transactions A transaction is a timestamped, permanently recorded grouping of one or more postings. For example, Alice trading coins for gems at a teller involves two postings in one transaction: ```json { "postings": [ { "source": "alice", "destination": "teller", "amount": 100, "asset": "COIN" }, { "source": "teller", "destination": "alice", "amount": 5, "asset": "GEM" } ] } ``` ![A transaction grouping two postings between alice and teller](/assets/images/ledger/core-concepts/transaction.svg) ### Atomicity Transactions are **atomic**: all of their postings are applied together, or none are. There is no intermediate state in which some postings have been committed and others have not. ![Atomicity: a transaction either applies all postings or rolls back entirely](/assets/images/ledger/core-concepts/atomicity.svg) The diagram above shows a customer payment split between a merchant and the platform. If the posting to `platform:fees` fails, atomicity means the posting to `merchant:account` is not applied either: the customer is not charged and the merchant receives nothing. There is no intermediate state where the merchant was paid but the platform was not. ## Single source and destination postings Every posting in the Formance Ledger has exactly one source and one destination. When a split or fan-out is needed, use a transient intermediary account: funds move into it in one posting and out to each destination in separate postings, all within the same atomic transaction: ![A split payment via a transient order:hold account](/assets/images/ledger/core-concepts/split-payment.svg) ```json { "postings": [ { "source": "customer:wallet", "destination": "order:hold", "asset": "USD/2", "amount": 2000 }, { "source": "order:hold", "destination": "merchant:account", "asset": "USD/2", "amount": 1800 }, { "source": "order:hold", "destination": "rider:earnings", "asset": "USD/2", "amount": 100 }, { "source": "order:hold", "destination": "platform:fees", "asset": "USD/2", "amount": 100 } ] } ``` `order:hold` is the transient intermediary. It receives the full amount in the first posting and distributes it in the three that follow. All four postings commit atomically: either the full split happens, or nothing does. The split logic ("5% to fees, 5% to the rider, the rest to the merchant") belongs in [Numscript](/modules/numscript), which compiles your intent into unambiguous single-destination postings automatically. Numscript runs on the `machine` interpreter by default. Experimental language features require the new interpreter — set `runtime: experimental-interpreter` on the transaction request. See [Selecting an Interpreter](/modules/numscript/interpreter). ### Why not a single posting with multiple destinations? An alternative design would allow one posting to fan out to multiple destinations directly: ![A multi-destination posting where amounts are ambiguous](/assets/images/ledger/core-concepts/multi-destination.svg) This is mathematically valid, but consider the questions you need a ledger to answer. Looking at alice's account: *how much did alice pay in this transaction?* Looking at the fees account: *how much of this transaction went to fees?* With a multi-destination posting, neither question has a clean answer; the `amount` field on the posting is ambiguous, and reconstructing per-destination flows requires parsing the posting's structure rather than reading a single number. With single-destination postings, every question resolves to one posting with one unambiguous amount: alice moved exactly `2000 USD/2` to `order:hold`, and `platform:fees` received exactly `100 USD/2`. Crediting a user from multiple source accounts, for example, can be expressed as a single transaction rather than a series of independent operations that must succeed or fail together. Each posting has one source, one destination, and one unambiguous amount. Every account balance question resolves to a single number on a single line. ## Volumes and Balances When a transaction commits, the ledger records the running totals (called **volumes**) on every account involved. The Ledger distinguishes between volumes and balances: | Concept | Description | |---------|-------------| | **Volumes** | The total input and output amounts processed for an account. Stored as unsigned integers, always non-negative. | | **Balances** | The available amount in an account. Can be positive, negative, or zero. | The ledger does not store balances directly. Instead, it records cumulative volumes. Balances are derived by subtracting output from input. ### Terminology across the API The Formance Ledger uses different terms depending on context: | Concept | Account component | API (postings) | API (volumes) | |---|---|---|---| | Value arriving in an account | Destination component | `destination` field | `input` field | | Value leaving an account | Source component | `source` field | `output` field | Postings (`source`/`destination` fields) describe movement between accounts. Volumes (`input`/`output` in `preCommitVolumes` and `postCommitVolumes`) are the running totals on each account before and after the transaction commits. ```json { "preCommitVolumes": { "world": { "USD/2": { "input": 0, "output": 0 } }, "bank": { "USD/2": { "input": 0, "output": 0 } } }, "postCommitVolumes": { "world": { "USD/2": { "input": 0, "output": 100 } }, "bank": { "USD/2": { "input": 100, "output": 0 } } } } ``` ## Metadata Transactions support metadata: key-value pairs you can attach at creation time or update later. ### Setting transaction metadata Use `set_tx_meta()` in Numscript to attach metadata to a transaction at creation time: ```numscript set_tx_meta("reference", "order-12345") set_tx_meta("type", "payment") send [USD/2 1000] ( source = @customers:123 destination = @merchants:456 ) ``` ### Setting account metadata in a transaction Use `set_account_meta()` in Numscript to set or update account metadata as part of a transaction: ```numscript set_account_meta(@users:alice, "tier", "premium") ``` For managing account metadata outside of transactions, see [Accounts](/modules/ledger/core-concepts/accounts#using-metadata). --- ## Create a Mirror Source: https://docs.formance.com/modules/ledger/migration/create-mirror Mirror mode is a creation-time choice. Create a new target with its source and optional transformations in one operation. If the target model must differ from Ledger v2, finish [Transform Mirrored Data](/modules/ledger/migration/transformations) before running the commands on this page. Rewrite rules cannot be added or changed after creation. You cannot convert an existing normal ledger into a mirror. While the target is in mirror mode, application writes are rejected; only the mirror worker imports Ledger v2 logs. ## Choose a creation surface | Surface | Use it for | Credential behavior | | --- | --- | --- | | `ledgerctl` | Controlled administration, including clusters that require signed writes, or a non-Kubernetes deployment. | Accepts request-signing keys, secrets, and DSNs as CLI or profile values. Protect the environment, process list, and command output. | | HTTP API | Programmatic creation only when the target accepts unsigned writes. | Accepts OAuth2 secrets or a PostgreSQL DSN inline in the request body. Request signing and RDS IAM fields are not exposed over HTTP. | The source configuration and rewrite rules become part of the immutable ledger configuration. Restrict access to ledger configuration and audit data accordingly. Operator-managed mirror creation is not supported in the Stack v4 preview. Use the current `ledgerctl` or HTTP workflow only when direct management is approved for the target. ## Create a mirror with `ledgerctl` Set the values for this `ledgerctl` session. Configure the target Ledger endpoint and authentication through your normal `ledgerctl` profile. When the target has `require_signatures=true`, configure that profile with a registered Ed25519 key or add `--signing-key --signing-key-id ` before `ledgers create`. `ledgerctl` signs the complete gRPC Apply batch. The target still requires the caller's normal authorization scopes. The examples below assume the active profile supplies these values when mandatory signing is enabled; see [Request and Response Signing](/modules/ledger/operations/signing). The current `ledgerctl ledgers create` command does not attach an idempotency key. Do not put it in a retry loop. If the command times out or loses its response, read the exact target ledger and resolve the outcome before deciding whether another creation attempt is safe. ```bash ``` For an HTTP source without OAuth2: ```bash ledgerctl ledgers create \ --name "$TARGET_LEDGER" \ --mode mirror \ --mirror-source-type http \ --mirror-ledger-name "$SOURCE_LEDGER" \ --mirror-base-url "$V2_URL" ``` For OAuth2, add all credential fields together: ```bash ledgerctl ledgers create \ --name "$TARGET_LEDGER" \ --mode mirror \ --mirror-source-type http \ --mirror-ledger-name "$SOURCE_LEDGER" \ --mirror-base-url "$V2_URL" \ --mirror-oauth2-client-id "$V2_CLIENT_ID" \ --mirror-oauth2-client-secret "$V2_CLIENT_SECRET" \ --mirror-oauth2-token-endpoint "$V2_TOKEN_ENDPOINT" \ --mirror-oauth2-scopes "$V2_SCOPE" ``` For PostgreSQL password authentication, use an ephemeral environment variable to keep the URI DSN out of shell history: ```bash ledgerctl ledgers create \ --name "$TARGET_LEDGER" \ --mode mirror \ --mirror-source-type postgres \ --mirror-ledger-name "$SOURCE_LEDGER" \ --mirror-dsn "$V2_MIRROR_DSN" ``` The shell still expands secrets and DSNs into command-line arguments. Run `ledgerctl` from a restricted administration environment, prevent untrusted local process inspection, and unset the variables afterward. The source database user needs permission to connect to the database, resolve the source ledger's bucket, and read that bucket's logs. Ask the source database administrator to replace the database, role, and bucket placeholders before applying grants: ```sql GRANT CONNECT ON DATABASE ledger TO ledger_mirror; GRANT USAGE ON SCHEMA _system, "" TO ledger_mirror; GRANT SELECT ON TABLE _system.ledgers, "".logs TO ledger_mirror; ``` Ledger reads `_system.ledgers` to find `` for `SOURCE_LEDGER`, then queries that schema's `logs` table. A table-level `SELECT` grant without database `CONNECT` and schema `USAGE` is insufficient. For RDS IAM, the DSN must use URI form, contain no password, and declare an explicit TLS mode: ```bash ledgerctl ledgers create \ --name "$TARGET_LEDGER" \ --mode mirror \ --mirror-source-type postgres \ --mirror-ledger-name "$SOURCE_LEDGER" \ --mirror-dsn "$V2_MIRROR_DSN" \ --mirror-aws-iam-region "$V2_AWS_REGION" ``` The target Ledger runtime identity needs `rds-db:connect` permission for the exact RDS database-user ARN. Add `--mirror-aws-iam-assume-role-arn ` when that identity must assume another role; the base identity then needs `sts:AssumeRole`, and the assumed role needs `rds-db:connect`. ### Resolve an ambiguous `ledgerctl` result After a timeout, connection loss, or unknown CLI result, retry only the read: ```bash set -euo pipefail : "${TARGET_LEDGER:?set TARGET_LEDGER}" command -v ledgerctl >/dev/null LEDGER_JSON="$(ledgerctl ledgers get "$TARGET_LEDGER" --json)" ``` Then run the mode, source, progress, and read-only assertions in [Verify creation and read-only behavior](#verify-creation-and-read-only-behavior). Resolve the outcome as follows: | Read outcome | Decision | | --- | --- | | The exact target exists and every assertion matches the requested source and rules. | Creation committed. Continue without another create request. | | The target exists but its mode, source, credentials, batch size, or rules differ. | Stop. Treat it as a name collision or unexpected state; do not delete or overwrite it automatically. | | The read times out, is unavailable, or returns an unclassified error. | State remains unknown. Retry the read, not the create request. | | The authoritative target returns an explicit `NotFound`. | The target is definitively absent. A new, approved create attempt is allowed. | The complete ledger response can contain the source OAuth2 secret or PostgreSQL DSN. Compare it only in the restricted administration environment, emit a Boolean result, and do not store or print the raw response. ## Create an HTTP mirror through the HTTP API The caller needs the effective `ledger:LedgerWrite` scope. With the default scope mapping, the virtual `ledger:write` scope includes it. The HTTP create handler submits an unsigned request. It is rejected when the target has `require_signatures=true`, regardless of the bearer token's scopes. Use `ledgerctl` with a registered signing key or a client that sends a signed gRPC `ApplyRequest`, as described in [Request and Response Signing](/modules/ledger/operations/signing). Do not disable mandatory signing to create the mirror. Build the body once, save it with restrictive permissions, and generate one stable idempotency key for this creation attempt. The body contains the source client secret, so keep these files only in the restricted administration environment and never add them to the migration record. ```bash set -euo pipefail umask 077 for value in \ TARGET_URL TARGET_TOKEN TARGET_LEDGER SOURCE_LEDGER \ V2_URL V2_CLIENT_ID V2_CLIENT_SECRET V2_TOKEN_ENDPOINT V2_SCOPE \ MIRROR_CREATE_TIMEOUT_SECONDS MIRROR_CREATE_BODY_FILE \ MIRROR_CREATE_KEY_FILE MIRROR_CREATE_URL_FILE \ MIRROR_CREATE_BODY_SHA256_FILE \ MIRROR_CREATE_RESPONSE_PREFIX; do test -n "${!value:-}" || { printf 'missing creation variable: %s\n' "$value" >&2 exit 1 } done [[ "$MIRROR_CREATE_TIMEOUT_SECONDS" =~ ^[1-9][0-9]*$ ]] || { printf 'MIRROR_CREATE_TIMEOUT_SECONDS must be a positive integer\n' >&2 exit 1 } command -v awk >/dev/null command -v curl >/dev/null command -v jq >/dev/null command -v openssl >/dev/null for file in \ "$MIRROR_CREATE_BODY_FILE" \ "$MIRROR_CREATE_KEY_FILE" \ "$MIRROR_CREATE_URL_FILE" \ "$MIRROR_CREATE_BODY_SHA256_FILE"; do test ! -e "$file" || { printf 'creation evidence already exists; do not overwrite it: %s\n' "$file" >&2 exit 1 } done jq -n \ --arg sourceLedger "$SOURCE_LEDGER" \ --arg sourceUrl "$V2_URL" \ --arg clientId "$V2_CLIENT_ID" \ --arg clientSecret "$V2_CLIENT_SECRET" \ --arg tokenEndpoint "$V2_TOKEN_ENDPOINT" \ --arg scope "$V2_SCOPE" \ '{ mode: "MIRROR", mirrorSource: { ledgerName: $sourceLedger, type: "http", baseUrl: $sourceUrl, oauth2ClientId: $clientId, oauth2ClientSecret: $clientSecret, oauth2TokenEndpoint: $tokenEndpoint, oauth2Scopes: [$scope], batchSize: 100 } }' > "$MIRROR_CREATE_BODY_FILE" MIRROR_CREATE_KEY="mirror-create-$(openssl rand -hex 16)" test "${#MIRROR_CREATE_KEY}" -le 256 printf '%s\n' "$MIRROR_CREATE_KEY" > "$MIRROR_CREATE_KEY_FILE" printf '%s\n' "$TARGET_URL/v3/$TARGET_LEDGER" > "$MIRROR_CREATE_URL_FILE" openssl dgst -sha256 -r "$MIRROR_CREATE_BODY_FILE" \ > "$MIRROR_CREATE_BODY_SHA256_FILE" ``` Submit the saved bytes. This block can be repeated after an ambiguous result because it reloads the same key and refuses to continue if the body hash changed: ```bash set -euo pipefail umask 077 : "${TARGET_URL:?set TARGET_URL}" : "${TARGET_TOKEN:?set TARGET_TOKEN}" : "${TARGET_LEDGER:?set TARGET_LEDGER}" : "${MIRROR_CREATE_TIMEOUT_SECONDS:?set MIRROR_CREATE_TIMEOUT_SECONDS}" : "${MIRROR_CREATE_BODY_FILE:?set MIRROR_CREATE_BODY_FILE}" : "${MIRROR_CREATE_KEY_FILE:?set MIRROR_CREATE_KEY_FILE}" : "${MIRROR_CREATE_URL_FILE:?set MIRROR_CREATE_URL_FILE}" : "${MIRROR_CREATE_BODY_SHA256_FILE:?set MIRROR_CREATE_BODY_SHA256_FILE}" : "${MIRROR_CREATE_RESPONSE_PREFIX:?set MIRROR_CREATE_RESPONSE_PREFIX}" command -v awk >/dev/null command -v curl >/dev/null command -v date >/dev/null command -v openssl >/dev/null command -v tr >/dev/null test -f "$MIRROR_CREATE_BODY_FILE" test -f "$MIRROR_CREATE_KEY_FILE" test -f "$MIRROR_CREATE_URL_FILE" test -f "$MIRROR_CREATE_BODY_SHA256_FILE" MIRROR_CREATE_KEY="$(tr -d '\n' < "$MIRROR_CREATE_KEY_FILE")" : "${MIRROR_CREATE_KEY:?saved idempotency key is empty}" test "${#MIRROR_CREATE_KEY}" -le 256 MIRROR_CREATE_URL="$(tr -d '\n' < "$MIRROR_CREATE_URL_FILE")" : "${MIRROR_CREATE_URL:?saved request URL is empty}" test "$MIRROR_CREATE_URL" = "$TARGET_URL/v3/$TARGET_LEDGER" EXPECTED_BODY_SHA256="$(awk '{print $1}' "$MIRROR_CREATE_BODY_SHA256_FILE")" ACTUAL_BODY_SHA256="$(openssl dgst -sha256 -r "$MIRROR_CREATE_BODY_FILE" | awk '{print $1}')" test "$ACTUAL_BODY_SHA256" = "$EXPECTED_BODY_SHA256" MIRROR_CREATE_RESPONSE_FILE="${MIRROR_CREATE_RESPONSE_PREFIX}.$(date -u +%Y%m%dT%H%M%SZ)-$$-$RANDOM.body" test ! -e "$MIRROR_CREATE_RESPONSE_FILE" set +e MIRROR_CREATE_STATUS="$(curl --silent --show-error \ --max-time "$MIRROR_CREATE_TIMEOUT_SECONDS" \ --output "$MIRROR_CREATE_RESPONSE_FILE" \ --write-out '%{http_code}' \ --request POST \ --header "Authorization: Bearer $TARGET_TOKEN" \ --header "Content-Type: application/json" \ --header "Idempotency-Key: $MIRROR_CREATE_KEY" \ --data-binary @"$MIRROR_CREATE_BODY_FILE" \ "$MIRROR_CREATE_URL")" MIRROR_CREATE_EXIT="$?" set -e printf 'curlExit=%s httpStatus=%s bodySha256=%s\n' \ "$MIRROR_CREATE_EXIT" \ "$MIRROR_CREATE_STATUS" \ "$ACTUAL_BODY_SHA256" printf 'curlExit=%s httpStatus=%s bodySha256=%s\n' \ "$MIRROR_CREATE_EXIT" \ "$MIRROR_CREATE_STATUS" \ "$ACTUAL_BODY_SHA256" \ > "$MIRROR_CREATE_RESPONSE_FILE.status" ``` A `201` with curl exit `0` is a successful response. For any timeout, connection loss, or unexpected status, read and verify the target before deciding what happened. If a retry is required, repeat only the second block with the saved URL, body, and key, within the server's configured idempotency-retention window. Keep the destination cluster, method, target URL, content type, idempotency key, and body bytes unchanged. The URL check catches changed environment variables; independently confirm that DNS and routing still select the same target cluster. A short-lived bearer token may be refreshed under the same approved caller identity. `IDEMPOTENCY_KEY_CONFLICT` means the key was reused with different request content; stop and investigate. After the outcome is resolved, delete the request body and response files because both can contain the source secret. Keep only the request URL, non-secret body hash, idempotency key, final status, and filtered verification evidence. For PostgreSQL, set `type` to `postgres` and provide `dsn`. The HTTP request does not expose RDS IAM configuration; use `ledgerctl` or the gRPC contract for that authentication method. The create body above is flat: `type` selects the source, and the source's own fields (`baseUrl`, `oauth2ClientId`, `dsn`) sit beside it. A read of the ledger returns the stored configuration in a nested form instead — `mirrorSource.http.baseUrl` with credentials under `mirrorSource.http.oauth2ClientCredentials`, or `mirrorSource.postgres.dsn` with `mirrorSource.postgres.awsIamAuth`. The response carries **no `type` field at all**: which sub-object is present is what identifies the source. So a verification cannot be written by mirroring the request body, and it must not look for `mirrorSource.type`. The verification block below reads the nested form for that reason. ## Optional creation settings - `batchSize` defaults to 100. The server also enforces a maximum batch size. Tune it only from rehearsal measurements. - Add `rewriteRules` only when source data must produce a deliberately different target projection. See [Transform Mirrored Data](/modules/ledger/migration/transformations). - Create and verify the indexes required by validation before switching production readers. ## Verify creation and read-only behavior Use the same target ledger variable as the creation command. With `ledgerctl`, fail unless the ledger is in mirror mode and exposes a valid progress object: ```bash LEDGER_JSON="$(ledgerctl ledgers get "$TARGET_LEDGER" --json)" jq -e \ --arg target "$TARGET_LEDGER" \ ' def enum_tail: split("_")[-1]; def uint64: ((. // 0) | tostring) | test("^[0-9]+$"); . as $ledger | $ledger.mirrorSyncProgress as $progress | ($ledger.name == $target) and (($ledger.mode | enum_tail) == "MIRROR") and ($progress != null) and (((($progress.state // "MIRROR_SYNC_STATE_SYNCING") | enum_tail) == "SYNCING") or ((($progress.state // "MIRROR_SYNC_STATE_SYNCING") | enum_tail) == "FOLLOWING")) and ($progress.cursor | uint64) and ($progress.sourceLogCount | uint64) and ($progress.remainingLogs | uint64) ' <<<"$LEDGER_JSON" ``` For HTTP, define this verification surface even when you created the mirror with `ledgerctl`: ```bash LEDGER_JSON="$(curl --fail --silent --show-error \ --header "Authorization: Bearer $TARGET_TOKEN" \ "$TARGET_URL/v3/$TARGET_LEDGER")" jq -e \ --arg target "$TARGET_LEDGER" \ ' def enum_tail: split("_")[-1]; def uint64: ((. // 0) | tostring) | test("^[0-9]+$"); .data as $ledger | $ledger.mirrorSyncProgress as $progress | ($ledger.name == $target) and (($ledger.mode | enum_tail) == "MIRROR") and ($progress != null) and (((($progress.state // "MIRROR_SYNC_STATE_SYNCING") | enum_tail) == "SYNCING") or ((($progress.state // "MIRROR_SYNC_STATE_SYNCING") | enum_tail) == "FOLLOWING")) and ($progress.cursor | uint64) and ($progress.sourceLogCount | uint64) and ($progress.remainingLogs | uint64) ' <<<"$LEDGER_JSON" ``` Both surfaces this page uses carry the full protobuf spelling: `mode` reads `LEDGER_MODE_MIRROR` and `state` reads `MIRROR_SYNC_STATE_FOLLOWING`. There is no short spelling to accept — the enum checks below tolerate one defensively, but do not write new automation against `MIRROR` or `FOLLOWING`, because nothing emits them. Fixed-width progress values arrive as JSON strings for the same reason, so the checks validate either representation as an unsigned integer rather than asserting a JSON number. The encoding also omits a field sitting at its default, which covers the zero-value syncing state and zero-valued progress counters; only those omissions are interpreted as protobuf defaults. Verify the immutable source configuration against the values used for this attempt. Normalize either the top-level `ledgerctl` record or the HTTP `{data: ...}` envelope without printing it. Select one source type, set the stored batch size (`0` for the CLI default or `100` in the HTTP example above), and require no rewrite rules on this page. `EXPECTED_SOURCE_TYPE` is this script's own selector, naming the four combinations it knows how to assert — it is not an API value, and it does not correspond to the request's `type` or to `--mirror-source-type`, both of which accept only `http` or `postgres`: ```bash set -euo pipefail : "${LEDGER_JSON:?read the target ledger into LEDGER_JSON first}" : "${TARGET_LEDGER:?set TARGET_LEDGER}" : "${SOURCE_LEDGER:?set SOURCE_LEDGER}" [[ "$EXPECTED_BATCH_SIZE" =~ ^[0-9]+$ ]] [[ "$EXPECTED_RULE_COUNT" =~ ^[0-9]+$ ]] command -v jq >/dev/null case "$EXPECTED_SOURCE_TYPE" in http) : "${V2_URL:?set V2_URL}" ;; http-oauth) : "${V2_URL:?set V2_URL}" : "${V2_CLIENT_ID:?set V2_CLIENT_ID}" : "${V2_CLIENT_SECRET:?set V2_CLIENT_SECRET}" : "${V2_TOKEN_ENDPOINT:?set V2_TOKEN_ENDPOINT}" : "${V2_SCOPE:?set V2_SCOPE}" ;; postgres-password) : "${V2_MIRROR_DSN:?set V2_MIRROR_DSN}" ;; postgres-rds-iam) : "${V2_MIRROR_DSN:?set V2_MIRROR_DSN}" : "${V2_AWS_REGION:?set V2_AWS_REGION}" ;; *) printf 'unsupported EXPECTED_SOURCE_TYPE: %s\n' "$EXPECTED_SOURCE_TYPE" >&2 exit 1 ;; esac LEDGER_RECORD_JSON="$(jq -ce ' if (.data? | type) == "object" then .data else . end ' <<<"$LEDGER_JSON")" jq -e \ --arg sourceType "$EXPECTED_SOURCE_TYPE" \ --arg batchSize "$EXPECTED_BATCH_SIZE" \ --arg ruleCount "$EXPECTED_RULE_COUNT" \ ' def enum_tail: split("_")[-1]; (.name == $ENV.TARGET_LEDGER) and ((.mode | enum_tail) == "MIRROR") and (.mirrorSyncProgress != null) and (.mirrorSource.ledgerName == $ENV.SOURCE_LEDGER) and (((.mirrorSource.batchSize // 0) | tostring) == $batchSize) and (((.mirrorSource.rewriteRules // []) | length) == ($ruleCount | tonumber)) and ( if $sourceType == "http" then (.mirrorSource.http.baseUrl == $ENV.V2_URL) and (.mirrorSource.http.oauth2ClientCredentials == null) and (.mirrorSource.postgres == null) elif $sourceType == "http-oauth" then (.mirrorSource.http.baseUrl == $ENV.V2_URL) and (.mirrorSource.http.oauth2ClientCredentials.clientId == $ENV.V2_CLIENT_ID) and (.mirrorSource.http.oauth2ClientCredentials.clientSecret == $ENV.V2_CLIENT_SECRET) and (.mirrorSource.http.oauth2ClientCredentials.tokenEndpoint == $ENV.V2_TOKEN_ENDPOINT) and (.mirrorSource.http.oauth2ClientCredentials.scopes == [$ENV.V2_SCOPE]) and (.mirrorSource.postgres == null) elif $sourceType == "postgres-password" then (.mirrorSource.postgres.dsn == $ENV.V2_MIRROR_DSN) and (.mirrorSource.postgres.awsIamAuth == null) and (.mirrorSource.http == null) elif $sourceType == "postgres-rds-iam" then (.mirrorSource.postgres.dsn == $ENV.V2_MIRROR_DSN) and (.mirrorSource.postgres.awsIamAuth.region == $ENV.V2_AWS_REGION) and ((.mirrorSource.postgres.awsIamAuth.assumeRoleArn // "") == $ENV.V2_AWS_ASSUME_ROLE_ARN) and (.mirrorSource.http == null) else false end ) ' <<<"$LEDGER_RECORD_JSON" >/dev/null ``` Every comparison is exact, including the source secret, but `jq` emits only its exit status. For multiple OAuth2 scopes, replace `[$ENV.V2_SCOPE]` with the exact ordered array supplied at creation. A transformed mirror must additionally compare the complete ordered rule list and accepted rule hash as described on [Transform Mirrored Data](/modules/ledger/migration/transformations). Finally, prove that an application write is rejected with the exact `LEDGER_IN_MIRROR_MODE` reason. The probe has to submit a structurally valid order. Ledger rejects a malformed one during admission, before the mirror-mode gate in the state machine runs, so an intentionally invalid body returns a content error on a mirror and on a normal ledger alike and proves nothing. Only a well-formed order reaches the gate. Use a metadata deletion for a key that does not exist. Deleting account metadata is not on the mirror-safe list, so on a mirror it reaches the gate and is refused with `LEDGER_IN_MIRROR_MODE`. On a normal ledger the same request fails its existence check with `METADATA_NOT_FOUND` before anything is written — no log, no account, no balance change. The probe therefore cannot commit value even if the target turns out to be writable, which a transaction probe from `world` would. Re-read the ledger record immediately before the probe and refuse to run unless the target is still a mirror: ```bash PROBE_LEDGER_JSON="$( curl --fail-with-body --silent --show-error \ --header "Authorization: Bearer $TARGET_TOKEN" \ "$TARGET_URL/v3/$TARGET_LEDGER" )" jq -e ' (if (.data? | type) == "object" then .data else . end) | .mode == "LEDGER_MODE_MIRROR" ' <<<"$PROBE_LEDGER_JSON" >/dev/null ``` `LEDGER_MODE_NORMAL` is the zero value of the enum, so a normal ledger omits `mode` entirely and this check fails closed. Read it here rather than reusing the record fetched for the configuration comparison above, so the value cannot be stale by the time the probe runs. The probe needs the effective `ledger:MetadataWrite` scope — the metadata routes sit behind a different scope from the transaction routes. ```bash PROBE_ACCOUNT="migration:read-only-probe" PROBE_KEY="mirror-read-only-probe-$(date +%s)-${RANDOM}" READ_ONLY_BODY_FILE="$(mktemp)" trap 'rm -f "$READ_ONLY_BODY_FILE"' EXIT HTTP_STATUS="$( curl --silent --show-error \ --output "$READ_ONLY_BODY_FILE" \ --write-out '%{http_code}' \ --request DELETE \ --header "Authorization: Bearer $TARGET_TOKEN" \ "$TARGET_URL/v3/$TARGET_LEDGER/accounts/$PROBE_ACCOUNT/metadata/$PROBE_KEY" )" test "$HTTP_STATUS" = "409" jq -e '.errorCode == "LEDGER_IN_MIRROR_MODE"' "$READ_ONLY_BODY_FILE" ``` Both assertions must pass. `PROBE_KEY` carries a timestamp and a random suffix so it is a key no writer has set; the probe proves admission is shut, not that the key is absent. A `400` with `METADATA_NOT_FOUND` means the request passed the gate and reached the existence check — the target is accepting writes and is no longer a mirror. Treat that as a stop condition: preserve the response, block the migration, and establish the target's mode before continuing. Nothing was written, so there is no probe state to clean up. ## Change or rotate the configuration The source, credentials, batch size, and rewrite rules are immutable after creation. Changing a value in the deployment's secret manager does not rewrite the credential already stored in the mirror configuration. To change any of those values: 1. keep Ledger v2 authoritative; 2. save the current evidence and rule file; 3. delete the target mirror; 4. update the secret-manager value or source configuration; 5. choose a new target ledger name; 6. create the replacement mirror under that new name; 7. update staged client, validation, and cutover configuration to the replacement name; 8. replay and validate from the beginning. Ledger retains a tombstone after deletion, so the deleted target ledger name cannot be reused. Ledger also has no supported update operation for the source or rewrite configuration. Changing it requires a replacement mirror with a new target name. Promotion is the only supported transition from a mirror to a normal ledger. Continue with [Transform Mirrored Data](/modules/ledger/migration/transformations) when the target model must differ from Ledger v2. Otherwise, continue with [Monitor the Mirror](/modules/ledger/migration/monitor-mirror). --- ## Account Types and Enforcement Source: https://docs.formance.com/modules/ledger/modeling/account-types Account types turn a [chart of accounts](/modules/ledger/modeling/chart-of-accounts) into rules Ledger can enforce. Each type combines a name, an address pattern, optional variable constraints, and a volume persistence mode. Use them to reject address typos before those typos create new balance buckets. ## Before you start The STRICT rollout commands below change account-type configuration and create two probe transactions: one write to an address that must match an active type, and one write to an address that must match no type at all. Together they prove that enforcement accepts what it should and rejects what it should. Neither is meant to leave value behind: the first is reverted as soon as it succeeds, and the second is expected to be rejected. Introduce account types in `AUDIT` mode first: adding the first type while the default `STRICT` mode is active can immediately reject every address that the type does not cover. Run these commands against a dedicated ledger that contains a representative copy of your address patterns and transaction flows, and do not point `LEDGER_NAME` at production until you have reviewed each write. This rollout uses `ledgerctl query-checkpoint create` and `delete`. If mandatory request signing is enabled, both commands fail: in the current 3.0 preview they submit unsigned mutations. Complete this rollout and its checkpoint cleanup before you enable that policy, or wait for a release with a tested signed checkpoint-management path. This rollout also needs `ledgerctl`, and targets an existing ledger rather than a disposable one: ```bash ``` The token needs these granular scopes: | Operation | Scope | | --- | --- | | List accounts and account types | `ledger:AccountRead` | | List historical transactions | `ledger:TransactionRead` | | Read the active enforcement mode | `ledger:LedgerRead` | | Add or remove account types, or change enforcement, through HTTP | `ledger:MetadataWrite` | | Add or remove account types, or change enforcement, through the current `ledgerctl` gRPC commands | `ledger:OpsWrite` | | Create and revert probe transactions | `ledger:TransactionWrite` | | Create, inspect, and delete the inventory checkpoint | `ledger:ClusterWrite`, `ledger:ClusterRead` | Your deployment may map the virtual `ledger:read` and `ledger:write` scopes to some of these permissions. Confirm the effective mapping in [Authentication and Authorization](/modules/ledger/operations/authentication#understand-scopes). Configure the gRPC endpoint and TLS settings as described in [Choose Read Consistency and Use Checkpoints](/modules/ledger/querying/consistency-and-checkpoints). See [Read Ledger Data](/modules/ledger/querying/read-ledger-data) for the read routes and [Create Transactions with Postings](/modules/ledger/transactions/create-with-postings) for the write contract. ## Define an address pattern A pattern has colon-separated fixed and variable segments: ```text customers:{customerId}:wallets:{wallet} merchants:{merchantId}:earnings platform:fees ``` Variables use `{name}` and match one non-empty segment. A pattern must match the full address, not only its prefix. You can constrain a variable with one of these segment types: | Constraint | Accepted value | | --- | --- | | `regex` | A value matching the supplied regular expression | | `uuid` | An RFC 4122 UUID string | | `uint64` | A non-empty decimal unsigned integer | | `bytes` | An even-length hexadecimal string | Add a type through the HTTP API: ```bash curl --fail-with-body -X POST \ "${CURL_AUTH[@]}" \ -H "Content-Type: application/json" \ -d '{ "name": "customer-wallet", "pattern": "customers:{customerId}:wallets:{wallet}", "persistence": "NORMAL", "segmentTypes": { "customerId": { "type": "uuid" }, "wallet": { "type": "regex", "regex": "main|rewards" } } }' \ "${LEDGER_URL}/v3/${LEDGER_NAME}/account-types" ``` The CLI supports account types without segment constraints: ```bash ledgerctl account-types add merchant-earnings \ 'merchants:{merchantId}:earnings' \ --ledger "${LEDGER_NAME}" \ --persistence normal ``` Do not put a regular expression inside `{...}`. The pattern contains only the variable name; HTTP `segmentTypes` carries its constraint. ## Understand overlapping patterns When several types can match an address, Ledger selects the pattern with the greatest number of fixed segments. This permits a general pattern and a more specific exception: ```text wallets:{id} wallets:treasury ``` Ledger rejects patterns that can match the same address with the same specificity. For example, `users:{id}` and `users:{name}` conflict because neither pattern is more specific. The built-in `world` account is always accepted and does not require an account type. ## Choose a persistence mode Every account type also carries a persistence mode. It controls whether Ledger keeps a stored volume row for the accounts that match the type, and whether those accounts may still hold value once a batch ends. A volume row is the per-asset, per-color `input`/`output` pair that account reads report; no mode affects transaction history. | Persistence | Behavior | Use it when | | --- | --- | --- | | `NORMAL` | Keep cumulative volumes, including after the balance returns to zero. | Direct account reads must retain input and output totals. | | `EPHEMERAL` | Keep a non-zero volume and remove its row when input equals output at commit. | The account only needs a stored volume while it holds value. | | `TRANSIENT` | Do not keep a volume row and require the account to finish the atomic Apply batch at zero. | The account is an intermediary that must not retain value beyond one atomic batch. | `EPHEMERAL` removes the zero-balance volume row, not transaction history. `TRANSIENT` checks the final net balance across the entire atomic Apply batch (the proposal), not after each transaction request inside that batch. A regular create-transaction request is a one-request batch, so the account must still finish that transaction at zero. In a multi-request atomic batch, a later request can return it to zero before the batch ends. Every asset and color touched on the intermediary must end at zero. An account read assembles its `volumes` from the stored rows and from nothing else, which is what makes the mode visible to callers. Under `NORMAL`, a drained account still reports its cumulative `input` and `output` alongside a zero `balance`. Under `EPHEMERAL`, the purged asset and color is absent from the response once input equals output, so those totals are no longer readable even though the transactions that produced them remain in history. Under `TRANSIENT`, the account never reports volumes at all. The address itself still resolves in every mode: a direct read of an account with no remaining rows returns its metadata and no `volumes`. Do not choose `EPHEMERAL` or `TRANSIENT` only to reduce storage. First confirm that direct reads, indexes, reconciliations, and recovery procedures do not depend on the volume row. ## Move from AUDIT to STRICT The ledger's default enforcement mode controls addresses that match no type: | Mode | Behavior | | --- | --- | | `AUDIT` | Accept unmatched addresses. | | `STRICT` | Reject unmatched addresses with `ACCOUNT_NOT_MATCHING_TYPE`. | If no account types exist, address-type enforcement is inactive. As soon as the first type exists, the default mode applies to every unmatched non-`world` address. `AUDIT` accepts an unmatched address without returning a warning list. Ledger also has no endpoint that reports which type matched an existing address. Treat `AUDIT` as a permissive mode while you build and review your own coverage report. ### 1. Set AUDIT and prove the active mode Set `AUDIT` before adding the first type, then read the ledger configuration independently: ```bash set_enforcement_mode() { curl --fail-with-body --silent --show-error \ --request PUT \ "${CURL_AUTH[@]}" \ --header "Content-Type: application/json" \ --data "{\"enforcementMode\":\"$1\"}" \ "$LEDGER_URL/v3/$LEDGER_NAME/account-types/default-enforcement-mode" } read_enforcement_mode() { curl --fail-with-body --silent --show-error \ "${CURL_AUTH[@]}" \ "$LEDGER_URL/v3/$LEDGER_NAME" \ | jq -er '.data.defaultEnforcementMode' } set_enforcement_mode AUDIT test "$(read_enforcement_mode)" = "CHART_ENFORCEMENT_AUDIT" ``` Add the required types while this check still passes. List them and keep the response as STRICT rollout evidence: ```bash curl --fail-with-body --silent --show-error \ "${CURL_AUTH[@]}" \ "$LEDGER_URL/v3/$LEDGER_NAME/account-types" \ > /tmp/ledger-account-types.json jq -e '.data.types | length > 0' /tmp/ledger-account-types.json ``` ### 2. Inventory addresses that must remain valid For the final production check, pause application writers before creating the inventory and keep them paused until the STRICT probes pass or you return to `AUDIT`. Without that pause, a writer can introduce a new address after the inventory and before the mode switch. Create a query checkpoint so the account and transaction exports come from the same committed dataset: ```bash CHECKPOINT_JSON="$(ledgerctl query-checkpoint create --json)" CHECKPOINT_ID="$(jq -er '.checkpointId | tostring' <<<"$CHECKPOINT_JSON")" cleanup_checkpoint() { if [ -n "${CHECKPOINT_ID:-}" ]; then ledgerctl query-checkpoint delete "$CHECKPOINT_ID" >/dev/null 2>&1 || true fi } trap cleanup_checkpoint EXIT INT TERM ledgerctl query-checkpoint info "$CHECKPOINT_ID" --json \ | jq -e --arg id "$CHECKPOINT_ID" \ '(.checkpointId | tostring) == $id' ``` A checkpoint can take a short time to become readable on the contacted replica. Retry only `CHECKPOINT_NOT_READY`, then export every account and transaction: ```bash CHECKPOINT_READY=false for attempt in $(seq 1 30); do if ledgerctl accounts list \ --ledger "$LEDGER_NAME" \ --all \ --checkpoint-id "$CHECKPOINT_ID" \ --json \ > /tmp/ledger-accounts.json \ 2> /tmp/ledger-checkpoint-read.err then CHECKPOINT_READY=true break fi if ! grep -q 'CHECKPOINT_NOT_READY' /tmp/ledger-checkpoint-read.err; then cat /tmp/ledger-checkpoint-read.err >&2 exit 1 fi sleep 2 done test "$CHECKPOINT_READY" = true TRANSACTIONS_READY=false for attempt in $(seq 1 30); do if ledgerctl transactions list \ --ledger "$LEDGER_NAME" \ --all \ --reverse \ --checkpoint-id "$CHECKPOINT_ID" \ --json \ > /tmp/ledger-transactions.json \ 2> /tmp/ledger-checkpoint-read.err then TRANSACTIONS_READY=true break fi if ! grep -q 'CHECKPOINT_NOT_READY' /tmp/ledger-checkpoint-read.err; then cat /tmp/ledger-checkpoint-read.err >&2 exit 1 fi sleep 2 done test "$TRANSACTIONS_READY" = true { jq -r '.[].address | select(. != "world")' /tmp/ledger-accounts.json jq -r \ '.[].postings[]? | .source, .destination | select(. != "world")' \ /tmp/ledger-transactions.json } | sort -u > /tmp/ledger-account-addresses.txt wc -l /tmp/ledger-account-addresses.txt ``` The transaction export matters because `EPHEMERAL` and `TRANSIENT` accounts may not appear in current account results, while a later revert still validates their original addresses. Also add addresses from: - every transaction template and metadata writer that is still active; - bulk jobs, repair tools, imports, and scheduled workflows; - planned account shapes that will appear after switching to `STRICT`. The team that owns each writer should confirm the type expected for every address it can produce. Ledger does not produce this match report for you. Store the review result as JSON so switching to `STRICT` has a concrete gate: ```json [ { "address": "customers:8bfcf61d-14ea-4e41-bb11-f064b8949758:wallets:main", "matchedType": "customer-wallet", "source": "current-account" } ] ``` Save the complete report as `/tmp/account-type-coverage.json`. The following gate checks that every current account appears in the report and that every reviewed row names a type: ```bash jq -e ' length > 0 and all(.[]; ((.address | type) == "string" and (.address | length) > 0) and ((.matchedType | type) == "string" and (.matchedType | length) > 0) and ((.source | type) == "string" and (.source | length) > 0) ) ' /tmp/account-type-coverage.json MISSING_CURRENT="$( comm -23 \ /tmp/ledger-account-addresses.txt \ <(jq -r '.[].address' /tmp/account-type-coverage.json | sort -u) )" test -z "$MISSING_CURRENT" UNKNOWN_TYPES="$( comm -23 \ <(jq -r '.[].matchedType' /tmp/account-type-coverage.json | sort -u) \ <(jq -r '.data.types[].name' /tmp/ledger-account-types.json | sort -u) )" test -z "$UNKNOWN_TYPES" ``` This gate proves that the review is complete and refers only to active type names. The application owner must still confirm that each address matches the named type's full pattern and segment constraints; Ledger does not expose that check through HTTP or `ledgerctl`. ### 3. Switch to STRICT and run positive and negative probes Switch to `STRICT` only after the inventory gate passes: ```bash set_enforcement_mode STRICT test "$(read_enforcement_mode)" = "CHART_ENFORCEMENT_STRICT" ``` Choose a `NORMAL` or `EPHEMERAL` address that must match an active type. Generate a fresh run identifier every time you repeat the probes: references are immutable and idempotency outcomes are retained, so reusing either value could return an earlier result instead of testing the current rules. The positive probe deposits one unit and immediately reverts it: ```bash RUN_ID="$(date -u +%Y%m%dT%H%M%SZ)-${BASHPID:-$$}-${RANDOM}" POSITIVE_STATUS="$(curl --silent --show-error \ --output /tmp/account-type-positive.json \ --write-out '%{http_code}' \ --request POST "$LEDGER_URL/v3/$LEDGER_NAME/transactions" \ "${CURL_AUTH[@]}" \ --header "Content-Type: application/json" \ --header "Idempotency-Key: account-type-positive:$RUN_ID:create" \ --data "$(jq -n \ --arg destination "$VALID_ACCOUNT" \ --arg reference "account-type-positive:$RUN_ID" '{ reference: $reference, postings: [{ source: "world", destination: $destination, asset: "USD/2", amount: 1 }] }')")" test "$POSITIVE_STATUS" = "201" POSITIVE_TRANSACTION_ID="$(jq -er '.data.transaction.id' /tmp/account-type-positive.json)" REVERT_STATUS="$(curl --silent --show-error \ --output /tmp/account-type-positive-revert.json \ --write-out '%{http_code}' \ --request POST \ "${CURL_AUTH[@]}" \ --header "Content-Type: application/json" \ --header "Idempotency-Key: account-type-positive:$RUN_ID:revert" \ --data '{"metadata":{"reason":"account_type_validation"}}' \ "$LEDGER_URL/v3/$LEDGER_NAME/transactions/$POSITIVE_TRANSACTION_ID/revert")" test "$REVERT_STATUS" = "201" ``` The deposit exists between those two requests, and nothing reverts it automatically. If the run stops before the revert returns `201`, take the transaction ID from `/tmp/account-type-positive.json` and revert it by hand before you repeat the probes. Next, choose an address that you have confirmed cannot match any active pattern. The rejected request must not create a transaction: ```bash NEGATIVE_STATUS="$(curl --silent --show-error \ --output /tmp/account-type-negative.json \ --write-out '%{http_code}' \ --request POST "$LEDGER_URL/v3/$LEDGER_NAME/transactions" \ "${CURL_AUTH[@]}" \ --header "Content-Type: application/json" \ --header "Idempotency-Key: account-type-negative:$RUN_ID:create" \ --data "$(jq -n \ --arg destination "$INVALID_ACCOUNT" \ --arg reference "account-type-negative:$RUN_ID" '{ reference: $reference, postings: [{ source: "world", destination: $destination, asset: "USD/2", amount: 1 }] }')")" if [ "$NEGATIVE_STATUS" = "201" ]; then UNEXPECTED_ID="$(jq -r '.data.transaction.id // empty' /tmp/account-type-negative.json)" if [ -z "$UNEXPECTED_ID" ]; then printf 'negative probe committed a transaction; read its id from %s and revert it by hand\n' \ /tmp/account-type-negative.json >&2 else printf 'negative probe committed transaction %s; reverting\n' "$UNEXPECTED_ID" >&2 CLEANUP_STATUS="$(curl --silent --show-error \ --output /tmp/account-type-negative-revert.json \ --write-out '%{http_code}' \ --request POST \ "${CURL_AUTH[@]}" \ --header "Content-Type: application/json" \ --header "Idempotency-Key: account-type-negative:$RUN_ID:revert" \ --data '{"metadata":{"reason":"unexpected_negative_probe"}}' \ "$LEDGER_URL/v3/$LEDGER_NAME/transactions/$UNEXPECTED_ID/revert" || true)" if [ "$CLEANUP_STATUS" = "201" ]; then printf 'reverted transaction %s\n' "$UNEXPECTED_ID" >&2 else printf 'REVERT FAILED (status %s) for transaction %s; revert it by hand before repeating the rollout\n' \ "$CLEANUP_STATUS" "$UNEXPECTED_ID" >&2 fi fi fi test "$NEGATIVE_STATUS" = "400" jq -e --arg account "$INVALID_ACCOUNT" ' .errorCode == "ACCOUNT_NOT_MATCHING_TYPE" and .errorMessage == ("account does not match any account type pattern: " + $account) ' \ /tmp/account-type-negative.json ``` The `201` branch runs before the assertion on purpose. `set -eo pipefail` is active from the runnable setup, so a bare failing assertion would stop the shell with the probe's unit still in the ledger and no chance to act on remediation further down the page. The revert captures its own HTTP status and asserts `201`, the same way the positive probe does, because `--fail-with-body` treats every status below `400` as success — a redirect or a `200` would otherwise be reported as a revert that never happened. Neither the request nor a missing ID can stop the shell, so the transaction ID reaches stderr whichever way it goes. If your chart intentionally has a pattern that matches every possible address, there is no negative address to test. Record that decision explicitly: `STRICT` cannot reject an unmatched address when every address matches by design. After the required probes pass, delete the captured checkpoint and verify that its ID is absent. Resume application writers only after this check: ```bash ledgerctl query-checkpoint delete "$CHECKPOINT_ID" ledgerctl query-checkpoint list --json \ | jq -e --arg id "$CHECKPOINT_ID" \ 'all(.[]; (.checkpointId | tostring) != $id)' CHECKPOINT_ID="" trap - EXIT INT TERM ``` ### 4. Roll back admission when a canary fails If an expected application write starts returning `ACCOUNT_NOT_MATCHING_TYPE`, restore permissive admission before changing any type: ```bash set_enforcement_mode AUDIT test "$(read_enforcement_mode)" = "CHART_ENFORCEMENT_AUDIT" ``` Changing the mode affects future writes only. It does not alter committed transactions, and a request rejected in `STRICT` made no accounting change. Add the missing coverage, repeat the inventory gate on the same dataset, then switch to `STRICT` again. Ledger remembers a final failure under its idempotency key. After correcting coverage, retry a rejected business operation with a new idempotency key; reusing the rejected key returns the stored failure even in `AUDIT`. If the negative probe returned `201`, the block above reports the transaction ID and attempts the revert. Read its stderr output: `reverted transaction ` means the revert returned `201` and the ledger is clean, while `REVERT FAILED (status ...)` means that transaction is still committed and needs reverting by hand. Either way, correct the overly broad pattern or the probe address before repeating the rollout — a `201` there means an address you expected `STRICT` to reject is matching an active type. Before switching a production ledger to `STRICT`, also exercise standalone account-metadata saves, reversals, bulk operations, transient zero-balance checks, and ephemeral reads with representative application requests. A metadata deletion does not prove address matching, and `accountMetadata` attached to a transaction is not a replacement for the posting-address probes above. ## Verify the active rules List account types through HTTP: ```bash curl --fail-with-body \ "${CURL_AUTH[@]}" \ "${LEDGER_URL}/v3/${LEDGER_NAME}/account-types" ``` Or through the CLI: ```bash ledgerctl account-types list --ledger "${LEDGER_NAME}" ``` The HTTP response contains the stored pattern, persistence, and segment constraints: ```json { "data": { "types": [ { "name": "customer-wallet", "pattern": "customers:{customerId}:wallets:{wallet}", "persistence": "NORMAL", "segmentTypes": { "customerId": { "type": "uuid" }, "wallet": { "type": "regex", "regex": "main|rewards" } } } ] } } ``` Test both sides of every rule: one address that must match and one that must fail. For persistence modes, also test the zero-balance boundary. ## Change or remove a type Account types have add, get, list, and remove operations. There is no in-place update operation. To evolve a chart, add compatible new types before removing old coverage whenever possible. ```bash ledgerctl account-types remove legacy-wallet --ledger "${LEDGER_NAME}" ``` Removing a type does not rewrite or delete existing accounts and transactions. In `STRICT`, it can make future metadata changes or reversals fail when they refer to an address that no remaining type covers. ## Common failure modes | Symptom | Cause | Correction | | --- | --- | --- | | Most writes fail after the first type is added | Default enforcement was still `STRICT` and the chart is incomplete | Set `AUDIT`, add complete coverage, then repeat the STRICT rollout. | | Adding a second type returns a conflict | Two patterns overlap with equal specificity | Merge them or add a genuinely more specific fixed pattern. | | A valid-looking ID does not match | The segment constraint rejects its format | Test UUID, uint64, bytes, or regex values independently. | | A transient transaction fails at commit | An intermediary retains a balance for an asset or color | Add the missing outgoing posting or use `NORMAL` when value may remain. | | An ephemeral account disappears from account-volume reads at zero | Its zero-balance volume row was removed as configured | Use `NORMAL` when a stored zero-balance row is required. | | A historical reversal fails in `STRICT` | Its original address no longer matches any active type | Keep historical patterns covered while those operations remain possible. | | `AUDIT` returns no warning array | The mode is permissive but does not emit per-write violations | Validate addresses from representative traffic before switching to `STRICT`. | When funds from different origins must remain separate inside one matching account and asset, add [colors](/modules/ledger/modeling/color-of-money). Account types do not separate balances by themselves. --- ## TLS and Network Security Source: https://docs.formance.com/modules/ledger/operations/tls-and-network-security Use TLS for Ledger's service gRPC API and inter-node traffic. In a multi-node authenticated cluster, TLS also protects the shared cluster secret that followers use when they forward a caller's write to the leader. TLS does not decide which user may call an API. Configure [authentication and authorization](/modules/ledger/operations/authentication) separately, and terminate HTTPS for the HTTP API at your ingress or load balancer. ## Prepare the certificate The server certificate must cover every DNS name that Ledger nodes and clients use. Take the exact service and pod names from your Stack release. If it uses the example names below for a deployment named `payments` in the `formance` namespace, include both names: ```text ledger-payments.formance.svc.cluster.local *.ledger-payments-headless.formance.svc.cluster.local ``` Add the external gRPC hostname when the connection reaches Ledger with that server name: ```text grpc.ledger.example.com ``` The wildcard covers pod addresses such as: ```text ledger-payments-0.ledger-payments-headless.formance.svc.cluster.local ``` Do not issue a certificate only for `localhost` or a load-balancer address. Raft members and Stack jobs dial internal service and pod names. A missing SAN can leave pods running while inter-node calls fail certificate verification. The leaf certificate and key must use PEM encoding. Protect the private key; clients only need the CA bundle. Use only the Secret reference and key names documented by your Stack release. ## Prepare the Stack v4 TLS rollout Operator-managed TLS is not available in this Stack v4 preview. Use the direct-server workflow below only for evaluation, and wait for your release's Stack Operator reference before changing production TLS. Ledger supports three modes. A production rollout must use the compatibility mode to move a running cluster through two rolling updates: ```mermaid flowchart LR D["disabledplaintext"] --> O["optionalTLS and plaintext"] O --> R["requiredTLS only"] ``` In `optional` mode, an updated node accepts TLS and plaintext while other members restart. After every node has converged, the controller can roll the cluster again in `required` mode. Before production, verify that the released Operator owns both passes, references the Secret, and reports which member prevents convergence. Do not edit the generated workload or its environment variables. Let the released Stack Operator own both passes. ### Verify the direct server behavior Use a disposable single-node deployment to verify the Ledger flags. This command creates a new evaluation cluster and requires TLS from its first start: ```bash ledger run \ --node-id 1 \ --cluster-id tls-evaluation \ --bind-addr 127.0.0.1:7777 \ --grpc-port 8888 \ --http-port 9000 \ --wal-dir ./tls-evaluation/wal/node-1 \ --data-dir ./tls-evaluation/data/node-1 \ --bootstrap \ --tls-mode required \ --tls-cert-file ./tls.crt \ --tls-key-file ./tls.key \ --tls-ca-cert-file ./ca.crt \ --cluster-secret "$LEDGER_CLUSTER_SECRET" ``` Ledger refuses to start with `--cluster-secret` while TLS is disabled. Use `--bootstrap` only on the first start; keep the same identity, storage paths, TLS files, and cluster secret when restarting without that flag. ## Understand cluster trust When a cluster secret is configured, every Ledger node must receive the same value. Inter-node gRPC calls use that static bearer token to identify a trusted peer. The Stack Operator must generate or reference this value and distribute it without exposing it in the resource. This internal identity matters when a client sends a mutation to a follower: 1. The follower authenticates the client and records its subject and scopes. 2. The follower forwards the mutation and caller snapshot to the leader. 3. The follower authenticates that internal call with the cluster secret. 4. The leader accepts the forwarded caller snapshot only from a trusted cluster connection. Ledger refuses to start with a cluster secret while TLS is disabled because the secret would cross the network in plaintext. Do not copy the cluster secret into application clients; it grants the internal path all Ledger scopes. ## Require client certificates By default Ledger verifies an inbound client certificate only if one is presented. A peer that presents none still completes the TLS handshake and falls through to cluster-secret or JWT authentication. That is the permissive posture, and it is the one you get unless you change it. `--tls-require-client-cert` switches inbound verification to reject any connection without a CA-signed client certificate. It needs a CA to verify against, so Ledger refuses to start if `--tls-ca-cert-file` is not also set. ```bash ledger run \ --tls-mode required \ --tls-cert-file ./tls.crt \ --tls-key-file ./tls.key \ --tls-ca-cert-file ./ca.crt \ --tls-require-client-cert \ ... ``` Enabling this rejects every client that does not present a CA-signed certificate — including a `ledgerctl` invocation that passes only `--tls-ca-cert`, which verifies the *server* and sends no client certificate of its own. Issue client certificates to every caller, including your own administrative tooling, before you turn it on. ## Connect with ledgerctl Dial a DNS name present in the certificate and pass the CA bundle: ```bash ledgerctl \ --server ledger-payments.formance.svc.cluster.local:8888 \ --tls-ca-cert ./ca.crt \ ledgers list ``` If the dial address cannot match a certificate SAN, keep the network address and verification name separate: ```bash ledgerctl \ --server 127.0.0.1:8888 \ --tls-server-name ledger-payments.formance.svc.cluster.local \ --tls-ca-cert ./ca.crt \ ledgers list ``` `--tls-server-name` cannot be combined with `--insecure`. A profile can store `tlsCaCert` and `tlsServerName` so operators do not need to repeat them. ## Verify every replica After the released Stack v4 status reports that every member requires TLS, test TLS and a forwarded write through each directly reachable replica. The example assumes `security-smoke` already exists and that `LEDGERCTL_AUTH_TOKEN` contains `ledger:read` and `ledger:TransactionWrite`. If the cluster requires request signatures, add the signing key and key ID described in [Request and Response Signing](/modules/ledger/operations/signing). ```bash for server in \ "ledger-0.internal.example.com:8888" \ "ledger-1.internal.example.com:8888" \ "ledger-2.internal.example.com:8888" do ledgerctl \ --server "$server" \ --tls-ca-cert ./ca.crt \ transactions create \ --ledger security-smoke \ --posting "world,security:tls-probe:${server%%.*},1,TEST" done ``` Every command must succeed regardless of which node is the current leader. This validates more than pod readiness: it exercises certificate verification, the cluster secret, caller forwarding, leader routing, and the response path. Verify the resulting accounts through the stable service: ```bash for account in ledger-0 ledger-1 ledger-2; do ledgerctl \ --server ledger.internal.example.com:8888 \ --tls-ca-cert ./ca.crt \ accounts get "security:tls-probe:${account}" \ --ledger security-smoke done ``` ## Restrict network access Keep network policy in platform-owned configuration unless your Stack release explicitly manages it. Permit only the paths Ledger requires: - Raft, gRPC, and HTTP traffic between members of the same Ledger deployment; - DNS over TCP and UDP; - the exact OIDC, event broker, object-store, and OpenTelemetry destinations configured for the deployment. Define ingress rules separately to limit callers to the gateway, Stack Operator, monitoring, and approved application namespaces. Private OIDC providers, databases, event brokers, object stores, and telemetry collectors need explicit egress where applicable. Test every dependency after enabling the policy; a healthy Ledger API does not prove that backups or event delivery can leave the pod. ## Terminate external traffic intentionally Ledger's server TLS protects service and inter-node gRPC. Treat external endpoints separately: | Endpoint | Recommended boundary | | --- | --- | | HTTP API | HTTPS terminated by the ingress or load balancer, with TLS from the edge to Ledger according to your cluster policy | | External gRPC | A gRPC-capable ingress or load balancer whose certificate and backend mode match the connection path | | Headless and Raft services | Cluster-internal only; do not expose them through an external load balancer | Avoid a TLS mode where the ingress speaks plaintext to a Ledger gRPC service that already requires TLS. This usually surfaces as `error reading server preface` or a connection reset rather than an HTTP error. ## Rotate certificates Ledger watches the mounted leaf certificate, private key, and inbound CA bundle and reloads them for new server handshakes. Outbound gRPC trust pools do not reload a changed CA bundle in place. For a leaf-only rotation signed by the same CA: 1. issue a certificate with the same required SANs; 2. update `tls.crt` and `tls.key` together; 3. open a new connection to every replica and verify the presented certificate; 4. repeat the forwarded-write test. For a CA rotation, publish a bundle containing both old and new CA certificates, restart one node at a time so outbound trust picks up the bundle, rotate leaf certificates, verify every peer, then remove the old CA in a second rolling update. Preserve quorum throughout both passes. ## Disable TLS The production disable workflow must reverse the two passes: `required` → `optional` → `disabled`. The Stack Operator must remove the cluster secret only after every node has stopped referencing it. Do not attempt this rollout unless the released workflow explicitly supports and reports both passes. Do not disable TLS on an externally reachable production cluster. In particular, do not disable it while a multi-node deployment relies on authenticated forwarded writes. The final state removes the encrypted inter-node trust channel and accepts plaintext gRPC. ## Diagnose connection failures | Symptom | Likely cause | Check | | --- | --- | --- | | `x509: certificate is valid for ...` | The dialed name is missing from the SAN list | Dial a covered DNS name or set `--tls-server-name` to one | | `certificate signed by unknown authority` | Wrong or incomplete CA bundle | Compare the server certificate chain with the file passed to `--tls-ca-cert` | | Stack TLS rollout does not converge | A member cannot become ready in the current compatibility pass | Inspect the released Stack conditions and the affected Ledger node's TLS startup logs | | `error reading server preface` | Client and server disagree about plaintext versus TLS | Remove `--insecure` and provide the CA when the cluster is `required`; check ingress backend protocol | | Direct leader writes work, follower writes fail | Cluster secret drift, TLS failure between nodes, or an untrusted forwarded caller snapshot | Confirm every member received the same TLS files and cluster secret; repeat the per-replica write test | | Server rejects `--cluster-secret` at startup | TLS mode is `disabled` or the certificate/key flags are missing | Move through `optional` before `required`, or let the released Stack controller coordinate the change | | Private IdP or broker becomes unreachable after NetworkPolicy activation | The egress policy blocks its private address | Add a narrowly scoped platform-owned egress rule and retest that dependency | --- ## Create and Manage Indexes Source: https://docs.formance.com/modules/ledger/querying/indexes Ledger maintains a small set of read paths for every ledger and builds other indexes only when you request them. Start from a production query, identify the fields it uses, and create the corresponding indexes. Each additional index consumes storage, and adds work to the write path and to the historical backfill that makes the index usable. The HTTP examples use the `LEDGER_URL`, `LEDGER_NAME`, and `CURL_AUTH` variables defined in [Read Ledger Data](/modules/ledger/querying/read-ledger-data), plus `jq`. The per-ledger HTTP routes for listing, inspecting, and checking an index require `ledger:LedgerRead`; creating and dropping one require `ledger:LedgerWrite`. Bucket-wide index inventory and status require `ledger:OpsRead`. See [Authentication and Authorization](/modules/ledger/operations/authentication). ## Choose the required index | Query condition | Index to create | `ledgerctl --type` | | --- | --- | --- | | Transaction reference | Transaction reference | `reference` | | Transaction effective timestamp | Transaction timestamp | `timestamp` | | Transaction insertion time | Transaction inserted-at | `inserted-at` | | Transaction reversal time | Transaction reverted-at | `reverted-at` | | Any posting account | Transaction address | `address` | | Posting source account | Transaction source address | `source-address` | | Posting destination account | Transaction destination address | `destination-address` | | Account has held an asset | Account asset | `account-asset` | | Account or transaction metadata | Metadata for the exact target and key | `metadata` plus `--target` and `--key` | | Ledger log date | Log date | REST canonical ID only | Account address matching, transaction ID ranges, and transaction `reverted` checks do not require an opt-in index. Metadata queries require two declarations in this order: 1. declare the metadata key and type in the ledger schema; 2. create an index for the same target and key. See [Metadata and Typed Values](/modules/ledger/modeling/metadata) before indexing a metadata key whose existing values do not have one stable representation. ## Choose a management surface Operator-managed index configuration is not available in this Stack v4 preview. When your released Operator supports indexes, manage each index through one owner only and follow [Configuration as Code](/modules/ledger/operations/configuration-as-code). A reconciler will recreate a directly deleted index while its desired configuration still declares it. Use the HTTP or `ledgerctl` workflows below to create, check, and remove an index. ## Create an index with ledgerctl `ledgerctl` uses the gRPC administration surface: ```bash ledgerctl indexes create --ledger "$LEDGER_NAME" --type reference ledgerctl indexes create --ledger "$LEDGER_NAME" --type address ledgerctl indexes create \ --ledger "$LEDGER_NAME" \ --type metadata \ --target account \ --key status ``` List the declared indexes and the readiness status reported by the contacted node: ```bash ledgerctl indexes list --ledger "$LEDGER_NAME" ``` The command needs `ledger:LedgerRead` to list registry definitions. Its human-readable table also calls the bucket-wide gRPC status service, which requires `ledger:OpsRead`, to add the contacted replica's readiness status. Without `ledger:OpsRead`, the definitions remain available but the status column is `UNKNOWN`. Structured output such as `--json` returns only the registry definitions; use the per-ledger HTTP status route with `ledger:LedgerRead` when automation needs `currentVersion` or `pendingVersion`. ## Create an index with HTTP The HTTP create and delete routes submit unsigned mutations. Ledger rejects them when `require_signatures=true`. In that mode, use `ledgerctl indexes create` and `ledgerctl indexes drop` through the signed gRPC `Apply` path, and configure the approved key and key ID in the CLI profile or with the global `--signing-key` and `--signing-key-id` options. See [Request and Response Signing](/modules/ledger/operations/signing). The REST route accepts a canonical index ID. Capture the normalized ID returned by the successful `201` response: ```bash REQUESTED_INDEX_ID='metadata:TARGET_TYPE_ACCOUNT:status' INDEX_RESPONSE="$(curl --fail-with-body --silent --show-error \ --request POST \ "${CURL_AUTH[@]}" \ --header "Content-Type: application/json" \ --header "Idempotency-Key: create-$LEDGER_NAME-account-status-index" \ --data "{\"id\":\"$REQUESTED_INDEX_ID\"}" \ "$LEDGER_URL/v3/$LEDGER_NAME/indexes")" INDEX_ID="$(jq -er '.data.id' <<<"$INDEX_RESPONSE")" test "$INDEX_ID" = "$REQUESTED_INDEX_ID" ``` Common canonical IDs include: ```text tx_builtin:TX_BUILTIN_INDEX_REFERENCE tx_builtin:TX_BUILTIN_INDEX_TIMESTAMP tx_builtin:TX_BUILTIN_INDEX_ADDRESS tx_builtin:TX_BUILTIN_INDEX_SOURCE_ADDRESS tx_builtin:TX_BUILTIN_INDEX_DESTINATION_ADDRESS tx_builtin:TX_BUILTIN_INDEX_INSERTED_AT tx_builtin:TX_BUILTIN_INDEX_REVERTED_AT account_builtin:ACCT_BUILTIN_INDEX_ASSET log_builtin:LOG_BUILTIN_INDEX_DATE metadata:TARGET_TYPE_ACCOUNT: metadata:TARGET_TYPE_TRANSACTION: ``` The `201` response returns the normalized canonical ID. Store that value for status, inspection, and deletion calls. ## Wait for readiness Creating an index registers it and starts an asynchronous backfill. It does not make the index immediately available to queries. ```mermaid flowchart LR Create["Create index"] --> Registered["Index registered"] Registered --> Backfill["Each replica backfills history"] Backfill --> Switch["Replica switches currentVersion"] Switch --> Ready["Query can use index on that replica"] ``` Poll the status route using the canonical ID returned at creation: ```bash curl --fail-with-body --silent --show-error \ "${CURL_AUTH[@]}" \ "$LEDGER_URL/v3/$LEDGER_NAME/indexes/$INDEX_ID/status" ``` Interpret the version fields as follows: | Field | Meaning | | --- | --- | | `currentVersion: 0` | The contacted replica cannot serve this index yet | | `currentVersion > 0` | The contacted replica can serve its current index version | | `pendingVersion > 0` | That replica is building a newer version while continuing to serve `currentVersion` | `ledgerctl indexes list --json` returns registry definitions only; they carry no readiness signal. Query execution uses the contacted replica's `currentVersion`, which only the status route reports. Index status is local to the node that serves the request. In a multi-replica cluster, query each pod directly and wait for `currentVersion > 0` on every pod that can receive reads. Repeating one request through a load balancer does not prove that all replicas are ready. Set `LEDGER_REPLICA_URLS` to endpoints that route directly to every Ledger pod that serves reads. Do not put a load-balanced URL in this list. ```bash ``` Run the check and print the version observed on each replica: ```bash ALL_READY=true printf 'endpoint\tcurrent\tpending\tforward\n' for endpoint in $LEDGER_REPLICA_URLS; do STATUS="$(curl --fail-with-body --silent --show-error \ "${CURL_AUTH[@]}" \ "$endpoint/v3/$LEDGER_NAME/indexes/$INDEX_ID/status")" || exit 1 jq -r --arg endpoint "$endpoint" ' .data | [$endpoint, .currentVersion, .pendingVersion, .index.forwardEncodingVersion] | @tsv ' <<<"$STATUS" if ! jq -e '.data.currentVersion > 0' <<<"$STATUS" >/dev/null; then ALL_READY=false fi done test "$ALL_READY" = true ``` For an initial build, the acceptance criterion is `current > 0` on every printed row and a final exit status of `0`. The version numbers can differ from the example below, but a completed initial build commonly looks like: ```text endpoint current pending forward https://ledger-0.internal.example.com 1 0 1 https://ledger-1.internal.example.com 1 0 1 https://ledger-2.internal.example.com 1 0 1 ``` Keep the pod-direct routing in deployment checks. Replacing these endpoints with one load-balanced address removes the per-replica guarantee. Changing a metadata field declaration does not convert stored metadata values. When the declaration changes, replace the `jq` readiness condition inside the per-replica loop with this stricter criterion before depending on the new index encoding and comparison semantics: ```bash jq -e ' .data.currentVersion > 0 and .data.pendingVersion == 0 and .data.currentVersion == .data.index.forwardEncodingVersion ' <<<"$STATUS" ``` ## Inspect a metadata index Inspection answers data-modeling questions without changing the index. It is available only for metadata indexes whose current version is ready. ```bash # Cardinality, min/max, and null/existence counts ledgerctl indexes inspect \ --ledger "$LEDGER_NAME" \ --target account \ --key status \ --mode summary # Values and counts ledgerctl indexes inspect \ --ledger "$LEDGER_NAME" \ --target account \ --key status \ --mode facets \ --page-size 100 ``` The HTTP inspection modes are `summary`, `distinctValues`, and `facets`: ```bash curl --fail-with-body --silent --show-error --get \ "${CURL_AUTH[@]}" \ --data-urlencode 'mode=facets' \ --data-urlencode 'pageSize=100' \ "$LEDGER_URL/v3/$LEDGER_NAME/indexes/$INDEX_ID/inspect" ``` `summary` scans the complete metadata index to compute its result. `distinctValues` and `facets` return an opaque `nextCursor` when another page exists. Treat inspection as an operator or design-time tool, not as a high-frequency application query. Built-in transaction, account, and log indexes cannot be inspected through this route. ## Remove an index safely Before removing an index: 1. identify every direct and prepared query that uses it; 2. deploy callers that no longer issue those queries; 3. confirm that no reconciliation or reporting job still depends on it; 4. remove the index; 5. verify that the registry no longer lists it. With `ledgerctl`, identify the same type, target, and key used at creation: ```bash ledgerctl indexes drop \ --ledger "$LEDGER_NAME" \ --type metadata \ --target account \ --key status ``` With HTTP, delete by canonical ID: ```bash curl --fail-with-body --silent --show-error \ --request DELETE \ "${CURL_AUTH[@]}" \ --header "Idempotency-Key: drop-$LEDGER_NAME-account-status-index" \ "$LEDGER_URL/v3/$LEDGER_NAME/indexes/$INDEX_ID" ``` Verify that the registry no longer resolves that exact ID: ```bash HTTP_STATUS="$(curl --silent --show-error \ --output /dev/null \ --write-out '%{http_code}' \ "${CURL_AUTH[@]}" \ "$LEDGER_URL/v3/$LEDGER_NAME/indexes/$INDEX_ID")" test "$HTTP_STATUS" = 404 ``` An exit status of `0` confirms that the captured ID is absent. A different status means the deletion is not verified: check authentication, connectivity, and any configuration reconciler before continuing. A caller that still depends on the removed index receives an index-not-found error. Recreating the index starts a new backfill; it does not restore immediate readiness. ## Diagnose index failures | Result | Meaning | Action | | --- | --- | --- | | `INDEX_NOT_FOUND` | The required index is not declared, or a metadata key is absent from the schema | Declare the schema field and index for the exact query target | | HTTP `503` with `INDEX_BUILDING` | The contacted replica has not switched to a usable version | Retry with bounded backoff and inspect each serving replica | | HTTP `409` on create | An index with that identity already exists or the request conflicts | List indexes and reuse the existing canonical ID | | HTTP `400` on inspect | The canonical ID is malformed, or the selected operation does not support a built-in index | Inspect metadata indexes only; treat a not-ready index as `INDEX_BUILDING` instead | Continue with [Create Prepared Queries](/modules/ledger/querying/prepared-queries) when several callers must execute the same reviewed filter. --- ## Retry Writes Safely Source: https://docs.formance.com/modules/ledger/transactions/idempotency-and-references Network failures make a write outcome uncertain: the client may lose the response after Ledger committed the transaction. An idempotency key gives every retry of one logical operation the same outcome. A transaction reference adds a permanent business-level uniqueness rule inside one ledger. Use both for production transactions. They solve different problems. | Control | Scope | Retention | Purpose | | --- | --- | --- | --- | | `Idempotency-Key` | Entire Ledger cluster, across all ledgers | Configurable; 24 hours by default | Replays the original outcome for a technical retry. | | `reference` | One ledger | Retained with the transaction | Prevents two transactions from claiming the same business identifier. | ## Prerequisites You need an existing normal ledger and a token with `ledger:write` for the transaction plus `ledger:read` for the account check. If the ledger enforces account types, replace the demo address with one that matches its configured patterns. ```bash set -eo pipefail ``` Choose an idempotency key before the first attempt. Keep it stable through every retry, and keep the full request body with the operation in your application state or durable job record. ## Submit and replay one operation Store the body in one shell variable so both attempts send identical content: ```bash PAYMENT_BODY=$(jq --compact-output --null-input \ --arg reference "$PAYMENT_REFERENCE" \ --arg account "$WALLET_ACCOUNT" \ '{ reference: $reference, postings: [ { source: "world", destination: $account, amount: 5000, asset: "EUR/2" } ] }') ``` Send the first attempt: ```bash curl --fail-with-body --silent --show-error \ --request POST "$LEDGER_URL/v3/$LEDGER_NAME/transactions" \ --header "Authorization: Bearer $LEDGER_TOKEN" \ --header "Content-Type: application/json" \ --header "Idempotency-Key: $IDEMPOTENCY_KEY" \ --data "$PAYMENT_BODY" \ > /tmp/idempotency-first.json ``` Then replay the same request with the same key: ```bash curl --fail-with-body --silent --show-error \ --request POST "$LEDGER_URL/v3/$LEDGER_NAME/transactions" \ --header "Authorization: Bearer $LEDGER_TOKEN" \ --header "Content-Type: application/json" \ --header "Idempotency-Key: $IDEMPOTENCY_KEY" \ --data "$PAYMENT_BODY" \ > /tmp/idempotency-replay.json ``` Both responses identify the same transaction: ```bash FIRST_ID=$(jq -er \ '.data.transaction.id | tostring | select(test("^[1-9][0-9]*$"))' \ /tmp/idempotency-first.json) && REPLAY_ID=$(jq -er \ '.data.transaction.id | tostring | select(test("^[1-9][0-9]*$"))' \ /tmp/idempotency-replay.json) && test -n "$FIRST_ID" && test -n "$REPLAY_ID" && test "$FIRST_ID" = "$REPLAY_ID" ``` Verify the accounting state independently. The wallet must receive EUR 50.00 once, not twice: ```bash curl --fail-with-body --silent --show-error \ "$LEDGER_URL/v3/$LEDGER_NAME/accounts/$WALLET_ACCOUNT" \ --header "Authorization: Bearer $LEDGER_TOKEN" \ | jq -e ' .data.volumes[] | select(.asset == "EUR/2" and .color == "") | .volumes.input == "5000" and .volumes.output == "0" and .volumes.balance == "5000"' ``` Expected volume: ```json { "input": "5000", "output": "0", "balance": "5000" } ``` ## Detect an incorrect retry Ledger hashes the logical request content. Reusing the key with a changed amount returns `409 Conflict` instead of applying a different operation: ```bash CHANGED_BODY=$(jq --compact-output \ '.postings[0].amount = 9000' \ <<<"$PAYMENT_BODY") HTTP_STATUS=$(curl --silent --show-error \ --output /tmp/idempotency-conflict.json \ --write-out "%{http_code}" \ --request POST "$LEDGER_URL/v3/$LEDGER_NAME/transactions" \ --header "Authorization: Bearer $LEDGER_TOKEN" \ --header "Content-Type: application/json" \ --header "Idempotency-Key: $IDEMPOTENCY_KEY" \ --data "$CHANGED_BODY") test "$HTTP_STATUS" = "409" && jq -e ' select(.errorCode == "IDEMPOTENCY_KEY_CONFLICT") | {errorCode, errorMessage}' \ /tmp/idempotency-conflict.json ``` The machine-readable reason is `IDEMPOTENCY_KEY_CONFLICT`. Do not respond by generating a new key automatically: first determine whether the caller intended a new operation or sent a corrupted retry. ## Enforce business uniqueness with a reference The reference remains unique for the lifetime of the ledger, independent of the idempotency-key window. A second request with a new key but the same reference returns a transaction-reference conflict: ```bash HTTP_STATUS=$(curl --silent --show-error \ --output /tmp/reference-conflict.json \ --write-out "%{http_code}" \ --request POST "$LEDGER_URL/v3/$LEDGER_NAME/transactions" \ --header "Authorization: Bearer $LEDGER_TOKEN" \ --header "Content-Type: application/json" \ --header "Idempotency-Key: $IDEMPOTENCY_KEY:replacement" \ --data "$PAYMENT_BODY") test "$HTTP_STATUS" = "409" && jq -e ' select(.errorCode == "TRANSACTION_REFERENCE_CONFLICT") | {errorCode, errorMessage}' \ /tmp/reference-conflict.json ``` The reason is `TRANSACTION_REFERENCE_CONFLICT`. Treat it as success only after you verify that the existing transaction represents the same business event and amount. A shared reference alone does not prove that two request bodies were equivalent. The reference conflict is reported only when the transaction commits, which happens after its Numscript has executed. Because funds are checked during execution, a resubmission can fail before the duplicate reference is ever evaluated, raising an insufficient-funds error (`INSUFFICIENT_FUNDS` in v3, `INSUFFICIENT_FUND` in v2), for example, instead of `TRANSACTION_REFERENCE_CONFLICT`. This is most visible with scripts whose postings depend on live balances (for example an amount derived from `overdraft()` or `balance()`), but it can affect any script whose replay no longer has sufficient funds at the source. Do not depend on receiving `TRANSACTION_REFERENCE_CONFLICT` from a resubmission to tell you an operation already ran. Prefer a durable idempotency key, which is evaluated before execution and replays the original outcome. Querying for the reference beforehand is a useful guard, but only a positive match is conclusive: an empty result is inconclusive, because the original request may still be committing or a reference-filtered read may lag the write. ## Design keys that survive retries An HTTP idempotency key can contain up to 256 UTF-8 bytes. Use a stable namespace plus an opaque operation ID, for example `payments-api:evt-7e49c4c9`. ASCII keys make this limit unambiguous. Do not include access tokens, customer names, or other sensitive values. Idempotency keys are cluster-global. The same key used for `ledger-a` and `ledger-b` refers to one operation and can conflict. Include the producing service or tenant in the key when several systems generate IDs independently. The server retains keys for the configured TTL: - the default is 24 hours; - operators can choose another duration; - `0` disables expiration; - as soon as the TTL expires, Ledger can treat the same key as new; background eviction only removes the expired storage entry. Set the idempotency TTL longer than the maximum retry and delayed-delivery window of every writer. Once a key expires, a retry can create a new transaction. A transaction reference remains the durable guard for operations that have a business identifier. ## Decide whether to retry | Outcome | Retry decision | | --- | --- | | Timeout or connection reset | Retry the identical request with the same key after bounded backoff. | | `503` | Inspect `errorCode`, honor `Retry-After` when present, and retry the identical request with the same key only while the outcome is uncertain. Stop repeated deterministic input failures such as Numscript `PRELOAD_UNAVAILABLE` and correct the request. | | Invalid JSON, invalid path parameters, authentication failure, or another rejection before Ledger accepts the write | Correct the request or credentials. Do not infer from the HTTP status that Ledger retained the key. Use a new key when the logical request content changes. | | Definitive business failure recorded as the operation's outcome | Keep the failure as the result of that logical operation. Ledger retains and replays these failures while the key remains live, even if account state later changes. A corrected operation needs a new key. | | `409` idempotency conflict | Stop. Compare the attempted request with the original operation. | | `409` reference conflict | Retrieve and verify the transaction that already owns the reference. | | `401` or `403` | Repair credentials or permissions before retrying. Keep the same key and body if the logical operation is unchanged. | Persist every response with your operation record. That record, not the HTTP status, is what tells a later retry which key and which body to reuse. The literal `latest` selector is hashed as `latest` and re-resolved on every execution: a retained key replays its first outcome, while an operation without a retained key executes whatever version is greatest at that moment. Ledger never rewrites a submitted selector to a concrete version, so a changed resolution cannot turn an unchanged body into an idempotency conflict. Pin an exact semantic version for retryable production writes. ## Apply the same rule to other writes The HTTP contract also accepts `Idempotency-Key` on revert, on every transaction, account, and ledger metadata write and delete, and on the numscript save. The REST bulk endpoint uses the `ik` field on each element when it processes elements sequentially. See [Process Operations in Bulk](/modules/ledger/transactions/bulk) before retrying a partial batch. Continue with [Revert Transactions](/modules/ledger/transactions/revert) when a committed operation needs an auditable correction. --- ## Double-entry Bookkeeping Source: https://docs.formance.com/modules/ledger/core-concepts/double-entry Every transaction in Formance must balance: assets leaving one account must arrive in another. This is the principle of double-entry bookkeeping. Assets are never created or destroyed inside the ledger; they only move. ## What it's for Double-entry is what makes the ledger trustworthy. It enforces conservation of assets at the database level, so any discrepancy between what entered the system and what accounts hold is structurally impossible, not a convention your code tries to maintain. ## The three rules ### 1. Transactions must balance Within a single transaction, the total amount added to source components must equal the total amount added to destination components. No assets are created or destroyed. In a multi-asset ledger this holds independently for each asset — USD/2 sources must balance USD/2 destinations, EUR/2 sources must balance EUR/2 destinations, and so on. **Invalid example:** A posting that credits `account:1` with 100 USD/2 without a corresponding debit violates this rule. The total destination increase (100) does not equal the total source increase (0), so the transaction is rejected. ### 2. The ledger-wide sum is always zero Across all accounts, the sum of all destination components always equals the sum of all source components at all times. Assets are never created or destroyed. This invariant holds per asset — the ledger-wide balance of USD/2 is always zero, the ledger-wide balance of EUR/2 is always zero, and so on independently. ### 3. Accounts cannot go negative by default An account's balance cannot fall below zero unless it is explicitly permitted. This prevents accounts from spending assets they do not have. **Invalid example:** An account with a balance of 100 USD/2 cannot send 150 USD/2. Doing so would leave it with a balance of −50 USD/2, violating this rule. To allow an account to go below zero, use the `allowing overdraft` clause in Numscript. See [Overdraft](/modules/numscript/reference/overdraft). ## The world account Since no transaction can create assets, there must be a designated way to introduce new assets from outside the system. Note that the ledger tracks assets, not value — for currencies, this typically maps 1:1, but value (e.g. accounting in GBP while holding USD) is not represented in the ledger. There are two patterns. ### Using the world account The `@world` account is a built-in account permitted to carry a negative balance. Sending from `@world` to any destination introduces assets into the ledger; `@world`'s output increases while the recipient's input increases by the same amount. `@world` is Formance-specific and has no counterpart in conventional accounting, so it does not map cleanly onto a real-world funding source. It is the fastest way to get a ledger working and it is the right tool for tests, prototypes, and the specific cases where the origin of the assets genuinely is outside your books. For production models where each inflow has an identifiable external counterparty, prefer counter-part accounts below. **Before:** | Account | Input | Output | Balance | |---------|-------|--------|---------| | `account:1` | 0 | 0 | 0 | | `world` | 0 | 0 | 0 | **Transaction:** send 100 USD/2 from `world` to `account:1` **After:** | Account | Input | Output | Balance | |---------|-------|--------|---------| | `account:1` | 100 | 0 | 100 | | `world` | 0 | 100 | -100 | The `world` account carries −100. Total ledger-wide balance: `100 + (−100) = 0`. ### Using counter-part accounts Instead of `@world`, use named counter-part accounts — one per payment method or external funding source. In accounting terms, these can be treated as Normal Debit asset accounts; their balance is negative because they are the counter-part. They represent the platform's obligation to those external parties. **Example:** A customer pays 100 USD/2: 50 via credit card, 50 via bank transfer. **Before:** | Account | Input | Output | Balance | |---------|-------|--------|---------| | `order:1234` | 0 | 0 | 0 | | `payment-method:credit-card` | 0 | 0 | 0 | | `payment-method:bank-transfer` | 0 | 0 | 0 | **After:** | Account | Input | Output | Balance | |---------|-------|--------|---------| | `order:1234` | 100 | 0 | 100 | | `payment-method:credit-card` | 0 | 50 | -50 | | `payment-method:bank-transfer` | 0 | 50 | -50 | The counter-part accounts carry negative balances representing what was received from each payment method. Reconciliation is straightforward: the balance of `payment-method:credit-card` is exactly the amount to reconcile against your credit card processor. By mapping counter-part accounts to your financial partners, you can reconcile your ledger against external systems by comparing each counter-part account's balance against the partner's reported balance. ## Overdraft and negative balances Some accounts are designed to carry a negative balance: liability accounts, credit lines, or accounts that represent external obligations. The ledger permits this when you explicitly declare it in Numscript using the `allowing overdraft` clause. ```numscript send [USD/2 1000] ( source = @users:alice allowing unbounded overdraft destination = @vendors:acme ) ``` You can also cap the overdraft to a specific amount: ```numscript send [USD/2 1000] ( source = @users:alice allowing overdraft up to [USD/2 500] destination = @vendors:acme ) ``` **Example:** Alice's account has a zero balance but is permitted to go negative. She purchases USD/2 8000 of software from a vendor: ```numscript send [USD/2 8000] ( source = @users:alice allowing unbounded overdraft destination = @vendors:acme-software ) ``` Alice's account balance is now −USD/2 8000. When her salary is credited and the balance is settled: ```numscript send [USD/2 8000] ( source = @payroll destination = @users:alice ) ``` Use `allowing overdraft` when modeling credit lines, loans, liability accounts (such as counter-part payment method accounts), or any account that legitimately represents an obligation. ## Mapping to traditional debit/credit accounting This section is for readers familiar with traditional double-entry accounting. If you're new to accounting, you can skip this section. ### The golden rule The golden rule for converting a credit/debit accounting model to Formance's source/destination model is: - **Credit** becomes **Destination** - **Debit** becomes **Source** ### Normal accounts In accounting practice, accounts are designated as _normal debit_ or _normal credit_ depending on whether they are on the asset or liability side of the balance sheet: | Entry type | Formance role | Normal Debit (trad. balance) | Normal Credit (trad. balance) | |------------|---------------|------------------------------|-------------------------------| | Debit Entry | **Source** | Increases | Decreases | | Credit Entry | **Destination** | Decreases | Increases | ### Conversion examples #### Topping up a wallet with a bank transfer A user tops up their wallet with a bank transfer. The user wallet is a liability (_normal credit_); the bank account is an asset (_normal debit_). In the credit/debit model: | Account | Debit | Credit | |---------|-------|--------| | User Wallet | `$0` | `$100` | | Bank Account | `$100` | `$0` | Applying the golden rule: | Account | Source | Destination | Balance | |---------|--------|-------------|---------| | User Wallet | `$0` | `$100` | `$100` | | Bank Account | `$100` | `$0` | `-$100` | The bank account carries a negative balance here. As an asset account (_normal debit_), its balance in the source/destination model is negative. Formance supports this with the [`allowing overdraft` clause](/modules/numscript/reference/overdraft) in Numscript. #### Transferring money between two wallets User 1 transfers `$100` from their wallet to User 2's wallet. Both wallets are liabilities (_normal credit_). Starting balances: | Account | Balance | |---------|---------| | User 1 Wallet | `$200` | | User 2 Wallet | `$0` | In the credit/debit model: | Account | Debit | Credit | |---------|-------|--------| | User 1 Wallet | `$100` | `$0` | | User 2 Wallet | `$0` | `$100` | Applying the golden rule: | Account | Source | Destination | Balance | |---------|--------|-------------|---------| | User 1 Wallet | `$100` | `$200` | `$100` | | User 2 Wallet | `$0` | `$100` | `$100` | #### Withdrawing money from a wallet A merchant withdraws `$50` from their wallet to their bank account. The wallet is a liability (_normal credit_); the bank account is an asset (_normal debit_). Starting balances: | Account | Balance | |---------|---------| | Merchant Wallet | `$100` | | Bank Account | `-$1000` | In the credit/debit model: | Account | Debit | Credit | |---------|-------|--------| | Merchant Wallet | `$50` | `$0` | | Bank Account | `$0` | `$50` | Applying the golden rule: | Account | Source | Destination | Balance | |---------|--------|-------------|---------| | Merchant Wallet | `$50` | `$100` | `$50` | | Bank Account | `$1000` | `$50` | `-$950` | #### Treasury movement between two bank accounts A business moves `$250` from a US bank account to a UK bank account. Both are assets (_normal debit_). Starting balances: | Account | Balance | |---------|---------| | US Bank Account | `-$1000` | | UK Bank Account | `-$500` | In the credit/debit model: | Account | Debit | Credit | |---------|-------|--------| | US Bank Account | `$0` | `$250` | | UK Bank Account | `$250` | `$0` | Applying the golden rule: | Account | Source | Destination | Balance | |---------|--------|-------------|---------| | US Bank Account | `$1000` | `$250` | `-$750` | | UK Bank Account | `$750` | `$0` | `-$750` | Asset accounts (_normal debit_) will have a negative balance in the source/destination model, because their source component (outflows) exceeds their destination component (inflows). ## Do / Don't - **Do:** Use `@world` as the source when introducing new funds whose real-world origin is tracked outside the ledger, and when getting started or testing. - **Do:** Prefer named counter-part accounts over `@world` in production models where each inflow has an identifiable external counterparty; they reconcile against your partners, `@world` does not. - **Do:** Use `allowing overdraft` explicitly when modeling credit lines, debts, or liability accounts. - **Don't:** Assume any account can go negative; it will fail without the overdraft clause. - **Don't:** Create single-sided postings (source without a destination, or vice versa); the ledger will reject them. --- ## Transform Mirrored Data Source: https://docs.formance.com/modules/ledger/migration/transformations A transformation is an ordered list of rules Ledger runs over each Ledger v2 log entry as it translates that entry into a target operation. A rule can rename account addresses, add, retype or remove metadata, or skip the entry entirely. Ledger's own vocabulary for these is *rewrite rules*: they are the `rewriteRules` field on a mirror's source configuration and the contents of `--mirror-rewrite-file`, which is the naming you will see in every field, flag, and error message. Most migrations need none of this. Add rules only when the target must use a different account structure, metadata model, or import policy. Rules apply to every entry the mirror ingests, for the mirror's whole life — the initial backfill and every entry it picks up afterwards while it follows Ledger v2. They change only the target projection; Ledger v2 remains unchanged. An entry that no rule matches is imported unchanged, which is what makes "add rules only when you must" a safe default. Rewrite rules are immutable and there is no transformation dry-run. Test the exact ordered rule file by creating a disposable mirror, then compare its output with recorded fixtures. Any rule change requires a new mirror and a complete replay. ## Decide whether to transform | Requirement | Preferred decision | | --- | --- | | Preserve the existing accounting model | Do not add rules. | | Rename account segments for the target model | Use an anchored `rewriteAddress` rule after checking for address collisions. | | Normalize or type legacy metadata | Use `setMetadata`, `deleteMetadata`, or account-metadata actions. | | Derive target account metadata from address segments | Use `setAccountMetadataFromAddress` on created transactions. | | Exclude a known, approved class of source entries | Use `drop` and maintain an explicit drop register. | | Correct arbitrary historical business data | Correct Ledger v2 first, or design a separate reconciliation process. A mirror rule is not a general data-editing tool. | Do not transform data only to make it look cleaner. Every intentional difference increases the validation surface and makes source-to-target comparisons less direct. ## Understand a rule Rules run from top to bottom. Each rule has: - exactly one scope; - an optional CEL `match` expression; - one or more typed actions; - optional `stop: true`, which prevents later rules from running after that rule matches. The scope determines the type of `log` in CEL and the actions the rule can carry. | Scope | Use it for | Available actions | | --- | --- | --- | | `createdTransaction` | New transactions, postings, transaction metadata, and account metadata. | Rewrite address; set/delete transaction metadata; set/delete account metadata; derive account metadata from addresses; drop. | | `revertedTransaction` | Reversion operations and their metadata. | Rewrite address; set/delete metadata; drop. | | `savedMetadata` | A Ledger v2 metadata addition or update. | Rewrite the target address; set/delete metadata; drop. | | `deletedMetadata` | A Ledger v2 metadata deletion. | Rewrite the target address; drop. | | `anyVariant` | Address rewrites or drops that must cover every supported log variant. | Rewrite address; drop. | A rule with a different scope from the current source entry is skipped. An empty `match` means the rule matches every entry in its scope. ```mermaid flowchart LR Source["Ledger v2 log entry"] --> Rule["Next rule in order"] Rule --> Match{"Scope and match apply?"} Match -- "No" --> More{"Another rule?"} Match -- "Yes" --> Actions["Apply actions in order"] Actions --> Error{"Action failed?"} Error -- "Yes" --> Retry["Keep cursor in place and retry the batch"] Error -- "No" --> Drop{"Entry dropped?"} Drop -- "Yes" --> Gap["Preserve the source position without a target transaction"] Drop -- "No" --> Stop{"stop is true?"} Stop -- "No" --> More Stop -- "Yes" --> Target["Write the transformed target operation"] More -- "Yes" --> Rule More -- "No" --> Target ``` ## Write safe CEL predicates CEL reads a scope-specific `log`. Guard optional map keys before accessing their typed value: ```cel "kind" in log.metadata && log.metadata["kind"].string_value == "payout" ``` Use direct key access. Do not iterate metadata maps to produce a value because map iteration is not a stable ordering contract. Rules should not depend on time, randomness, network calls, or other external state. The error behavior is deliberate: | Failure | Mirror behavior | | --- | --- | | `match` cannot evaluate for one entry | Ledger skips that rule for the entry and continues. Guard optional fields to avoid an unintended skip. | | `valueExpr` cannot evaluate | Ledger rejects the batch, records the error, and retries the same batch. | | An action produces an invalid address or value | Ledger rejects the batch, records the error, and retries the same batch. | | A rule does not match | Ledger evaluates the next rule. | | `drop` runs | Ledger records a gap for that source position and stops evaluating the entry. | Ledger never skips a failing batch automatically, and it never gives up on one either. Retries back off exponentially — one second, doubling, capped at sixty — and the backoff resets on the first success. There is no attempt limit and no dead-letter path. That applies to the failures in the table that reject the batch: a `valueExpr` that cannot evaluate, or an action producing an invalid address or value. Any of those stalls the mirror at its current cursor indefinitely, with the error visible in its status, and it will not recover on its own — fix the rule file and rebuild the mirror. A `match` that cannot evaluate is not in this category: Ledger skips that rule for the entry and carries on, which is quieter and easier to miss. ## Recipe 1: rewrite account addresses This rule changes `payments:merchant:m_123:available` to `merchants:m_123` wherever a supported source entry carries that address: ```yaml - anyVariant: actions: - rewriteAddress: pattern: "^payments:merchant:([^:]+):available$" replacement: "merchants:$1" ``` Use RE2-compatible patterns and anchor them with `^` and `$` when the full address must match. Before using this recipe, prove that two distinct source accounts cannot collapse into one target account unless that merge is intentional. A rewrite affects postings and metadata targets, so test transaction creation, reversion, metadata save, and metadata deletion fixtures. | Fixture | Source | Expected target | | --- | --- | --- | | Posting source | `payments:merchant:m_123:available` | `merchants:m_123` | | Posting destination | `payments:merchant:m_456:available` | `merchants:m_456` | | Non-matching account | `payments:customer:c_123:available` | Unchanged | | Similar suffix | `payments:merchant:m_123:pending` | Unchanged | ## Recipe 2: normalize and type metadata This rule identifies payout transactions, adds a target category, copies the reference, converts a legacy string counter to `int64`, and removes a legacy key: ```yaml - createdTransaction: match: >- "kind" in log.metadata && log.metadata["kind"].string_value == "payout" actions: - setMetadata: key: category value: external - setMetadata: key: original_reference valueExpr: log.reference - setMetadata: key: attempt_count valueExpr: log.metadata["attempt"].string_value type: int64 - deleteMetadata: key: legacy_payout stop: true ``` The `value` field is a literal string. `valueExpr` evaluates against the current, already-mutated entry. `type` can coerce to `string`, signed or unsigned integer widths, `bool`, or `datetime`. When a value cannot be parsed as the declared type, the target stores a typed null that preserves the original string for diagnosis. Include valid, missing, and invalid values in the fixture set. In this example, a missing `attempt` causes the action to fail and blocks the batch; add a separate guarded rule or omit the action when missing values are valid source data. | Fixture | Source metadata | Expected target | | --- | --- | --- | | Valid payout | `kind=payout`, `attempt=3`, `legacy_payout=yes` | `category=external`, `attempt_count=3` as `int64`, legacy key absent | | Other transaction | `kind=refund` | Unchanged | | Invalid counter | `kind=payout`, `attempt=three` | `attempt_count` is typed null preserving `three` | | Missing counter | `kind=payout`, no `attempt` | Batch blocks unless the rules handle this case explicitly | To derive account metadata from an address, use the created-transaction-only action: ```yaml - createdTransaction: actions: - setAccountMetadataFromAddress: pattern: "^merchants:([^:]+):wallet:(\\d+)$" replacements: - key: merchant_id replacement: "$1" - key: wallet_id replacement: "$2" type: int64 ``` ## Recipe 3: drop an approved entry class This rule excludes created transactions explicitly marked as internal: ```yaml - createdTransaction: match: >- "internal" in log.metadata && log.metadata["internal"].string_value == "true" actions: - drop: {} stop: true ``` Dropping an entry does not remove it from Ledger v2. The mirror preserves the source log position and transaction-ID progression so later history can continue. A `createdTransaction` rule never sees the reversal of the transaction it dropped: a Ledger v2 reversal arrives as a separate `revertedTransaction` entry, and the metadata on that entry is the *reversal's* own, not the original's. So a predicate keyed on the dropped transaction's metadata cannot match it. Ledger applies the reverse postings regardless of whether the original was imported — it forces them through without requiring the original transaction to exist — so the mirror does not stall or raise an error. The reverse postings are themselves a balanced transaction, so nothing in the ledger is left unbalanced; what breaks is agreement with the source. The target carries a compensating transaction with nothing to compensate, and the affected balances diverge from Ledger v2 by the amount of the original. Validation catches it as a balance mismatch, not as a structural error. Closing this means the dropped class has to be identifiable from **both** entry variants. A `createdTransaction` rule sees the original's `metadata`; a `revertedTransaction` rule sees only `reverted_transaction_id`, `reverse_postings`, the reversal's own `metadata`, and its timestamps — never the original's metadata. There is no registry of dropped IDs for a predicate to consult at run time, and rules are fixed when the mirror is created. For a backfill of history that already exists, that is workable: query Ledger v2 for the IDs of the class you intend to drop before you create the mirror, and match `reverted_transaction_id` against that fixed set. It only covers originals that exist at authoring time, so a reversal arriving later still matches as long as its original was in the set. What it cannot cover is a transaction created *after* the mirror exists and reverted while the mirror follows live writes — that ID was never available to embed. So key the drop on something both variants carry. Posting addresses are the practical choice, because the creation carries `postings` and the reversal carries `reverse_postings`: ```yaml - createdTransaction: match: >- log.postings.exists(p, p.destination.startsWith("internal:")) actions: - drop: {} stop: true - revertedTransaction: match: >- log.reverse_postings.exists(p, p.source.startsWith("internal:")) actions: - drop: {} stop: true ``` The two predicates must describe the same class, and it is on you to make sure they do. Note the reversal's postings run the other way, so the address that was a `destination` on the way in is a `source` on the way back. If the class cannot be expressed as an address invariant, fall back to the fixed-ID approach above: enumerate the originals from Ledger v2 before creating the mirror and match `reverted_transaction_id` against that set. Only a transaction created after the mirror exists and reverted while it follows live writes escapes both approaches, and such a class has to be balance-neutral or reconciled downstream. Do not pair a metadata-keyed drop with an address-keyed reversal rule: they will select different sets, missing some reversals and dropping others they should not. A dropped financial transaction changes target balances unless the dropped class is balance-neutral or its complete accounting effect is handled elsewhere. Record every expected drop and its balance impact before approving the rule. | Fixture | Source | Expected target | | --- | --- | --- | | Approved internal transaction | `internal=true` | No target transaction; source log position appears in the drop register. | | External transaction | `internal=false` | Imported normally. | | Missing flag | No `internal` key | Imported normally. | | Reversion of a dropped transaction | Matching reversion fixture | The recipe above is keyed on metadata, so its reversals are **not** dropped and the reverse postings reach the target. Either re-key the class on an address invariant and drop both variants, or assert that the compensating effect is acceptable. | ## Order interacting rules Later actions see changes made by earlier actions for the same source entry. Use this intentionally: 1. normalize an address or metadata value; 2. derive additional metadata from the normalized entry; 3. stop when no later rule should apply. Keep related rules together and add a one-sentence reason above each rule in the version-controlled file. Reordering rules changes the target projection and invalidates previous validation. ## Run an executable fixture rehearsal There is no dry-run endpoint. Use a disposable Ledger v2 source, stop writes after seeding it, then create a disposable mirror with the exact candidate rule file. Seed at least these source transactions and record their real transaction IDs: | Fixture | Source requirement | Assertion below | | --- | --- | --- | | Matching rewrite | Contains `SOURCE_REWRITE_ADDRESS`. | The target transaction contains `TARGET_REWRITE_ADDRESS` and no longer contains the source address. | | Non-match | Contains `PASSTHROUGH_ADDRESS`. | The target transaction keeps that address unchanged. | | Deliberate drop | Matches the approved `drop` predicate. | Reading the same transaction ID from the target returns `404`. | | Rewritten balance | Produces known input, output, and balance totals on the rewritten account. | The target account matches all three expected totals. | The harness does not seed Ledger v2 because the source write endpoint and authorization policy belong to your existing deployment. Source seeding is a required gate: do not continue with invented IDs or expected balances. ### Set the fixture and target variables The configured `ledgerctl` profile must point to the disposable target cluster. Install Mike Farah `yq` v4 so the harness can compare the submitted YAML with the rules stored by Ledger. This example uses an OAuth2-protected HTTP source; omit the four OAuth2 flags at creation only when the fixture source intentionally allows unauthenticated log reads. ```bash set -euo pipefail for value in \ V2_URL V2_TOKEN SOURCE_LEDGER \ V2_CLIENT_ID V2_CLIENT_SECRET V2_TOKEN_ENDPOINT V2_SCOPE \ TARGET_URL TARGET_TOKEN TARGET_LEDGER RULE_FILE MIRROR_BATCH_SIZE \ MATCH_TX_ID NON_MATCH_TX_ID DROP_TX_ID \ SOURCE_REWRITE_ADDRESS TARGET_REWRITE_ADDRESS PASSTHROUGH_ADDRESS \ FIXTURE_ASSET EXPECTED_INPUT EXPECTED_OUTPUT EXPECTED_BALANCE; do test -n "${!value:-}" || { printf 'missing fixture variable: %s\n' "$value" >&2 exit 1 } done [[ "$MIRROR_BATCH_SIZE" =~ ^[1-9][0-9]*$ ]] || { printf 'MIRROR_BATCH_SIZE must be a positive integer\n' >&2 exit 1 } for transaction_id in "$MATCH_TX_ID" "$NON_MATCH_TX_ID" "$DROP_TX_ID"; do [[ "$transaction_id" =~ ^[0-9]+$ ]] || { printf 'replace every transaction ID placeholder with a numeric source ID\n' >&2 exit 1 } done for amount in "$EXPECTED_INPUT" "$EXPECTED_OUTPUT"; do [[ "$amount" =~ ^[0-9]+$ ]] || { printf 'expected input and output must be unsigned integer strings\n' >&2 exit 1 } done [[ "$EXPECTED_BALANCE" =~ ^-?[0-9]+$ ]] || { printf 'expected balance must be an integer string\n' >&2 exit 1 } test -f "$RULE_FILE" command -v awk >/dev/null command -v cmp >/dev/null command -v curl >/dev/null command -v date >/dev/null command -v grep >/dev/null command -v jq >/dev/null command -v ledgerctl >/dev/null command -v openssl >/dev/null command -v yq >/dev/null yq --version | grep -q 'mikefarah/yq.*version v4' ``` ### Hash the rules and create the disposable mirror ```bash for artifact in \ mirror-rules.sha256 \ mirror-rules.candidate.canonical.json \ mirror-rules.candidate.canonical.sha256 \ mirror-rules.target.canonical.json \ mirror-rules.target.sha256; do test ! -e "$artifact" || { printf 'rule evidence already exists; do not overwrite it: %s\n' \ "$artifact" >&2 exit 1 } done RULE_SHA256="$(openssl dgst -sha256 -r "$RULE_FILE" | awk '{print $1}')" printf '%s %s\n' "$RULE_SHA256" "$RULE_FILE" > mirror-rules.sha256 yq -o=json -I=0 '.' "$RULE_FILE" \ | jq -S -c ' def strip_proto_defaults: if type == "array" then map(strip_proto_defaults) elif type == "object" then with_entries(.value |= strip_proto_defaults) | with_entries(select( .value != null and .value != "" and .value != false and .value != [] )) else . end; strip_proto_defaults ' \ > mirror-rules.candidate.canonical.json CANDIDATE_RULES_SHA256="$(openssl dgst -sha256 -r \ mirror-rules.candidate.canonical.json | awk '{print $1}')" printf '%s %s\n' \ "$CANDIDATE_RULES_SHA256" \ mirror-rules.candidate.canonical.json \ > mirror-rules.candidate.canonical.sha256 ledgerctl ledgers create \ --name "$TARGET_LEDGER" \ --mode mirror \ --mirror-source-type http \ --mirror-ledger-name "$SOURCE_LEDGER" \ --mirror-base-url "$V2_URL" \ --mirror-oauth2-client-id "$V2_CLIENT_ID" \ --mirror-oauth2-client-secret "$V2_CLIENT_SECRET" \ --mirror-oauth2-token-endpoint "$V2_TOKEN_ENDPOINT" \ --mirror-oauth2-scopes "$V2_SCOPE" \ --mirror-batch-size "$MIRROR_BATCH_SIZE" \ --mirror-rewrite-file "$RULE_FILE" ``` `ledgerctl ledgers create` does not attach an idempotency key. After every attempt, recompute the input-file hash and read the unique target before continuing. If the command times out or loses its response, do not rerun it before this check: ```bash set -euo pipefail : "${RULE_FILE:?set RULE_FILE}" : "${TARGET_LEDGER:?set TARGET_LEDGER}" : "${SOURCE_LEDGER:?set SOURCE_LEDGER}" : "${V2_URL:?set V2_URL}" : "${V2_CLIENT_ID:?set V2_CLIENT_ID}" : "${V2_CLIENT_SECRET:?set V2_CLIENT_SECRET}" : "${V2_TOKEN_ENDPOINT:?set V2_TOKEN_ENDPOINT}" : "${V2_SCOPE:?set V2_SCOPE}" : "${MIRROR_BATCH_SIZE:?set MIRROR_BATCH_SIZE}" [[ "$MIRROR_BATCH_SIZE" =~ ^[1-9][0-9]*$ ]] command -v awk >/dev/null command -v cmp >/dev/null command -v jq >/dev/null command -v ledgerctl >/dev/null command -v openssl >/dev/null test -f "$RULE_FILE" test -f mirror-rules.sha256 test -f mirror-rules.candidate.canonical.json test -f mirror-rules.candidate.canonical.sha256 for artifact in \ mirror-rules.target.canonical.json \ mirror-rules.target.sha256; do test ! -e "$artifact" || { printf 'target-rule evidence already exists; do not overwrite it: %s\n' \ "$artifact" >&2 exit 1 } done EXPECTED_RULE_SHA256="$(awk '{print $1}' mirror-rules.sha256)" ACTUAL_RULE_SHA256="$(openssl dgst -sha256 -r "$RULE_FILE" | awk '{print $1}')" test "$ACTUAL_RULE_SHA256" = "$EXPECTED_RULE_SHA256" EXPECTED_CANDIDATE_RULES_SHA256="$(awk '{print $1}' \ mirror-rules.candidate.canonical.sha256)" ACTUAL_CANDIDATE_RULES_SHA256="$(openssl dgst -sha256 -r \ mirror-rules.candidate.canonical.json | awk '{print $1}')" test "$ACTUAL_CANDIDATE_RULES_SHA256" = "$EXPECTED_CANDIDATE_RULES_SHA256" FIXTURE_LEDGER_JSON="$(ledgerctl ledgers get "$TARGET_LEDGER" --json)" jq -e ' def enum_tail: split("_")[-1]; (.name == $ENV.TARGET_LEDGER) and ((.mode | enum_tail) == "MIRROR") and (.mirrorSyncProgress != null) and (.mirrorSource.ledgerName == $ENV.SOURCE_LEDGER) and (.mirrorSource.http.baseUrl == $ENV.V2_URL) and (.mirrorSource.http.oauth2ClientCredentials.clientId == $ENV.V2_CLIENT_ID) and (.mirrorSource.http.oauth2ClientCredentials.clientSecret == $ENV.V2_CLIENT_SECRET) and (.mirrorSource.http.oauth2ClientCredentials.tokenEndpoint == $ENV.V2_TOKEN_ENDPOINT) and (.mirrorSource.http.oauth2ClientCredentials.scopes == [$ENV.V2_SCOPE]) and (((.mirrorSource.batchSize // 0) | tostring) == $ENV.MIRROR_BATCH_SIZE) and (((.mirrorSource.rewriteRules // []) | length) > 0) ' <<<"$FIXTURE_LEDGER_JSON" >/dev/null jq -S -c ' def strip_proto_defaults: if type == "array" then map(strip_proto_defaults) elif type == "object" then with_entries(.value |= strip_proto_defaults) | with_entries(select( .value != null and .value != "" and .value != false and .value != [] )) else . end; .mirrorSource.rewriteRules | strip_proto_defaults ' <<<"$FIXTURE_LEDGER_JSON" \ > mirror-rules.target.canonical.json cmp -s \ mirror-rules.candidate.canonical.json \ mirror-rules.target.canonical.json TARGET_RULES_SHA256="$(openssl dgst -sha256 -r \ mirror-rules.target.canonical.json | awk '{print $1}')" test "$TARGET_RULES_SHA256" = "$EXPECTED_CANDIDATE_RULES_SHA256" printf '%s %s\n' \ "$TARGET_RULES_SHA256" \ mirror-rules.target.canonical.json \ > mirror-rules.target.sha256 ``` The canonical form removes explicitly written protobuf defaults such as `stop: false` and `match: ""`, which the structured ledger response normally omits. It does not remove objects, so action selectors such as `drop: {}` remain part of the comparison. The `cmp` and hash assertions prove that the complete normalized rule list, including order, matches the candidate. Keep the raw file hash and both canonical hashes with the fixture evidence. If the target exists and every source, progress, and rule check matches, creation committed; continue without another create request. If it exists with any mismatch, stop and preserve it for investigation. If the read is unavailable, retry only the read. Start a new creation attempt only after the target returns an explicit `NotFound`, using a new unique target name and fresh evidence files. The HTTP API accepts the same ordered list as `mirrorSource.rewriteRules` in the create-ledger body. ### Wait for the frozen fixture boundary ```bash SOURCE_HEAD_JSON="$(curl --fail --silent --show-error \ --header "Authorization: Bearer $V2_TOKEN" \ "$V2_URL/v2/$SOURCE_LEDGER/logs?pageSize=1")" FIXTURE_SOURCE_HEAD="$(jq -er ' .cursor.data | if type != "array" then error("source response has no cursor.data array") elif length == 0 then error("fixture source is empty") else (.[0].id | tostring) as $id | if $id | test("^[1-9][0-9]*$") then $id else error("newest fixture source log has an invalid id") end end ' <<<"$SOURCE_HEAD_JSON")" MIRROR_WAIT_DEADLINE=$((SECONDS + MIRROR_WAIT_TIMEOUT_SECONDS)) while :; do TARGET_LEDGER_JSON="$(ledgerctl ledgers get "$TARGET_LEDGER" --json)" jq -e '.mirrorSyncProgress != null' <<<"$TARGET_LEDGER_JSON" >/dev/null TARGET_MODE="$(jq -er '.mode | split("_")[-1]' <<<"$TARGET_LEDGER_JSON")" TARGET_CURSOR="$(jq -er '(.mirrorSyncProgress.cursor // 0) | tostring' <<<"$TARGET_LEDGER_JSON")" TARGET_SOURCE_HEAD="$(jq -er '(.mirrorSyncProgress.sourceLogCount // 0) | tostring' <<<"$TARGET_LEDGER_JSON")" TARGET_REMAINING="$(jq -er '(.mirrorSyncProgress.remainingLogs // 0) | tostring' <<<"$TARGET_LEDGER_JSON")" TARGET_ERROR_JSON="$(jq -c '.mirrorSyncProgress.error // null' <<<"$TARGET_LEDGER_JSON")" if [ "$TARGET_ERROR_JSON" != "null" ]; then printf 'fixture mirror error: %s\n' "$TARGET_ERROR_JSON" >&2 exit 1 fi if [ "$TARGET_MODE" != "MIRROR" ]; then printf 'fixture target left MIRROR mode\n' >&2 exit 1 fi if [ "$TARGET_CURSOR" = "$FIXTURE_SOURCE_HEAD" ] && [ "$TARGET_SOURCE_HEAD" = "$FIXTURE_SOURCE_HEAD" ] && [ "$TARGET_REMAINING" = "0" ]; then break fi if [ "$SECONDS" -ge "$MIRROR_WAIT_DEADLINE" ]; then printf 'fixture mirror timed out: cursor=%s observedSource=%s head=%s remaining=%s\n' \ "$TARGET_CURSOR" \ "$TARGET_SOURCE_HEAD" \ "$FIXTURE_SOURCE_HEAD" \ "$TARGET_REMAINING" >&2 exit 1 fi sleep 5 done ``` Keep the same Bash session. Define one fail-closed boundary check and run it immediately before querying fixture results: ```bash assert_fixture_boundary() { LATEST_SOURCE_HEAD_JSON="$(curl --fail --silent --show-error \ --header "Authorization: Bearer $V2_TOKEN" \ "$V2_URL/v2/$SOURCE_LEDGER/logs?pageSize=1")" LATEST_SOURCE_HEAD="$(jq -er ' .cursor.data | if type != "array" then error("source response has no cursor.data array") elif length == 0 then error("fixture source is empty") else (.[0].id | tostring) as $id | if $id | test("^[1-9][0-9]*$") then $id else error("newest fixture source log has an invalid id") end end ' <<<"$LATEST_SOURCE_HEAD_JSON")" TARGET_LEDGER_JSON="$(ledgerctl ledgers get "$TARGET_LEDGER" --json)" jq -e '.mirrorSyncProgress != null' <<<"$TARGET_LEDGER_JSON" >/dev/null LATEST_TARGET_MODE="$(jq -er '.mode | split("_")[-1]' <<<"$TARGET_LEDGER_JSON")" LATEST_TARGET_CURSOR="$(jq -er '(.mirrorSyncProgress.cursor // 0) | tostring' <<<"$TARGET_LEDGER_JSON")" LATEST_TARGET_SOURCE_HEAD="$(jq -er '(.mirrorSyncProgress.sourceLogCount // 0) | tostring' <<<"$TARGET_LEDGER_JSON")" LATEST_TARGET_REMAINING="$(jq -er '(.mirrorSyncProgress.remainingLogs // 0) | tostring' <<<"$TARGET_LEDGER_JSON")" LATEST_TARGET_ERROR_JSON="$(jq -c '.mirrorSyncProgress.error // null' <<<"$TARGET_LEDGER_JSON")" if [ "$LATEST_SOURCE_HEAD" != "$FIXTURE_SOURCE_HEAD" ] || [ "$LATEST_TARGET_MODE" != "MIRROR" ] || [ "$LATEST_TARGET_CURSOR" != "$FIXTURE_SOURCE_HEAD" ] || [ "$LATEST_TARGET_SOURCE_HEAD" != "$FIXTURE_SOURCE_HEAD" ] || [ "$LATEST_TARGET_REMAINING" != "0" ] || [ "$LATEST_TARGET_ERROR_JSON" != "null" ]; then printf 'fixture boundary changed; discard these comparison results\n' >&2 return 1 fi } assert_fixture_boundary ``` ### Assert match, non-match, drop, and balances ```bash MATCH_JSON="$(curl --fail --silent --show-error \ --header "Authorization: Bearer $TARGET_TOKEN" \ "$TARGET_URL/v3/$TARGET_LEDGER/transactions/$MATCH_TX_ID")" jq -e \ --arg source "$SOURCE_REWRITE_ADDRESS" \ --arg target "$TARGET_REWRITE_ADDRESS" \ ' .data.transaction.postings as $postings | (any($postings[]; .source == $target or .destination == $target)) and (all($postings[]; .source != $source and .destination != $source)) ' <<<"$MATCH_JSON" >/dev/null NON_MATCH_JSON="$(curl --fail --silent --show-error \ --header "Authorization: Bearer $TARGET_TOKEN" \ "$TARGET_URL/v3/$TARGET_LEDGER/transactions/$NON_MATCH_TX_ID")" jq -e \ --arg address "$PASSTHROUGH_ADDRESS" \ ' .data.transaction.postings | any(.[]; .source == $address or .destination == $address) ' <<<"$NON_MATCH_JSON" >/dev/null DROP_BODY_FILE="$(mktemp)" DROP_STATUS="$(curl --silent --show-error \ --output "$DROP_BODY_FILE" \ --write-out '%{http_code}' \ --header "Authorization: Bearer $TARGET_TOKEN" \ "$TARGET_URL/v3/$TARGET_LEDGER/transactions/$DROP_TX_ID")" test "$DROP_STATUS" = "404" jq -e '.errorCode == "TRANSACTION_NOT_FOUND"' "$DROP_BODY_FILE" >/dev/null TARGET_REWRITE_ADDRESS_URI="$(jq -rn \ --arg address "$TARGET_REWRITE_ADDRESS" \ '$address | @uri')" BALANCE_JSON="$(curl --fail --silent --show-error \ --header "Authorization: Bearer $TARGET_TOKEN" \ "$TARGET_URL/v3/$TARGET_LEDGER/accounts/$TARGET_REWRITE_ADDRESS_URI?collapseColors=true")" jq -e \ --arg address "$TARGET_REWRITE_ADDRESS" \ --arg asset "$FIXTURE_ASSET" \ --arg input "$EXPECTED_INPUT" \ --arg output "$EXPECTED_OUTPUT" \ --arg balance "$EXPECTED_BALANCE" \ ' (.data.address == $address) and any( .data.volumes[]; .asset == $asset and .color == "" and .volumes.input == $input and .volumes.output == $output and .volumes.balance == $balance ) ' <<<"$BALANCE_JSON" >/dev/null ``` Every command above must exit with status `0`. A missing transaction, unexpected `200` for the dropped ID, address mismatch, or volume mismatch fails the rehearsal. Rerun the boundary check after the result queries. This prevents the evidence bundle from combining an older source head with a newer target state: ```bash assert_fixture_boundary ``` ### Preserve the evidence ```bash EVIDENCE_DIR="rewrite-evidence-$TARGET_LEDGER" test ! -e "$EVIDENCE_DIR" || { printf 'evidence directory already exists; do not overwrite it: %s\n' \ "$EVIDENCE_DIR" >&2 exit 1 } mkdir "$EVIDENCE_DIR" cp "$RULE_FILE" "$EVIDENCE_DIR/rules.yaml" cp mirror-rules.sha256 "$EVIDENCE_DIR/rules.sha256" cp mirror-rules.candidate.canonical.json \ "$EVIDENCE_DIR/candidate-rules.canonical.json" cp mirror-rules.candidate.canonical.sha256 \ "$EVIDENCE_DIR/candidate-rules.canonical.sha256" jq -n \ --arg sourceLogId "$FIXTURE_SOURCE_HEAD" \ '{sourceLogId: $sourceLogId}' \ > "$EVIDENCE_DIR/source-head.initial.json" jq -n \ --arg sourceLogId "$LATEST_SOURCE_HEAD" \ '{sourceLogId: $sourceLogId}' \ > "$EVIDENCE_DIR/source-head.confirmed.json" printf '%s\n' "$TARGET_LEDGER_JSON" \ | jq '{name, mode, mirrorSyncProgress}' \ > "$EVIDENCE_DIR/target-progress.json" cp mirror-rules.target.canonical.json \ "$EVIDENCE_DIR/target-rules.canonical.json" cmp -s \ "$EVIDENCE_DIR/candidate-rules.canonical.json" \ "$EVIDENCE_DIR/target-rules.canonical.json" openssl dgst -sha256 -r "$EVIDENCE_DIR/target-rules.canonical.json" \ > "$EVIDENCE_DIR/target-rules.sha256" printf '%s\n' "$MATCH_JSON" > "$EVIDENCE_DIR/match-transaction.json" printf '%s\n' "$NON_MATCH_JSON" > "$EVIDENCE_DIR/non-match-transaction.json" printf '%s\n' "$BALANCE_JSON" > "$EVIDENCE_DIR/rewritten-balance.json" cp "$DROP_BODY_FILE" "$EVIDENCE_DIR/drop-response.json" ``` Keep the disposable mirror until a reviewer accepts the evidence. Delete it afterward with `ledgerctl ledgers delete "$TARGET_LEDGER" --yes`; never promote a fixture mirror. The Stack v4 preview does not surface rewrite rules in its own manifests. If you drive Ledger through the ledger operator's `Ledger` CRD directly, set `spec.mirrorSource.rewriteRules`, which the controller forwards to `ledgerctl` as `--mirror-rewrite-rule`. Otherwise create the mirror with `ledgerctl` or the HTTP API. Do not infer Stack manifest fields. ## Promotion gate for transformed mirrors - [ ] The exact rule file and its hash are in the migration record. - [ ] Every rule has positive, negative, missing-field, and boundary fixtures. - [ ] Every output address is valid and collision behavior is approved. - [ ] Typed metadata includes invalid-value behavior. - [ ] Every dropped log and its accounting impact is explained. - [ ] Balance, volume, reference, reversion, and metadata comparisons account for the rules. - [ ] The production mirror uses the same rule order as the accepted rehearsal. Continue with [Monitor the Mirror](/modules/ledger/migration/monitor-mirror). --- ## Track Fund Origin with Colors Source: https://docs.formance.com/modules/ledger/modeling/color-of-money A color adds an origin or restriction to a posting. Ledger then keeps a separate balance for each `(account, asset, color)` instead of grouping everything by account and asset. Use colors when funds with the same asset must not satisfy each other's spending rules. Examples include restricted grants, promotional balances, reserved funds, or funds tied to a specific source program. Do not use colors as display labels. Use [metadata](/modules/ledger/modeling/metadata) when values may be grouped together for spending and only need descriptive context. To create colored balances and watch isolation enforced against a live ledger, follow [Colors](/modules/ledger/hands-on#colors-isolated-balances-within-one-asset) in the hands-on walkthrough. Read [Create Transactions with Postings](/modules/ledger/transactions/create-with-postings) for retry rules and [Read Ledger Data](/modules/ledger/querying/read-ledger-data#read-an-account) for the account response contract. ## Balance buckets are isolated Suppose Alice holds three `USD/2` buckets: | Color | Balance | Meaning | | --- | ---: | --- | | `""` | 10000 | Uncolored funds | | `GRANTS` | 5000 | Restricted grant funds | | `OPS` | 2500 | Operations budget | A posting that requests `GRANTS` can spend only the `GRANTS` balance. It cannot fall back to the uncolored or `OPS` buckets, even when their combined balance would cover the amount. The empty string is a real bucket. Omitting `color` on a posting selects the uncolored bucket; it does not mean “use any color.” ## Write a colored posting Color follows the value from source to destination. One posting cannot debit `GRANTS` and credit `OPS`; changing color in place would break conservation inside each bucket. Color identifiers must contain only uppercase ASCII letters (`A` through `Z`) and can contain at most 32 characters. The empty string remains valid for uncolored funds. ## Balances are returned per bucket Ledger returns one volume row per `(asset, color)`, including `color: ""` for the uncolored bucket: ```json { "data": { "address": "customers:alice:wallets:main", "volumes": [ { "asset": "USD/2", "color": "", "volumes": { "input": "10000", "output": "0", "balance": "10000" } }, { "asset": "USD/2", "color": "GRANTS", "volumes": { "input": "5000", "output": "0", "balance": "5000" } }, { "asset": "USD/2", "color": "OPS", "volumes": { "input": "2500", "output": "0", "balance": "2500" } } ] } } ``` Ledger enforces sufficiency per bucket: spending `5001` from a `GRANTS` bucket holding `5000` is rejected with `INSUFFICIENT_FUNDS`, and the rejection is atomic — neither the source nor the destination account changes. The [hands-on walkthrough](/modules/ledger/hands-on#colors-isolated-balances-within-one-asset) runs that probe and asserts both accounts are untouched. ## Use collapsed views only for reporting Set `collapseColors=true` when a report needs one total per asset; the same option works on aggregate volume queries. Collapsed responses sum the colored rows and return the result with `color: ""`. They do not merge or recolor stored balances. Never use a collapsed balance to authorize a spend that must respect color restrictions. Treat `collapseColors=true` as a presentation choice. The returned empty color represents a summary across buckets, while an uncollapsed empty color represents the actual uncolored bucket. ## Spend a color with Numscript Numscript can restrict a source to one color. The current syntax requires the asset-colors feature directive: ```numscript #![feature("experimental-asset-colors")] send [USD/2 1000] ( source = @customers:alice:wallets:main \ "GRANTS" destination = @merchants:acme:earnings ) ``` The emitted posting carries `color: "GRANTS"`, and the destination receives the same color. A source without a color restriction reads only the uncolored bucket: `@account \ ""` and `@account` name the same source. There is no "any color" source. A script that may draw from more than one bucket lists those buckets in an ordered source, and each posting it emits carries the color of the bucket the funds came from. Ledger keeps each color as a separate balance even though the current Numscript syntax requires an explicit feature directive. ## Reversals keep the original color Reverting a colored transaction creates reverse postings with the same color. Funds return to the bucket from which they moved; they do not return as uncolored value. Verify this behavior in tests that read the original source after a reversal. Its matching colored balance should recover, while other colors remain unchanged. ## Change a business classification explicitly Ledger has no in-place recoloring primitive. Reclassify value with one transaction that contains two equal postings: 1. retire the old color by sending it to `world`; 2. issue the same amount from `world` with the new color. Keeping both postings in one transaction is mandatory. Ledger commits both or neither, and one idempotency key protects the whole exchange. Crediting the new color in a separate transaction can temporarily double the customer's value if the old-color debit fails or is delayed. The [hands-on walkthrough](/modules/ledger/hands-on#reclassify-value-between-colors) moves `1000 USD/2` from `GRANTS` to `OPS` and asserts both bucket changes plus the invariant total. Production clients must parse Ledger's balance strings with arbitrary-precision integer arithmetic. Do not replace `world` with a two-posting clearing hop. `GRANTS` and `OPS` are separate balances, so they cannot cancel each other inside the clearing account. A clearing design needs additional postings that return every `(asset, color)` bucket to zero in the same transaction. Keep the approval reference in transaction metadata so reviewers can explain why the classification changed. ## Current limits - List-accounts and list-transactions do not expose a native color filter. Read the relevant account or aggregate volumes and filter the returned rows in your application. - Color is not derived from account or transaction metadata. Every posting that must remain separate must carry the intended color. - There is no primitive that changes the color of existing funds in place. - A direct-posting insufficient-funds error identifies the exact color bucket. A Numscript insufficient-funds error cannot yet identify which color was short and omits the color from its error metadata. When Numscript error metadata has no color field, read it as “color unresolved,” not as “the uncolored bucket failed.” Direct-posting errors can identify the empty uncolored bucket explicitly. ## Common failure modes | Symptom | Cause | Correction | | --- | --- | --- | | A spend fails although the account total is sufficient | The requested color bucket is short | Read uncollapsed volumes and fund the intended bucket. | | A posting is rejected as invalid | The color contains lowercase letters, digits, separators, or exceeds 32 characters | Use a stable uppercase identifier such as `GRANTS`. | | A recipient gets an unexpected colored balance | Color follows the posting to the destination | Include color handling in the destination's spending and reconciliation logic. | | A collapsed response appears to lose provenance | The response intentionally summed buckets under `color: ""` | Remove `collapseColors=true` when provenance matters. | | A Numscript error has no `color` metadata | The interpreter cannot resolve the failing bucket in that error path | Inspect the script source and uncollapsed source balances. | | Reports cannot list every transaction for one color server-side | Native list filtering by color is not available | Filter posting colors client-side or store a separate lookup in your application. | Before production use, test direct postings, Numscript, reversals, collapsed reads, and insufficient-funds behavior with at least one uncolored and two colored buckets. --- ## Request and Response Signing Source: https://docs.formance.com/modules/ledger/operations/signing Use request signing when you need evidence that a registered key approved the exact composition and order of an admission batch. Response signing places a signed copy of each response log in an embedded envelope. The current `ledgerctl` verification path authenticates that envelope, but it does not prove that the outer `Log` consumed by the command matches the signed copy. Both mechanisms use Ed25519, but they use different keys and protect different directions. Request signing is available on the gRPC `Apply` path and on `ledgerctl` mutation commands that use that path. The HTTP API does not expose the signed protobuf envelope, and some current `ledgerctl` commands call unsigned unary RPCs instead. ## Keep the security layers separate | Mechanism | Question it answers | What it does not prove or replace | | --- | --- | --- | | [Authentication](/modules/ledger/operations/authentication) | Who is the caller, and which operations may it perform? | TLS, payload approval, or replay protection | | [TLS](/modules/ledger/operations/tls-and-network-security) | Is the connection encrypted, and is the peer certificate trusted? | User authorization or payload approval | | Request signature | Did a registered private key approve these exact ordered mutations? | JWT scopes, TLS, or idempotency | | Response signature | Did the expected Ledger response key sign the bytes in this embedded `SignedLog` envelope? | Proof that the outer returned `Log` matches those bytes, or persistence of the proof | Request signatures cover the serialized `ApplyBatch`, including its ordered requests and idempotency key. Ledger persists the signed batch proof once per batch on that batch's audit entry, and binds it into the audit hash chain, so the proof is tamper-evident and stays verifiable long after the call. Retrieve it through the audit API; it is not attached to individual logs. A signature alone does not prevent replay. A gRPC client can make a retry idempotent only by persisting the original idempotency key and resending the exact same batch. For each log in an `ApplyResponse`, Ledger signs a serialized copy and places that copy in the outer log's `SignedLog` envelope. Ledger creates the envelope after consensus and does not persist it. Clients that need the envelope as evidence must retain the signed response. `ledgerctl --response-verify-key` verifies the signature over `SignedLog.payload`, but the verification helper does not compare that payload with the separate outer `Log`. Commands such as `transactions create` then read data from the outer log. A successful command therefore authenticates the embedded envelope only. It does not establish that the transaction or other data displayed or consumed from the outer log is the data that Ledger signed. Do not use this check alone as a tamper-rejection guarantee. Until a released client binds the two representations and consumes the signed payload, rely on TLS and your established peer trust for the outer response. ## Bootstrap request signing Request signing is optional until you explicitly require it. Plan the bootstrap as a controlled change: 1. restrict access to the cluster; 2. inventory every application, administrative command, and scheduled mutation against the compatibility table below; 3. complete or remove incompatible scheduled and background work — client-driven work only; the leader's own scheduled proposals are exempt; 4. register the first public key; 5. verify signed writes; 6. require signatures; 7. verify that an unsigned admission mutation fails; 8. when authentication is enabled, verify that a direct cluster-control RPC rejects a JWT without `ledger:ClusterWrite`; 9. reopen normal access. ### Check mutation compatibility Mandatory request signing in the current 3.0 preview gates admission batches only. It rejects several public and internal paths that submit unsigned `Apply` requests, but it does not add signature checks to direct cluster-control RPCs. Enabling the policy without this preflight can stop configuration changes while other administrative mutations remain outside the signing policy. | Mutation surface | Behavior while signatures are mandatory | Safe gate before enabling | | --- | --- | --- | | HTTP mutations | Rejected because HTTP always creates an unsigned `ApplyRequest` | Move every writer to a tested signed gRPC `Apply` path | | Query-checkpoint create or delete through the unary gRPC methods used by `ledgerctl query-checkpoint` | The server submits an unsigned mutation and rejects it | Finish checkpoint-based jobs and delete their checkpoints before enabling; see [Consistency and Checkpoints](/modules/ledger/querying/consistency-and-checkpoints#request-signing-limitations) | | Scheduled query-checkpoint creation | Unaffected. The scheduler's proposal is leader-internal and exempt from the signature requirement, because admission returns early for a system actor before it reads `require_signatures` | None | | Prepared-query create, update, or delete through HTTP or the unary gRPC methods used by `ledgerctl queries` | Both direct mutation surfaces submit unsigned requests and are rejected | Use the signed [configuration-as-code](/modules/ledger/operations/configuration-as-code) apply path or a version-pinned client that signs the exact `ApplyBatch` | | Cluster membership changes, leadership transfer, primary or secondary compaction, primary-store checkpoint creation, and full or incremental backup | These direct cluster RPCs do not enter Admission and accept no request-signature envelope. `require_signatures` does not cover them; with authentication enabled, they require a JWT carrying `ledger:ClusterWrite` | Protect cluster administration credentials and network access, and test scope denial separately. Do not treat the signing policy as approval evidence for these operations | Reads, including prepared-query execution and checkpoint-scoped reads, remain unsigned and continue to work. Direct cluster-control commands also remain outside request signing, even when the CLI receives `--signing-key` flags. Record the Ledger and `ledgerctl` versions used for this inventory; do not assume that a global signing flag changes a command whose implementation does not use `Apply`. The first signing key can be registered without a **request signature** because no trusted signing key exists yet. It still requires a valid JWT and `ledger:OpsWrite` when authentication is enabled. Register the first key in a controlled maintenance window. Generate the root keypair: ```bash ROOT_KEY_DIR="./root-signing-key" test ! -e "$ROOT_KEY_DIR" || { printf 'refusing to overwrite existing root key directory: %s\n' "$ROOT_KEY_DIR" >&2 exit 1 } ledgerctl signing generate-key "$ROOT_KEY_DIR" ``` Store `seed.hex` in your approved secret manager and verify that an authorized operator can recover it before registering the public key. After signatures become mandatory, losing or overwriting the only trusted root seed can prevent every future key-management mutation while its public-key registration remains present. Register the public key. The unsigned bootstrap path is available whenever Ledger has no signing-key registrations: ```bash ledgerctl signing register-key \ --key-id root \ --public-key-file ./root-signing-key/pubkey.hex ``` Create a ledger through a signed batch. The `--signing-key-id` must match the ID registered above: ```bash ledgerctl \ --signing-key ./root-signing-key/seed.hex \ --signing-key-id root \ ledgers create --name signing-smoke ``` List the registered keys and confirm that `root` has `(root)` as its parent: ```bash ledgerctl signing list-keys ``` Once the signed write succeeds, require a valid request signature for admission mutation batches: ```bash ledgerctl \ --signing-key ./root-signing-key/seed.hex \ --signing-key-id root \ signing require true ``` Reads remain unsigned. Admission mutations must now provide a registered key ID and valid signature. The direct cluster-control RPCs listed above remain outside this policy and continue to depend on authentication and scope enforcement. ## Verify admission enforcement An unsigned admission mutation must fail after `signing require true`: ```bash ledgerctl ledgers create --name unsigned-must-fail ``` Ledger maps a missing required signature to gRPC `Unauthenticated`. A mutation signed with an unknown key ID, the wrong private key, or a modified payload fails with gRPC `PermissionDenied`. The same mutation succeeds when the client provides the registered key: ```bash ledgerctl \ --signing-key ./root-signing-key/seed.hex \ --signing-key-id root \ ledgers create --name signed-write ``` If authentication is also enabled, repeat the test with a JWT that lacks `ledger:LedgerWrite`. A cryptographically valid request must still fail with `PermissionDenied`: signing approves the payload, while authorization decides whether the caller may execute it. ## Delegate with child keys Register operational keys under the root instead of distributing the root seed. The key that signs a registration becomes the immutable parent of the new key. ```bash APPLICATION_KEY_DIR="./application-signing-key" test ! -e "$APPLICATION_KEY_DIR" || { printf 'refusing to overwrite existing application key directory: %s\n' "$APPLICATION_KEY_DIR" >&2 exit 1 } ledgerctl signing generate-key "$APPLICATION_KEY_DIR" ledgerctl \ --signing-key ./root-signing-key/seed.hex \ --signing-key-id root \ signing register-key \ --key-id application \ --public-key-file ./application-signing-key/pubkey.hex ``` Use the child key for application writes: ```bash ledgerctl \ --signing-key ./application-signing-key/seed.hex \ --signing-key-id application \ transactions create \ --ledger signing-smoke \ --posting "world,users:001,1000,USD/2" ``` List the hierarchy and verify that `application` reports `root` as its parent: ```bash ledgerctl signing list-keys ``` ## Revoke keys Revoke one key without affecting its descendants: ```bash ledgerctl \ --signing-key ./root-signing-key/seed.hex \ --signing-key-id root \ signing revoke-key --key-id application ``` Use `--cascade` only when every descendant must stop signing immediately: ```bash ledgerctl \ --signing-key ./root-signing-key/seed.hex \ --signing-key-id root \ signing revoke-key --key-id compromised-team-key --cascade ``` After the command commits, retry a mutation with each affected key and confirm that Ledger rejects it. Key registration, revocation, and signing policy changes pass through Raft; consider the change complete only after the committed result is visible and the behavioral check fails as expected. Cascade revocation follows the complete descendant tree. Do not register a replacement below a compromised key and then cascade-revoke that parent: the replacement will be revoked as well. Revocation removes keys from the active registry; Ledger does not retain an inactive registry entry for them. The revocation remains in log and audit history. If a revocation or cascade removes the last registration, the unsigned first-key bootstrap path opens again even when `require_signatures` remains `true`. Before removing the last key, restrict access to the cluster and prepare a replacement public key. Register that replacement with a valid JWT carrying `ledger:OpsWrite` when authentication is enabled, verify a signed admission mutation, and confirm the new registration before reopening access. ## Enable response signing Response signing uses one server seed supplied at startup. Generate a separate keypair; do not reuse a request-signing key: ```bash RESPONSE_KEY_DIR="./response-signing-key" test ! -e "$RESPONSE_KEY_DIR" || { printf 'refusing to overwrite existing response key directory: %s\n' "$RESPONSE_KEY_DIR" >&2 exit 1 } ledgerctl signing generate-key "$RESPONSE_KEY_DIR" ``` Operator-managed response signing is not available in this Stack v4 preview. Before clients require response verification, use a released workflow that mounts one protected seed on every replica and reports full convergence. To verify the current server contract in a disposable direct deployment, add the seed path to every Ledger process and restart one member at a time: ```bash --response-signing-key ./response-signing-key/seed.hex ``` This is a Ledger server flag, not a Stack v4 manifest field. Keep the seed out of command output and source control. Wait until every replica has restarted with the same key before requiring clients to verify signatures. ## Verify signed responses Pass the response **public** key to `ledgerctl`. This example checks the embedded response envelope and signs the request when request signatures are mandatory: ```bash ledgerctl \ --signing-key ./root-signing-key/seed.hex \ --signing-key-id root \ --response-verify-key ./response-signing-key/pubkey.hex \ transactions create \ --ledger signing-smoke \ --posting "world,response:probe,1,TEST" ``` The command fails if any outer log has no embedded response signature, or if that envelope carries a malformed signature or fails verification with the supplied public key. As described above, this check does not compare the signed payload with the outer log that the command consumes. SDKs can retrieve the current server public key through the unauthenticated Discovery RPC: ```bash grpcurl \ -cacert ./ca.crt \ ledger-payments.formance.svc.cluster.local:8888 \ ledger.BucketService/Discovery ``` Pin the discovered key according to your trust policy. Discovery transports the key; it does not by itself establish that a newly observed key is authorized. ## Rotate signing keys ### Request keys Register the replacement as a child of a still-valid key, deploy the new private seed to its client, and verify a signed mutation. Revoke the old key without `--cascade` only after all intended descendants have been reviewed. Keep the JWT credential rotation separate; a request-signing key does not mint client permissions. ### Response key Response signing accepts one server key at a time. Distribute and trust the new public key before updating the Kubernetes Secret and rolling Ledger nodes. During the rollout, clients pinned to only one key cannot verify envelopes from both old and new replicas. Route clients that require envelope verification to a converged deployment or coordinate a client trust window that accepts both keys. After convergence, run the envelope check through every reachable replica, then retire the old public key. This confirms that each replica uses the expected response key; it does not remove the current outer-log binding limitation. ## Recover from signing failures | Symptom | Likely cause | Recovery | | --- | --- | --- | | `missing signature` / `Unauthenticated` | Signatures are required but the mutation was unsigned | For a command documented as signing-capable, add `--signing-key` and the matching `--signing-key-id`; otherwise follow the owning page's compatibility guidance | | `unknown key ID` / `PermissionDenied` | Client key ID differs from the registered ID, or the key was revoked | Run `signing list-keys`; correct the ID or register a replacement through a valid parent | | `invalid signature` / `PermissionDenied` | Wrong seed, modified payload, or corrupted signature | Stop retrying the modified batch; rebuild it from the intended inputs and sign once | | Signed request receives an authorization error | JWT is missing or lacks the operation's scope | Fix authentication and scopes; do not weaken signing policy | | `missing response signature` | Client envelope verification is enabled before response signing has converged | Complete the deployment rollout and confirm that every replica returns an envelope signed by the expected key | | `response signature verification failed` | Client has the wrong public key or reached a replica using another response key | Compare Discovery output, the configured Secret, and rollout revision | | Ambiguous timeout after a signed gRPC batch | The server may have committed the batch before the client lost the response | Reconcile the result; if it is still unknown, resend the persisted signed envelope byte for byte with the original idempotency key | | Ambiguous timeout after a `ledgerctl` mutation | The current CLI request path has no idempotency-key option and puts an empty idempotency key in the batch | Read the affected state and audit or log evidence; do not rerun the CLI mutation blindly | | No signing-key registrations remain | The last registration was removed by a revocation or cascade | Keep the cluster isolated, register a new root through the unsigned bootstrap path using a valid JWT with `ledger:OpsWrite` when authentication is enabled, then verify a signed admission mutation before reopening access | | Registrations remain, but no matching usable private key remains | The corresponding seeds were lost, destroyed, or made unusable | Keep the cluster isolated and use the release-specific break-glass recovery procedure. Unsigned bootstrap remains closed while any registration exists | The gRPC retry rule applies only when the client retained the exact serialized batch or signed envelope. Rebuilding a protobuf message and changing its requests, ordering, or idempotency key creates a different operation. Current `ledgerctl` commands that use `BucketService` can automatically retry gRPC `Unavailable` failures within one invocation, while their generated batches still carry an empty idempotency key. A failed or timed-out command can therefore have applied its mutation more than once. Reconcile the intended state and audit or log evidence even before considering a manual retry. Retry through the CLI only after you have proved that the intended state change did not commit and have accepted that the new call is not protected by a batch idempotency key. To turn mandatory request signing off during a planned rollback, sign the policy change with a currently valid key: ```bash ledgerctl \ --signing-key ./root-signing-key/seed.hex \ --signing-key-id root \ signing require false ``` Record who approved the change and when unsigned mutations became possible again. --- ## Create Prepared Queries Source: https://docs.formance.com/modules/ledger/querying/prepared-queries A prepared query stores a named filter inside one ledger. Applications execute the name with runtime parameters instead of rebuilding the filter in every service. Use a prepared query when a filter is a shared application contract, needs review before deployment, or must remain consistent across several callers. Keep an ad hoc filter when one caller owns a short-lived operational search. The HTTP examples use `jq` and the `LEDGER_URL`, `LEDGER_NAME`, and `CURL_AUTH` variables defined in [Read Ledger Data](/modules/ledger/querying/read-ledger-data). Listing and executing prepared queries require `ledger:QueryRead`; creating, updating, and deleting them require `ledger:QueryWrite`. See [Authentication and Authorization](/modules/ledger/operations/authentication). When mandatory request signing is enabled in the current 3.0 preview, the create, update, and delete examples on this page fail through both HTTP and `ledgerctl queries`: those direct paths submit unsigned mutations. Manage prepared-query definitions through a reviewed [configuration-as-code](/modules/ledger/operations/configuration-as-code) file and its signed `ledgerctl ledgers configuration apply` path, or use a version-pinned gRPC client that signs the exact `ApplyBatch`. Listing and executing queries remain read operations and continue to work. ## Prerequisites Choose one target: | Target | `LIST` result | `AGGREGATE_VOLUMES` | | --- | --- | --- | | `ACCOUNTS` | Accounts | Supported | | `TRANSACTIONS` | Transactions | Not supported | | `LOGS` | Ledger logs | Not supported | The server supports `LIST` execution for all three targets over HTTP and gRPC. Then verify that: - every metadata field has a declared type for the selected target; - every condition that needs an opt-in index has that index ready on the serving replica; - the query name identifies its business purpose rather than its current implementation; - callers have query-read permission and the deployment path has query-write permission. Prepared queries are scoped to one ledger. They do not read across ledgers. The create and update APIs accept textual or structured filters, but the representations are not equivalent for every condition. In particular, a transaction reference is available only as a structured `$match`. Review the [filter representation table](/modules/ledger/querying/filters-and-pagination#choose-a-filter-representation) before storing a query. ## Create a parameterized query with ledgerctl The textual filter grammar represents a parameter as `$name`: ```bash ledgerctl queries create accounts-by-status \ --ledger "$LEDGER_NAME" \ --target accounts \ --filter 'metadata[status] == $status' ``` Ledger parses and validates the filter before storing it. A duplicate name is rejected; creation does not update an existing query. List the stored definitions to verify the target and normalized filter: ```bash ledgerctl queries list --ledger "$LEDGER_NAME" --json ``` There is no single-query read operation. Use the list command or HTTP list route and select the name in the client. ## Create a query with HTTP The REST request can carry a structured filter: ```bash curl --fail-with-body --silent --show-error \ --request POST \ "${CURL_AUTH[@]}" \ --header "Content-Type: application/json" \ --header "Idempotency-Key: create-accounts-by-status-query" \ --data '{ "name": "accounts-by-status", "target": "ACCOUNTS", "filter": { "$match": { "metadata[status]": {"$param": "status"} } } }' \ "$LEDGER_URL/v3/$LEDGER_NAME/prepared-queries" ``` A successful creation returns `204 No Content`. Confirm it with: ```bash curl --fail-with-body --silent --show-error \ "${CURL_AUTH[@]}" \ "$LEDGER_URL/v3/$LEDGER_NAME/prepared-queries" ``` The create and update bodies also accept a textual filter as a JSON string. The list response exposes the stored filter in structured form. ## Execute a list query The `ledgerctl` result renderer handles accounts and transactions only. Execute a `LOGS` query through HTTP or a gRPC client so log rows are returned to the caller. Do not use `ledgerctl queries execute --all` for an export in this release; follow `cursor.next` yourself until `hasMore` is false. For an account or transaction query, execute with `ledgerctl` and provide each parameter as `key=value`: ```bash ledgerctl queries execute accounts-by-status \ --ledger "$LEDGER_NAME" \ --param status=active \ --page-size 100 \ --json ``` The CLI sends parameter values as strings. Ledger converts them according to the typed field used by the filter. A value that cannot be converted is rejected rather than compared as a different type. The equivalent HTTP request is: ```bash curl --fail-with-body --silent --show-error \ --request POST \ "${CURL_AUTH[@]}" \ --header "Content-Type: application/json" \ --data '{ "mode": "LIST", "parameters": {"status": "active"}, "pageSize": 100 }' \ "$LEDGER_URL/v3/$LEDGER_NAME/prepared-queries/accounts-by-status/execute" ``` The execution response is not wrapped in a top-level `data` field. A list result has this shape: ```json { "cursor": { "pageSize": 100, "hasMore": true, "next": "opaque-cursor", "accountData": [ {"address": "users:alice", "metadata": {"status": "active"}} ] } } ``` The row field follows the target: `accountData`, `transactionData`, or `logData`. When `hasMore` is true, send `cursor.next` unchanged in the next request: ```json { "mode": "LIST", "parameters": {"status": "active"}, "pageSize": 100, "cursor": "opaque-cursor" } ``` The HTTP route also accepts `pageSize` and `cursor` as query parameters. Query-parameter values override the request body. Use one location consistently so application logs show the effective request clearly. Keep the query name, parameters, page size, and any minimum sequence unchanged throughout the cursor chain. Follow `cursor.next` yourself until `hasMore` is false. ## Include a completed write Add `minLogSequence` only when the prepared query must include a write that just completed. Derive it from the full gRPC `ApplyResponse`; do not use a transaction ID, timestamp, or estimated sequence: ```bash MIN_LOG_SEQUENCE="$( jq -er ' .logs[-1].sequence // error("the response contains no new log sequence") | tostring ' apply-response.json )" ``` `ApplyResponse.logs` is returned in commit order, so the last item is the highest sequence produced by the proposal. If the response has no fresh log, that call produced no new sequence to wait for, so the filter raises an error rather than yielding the string `null`, which `minLogSequence` would otherwise accept. Prepared-query execution checks the local read-index position once. When it is behind, the gRPC API returns `READ_INDEX_NOT_CAUGHT_UP` instead of waiting. Application clients should retry only that structured `google.rpc.ErrorInfo.reason`, with a bound. In Ledger v3.0, `ledgerctl` does not print the structured reason. Its stable signal for that same error is the server message containing the requested sequence. The loop below matches both that message and the exact captured sequence; every other failure stops immediately: ```bash QUERY_READY=false EXPECTED_LAG_PREFIX="read index has not caught up to sequence $MIN_LOG_SEQUENCE (current:" for attempt in {1..30}; do if ledgerctl queries execute accounts-by-status \ --ledger "$LEDGER_NAME" \ --param status=active \ --page-size 100 \ --min-log-sequence "$MIN_LOG_SEQUENCE" \ --json \ >prepared-page.json 2>prepared-query.err then QUERY_READY=true break fi if ! grep -Fq "$EXPECTED_LAG_PREFIX" prepared-query.err; then cat prepared-query.err >&2 exit 1 fi sleep 2 done if [ "$QUERY_READY" != true ]; then cat prepared-query.err >&2 exit 1 fi ``` The HTTP execution body accepts the same number in its `minLogSequence` field; it is also a one-shot freshness gate, not a wait. Use the gRPC reason above when a client needs to distinguish the retryable lag from other failures in v3.0. This freshness gate does not cover an index backfill or metadata-index version transition. Verify the field declaration and index readiness separately. Prepared-query execution does not accept a query checkpoint ID in Ledger v3.0. See [Choose Read Consistency and Use Checkpoints](/modules/ledger/querying/consistency-and-checkpoints#wait-for-the-read-index) for the complete sequence-capture workflow. ## Aggregate matching account volumes An account prepared query can aggregate the input, output, and balance of every matching account: ```bash ledgerctl queries execute accounts-by-status \ --ledger "$LEDGER_NAME" \ --param status=active \ --mode aggregate \ --json ``` With HTTP, set `mode` to `AGGREGATE_VOLUMES`. The response contains `aggregateResult` rather than `cursor`: ```json { "aggregateResult": { "volumes": [ { "asset": "USD/2", "color": "", "input": "15000", "output": "2500", "balance": "12500" } ] } } ``` Aggregation is computed when the request runs. It is not a precomputed reporting table, so include its scan cost in capacity tests. The HTTP response computes `balance` as `input - output`. The gRPC `AggregateResult` used by `ledgerctl` carries `input` and `output` but no balance field; compute that difference in the client when using gRPC. ## Evolve a query without surprising callers An update replaces the filter atomically and keeps the target unchanged: ```bash ledgerctl queries update accounts-by-status \ --ledger "$LEDGER_NAME" \ --filter 'metadata[status] == $status and address ^= "users:"' ``` Prepared queries do not expose version history through the read API. The audit trail records updates, but callers can retrieve only the current filter. For a business-critical change, prefer a versioned rollout: 1. create a new name such as `accounts-by-status-v2`; 2. execute it with representative parameter sets; 3. compare its results with the existing query; 4. move callers to the new name; 5. remove the old query after the rollback window. Changing a metadata field declaration does not convert existing stored values or validate every prepared query that uses the field. Review the existing values and prepared queries, then wait for the metadata index's new local version before depending on the new comparison semantics. ## Delete a query Delete a query only after every caller has stopped using its name: ```bash ledgerctl queries delete accounts-by-status --ledger "$LEDGER_NAME" ``` Or use HTTP: ```bash curl --fail-with-body --silent --show-error \ --request DELETE \ "${CURL_AUTH[@]}" \ "$LEDGER_URL/v3/$LEDGER_NAME/prepared-queries/accounts-by-status" ``` A later execution of that name returns `NotFound`. Recreating the same name creates a new current definition; it does not restore the deleted definition's history. ## Diagnose execution failures | Failure | Check | | --- | --- | | Missing parameter | Send every `$param` referenced by the stored filter | | Parameter conversion error | Match the value to the declared metadata type | | Index not found | Declare the index for the exact target and condition | | HTTP `503` with `INDEX_BUILDING` | Wait for readiness on the serving replica and retry with bounded backoff | | `READ_INDEX_NOT_CAUGHT_UP` | Retry the same request with bounded backoff; stop on every other error | | Invalid mode | Use `LIST` for any target; use `AGGREGATE_VOLUMES` only for accounts | | Query no longer compiles | Review recent metadata schema and index changes | | Unexpected page contents | Keep parameters, mode, page size, and cursor chain stable | Continue with [Analyze and Aggregate Ledger Data](/modules/ledger/querying/analyze-and-aggregate) when an application needs totals rather than rows. --- ## Revert Transactions Source: https://docs.formance.com/modules/ledger/transactions/revert A revert creates a new transaction whose postings reverse the original transaction. Ledger keeps the original record, marks it as reverted, and links the two transactions. Use a revert when the whole accounting effect must be undone. Create a separate correcting transaction when only part of the amount or destination changes. ## Prerequisites You need an existing normal ledger and a token with `ledger:write` for the create and revert operations plus `ledger:read` for the transaction and account checks. The virtual scopes assume the default scope mapping. If the ledger enforces account types, replace the demo wallet with an address that matches its configured patterns. ```bash set -eo pipefail ``` The example creates a USD 25.00 deposit that can be reverted safely: ```bash CREATE_RESPONSE=$(curl --fail-with-body --silent --show-error \ --request POST "$LEDGER_URL/v3/$LEDGER_NAME/transactions" \ --header "Authorization: Bearer $LEDGER_TOKEN" \ --header "Content-Type: application/json" \ --header "Idempotency-Key: $DEPOSIT_REFERENCE:create" \ --data "$(jq --compact-output --null-input \ --arg reference "$DEPOSIT_REFERENCE" \ --arg destination "$WALLET_ACCOUNT" \ '{ reference: $reference, postings: [ {source: "world", destination: $destination, amount: 2500, asset: "USD/2"} ] }')") TRANSACTION_ID=$(jq -er \ '.data.transaction.id | tostring | select(test("^[1-9][0-9]*$"))' \ <<<"$CREATE_RESPONSE") ``` ## Revert the transaction Send a new idempotency key for the revert. Metadata belongs to the compensating transaction, not to the original one. ```bash jq --null-input \ --arg caseId "$REVERT_CASE_ID" \ '{ metadata: { reason: "customer_refund", case_id: $caseId } }' \ | curl --fail-with-body --silent --show-error \ --request POST "$LEDGER_URL/v3/$LEDGER_NAME/transactions/$TRANSACTION_ID/revert" \ --header "Authorization: Bearer $LEDGER_TOKEN" \ --header "Content-Type: application/json" \ --header "Idempotency-Key: $DEPOSIT_REFERENCE:revert" \ --data @- \ > /tmp/reverted-transaction.json ``` Ledger returns `201 Created`. Inspect the relationship and the new postings: ```bash jq '.data | { revertedTransactionId, revertTransaction: { id: .revertTransaction.id, revertsTransactionId: .revertTransaction.revertsTransactionId, timestamp: .revertTransaction.timestamp, metadata: .revertTransaction.metadata, postings: .revertTransaction.postings, postCommitVolumes: .revertTransaction.postCommitVolumes } }' /tmp/reverted-transaction.json ``` Expected result: - `revertedTransactionId` equals the original transaction ID; - `revertTransaction.revertsTransactionId` also points to the original; - the new posting sends `2500` units of `USD/2` from `$WALLET_ACCOUNT` to `world`; - the new transaction contains the refund metadata; - the wallet returns to a zero balance. ## Verify both records Read the original transaction: ```bash curl --fail-with-body --silent --show-error \ "$LEDGER_URL/v3/$LEDGER_NAME/transactions/$TRANSACTION_ID" \ --header "Authorization: Bearer $LEDGER_TOKEN" \ | jq -e --arg id "$TRANSACTION_ID" ' .data.transaction | select( (.id | tostring) == $id and .reverted == true and (.revertedByTransactionId | tostring | test("^[1-9][0-9]*$")) ) | {id, reverted, revertedAt, revertedByTransactionId}' ``` `reverted` must be `true`, and `revertedByTransactionId` must identify the compensating transaction. Then verify the balance independently: ```bash curl --fail-with-body --silent --show-error \ "$LEDGER_URL/v3/$LEDGER_NAME/accounts/$WALLET_ACCOUNT" \ --header "Authorization: Bearer $LEDGER_TOKEN" \ | jq -e ' .data.volumes[] | select(.asset == "USD/2" and .color == "") | .volumes.input == "2500" and .volumes.output == "2500" and .volumes.balance == "0"' ``` Expected volume: ```json { "input": "2500", "output": "2500", "balance": "0" } ``` ## Choose the effective timestamp By default, the compensating transaction uses the time of the revert. Set `atEffectiveDate` when reports must place the compensation at the original transaction's effective timestamp: ```json { "atEffectiveDate": true, "metadata": { "reason": "effective_date_correction" } } ``` `insertedAt` still shows when Ledger recorded the revert. Review historical reports and date-based reconciliation rules before choosing the original effective date. ## Revert when funds have moved A normal revert fails when reversing the postings would take an account below its allowed balance. Investigate where the funds moved first. A separate correction may represent the business event more accurately. If policy requires a full reversal despite the resulting negative balance, the REST request accepts `force: true`: ```json { "force": true, "metadata": { "reason": "operator_approved_forced_revert", "case_id": "case-0045" } } ``` A forced revert bypasses the normal balance check. It can create negative balances in accounts that spent the original funds. Require explicit approval, capture the case ID in metadata, and verify every account in the returned `revertTransaction.postCommitVolumes`. ## Handle errors | Result | Meaning | Action | | --- | --- | --- | | `400` | Invalid body or the reversal violates a balance or account rule | Inspect the structured reason. Correct the request or follow the approved forced-revert procedure. | | `401` / `403` | Missing credentials or insufficient scope | Repair authentication or grant `ledger:write`. | | `404` | The ledger or original transaction cannot be found | Check the environment and ID. | | `409` with `IDEMPOTENCY_KEY_CONFLICT` | The key was already used with different content | Restore the exact original body for an unchanged retry. If the original operation record is unavailable, stop; reading the target transaction does not identify the outcome stored under that key. | | `409` with `TRANSACTION_ALREADY_REVERTED` | A compensating transaction already exists | Read the original and verify `revertedByTransactionId`. Do not issue another correction automatically. | | `503` | Ledger cannot currently accept the revert; the outcome may be unknown | Retry the identical request with the same idempotency key, over HTTP or direct gRPC. | ## Recover after a timeout or lost response Read the original after the cluster is healthy enough to serve a fresh read: ```bash ORIGINAL=$(curl --fail-with-body --silent --show-error \ "$LEDGER_URL/v3/$LEDGER_NAME/transactions/$TRANSACTION_ID" \ --header "Authorization: Bearer $LEDGER_TOKEN") jq -e --arg id "$TRANSACTION_ID" ' .data.transaction | select((.id | tostring) == $id) | {id, reverted, revertedAt, revertedByTransactionId}' \ <<<"$ORIGINAL" ``` If `reverted` is `true`, the revert committed. Read the compensating transaction identified by `revertedByTransactionId`; do not send another revert: ```bash ( REVERT_TRANSACTION_ID=$(jq -er \ '.data.transaction.revertedByTransactionId // 0 | tostring' \ <<<"$ORIGINAL") if [[ -z "$REVERT_TRANSACTION_ID" || "$REVERT_TRANSACTION_ID" == "0" ]]; then echo "No compensating transaction is linked; do not retry yet." >&2 exit 1 fi curl --fail-with-body --silent --show-error \ "$LEDGER_URL/v3/$LEDGER_NAME/transactions/$REVERT_TRANSACTION_ID" \ --header "Authorization: Bearer $LEDGER_TOKEN" \ | jq -e --arg original "$TRANSACTION_ID" ' .data.transaction | select((.revertsTransactionId | tostring) == $original) | {id, revertsTransactionId, postings, metadata}' ) ``` Then read every account affected by the original postings. For the example on this page: ```bash curl --fail-with-body --silent --show-error \ "$LEDGER_URL/v3/$LEDGER_NAME/accounts/$WALLET_ACCOUNT" \ --header "Authorization: Bearer $LEDGER_TOKEN" \ | jq -e ' .data.volumes[] | select(.asset == "USD/2" and .color == "") | .volumes.input == "2500" and .volumes.output == "2500" and .volumes.balance == "0"' ``` Use these outcomes to decide what happens next: - If the original is reverted, the compensating transaction points back to it, and balances match the reversal, record success and stop. - If the original is not reverted, wait for the cluster to recover, read it and the affected balances again, then make an explicit retry decision. - If the transaction link or balances are inconsistent, or a fresh read is unavailable, stop the workflow and investigate. A new revert or correction would make the accounting state harder to reconcile. For the HTTP example earlier on this page, keep the same `Idempotency-Key` and body after a timeout or `503`. For direct gRPC automation, keep the same `ApplyBatch.idempotency_key` and batch. Never replace the key while the first outcome is unknown. ## Correct a completed revert A revert is itself an immutable transaction. Ledger does not delete it or restore the original transaction to an unreverted state. If a revert was incorrect, model the required business correction as a new reviewed transaction and preserve the complete audit trail. Continue with [Process Operations in Bulk](/modules/ledger/transactions/bulk) when several independent corrections are submitted together. --- ## Capabilities Source: https://docs.formance.com/modules/payments/capabilities The Payments module provides a connector framework that integrates with payment providers and exposes a stable set of **capabilities** per connector — fetching accounts, balances, payments, initiating transfers, and so on. This page is the per-connector capability inventory. You can also create your own custom connectors by implementing the [Connector Framework](/modules/payments/connectors/build-connector) and taking inspiration from the existing connectors in the GitHub repository. For operational guidance — polling cadence, monitoring schedules, health checks, and recovery — see [Operations](/modules/payments/operations). ## How to read this page The matrix below is generated at build time from [`docs/other/connector-capabilities.json`](https://github.com/formancehq/payments/blob/main/docs/other/connector-capabilities.json) in `formancehq/payments`, one snapshot per Payments release. Flip the **Payments** picker in the sidebar to see what a stack pinned to that release exposes — new connectors appear, capabilities added later light up. The **Edition** column shows which connectors ship in the open-source Community edition (CE) versus the Enterprise edition (EE). EE connectors require an EE license to install. ## PSP connectors PSP connectors model traditional payment service providers — single-asset balance movements on a single account leg. See [PSP Connectors](/modules/payments/connectors/psp) for the catalogue. ## Exchange connectors Exchange connectors surface trading-venue activity. Two primitives that don't exist on traditional PSPs are surfaced as their own columns from Payments 3.3.0: **Orders** (instructions placed against an order book) and **Conversions** (atomic two-asset swaps that bypass the order book). See [Exchange Connectors](/modules/payments/connectors/exchange). ## Open Banking connectors Open Banking connectors are read-only data aggregators backed by [Payment Service Users](/modules/payments/payment-service-users). See [Open Banking](/modules/payments/connectors/open-banking). ## Capability definitions #### Accounts Fetch the internal accounts from the payment provider. Internal accounts are accounts where you have operational control, such as digital e-wallets or payment service provider accounts. Maps to `CAPABILITY_FETCH_ACCOUNTS` upstream. #### External Accounts Fetch external accounts, which represent accounts used as destinations for transfers, such as the user's bank accounts. Maps to `CAPABILITY_FETCH_EXTERNAL_ACCOUNTS`. #### Balances Fetch the internal-account balances. None of the upstream providers expose historical balance series, so balances are always point-in-time. Maps to `CAPABILITY_FETCH_BALANCES`. #### Payments Fetch payments happening on the provider and relate them to the accounts previously fetched. Maps to `CAPABILITY_FETCH_PAYMENTS`. #### Other Fetches Pull provider-specific objects that don't fit any of the standard streams (mandates, beneficiaries, …). Surfaced under `metadata` on the relevant resource. Maps to `CAPABILITY_FETCH_OTHERS`. #### Orders Fetch trading orders placed on the provider — instructions to exchange one asset for another at a price. Each order carries a direction (BUY or SELL), a source and destination asset, a type (MARKET, LIMIT, STOP_LIMIT, TWAP, VWAP, PEG, BLOCK, RFQ), a status that follows the fill lifecycle (PENDING, OPEN, PARTIALLY_FILLED, FILLED, CANCELLED, EXPIRED, FAILED), ordered and filled quantities, time-in-force, fees, source and destination account references, and an append-only adjustment list capturing every observed state change. Maps to `CAPABILITY_FETCH_ORDERS` (Payments 3.3.0+). #### Conversions Fetch atomic two-asset swaps — moving a balance from one asset to another at the provider, with distinct source and destination amounts. Each conversion carries source and destination assets and amounts at each side's precision, fees, a status (PENDING, COMPLETED, FAILED), and source and destination account references. Conversions are commonly used for stablecoin redemption (1:1 swaps such as USDC ↔ USD) and for FX-style asset swaps. Maps to `CAPABILITY_FETCH_CONVERSIONS` (Payments 3.3.0+). #### Bank Account Creation Create a bank account on the payment provider side. Useful when you want to create a bank account for a user and then initiate a payout to that bank account. Maps to `CAPABILITY_CREATE_BANK_ACCOUNT`. #### Transfer Initiation Initiate a transfer from one internal account to another internal account. Maps to `CAPABILITY_CREATE_TRANSFER`. #### Payout Initiation Initiate a payout to an external account from an internal account. Maps to `CAPABILITY_CREATE_PAYOUT`. #### Webhooks Subscribe to provider webhooks and translate them into Payments events. The matrix collapses the upstream `CAPABILITY_CREATE_WEBHOOKS` and `CAPABILITY_TRANSLATE_WEBHOOKS` pair into a single column — they always ship together. --- ## Generic Connector Source: https://docs.formance.com/modules/payments/connectors/generic The Generic connector is a thin shim over an HTTP service you run yourself. Use it when no Formance-shipped connector exists for the upstream you need to integrate, or when you want the integration to live in your own infrastructure for compliance, vendor-neutrality, or operational reasons. Formance's side stays generic: the platform calls a small set of HTTP endpoints on your service to fetch accounts, balances, and payments, and forwards transfer / payout initiation requests when configured. Your service is responsible for the upstream-specific glue — auth, rate limiting, format conversion. ## Available pages Install the Generic connector pointed at your HTTP service. The wire protocol Formance expects from your service — endpoints, payloads, error semantics. ## When to reach for it The Generic connector is the right tool when: - You're integrating with a provider Formance doesn't ship a connector for, and waiting for one isn't an option. - You already operate a service that talks to the upstream, and want Payments to consume it through a stable contract instead of duplicating the integration. - You need the integration in your own VPC for regulatory reasons, with Formance reading from it across a controlled network boundary. For first-party Formance-shipped integrations, see [PSP Connectors](/modules/payments/connectors/psp), [Exchange Connectors](/modules/payments/connectors/exchange), and [Open Banking](/modules/payments/connectors/open-banking). To contribute a new first-party connector upstream, see [Build a connector](/modules/payments/connectors/build-connector). --- ## Chart of Accounts Source: https://docs.formance.com/modules/ledger/core-concepts/chart-of-accounts A chart of accounts is the set of accounts you model in your ledger and the conventions you use to name them. Get this right early and the rest of your integration falls into place — Numscripts read naturally, queries filter cleanly, and an auditor can read your books without a tour guide. ## Structuring Account Hierarchies Use colons (`:`) to compose address segments into a hierarchy. The segments are just strings — Ledger doesn't interpret them — but consistent conventions let you query, group, and reason about accounts at scale. ``` users:{user_id}:wallet:main users:{user_id}:wallet:pending users:{user_id}:payment:{payment_id} merchants:{merchant_id}:earnings merchants:{merchant_id}:settlements platform:fees platform:revenue platform:taxes:{tax_type} orders:{order_id}:authorization orders:{order_id}:capture orders:{order_id}:refund ``` ### Why this works 1. **Filtering**: query every account under a prefix (e.g. `users:123:*`). 2. **Organization**: related accounts live next to each other in any listing. 3. **Scalability**: adding new account types is just another segment — no restructure. ## How much of your treasury do you model in the ledger? The `world` account is a built-in "infinite source" — it can always send funds, regardless of balance. It's the natural placeholder for "money came from outside" without committing to *where* outside. Whether to lean on `world` or to model your asset side explicitly is the central design question for a product ledger. There's a spectrum, and where you sit depends on what role you want the ledger to play. ### Use `world` as the asset side (default for fintech product ledgers) For most fintech use cases — third-party funds ledgering, embedded finance, neobank-style products — the product ledger's job is to track **liabilities**: what the platform owes to each user, merchant, partner, or internal pool. The asset side (the actual cash sitting in safeguarding accounts, FBO arrangements, custody providers) is a treasury concern, often spread across many providers, and changing that map shouldn't ripple through your product code. A user deposit might land across six different safeguarding accounts depending on currency, region, partner bank, and risk policy. Folding all of that into the same ledger means every product feature has to know about treasury topology. Instead, the product ledger treats funds as coming **from `world`** and lets a separate system reconcile against the actual asset-side accounts. ```numscript // Product ledger only cares that Alice now has a claim worth $100. send [USD/2 10000] ( source = @world destination = @users:alice:wallet ) ``` This keeps the product ledger small, focused on the liability side, and decoupled from the asset-side complexity. ### Model the asset side explicitly (when treasury belongs in the ledger) If you *do* want the ledger to double as your treasury view — to know exactly which bank account funds sit in, or to enforce that liabilities match asset coverage — replace `world` with named asset accounts: ``` treasury:bank:main treasury:bank:secondary treasury:bank:eu treasury:bank:us:main treasury:custody:circle ``` A deposit then names its actual landing place: ```numscript send [USD/2 10000] ( source = @treasury:bank:us:main allowing unbounded overdraft destination = @users:alice:wallet ) ``` This trades simplicity for tighter coupling: every deposit / payout / settlement flow has to know which asset account moved, and a treasury reshuffle (new safeguarding partner, account migration) is a ledger change. It's the right call when the ledger *is* your treasury system, less so when treasury is a distinct concern handled upstream. ### Pool the asset side (paired with Payments cash pools) The two approaches above pull in opposite directions — one keeps the ledger blissfully unaware of treasury topology, the other pulls every bank account into the chart. A pairing with the Payments module's [cash pools](/modules/payments/cash-pools) lets you have both at once. Model a small, stable set of **pool accounts** in the ledger — each one standing for "value held somewhere across this group of underlying accounts": ``` treasury:pool:eu treasury:pool:us treasury:pool:cards treasury:pool:wires ``` A deposit names the pool, not the specific bank: ```numscript send [USD/2 10000] ( source = @treasury:pool:us destination = @users:alice:wallet ) ``` Each ledger pool has a matching Payments cash pool on the operational side, aggregating the real-world PSP / safeguarding / FBO accounts. A [reconciliation policy](/modules/reconciliation/getting-started) compares the ledger pool's balance against the Payments cash pool's aggregate balance and flags any drift. The product ledger stays small and stable; treasury reshuffles happen entirely on the Payments side (new partner bank? add it to the cash pool query — the ledger doesn't change). This is usually the right answer for fintechs that have grown past "one account at one partner" but don't want every flow in the product to depend on the precise asset layout. ### Other legitimate `world` uses Regardless of the broader choice, `world` is the natural source for: - Initial system bootstrapping. - Minting new assets you control end-to-end (loyalty points, tokens, in-app credits). - Non-funded currency conversions (see [Currency Conversion](/modules/ledger/working-with/currency-conversion)). ## Specific account addresses on the liability side Whichever side of the asset-modeling spectrum you pick, name your **liability-side** accounts with as much context as you can: ``` @users:alice:wallet:main @users:alice:payment:order-123 @merchants:acme:earnings @platform:fees ``` The address is your audit trail and your query surface — `users:alice:*` should list every account belonging to Alice, `payment:order-123` should let you trace a single business event end-to-end. Generic names like `revenue` or `pending` lose that context the moment you have two of them. For handling negative balances and overdraft in your account model, see [Double-entry Bookkeeping](/modules/ledger/core-concepts/double-entry#overdraft-and-negative-balances). For worked examples of complete account hierarchies for e-commerce, lending, and wallet products, see [Account Hierarchy Examples](/examples/account-hierarchies). --- ## Monitor the Mirror Source: https://docs.formance.com/modules/ledger/migration/monitor-mirror Monitor deployment provisioning and Ledger mirror progress as separate signals. They answer different questions: - deployment provisioning confirms that the target service and ledger were created; - a mirror is *following* once it has applied every source entry it has seen — formally, the cursor is at or past a non-zero source head the target has persisted; - neither state proves that source and target business data match. The Stack v4 preview does not define supported Operator status fields for this workflow. Keep deployment provisioning and Ledger mirror progress as separate signals; the fields below describe Ledger progress only. ## Read progress Read the ledger's mirror progress with `ledgerctl`: ```bash set -euo pipefail : "${TARGET_LEDGER:?set TARGET_LEDGER}" command -v ledgerctl >/dev/null ledgerctl ledgers get "$TARGET_LEDGER" ``` For automation: ```bash set -euo pipefail : "${TARGET_LEDGER:?set TARGET_LEDGER}" command -v jq >/dev/null command -v ledgerctl >/dev/null ledgerctl ledgers get "$TARGET_LEDGER" --json \ | jq -e '.mirrorSyncProgress' ``` Or use HTTP: ```bash set -euo pipefail : "${TARGET_URL:?set TARGET_URL}" : "${TARGET_TOKEN:?set TARGET_TOKEN}" : "${TARGET_LEDGER:?set TARGET_LEDGER}" command -v curl >/dev/null command -v jq >/dev/null curl --fail --silent --show-error \ --header "Authorization: Bearer $TARGET_TOKEN" \ "$TARGET_URL/v3/$TARGET_LEDGER" \ | jq -e '.data.mirrorSyncProgress' ``` ## Interpret the fields | Field | Meaning | Healthy behavior | | --- | --- | --- | | `state` | `MIRROR_SYNC_STATE_SYNCING` while the cursor is below the persisted source head; `MIRROR_SYNC_STATE_FOLLOWING` when the head is non-zero and `cursor >= sourceLogCount`. An empty source stays syncing. The field carries the full declared enum name, and because syncing is the enum's default value it is omitted from the JSON entirely — treat an absent `state` as syncing rather than as a contract violation. | It may alternate while Ledger v2 continues receiving writes. Following with `cursor > sourceLogCount` is anomalous. | | `cursor` | Highest Ledger v2 log ID successfully applied to the target. | Increases monotonically during replay. | | `sourceLogCount` | Source head observed by the most recent successful source query and persisted by the target — by an ingestion batch, or on its own by an idle, caught-up worker. It is not read live at request time. | Advances as the worker observes new source logs. If the source is unreachable it stays stale, because a failed head query is logged but never recorded as an error. | | `remainingLogs` | `sourceLogCount - cursor` when the persisted source head is ahead; otherwise `0`. | Trends down during catch-up and repeatedly returns to zero while following. A zero value does not detect `cursor > sourceLogCount`. | | `error` | Most recent source, translation, transformation, or apply error that Ledger persisted. | The next successful poll clears it, whether or not that poll ingested any logs. An error that persists across polls means the failure is still active. | `mode` differs between directions: a create request takes the bare `MIRROR`, while a read returns the full `LEDGER_MODE_MIRROR` and omits the field entirely for a normal ledger. Compare reads against the full value. Ledger refreshes the source head in memory while polling and publishes it — together with an error clear — on the next poll, including a poll that returns no logs. A worker that has never successfully read the source head publishes nothing at all, so the progress object cannot distinguish an idle healthy source from a worker that failed before its first source read. Following compares the cursor with a source head persisted by the target. It does not refresh the source, query business invariants, compare balances, or certify promotion safety. Treat `cursor > sourceLogCount` as an inconsistent progress tuple even though the current state calculation reports `FOLLOWING` and zero remaining logs. ## Decide what state you are in | Observation | Interpretation | Next action | | --- | --- | --- | | Cursor increases and remaining logs decrease | Replay is progressing. | Continue monitoring source and target load. | | State alternates between following and syncing under live writes | The target catches new source tails. | Confirm it repeatedly returns to zero remaining logs. | | Cursor is unchanged and an error is present | The failure is still active: a successful poll would have cleared the error even with no logs to ingest. | Read the error message and act on it; Ledger retries without skipping the failed entry. | | All counters are unchanged and no error is visible | The source may be idle, the worker may not be running because of initialization, leadership, or storage pressure, or the source may be unreachable for head queries — that last case is logged as `Failed to query source head` and deliberately not recorded as an error. | Read the source head directly, inspect target logs, and search them for `Failed to query source head`. For a new PostgreSQL mirror, check for `Failed to create mirror source`. | | Following, remaining zero, no error | The target reached its persisted source head. | Compare the cursor with a fresh source read, then run independent validation; do not promote from status alone. | | Following with cursor greater than source head | The persisted progress tuple is inconsistent or stale. `remainingLogs` still reports zero. | Stop status-based automation, capture the tuple and server version, compare the source directly, and investigate before validation. | | Ledger reports `LEDGER_MODE_MIRROR`, progress missing | The response violates the current Ledger contract: every readable mirror ledger receives a progress object, including when all counters are zero. | Stop status-based automation, capture a credential-safe response and server version, compare another read surface, and investigate serialization or runtime health. | For an intentionally empty source, confirm directly that the Ledger v2 log list is empty and that `cursor`, `sourceLogCount`, and `remainingLogs` are all `0`. Do not wait for the following state in that case. A visible error means the current poll cycle is still failing, so treat it as a live condition rather than as history: the worker clears it on its own once a poll succeeds, and it needs no synthetic financial log or manual intervention to do so. The one case that produces no error at all is a source the worker cannot query for its head, which is logged rather than recorded — so a wholly static tuple with no error still warrants a direct source read. ## Diagnose a stalled mirror Work from non-destructive checks toward configuration changes. 1. Record the cursor, source head, error message, and error timestamp. 2. Confirm the target cluster has a leader and sufficient disk headroom. 3. Test the source path from the target environment. 4. Inspect the Ledger v2 entry immediately after the cursor. 5. Identify the rule scope and actions that apply to that entry. 6. Decide whether the source can be corrected safely or the mirror must be recreated. | Error area | Check | Recovery | | --- | --- | --- | | HTTP connectivity | DNS, TLS, route, source status, and `/v2/{ledger}/logs` response. | Restore reachability; the worker retries automatically. | | OAuth2 | Client ID, token endpoint, client secret, scopes, and token expiry. | If immutable credentials must change, recreate the mirror. | | PostgreSQL | DNS, TLS mode, database, `_system.ledgers`, bucket table, `CONNECT`, schema `USAGE`, and table `SELECT` grants. | Restore access; recreate only when the stored connection configuration changes. | | RDS IAM | Target Ledger runtime identity, region, `rds-db:connect`, optional assume role, and TLS mode. | Correct IAM or recreate when region/role configuration changes. | | Source log | Log type, JSON shape, posting amount, target, timestamp, and referenced transaction. | Correct the source condition only with an approved Ledger v2 procedure. | | Transformation | Missing field, CEL type, `valueExpr`, RE2 pattern, typed conversion, or invalid output address. | Correct the rule and create a new mirror. Rules cannot change in place. | | Target storage | Disk pressure, write stall, compaction, and cluster availability. | Restore storage headroom; ingestion resumes after the stall clears. | | Leadership | Leader election or member availability. | Restore quorum and leadership; the new leader resumes the mirror worker. | For a PostgreSQL source, Ledger resolves the source ledger's bucket before it starts the mirror worker. A connection, authentication, or bucket-lookup failure at this stage is written to the target Ledger logs as `Failed to create mirror source`; it may leave `cursor`, `sourceLogCount`, and `remainingLogs` at `0` with no progress error. Inspect target logs whenever a new PostgreSQL mirror remains at zero, even when `.mirrorSyncProgress.error` is absent. ## Understand retry behavior When a source fetch, translation, rule action, or target apply fails: - the cursor does not advance past the failed batch; - Ledger records the error and retries with backoff; - Ledger does not automatically discard the failing source entry; - the next successful poll clears the recorded error and resumes progress, whether or not it ingested any logs; - an idle poll that returns no logs still publishes the observed source head, so a visible error means the failure is still live rather than historical. Do not promote to bypass a blocked batch. Promotion stops replication at the current cursor and makes that incomplete projection writable. ## Establish alerts Alert on user-visible conditions rather than a single instantaneous state: - cursor unchanged for longer than the measured healthy interval; - remaining logs increasing across several samples; - any mirror error that persists across retries; - target disk pressure or write stalls; - source authorization or TLS failures; - target without a leader or quorum; - production mirror no longer reporting `LEDGER_MODE_MIRROR` before the approved cutover. Choose thresholds from the rehearsal throughput and source-write profile. A fixed lag threshold without that context can hide a slow migration or alert on normal bursts. ## Ready for validation Proceed to [Validate the Migration](/modules/ledger/migration/validation) when: - the cursor advances normally; - the target repeatedly reaches the following state with zero remaining logs, or both ledgers are confirmed empty; - the cursor never exceeds the persisted source head; - a fresh Ledger v2 read confirms that the cursor reached the current source head; - no unresolved mirror error remains; - the exact source and rewrite configuration matches the accepted rehearsal; - required indexes are ready; - source and target load remain inside approved limits. These conditions start validation. They do not replace it. Continue with [Validate the Migration](/modules/ledger/migration/validation) once every condition above holds. --- ## Monitor Ledger Source: https://docs.formance.com/modules/ledger/operations/monitoring Monitor Ledger from the user-visible outcome back to the cluster. A useful alert says which capability is at risk, which cluster or member caused it, what evidence confirms it, and which runbook owns the response. ## Before you connect telemetry Decide and record: - the OpenTelemetry collector and storage backend; - the metric naming mode used by the Ledger server and collector; - the stable environment, Stack, cluster, and node attributes; - retention for metrics, traces, and logs; - service-level objectives for write availability, current reads, and latency; - an owner and runbook for every paging alert. The Stack Operator must preserve `service.cluster` and `service.node_id` on every replica. Without both attributes, a slow member and a cluster-wide outage look alike. The dashboards Ledger ships use exactly these two attributes as their template variables, so treat the names as a contract rather than an implementation detail. ## Choose one metric naming path Ledger emits application metrics in OpenTelemetry dot notation by default. For example: - `raft.node.lead`; - `raft.fsm.logs_appended`; - `raft.apply_entries.duration`; - `admission.propose_queue.full`; - `pebble.write_stall.active`; - `storage.disk.volume.bytes`; - `usage.builder.lag`. With the server in `prom` naming mode (`--metrics-naming`, `otel` by default), Ledger prefixes application metrics with `ledger_` and replaces dots with underscores. A collector may also append unit or counter suffixes. OpenTelemetry auto-instrumented metrics such as `http.server.request.duration`, `go.*`, and `process.*` follow the collector's normalization rules and do not receive the Ledger prefix. Choose the dashboard that matches both the Ledger naming mode and the collector's normalization. Do not rename an existing production stream without migrating dashboards and alerts in the same change; a silent name mismatch removes monitoring while the service keeps running. Ledger ships a generated Grafana dashboard for each combination of naming mode, collector normalization, and histogram representation, so you do not have to rewrite queries by hand. The seven variants are committed to [`misc/devenv/monitoring-dashboards/config/dashboards`](https://github.com/formancehq/ledger/tree/release/v3.0/misc/devenv/monitoring-dashboards/config/dashboards) in the Ledger repository, and its [README](https://github.com/formancehq/ledger/blob/release/v3.0/misc/devenv/monitoring-dashboards/README.md) maps every file to the server flag, collector behavior, and histogram representation it expects. Pick the one matching your pipeline: `ledger-metrics-prom-noprefix-normalized-native.json` covers the common case of default `otel` naming with full OpenTelemetry-to-Prometheus normalization and native histograms. Configure OTLP export, authentication, naming mode, and resource attributes only through the release-specific Stack Operator fields documented for your selected Stack version. Do not copy provisional field names into a production resource. ## Verify the telemetry path Run this precheck before loading dashboards: 1. Confirm that every voter exports a process heartbeat and a distinct node label. 2. Confirm that all voters share the intended cluster label. 3. Send a controlled read and write in a test ledger. 4. Find the corresponding HTTP or gRPC trace and the server-side span. HTTP server spans come from the `ledger-http-server` instrumentation. A write then shows the `admission.*` span family — `admission.verify_signatures`, `admission.preload`, `admission.propose`, `admission.fsm_wait`, `admission.marshal` — and a read shows the `ctrl.*` family. 5. Confirm that request duration, response status, and route attributes appear without unbounded IDs in labels. 6. Compare `raft.fsm.logs_appended` before and after the write. 7. Compare `ledgerctl cluster status` with the reported leader and member signals. 8. Stop one telemetry collector in staging and confirm that Ledger requests continue while telemetry delivery recovers. Telemetry export must not become a dependency of the write path. ## Build the operational dashboard Keep the primary dashboard focused on decisions: | Question | Signals to correlate | | --- | --- | | Can the cluster commit? | Known leader, healthy voter count, commit/apply positions, proposal errors. | | Are users succeeding? | Request rate, latency by operation, response status, retry reasons. | | Is consensus under pressure? | Apply duration, WAL append duration, peer latency, queue load and full events. | | Can storage sustain the workload? | WAL/data bytes, disk headroom, flush/compaction duration, active write stalls. | | Will a process be killed? | Working memory, Go heap goal, container limit, restarts and OOM events. | | Can applications query? | Required index state and lag, read latency, consistency failures. | | Are usage statistics current? | Usage-builder processed position, available audit head, and lag on every serving member. | | Are asynchronous workflows current? | Event-sink cursor/error, backup job state and last verified artifact. | Use separate detail dashboards for consensus queues, Pebble, indexes, usage projection, event sinks, and backup jobs. The landing dashboard should answer whether users can write and read now. For Mirror progress and promotion proofs, use the migration-specific [Mirror monitoring procedure](/modules/ledger/migration/monitor-mirror). ## Monitor usage statistics Ledger serves per-ledger counters through the ledger `stats` endpoint and `ledgerctl ledgers stats`. Applications and finance dashboards read them, so a stalled projection surfaces as numbers that quietly stop moving rather than as a failed request. The nine returned fields do not share one source. `transactionCount` and `logCount` come from Raft-applied ledger boundaries and are always current. The projected counters — `volumeCount`, `postingCount`, `referenceCount`, `revertCount`, `numscriptExecutionCount`, `ephemeralEvictedCount`, and `transientUsedCount` — along with Numscript template usage, are rebuilt from the audit chain into a local derived store on each member. Those can trail committed accounting data briefly during normal operation and for longer after an upgrade or restore. When a stats read carries a checkpoint, every projected counter returns `0`. Read that `0` as "not available at this checkpoint", not as an empty ledger; only `transactionCount` and `logCount` are checkpoint-consistent. `metadataCount` is not returned at all, and `ephemeralEvictedCount` and `transientUsedCount` are informational — the integrity checker excludes them when it re-derives ledger boundaries. | Metric | Healthy interpretation | | --- | --- | | `usage.builder.last_indexed_sequence` | Advances as audit entries are reflected in the local usage store. | | `usage.builder.audit_last_sequence` | Tracks the latest audit sequence available from the primary store. | | `usage.builder.lag` | Returns to zero after normal writes, startup catch-up, an upgrade, or a restore. | Compare all three gauges on every serving member. `usage.builder.lag` counts sequences, not time: it is the latest available audit sequence minus the last one indexed locally. A large but decreasing lag is expected during a known rebuild; an unchanged or growing lag needs investigation, and so does `last_indexed_sequence` standing still while `audit_last_sequence` keeps moving. `usage.builder.lag = 0` means the member processed the audit history still available to it. It does not prove that archived pre-reset history contributed to the counters, and neither the primary-store integrity checker nor `restore validate` checks this local projection. ## Start with actionable alerts The conditions below are initial triggers, not universal capacity promises. Validate them under representative load and tune them against your service objectives. | Alert | Initial trigger | User impact | Verify | First action | | --- | --- | --- | --- | --- | | No leader | Every replica reports `raft.node.lead = 0` for 30 seconds | Writes and linearizable reads cannot complete | `ledgerctl cluster status` | Restore a voter majority and peer connectivity; honor `Retry-After` while elections remain transient. | | Lost failure tolerance | Fewer healthy voters than the intended topology | The next failure can stop commits | Cluster and Operator status | Stop rollout or disruption and recover the missing voter. | | Proposal queue full | Any increase in `admission.propose_queue.full` when admission metrics are enabled (`--admission-metrics`, off by default), or any corresponding overload error | Writes are rejected or delayed | Queue load, apply and WAL latency | Reduce incoming load, then find consensus or storage pressure. | | Active write stall | `pebble.write_stall.active = 1` for 10 seconds | Write latency rises and requests may fail | Compaction, flush, CPU and disk IOPS | Reduce writes and restore storage throughput or headroom. | | Disk headroom low | Used bytes on `storage.disk.volume.bytes` cross the warning level you compute against volume capacity, initially no later than 70% | Ledger can block writes at its configured high-water mark | `ledgerctl cluster disk-usage` on each member | Expand the affected volume and confirm usage falls below the resume mark. | | Memory headroom low | Working memory remains near the container limit or startup estimate exceeds `GOMEMLIMIT` | Kubernetes can kill a voter | Process, Go and container metrics | Reduce configured memory consumers or increase the limit; keep quorum during restart. | | Apply latency outside SLO | p99 `raft.apply_entries.duration` exceeds the tested baseline for five minutes | Write latency and catch-up time rise | WAL, compaction, queue and peer latency | Identify CPU, disk, network, or payload pressure before tuning. | | HTTP failures outside SLO | 5xx rate exceeds the service objective by route | Users receive failed requests | Status code, structured error reason, leader and write gates | Route to [Troubleshoot Ledger](/modules/ledger/operations/troubleshooting); do not retry every 5xx identically. | | Required index not ready | A required index is building, failed, or behind on any serving member | Queries fail or return an unavailable view | Index status per replica | Keep dependent traffic disabled and recover the index build. | | Usage projection behind | `usage.builder.lag` remains above the tested sequence backlog or grows continuously | Ledger statistics and Numscript template usage are stale or incomplete | The three usage-builder gauges on the affected member | Preserve logs, reduce competing resource pressure, and let the online builder catch up; do not delete its store as an alert response. | | Async workflow stalled | A sink or backup cursor stops advancing and reports an error | Downstream delivery or the recovery point falls behind | Workflow-specific status | Follow its owning runbook; a committed transaction is not rolled back by sink failure. | `storage.disk.volume.bytes` reports used bytes only, once per volume, with a `volume` attribute of `wal` or `data`. Ledger publishes no capacity series, so the disk-headroom percentage cannot be computed from Ledger telemetry alone. Pair the gauge with your platform's volume-capacity series — `kubelet_volume_stats_capacity_bytes` on Kubernetes, for example — or read used and total together from `ledgerctl cluster disk-usage`. The server does compare used against total internally to apply its own high- and low-water marks; it just does not export the ratio. Attach a dashboard link, owner, severity, expected response time, and runbook to each alert. Page only on conditions that require an operator action; use tickets or dashboards for trends that need capacity planning. ## Diagnose an alert Begin with non-destructive evidence: ```bash ledgerctl cluster status ledgerctl cluster disk-usage --json ``` `cluster disk-usage` reports the node to which `ledgerctl` is directly connected; run it against each member endpoint or profile when the cluster-level alert does not identify the affected volume. Then correlate the alert window with the two things this page owns: Stack and Kubernetes reconciliation events, and any deployment, index build, usage-projection rebuild, backup, or restore in flight. For the full signal-by-signal chain — request reasons, queue load, WAL and apply duration, membership changes, compaction, disk, memory, and changes in client request shape — follow the latency procedure in [Troubleshoot Ledger](/modules/ledger/operations/troubleshooting) instead of keeping a second copy of that order here. Do not compact, restart, remove a member, or change thresholds only to clear the alert. Identify the resource or availability constraint first. ## Verify and roll back monitoring changes After changing export, dashboards, or alerts: 1. repeat the controlled read and write; 2. confirm every member still reports under the same cluster identity; 3. exercise one warning and one paging route in staging; 4. confirm the notification reaches the documented owner; 5. verify that the linked runbook contains a non-destructive first step; 6. watch telemetry delivery and Ledger request latency through the change window. If telemetry disappears, restore the previous exporter and naming configuration. Keep the previous dashboards and alerts active until the replacement has produced continuous data for an agreed overlap period. A collector failure must never be handled by disabling Ledger health or capacity protection. --- ## Analyze and Aggregate Ledger Data Source: https://docs.formance.com/modules/ledger/querying/analyze-and-aggregate Use analysis to understand the shape of existing accounts and transactions. Use statistics for broad counts, and use volume aggregation when an application needs totals over a defined account set. ## Prerequisites The HTTP examples below use that connection. In addition: - Configure an authenticated `ledgerctl` profile or provide its connection and authentication flags. - Select a ledger that already contains representative data, and export its name as `LEDGER_NAME`. - Before filtering on metadata, declare the field type and wait for its index to become ready. Grant the scope required by the exact interface: | Operation | HTTP scope | gRPC or `ledgerctl` scope | | --- | --- | --- | | Analyze accounts | `ledger:AccountRead` | `ledger:AccountRead` | | Analyze transactions | `ledger:AccountRead` in v3.0 | `ledger:TransactionRead` | | Read statistics | `ledger:LedgerRead` | `ledger:LedgerRead` | | Aggregate account volumes | `ledger:AccountRead` | `ledger:AccountRead` | The HTTP and gRPC transaction-analysis scopes differ in Ledger v3.0. Grant only the scope used by the application's chosen interface. See [Authentication and Authorization](/modules/ledger/operations/authentication). These operations answer different questions: | Operation | Question | Read behavior | | --- | --- | --- | | Account analysis | Which address patterns and metadata keys exist? | Full account scan | | Transaction analysis | Which movement patterns and volume distributions exist? | Full transaction scan | | Ledger statistics | How many records or usage events have been counted? | Synchronous boundaries plus local asynchronous usage counters | | Volume aggregation | What are the input, output, and balance totals for selected accounts? | Reads every matching account volume | ## Analyze account structure Account analysis groups addresses into patterns and suggests variable segments. Run it before formalizing account types or when reviewing whether the live chart of accounts still matches the intended model. ```bash ledgerctl accounts analyze \ --ledger "$LEDGER_NAME" \ --threshold 10 \ --json ``` The threshold controls how many distinct children the analyzer accepts before it classifies a segment as variable. A value of `0` uses the server default of `10`. Capture the HTTP result and verify that the pattern counts cover the scanned accounts: ```bash ACCOUNT_ANALYSIS="$(curl --fail-with-body --silent --show-error --get \ "${CURL_AUTH[@]}" \ --data-urlencode 'variableThreshold=10' \ "$LEDGER_URL/v3/$LEDGER_NAME/analyze-accounts")" jq -e ' .data.totalAccounts as $total | ([.data.patterns[].accountCount] | add // 0) == $total ' <<<"$ACCOUNT_ANALYSIS" ``` The check prints `true` and exits with status `0` when every scanned account belongs to one returned pattern. The response also provides fixed and variable segments, examples, distinct-value counts, assets, and metadata keys. Review each suggestion before turning it into an account type. A changing identifier and a finite business category can look similar to a statistical analyzer but need different constraints in the model. ## Analyze transaction flows Transaction analysis normalizes posting addresses and groups transactions by movement structure: ```bash ledgerctl transactions analyze \ --ledger "$LEDGER_NAME" \ --threshold 10 \ --json ``` Capture the HTTP result and verify its totals: ```bash TRANSACTION_ANALYSIS="$(curl --fail-with-body --silent --show-error --get \ "${CURL_AUTH[@]}" \ --data-urlencode 'variableThreshold=10' \ "$LEDGER_URL/v3/$LEDGER_NAME/analyze-transactions")" jq -e ' .data.totalTransactions as $total | .data.totalReverted as $reverted | (([.data.flowPatterns[].transactionCount] | add // 0) == $total) and ($reverted <= $total) ' <<<"$TRANSACTION_ANALYSIS" ``` The check prints `true` only when the flow groups cover the scanned transactions and the reverted count does not exceed the total. Each flow also reports: - its normalized posting signature; - a `simple`, `multiSource`, `multiDestination`, or `complex` structure; - transaction and temporal statistics; - source and destination patterns; - asset and color dimensions; - exact volume strings and metadata keys. Use the result to find undocumented flows and select representative test cases. The analyzer does not declare an account type, create an index, or change a transaction. ## Make analysis repeatable Each analysis request uses one internally consistent snapshot, but two separate requests do not share a snapshot. Writes committed between account analysis and transaction analysis can therefore make the two totals describe different moments. The analysis routes accept neither `minLogSequence` nor a query checkpoint. When a paired result must be reproducible: 1. enter [Maintenance Mode](/modules/ledger/operations/maintenance-mode) to pause writes; 2. run both analyses and save their JSON responses; 3. verify the count invariants above; 4. leave maintenance mode only after every required analysis finishes. If writes cannot pause, treat the result as discovery data. Record when each request started and ended, but do not present the pair as one frozen reconciliation. Account and transaction analysis scan the complete target. Run them during a planned capacity window on a large ledger. ## Read ledger statistics Read the live counters with: ```bash ledgerctl ledgers stats --ledger "$LEDGER_NAME" --json ``` Or use HTTP: ```bash LEDGER_STATS="$(curl --fail-with-body --silent --show-error \ "${CURL_AUTH[@]}" \ "$LEDGER_URL/v3/$LEDGER_NAME/stats")" jq '.data' <<<"$LEDGER_STATS" ``` The response contains exactly these counters: | Counter | Source and freshness | | --- | --- | | `transactionCount`, `logCount` | Ledger boundaries maintained synchronously with committed state | | `volumeCount`, `referenceCount`, `postingCount`, `ephemeralEvictedCount`, `transientUsedCount`, `revertCount`, `numscriptExecutionCount` | Local usage projection updated asynchronously on each replica | The seven usage counters can briefly lag a completed write and can differ between replicas while they catch up. They describe volume cells and usage events; there is no account-count or metadata-count field. Do not use them as an instant write receipt or a single-request cutover gate. The gRPC statistics request accepts `--checkpoint-id`; the HTTP route does not. At a query checkpoint, only `transactionCount` and `logCount` describe the frozen state. Ledger returns the seven usage counters as `0` because they are unavailable at that checkpoint, not because the historical ledger had no such activity. When data is changing or a replica's derived usage view is rebuilding, use the transaction and log boundaries together with specific logs, audit entries, and integrity checks. Allow the local usage projections to converge before comparing their seven counters. Statistics show broad growth; they do not identify which records changed. Use [Read Ledger Data](/modules/ledger/querying/read-ledger-data) for a transaction-to-audit trace and [Verify Ledger Integrity](/modules/ledger/operations/integrity) for an integrity assessment. ## Aggregate volumes with HTTP Without a selector, the volume route aggregates every account in the ledger and returns input, output, and balance totals by asset and color: ```bash VOLUME_RESULT="$(curl --fail-with-body --silent --show-error \ "${CURL_AUTH[@]}" \ "$LEDGER_URL/v3/$LEDGER_NAME/volumes")" jq '.data' <<<"$VOLUME_RESULT" ``` Limit and group the account set with HTTP query parameters. `filter` is the only account selector this route accepts: ```bash SCOPED_VOLUMES="$(curl --fail-with-body --silent --show-error --get \ "${CURL_AUTH[@]}" \ --data-urlencode 'filter=address ^= "users:"' \ --data-urlencode 'groupByPrefixes=users:retail:,users:business:' \ --data-urlencode 'useMaxPrecision=true' \ "$LEDGER_URL/v3/$LEDGER_NAME/volumes")" jq '.data' <<<"$SCOPED_VOLUMES" ``` | Parameter | Effect | | --- | --- | | `filter` | Selects the accounts to aggregate, using the account filter grammar | | `groupByPrefixes` | Assigns each matching account to the first declared prefix it matches | | `useMaxPrecision` | Merges the same asset base at the highest observed precision | | `collapseColors` | Sums every color for an asset and returns the result with `color: ""` | The route ignores an unrecognized query parameter rather than rejecting it, and the aggregate response carries no account addresses. A misspelled or unsupported selector therefore returns `200` with a whole-ledger total that looks exactly like a scoped one. Assert the scope rather than assuming it. Prove that the selector narrowed the account set by running the same aggregate twice, changing only the filter, and requiring the two totals to differ. Compare the amounts as the decimal strings the route returns, never as converted numbers: ```bash aggregate_totals() { curl --fail-with-body --silent --show-error --get \ "${CURL_AUTH[@]}" \ "$@" \ "$LEDGER_URL/v3/$LEDGER_NAME/volumes" \ | jq -cS '[.data.volumes[] | {asset, color, input, output}] | sort' } SCOPED_TOTALS="$(aggregate_totals --data-urlencode "filter=$ACCOUNT_SELECTOR")" WHOLE_LEDGER_TOTALS="$(aggregate_totals)" if [ "$SCOPED_TOTALS" != "$WHOLE_LEDGER_TOTALS" ]; then echo "scope applied" else echo "scope NOT applied: the selector was ignored and this total covers every account" >&2 exit 1 fi ``` Both requests pass the same aggregation options, so the only difference between them is the selector. Equal totals mean the selector did not narrow anything. Run this against a ledger that holds accounts outside the selected subtree: where the subtree is the entire ledger the two totals are legitimately equal, and the check cannot distinguish that from an ignored selector. Prefix matching is not segment-aware. `accounts:1` also matches `accounts:10` and `accounts:11`; include the delimiter when the chart of accounts requires a whole segment. When group prefixes overlap, declaration order matters: the first match wins, so an account is not double-counted. Accounts that match none of the group prefixes are omitted from the grouped result. By default, colors remain separate. Use `collapseColors=true` only when the report intentionally removes that dimension. The HTTP response keeps amounts as decimal strings and includes the computed balance: ```json { "data": { "volumes": [ { "asset": "USD/2", "color": "", "input": "15000", "output": "2500", "balance": "12500" } ] } } ``` For a known fixture, compare exact strings rather than converting amounts to a JavaScript `Number`: ```bash jq -e \ --arg asset "$EXPECTED_ASSET" \ --arg color "$EXPECTED_COLOR" \ --arg input "$EXPECTED_INPUT" \ --arg output "$EXPECTED_OUTPUT" \ --arg balance "$EXPECTED_BALANCE" ' any(.data.volumes[]; .asset == $asset and .color == $color and .input == $input and .output == $output and .balance == $balance ) ' <<<"$VOLUME_RESULT" ``` The check prints `true` only when the expected asset and color bucket has all three exact values. Use an arbitrary-precision integer or decimal library when the application must calculate with these strings. ## Aggregate a filtered account set with gRPC `ledgerctl` accepts the account filter grammar as well as minimum-sequence and checkpoint controls. A metadata filter requires a declared field and a ready account metadata index. When the aggregate must include a completed write, derive the sequence from the full gRPC `ApplyResponse`: ```bash MIN_LOG_SEQUENCE="$( jq -er ' .logs[-1].sequence // error("the response contains no new log sequence") | tostring ' apply-response.json )" ledgerctl accounts aggregate-volumes \ --ledger "$LEDGER_NAME" \ --filter 'metadata[status] == "active"' \ --min-log-sequence "$MIN_LOG_SEQUENCE" \ --json ``` `ApplyResponse.logs` is returned in commit order, so its last entry provides the highest sequence from that proposal. A response with no fresh log produced no new sequence to wait for, so the filter raises an error rather than yielding the string `null`, which `--min-log-sequence` would otherwise accept. This gRPC read waits for the local read index to reach the captured sequence. It does not wait for a metadata-index backfill or version transition. For a point-in-time reconciliation, first run the complete [checkpoint capture, readiness, and cleanup workflow](/modules/ledger/querying/consistency-and-checkpoints#freeze-a-view-with-a-query-checkpoint). That workflow derives `CHECKPOINT_ID` from the create response, probes the serving replica, and installs a cleanup trap. Then use the captured value rather than selecting an ID from a list: ```bash : "${CHECKPOINT_ID:?create and verify a query checkpoint first}" ledgerctl accounts aggregate-volumes \ --ledger "$LEDGER_NAME" \ --filter 'address ^= "users:"' \ --checkpoint-id "$CHECKPOINT_ID" \ --json ``` Repeat every related read with that same checkpoint ID, and let the capture workflow delete that exact ID after all consumers finish. The direct HTTP volume route accepts the generic `filter` parameter but exposes neither `minLogSequence` nor checkpoint controls. The raw gRPC `AggregateResult` carries `input` and `output`, but no `balance` field. A client that consumes this message directly must compute `input - output` with an arbitrary-precision type. `ledgerctl accounts aggregate-volumes --json` performs that calculation and includes `balance` in its JSON output. When an aggregate filter becomes a shared application contract, store it as an account [prepared query](/modules/ledger/querying/prepared-queries) and execute it in `AGGREGATE_VOLUMES` mode; prepared-query execution supports a minimum sequence but not a query checkpoint in v3.0. ## Interpret results responsibly - `input` and `output` are cumulative movements over the selected account set. - The HTTP volume route and `ledgerctl accounts aggregate-volumes --json` report `balance` as `input - output` for the returned asset and color bucket. - A zero balance does not prove that no transactions exist; equal input and output can net to zero. - Analysis, statistics, and aggregation describe data. They do not prove ledger integrity. See [Filter and Paginate Results](/modules/ledger/querying/filters-and-pagination) and [Create and Manage Indexes](/modules/ledger/querying/indexes) when an aggregate filter depends on an opt-in index. --- ## Process Operations in Bulk Source: https://docs.formance.com/modules/ledger/transactions/bulk The REST bulk endpoint reduces request overhead when several independent operations target the same ledger. Use one transaction with several postings when the movements form one accounting event. Use bulk when each element is an independent operation with its own identity and result. ## Prerequisites You need an existing normal ledger, a token with the write scopes required by every element, and `ledger:read` for the account checks in this guide. With the default scope mapping, the virtual `ledger:write` scope covers all four: `ledger:TransactionWrite` for `CREATE_TRANSACTION` and `REVERT_TRANSACTION`, `ledger:MetadataWrite` for `ADD_METADATA` and `DELETE_METADATA`. Bulk checks the required scope per element. If the ledger enforces account types, replace the demo addresses with values that match its configured patterns. ```bash set -eo pipefail ``` Give every element a distinct `ik` value. This is the per-operation idempotency key used by the normal sequential REST path. ## REST surface This guide covers `POST /v3/{ledgerName}/bulk` with an `application/json` array. The ledger name in the path applies to every element, so one REST bulk request cannot span ledgers. The REST endpoint accepts these actions: | `action` | `data` | | --- | --- | | `CREATE_TRANSACTION` | The same `postings`, `script`, or `scriptReference` fields as a transaction create request. | | `ADD_METADATA` | `targetType`, `targetId`, and `metadata` for an account or transaction. | | `REVERT_TRANSACTION` | `id` plus optional `force`, `atEffectiveDate`, and `metadata`. | | `DELETE_METADATA` | `targetType`, `targetId`, and `key` for an account or transaction. | Account-type bulk actions exist on the gRPC surface but are not accepted by this REST endpoint. Do not infer parity between the two transports. A bulk result exposes only the created transaction for a `CREATE_TRANSACTION` element, and a successful non-create element omits `data` entirely. Post-commit volumes ride on that created transaction, and no request field adds volumes to any other element. Verify affected accounts with separate reads. ## Observe the default stop behavior By default, Ledger processes elements in order and stops after the first failure; operations that completed before it remain committed. The second element below deliberately reuses the first element's transaction reference. The third element is valid but appears after the failure. ```bash BULK_BODY=$(jq --compact-output --null-input \ --arg prefix "$BULK_PREFIX" \ --arg duplicateReference "$BULK_DUPLICATE_REFERENCE" \ --arg wallet1 "$BULK_WALLET_1" \ --arg wallet2 "$BULK_WALLET_2" \ --arg wallet3 "$BULK_WALLET_3" \ '[ { action: "CREATE_TRANSACTION", ik: ($prefix + ":1"), data: { reference: $duplicateReference, postings: [ {source: "world", destination: $wallet1, amount: 1000, asset: "USD/2"} ] } }, { action: "CREATE_TRANSACTION", ik: ($prefix + ":2"), data: { reference: $duplicateReference, postings: [ {source: "world", destination: $wallet2, amount: 2000, asset: "USD/2"} ] } }, { action: "CREATE_TRANSACTION", ik: ($prefix + ":3"), data: { reference: ($prefix + ":third"), postings: [ {source: "world", destination: $wallet3, amount: 3000, asset: "USD/2"} ] } } ]') ``` Submit the request without `continueOnFailure`: ```bash HTTP_STATUS=$(curl --silent --show-error \ --output /tmp/bulk-stop.json \ --write-out "%{http_code}" \ --request POST "$LEDGER_URL/v3/$LEDGER_NAME/bulk" \ --header "Authorization: Bearer $LEDGER_TOKEN" \ --header "Content-Type: application/json" \ --data "$BULK_BODY") test "$HTTP_STATUS" = "409" && jq -e ' .data as $results | ($results | length) == 3 and $results[0].responseType == "CREATE_TRANSACTION" and ($results[0].logID > 0) and $results[1].responseType == "ERROR" and $results[1].errorCode == "TRANSACTION_REFERENCE_CONFLICT" and $results[2].responseType == "ERROR" and $results[2].logID == 0' \ /tmp/bulk-stop.json ``` Expected result: 1. the first element has `responseType: "CREATE_TRANSACTION"` and a non-zero `logID`; 2. the second has `responseType: "ERROR"` and `errorCode: "TRANSACTION_REFERENCE_CONFLICT"`; 3. the third has no committed log because processing stopped; 4. the top-level status is a business `4xx`, driven by the second element. Verify the state rather than inferring it from the top-level status: ```bash curl --fail-with-body --silent --show-error \ "$LEDGER_URL/v3/$LEDGER_NAME/accounts/$BULK_WALLET_1" \ --header "Authorization: Bearer $LEDGER_TOKEN" \ | jq -e ' .data.volumes[] | select(.asset == "USD/2" and .color == "") | .volumes.input == "1000" and .volumes.output == "0" and .volumes.balance == "1000"' && curl --fail-with-body --silent --show-error \ "$LEDGER_URL/v3/$LEDGER_NAME/accounts/$BULK_WALLET_3" \ --header "Authorization: Bearer $LEDGER_TOKEN" \ | jq -e ' (.data.volumes | type) == "array" and (.data.volumes | length) == 0' ``` Both assertions return `true`. The first account has USD 10.00. Ledger returns an empty account view for an address with no volumes or metadata, so the empty volume list—not a `404` status—shows that the third operation did not run. ## Continue after business failures Set `continueOnFailure=true` when independent later elements should still run after a per-element business `4xx`: ```bash HTTP_STATUS=$(curl --silent --show-error \ --output /tmp/bulk-continue.json \ --write-out "%{http_code}" \ --request POST "$LEDGER_URL/v3/$LEDGER_NAME/bulk?continueOnFailure=true" \ --header "Authorization: Bearer $LEDGER_TOKEN" \ --header "Content-Type: application/json" \ --data "$BULK_BODY") test "$HTTP_STATUS" = "200" && jq -e ' .data as $results | ($results | length) == 3 and $results[0].responseType == "CREATE_TRANSACTION" and ($results[0].logID > 0) and $results[1].responseType == "ERROR" and $results[1].errorCode == "TRANSACTION_REFERENCE_CONFLICT" and $results[2].responseType == "CREATE_TRANSACTION" and ($results[2].logID > 0)' \ /tmp/bulk-continue.json ``` The first element replays its original result through `$BULK_PREFIX:1`. The second replays its reference-conflict outcome through `$BULK_PREFIX:2`. The third now commits under `$BULK_PREFIX:3`. `continueOnFailure` changes the top-level status and whether later elements run. It does not hide per-element errors. Inspect every item in `.data`, even when the HTTP response is `200`. ## Turn an expected conflict into a recorded skip Some import and replay workflows expect a specific business conflict. Add `skippableReasons` to that element to record a no-op instead of an error: ```bash jq --null-input \ --arg ik "$BULK_PREFIX:expected-duplicate" \ --arg reference "$BULK_DUPLICATE_REFERENCE" \ --arg destination "bulk:$RUN_ID:wallet:ignored" \ '[ { action: "CREATE_TRANSACTION", ik: $ik, skippableReasons: ["TRANSACTION_REFERENCE_CONFLICT"], data: { reference: $reference, postings: [ {source: "world", destination: $destination, amount: 9999, asset: "USD/2"} ] } } ]' \ | curl --fail-with-body --silent --show-error \ --request POST "$LEDGER_URL/v3/$LEDGER_NAME/bulk" \ --header "Authorization: Bearer $LEDGER_TOKEN" \ --header "Content-Type: application/json" \ --data @- \ > /tmp/bulk-skip.json jq -e \ --arg ledger "$LEDGER_NAME" \ --arg reference "$BULK_DUPLICATE_REFERENCE" \ '.data[0] as $result | $result.responseType == "CREATE_TRANSACTION" and ($result.logID > 0) and $result.data.skipped and ($result.data.reason == "TRANSACTION_REFERENCE_CONFLICT") and ($result.data.context.ledger == $ledger) and ($result.data.context.reference == $reference) and ($result.data.context.existingTransactionId | test("^[1-9][0-9]*$"))' \ /tmp/bulk-skip.json ``` The skipped result contains this information; the concrete reference and transaction ID depend on the run: ```json { "skipped": true, "reason": "TRANSACTION_REFERENCE_CONFLICT", "context": { "ledger": "main", "reference": "bulk-demo::duplicate-reference", "existingTransactionId": "" } } ``` A skipped element creates an auditable skip log but does not apply the attempted posting. The REST whitelist is intentionally narrow: | Action | Supported skippable reason | | --- | --- | | `CREATE_TRANSACTION` | `TRANSACTION_REFERENCE_CONFLICT` | | `REVERT_TRANSACTION` | `TRANSACTION_ALREADY_REVERTED` | | `DELETE_METADATA` | `METADATA_NOT_FOUND` | Ledger rejects reasons outside the action's whitelist when it processes that element. In sequential mode, earlier elements may already have committed; in atomic mode, the whole batch fails. `ADD_METADATA` has no REST skippable reason. ## Make the whole batch atomic Set `atomic=true` when every element must commit or none may commit. Ledger sends the array as one atomic batch. Use one `Idempotency-Key` header for that batch; per-element `ik` values belong to sequential mode and do not identify an atomic request. The example below deliberately repeats a reference inside a new atomic batch. The conflict rejects both elements: ```bash ATOMIC_PREFIX="$BULK_PREFIX:atomic" ATOMIC_WALLET_1="bulk:$RUN_ID:atomic:wallet:1" ATOMIC_WALLET_2="bulk:$RUN_ID:atomic:wallet:2" ATOMIC_REFERENCE="$ATOMIC_PREFIX:duplicate-reference" ATOMIC_BODY=$(jq --compact-output --null-input \ --arg reference "$ATOMIC_REFERENCE" \ --arg wallet1 "$ATOMIC_WALLET_1" \ --arg wallet2 "$ATOMIC_WALLET_2" \ '[ { action: "CREATE_TRANSACTION", data: { reference: $reference, postings: [ {source: "world", destination: $wallet1, amount: 1000, asset: "USD/2"} ] } }, { action: "CREATE_TRANSACTION", data: { reference: $reference, postings: [ {source: "world", destination: $wallet2, amount: 2000, asset: "USD/2"} ] } } ]') HTTP_STATUS=$(curl --silent --show-error \ --output /tmp/bulk-atomic.json \ --write-out "%{http_code}" \ --request POST "$LEDGER_URL/v3/$LEDGER_NAME/bulk?atomic=true" \ --header "Authorization: Bearer $LEDGER_TOKEN" \ --header "Content-Type: application/json" \ --header "Idempotency-Key: $ATOMIC_PREFIX:batch" \ --data "$ATOMIC_BODY") test "$HTTP_STATUS" = "409" && jq -e ' (.data | length) == 2 and all(.data[]; .responseType == "ERROR" and .errorCode == "TRANSACTION_REFERENCE_CONFLICT" and .logID == 0)' \ /tmp/bulk-atomic.json ``` Verify that neither account received funds: ```bash ( for ACCOUNT in "$ATOMIC_WALLET_1" "$ATOMIC_WALLET_2"; do curl --fail-with-body --silent --show-error \ "$LEDGER_URL/v3/$LEDGER_NAME/accounts/$ACCOUNT" \ --header "Authorization: Bearer $LEDGER_TOKEN" \ | jq -e ' (.data.volumes | type) == "array" and (.data.volumes | length) == 0' \ || exit 1 done ) ``` Do not combine `atomic=true` with `continueOnFailure=true`: atomic execution has no successful prefix to continue after. If several postings describe one accounting event, a single transaction remains the clearer model; use atomic bulk for distinct operations that share one commit boundary. ## Handle response classes Bulk failures fall into three groups: | Response | Meaning | Client action | | --- | --- | --- | | Request-level `400`, `401`, `403`, or `413` | Ledger rejected the request before processing elements | Correct the body, credentials, scopes, or configured batch-size limit. Do not assume any element result from a missing `.data` array. | | Per-element business `4xx` | One operation failed validation, lookup, conflict, or a precondition | Inspect each item. In sequential mode, preceding items may have committed and later items did not run. In atomic mode, no element commits. | | `429`, `500`, or `503` | A rate limit, service failure, stale input resolution, or unavailable Numscript preload occurred | `continueOnFailure` does not turn these into `200`. Inspect `errorCode`, honor `Retry-After` when present, and retry only with bounded backoff. A repeated `PRELOAD_UNAVAILABLE` can indicate invalid Numscript input rather than an infrastructure outage. | The maximum number of elements is a server setting. Ledger returns `413` with `BULK_SIZE_EXCEEDED` when the request exceeds that configured limit. Split work into bounded batches instead of assuming one fixed limit across environments. ## Retry a bulk request For a sequential top-level `500` or `503`, inspect every `errorCode`, then retry the same array after bounded backoff while the outcome is uncertain. Preserve every element's `ik` and original `data`. Elements that committed replay their outcomes; elements that did not run can proceed. Stop and validate the program and variables if Numscript `PRELOAD_UNAVAILABLE` persists. For an atomic timeout or retryable service failure, resend the exact array with the same batch-level `Idempotency-Key` header. Do not add or change per-element keys. No successful prefix can escape a failed atomic batch, but the stable batch key protects a commit whose response was lost. For a completed response with business errors: 1. persist the result for every element; 2. treat a successful or explicitly skipped item as complete; 3. diagnose each failed item by `errorCode`; 4. create a corrected operation with a new `ik` only when its logical content changes; 5. verify balances, metadata, or reversion state independently. Do not generate fresh keys for an unchanged whole-batch retry. That can duplicate the elements that committed before the failure. Continue with [Write Transactions](/modules/ledger/transactions) to choose between a single transaction, a saved Numscript program, and a bulk request. --- ## Validate the Migration Source: https://docs.formance.com/modules/ledger/migration/validation A mirror can be `FOLLOWING` and still contain an unacceptable business result. Ledger does not compare source and target balances or state hashes for you. Use the same validation specification for the rehearsal, the live-following period, and the final frozen-source check. Only the final check runs against a source that can no longer move. ## Define materiality before comparing For each ledger, record: - accounts and account groups whose balances must match exactly; - assets, precisions, and colors to compare; - transaction and reference samples, including boundaries and high-value cases; - reverted and backdated transactions; - metadata keys, expected types, deleted keys, and null behavior; - transformations and drops that deliberately change the target; - query, index, authentication, event, and performance requirements; - who approves each check and what constitutes a blocking difference. Do not decide that a difference is immaterial during the production cutover without the validation owner. ## Enforce the final comparison boundary Run this gate after every Ledger v2 writer is frozen. During rehearsal, use it only against a quiesced source; otherwise the source can advance between the two reads. Set every prerequisite explicitly. The configured `ledgerctl` profile must point to the candidate target cluster and have read access to `$TARGET_LEDGER`. Choose the source transport used by the mirror; do not configure HTTP credentials for a PostgreSQL source. The three shell blocks on this page are one program, not independent snippets. `set -euo pipefail`, the `V2_*` exports, the helper functions, and `FINAL_SOURCE_LOG_ID` are defined in earlier blocks and consumed by later ones, and the gate exits with `exit 1` on every failure. Append them in order to a single file and run that file. Pasting them into an interactive shell works until something fails, at which point `exit 1` closes the session — taking the exported source configuration and the helper functions with it, at the moment a frozen-source window is open and you most need them. ```bash set -euo pipefail set +x # HTTP source only: # export V2_URL="https://ledger-v2.example.com" # export V2_TOKEN="" # PostgreSQL source only. Build a passwordless, hostname-verifying DSN from # the same endpoint, database, and identity configured on the mirror. The # preflight below reassembles V2_PG_DSN from these four variables, so edit # them rather than the DSN string. # Password authentication only. Mount this file from a secret manager with # mode 0600; never put the password in V2_PG_DSN. # RDS IAM authentication only. Leave the role ARN empty when the current # runtime identity already has rds-db:connect. : "${SOURCE_TRANSPORT:?set SOURCE_TRANSPORT}" : "${SOURCE_LEDGER:?set SOURCE_LEDGER}" : "${TARGET_LEDGER:?set TARGET_LEDGER}" command -v jq >/dev/null command -v ledgerctl >/dev/null case "$SOURCE_TRANSPORT" in http) : "${V2_URL:?set V2_URL for an HTTP source}" : "${V2_TOKEN:?set V2_TOKEN for an HTTP source}" command -v curl >/dev/null ;; postgres) : "${SOURCE_POSTGRES_AUTH:?set SOURCE_POSTGRES_AUTH}" : "${V2_PG_HOST:?set V2_PG_HOST}" : "${V2_PG_PORT:?set V2_PG_PORT}" : "${V2_PG_DATABASE:?set V2_PG_DATABASE}" : "${V2_PG_USER:?set V2_PG_USER}" # Constrain the parts, then build the DSN. Do not inspect a supplied one: # libpq percent-decodes URI parameter names and honours quoting in # keyword/value conninfo, so no shell-level check can establish the sslmode # libpq will actually apply. Assembling the URI from validated components # makes verify-full true by construction, with nothing left to parse. [[ "$V2_PG_HOST" =~ ^[A-Za-z0-9.-]+$ ]] || { printf 'V2_PG_HOST must be a bare hostname\n' >&2 exit 1 } [[ "$V2_PG_PORT" =~ ^[1-9][0-9]*$ ]] || { printf 'V2_PG_PORT must be a port number\n' >&2 exit 1 } [[ "$V2_PG_DATABASE" =~ ^[A-Za-z0-9_-]+$ ]] || { printf 'V2_PG_DATABASE must be a bare database name\n' >&2 exit 1 } [[ "$V2_PG_USER" =~ ^[A-Za-z0-9_-]+$ ]] || { printf 'V2_PG_USER must be a bare role name\n' >&2 exit 1 } V2_PG_DSN="postgresql://${V2_PG_USER}@${V2_PG_HOST}:${V2_PG_PORT}/${V2_PG_DATABASE}?sslmode=verify-full" command -v psql >/dev/null case "$SOURCE_POSTGRES_AUTH" in password) : "${PGPASSFILE:?set PGPASSFILE for password authentication}" if [ ! -r "$PGPASSFILE" ]; then printf 'PGPASSFILE must be a readable, secret-backed mode-0600 file\n' >&2 exit 1 fi pgpass_mode="$(stat -c '%a' "$PGPASSFILE" 2>/dev/null || stat -f '%Lp' "$PGPASSFILE")" if [ "$pgpass_mode" != "600" ]; then printf 'PGPASSFILE must have mode 0600; found %s\n' "$pgpass_mode" >&2 exit 1 fi ;; rds-iam) : "${V2_AWS_REGION:?set V2_AWS_REGION}" : "${V2_AWS_ROLE_SESSION_NAME:?set V2_AWS_ROLE_SESSION_NAME}" command -v aws >/dev/null ;; *) printf 'unsupported SOURCE_POSTGRES_AUTH: %s\n' \ "$SOURCE_POSTGRES_AUTH" >&2 exit 1 ;; esac ;; *) printf 'unsupported SOURCE_TRANSPORT: %s\n' "$SOURCE_TRANSPORT" >&2 exit 1 ;; esac ``` For a PostgreSQL source, the reader needs `CONNECT` on the database, `USAGE` on `_system` and the selected bucket schema, `SELECT` on `_system.ledgers`, and `SELECT` on the selected bucket's `logs` table. It does not need write access. Protect credentials as secrets, keep shell tracing disabled, and use the same hostname, database, TLS policy, and identity that the mirror uses. Define the PostgreSQL authentication helpers and one source-head function, then use them for this gate. `V2_PG_HOST` must be the exact RDS endpoint used in the DSN, not a different alias. The RDS IAM helper generates a fresh 15-minute token for every psql process. When an assume-role ARN is set, it also obtains fresh role credentials for that invocation. Neither the token nor the temporary AWS credentials enter command-line arguments, global environment variables, or evidence files. ```bash generate_source_postgres_iam_token() { local credentials_json local access_key_id local secret_access_key local session_token if [ -z "$V2_AWS_ASSUME_ROLE_ARN" ]; then AWS_PAGER="" aws rds generate-db-auth-token \ --no-cli-pager \ --hostname "$V2_PG_HOST" \ --port "$V2_PG_PORT" \ --region "$V2_AWS_REGION" \ --username "$V2_PG_USER" || return return fi credentials_json="$(AWS_PAGER="" aws sts assume-role \ --no-cli-pager \ --role-arn "$V2_AWS_ASSUME_ROLE_ARN" \ --role-session-name "$V2_AWS_ROLE_SESSION_NAME" \ --duration-seconds 900 \ --output json)" || return access_key_id="$(jq -er '.Credentials.AccessKeyId' \ <<<"$credentials_json")" || return secret_access_key="$(jq -er '.Credentials.SecretAccessKey' \ <<<"$credentials_json")" || return session_token="$(jq -er '.Credentials.SessionToken' \ <<<"$credentials_json")" || return AWS_ACCESS_KEY_ID="$access_key_id" \ AWS_SECRET_ACCESS_KEY="$secret_access_key" \ AWS_SESSION_TOKEN="$session_token" \ AWS_PAGER="" \ aws rds generate-db-auth-token \ --no-cli-pager \ --hostname "$V2_PG_HOST" \ --port "$V2_PG_PORT" \ --region "$V2_AWS_REGION" \ --username "$V2_PG_USER" || return } source_psql() { local iam_token case "$SOURCE_POSTGRES_AUTH" in password) psql "$V2_PG_DSN" "$@" ;; rds-iam) iam_token="$(generate_source_postgres_iam_token)" || return if [ -z "$iam_token" ] || [[ "$iam_token" == *$'\n'* ]]; then printf 'RDS IAM token generation returned an invalid value\n' >&2 return 1 fi PGPASSWORD="$iam_token" psql "$V2_PG_DSN" "$@" ;; esac } read_source_log_id() { local source_head_json local source_log_id case "$SOURCE_TRANSPORT" in http) source_head_json="$(curl --fail --silent --show-error \ --header "Authorization: Bearer $V2_TOKEN" \ "$V2_URL/v2/$SOURCE_LEDGER/logs?pageSize=1")" source_log_id="$(jq -er ' .cursor.data | if type != "array" then error("source response has no cursor.data array") elif length == 0 then "0" else (.[0].id | tostring) as $id | if $id | test("^[1-9][0-9]*$") then $id else error("newest source log has an invalid id") end end ' <<<"$source_head_json")" ;; postgres) source_log_id="$( source_psql \ --no-psqlrc \ --no-password \ --set=ON_ERROR_STOP=1 \ --set=source_ledger="$SOURCE_LEDGER" \ --quiet \ --tuples-only \ --no-align <<'SQL' SELECT bucket AS source_bucket FROM _system.ledgers WHERE name = :'source_ledger' \gset \if :{?source_bucket} SELECT COALESCE(MAX(id), 0)::text FROM :"source_bucket".logs WHERE ledger = :'source_ledger'; \else \warn 'source ledger did not resolve to exactly one bucket' \endif SQL )" ;; esac if [[ "$source_log_id" == *$'\n'* ]] || ! [[ "$source_log_id" =~ ^(0|[1-9][0-9]*)$ ]]; then printf 'source head must be exactly one unsigned integer row\n' >&2 return 1 fi printf '%s\n' "$source_log_id" } FINAL_SOURCE_LOG_ID="$(read_source_log_id)" jq -n \ --arg finalSourceLogId "$FINAL_SOURCE_LOG_ID" \ '{finalSourceLogId: $finalSourceLogId}' \ > source-head.json ``` The two paths do not rest on the same guarantee. The PostgreSQL query takes `MAX(id)`, so it is order-independent. The HTTP path reads the first element of a one-item page and treats it as the newest log, which assumes the endpoint returns logs newest-first — an ordering the v2 API contract does not state. Confirm it holds on the Ledger v2 version you are migrating from before you trust the recorded `finalSourceLogId`: if the endpoint ever returns oldest-first, the gate still fails closed, but the evidence file records the wrong boundary while appearing complete. Read the target once, capture the complete progress tuple, and stop unless the target is a mirror at the same frozen boundary with no active error: ```bash TARGET_LEDGER_JSON="$(ledgerctl ledgers get "$TARGET_LEDGER" --json)" jq -e '.mirrorSyncProgress != null' <<<"$TARGET_LEDGER_JSON" >/dev/null TARGET_MODE="$(jq -r '.mode // "(absent)"' <<<"$TARGET_LEDGER_JSON")" if ! jq -e ' .mirrorSyncProgress as $p | ["cursor", "sourceLogCount", "remainingLogs"] | all(. as $k | if ($p | has($k)) then ($p[$k] | tostring | test("^[0-9]+$")) else true end) ' <<<"$TARGET_LEDGER_JSON" >/dev/null; then printf 'mirrorSyncProgress counters must be omitted or unsigned integers\n' >&2 exit 1 fi TARGET_CURSOR="$(jq -er '(.mirrorSyncProgress.cursor // 0) | tostring' <<<"$TARGET_LEDGER_JSON")" TARGET_SOURCE_HEAD="$(jq -er '(.mirrorSyncProgress.sourceLogCount // 0) | tostring' <<<"$TARGET_LEDGER_JSON")" TARGET_REMAINING="$(jq -er '(.mirrorSyncProgress.remainingLogs // 0) | tostring' <<<"$TARGET_LEDGER_JSON")" TARGET_ERROR_JSON="$(jq -c '.mirrorSyncProgress.error // null' <<<"$TARGET_LEDGER_JSON")" if [ "$TARGET_MODE" != "LEDGER_MODE_MIRROR" ] || [ "$TARGET_CURSOR" != "$FINAL_SOURCE_LOG_ID" ] || [ "$TARGET_SOURCE_HEAD" != "$FINAL_SOURCE_LOG_ID" ] || [ "$TARGET_REMAINING" != "0" ] || [ "$TARGET_ERROR_JSON" != "null" ]; then printf '%s\n' \ "final mirror gate failed" \ "source=$FINAL_SOURCE_LOG_ID" \ "mode=$TARGET_MODE" \ "cursor=$TARGET_CURSOR" \ "observedSource=$TARGET_SOURCE_HEAD" \ "remaining=$TARGET_REMAINING" \ "errorPresent=$([ "$TARGET_ERROR_JSON" = "null" ] && printf no || printf yes)" >&2 if [ "$TARGET_ERROR_JSON" != "null" ]; then if (set -o noclobber; umask 077; printf '%s\n' "$TARGET_ERROR_JSON" \ > mirror-error.json) 2>/dev/null then printf 'persisted mirror error written to mirror-error.json (mode 0600)\n' >&2 else printf 'refusing to write mirror-error.json: the path already exists\n' >&2 fi fi exit 1 fi jq \ --arg finalSourceLogId "$FINAL_SOURCE_LOG_ID" \ '{ name, mode, finalSourceLogId: $finalSourceLogId, mirrorSyncProgress }' <<<"$TARGET_LEDGER_JSON" > target-ledger.json ``` The failure dump reports only whether an error is present. The persisted mirror error is the raw source, translation or apply error, so it can echo the source endpoint or a PostgreSQL DSN, and it must stay out of CI logs and transcripts. Since the gate exits on failure, the error cannot be left in a shell variable for the operator to inspect: it is written to `mirror-error.json` with mode `0600` instead. `set -o noclobber` is deliberate. `umask` only applies when a file is created, so a plain redirect into an existing path would keep that file's mode while claiming `0600`, or follow a symlink left there. Under `noclobber` the shell opens with `O_EXCL`, so an existing path — regular file or symlink — makes the write fail rather than succeed somewhere unintended. That is one atomic operation, unlike removing the path first and then creating it, which leaves a window for a symlink to be planted in between. If the write is refused, inspect or remove the existing file and rerun. Read that file, then delete it with the rest of the credential-bearing artifacts. Two details of the wire format shape those reads, and a gate written without both rejects the one state it exists to accept. The counters are 64-bit integers, so they arrive as JSON strings rather than numbers — assert on their digits, not on `type == "number"`. And a field sitting at its default value is omitted from the response entirely, which is exactly what happens to `remainingLogs` on a fully caught-up mirror and to `mode` on a normal ledger. Absent and present-but-wrong are not the same thing, which is why the check tests `has()` rather than defaulting with `//`. An omitted counter is a protobuf default and is accepted as zero; a counter that is present as `null`, `false`, or a non-integer is a contract violation and fails the gate. `// 0` alone cannot tell those apart, because jq treats `null` and `false` as absent — so a malformed response would have been read as a legitimately empty source. Once the check has established that every present counter is digits, the reads below can default with `//` safely. Keep the two filtered evidence files with the migration record. Do not save the complete structured ledger response: it contains the immutable mirror source configuration, which can include an OAuth2 client secret or a PostgreSQL DSN. ## Run the technical checks | Check | Evidence | Pass condition | | --- | --- | --- | | Log coverage | Final source log ID, cursor, observed source head, and remaining logs. | Cursor equals the recorded source ID; remaining logs are zero; no error is present. | | Ledger identity | Source worksheet and target ledger info. | Source and target names map as planned; target remains in mirror mode before approval. | | Transaction coverage | Counts by agreed range plus sampled IDs and references. | Matches after documented drops and transformations. | | Reversions | Source transaction, reversion, and resulting target state. | Original and compensating effects are represented as expected. | | Account coverage | Counts by material prefix and selected boundary accounts. | All required accounts exist at their expected target addresses. | | Balances and volumes | Per account/group, asset, and color comparison. | Exact equality except for pre-approved transformation effects. | | Metadata | Account and transaction key/value/type samples. | Values, deletions, typed nulls, and conversions match the specification. | | Rewrite evidence | Rule-file hash and fixture results. | Every expected match, non-match, collision, and drop is explained. | | Indexes | Index status from every target replica. | Every index required by a production query is ready. | | Mirror health | Time series of cursor/head/error. | The target follows normally and has no unresolved error. | Counts are useful for coverage but do not prove accounting equality. Pair them with balance, volume, and representative transaction checks. ## Compare business invariants Translate each domain invariant into a repeatable source query and target query. Examples include: - customer liabilities equal the corresponding safeguarding or settlement accounts; - clearing account balances remain within an approved range; - the sum of wallet balances per asset matches the source result; - a payout transaction preserves principal, fee, and destination totals; - every settled business reference remains unique; - no account moved into an unintended prefix after address rewriting. For each invariant, save: 1. query or command version; 2. exact source and target parameters; 3. source log boundary; 4. normalized result; 5. expected transformation adjustment; 6. pass/fail result and reviewer. Do not compare formatted monetary values with floating-point arithmetic. Compare Ledger integer amounts at the declared asset precision. ## Validate transformed data For each rewrite rule, reconcile the fixture register against the target: | Rule effect | Required proof | | --- | --- | | Address rewrite | Positive and negative fixtures, collision analysis, and account-group balance comparison. | | Metadata set/delete | Original value, expected target value/type, missing-field behavior, and deletion result. | | Typed coercion | Valid conversion and invalid value represented as the expected typed null. | | Derived account metadata | Captured segments, target account, type, and non-matching address. | | Drop | Source log ID, reason, transaction ID when present, and accounting impact. | An unexplained difference is a failed validation, even if it appears to match the intent of a rule. ## Validate application workflows Connect a non-production application instance to the target. A mirror rejects writes, so split the test: - run read, filter, pagination, metadata-read, and reporting workflows against the production-shaped mirror; - run create, retry, revert, metadata-write, and bulk workflows against a disposable normal target built from the same release; - test the final endpoint and authentication switch in the cutover rehearsal; - verify every target identity has the intended scope and no broader one; - test timeout, conflict, unavailable, and rate-limit handling where relevant. The application must use the target contract. A correct data copy does not make a Ledger v2 client compatible. ## Validate event consumers Mirrored Ledger v2 entries are ordinary target logs. Every supported imported change produces a standard target event, and a newly created sink starts at cursor `0` and replays all supported retained history. The current sink configuration does not provide a start cursor that opts out of that replay. Treat historical replay as part of the migration, not as an optional event mode. Keep the target destination or consumer quarantined while Ledger v2 remains the production authority: it may durably receive and inspect target events, but it must not repeat business effects already produced from the Ledger v2 pipeline. For every downstream consumer: - confirm the target sink configuration; - verify the standard target event type and payload for imported transactions, reversions, and metadata changes; - persist and deduplicate target at-least-once delivery using a stable target source identity plus `logSequence`; - deduplicate business effects across versions with a key that survives the Ledger v2-to-target translation; - run the full retained replay through quarantine and prove that no historical message produces a second effect; - after the mirror reaches the frozen source boundary, record the target sink cursor and destination or consumer checkpoint that prove replay has drained through the corresponding target event history; - verify recovery and redelivery after temporarily interrupting the destination; - activate effects only for events after the approved cutover boundary, then test the first post-boundary event. Do not use only the target `logSequence` as the cross-version business key: it belongs to the target cluster and does not match the identity used by an existing Ledger v2 pipeline. Document the business key for every effecting event type. If one cannot be defined, the consumer is not ready for cutover. The mirror cursor is a Ledger v2 log ID; the sink cursor is a target global log sequence. They are different sequence spaces and must not be compared numerically. Record the point at which the Ledger v2 consumer stops, the replay checkpoint reached in quarantine, and the point at which target effects become active. The activation boundary is migration evidence enforced by your consumer workflow; it is not a configurable sink start cursor. ## Validate operational readiness - target cluster has a stable leader and quorum; - disk, WAL, memory, and compaction headroom remain within the approved budget; - replay and query latency meet the migration objectives; - required indexes are ready on every replica; - authentication, TLS, monitoring, alerts, backup, and incident ownership are active; - administrators can read mirror progress and promotion status without exposing source credentials. ## Store the evidence Use a durable record rather than a chat transcript. One check can be represented as: ```yaml check: material-customer-liability-balance sourceLedger: "" sourceLogId: 4821931 targetLedger: "" targetCursor: 4821931 queryVersion: 4f36a9e expectedAdjustment: none result: pass evidence: s3://migration-evidence//2026-08-01/liability.json reviewer: finance-platform-owner reviewedAt: 2026-08-01T10:24:00Z ``` Keep secrets and access tokens out of the record. ## Final approval gate - [ ] The source is frozen and the final source log ID is recorded. - [ ] Target cursor equals that final ID, remaining logs are zero, and no error is present. - [ ] Transactions, references, accounts, balances, volumes, and reversions pass. - [ ] Metadata, typed nulls, transformations, and drops pass. - [ ] Required indexes and queries pass. - [ ] Client authentication and production workflows pass. - [ ] Target event replay has drained through the event boundary captured after the final frozen-source check. - [ ] Historical target events cannot repeat Ledger v2 business effects. - [ ] Sink and consumer checkpoints, stable deduplication keys, and the activation boundary are recorded. - [ ] Recovery, redelivery, and the rehearsal's first post-boundary event pass. - [ ] Performance and operational readiness pass. - [ ] Every difference has an owner-approved explanation. - [ ] The validation and migration owners sign the evidence. Do not promote because the maintenance window is ending. If a material check fails or lacks evidence, keep Ledger v2 frozen only as long as the approved window allows, then resume it and schedule another cutover. Continue with [Cut Over and Promote](/modules/ledger/migration/cut-over-and-promote) only after the final approval gate passes. --- ## Plan Capacity and Tune Performance Source: https://docs.formance.com/modules/ledger/operations/capacity-and-performance Plan capacity from the workload and recovery objective, not from a universal transactions-per-second number. Every voter stores the cluster state, applies every committed write, and needs enough headroom to compact, build indexes, create [checkpoints](/modules/ledger/querying/consistency-and-checkpoints), and catch up after an interruption. ## Build the workload profile Measure or estimate these inputs before choosing resources: | Input | Why it changes capacity | | --- | --- | | Peak and sustained write rate | Sets the CPU, WAL, consensus, and compaction load. | | Postings and accounts per transaction | Increases validation, state updates, serialized size, and cache lookups. | | Numscript programs and variables | Unique script text consumes parse and cache work; account fan-out increases preload work. | | Bulk request size and concurrency | Trades network and consensus overhead for request memory, latency, and retry blast radius. | | Active accounts and metadata | Determines hot-cache working set and point-read pressure. | | Retention requirements | Determines long-term data volume; history is permanent, so see the growth note below. | | Required indexes and query rate | Consumes read-index disk, memory, I/O, and background build work. | | Event delivery, backup, restore, and usage reconstruction | Adds source reads, serialization, network transfer, background writes, and temporary storage. | | Recovery objective | Determines how quickly a voter or full cluster must catch up or restore. | Include normal peak traffic and a recovery scenario in the same test. A cluster that meets latency targets only when every voter is current has not proved its catch-up capacity. ## Size storage per voter Every voting member needs space for: - the latency-sensitive Raft WAL and its spool; - Ledger data and retained history; - derived read indexes; - compaction overlap and temporary files; - checkpoints and backup staging; - the local usage projection at `/usage/usagedb`; - catch-up or restore work; - growth during the time needed to approve and complete a volume expansion. Use fast persistent storage for the WAL. Separate WAL and data I/O when the platform and Stack Operator support it, then monitor both volumes independently. Ledger 3.0 has no retention, archival, or cold-storage tier: the global log and audit history grow for the life of the cluster and are never truncated. Deleting a ledger reclaims that ledger's own projections but keeps its entries in the shared history, and the deletion is itself appended to the chain. Plan for monotonic growth — the levers are a compression profile and a volume-expansion path, not a retention policy. Multiplying the current dataset by the voter count estimates cluster-wide storage cost, not the required size of one voter. Each voter still needs the complete per-node budget. The usage store is normally small because it contains per-ledger counters and per-template records rather than transaction history. It still consumes data-volume I/O and grows independently on every member. Include its `usagestore.level.bytes` and `usagestore.memtable.bytes` signals when estimating headroom, especially while a node folds retained audit history after an upgrade or restore. Ledger protects the cluster with independent high- and low-water marks. The current server defaults block writes at 80% use and resume only after use falls below 75%. This hysteresis prevents repeated open/close cycles near a limit. Confirm how your Stack version's storage preset maps to these Ledger thresholds before sizing a volume. The Ledger server defaults apply wherever the Operator does not override them. If disk or clock protection blocks writes, the member stays ready and keeps serving reads. Kubernetes readiness is therefore not proof that the cluster accepts mutations. Inspect the structured error reason and disk state. | Write gate | HTTP result | Structured reason | Required response | | --- | --- | --- | --- | | WAL or data volume at the block mark | `429` | `WRITES_BLOCKED_DISK_FULL` | Stop non-essential writes, expand or reclaim supported storage, then verify use below the resume mark. | | Peer clock outside the configured threshold | `503` | `WRITES_BLOCKED_CLOCK_SKEW` | Restore time synchronization across members; do not disable the check as the first response. | Check the member reached by the active `ledgerctl` profile: ```bash ledgerctl cluster disk-usage --json ``` Repeat the command against every member endpoint because disk usage is node-local. ## Size memory per voter Ledger reports a conservative startup memory estimate. Use that estimate as a precheck and leave room for the Go runtime, gRPC connections, request bursts, telemetry, and Kubernetes overhead. The largest configurable consumers at their current defaults are: | Consumer | Current default budget | Main tradeoff | | --- | --- | --- | | Pebble block cache | 1 GiB | More cache reduces data reads but directly increases resident memory. | | Pebble memtables | Up to 1.5 GiB (`256 MiB × 6`) | Smaller buffers save memory but flush more often and reach write stalls sooner. | | Read-index cache and memtables | About 320 MiB | Smaller values reduce memory but increase query and index-build I/O. | | Usage-store cache and memtables | 16 MiB cache plus up to four 16 MiB memtables before projection writes stop: about 80 MiB before runtime overhead | Usually small, but every member needs the headroom and catch-up adds background I/O. | | Raft transport buffer | 10 MiB per peer | Larger buffers absorb network bursts but scale with member count. | | FSM generation cache | About 17 MiB at threshold 1,000 (2 generations x 1,000 entries x ~30 keys x ~300 bytes) | A larger working set reduces preloads but retains more entries. | | Go runtime | ~200 MiB in the startup estimate; actual use is workload-dependent | `GOMEMLIMIT` guides garbage collection but is not a hard container limit. | Set `GOMEMLIMIT` below the container memory limit so the runtime reacts before Kubernetes does. Approximately 90% is a starting point, not a guarantee: validate it with your connection count, request sizes, indexes, and telemetry. Fail the deployment precheck when the startup estimate already exceeds `GOMEMLIMIT`. If a node cannot fit its memory budget, reduce consumers in this order and remeasure after each change: 1. `--pebble-cache-size`; 2. `--pebble-memtable-size`; 3. `--pebble-memtable-stop-writes-threshold`; 4. `--read-index-cache-size` and read-index memtable settings; 5. `--cache-rotation-threshold`. Each reduction trades memory for disk I/O, preload latency, or earlier write stalls. Increasing memory without fixing insufficient disk throughput can move the failure rather than remove it. Ledger 3.0 does not expose usage-store tuning as a public server or Stack Operator contract. Budget its default cache, memtables, disk, and catch-up work rather than inventing an unsupported deployment field. ## Establish a baseline Use a staging cluster with the same voter count, storage class, failure-domain network, security, and telemetry path as production. 1. Load representative history, build the required indexes, and wait for the usage projection to catch up on every member — `usage.builder.lag` at zero, meaning every available audit sequence has been indexed locally. See [Monitor Ledger](/modules/ledger/operations/monitoring). 2. Warm the cluster with normal reads and writes. 3. Run sustained and peak traffic with representative posting width and Numscript variables. 4. Record user request latency and error rate with CPU, memory, queue, WAL, apply, compaction, and disk signals. 5. Stop or restart one follower and measure both Raft catch-up and usage-projection catch-up while traffic continues. 6. Run a backup or index build during controlled load. 7. Keep the test running long enough to observe flush and compaction cycles. Record the exact Stack release, Ledger version, resource limits, storage class, data size, client concurrency, and request distribution with the result. A number without this context is not a capacity commitment. ## Tune one constraint at a time | Observed constraint | Candidate change | Risk | Verification and rollback | | --- | --- | --- | --- | | Read I/O and cache misses are high with memory available | Increase `--pebble-cache-size` | Higher resident memory and OOM risk | Compare read latency, disk reads, and memory; restore the previous value if headroom or latency worsens. | | Write stalls coincide with memtable flush pressure | Increase memtable budget or storage throughput | More memory or cost | Require fewer stalls without breaching memory; roll back the setting if GC or OOM pressure rises. | | Preload latency is high for an active working set | Increase `--cache-rotation-threshold` | More FSM cache memory and a rolling cluster change | Compare preload latency and memory; follow the rolling-change procedure to restore the prior threshold. | | Query and index-build reads are I/O-bound | Increase read-index cache or improve disk | More memory or storage cost | Compare query latency, build time, memory, and compaction before and after. | | Peer queues fill during network bursts | Fix peer latency or review transport buffers | Larger buffers can hide a persistently slow peer and use more memory | Require queue-full events to stop without increasing catch-up time or memory beyond budget. | | Numscript cache churn is high | Reuse program text with variables, then review `--numscript-cache-size` | A larger cache does not fix generated scripts | Verify stable script text and parse/cache behavior; restore the prior size if memory rises without a hit-rate benefit. | | Large requests dominate tail latency | Reduce bulk size or concurrency | Lower throughput | Compare throughput, p99 latency, memory, and retry blast radius; restore the previous client setting if the service objective worsens. | Do not disable the Pebble WAL (`--pebble-disable-wal`), weaken hash integrity, relax durability, or bypass validation to reach a benchmark target. Treat every one of those as a correctness change, not performance tuning. ## Apply a capacity change ### Precheck 1. Capture the baseline and the exact limiting signal. 2. Define the expected improvement, safety limit, and rollback trigger. 3. Confirm healthy quorum and current backups. 4. Confirm whether the setting is node-local, cluster-wide, persisted, or immutable. 5. Confirm the setting is supported by your Stack version's Operator fields; do not add an unsupported server flag to a workload template. ### Change and verify Apply one declared change through the Stack Operator. For a rolling setting, change one follower, wait for catch-up, transfer leadership to an updated voter when required, then change the former leader. Follow [Upgrade Ledger](/modules/ledger/operations/upgrade) for the full sequence. After the rollout: ```bash ledgerctl cluster status ledgerctl cluster disk-usage --json ``` Repeat the same workload and compare the agreed user and resource metrics. Keep the change only if it improves the measured constraint without reducing quorum, recovery headroom, correctness, or another service objective. ### Roll back Stop the rollout at the first safety trigger. Restore the previous declared setting through the same Operator path, one member at a time when required. Wait for every member to catch up and repeat the controlled write/read and integrity checks. Do not combine several compensating tuning changes during rollback; that removes the baseline needed to understand the result. --- ## Cut Over and Promote Source: https://docs.formance.com/modules/ledger/migration/cut-over-and-promote Cutover changes the write authority from Ledger v2 to the target. Run it as an approved maintenance procedure with one migration lead controlling the sequence. Promotion does not check `FOLLOWING`, remaining logs, errors, balances, or business invariants. It stops replication immediately, clears the source configuration, and makes the target writable. There is no demotion operation. ## Preconditions Do not start the maintenance window until: - the complete runbook has succeeded in a rehearsal; - every Ledger v2 writer can be stopped and its owner is present; - the target repeatedly follows the live source without an unresolved error; - final validation commands and evidence storage are ready; - target clients, authentication, and routing are staged; - every target sink has completed its initial replay into a quarantined destination or consumer, follows the target, and has not repeated Ledger v2 business effects; - stable cross-version deduplication keys, sink and consumer checkpoints, and an event activation boundary are approved; - promotion access belongs to the named target administrator; - the migration lead, validation owner, and incident owner are present; - the source-resume deadline and no-go criteria are agreed. ## Assign the live roles | Role | Live responsibility | | --- | --- | | Migration lead | Calls each step, records timestamps, and makes the go/no-go decision. | | Source administrator | Stops writers, proves the freeze, records the final source head, and can resume Ledger v2 before promotion. | | Target administrator | Watches catch-up, promotes once, and verifies target mode. | | Application owner | Switches endpoint and authentication, then releases traffic. | | Validation owner | Runs final checks and signs the evidence. | | Event owner | Switches consumers and verifies one controlled event. | | Incident owner | Stops the procedure when a no-go condition is met. | ## Run the cutover Keep the following commands in one Bash session. The configured `ledgerctl` profile must point to the candidate target cluster and have permission to read and promote `$TARGET_LEDGER`. ```bash set -euo pipefail set +x # HTTP source only: # export V2_URL="https://ledger-v2.example.com" # export V2_TOKEN="" # PostgreSQL source only. Build a passwordless, hostname-verifying DSN from # the same endpoint, database, and identity configured on the mirror. The # preflight below reassembles V2_PG_DSN from these four variables, so edit # them rather than the DSN string. # Password authentication only. Mount this file from a secret manager with # mode 0600; never put the password in V2_PG_DSN. # RDS IAM authentication only. Leave the role ARN empty when the current # runtime identity already has rds-db:connect. : "${SOURCE_TRANSPORT:?set SOURCE_TRANSPORT}" : "${SOURCE_LEDGER:?set SOURCE_LEDGER}" : "${TARGET_LEDGER:?set TARGET_LEDGER}" : "${MIRROR_WAIT_TIMEOUT_SECONDS:?set MIRROR_WAIT_TIMEOUT_SECONDS}" : "${MIRROR_POLL_INTERVAL_SECONDS:?set MIRROR_POLL_INTERVAL_SECONDS}" [[ "$MIRROR_WAIT_TIMEOUT_SECONDS" =~ ^[1-9][0-9]*$ ]] [[ "$MIRROR_POLL_INTERVAL_SECONDS" =~ ^[1-9][0-9]*$ ]] command -v jq >/dev/null command -v ledgerctl >/dev/null case "$SOURCE_TRANSPORT" in http) : "${V2_URL:?set V2_URL for an HTTP source}" : "${V2_TOKEN:?set V2_TOKEN for an HTTP source}" command -v curl >/dev/null ;; postgres) : "${SOURCE_POSTGRES_AUTH:?set SOURCE_POSTGRES_AUTH}" : "${V2_PG_HOST:?set V2_PG_HOST}" : "${V2_PG_PORT:?set V2_PG_PORT}" : "${V2_PG_DATABASE:?set V2_PG_DATABASE}" : "${V2_PG_USER:?set V2_PG_USER}" # Constrain the parts, then build the DSN. Do not inspect a supplied one: # libpq percent-decodes URI parameter names and honours quoting in # keyword/value conninfo, so no shell-level check can establish the sslmode # libpq will actually apply. Assembling the URI from validated components # makes verify-full true by construction, with nothing left to parse. [[ "$V2_PG_HOST" =~ ^[A-Za-z0-9.-]+$ ]] || { printf 'V2_PG_HOST must be a bare hostname\n' >&2 exit 1 } [[ "$V2_PG_PORT" =~ ^[1-9][0-9]*$ ]] || { printf 'V2_PG_PORT must be a port number\n' >&2 exit 1 } [[ "$V2_PG_DATABASE" =~ ^[A-Za-z0-9_-]+$ ]] || { printf 'V2_PG_DATABASE must be a bare database name\n' >&2 exit 1 } [[ "$V2_PG_USER" =~ ^[A-Za-z0-9_-]+$ ]] || { printf 'V2_PG_USER must be a bare role name\n' >&2 exit 1 } V2_PG_DSN="postgresql://${V2_PG_USER}@${V2_PG_HOST}:${V2_PG_PORT}/${V2_PG_DATABASE}?sslmode=verify-full" command -v psql >/dev/null case "$SOURCE_POSTGRES_AUTH" in password) : "${PGPASSFILE:?set PGPASSFILE for password authentication}" if [ ! -r "$PGPASSFILE" ]; then printf 'PGPASSFILE must be a readable, secret-backed mode-0600 file\n' >&2 exit 1 fi pgpass_mode="$(stat -c '%a' "$PGPASSFILE" 2>/dev/null || stat -f '%Lp' "$PGPASSFILE")" if [ "$pgpass_mode" != "600" ]; then printf 'PGPASSFILE must have mode 0600; found %s\n' "$pgpass_mode" >&2 exit 1 fi ;; rds-iam) : "${V2_AWS_REGION:?set V2_AWS_REGION}" : "${V2_AWS_ROLE_SESSION_NAME:?set V2_AWS_ROLE_SESSION_NAME}" command -v aws >/dev/null ;; *) printf 'unsupported SOURCE_POSTGRES_AUTH: %s\n' \ "$SOURCE_POSTGRES_AUTH" >&2 exit 1 ;; esac ;; *) printf 'unsupported SOURCE_TRANSPORT: %s\n' "$SOURCE_TRANSPORT" >&2 exit 1 ;; esac generate_source_postgres_iam_token() { local credentials_json local access_key_id local secret_access_key local session_token if [ -z "$V2_AWS_ASSUME_ROLE_ARN" ]; then AWS_PAGER="" aws rds generate-db-auth-token \ --no-cli-pager \ --hostname "$V2_PG_HOST" \ --port "$V2_PG_PORT" \ --region "$V2_AWS_REGION" \ --username "$V2_PG_USER" || return return fi credentials_json="$(AWS_PAGER="" aws sts assume-role \ --no-cli-pager \ --role-arn "$V2_AWS_ASSUME_ROLE_ARN" \ --role-session-name "$V2_AWS_ROLE_SESSION_NAME" \ --duration-seconds 900 \ --output json)" || return access_key_id="$(jq -er '.Credentials.AccessKeyId' \ <<<"$credentials_json")" || return secret_access_key="$(jq -er '.Credentials.SecretAccessKey' \ <<<"$credentials_json")" || return session_token="$(jq -er '.Credentials.SessionToken' \ <<<"$credentials_json")" || return AWS_ACCESS_KEY_ID="$access_key_id" \ AWS_SECRET_ACCESS_KEY="$secret_access_key" \ AWS_SESSION_TOKEN="$session_token" \ AWS_PAGER="" \ aws rds generate-db-auth-token \ --no-cli-pager \ --hostname "$V2_PG_HOST" \ --port "$V2_PG_PORT" \ --region "$V2_AWS_REGION" \ --username "$V2_PG_USER" || return } source_psql() { local iam_token case "$SOURCE_POSTGRES_AUTH" in password) psql "$V2_PG_DSN" "$@" ;; rds-iam) iam_token="$(generate_source_postgres_iam_token)" || return if [ -z "$iam_token" ] || [[ "$iam_token" == *$'\n'* ]]; then printf 'RDS IAM token generation returned an invalid value\n' >&2 return 1 fi PGPASSWORD="$iam_token" psql "$V2_PG_DSN" "$@" ;; esac } read_source_log_id() { local source_head_json local source_log_id case "$SOURCE_TRANSPORT" in http) source_head_json="$(curl --fail --silent --show-error \ --header "Authorization: Bearer $V2_TOKEN" \ "$V2_URL/v2/$SOURCE_LEDGER/logs?pageSize=1")" source_log_id="$(jq -er ' .cursor.data | if type != "array" then error("source response has no cursor.data array") elif length == 0 then "0" else (.[0].id | tostring) as $id | if $id | test("^[1-9][0-9]*$") then $id else error("newest source log has an invalid id") end end ' <<<"$source_head_json")" ;; postgres) source_log_id="$( source_psql \ --no-psqlrc \ --no-password \ --set=ON_ERROR_STOP=1 \ --set=source_ledger="$SOURCE_LEDGER" \ --quiet \ --tuples-only \ --no-align <<'SQL' SELECT bucket AS source_bucket FROM _system.ledgers WHERE name = :'source_ledger' \gset \if :{?source_bucket} SELECT COALESCE(MAX(id), 0)::text FROM :"source_bucket".logs WHERE ledger = :'source_ledger'; \else \warn 'source ledger did not resolve to exactly one bucket' \endif SQL )" ;; esac if [[ "$source_log_id" == *$'\n'* ]] || ! [[ "$source_log_id" =~ ^(0|[1-9][0-9]*)$ ]]; then printf 'source head must be exactly one unsigned integer row\n' >&2 return 1 fi printf '%s\n' "$source_log_id" } ``` For a PostgreSQL source, the reader needs `CONNECT` on the database, `USAGE` on `_system` and the selected bucket schema, `SELECT` on `_system.ledgers`, and `SELECT` on the selected bucket's `logs` table. It does not need write access. Protect credentials as secrets, keep shell tracing disabled, and use the same hostname, database, TLS policy, and identity that the mirror uses. The PostgreSQL branch does not read `V2_URL` or `V2_TOKEN`. `V2_PG_HOST` must be the exact RDS endpoint used in the DSN, not a different alias. The RDS IAM helper generates a fresh 15-minute token for every psql process. When an assume-role ARN is set, it also obtains fresh role credentials for that invocation. Neither the token nor the temporary AWS credentials enter command-line arguments, global environment variables, or evidence files. The shared source-head function resolves the exact bucket recorded for `$SOURCE_LEDGER`, passes that server-provided name through psql's identifier quoting, and rejects absent, duplicate, or non-numeric results. ### 1. Freeze Ledger v2 writes Stop API writers, background workers, scheduled jobs, support tools, and administrative scripts. Reject or queue new business operations according to the maintenance plan. Prove the freeze with application metrics and Ledger v2 logs. Waiting for traffic to become quiet is not the same as blocking every writer. ### 2. Record the final source log ID After the freeze is effective, derive `FINAL_SOURCE_LOG_ID` from the selected source. The function returns `0` for an empty source and fails if the HTTP response or PostgreSQL result is invalid: ```bash FINAL_SOURCE_LOG_ID="$(read_source_log_id)" jq -n \ --arg finalSourceLogId "$FINAL_SOURCE_LOG_ID" \ '{finalSourceLogId: $finalSourceLogId}' \ > final-source-head.json printf 'final source log ID: %s\n' "$FINAL_SOURCE_LOG_ID" ``` Store `final-source-head.json` and `FINAL_SOURCE_LOG_ID` in the migration record. ### 3. Ingest the final tail Poll the target until the complete progress tuple reaches the frozen boundary. An active mirror error fails immediately. A timeout fails with the last cursor, observed source head, remaining count, and error; it never falls through to promotion. ```bash MIRROR_WAIT_DEADLINE=$((SECONDS + MIRROR_WAIT_TIMEOUT_SECONDS)) while :; do TARGET_LEDGER_JSON="$(ledgerctl ledgers get "$TARGET_LEDGER" --json)" jq -e '.mirrorSyncProgress != null' <<<"$TARGET_LEDGER_JSON" >/dev/null TARGET_MODE="$(jq -er '.mode | split("_")[-1]' <<<"$TARGET_LEDGER_JSON")" TARGET_CURSOR="$(jq -er '(.mirrorSyncProgress.cursor // 0) | tostring' <<<"$TARGET_LEDGER_JSON")" TARGET_SOURCE_HEAD="$(jq -er '(.mirrorSyncProgress.sourceLogCount // 0) | tostring' <<<"$TARGET_LEDGER_JSON")" TARGET_REMAINING="$(jq -er '(.mirrorSyncProgress.remainingLogs // 0) | tostring' <<<"$TARGET_LEDGER_JSON")" TARGET_ERROR_JSON="$(jq -c '.mirrorSyncProgress.error // null' <<<"$TARGET_LEDGER_JSON")" printf 'mode=%s cursor=%s observedSource=%s remaining=%s error=%s\n' \ "$TARGET_MODE" \ "$TARGET_CURSOR" \ "$TARGET_SOURCE_HEAD" \ "$TARGET_REMAINING" \ "$TARGET_ERROR_JSON" if [ "$TARGET_MODE" != "MIRROR" ]; then printf 'target left MIRROR mode before the approved promotion\n' >&2 exit 1 fi if [ "$TARGET_ERROR_JSON" != "null" ]; then printf 'mirror reported an active error; do not promote\n' >&2 exit 1 fi if [ "$TARGET_CURSOR" = "$FINAL_SOURCE_LOG_ID" ] && [ "$TARGET_SOURCE_HEAD" = "$FINAL_SOURCE_LOG_ID" ] && [ "$TARGET_REMAINING" = "0" ]; then break fi if [ "$SECONDS" -ge "$MIRROR_WAIT_DEADLINE" ]; then printf 'mirror did not reach the final source boundary before timeout\n' >&2 exit 1 fi sleep "$MIRROR_POLL_INTERVAL_SECONDS" done ``` Confirm that the source head did not move after the freeze, then save a credential-safe target snapshot: ```bash CONFIRMED_SOURCE_LOG_ID="$(read_source_log_id)" if [ "$CONFIRMED_SOURCE_LOG_ID" != "$FINAL_SOURCE_LOG_ID" ] || [ "$TARGET_CURSOR" != "$FINAL_SOURCE_LOG_ID" ] || [ "$TARGET_SOURCE_HEAD" != "$FINAL_SOURCE_LOG_ID" ] || [ "$TARGET_REMAINING" != "0" ] || [ "$TARGET_ERROR_JSON" != "null" ]; then printf 'final source or mirror gate changed; do not promote\n' >&2 exit 1 fi jq \ --arg finalSourceLogId "$FINAL_SOURCE_LOG_ID" \ '{ name, mode, finalSourceLogId: $finalSourceLogId, mirrorSyncProgress }' <<<"$TARGET_LEDGER_JSON" > final-target-progress.json ``` Continue only when every assertion in the two blocks passes: - `cursor` equals the final source log ID; - `sourceLogCount` equals the same frozen ID; - `remainingLogs` is `0`; - no mirror error is present; - the source head remains unchanged because writes are frozen. `FOLLOWING` alone is not sufficient. ### 4. Run final validation Run the critical checklist from [Validate the Migration](/modules/ledger/migration/validation) against the final log boundary. Store the commands, normalized results, transformation adjustments, and signatures in the migration record. The migration lead announces `GO` only after the validation owner approves the result. Otherwise, announce `NO-GO`, do not promote, and follow the source-resume procedure. ### 5. Prepare the client and consumer switch Confirm the staged target configuration one last time: - target base URL or gRPC endpoint; - target authentication issuer, audience, and scopes; - target ledger name; - retry and idempotency-key behavior; - required indexes; - target sink cursor and destination or consumer replay checkpoint; - stable target-delivery and cross-version business-effect keys; - Ledger v2 consumer stop point and target effect-activation boundary; - alert and dashboard context. A newly created target sink starts at cursor `0` and replays every supported event retained in the target global log, including mirrored Ledger v2 history. There is no sink start-cursor option in the current configuration. Do not create or enable an effecting consumer here under the assumption that it will receive only new target writes. Before promotion, keep target delivery in quarantine and prove that it has drained through the target event boundary captured after final mirror validation. Historical messages must be durably accepted, inspected, and either matched to an existing Ledger v2 business-effect receipt or explicitly reconciled without executing the effect again. Record the sink cursor and the destination-specific consumer checkpoint together. Do not compare the target sink cursor numerically with the Ledger v2 mirror cursor; they belong to different sequence spaces. The activation boundary belongs to the migration record and consumer workflow; it is not a value configured on the Ledger sink. If you cannot prove that every event at or below that boundary is suppressed or deduplicated, announce `NO-GO` and keep writers stopped. Keep writers stopped until promotion, endpoint switching, and consumer activation are complete. ### 6. Promote exactly once This is the only irreversible step in the runbook. Before you run the command, confirm the recovery position in [No-go and recovery decisions](#no-go-and-recovery-decisions): after promotion you cannot demote, and once a target-only write lands, Ledger v2 no longer holds all production state. Use the same control surface that owns the ledger. Do not promote from the snapshot captured before final validation. Immediately before the promotion request, reread both sides and fail unless the complete frozen boundary still holds: ```bash PRE_PROMOTION_SOURCE_LOG_ID="$(read_source_log_id)" PRE_PROMOTION_TARGET_JSON="$(ledgerctl ledgers get "$TARGET_LEDGER" --json)" jq -e '.mirrorSyncProgress != null' <<<"$PRE_PROMOTION_TARGET_JSON" >/dev/null PRE_PROMOTION_MODE="$(jq -er '.mode | split("_")[-1]' <<<"$PRE_PROMOTION_TARGET_JSON")" PRE_PROMOTION_CURSOR="$(jq -er '(.mirrorSyncProgress.cursor // 0) | tostring' <<<"$PRE_PROMOTION_TARGET_JSON")" PRE_PROMOTION_SOURCE_HEAD="$(jq -er '(.mirrorSyncProgress.sourceLogCount // 0) | tostring' <<<"$PRE_PROMOTION_TARGET_JSON")" PRE_PROMOTION_REMAINING="$(jq -er '(.mirrorSyncProgress.remainingLogs // 0) | tostring' <<<"$PRE_PROMOTION_TARGET_JSON")" PRE_PROMOTION_ERROR_JSON="$(jq -c '.mirrorSyncProgress.error // null' <<<"$PRE_PROMOTION_TARGET_JSON")" if [ "$PRE_PROMOTION_SOURCE_LOG_ID" != "$FINAL_SOURCE_LOG_ID" ] || [ "$PRE_PROMOTION_MODE" != "MIRROR" ] || [ "$PRE_PROMOTION_CURSOR" != "$FINAL_SOURCE_LOG_ID" ] || [ "$PRE_PROMOTION_SOURCE_HEAD" != "$FINAL_SOURCE_LOG_ID" ] || [ "$PRE_PROMOTION_REMAINING" != "0" ] || [ "$PRE_PROMOTION_ERROR_JSON" != "null" ]; then printf 'pre-promotion boundary changed; do not promote\n' >&2 exit 1 fi jq \ --arg finalSourceLogId "$FINAL_SOURCE_LOG_ID" \ '{ name, mode, finalSourceLogId: $finalSourceLogId, mirrorSyncProgress }' <<<"$PRE_PROMOTION_TARGET_JSON" > pre-promotion-gate.json ``` If this check fails, return to catch-up and validation. A changed source head creates a new final boundary; it cannot be accepted by editing the evidence file. #### Operator-owned targets Operator-managed promotion is not supported in the Stack v4 preview. If an Operator owns the ledger resource, do not bypass its reconciliation with a direct Ledger call; wait for the published promotion workflow. The following commands apply only when `ledgerctl` or HTTP is the approved control surface for that target. #### `ledgerctl` Promotion cannot be undone: there is no demotion operation. The CLI promotion request carries no idempotency key, so a command that times out leaves the outcome unknown from the client side. Read the ledger mode after every attempt, including a timeout or an error, and never send a second promotion request on the strength of a failed first one. Set a bounded request timeout and record the output and exit status. When the target has `require_signatures=true`, use a `ledgerctl` profile configured with a registered signing key or add `--signing-key --signing-key-id ` before `ledgers promote`. The CLI signs the gRPC Apply batch. A signed gRPC client is the other supported control surface; the HTTP promotion route cannot carry the required request signature. The command block below assumes the active profile supplies the signing key when required; see [Request and Response Signing](/modules/ledger/operations/signing). ```bash : "${PROMOTION_TIMEOUT:?set PROMOTION_TIMEOUT}" set +e PROMOTION_OUTPUT="$(ledgerctl ledgers promote "$TARGET_LEDGER" \ --yes \ --timeout "$PROMOTION_TIMEOUT" 2>&1)" PROMOTION_EXIT="$?" set -e printf 'promotion exit=%s\n%s\n' \ "$PROMOTION_EXIT" \ "$PROMOTION_OUTPUT" \ > promotion-ledgerctl-attempt.txt POST_PROMOTION_JSON="$(ledgerctl ledgers get "$TARGET_LEDGER" --json)" OBSERVED_MODE="$(jq -er ' if .mode != null then (.mode | split("_")[-1]) elif .mirrorSource == null and .mirrorSyncProgress == null then "NORMAL" else error("cannot determine ledger mode") end ' <<<"$POST_PROMOTION_JSON")" case "$OBSERVED_MODE" in NORMAL) printf 'promotion committed; do not retry\n' ;; MIRROR) printf 'promotion not observed; rerun every cutover gate before one retry\n' >&2 exit 75 ;; *) printf 'unknown ledger mode after promotion attempt: %s\n' "$OBSERVED_MODE" >&2 exit 1 ;; esac ``` The current structured output can omit the zero-value `NORMAL` enum; the check accepts that only when mirror source and progress are also absent. `NORMAL` means the request committed even if the first command timed out. Do not send another promotion request. #### HTTP API The token needs effective `ledger:LedgerWrite` and `ledger:LedgerRead` scopes. With the default scope mapping, the virtual `ledger:write` and `ledger:read` scopes include them. Set the complete HTTP surface: Use this HTTP workflow only when the target accepts unsigned writes. The HTTP promotion handler submits an unsigned request and is rejected when `require_signatures=true`. In that case, use the signed `ledgerctl` or gRPC workflow above; do not disable mandatory signing for cutover. ```bash : "${TARGET_URL:?set TARGET_URL}" : "${TARGET_TOKEN:?set TARGET_TOKEN}" : "${TARGET_LEDGER:?set TARGET_LEDGER}" : "${PROMOTION_TIMEOUT_SECONDS:?set PROMOTION_TIMEOUT_SECONDS}" : "${CUTOVER_ID:?set CUTOVER_ID}" : "${PROMOTION_KEY:?set PROMOTION_KEY once for this cutover}" [[ "$PROMOTION_TIMEOUT_SECONDS" =~ ^[1-9][0-9]*$ ]] test "${#PROMOTION_KEY}" -le 256 command -v curl >/dev/null command -v jq >/dev/null PROMOTION_BODY_FILE="$(mktemp)" trap 'rm -f "$PROMOTION_BODY_FILE"' EXIT set +e PROMOTION_STATUS="$(curl --silent --show-error \ --max-time "$PROMOTION_TIMEOUT_SECONDS" \ --output "$PROMOTION_BODY_FILE" \ --write-out '%{http_code}' \ --request POST \ --header "Authorization: Bearer $TARGET_TOKEN" \ --header "Idempotency-Key: $PROMOTION_KEY" \ "$TARGET_URL/v3/$TARGET_LEDGER/promote")" PROMOTION_EXIT="$?" set -e printf 'curlExit=%s httpStatus=%s idempotencyKey=%s\n' \ "$PROMOTION_EXIT" \ "$PROMOTION_STATUS" \ "$PROMOTION_KEY" \ > promotion-http-attempt.txt PROMOTION_KEY_CONFLICT="false" if [ "$PROMOTION_STATUS" = "409" ] && jq -e '.errorCode == "IDEMPOTENCY_KEY_CONFLICT"' \ "$PROMOTION_BODY_FILE" >/dev/null 2>&1; then PROMOTION_KEY_CONFLICT="true" fi POST_PROMOTION_JSON="$(curl --fail --silent --show-error \ --header "Authorization: Bearer $TARGET_TOKEN" \ "$TARGET_URL/v3/$TARGET_LEDGER")" OBSERVED_MODE="$(jq -er ' .data | if .mode != null then (.mode | split("_")[-1]) elif .mirrorSource == null and .mirrorSyncProgress == null then "NORMAL" else error("cannot determine ledger mode") end ' <<<"$POST_PROMOTION_JSON")" if [ "$PROMOTION_KEY_CONFLICT" = "true" ]; then printf 'promotion idempotency key belongs to different request content; stop\n' >&2 printf 'observed target mode: %s\n' "$OBSERVED_MODE" >&2 exit 1 fi case "$OBSERVED_MODE" in NORMAL) printf 'promotion committed; do not retry\n' ;; MIRROR) printf 'promotion not observed; rerun every cutover gate before one retry\n' >&2 exit 75 ;; *) printf 'unknown ledger mode after promotion attempt: %s\n' "$OBSERVED_MODE" >&2 exit 1 ;; esac ``` The API normally returns `201`, but the follow-up mode read decides whether the operation committed. If the POST timed out and the read reports `NORMAL`, record the timeout and continue without retrying. The current runtime handler accepts `Idempotency-Key` on promotion even though that header is not yet described by the OpenAPI operation. Keep `PROMOTION_KEY` stable in the approved cutover record. If an ambiguous result leads to one authorized manual retry, send the same method, target URL, empty body, and key. A changed key creates a different request identity. The mode read remains authoritative regardless of the POST status or idempotency response. `IDEMPOTENCY_KEY_CONFLICT` is not an ambiguous retry result. It means the cluster-level key is already bound to different request content. Do not reuse that key or choose another one inside the same attempt. If the target remains in `MIRROR`, start a separately approved attempt with a fresh key only after rerunning every cutover gate. If the target is already `NORMAL`, stop and investigate which request promoted it before opening traffic. Never put promotion in an automatic retry loop. If the follow-up read reports `MIRROR`, repeat the source freeze proof, derive the final source ID again, rerun the complete mirror gate, rerun final validation, and obtain a new `GO` decision before one manual retry. For an ambiguous HTTP result without an idempotency conflict, reuse the original `PROMOTION_KEY` and exact request. If that second attempt is ambiguous or still reports `MIRROR`, stop and escalate; do not send a third request. If the follow-up read itself fails, the state is unknown and no retry is safe until mode can be read. ### 7. Switch applications and consumers Stop the Ledger v2 consumer at its recorded checkpoint. Confirm once more that the quarantined target replay has drained through the approved activation boundary and that no event at or below it can produce an effect. Switch the target consumer from quarantine to active processing for events after that boundary. Keep both protections in place: - deduplicate target redelivery with the target source identity and `logSequence`; - deduplicate cross-version business effects with the stable business key validated during rehearsal. Only then route application traffic to the target endpoint and activate the staged target credentials. Do not recreate the sink during this step: a new sink would start again at cursor `0` and replay the retained history. Do not reopen all traffic yet. ### 8. Send one controlled production write Release one approved business operation with a unique reference and idempotency key. Do not add synthetic financial state merely to test the system. Verify independently: - the write succeeded once; - a retry returns the same accepted result; - the transaction is readable by ID and reference; - affected account balances and volumes match the expected amounts; - the audit or correlation data identifies the caller; - the expected downstream event arrived once after deduplication. When the operation supports a business-approved reversal, verify that procedure in rehearsal rather than improvising a production cleanup. ### 9. Reopen traffic gradually Increase traffic according to the cutover plan while watching: - write and read errors; - latency and retry rate; - idempotency or reference conflicts; - leader and quorum health; - disk and compaction pressure; - index readiness and query failures; - event delivery lag and duplicate handling; - business reconciliation signals. Record the promotion, first write, and full-traffic timestamps. ## No-go and recovery decisions | Point | Available action | | --- | --- | | Before promotion | Resume Ledger v2 writes, investigate, or recreate the mirror. The target is still disposable. | | After promotion, before a target-only write | You cannot demote. You may discard the promoted target and build a new mirror from the still-frozen Ledger v2 source, subject to the maintenance decision. | | After a target-only write | Ledger v2 no longer contains all production state. Routing writes back is not a data rollback. Use the incident and reconciliation plan. | Do not delete the Ledger v2 source immediately. Keep it read-only for the approved forensic and retention period, with write credentials revoked or disabled. ## Close the migration - [ ] Target mode is normal and mirror progress no longer advances. - [ ] First controlled write, retry, read, balance, and event checks pass. - [ ] Target replay checkpoints and the consumer activation boundary are archived. - [ ] No mirrored event repeated a Ledger v2 business effect. - [ ] Production traffic is fully routed to the target. - [ ] Ledger v2 remains read-only under the retention policy. - [ ] Final source log ID and preserved target cursor are recorded. - [ ] Validation evidence, rule hash, approvals, and timestamps are archived. - [ ] Temporary source credentials and network access are revoked. - [ ] Dashboards, alerts, backups, and on-call ownership reference the target. - [ ] The migration lead and application owner sign completion. The preserved cursor is useful forensic context after promotion, but it is not an ongoing replication position: the promoted ledger no longer follows Ledger v2. --- ## Use Maintenance Mode Source: https://docs.formance.com/modules/ledger/operations/maintenance-mode Maintenance mode blocks Ledger mutations across the cluster while reads continue. Use it when a backup, integrity check, upgrade step, or incident response requires a stable write boundary. Maintenance mode does not drain application requests, stop client retries, fix an unhealthy cluster, or make an unsafe operation reversible. Coordinate the boundary with application owners before enabling it. ## What changes | Operation | During maintenance mode | | --- | --- | | Create or delete a ledger | Blocked | | Create, revert, or bulk-process transactions | Blocked | | Save or delete metadata and other configuration | Blocked | | List or get Ledger data | Allowed | | Inspect cluster status | Allowed | | Disable maintenance mode | Allowed | Blocked writes return gRPC `Unavailable` or HTTP `503 Service Unavailable`. Clients must treat the response as an intentional service boundary and wait for the maintenance owner rather than retrying in a tight loop. The setting is committed through the cluster and survives process restarts. Restarting members does not clear it. ## Enter maintenance mode ### Precheck 1. Name the maintenance owner, affected applications, expected duration, and abort criteria. 2. Confirm that the procedure has its own rollback and that a recent backup exists when data could be affected. 3. Check that the cluster has a leader, the expected voters, and no member still catching up. 4. Confirm that disk and clock health are not already blocking writes. 5. Pause producers and background workers, then drain in-flight writes at the application boundary. 6. Confirm that clients will honor `503` without uncontrolled retries. ```bash ledgerctl cluster status ``` Do not enter planned maintenance to hide an unknown no-leader, disk, or clock-skew failure. Diagnose that condition first. ### Enable the write boundary ```bash ledgerctl cluster maintenance enable ``` If the cluster requires request signing, use the configured signing key: ```bash ledgerctl \ --signing-key /secure/path/maintenance.seed \ --signing-key-id maintenance \ cluster maintenance enable ``` Inspect the cluster again: ```bash ledgerctl cluster status ``` Proceed only after the status shows maintenance mode enabled and application telemetry shows no successful writes after the boundary. ### Verify independently Use a disposable, idempotent write in a non-production verification ledger to confirm the expected `503` result. Then read an existing account or transaction to confirm that required read access still works. Record the last accepted Ledger log or transaction position before starting the maintenance task. Do not use a real customer transaction as the negative write test. Client libraries may queue or retry it after maintenance ends. ## Run the maintenance task Follow the owning runbook rather than improvising inside the window. While the task runs: - keep the application write drain active; - keep the cluster status, voter count, disk, memory, and alert state visible; - record any leadership change or member restart; - stop at the procedure's rollback trigger; - keep maintenance mode enabled when validation is incomplete. Reads may continue, but choose the required consistency deliberately. A long checkpoint, restore, or synchronization can delay a current read even though maintenance mode permits it. ## Leave maintenance mode ### Precheck Before reopening writes, confirm: - the maintenance procedure reports a successful result; - the cluster has a leader and the expected caught-up voters; - no restore, synchronization, or rolling change remains in progress; - required indexes and event workflows are in their expected state; - the integrity and task-specific checks pass; - application owners are ready to resume producers in a controlled order. ### Disable the write boundary ```bash ledgerctl cluster maintenance disable ledgerctl cluster status ``` If signing is required: ```bash ledgerctl \ --signing-key /secure/path/maintenance.seed \ --signing-key-id maintenance \ cluster maintenance disable ``` ### Verify and reopen 1. Confirm that cluster status reports maintenance mode disabled. 2. Send one controlled write with a new idempotency key. 3. Read the result back with current-read consistency. 4. Confirm that the cluster applied the write on every expected voter. 5. Resume one producer or traffic segment. 6. Watch latency, structured errors, queue load, and downstream delivery. 7. Reopen the remaining traffic only after the signals return to baseline. The result is a recorded write boundary, a validated maintenance outcome, and a controlled first write after reopening. ## Recover when the procedure fails If the maintenance task fails, keep maintenance mode enabled and execute that task's rollback. Do not reopen because the original maintenance window has expired. If the disable command fails: 1. confirm that a leader and quorum exist; 2. confirm that authentication and any required request signature are valid; 3. inspect whether the request reached a follower that cannot forward to the leader; 4. retry with the same authorized procedure after restoring cluster connectivity; 5. escalate with cluster status and the structured error when the flag remains set. Do not edit persistent state or restart every member to clear maintenance mode. The flag is replicated state and must be changed through the maintenance command. --- ## Back Up a Cluster Source: https://docs.formance.com/modules/ledger/operations/backup A backup is useful only when you can identify its recovery point, restore it with the intended Ledger release, and verify the restored state. Keep the backup manifest and its sequence boundaries with every recovery record. Ledger provides two backup operations: - a **full backup** captures a point-in-time checkpoint of the cluster's primary hot data; - an **incremental backup** exports log and audit entries committed after the existing backup boundary. Create a full backup before the first incremental backup. A later full backup establishes a new base and replaces the previous incremental chain. ## Before you start Confirm that the active `ledgerctl` profile targets the intended cluster and that the cluster has a leader. When authentication is enabled, both `ledgerctl store backup` and `ledgerctl store incremental-backup` require the granular `ledger:ClusterWrite` scope. The virtual `ledger:admin` scope includes this permission; `ledger:write` does not. The active Ledger leader, not the `ledgerctl` host, opens the object-storage connection. Ensure every node that may become leader can reach the destination and has the required read, write, list, and delete permissions through the provider's default credential chain. When you pass static credentials as CLI flags, `ledgerctl` serializes them into the backup RPC instead; protect that invocation and do not expose the values in process logs or scheduler configuration. Verify the complete server-side path against a non-production destination before scheduling a production backup. Ledger's local usage projection is not part of a full or incremental backup. The store at `/usage/usagedb` contains derived ledger statistics and Numscript template usage; it is reconstructed from the restored audit chain after Ledger starts. A successful backup therefore does not preserve or certify those figures. ## Choose a destination Backend support depends on how you run the backup: | Surface | Supported destinations | | --- | --- | | `ledgerctl store backup` and `ledgerctl store incremental-backup` | Amazon S3, S3-compatible storage such as MinIO, and Azure Blob Storage | | The operator's scheduled `Backup` resource | Amazon S3 and S3-compatible storage only | Use workload identity or the provider's default credential chain on every possible Ledger leader where possible. If you use static credentials, store and rotate them through your deployment's secret-management process rather than committing them with the backup configuration or scheduler. ## Create a full backup The following command writes a full backup to S3: ```bash ledgerctl store backup \ --driver s3 \ --s3-bucket ledger-production-backups \ --s3-region eu-west-1 \ --bucket-id ledger-production ``` For an S3-compatible service, add its endpoint: ```bash ledgerctl store backup \ --driver s3 \ --s3-bucket ledger-backups \ --s3-endpoint https://minio.example.com \ --bucket-id ledger-production ``` For Azure Blob Storage: ```bash ledgerctl store backup \ --driver azure \ --azure-account-name financearchive \ --azure-container ledger-backups \ --bucket-id ledger-production ``` When you omit `--azure-account-key`, Ledger uses `DefaultAzureCredential`. When you omit static S3 credentials, Ledger uses the default AWS credential chain. The cluster continues to accept writes while it creates the checkpoint. The reported log, audit, and applied-index boundaries identify the state included in that backup. At a given `bucketId`, a full backup publishes a new manifest with an empty incremental-export list. Only after the new manifest is durable does Ledger remove the checkpoint objects and incremental exports that the previous manifest no longer references. This ordering protects the new backup from an interrupted cleanup, but it also means that one `bucketId` represents one current backup chain rather than a built-in history of full-backup generations. Object-store versioning and retention are separate from Ledger backup retention. If you need multiple independent recovery generations, use protected object versions, replication, or distinct `bucketId` values and test how you select each generation during restore. ## Add incremental backups Run incremental backups more frequently than full backups to reduce the recovery point objective: ```bash ledgerctl store incremental-backup \ --driver s3 \ --s3-bucket ledger-production-backups \ --s3-region eu-west-1 \ --bucket-id ledger-production ``` Ledger appends new log and audit segments to the backup manifest. It serializes backup work for the same destination. If leadership changes or a run fails, inspect the result and retry the command; do not infer completion from the presence of a partially uploaded object. That serialization is cluster-wide rather than per node, and full and incremental runs share one slot per destination. Whichever run reaches the destination second is rejected with `backup already in progress for this destination` instead of racing the first. Overlapping schedules therefore produce failed runs rather than damaged chains — but which run fails is not something you can predict. Nothing gives a full backup precedence over an incremental one, so a weekly full at `0 2 * * 0` and an hourly incremental at `0 * * * *` can lose either member of the Sunday 02:00 pair, and losing the full is the more expensive outcome: the next one is a week away. Offset the incremental schedule away from the full-backup slot by more than a full backup usually takes. That is a mitigation, not a guarantee — an overrunning run will still occasionally collide with the next slot — so treat this specific rejection as an expected event rather than paging on it, and alert instead on a destination that has gone too long without a successful run of either kind. ## Schedule backups with the Ledger operator The Ledger Kubernetes operator ships a `Backup` resource that runs both commands on cron schedules. Each firing creates a `BackupRun`, which executes the matching `ledgerctl` invocation in a one-shot Job and reports its outcome back on the parent `Backup`: ```yaml apiVersion: ledger.formance.com/v1alpha1 kind: Backup metadata: name: production spec: clusterRef: customer-ledger destination: driver: s3 bucketId: customer-production s3: bucket: customer-ledger-backups region: eu-west-1 schedule: full: "0 2 * * 0" incremental: "30 * * * *" successfulRunsHistoryLimit: 3 failedRunsHistoryLimit: 1 ``` The incremental schedule above is offset to the half hour so the two never fire in the same minute, which is the collision described under [Add incremental backups](#add-incremental-backups). An offset narrows that window but does not close it: each field takes a single five-field cron expression, so neither schedule can carve out a gap around the other, and a full backup running longer than thirty minutes still meets the following incremental. Size the offset against the full-backup duration you actually observe, and expect an occasional rejection when a run overruns it. Both schedule entries are optional, and the operator holds the incremental schedule back until at least one full run has succeeded. A `Backup` with neither entry is a valid destination template for `BackupRun` resources you create yourself. The resource status reports the same boundaries the CLI does. `status.lastFullBackup` and `status.lastIncrementalBackup` each carry the completion time, duration, and the last log and audit sequences, and the full-backup status adds the last applied index and the file counts uploaded, deleted, and pruned. Plan around four constraints: - the operator's destination driver accepts `s3` only. Azure Blob Storage is available through `ledgerctl` but not through a scheduled `Backup`; - `successfulRunsHistoryLimit` and `failedRunsHistoryLimit` retain Kubernetes `BackupRun` resources per run type. They do not retain backup artifacts or restore points in object storage; - one `Backup` carries one fixed `bucketId`, so rotating immutable retention points requires orchestration outside the operator; - `spec.timeout`, one hour by default, caps how long each `ledgerctl` invocation may run before the client aborts. Size it against the ledger and the throughput of the destination bucket. Whether a given Formance Stack release installs this operator is a separate question from whether the operator supports scheduled backups. Check which custom resources your Stack version provides before relying on either. Where the operator is unavailable, schedule the `ledgerctl` commands with your existing workload scheduler and reproduce the same properties: a stable `--bucket-id`, distinct full and incremental schedules, and a recorded terminal status carrying the sequence boundaries. Treat the `--bucket-id` and `spec.destination.bucketId` value as a stable operational identifier. Change it deliberately when starting a separate recovery chain, and record which chain each restore exercise used. ## Record proof of completion For every successful run, retain: - the destination and `bucketId`; - the backup manifest and its creation time; - the last log sequence, last audit sequence, and last applied index reported by the run; - the number of checkpoint files or incremental segments written; - the Ledger release used to create the backup; - the command result or scheduler status. When you schedule backups through the operator, require both the terminal success state and the sequence boundaries recorded on the `Backup` status. A completed Kubernetes Job alone is not sufficient evidence that the manifest was published. A successful upload proves that the backup operation completed. It does not prove that the data can be restored or that the recovered application state meets your requirements. Rehearse the procedure in [Restore a Cluster](/modules/ledger/operations/restore) and record the result. ## Define the recovery policy Document these decisions before production: 1. Set the maximum acceptable data loss and derive the incremental schedule from it. 2. Set the recovery-time objective and test it with production-scale data. 3. Retain enough independent recovery generations to survive a damaged latest chain; a new full backup at the same `bucketId` replaces the manifest and purges objects from the previous chain. 4. Apply object lock, versioning, encryption, and access controls according to your risk model. 5. Assign an owner to review failures and expired recovery exercises. The recovery set for a cluster is the latest usable backup chain on its own. Ledger 3.0 keeps every log and audit entry in the primary store for the life of the bucket, so a backup captures the whole of it — there is no separate archive tier to retain alongside it. After a restore, the usage projection folds the audit entries reachable from the recovered primary store. A backup chain that covers the whole store therefore reproduces the full counters; a partial or truncated recovery is the only way they settle below their historical totals, and `usage.builder.lag` reaching zero says nothing about whether the recovery was complete. Record which backup chain a restore came from whenever these figures feed reporting, billing, or limits. --- ## Restore a Cluster Source: https://docs.formance.com/modules/ledger/operations/restore Restore into a fresh data directory and an isolated environment. Keep application writers pointed at the existing cluster until the recovered cluster passes both Ledger integrity checks and your business checks. Ledger provides two restore workflows: | Workflow | Use it when | | --- | --- | | Restore service | An operator needs separate download, validation, preview, and finalization gates | | Offline bootstrap | Automation needs to build a ready data directory with one command and no running Ledger server | Both workflows support S3, S3-compatible storage, and Azure Blob Storage. Both apply incremental segments after the full checkpoint before they finalize the recovered state. ## Before you start Record: - the chosen full backup and every incremental segment that follows it; - the expected last log and audit sequences; - the Ledger release used to create the backup and the release selected for restore; - the target cluster identity, storage paths, and security configuration; - the application workflows and balances that must pass before cutover. Persisted-format compatibility depends on the Ledger release. Restore with a release combination that you have tested. Do not assume that development or prerelease builds can exchange persisted data. Use a dedicated target path. Before either workflow starts, Ledger checks for state that could make a normal startup select stale data under a restored boundary: | Guard | Required state | | --- | --- | | `RESTORED` | The marker must not exist. Its presence means a restore has already committed. | | `live/`, `live.staging/`, `live.discard/` | None of these directories may exist. | | `checkpoints/` | No numbered checkpoint may exist. The only recoverable exception is an orphaned `checkpoints/0` with no marker and no `live*` directory; Ledger removes that half-finalized checkpoint before retrying. | Do not copy a backup over an existing data directory to bypass these guards. Use another path when the ownership of a leftover directory is unclear. ## Use the restore service ### 1. Start restore mode Start one node with a fresh data directory. Do not combine `--restore` with `--bootstrap` or `--join`. ```bash ledger run \ --node-id 1 \ --cluster-id recovered-production \ --data-dir ./recovered-data \ --restore \ --grpc-port 8888 ``` Restore RPCs are unauthenticated and bind to `127.0.0.1` by default. Keep this default for local operations. If you must expose restore mode remotely, restrict network access and configure TLS before changing `--restore-listen`. ### 2. Download the recovery set From S3: ```bash ledgerctl --server localhost:8888 --insecure restore download \ --driver s3 \ --s3-bucket ledger-production-backups \ --s3-region eu-west-1 \ --bucket-id ledger-production ``` From Azure Blob Storage: ```bash ledgerctl --server localhost:8888 --insecure restore download \ --driver azure \ --azure-account-name financearchive \ --azure-container ledger-backups \ --bucket-id ledger-production ``` The command starts a server-side download and reports progress until it reaches a terminal state. `--insecure` is required here because this local example starts the restore service without TLS. If the command is interrupted with Ctrl+C, Ledger cancels the server-side job and clears its staging directory. A failed download also clears staging. Retry `restore download` against the same running restore service; you do not need to restart it. ### 3. Validate the staged data ```bash ledgerctl --server localhost:8888 --insecure restore validate ``` Validation checks the staged log sequence and compares replayed volumes and metadata with the staged state. It returns a non-zero exit code when it finds an integrity error. This check does not validate your external event consumers, application database, access policies, business-level aggregates, or Ledger's local usage projection. The backup does not contain `/usage/usagedb`; Ledger reconstructs those counters and Numscript template-usage records after normal startup. ### 4. Preview the recovery point ```bash ledgerctl --server localhost:8888 --insecure restore preview ``` Record the previewed: - last applied index and timestamp; - last log sequence; - ledger names and count. Compare the last log sequence with the boundary recorded by the selected backup chain. Stop if the preview identifies a different recovery set. ### 5. Finalize and restart ```bash ledgerctl --server localhost:8888 --insecure restore finalize ``` Finalization first places `checkpoints/0`, then atomically writes `RESTORED`. The marker is the commit point. After the command succeeds, stop restore mode and start Ledger normally with a fresh WAL: ```bash ledger run \ --node-id 1 \ --cluster-id recovered-production \ --data-dir ./recovered-data \ --wal-dir ./recovered-wal \ --bootstrap \ --grpc-port 8888 ``` Ledger consumes the restore marker during the first normal startup and establishes the recovered state as the beginning of the new cluster history. ### Resume after an interruption Use the filesystem state to decide what to do; do not rerun finalization blindly: | State after interruption | Next action | | --- | --- | | Download failed or was canceled | Retry `restore download` against the running restore service. Staging was cleared. | | Validation or preview failed | Keep traffic isolated. Fix the selected recovery set, download it again when needed, then repeat validation and preview. Do not finalize a mismatched set. | | `RESTORED` exists | Finalization committed. Do not restart in restore mode; start Ledger normally with `--bootstrap` and a fresh WAL. | | No marker, no `live*`, and only `checkpoints/0` remains | Restart restore mode. Ledger recognizes and removes this single half-finalized checkpoint, then you can download and finalize again. | | A marker write failed and Ledger reports that cleanup also failed | Remove only the exact `checkpoints/0` path named in the error after confirming that `RESTORED` is absent, then restart restore mode. | | Any `live/`, `live.staging/`, `live.discard/`, or other checkpoint exists | Stop. Preserve the directory and use a new target path unless you have established where that state came from. | The offline bootstrap command applies the same freshness guards. If it completed the `RESTORED` marker, proceed to normal startup. If it left only the recoverable `checkpoints/0` state, rerunning bootstrap reclaims it; other leftovers require a fresh target path or an explicit investigation. ## Use offline bootstrap Offline bootstrap downloads, applies incrementals, previews, optionally validates, and finalizes the recovered data directory without a restore server: ```bash ledgerctl store bootstrap \ --driver s3 \ --s3-bucket ledger-production-backups \ --s3-region eu-west-1 \ --bucket-id ledger-production \ --data-dir ./recovered-data \ --validate ``` Review the preview before confirming. Use `--yes` only in automation that already verifies the selected bucket, expected sequences, target path, and command exit status. After bootstrap completes, start Ledger normally with `--bootstrap`, the recovered data directory, and a fresh WAL directory as shown above. ## Wait for usage statistics to catch up On every recovered member, Ledger creates a new local usage store and folds the audit entries available in the restored primary store. During this period, ledger statistics and Numscript template usage can be zero or incomplete even though transactions and balances are ready. Monitor these gauges until the background fold completes: ```text usage.builder.last_indexed_sequence usage.builder.audit_last_sequence usage.builder.lag ``` Require `usage.builder.lag = 0` on every serving member before accepting those statistics. There is no public full-rebuild command in Ledger 3.0. Ledger 3.0 keeps audit history permanently in the primary store, so a restored store carries the full range and zero lag means the whole of it was processed. ## Prove the recovery Do not route production traffic until you have captured evidence for each layer: ### Ledger state - `ledgerctl store check --json` completes without an integrity error; - the global log boundary matches the restore preview; - expected ledgers exist; - account and transaction counts match the recovery record; - representative balances, metadata, references, and reversions match known values; - required indexes have completed their backfill; - usage-builder lag is zero on every serving member, and any historically incomplete usage figures are documented and reconciled where required. ### Cluster operations - the cluster has a leader and the expected voting members; - a controlled write and a current read succeed through the intended endpoint; - authentication, authorization, request signing, response verification, and internal TLS behave as configured; - monitoring and alerts identify the recovered cluster correctly. ### Downstream systems - event-sink cursors are understood before delivery resumes; - consumers tolerate replay from the selected boundary; - reconciliation against the application source of truth passes. ## Rehearse disaster recovery Run the complete procedure on a schedule with production-scale data. Record: 1. backup identifiers and sequence boundaries; 2. start and completion times for download, validation, startup, and application verification; 3. the achieved recovery point and recovery time; 4. every manual intervention; 5. the final decision to accept or reject the recovered cluster. A recovery exercise fails when the cluster starts but cannot prove the expected accounting state. Keep the original cluster and recovery artifacts unchanged until the investigation is complete. --- ## Assets & currency conversion Source: https://docs.formance.com/modules/ledger/working-with/currency-conversion The ledger handles assets — the generic term for anything tracked on the ledger, including currencies, crypto, stocks (like AAPL), loyalty points, and more. Each asset has a name and an optional decimal precision. For example, `USD/2` means US dollars stored as integer cents. If you're not familiar with asset notation, read [Assets](/modules/ledger/core-concepts/assets) first. ## Funded vs non-funded The first thing to consider is whether you want the conversion to be funded or not, which is in simpler terms, simply the difference between conversion and exchange. ### Non-funded conversion When you're exchanging currency `A` for currency `B`, the latter needs to come from somewhere. If you're trading against the `@world` account, you're effectively introducing new units of `B` in an unbounded way into the ledger - which is what we refer to as a non-funded conversion. It is the simplest way to perform a conversion, albeit lacking more fine grained control over the amount of currency `B` you're introducing into the ledger. ```mermaid flowchart LR world[world] -->|GEM: 5| users[users:1234] users -->|USD/2: 100| world ``` ### Funded exchange The other way to perform a conversion is to trade against a funding account that you control, and you can ensure that the amount of currency `B` you're trading is limited by the amount of currency `B` you previously introduced in the ledger in a separate process, which is what we refer to as a funded conversion. ```mermaid flowchart LR reserves[reserves:liquidity] -->|GEM: 5| users[users:1234] users -->|USD/2: 100| reserves ``` An example case where a funded conversion is preferable could be a BTC buying platform. In this example, users acquire BTC from you and you will enable them to hold both BTC and USD in their accounts, letting them exchange their USD for BTC. In this case, the total amount of BTC held by your users should be limited by the amount of BTC you actually own in your reserves. By using a funded conversion and sourcing BTC granted to users from a previously provisioned account in the ledger, you can ensure that you are not accidentally creating BTC out of thin air. ## Single-phase vs multi-stages The other thing to consider is whether you want to perform the asset conversion in a single step, or in multiple stages. In the case of a single step conversion, you will be performing the conversion in a single atomic transaction posted to the ledger. This is the simplest way to perform a conversion, but it is not always possible and assumes that the desired asset is already available for exchange. A typical good case for multi-stage conversion, is a conversion that involves the intervention of different parties or a delay in the conversion process. For example, if you enable your users to trade some USD for AAPL, and you are using a broker behind the scenes for the acquiring of AAPL. In this scenario, the best practice is to use a swap account as described below: ```mermaid flowchart LR users[users:1234] -->|1 · USD/2: 100| swaps[swaps:XYZ1234] swaps -->|2 · USD/2: 100| brokering[brokering:main] brokering -->|2 · AAPL/6: 50| swaps swaps -->|3 · AAPL/6: 50| users ``` In this example, each edge is labeled with the step it belongs to: 1. The first step represents the user placing some USD in the swap account. 2. The second step represents a step where the logic of acquiring the AAPL would be happening in your backend, with them eventually being sent to the swap account in exchange for its whole balance of USD. 3. Finally, the third step is the user exchanging the AAPL acquired by the broker for AAPL in their account. Each of these steps can be performed in a separate ledger transaction, as the swap account is being used as a temporary holding location for the exchange process. ## Additional best practices ### Exchange rate For auditability purposes, it is recommended that you store the exchange rate used for the conversion in the transaction metadata. This will enable you to audit the conversion process and ensure that the conversion rate used is the one you intended to use. --- ## PSP Connectors Source: https://docs.formance.com/modules/payments/connectors/psp PSP connectors integrate Formance Payments with traditional payment service providers — banks, card processors, alternative payment methods, money-movement networks. They surface accounts, balances, and payments from the upstream provider and, where the provider supports it, initiate outbound transfers and payouts on your behalf. This is the broadest connector category in Payments. Most ship in the Community edition; a few specialized integrations require an Enterprise license. ## Available PSP connectors **Community edition** — Adyen, Atlar, Bankingcircle, Column, Currencycloud, Increase, Mangopay, Modulr, Moneycorp, Qonto, Stripe, Wise. **Enterprise edition** — Banking Bridge, Routable. Capability coverage varies per connector — some are read-only (accounts + balances + payments), others support write flows like transfer or payout initiation. The [Capabilities](/modules/payments/capabilities) page is the per-connector inventory; check it before picking an integration to make sure the provider exposes what you need. ## What's different about PSP connectors PSP connectors model the world of traditional payments — single-asset balance movements on a single account leg. The Payments module's `Payment` resource carries a `scheme` (SEPA, ACH, wire, card-network) where applicable, and optional initiation flows let your stack create payouts or transfers programmatically rather than only observing them. For asset-class flows — multi-currency swaps, trading orders, stablecoin redemption — see [Exchange Connectors](/modules/payments/connectors/exchange) instead. For aggregator-driven access to end-user bank accounts, see [Open Banking](/modules/payments/connectors/open-banking). --- ## Operations Source: https://docs.formance.com/modules/payments/operations Operational guidance for running installed connectors — how often they poll, how to find their schedules, how to monitor health, and how to recover when something drifts. For what each connector actually does, see [Capabilities](/modules/payments/capabilities). ## Polling cadence When installing a connector — on Formance Console or via the API — you can configure a `pollingPeriod` that defaults to **30 minutes** if not specified. It drives every periodic capability the connector exposes — Accounts, Balances, External Accounts, Payments, Orders, Conversions, plus the poll loop on in-flight Transfer / Payout statuses. Each connector declares a **minimum** polling period the platform enforces at install time. The common floor is 20 minutes (Stripe, Bankingcircle, Routable, Bitstamp, Mangopay, …); a few connectors with slower-moving data or stricter upstream rate limits raise the floor higher. Submitting a `pollingPeriod` below the floor is rejected with a clear validation error before the connector is installed — see the per-connector page for the exact value (look for the `pollingPeriod` row in the [Configuration fields](/modules/payments/connectors/psp/stripe#configuration-fields) table). ## Monitoring connector schedules To check when a connector last pulled data, use the Connector Schedule APIs. ### Finding the schedule ID Each connector has schedules that are automatically generated when the connector is installed. The schedule ID follows this format: ``` {stackID}-{connectorID}-{capability} ``` For example: `stack123-abc-FETCH_ACCOUNTS` To find the schedule IDs for your connectors, use the [List All Connector Schedules](/stack-api-reference/paymentsv3/list-all-connector-schedules) endpoint. ### Checking last pull time Once you have the schedule ID, call the [List Connector Schedule Instances](/stack-api-reference/paymentsv3/list-all-connector-schedule-instances) endpoint with the schedule ID to see the history of data pulls, including the most recent one. This is useful for: - Debugging synchronization issues - Monitoring connector health - Verifying data freshness --- ## Verify Ledger Integrity Source: https://docs.formance.com/modules/ledger/operations/integrity Run an integrity check after a restore, before and after high-risk maintenance, and whenever accounting state differs from an external source of truth. The checker reports inconsistencies; it does not repair them. ## Run a primary-store check Use JSON output when you need a durable result for an incident or recovery record: ```bash ledgerctl store check --json > ledger-integrity-check.json jq -e '.valid == true and .errorCount == 0' ledger-integrity-check.json ``` The command takes a consistent point-in-time view of Ledger's primary data store, verifies the available audit chain, and compares selected stored values with state reconstructed from that chain. With structured output, `ledgerctl` can exit successfully even when the result contains integrity errors; the `jq` assertion is therefore the gate for automation. Capture the following alongside the output: - the Ledger release and node you checked; - the cluster and environment identifiers; - the check start and completion times; - the latest expected log and audit sequences; - the backup or maintenance event that prompted the check. Run the check against a stable, healthy cluster. For an incident investigation, preserve the affected data and logs before restarting nodes or changing storage. ## Understand what the checker verifies The following table shows representative coverage in the current preview. It is a guide to the current checks, not a claim that every file and service in a deployment is covered. | Sample state | What the checker compares | | --- | --- | | Audit boundaries | Audit hashes and sequence continuity. | | Transactions and balances | Transaction state, posting-derived input and output volumes, reversion state and bitsets, and transaction references. | | Account metadata | Typed account metadata values and deletions reconstructed from the log. Metadata on accounts deliberately excluded by transient or purged-ephemeral processing is not compared. | | Ledger definition | The live ledger set, metadata schema, account types, and operational boundaries such as the next transaction and log IDs. | | Retry and indexing projections | Frozen idempotency outcomes, allowed skipped-order results, the index registry's presence and identity, and transient or purged-volume exclusion records. | Mirror acceptance has a separate set of migration proofs. Follow [Monitor Mirror Progress](/modules/ledger/migration/monitor-mirror) instead of treating this primary-store check as a migration approval. ## Know the limits The primary-store checker has these explicit implementation boundaries: - it opens the peer read-side index store for exactly one pass, the reverse-map orphan scan, and skips that pass with an informational log when no peer store is attached to the checker. That is the case for `restore validate` and `store bootstrap --validate`, which validate a staged store with no peer index beside it. The pass also narrows to key decoding when the peer index has not yet folded the whole verified log range, so a check run against a lagging replica proves less on this axis than one run against a caught-up replica; - it does not re-derive read-side index contents. No index type has content verification today, and per-replica build state is outside its scope: the live and pending forward-encoding versions and the backfill cursor that `ledgerctl indexes list` reports come from each replica's own index store, not from the verified registry. Registry entries are compared for presence and identity — the index ID and the ledger it belongs to — only; - it does not compare ledger-level metadata or the default account-type enforcement mode (`LedgerInfo.DefaultEnforcementMode`); restore rebuilds those values from retained verified state; - it does not compare stored prepared-query definitions. These definitions are authoritative business-shaping state because they determine which records named queries return. Export them and compare them with reviewed [configuration as code](/modules/ledger/operations/configuration-as-code) separately; - it does not open the local usage store at `/usage/usagedb`. None of the derived ledger statistics or Numscript template-usage records are checked there; - it does not verify bloom filters, snapshot files, or the transient spool. These exclusions are not a statement that the data cannot matter. For example, a corrupted read-side index can affect query results even when the primary-store check passes; verify index readiness and rebuild health separately. The checker also does not prove that: - an application sent the intended business transaction; - an external database, payment provider, or event consumer matches Ledger; - permissions, certificates, and signing keys follow your security policy; - every query index is ready or returns the result you expect; - a backup can meet your recovery-time objective. Combine the integrity result with business reconciliation, index readiness, event-sink status, and a restore exercise. The `restore validate` command applies the same class of primary-store checks to staged restore data before finalization. It validates the staged Ledger state, not the rest of your recovered application or the usage projection that will be reconstructed after startup. If usage figures look wrong while this check passes, inspect `usage.builder.last_indexed_sequence`, `usage.builder.audit_last_sequence`, and `usage.builder.lag` on the affected member. Audit history is permanent in the primary store, so a member holding the full range and reporting zero lag has processed all of it — a wrong figure there points at the projection or at the recovered store, not at missing history. The current preview has no public full-rebuild command, so reconcile critical figures against your application records without deleting the local store during the investigation. ## Respond to a finding Do not delete, compact, rebuild, or overwrite the affected data as an initial response. Those actions can remove evidence and make the recovery boundary harder to establish. When a check reports an error: 1. Stop routing new writes to the affected cluster. 2. Record the complete checker output, Ledger version, node, and sequence boundaries. 3. Preserve node logs, storage snapshots, and backup manifests. 4. Run the same read-only check on another healthy replica when available. 5. Determine whether the problem affects one replica, the shared committed history, or an external comparison. 6. Choose recovery from a previously verified backup only after identifying the last trusted boundary. 7. Reconcile the recovered state with application-level records before reopening traffic. Ledger does not auto-fix integrity errors. Treat a confirmed mismatch as a correctness incident. ## Use Sentinel mode in test environments Sentinel mode adds runtime assertions around volume updates and stored state during every applied write. It checks: - that cumulative volumes do not decrease; - that posting deltas match volume updates; - that aggregated input and output volumes remain balanced; - that values committed to disk match the values Ledger expected to write. Enable it on a test or staging cluster when you are investigating correctness or running fault-injection tests. Add the boolean `--sentinel-mode` server flag to that cluster's complete `ledger run` invocation. Keep its required identity, storage, network, bootstrap or join, TLS, and authentication flags unchanged. Sentinel mode adds work to the write path, including expensive scans for some checks. It is not a replacement for scheduled primary-store checks, backup validation, or business reconciliation. Keep Sentinel mode disabled for normal production traffic unless Formance support asks you to enable it for a bounded investigation. Measure its effect with production-like data before using it on a live workload. ## Keep verification evidence For every scheduled check or recovery exercise, retain: - the command output and exit status; - the verified log and audit boundaries; - representative business reconciliation results; - the backup identifiers used for recovery testing; - the decision and approver for reopening writes. This record establishes what you verified, with which artifacts, and at which point in the cluster history. --- ## Bi-temporality Source: https://docs.formance.com/modules/ledger/working-with/bi-temporality ## Definition The ledger supports bi-temporality, which means that each transaction is associated with two timestamps: - **Request time**: The time at which the transaction was submitted to the ledger. It is usually the machine clock time. - **Transaction time**: The time at which the transaction is considered to have occurred. In the API, the two timestamps map to these fields on a transaction: | Concept | API field | |------------------|--------------| | Request time | `insertedAt` | | Transaction time | `timestamp` | The ledger records both timestamps for each transaction, and the transaction time is used to determine the state of the ledger at a given point in time. Let's consider the following example to illustrate the difference between the two timestamps. ![Bi-temporality](/assets/images/ledger/bi-temporality/example.png) In this example, 3 transactions are submitted to the ledger at different times: - `TX1` is submitted on monday, but the transaction time is set to tuesday. The transaction is considered to have occurred on tuesday. We say that the transaction is **postdated**. - `TX2` is submitted on tuesday, and the transaction time is set to monday. The transaction is considered to have occurred on monday. We say that the transaction is **backdated**. - `TX3` is submitted on tuesday, and the transaction time is set to tuesday. The transaction is considered to have occurred on tuesday. This is **the default behavior**. Bi-temporality is useful in the following scenarios: - **Time travel**: The ability to query the ledger as it was at a specific point in time. - **Correction of errors**: The ability to correct errors in the ledger by backdating or postdating transactions. - **Auditing**: The ability to audit the ledger at a specific point in time. - **Data import**: The ability to import data from external systems that use different timestamps. ### Present time relativity Because transactions can either be backdated or postdated, the concept of "present time" may not match the current machine clock time. From the point of view of the ledger, the present time is the **transaction timestamp** the most in the future. In the example above, let's consider that we only inserted `TX1`. In that case, the **machine clock is on monday**, but **the present time is tuesday** because the transaction time of `TX1` is tuesday and it is the most recent transaction. ![Present time](/assets/images/ledger/bi-temporality/present-time.png) Likewise, if we insert only backdated transactions, the present time will be in the past compared to the machine clock. ## Implications of bi-temporality Bi-temporality allows users to insert transaction at any point in time in the past or future. When a user performs a query on the ledger, the ledger state is determined based on the requested point in time. The ledger state is a snapshot of the ledger at the requested point in time, and it includes all transactions that have a transaction time equal to or less than the requested point in time. This capability has some implications discussed below. From now on, we will only consider the **transaction time** when discussing the ledger state at a specific point in time. ### Account and Transaction Metadata Account and transaction metadata are not fixed in time. When a user queries the ledger at a specific point in time, the metadata associated with accounts and transactions is also determined based on the requested point in time. This means that the metadata associated with an account or transaction can change over time. #### Example: Fraud management Let's consider the following example. Suppose that a fraud engine is integrated with the ledger to flag suspicious account. It does so by adding `risk=high` to the account metadata. The fraud team regularly exports the suspicious accounts to an external system for further investigation. The fraud team queries the ledger at specific points in time to get the list of suspicious accounts. Let's consider the account `customer:123456`. At a time `t1` the fraud engine marks the account as suspicious by adding `risk=high` to the account metadata. The fraud team exports the list of suspicious accounts at time `t2` and `t3`. The account will be included in the list of suspicious accounts in both exports. ![Initial state](/assets/images/ledger/bi-temporality/01-initial-state.png) Now, let's consider that the fraud engine removes the `risk=high` metadata from the account at time `t4` located between the two exports `t2` and `t3`. It does so by removing the `risk` metadata using a backdated transaction. In this case, the account will not be included in the list of suspicious accounts in the export at time `t3`. ![Final state](/assets/images/ledger/bi-temporality/02-final-state.png) ### Backdated transaction validation #### Problem Statement When a user inserts a transaction with a transaction time in the past, there is a risk that the transaction is invalid because it might yield an invalid state of the ledger. Consider the following example account whose balance evolution is as follows: | Time | Transaction amount | New Balance | |------|-------------------|-------------| | 1 | 100 | 100 | | 2 | -50 | 50 | | 3 | -10 | 40 | | 4 | 50 | 90 | | 5 | -10 | 80 | Now, consider that a user wants to insert a backdated transaction with a transaction time of 2 and an amount of `-100`. This transaction would yield the following balance evolution: | Time | Transaction amount | New Balance | |------|-------------------|-------------| | 1 | 100 | 100 | | **2**| **-100** | **0** | | 3 | -50 | -50 | | 4 | -10 | -60 | | 5 | 50 | -10 | | 6 | -10 | **-20** | The new balance at time 6 is `-20`, which is invalid because the account has a negative balance. This is an example of an invalid backdated transaction. Now, consider that a user wants to insert a backdated transaction with a transaction time of 2 and an amount of `-50`, rather than `-100`. This transaction would yield the following balance evolution: | Time | Transaction amount | New Balance | |------|-------------------|-------------| | 1 | 100 | 100 | | **2**| **-50** | **50** | | 3 | -50 | 0 | | 4 | -10 | -10 | | 5 | 50 | 40 | | 6 | -10 | **30** | The new balance at time 6 is `30`, which is valid because the account has a positive balance. This is an example of a valid backdated transaction. Note that the intermediate balances might be negative, as it is the case at time 4 in the second example. The ledger does not validate the intermediate states of the ledger, only the final state. #### Validation Mechanism The ledger does not validate backdated transactions the same way it validates usual transactions. To check that a backdated transaction is valid, the ledger computes the new current state of the ledger by applying the backdated transaction and all the transactions that occurred after the backdated transaction. If the new state is valid, the backdated transaction is accepted. Otherwise, the backdated transaction is rejected. A backdated transaction is considered valid if it doesn't put any account in a negative balance in the new computed final state of the ledger. Keep in mind that the ledger does not validate the intermediate states of the ledger, only the final state. The only exception to this rule is the accounts that have been allowed to overdraft within the transaction. These accounts can have a negative balance in the new computed final state of the ledger. ## Setting transaction timestamps When creating transactions, you can specify a custom timestamp to backdate or postdate the transaction. Include the `timestamp` field in the request body (RFC3339 format): {"fctl ledger transactions num script.ns --ledger my-ledger --timestamp \"2024-09-07T00:00:00.000Z\""} Currently, it's not possible to set the transaction timestamp directly within a Numscript. While you can set transaction metadata using `set_tx_meta(key, value)`, there's no equivalent function for setting the timestamp in the script itself. To undo a committed transaction entirely, see [Reverting Transactions](/modules/ledger/working-with/reverting-transactions). ## Effective Volumes In Ledger v2, the concept of **effective volumes** was introduced to handle backdated transactions correctly. ### What are effective volumes? - **Regular volumes**: Represent volumes as they are at the current time - **Effective volumes**: Represent volumes as they were at the date a transaction was inserted, not the date it was created While regular volumes cannot change over time, effective volumes can change when looking into the past. If you insert a backdated transaction, the effective volumes of all subsequent transactions are recalculated. Effective volumes are the only data that escapes the control of historization. Even by freezing the ledger at a date T, effective volumes can move because you can insert transactions before that date. ### Enabling effective volumes Effective volumes are enabled when the following features are configured: | Feature | Value | Description | |---------|-------|-------------| | `MOVES_HISTORY` | `ON` | Historize funds movements by account | | `MOVES_HISTORY_POST_COMMIT_EFFECTIVE_VOLUMES` | `SYNC` | Compute and maintain post-commit effective volumes | ### Performance considerations While `MOVES_HISTORY_POST_COMMIT_EFFECTIVE_VOLUMES: SYNC` guarantees data accuracy, it introduces performance overhead. When set to `DISABLED`, you lose **pre/post commit effectiveVolumes** but speed up backdated transaction updates. The effectiveVolumes field is still updated. When inserting a backdated transaction with `MOVES_HISTORY_POST_COMMIT_EFFECTIVE_VOLUMES` enabled, the ledger performs these additional steps: 1. Compute **postCommitEffectiveVolumes** for the moves by searching for previous moves for the account/asset pair 2. Insert the new move 3. Update **postCommitEffectiveVolumes** for all future moves The more transactions are affected by a backdated insertion, the more updates need to be processed. ## Volume Query Consistency Formance Ledger provides **immediate consistency** for all write operations. The system relies on PostgreSQL's ACID properties — you should never need `time.Sleep` or similar delays. The ledger stores dates with **microsecond precision** (PostgreSQL's maximum precision). To avoid potential flakiness in tests, consider rounding your dates to microsecond precision. ### PIT timing issue (Ledger 2.0–2.1) In versions before 2.2, when no explicit end time (PIT) is provided in volume queries, the system auto-calculates one from the current wall clock. This causes flaky results when writing transactions with future timestamps and querying immediately. **Fix:** provide an explicit `endTime` in your volume queries: ```go endTime := startTime.Add(time.Minute) volumeReq := operations.V2GetVolumesWithBalancesRequest{ Ledger: ledgerName, RequestBody: filter, StartTime: &startTime, EndTime: &endTime, } ``` Or use past timestamps instead of future ones in your tests. Upgrade to Ledger 2.2+ to eliminate this issue entirely — the auto-calculated PIT was removed. --- ## Upgrade Ledger Source: https://docs.formance.com/modules/ledger/operations/upgrade Start with the procedure published for the exact source and target releases. A one-member-at-a-time rollout is valid only when that procedure explicitly supports mixed binaries and the persisted state remains compatible. Treat the Stack release, Ledger server, Stack Operator, and `ledgerctl` as separate versioned artifacts. The current Ledger 3.0 preview contains a persisted-state break for which its source requires a full wipe and replay from a trusted source; mixed-binary rolling upgrades are unsupported. The final Stack v4 source/target matrix and release procedure are blockers for this runbook. Do not approve a production upgrade from this page alone, and do not infer server compatibility from the `ledgerctl upgrade` command. ## Expect usage statistics to rebuild Ledger v3 moves the counters returned by ledger statistics, together with Numscript template usage, into a local derived store on every member. The first v3 start resets the migrated counters and rebuilds them from the audit entries still available to that member. This transition does not change transactions, balances, or account metadata. It can, however, make posting, revert, Numscript execution, reference, ephemeral-account, transient-account, volume, and template-usage figures start at zero and rise while the background rebuild runs. Watch these three gauges on every member: | Metric | What it tells you | | --- | --- | | `usage.builder.last_indexed_sequence` | Highest audit sequence already reflected in usage statistics. | | `usage.builder.audit_last_sequence` | Highest audit sequence currently available to the builder. | | `usage.builder.lag` | Number of available audit sequences still to process. | Wait for `usage.builder.lag` to reach zero before accepting the rebuilt statistics. Do not use the figures for billing, limits, or an upgrade comparison while they are still moving. Zero lag proves that the member processed the audit history it can still reach. It does not recover counter values from audit entries that were already archived or removed before the upgrade. In that case, the rebuilt values can remain below their historical totals. Ledger 3.0 does not provide a public full-rebuild command that can recover those missing values; record the limitation and reconcile critical figures against your own source of truth. ## Identify the four versions Record these values before planning the change: | Artifact | Source of truth | Why it matters | | --- | --- | --- | | Stack release | The selected Stack version and its `Versions` resource | Resolves the component set intended to run together. | | Formance Stack Operator | Installed controller and CRD versions | Determines which desired state and status fields are supported. | | Ledger server | Image digest and Ledger version resolved by Stack | Determines API behavior and persisted-format compatibility. | | `ledgerctl` | `ledgerctl version` or the packaged artifact | Determines available administrative commands; it does not change the server. | Pin image digests and released versions rather than mutable tags. The change record must contain both the source and target values. ## Precheck the upgrade Do not start until every item has evidence: 1. The release matrix explicitly supports the source-to-target Stack, Operator, Ledger, and CLI path. 2. Release notes identify API, configuration, security, storage, and operational changes. 3. The previous server version can read state written by the target version if binary rollback is part of the plan. 4. The current cluster has a leader, the expected voters, and no member catching up. 5. The cluster can lose one voter and remain within its service objective during the rollout. 6. WAL, data, memory, and node-pool headroom cover restart and catch-up. 7. Required authentication, TLS trust, signing keys, and client versions support the mixed-version window. 8. A recent backup has restored successfully into an isolated cluster using the documented release path. 9. Dashboards, paging alerts, and the maintenance owner are active. 10. Clients send idempotency keys and honor retry guidance for transient `Unavailable` responses. ```bash ledgerctl cluster status ledgerctl cluster disk-usage --json ``` Repeat the disk check against every member endpoint. Capture a controlled write and current read as the pre-upgrade baseline. ## Choose the upgrade class The published procedure for your source and target releases must name one of three upgrade classes. If it names none, stop and obtain a release decision rather than inferring the class from a successful single-node restart. | Class | When it applies | What it requires | | --- | --- | --- | | Rolling | The target revision explicitly supports mixed binaries on the existing storage schema, and the cluster can lose one voter while traffic continues. | Restart members one at a time under load, as in *Roll the cluster only when the release permits it* below. | | Coordinated restart | The persisted state is compatible, but two binaries can apply the same committed entry differently. | Stop every member behind a maintenance barrier before any target binary starts, as in *Restart the cluster in one coordinated window* below. | | Rebuild or restore | The persisted or backup format is incompatible. | Follow the target revision's own reset, replay, or restore instructions. This is the current Ledger 3.0 preview case. | Also use [maintenance mode](/modules/ledger/operations/maintenance-mode) within a rolling upgrade when the release notes require a stable write boundary, when a configuration or storage transition cannot accept concurrent mutations, or when the rollout owner cannot safely validate mixed members under load. Maintenance mode does not make an unsupported downgrade safe. ## Roll the cluster only when the release permits it This section applies only when the released compatibility matrix permits a mixed-version rolling change. For the coordinated-restart class, use the next section instead. For rebuild or restore, follow the target revision's own procedure. Change only the supported Stack version declaration. The Stack Operator should resolve the target Ledger image and coordinate the member sequence; do not edit the Ledger workload or membership manually underneath it. The rollout must follow this observable sequence: 1. Update one follower. 2. Wait for the process to become ready. 3. Wait for it to rejoin the same cluster and catch up to the committed position. 4. Confirm that it remains a voter and can serve the intended read path. 5. Send a controlled idempotent write and current read through the service. 6. Confirm that error rate, latency, queue, WAL, disk, and memory signals remain within the rollback thresholds. 7. Continue with the next follower. 8. Transfer leadership to an updated, caught-up voter when the workflow requires it. 9. Update the former leader last. Watch Ledger membership independently from Kubernetes reconciliation: ```bash ledgerctl cluster watch --interval 2s ``` Pause when a member cannot catch up, the cluster loses a leader, a second voter becomes unavailable, the resolved image differs from the target, or a safety threshold is crossed. A completed Kubernetes rollout is not proof that Ledger membership or state converged. ## Restart the cluster in one coordinated window Use this section for the coordinated-restart class: the persisted state is compatible, but two binaries must never apply the same committed entry. Every member stops before any target binary starts, so there is no mixed-version window to validate. Enable maintenance mode first, and confirm that the cluster state reports it and that writes are rejected: ```bash ledgerctl cluster maintenance enable ledgerctl cluster status --json ``` Read `raftStatus.commit` from the leader once maintenance mode is visible in that output. That index is the barrier; call it `C`. Then query every voter and require a normal `syncProgress.status` and a `raftStatus.lastPersistedIndex` at or above `C` before stopping it: ```bash ledgerctl cluster status --node-id --json ``` `lastPersistedIndex` is the storage-side cursor: it advances only after the member's write batch has committed to disk, so it, rather than `applied`, is what proves the member durably applied every entry through the barrier under the old binary. Then complete the window in order: 1. Stop or finish any cluster-internal job that can still propose work, and repeat the per-voter barrier check if the leader's commit index advanced. 2. Stop every member. Do not start a target binary while any member still runs the old one. 3. Replace every server binary or image. 4. Start enough voters to form quorum, then start the remaining members. 5. Verify cluster health and run the documented integrity check. 6. Disable maintenance mode last, then resume traffic in controlled stages. ## Verify the upgraded cluster After every member reports the target version: 1. Run `ledgerctl cluster status` and confirm the expected leader, voters, and applied positions. 2. Confirm that the Stack status and actual image digests match the target release. 3. Send a controlled write with a new idempotency key and read it back with current consistency. 4. Exercise the write through each externally reachable endpoint to detect follower-forwarding or trust failures. 5. Verify required indexes, prepared queries, event sinks, backup schedules, and `usage.builder.lag` on every member. 6. Test allowed and denied authorization paths and any required request or response signatures. 7. Run the documented integrity check at the scope and time approved for the cluster. 8. Compare request latency, error rate, queue, WAL, compaction, disk, and memory against the baseline. 9. Keep the change window open for at least one representative compaction and background-work cycle. If maintenance mode was enabled, complete these checks before reopening, then resume traffic in controlled stages. ## Roll back safely Stop the rollout first. Keep the healthy existing voters running and preserve every volume and node identity. Reapply the previous Stack release only when the published compatibility matrix confirms that: - the previous Operator understands the current CRDs or the downgrade sequence handles CRDs explicitly; - the previous Ledger binary can read all persisted state written by the target version; - no irreversible configuration or data transition has completed; - authentication and TLS still support the mixed window. Roll members back one at a time with the same quorum and catch-up checks. If any condition is not confirmed, do not attempt a binary downgrade. Keep writes stopped when required, preserve evidence, and follow the target release's forward-recovery or restore procedure. A backup is a disaster-recovery input, not proof that a runtime downgrade is safe. Restoring it replaces cluster state and requires the separate restore runbook and business approval. ## Update `ledgerctl` separately `ledgerctl upgrade` replaces the local CLI binary. It does not upgrade Ledger servers or the Stack. Check the latest stable CLI release without installing it: ```bash ledgerctl upgrade --channel stable --dry-run ``` Install the latest stable CLI only when that version is approved by the Stack compatibility matrix: ```bash ledgerctl upgrade --channel stable ``` The command verifies the downloaded archive checksum before replacing the binary. Its default channel is `nightly`, so always name `stable` in a production procedure. When the change requires an exact CLI version rather than the latest stable one, install the pinned release artifact through your normal package or image workflow instead. After changing the CLI, verify its version, connection profile, TLS trust, authentication, request signing, and response verification before using an administrative mutation. --- ## Reverting Transactions Source: https://docs.formance.com/modules/ledger/working-with/reverting-transactions A revert creates a compensatory transaction with the exact opposite postings, cancelling the effect of the original. The original transaction remains in the immutable log; reverts are corrections, not deletions. ## When to use it Use a revert when a transaction was committed by mistake, for example a duplicate payment, a wrong amount, or a processing error. Do not use reverts routinely as a workflow mechanism (e.g. to model a "pending → cancelled" state); model that with your account structure instead. ## How to revert To revert a transaction, use the revert endpoint: The ledger creates a compensatory transaction with opposite postings to cancel out the original transaction's effects. ## The atEffectiveDate parameter The `atEffectiveDate` parameter controls the transaction time of the compensatory transaction: Without `atEffectiveDate=true`, the compensatory transaction is created at the current time, which produces incorrect balances in historical reports and financial statements. For example, consider an account with transactions 1, 2, and 3. If you revert transaction 2 without `atEffectiveDate`, the compensatory transaction is created at the current time (after transaction 3). When generating reports that filter out reverted transactions, transaction 3 will show an incorrect balance of -9250 instead of -9750 because the revert isn't accounted for at the right point in time. With `atEffectiveDate=true`, the compensatory transaction is created with the same transaction time as the original, maintaining accurate historical balances: | TxId | Amount In | Amount Out | Balance | |------|-----------|------------|----------| | 1 | 0 | 10000 | -10000 | | 2 | 500 | 0 | -9500 | | 4 | 0 | 500 | -10000 | | 3 | 250 | 0 | -9750 | Always use `atEffectiveDate=true` when reverting transactions to maintain accurate historical balances. ## Force reverting By default, the system prevents reverting transactions that would result in insufficient funds. However, you can override this validation using the `force` parameter. To revert a transaction involving an account with a negative balance: Use the `force` parameter with caution: - It allows creating negative balances in accounts - Ensure proper controls and auditing are in place when using forced reversals - This feature is particularly useful for accounts that act as "always negative" balance accounts (like liability accounts) ## Querying without reverts When querying transactions, you may want to exclude both reverted transactions and their compensatory transactions. Use this filter: ```json { "$and": [ { "$match": { "account": "deals:XYZ:balances:" } }, { "$match": { "reverted": false } }, { "$not": { "$exists": { "metadata": "com.formance.spec/state/reverts" } } } ] } ``` This query: - Matches transactions for a specific account - Excludes transactions marked as reverted - Excludes compensatory transactions (identified by the `com.formance.spec/state/reverts` metadata) For performance, consider using the `/volumes` endpoint instead of `/transactions` when you only need balance information. The volumes endpoint supports `startTime`, `endTime`, `insertionDate`, and `groupBy` query parameters; pass `insertionDate=true` for bi-temporal point-in-time queries. --- ## Troubleshoot Ledger Source: https://docs.formance.com/modules/ledger/operations/troubleshooting Start with the response status and structured error reason, then correlate it with Ledger membership, storage, and telemetry. Several different protections return `503`; they do not all have the same retry or recovery action. ## Preserve evidence first Before restarting, scaling, compacting, removing a member, or changing configuration, record: - the start time, affected operation, ledger, endpoint, and request or trace identifier; - the Stack, Operator, Ledger server, and client versions; - the HTTP status or gRPC code, the structured reason, and the `Retry-After` value when the response is a `503`; - whether reads, writes, or both are affected; - the current leader, [voters and learners](/modules/ledger/operations/cluster-lifecycle), committed position, and applied positions; - recent rollout, maintenance, index, backup, restore, or event-sink activity; - member restarts, volume events, and time-synchronization state. Run non-destructive checks first: ```bash ledgerctl cluster status ledgerctl ledgers list --json ledgerctl cluster disk-usage --json ``` `cluster disk-usage` inspects the member reached by the active profile. Repeat it against each member endpoint when the affected node is not known. Do not use `ledgerctl cluster remove-node --force`, do not restart a node with the server's `--unsafe-skip-config-validation`, and do not hand-edit persistent data or delete a persistent volume as a diagnostic step. These actions change the evidence and can make recovery harder. The storage-schema-version check cannot be bypassed, not even with `--unsafe-skip-config-validation`, so that flag will not clear a schema failure. ## Route the symptom | Symptom | Confirm | Likely condition | Next action | | --- | --- | --- | --- | | HTTP `503` with `NO_LEADER` and `Retry-After: 1` | `ledgerctl cluster status` shows no leader | Election in progress, missing voter majority, or broken peer connectivity | Clients may retry after the advertised delay for a brief election. Restore quorum or peer connectivity when it persists. | | Pods are ready, but writes return `429` and `WRITES_BLOCKED_DISK_FULL` | Disk usage on each member; WAL/data high-water settings | At least one protected volume crossed its block mark | Stop non-essential writes, expand the affected volume through the supported platform path, and wait until use falls below the resume mark. | | Pods are ready, but writes return `503` and `WRITES_BLOCKED_CLOCK_SKEW` | Node clock offset and time-service health | A peer exceeds the configured clock-skew threshold (`--health-clock-skew-threshold`, 500 ms by default) | Restore time synchronization across members. Do not disable the gate before identifying the clock source. | | Writes return `503` and `MAINTENANCE_MODE` | Cluster status reports maintenance enabled | A planned or failed maintenance window still owns the write boundary | Contact the maintenance owner and follow [Use Maintenance Mode](/modules/ledger/operations/maintenance-mode). Do not retry until the boundary is intentionally removed. | | A follower endpoint fails writes while direct leader requests work | Request path through every exposed endpoint; internal TLS and cluster trust | The follower cannot forward authorized requests to the leader | Follow [Secure Cluster Traffic with TLS](/modules/ledger/operations/tls-and-network-security) and verify certificates, names, trust, and cluster identity on every member. | | `401` from all endpoints | Token signature, issuer, audience, expiry, and active profile | Authentication failed | Refresh or correct the credential using [Authenticate and Authorize Clients](/modules/ledger/operations/authentication). An invalid token must not fall back to anonymous access. | | `403` for one operation | Caller scopes and ledger target | The authenticated caller lacks permission | Compare the required scope with the assigned role; do not broaden access until the denied action is understood. | | TLS handshake or certificate-name failure | CA chain, certificate validity, SAN, endpoint name, and client profile | Client or inter-node trust does not match the endpoint | Correct the trust or name through the TLS runbook; do not use insecure transport as the production fix. | | Writes are slow and queue-full counters increase | Apply, WAL, peer latency, Pebble stall, CPU and disk IOPS | Consensus or storage cannot sustain the incoming rate | Reduce load, preserve quorum, then follow [Plan Capacity and Tune Performance](/modules/ledger/operations/capacity-and-performance). | | Member remains a learner or falls behind | Cluster status from leader and affected node; restart, disk, network, memory | Catch-up cannot complete or the member keeps restarting | Stop further lifecycle changes and follow [Plan Topology and Manage Cluster Lifecycle](/modules/ledger/operations/cluster-lifecycle). | | Query reports `INDEX_BUILDING` or required index is not ready | Index status on every serving member | Backfill has not completed | Keep dependent traffic disabled and follow [Create and Manage Indexes](/modules/ledger/querying/indexes). | | Current read reports `READ_INDEX_NOT_CAUGHT_UP` | Requested minimum sequence or checkpoint versus member state | The local read index has not reached the requested position | Retry on an appropriate member after catch-up or choose the documented consistency path; see [Choose Read Consistency and Use Checkpoints](/modules/ledger/querying/consistency-and-checkpoints). | | Filter returns an index error or unexpectedly empty result | Target, field, [typed schema](/modules/ledger/modeling/metadata), filter, index, cursor and checkpoint | Query contract or view does not match the request | Validate the filter and index rather than falling back to an unbounded scan. | | Transactions commit but downstream events stop | Sink cursor, last error, destination and consumer deduplication | Asynchronous delivery is retrying independently of Ledger commits | Follow [Consume, Monitor, and Recover Events](/modules/ledger/events/consume-and-recover); do not replay without a deduplication plan. | | Backup or restore stalls or fails validation | Job status, manifest, segment chain, object access, target cleanliness | Artifact, credential, concurrency, compatibility, or target-state problem | Keep the source cluster unchanged and follow the backup or restore runbook. Do not finalize an invalid restore. | ## Diagnose availability failures ### No leader A brief election can return `503`, `NO_LEADER`, and `Retry-After: 1`. Any request can hit it, including a linearizable read. Retry after the advertised delay: a mutation must be reissued byte-identical under its original idempotency key, while a read carries no key and can simply be repeated. Reusing a key with different content is rejected, not silently accepted — see the `IDEMPOTENCY_KEY_CONFLICT` row in [Retry and rollback rules](#retry-and-rollback-rules). When the condition lasts beyond the normal election window: 1. identify which voters are reachable; 2. confirm that a majority of the configured voter set is running; 3. compare peer connectivity and internal TLS from each voter; 4. inspect disk, memory, and restart events on missing members; 5. stop concurrent disruption or rollout; 6. recover existing members before considering membership changes. Do not bootstrap a replacement cluster or force-remove a member while the previous voter set can recover. Escalate a quorum-loss decision with the current membership and volume state. ### Ready but not writable If a disk or clock-skew write gate is active, readiness still reports healthy. Read the structured reason to tell the gates apart: - `WRITES_BLOCKED_DISK_FULL` needs volume headroom below the low-water resume mark; - `WRITES_BLOCKED_CLOCK_SKEW` needs clocks brought within the configured threshold; - `MAINTENANCE_MODE` needs the maintenance owner to release the boundary. Every `503` carries `Retry-After: 1` — `NO_LEADER`, `WRITES_BLOCKED_CLOCK_SKEW`, `MAINTENANCE_MODE`, `INDEX_BUILDING` and `READ_INDEX_NOT_CAUGHT_UP` among them. That value is a fixed default applied to the whole class, not an estimate of how long the condition will last, so read the header as "this response is retryable in principle" and let the structured reason set the actual retry policy. `NO_LEADER` clears within an election and does suit a one-second retry; `MAINTENANCE_MODE` does not, and a client that honors the header literally will retry every second for the length of the window. The disk-full `429` carries no header at all, so a client must pace those retries itself. Do not repeatedly restart ready members. The write gate is cluster state or a health decision, not a liveness failure. ## Diagnose latency Correlate the same time window in this order: 1. request latency and response reasons; 2. proposal queue load and full events; 3. Raft apply and WAL append duration; 4. peer round-trip latency and leadership changes; 5. Pebble flush, compaction, and write stalls; 6. WAL/data bytes and volume IOPS; 7. process memory, garbage collection, CPU throttling, and restarts; 8. index builds, backups, and restore work; 9. changes in request size, bulk concurrency, postings, or Numscript account fan-out. Reduce incoming concurrency when the queue or storage cannot drain. Tune only after identifying the limiting resource and capturing a repeatable baseline. ## Retry and rollback rules | Condition | Client behavior | Operator behavior | | --- | --- | --- | | Brief `NO_LEADER` with `Retry-After` | Retry after the advertised delay. A mutation keeps its original idempotency key and payload; a read has neither and is simply repeated. | Watch election and quorum; intervene if it persists. | | Maintenance mode | Stop automatic retries or back off until the maintenance owner reopens writes. | Keep the application drain in place and complete or roll back the maintenance task. | | Disk-full gate | Do not create a retry storm. Preserve the mutation and its idempotency key. | Restore headroom and verify use below the resume mark. | | Clock-skew gate | Back off; changing request payload does not help. | Restore node time synchronization. | | `409` with `IDEMPOTENCY_KEY_CONFLICT` | The key was reused with different request content. Reissue the original payload unchanged, or use a new key for a genuinely different request. | Compare the retried request body against the recorded first attempt; a conflicting retry is rejected, never applied twice. | | Auth or validation error | Do not retry unchanged credentials or payload. | Correct the client, policy, or request contract. | | Index building or read index behind | Retry only after the required view advances, or use an explicitly acceptable consistency mode. | Keep dependent traffic disabled and restore index progress. | If a deployment or configuration change caused the symptom, stop its rollout and follow that change's rollback procedure. Do not introduce an unrelated tuning change while evidence still points to the original change. ## Prepare an escalation bundle Include: - exact UTC time range and user impact; - Stack, Operator, server, and client versions and image digests; - sanitized request shape, response status, structured reason, and retry header; - cluster status from the leader and affected member; - Operator conditions and Kubernetes events; - relevant logs and traces with credentials, tokens, signing keys, and database DSNs removed; - dashboard snapshots for request, leader, queue, WAL, apply, storage, memory, and network signals; - the last known good change and every action already attempted; - backup status and persistent-volume state when recovery may be required. Preserve the original artifacts. An escalation should allow another operator to reconstruct the timeline without repeating a destructive action. --- ## Bulk processing Source: https://docs.formance.com/modules/ledger/working-with/bulk-processing The `_bulk` endpoint allows for efficient processing of multiple requests in a single operation. It supports streaming, parallel or sequential processing, and atomic execution. ## Request formats ### Non-streaming For standard bulk requests, send operations as a JSON array with `Content-Type: application/json`: ```json [ {"action": "CREATE_TRANSACTION", "data": {"postings": [{"source": "world", "amount": 100, "asset": "USD", "destination": "alice"}]}}, {"action": "CREATE_TRANSACTION", "data": {"postings": [{"source": "world", "amount": 200, "asset": "USD", "destination": "bob"}]}} ] ``` By default, the ledger has a maximum bulk transaction size of items. This limitation helps prevent timeouts when processing large bulk requests. ### Streaming Bulk requests can be streamed without requiring the entire request to be loaded into memory. Results are kept in memory until the full stream is complete, which may result in large responses for big datasets. Consider breaking very large operations into smaller batches. For processing more than 100 items, we recommend using streaming mode instead of increasing the bulk size limit. To enable streaming, include one of the following content type headers in your HTTP request: - For a script stream, include content type `application/vnd.formance.ledger.api.v2.bulk+script-stream` - For a JSON stream, include content type `application/vnd.formance.ledger.api.v2.bulk+json-stream` **Script stream format** For a script stream, each Numscript transaction must be wrapped with `//script` and `//end` delimiters: ```numscript //script send [USD 100] ( source = @world destination = @alice ) //end //script send [USD 100] ( source = @world destination = @bob ) //end ``` **JSON stream format** For a JSON stream, send an array of elements in the request body. Example: ```json [ {"action": "CREATE_TRANSACTION", "data": {"postings": [{"source": "world", "amount": 100, "asset": "USD", "destination": "bank"}]}}, {"action": "CREATE_TRANSACTION", "data": {"postings": [{"source": "world", "amount": 200, "asset": "USD", "destination": "bank"}]}} ] ``` Available actions for JSON bulk operations (script streams support `CREATE_TRANSACTION` only): ## Processing options The bulk endpoint accepts the following query parameters: You cannot set `parallel=true` and `atomic=true` at the same time. The `continueOnFailure` parameter only applies when `parallel=false`. In parallel mode, elements are processed independently regardless of this setting. Example with continue on failure: This is useful when you want to process as many elements as possible and handle failures separately, rather than stopping the entire batch on the first error. ### Idempotency Each bulk element can carry its own idempotency key via an `ik` field (JSON) or `//script ik=` header (Numscript stream), making it safe to retry a partially-failed batch without creating duplicate transactions. See [Idempotency in Bulk Operations](#idempotency-in-bulk-operations) below for the full details. ## Configuration ### Bulk size limits The 100 item limit applies only to non-streaming requests. If your use case requires a larger limit, you can configure it: Increasing the bulk size does not necessarily improve write performance. Test different values to find the optimal setting for your use case. ## Examples ### Script streaming To send a bulk request with script streaming: ### JSON streaming To send a bulk request with JSON streaming: For more details on the bulk API parameters and response format, see the [API Reference](/stack-api-reference/introduction). ## Idempotency in Bulk Operations Idempotency keys prevent duplicate transactions when replaying bulk requests after failures. Each bulk element can specify its own key, making it safe to retry a partially-failed bulk operation without creating duplicates. For script streams, add `ik=` to the script header: ```numscript //script ik=transaction-001 send [USD 100] ( source = @world destination = @alice ) //end ``` For JSON streams, add the `ik` field to each element: ```json {"action": "CREATE_TRANSACTION", "ik": "transaction-001", "data": {"postings": [{"source": "world", "amount": 100, "asset": "USD", "destination": "bank"}]}} ``` When a request uses an idempotency key that was already processed, the API returns the original response with an `Idempotency-Hit: true` header. This helps clients distinguish between new and replayed requests. For the full details on how idempotency keys behave, see [Idempotency](/modules/ledger/working-with/idempotency). --- ## Manage Ledger Configuration as Code Source: https://docs.formance.com/modules/ledger/operations/configuration-as-code Use `ledgerctl ledgers configuration` to keep the editable business configuration of a ledger in version control. The workflow reads the live state, exports an apply-compatible JSON or YAML document, computes a diff, and submits the required Ledger mutations after review. This is a ledger-level workflow. It does not configure Kubernetes resources, voter topology, storage, server flags, authentication, TLS, event sinks, or backup infrastructure. ## Know what the file owns The exported document includes: | Section | Managed state | | --- | --- | | `defaultEnforcementMode` | Default account-type enforcement mode. | | `accountTypes` | Account patterns and persistence modes. | | `metadataSchema` | Ledger, account, and transaction field declarations and their index intent. | | `indexes` | Editable built-in transaction indexes. | | `preparedQueries` | Query name, target, and filter. | | `numscripts` | Program name, content, and version. A content change requires a new, strictly greater canonical semver (`1.2.0`, not `1.2`); editing content under an existing version is rejected. | Read-only state such as creation time and index build status is not exported. Inspect it separately with `ledgerctl ledgers get`, index commands, and the owning workflow. Treat the exported file as complete desired state, not as a partial patch. Removing an account type, index, or prepared query from the file can plan its removal from Ledger. Always start from a fresh export and review every delete in the dry-run output. An omitted section is read as empty, not as unmanaged. A file with no `indexes:` key plans a drop of every built-in transaction index. The Numscript library is the exception, because it is append-only. Dropping a program from the file does not plan its removal: the diff fails with `the append-only library does not support removal`, and because one failing section aborts the whole diff, no plan is produced for any section. The file contains program and query definitions but no credentials. Do not add tokens, signing seeds, TLS private keys, database DSNs, or other secrets to it. ## Export the current state The examples use a ledger named `payments`. Replace that name with the ledger you intend to manage. ### Precheck 1. Confirm that your `ledgerctl` profile targets the intended environment. 2. Confirm that the caller can read every managed section and is allowed to apply the intended mutations. 3. Check for active maintenance or another configuration deployment. 4. Choose one owner for this state. Do not run a CLI reconciler and an Operator reconciler against the same objects. 5. Create or select a reviewed repository path that contains no secrets. Inspect the human-readable summary first: ```bash ledgerctl ledgers configuration payments --expand ``` Export YAML for review: ```bash test ! -e payments.before.yaml || { printf 'refusing to overwrite rollback candidate: payments.before.yaml\n' >&2 exit 1 } ledgerctl ledgers configuration export payments --yaml > payments.before.yaml cp payments.before.yaml payments.ledger.yaml ``` Keep `payments.before.yaml` unchanged as the rollback candidate, and edit only `payments.ledger.yaml`. A Numscript change is the exception to that rollback plan: it cannot be reversed by re-applying the previous file. The export command defaults to JSON when neither `--yaml` nor `--json` is provided. Commit the generated document with the Stack and Ledger versions against which it was exported. ## Edit and review Make one business change per review when possible. The reviewer should answer: - Does a new or changed account type still accept existing valid addresses? - Does an `AUDIT` to `STRICT` change have evidence that unmatched traffic is zero? - Does each declared metadata type match existing values and application expectations? - Does a new query have every required index? - Will dropping an index break a deployed filter or prepared query? - Does the required Numscript version bump preserve callers that pin an older version? - Is any delete intentional, owned, and recoverable? Index builds can continue after the configuration mutation commits. Metadata schema changes declare how Ledger interprets and validates fields; they do not convert or rewrite historical values. The apply command confirms that Ledger accepted the declarations and other changes, not that every asynchronous index result is ready. ## Preview the live diff Run the dry-run against the target environment immediately before approval: ```bash ledgerctl ledgers configuration apply payments \ --file payments.ledger.yaml \ --dry-run ``` The command fetches the current live configuration and prints the planned add, update, and remove actions without applying them. Review the plan for: - unexpected deletes; - a default enforcement change; - metadata type changes; - index removal or creation; - prepared-query replacement; - a Numscript version change. The dry-run does not reserve the state. Another writer can change the ledger before the real apply, so run it again when the approval or deployment queue introduces a delay. ## Apply the configuration Apply interactively after reviewing the current diff: ```bash ledgerctl ledgers configuration apply payments \ --file payments.ledger.yaml ``` The prompt defaults to no. For an approved non-interactive deployment, use `--yes` and preserve the dry-run and approval as CI artifacts: ```bash ledgerctl ledgers configuration apply payments \ --file payments.ledger.yaml \ --yes ``` If Ledger requires request signing, provide both the approved signing key and its registered key ID through the secure CLI profile or the global `--signing-key` and `--signing-key-id` options. This configuration apply command submits its complete change set through the signed `Apply` path, including prepared-query changes that the direct `ledgerctl queries` commands cannot sign in the current preview. Never place the seed in the desired-state file or CI log. ## Verify independently Export the result again: ```bash ledgerctl ledgers configuration export payments --yaml > payments.actual.yaml diff -u payments.ledger.yaml payments.actual.yaml ``` Then verify each affected behavior: 1. List account types and confirm the expected default enforcement mode. 2. Confirm the metadata schema declaration and test representative current and historical values against the intended type. 3. Wait for each required index to become ready on every serving replica. 4. Execute each changed prepared query with representative parameters. 5. Execute each changed Numscript program with test variables and verify the resulting postings. 6. Test one expected rejection as well as one accepted request when enforcement changed. 7. Confirm that request errors, index lag, and latency remain within the deployment thresholds. The change succeeds when the exported live desired state matches the reviewed file and every asynchronous or business check has reached its expected result. ## Roll back Before applying a change, preserve the previous export as the rollback candidate. Preview it against the current ledger before applying it: If the change published a new Numscript version, the previous export can no longer be applied at all. The diff requires a version strictly greater than the greatest one stored, so a file naming the older version fails before any request is built, and that one failure aborts the diff for every other section too. Rolling a program back means publishing a higher version whose content matches the old one. Plan that forward correction before you apply a Numscript change, not after. ```bash ledgerctl ledgers configuration apply payments \ --file payments.before.yaml \ --dry-run ``` Apply it only after reviewing the new rollback plan. Reapplying the previous file can restore declarations, but it does not automatically reverse all consequences: - an index may need to rebuild; - restoring a previous metadata type declaration does not rewrite values accepted under the newer declaration, so validate their compatibility first; - a removed prepared query or program may have interrupted callers; - switching to `STRICT` may already have rejected requests; - removing configuration does not delete or rewrite historical transactions. When rollback would remove newly accepted business state or when the previous declaration no longer validates, stop automated reconciliation and use a forward correction. Keep the pre-change export, dry-run, apply result, post-change export, and verification evidence with the deployment record. --- ## Filtering queries Source: https://docs.formance.com/modules/ledger/working-with/filtering-queries Ledger V2 comes with a number of operations allowing you to list resources such as transactions or accounts. In some cases, you may want to filter the results to only include resources that match certain criteria. This is where filtering comes in. ## Filtering syntax Filtering is done either: - by using a query parameter named `query` and containing a JSON object describing the filter criteria. - by sending the filter criteria as a JSON object in the body of the request. Here is a value example: ```json { "$match": { "source": "order::pending" } } ``` In this example, we are filtering on a ledger account. The naming convention set on this account is the following: `order:XXXX:pending` In order to get all accounts, with unique value `XXXX` of order segment, we use this regex-like syntax `order::pending`. An address is divided into segments delimited by colons `:`. You can use wildcards for partial address matching: - Empty segment matches any single segment: `order::pending` matches `order:123:pending`, `order:456:pending`, etc. - Trailing colon matches exactly one more segment: `order:` matches `order:123`, `order:456`, etc., but **not** `order:123:pending`. - Trailing `...` matches any remaining segments (the whole subtree): `wallet:user123:...` matches `wallet:user123:main`, `wallet:user123:pending:hold`, etc. For a metadata filter, the syntax is the following: ```json { "$match": { "metadata[foo]": "bar"} } ``` In this example, we are filtering on a metadata field named `foo` with the value `bar`. ## Combining filters The root of the filter object has always a single key. This key is the operation to apply to the filter. In order to combine filters, you can use the `$and` and `$or` operations. Here is an example: ```json { "$and": [ { "$match": { "source": "order::pending" } }, { "$match": { "metadata[foo]": "bar" } } ] } ``` In this example, we are combining two filters using the `$and` operation. The result will only include resources that match both filters. ## Cross-posting filter behavior When filtering transactions based on posting fields (like `source` and `destination` addresses), filters apply **across all postings** within a transaction, not just individual postings. ### How it works If you filter for transactions where: - `source = "world"` AND - `destination = "user:123"` The transaction will match if **any** of its postings have `source="world"` AND **any** of its postings have `destination="user:123"`. The source and destination **don't need to be in the same posting**. ### Example Consider this transaction with multiple postings: ```json { "postings": [ { "source": "world", "destination": "order:abc:payment", "amount": 100, "asset": "USD/2" }, { "source": "order:abc:payment", "destination": "platform:fee", "amount": 1, "asset": "USD/2" }, { "source": "order:abc:payment", "destination": "user:123", "amount": 99, "asset": "USD/2" } ] } ``` This transaction **would match** the filter `source="world" AND destination="user:123"` because: - The first posting has `source="world"` (matches source filter) - The third posting has `destination="user:123"` (matches destination filter) Even though no single posting has both `source="world"` AND `destination="user:123"`, the transaction matches because the criteria are met across multiple postings. This behavior can lead to unexpected results when filtering multi-posting transactions. If you need to match source and destination within the same posting, you may need to query at a more granular level or use additional criteria. ## Using the `$in` operator The `$in` operator matches resources where a field equals any value in a provided list. This is useful when filtering by multiple specific values. Instead of using multiple `$or` and `$match` filters: ```json { "$or": [ { "$match": { "address": "account:1" } }, { "$match": { "address": "account:2" } }, { "$match": { "address": "account:3" } } ] } ``` You can use the `$in` operator: ```json { "$in": { "address": ["account:1", "account:2", "account:3"] } } ``` The `$in` operator only supports full addresses, not partial matches like `account:`. This is enforced at validation time. ## Filtering operations reference ### Operations `$like` only does SQL `LIKE` on a few free-form string fields — `transactions.id`, `transactions.reference`, `logs.type`, `ledgers.name`. On address-shaped fields (`address` / `account` / `source` / `destination`) it's silently treated like `$match`; use the segment syntax above for partial-address matching, not `%`. ### Filterable fields Depending on the resource you are querying, you can filter on different fields: #### Accounts #### Transactions #### Volumes #### Aggregated balances #### Logs ## Using the Volumes Endpoint For performance, consider using the `/volumes` endpoint instead of `/transactions` when you only need balance information. The volumes endpoint is optimized for aggregate balance queries. The volumes endpoint supports the following query parameters: ### Example: Querying volumes up to a point in time ### Example: Querying volumes within a time range When working with [bi-temporality](/modules/ledger/working-with/bi-temporality), pass `insertionDate=true` to filter the time range against each transaction's **insertion timestamp** rather than its effective (`date`) timestamp. The `/volumes` endpoint does not accept a `pit` query parameter — use `endTime` (optionally combined with `insertionDate=true`) to query the ledger state as it was at a given point in time. --- ## Idempotency Source: https://docs.formance.com/modules/ledger/working-with/idempotency When building financial applications, preventing duplicate transactions is critical. The Formance Ledger provides two mechanisms to ensure transaction uniqueness: **idempotency keys** and **references**. ## Idempotency Keys Add an `Idempotency-Key` header to your request. If you execute the same request twice with the same key, the system will skip processing and return the original successful response. ### How it works 1. When you send a request with an idempotency key, the ledger stores a hash of the request along with the response 2. If you retry with the same key and identical request, the ledger returns the cached response with an `Idempotency-Hit: true` header 3. If you retry with the same key but different request parameters, the ledger returns a validation error ### Supported endpoints Idempotency keys work on all write endpoints: - Create transaction (including batch and Numscript) - Update metadata (account or transaction) - Revert transaction ### Scope Idempotency keys are scoped to a specific ledger. The same key can be used on different ledgers without conflict. ### Example ### Replay Retrying the same request with the same key returns the cached response — no second transaction is created. The response body is identical to the original, and the response includes an `Idempotency-Hit: true` header: ```http HTTP/1.1 200 OK Idempotency-Hit: true ``` If the request body differs from the original, the ledger returns a `VALIDATION` error instead. See [Validation Errors](#validation-errors). ## References A `reference` is a field in the transaction body that acts as a unique identifier. If a transaction with the same reference already exists, the ledger returns an error. ### Example {"fctl ledger transactions num - --confirm --ledger my-ledger --reference order-12345-payment <<< 'send [USD/2 100] ( source = @world destination = @users:alice )'"} ## Choosing Between Idempotency Keys and References | Use Case | Recommended | Reason | |----------|-------------|--------| | Retry logic for network failures | Idempotency Key | System returns success on retry | | Mapping to external entities (e.g., orders, refunds) | Reference | Get an error if duplicate, useful for debugging | | Unknown or dynamic transaction identity | Idempotency Key | Works with any unique value | Use a **reference** when you have a unique entity in your system that the transaction should match (e.g., a `refund` object creating a unique ledger transaction). You'll get an error if you try to re-submit with the same reference. Use an **idempotency key** when the identity is less obvious or you want silent duplicate handling. The system will skip processing and return success on retry. ## Validation Errors Reusing an idempotency key with different request content returns a `VALIDATION` error (HTTP 400). The ledger compares a hash of the request inputs against the hash stored with the original log entry; a mismatch is rejected. ### Error format ```json { "errorCode": "VALIDATION", "errorMessage": "invalid idempotency hash when using idempotency key 'unique-key-123', has computed 'abc123...' but 'xyz789...' is stored" } ``` ### Common causes - Changing any field in the request body (including metadata) - Using a different endpoint path - Modifying the request after a previous successful call ### Best practices 1. **Use unique keys for each distinct transaction** - Generate a new key for each new transaction 2. **Keep request parameters consistent** - If retrying, ensure all parameters match the original request exactly 3. **Use a new key if parameters change** - If you need to modify the transaction, generate a new idempotency key Changing **any** part of the transaction, including metadata, will trigger a `VALIDATION` error when reusing an idempotency key. Transactions written by Ledger 2.1 or earlier have no stored idempotency hash. For those entries the hash comparison is skipped and the original response is replayed, for backward compatibility. For idempotency keys in bulk operations, see [Bulk Processing](/modules/ledger/working-with/bulk-processing#idempotency-in-bulk-operations). --- ## Accounts Source: https://docs.formance.com/modules/payments/accounts An **account** in Payments is a stable representation of an account that lives on an upstream payment provider — a Stripe balance, a Bankingcircle settlement account, a Mangopay wallet, a Coinbase Prime trading wallet. Each PSP-side account maps one-to-one to a Payments `PSPAccount`, so the rest of the platform can reason about provider balances and flows through a uniform shape regardless of which connector they came from. ## Account model Every account carries: | Field | Description | | --- | --- | | `id` | Platform-assigned UUID. Stable across reinstalls of the same connector. | | `reference` | The account ID on the provider. Unique within the connector. | | `connectorID`, `provider` | The connector instance the account belongs to. | | `type` | `INTERNAL` (you can read balances, source transfers) or `EXTERNAL` (destination-only, typically a bank account). | | `createdAt` | Provider-side creation date when available; sentinel value when the provider doesn't expose one (e.g. Bitstamp defaults to the venue's launch date). | | `defaultAsset` | UMN-formatted asset code (`USD/2`, `EUR/2`, `BTC/8`). Optional — multi-asset accounts leave it null. | | `name` | Human-readable name when the provider exposes one. | | `metadata` | Connector-specific keys under `com..spec/` — chain IDs for crypto, IBAN for bank accounts, payable type for Routable, etc. | | `raw` | The full upstream payload, preserved verbatim for fields the typed surface doesn't cover. | Internal accounts are operationally relevant — the connector can fetch balances, surface payment history, and (where supported) initiate outbound transfers from them. External accounts are destination-only: funds can land in them via [Payment Initiation](/modules/payments/transfer-initiation), but no balance or transaction history is fetched. ## Listing accounts ### Formance Console Open the **Accounts** tab in the sidebar. The page lists every account synced from every installed connector with filterable columns for connector, asset, type, and creation date. Click an account to see its metadata, raw upstream payload, and (for internal accounts) the latest balance per asset. ### API To filter by connector or type, post a query body: ```bash curl -s "$STACK/api/payments/v3/accounts?pageSize=15" \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{"$match": {"connectorID": "'$CONNECTOR_ID'", "type": "INTERNAL"}}' | jq ``` A single account by id: " }} /> The same `$match` operator works against `default_asset`, `metadata.`, and `psu_id` for Open Banking accounts — see [Cash pools](/modules/payments/cash-pools#query-syntax) for the full query syntax. ## Reading balances Balances are fetched per cycle by the connector's `FetchBalances` capability and stored as point-in-time snapshots. No provider exposes a historical balance series, so Payments surfaces only the latest value per `(account, asset)`: " }} /> Returns one row per asset the account holds. Empty when the account type is `EXTERNAL` (those don't carry balances). For aggregating balances across many accounts (treasury views, reconciliation against a ledger), wrap them in a [cash pool](/modules/payments/cash-pools) — the pool endpoints sum balances by asset across every account in the pool. ## Internal accounts INTERNAL accounts are operationally controlled accounts at the provider: - a Stripe `acct_*` balance, - a Bankingcircle settlement account, - a Mangopay wallet, - a Coinbase Prime trading wallet, - etc. You can read balances, see the full transaction history, and (when the connector supports it) initiate transfers and payouts from them: 1. **Transfer** — INTERNAL → INTERNAL movement at the same connector. 2. **Payout** — INTERNAL → EXTERNAL movement (e.g. settlement to a bank account). See [Payment Initiation](/modules/payments/transfer-initiation) for the v3 API. ## External accounts EXTERNAL accounts are destination-only: bank accounts, beneficiaries, or counterparty wallets the provider knows about but doesn't operationally control on your behalf. No balance or transaction history is fetched. You move funds *to* them via Payment Initiation. ### Creating an external account Some connectors let you create an external account on the provider side from the Payments API — useful when you want to register a new beneficiary before payouting to it: The connector must declare `CAPABILITY_CREATE_BANK_ACCOUNT` — see the [capability matrix](/modules/payments/capabilities). The response links the freshly-created bank account to the connector's account list; fetch it back with `v3GetAccount` against the returned ID. External account fields and provider-specific requirements vary — Bankingcircle wants an IBAN + BIC, Column wants a routing number + account number, Mangopay wants both depending on the destination country. Refer to the per-connector reference page for the exact field set each provider requires. --- ## Exchange Connectors Source: https://docs.formance.com/modules/payments/connectors/exchange Exchange connectors integrate Formance Payments with crypto and asset exchanges. They surface wallets, balances, and payments alongside two primitives that don't exist on traditional PSPs — **trading orders** (instructions placed against an order book) and **conversions** (atomic two-asset swaps that bypass the order book). Every connector in this category ships in the **Enterprise edition** of the Formance stack. An EE license is required to install or run them. ## Available exchange connectors Connect a Coinbase Prime portfolio. Available from Payments module 3.2.0; Orders and Conversions added in 3.3.0. Connect a Fireblocks workspace. Read-only; available from Payments module 3.2.0. Connect a Bitstamp account. Read-only spot-only; available from Payments module 3.3.0. Connect a Kraken Pro account. Read-only spot-only; available from Payments module 3.4.0. ## What's different about Exchange connectors Two model differences are worth knowing upfront, both surfaced as their own resources in the Payments API: - **[Orders](/modules/payments/orders)** — an instruction to swap one asset for another at a price, with a fill lifecycle (`PENDING` → `OPEN` → `PARTIALLY_FILLED` → `FILLED` / `CANCELLED` / `EXPIRED`). Each Order pairs two assets and two account legs on the same venue. A standard PSP Payment moves a single asset on a single leg. - **[Conversions](/modules/payments/conversions)** — atomic two-asset swaps with distinct source and destination amounts, bypassing the order book entirely. Use them for stablecoin redemption (USDC ↔ USD), wrapped-asset swaps, and direct asset-to-asset exchanges that don't need price discovery. Both Orders and Conversions are gated to Payments module 3.3.0 and above. Exchange connectors on earlier module versions expose Accounts, Balances, and Payments only. --- ## Data isolation with buckets Source: https://docs.formance.com/modules/ledger/working-with/data-isolation-buckets Buckets let you isolate ledgers within the same database server. This is useful for separating data by client, application, or environment. ## Understanding PostgreSQL Schemas At a technical level, **a bucket in Formance is directly mapped to a PostgreSQL schema**. In PostgreSQL, a schema is a **namespace** that groups together database objects (tables, views, indexes, functions): - **Without schemas**: All tables would live in one big global namespace, which could quickly become messy - **With schemas**: You can organize and separate data logically, avoid name collisions, and apply different permissions Two tables with the same name can exist in different schemas, as they live under different namespaces. ## How Formance Uses Schemas When you create a ledger, Formance stores its data inside a PostgreSQL schema (the bucket). This has several implications: | Aspect | Description | |--------|-------------| | **Isolation** | Each bucket/schema provides clean separation of ledger data. Two ledgers in different buckets won't share tables, reducing risk of accidental data mixing. | | **Shared storage within a bucket** | Multiple ledgers inside the same bucket share the same underlying tables, which can be efficient but means their data is less isolated. | | **Scaling strategy** | By using multiple buckets, you can spread data across different schemas to avoid a single schema growing too large and becoming a performance bottleneck. | ### System Schema Formance uses a special internal schema called `_system` to track metadata about your ledgers and buckets. If you ever delete a bucket manually in PostgreSQL (via `DROP SCHEMA`), you must also remove the corresponding entry in the `_system` schema. Skipping this step can leave your ledger registry in an inconsistent state. ## Practical Considerations - **Data Isolation**: Buckets are a good fit if you need strong separation (e.g., per-tenant ledgers in a multi-tenant system) - **Performance Management**: For high-volume workloads, spreading ledgers across multiple buckets can reduce contention and keep queries fast - **Simplicity**: If isolation isn't critical, sticking with the `_default` bucket keeps things straightforward ## Creating a bucket Buckets are automatically created when you create a new ledger. By default, if the bucket is not specified, the ledger is created in the `_default` bucket. Create a ledger in a specific bucket: {"fctl ledger create testing --bucket bucket0"} **Response:** ```json { "data": { "bucket": "bucket0", "metadata": {}, "features": { "ACCOUNT_METADATA_HISTORY": "SYNC", "HASH_LOGS": "SYNC", "MOVES_HISTORY": "ON", "MOVES_HISTORY_POST_COMMIT_EFFECTIVE_VOLUMES": "SYNC", "TRANSACTION_METADATA_HISTORY": "SYNC" }, "id": 2, "name": "testing", "addedAt": "2024-10-03T08:27:11.540373Z" } } ``` ## Features Each ledger can be configured with a set of features at creation time. Features let you tune ledger behavior for different use cases (high write throughput, full audit trail, and so on). **Features are immutable after ledger creation.** You cannot change them once the ledger exists. Plan your feature set carefully. When you create a ledger without specifying features, all features are enabled with their default values. You only need to specify features you want to override; unspecified features receive their defaults. ### Features summary The current set of features is not stable; some can be added or removed. ### MOVES_HISTORY **Values:** `ON` | `OFF` **Default:** `ON` When enabled, the ledger tracks every individual fund movement for each account/asset pair in the `moves` table. Each move record includes account address, asset, amount, source/destination flag, insertion date, effective date, and post-commit volumes. **When enabled (ON):** Full balance history at any point in time, historical balance queries, point-in-time support. Required for effective volumes calculation. **When disabled (OFF):** Only current balances are available; better write performance and lower storage. ### MOVES_HISTORY_POST_COMMIT_EFFECTIVE_VOLUMES **Values:** `SYNC` | `DISABLED` **Default:** `SYNC` Maintains the `post_commit_effective_volumes` column in moves, which tracks volumes ordered by **effective date** (not insertion date). When you create backdated transactions, effective volumes for later moves are updated automatically. Depends on MOVES_HISTORY being ON. If MOVES_HISTORY is OFF, this feature has no effect. **When enabled (SYNC):** Transaction responses include `postCommitEffectiveVolumes`. Accurate historical balance queries and correct handling of backdated transactions. **When disabled (DISABLED):** The `postCommitEffectiveVolumes` property is not present on transaction responses. Better write performance when transactions are always inserted in chronological order. ### HASH_LOGS **Values:** `SYNC` | `ASYNC` | `DISABLED` **Default:** `SYNC` Provides cryptographic integrity verification for the ledger log chain. Each log entry is hashed (SHA-256) and chained to the previous log's hash. - **SYNC:** Hashes are computed synchronously during each log insert. Hash is immediately available; only one log can be inserted at a time per ledger (advisory lock). Best for strong integrity guarantees with lower throughput. - **ASYNC:** Logs are inserted without a hash; a background worker computes hashes in batches (blocks). No locking on insert, so higher write throughput. Requires the ledger worker process and configuration (e.g. `--worker-async-block-hasher-max-block-size`, `--worker-async-block-hasher-schedule`). - **DISABLED:** No hashing. Maximum write performance; no integrity verification. ### ACCOUNT_METADATA_HISTORY **Values:** `SYNC` | `DISABLED` **Default:** `SYNC` When enabled, every change to account metadata is historized with revision tracking. You can query account metadata at any point in time using the `pit` parameter. When disabled, only current metadata is stored and PIT queries return current values. ### TRANSACTION_METADATA_HISTORY **Values:** `SYNC` | `DISABLED` **Default:** `SYNC` When enabled, every change to transaction metadata is historized. You can query transaction metadata at any point in time using the `pit` parameter. When disabled, only current metadata is stored and PIT queries return current values. ### Setting features at ledger creation Use `POST /v2/{ledger}` and pass features in the request body. Only specify features you want to override. **Request body:** ```json { "bucket": "optional-bucket-name", "metadata": {}, "features": { "FEATURE_NAME": "VALUE" } } ``` **Create ledger with all defaults:** **Create high-throughput ledger (minimal features):** **Create ledger with async hashing:** **Override specific features only (others get defaults):** ### Feature sets **Default (full audit):** All features enabled. Use when you need full audit capabilities. **Minimal (performance):** All features disabled or minimized. Use for high-throughput scenarios where an audit trail is not required. **Async hashing (balanced):** `HASH_LOGS: "ASYNC"` with other features at defaults. Use for higher throughput with eventual integrity verification (requires the ledger worker). ### Feature impact summary | Feature | When enabled | When disabled | |---------|--------------|---------------| | MOVES_HISTORY | Full balance history available | Only current balances available | | MOVES_HISTORY_POST_COMMIT_EFFECTIVE_VOLUMES | postCommitEffectiveVolumes on transactions | postCommitEffectiveVolumes not available | | HASH_LOGS | Logs cryptographically chained | No hash verification | | ACCOUNT_METADATA_HISTORY | Full metadata revision history | Only current metadata stored | | TRANSACTION_METADATA_HISTORY | Full metadata revision history | Only current metadata stored | **ASYNC hashing:** If you use `HASH_LOGS: "ASYNC"`, you must run the ledger worker process with the appropriate configuration. **Dependencies:** MOVES_HISTORY_POST_COMMIT_EFFECTIVE_VOLUMES depends on MOVES_HISTORY. There is a trade-off between write performance and audit capabilities; choose based on your use case. ## Deleting and restoring buckets You can delete buckets and restore them within a retention period if needed. ### Deleting a bucket When you delete a bucket: - All ledgers in the bucket become hidden from normal queries - The data is preserved for 30 days (configurable) - You can restore the bucket during this period ### Restoring a bucket If you deleted a bucket by mistake, restore it before the retention period ends: All ledgers in the bucket become visible again. ### Listing deleted ledgers To see deleted ledgers, add `includeDeleted=true` to your query: Deleted ledgers include a `deletedAt` timestamp in the response: ```json { "cursor": { "data": [ { "name": "my-ledger", "bucket": "my-bucket", "addedAt": "2025-01-10T10:00:00Z", "deletedAt": "2025-01-14T15:30:00Z" } ] } } ``` ### Configuring retention By default, deleted buckets are permanently removed after 30 days. To change this, configure the worker: ```bash ledger worker \ --worker-bucket-cleanup-retention-period=2160h \ --worker-bucket-cleanup-schedule="0 0 */6 * * *" ``` | Flag | Description | Default | |------|-------------|---------| | `--worker-bucket-cleanup-retention-period` | How long to keep deleted buckets | 30 days (`720h`) | | `--worker-bucket-cleanup-schedule` | How often to check for expired buckets (cron format) | Every hour | After the retention period, deleted buckets are permanently removed and cannot be recovered. --- ## Streaming to analytics systems Source: https://docs.formance.com/modules/ledger/working-with/data-streaming Stream your ledger data in real-time to analytics systems like ClickHouse, Elasticsearch, or custom HTTP endpoints. This lets you build dashboards, enable full-text search, or trigger external workflows—without polling the API. ## Architecture The export system has two components: **exporters** define where data goes (driver + connection config), and **pipelines** connect a ledger to an exporter. Multiple ledgers can share the same exporter. ```mermaid flowchart LR L1[ledger-001] --> P1[Pipeline 1] L2[ledger-002] --> P2[Pipeline 2] L3[ledger-003] --> P3[Pipeline 3] P1 --> E[ClickHouse Exporter] P2 --> E P3 --> E ``` ## What gets streamed Every change to your ledger is recorded as a log entry. Pipelines stream these logs to your chosen destination: ## Quick start Here's the complete flow to start streaming data: **1. Create an exporter** (defines where data goes) Response: ```json { "data": { "id": "exp-abc123", "driver": "clickhouse", "config": { "dsn": "clickhouse://localhost:9000" }, "createdAt": "2025-01-15T10:00:00Z" } } ``` **2. Create a pipeline** (connects a ledger to the exporter) Pipelines start automatically. Logs begin streaming immediately. ## Exporter drivers ```json { "driver": "clickhouse", "config": { "dsn": "clickhouse://user:password@localhost:9000/database" } } ``` ```json { "driver": "elasticsearch", "config": { "endpoint": "https://localhost:9200", "index": "ledger-logs", "authentication": { "username": "elastic", "password": "secret" } } } ``` ```json { "driver": "http", "config": { "url": "https://your-service.example.com/ledger-logs" } } ``` ```json { "driver": "stdout", "config": {} } ``` Your endpoint receives a JSON array of log entries: ```json [ { "ledger": "my-ledger", "id": 42, "type": "NEW_TRANSACTION", "date": "2025-01-15T10:30:00Z", "data": { "transaction": { "id": 1, "postings": [...], "metadata": {} } } } ] ``` Return a 2xx status code to acknowledge receipt. ## Managing exporters **List all exporters:** **Get a specific exporter:** " }} noFctl /> **Delete an exporter:** " }} noFctl /> ## Managing pipelines ### Check pipeline status " }} noFctl /> The `lastLogID` shows how far the pipeline has progressed: ```json { "data": { "id": "pipe-xyz789", "ledger": "my-ledger", "exporterID": "exp-abc123", "lastLogID": 1042, "enabled": true, "createdAt": "2025-01-15T10:00:00Z" } } ``` ### Stop and restart Pipelines remember their position. Stop and start without losing progress: **Stop streaming:** " }} noFctl /> **Resume streaming:** " }} noFctl /> ### Replay from the beginning Reset the pipeline to re-stream all logs: " }} noFctl /> This replays all historical logs. Make sure your destination can handle duplicates or clear it first. ### Delete a pipeline " }} noFctl /> You cannot delete an exporter that has active pipelines connected to it. Delete or reassign the pipelines first. ## Scaling with multiple pipelines You can attach multiple ledgers to the same exporter. This helps avoid rate limits on external systems, spread load, and maintain independent data flows per ledger. " }} /> Repeat for each ledger — all pipelines share the same exporter. --- ## Ledger Schema Source: https://docs.formance.com/modules/ledger/working-with/ledger-schema ## What is a Ledger Schema? A Ledger Schema is an optional configuration that defines which account addresses are valid in your ledger. Think of it as a type system for your chart of accounts: you declare the patterns, and the ledger enforces them at write time. A schema has three parts: a `chart` (which account addresses are allowed), `transactions` (reusable Numscript templates), and `queries` (reusable parameterized queries). Schemas are versioned and immutable. Once you publish a version, it cannot be changed; create a new version to evolve your structure. Use a schema when you want to enforce transaction and account patterns, simplify reads and queries, and ensure the ledger does not contain things you don't expect. ## Why use a Ledger Schema? By default, the ledger accepts any account address. A schema adds structure: - **Catch errors early**: Reject typos like `users:alcie` before they create orphaned accounts - **Enforce naming conventions**: Require user IDs to match a specific format - **Auto-assign metadata**: New accounts automatically get default metadata values - **Audit trail**: Every transaction records which schema version validated it For guidance on designing your account hierarchy, see [Chart of Accounts](/modules/ledger/core-concepts/chart-of-accounts). ## Schema structure A schema consists of three required fields: - `chart`: Defines valid account patterns - `transactions`: Defines reusable transaction templates (can be `{}`). See [Transaction templates](#transaction-templates). - `queries`: Defines reusable query templates (can be `{}`). See [Query templates](#query-templates). ```json { "chart": { "world": {}, "banks": { "$iban": { ".pattern": "^[A-Z]{2}[0-9]{2}[A-Z0-9]{1,30}$", ".self": {}, "main": {}, "fees": {} } }, "users": { "$userId": { ".metadata": { "type": { "default": "customer" } } } } }, "transactions": {}, "queries": {} } ``` ## Defining your chart The chart uses a nested JSON structure with special prefixes to distinguish between account segments and properties. ### Fixed segments Fixed segments are literal account path components. In the example above, `world`, `banks`, `users`, `main`, and `fees` are fixed segments. ```json { "banks": { "main": {}, "fees": {} } } ``` This defines valid accounts: `banks:main` and `banks:fees`. ### Variable segments Variable segments start with `$` and match any value. The text after `$` is the variable name (e.g., `$userId`, `$orderId`), which helps document what the segment represents. ```json { "users": { "$userId": {} } } ``` This matches any account like `users:123`, `users:alice`, or `users:order-456`. ### Pattern validation Add a `.pattern` property to validate variable segments against a regular expression: ```json { "banks": { "$iban": { ".pattern": "^[A-Z]{2}[0-9]{2}[A-Z0-9]{1,30}$" } } } ``` This only matches accounts where the IBAN segment is valid, like `banks:GB82WEST12345698765432`. An account like `banks:abc123` would be rejected because it doesn't match the IBAN format. ### Leaf vs non-leaf accounts By default, a segment without children is a valid account (leaf). To define a segment that: - Has children AND - Is itself a valid account Use the `.self` property: ```json { "orders": { "$orderId": { ".self": {}, "pending": {}, "completed": {} } } } ``` This makes all of these valid: - `orders:123` (the order itself) - `orders:123:pending` (pending state) - `orders:123:completed` (completed state) Without `.self`, only `orders:123:pending` and `orders:123:completed` would be valid. ### Default metadata Define default metadata values for accounts matching a pattern: ```json { "users": { "$userId": { ".metadata": { "type": { "default": "customer" }, "tier": { "default": "standard" } } } } } ``` When an account like `users:alice` is created, it automatically receives `{"type": "customer", "tier": "standard"}`. Defaults only apply when an account is first created. They never overwrite existing metadata. ### Validation rules When defining your chart, keep these constraints in mind: - **Segment names** can only contain letters, numbers, underscores, and hyphens - **Root segments** must be fixed—you cannot start your chart with a variable (`$userId`) or property (`.pattern`) - **One variable per level** — each level in your chart can have at most one variable segment - **Patterns only on variables** — you cannot add `.pattern` to a fixed segment For example, this is invalid because it has two variable segments at the same level: ```json { "users": { "$userId": {}, "$username": {} } } ``` ## Managing schemas ### Create a schema Insert a schema with a version identifier: **Go SDK equivalent:** ```go _, err := client.Ledger.V2.InsertSchema(ctx, operations.V2InsertSchemaRequest{ Ledger: "my-ledger", Version: "v1.0.0", V2SchemaData: components.V2SchemaData{ Chart: map[string]any{ "world": map[string]any{}, "users": map[string]any{"$userId": map[string]any{}}, "merchants": map[string]any{ "$merchantId": map[string]any{ "revenue": map[string]any{}, "payouts": map[string]any{}, }, }, }, Transactions: map[string]any{}, }, }) ``` Schemas are immutable—once created, a version cannot be modified. Create a new version to evolve your account structure. Use semantic versioning (e.g., `v1.0.0`, `v1.1.0`, `v2.0.0`) to communicate breaking vs non-breaking changes. ### List schemas Retrieve all schema versions for a ledger. The endpoint supports cursor-based pagination and sorting. | Query parameter | Type | Default | Description | | --- | --- | --- | --- | | `cursor` | string | — | Pagination cursor from the previous response's `next` or `previous` cursor | | `pageSize` | integer | 15 | Maximum number of schema versions to return per page | | `sort` | string | `created_at` | Field to sort by (currently only `created_at` is supported) | | `order` | string | `desc` | Sort order: `asc` or `desc` | **With pagination and sort order:** The response includes a `next` cursor when more results are available; use it as the `cursor` query parameter to fetch the next page. ### Get a specific schema Retrieve a schema by version. The response returns the full schema JSON (chart, transactions, and queries) for that version. ## Creating transactions with a schema Once a schema exists on a ledger, you must pass the `schemaVersion` query parameter when creating transactions: The ledger validates that all accounts (source and destination) match the chart before committing. The schema version is recorded in the transaction log for audit purposes. When reverting a transaction, pass the same `schemaVersion` query parameter so the revert is validated against the schema and the version is recorded in the log: ## Enforcement modes Control what happens when validation fails: | Mode | Behavior | | --- | --- | | `audit` (default) | Allow the transaction (validation failures are logged for review) | | `strict` | Reject the transaction with an error | ### Behavior summary | Scenario | Strict | Audit | | --- | --- | --- | | Schema specified, validation passes | ✓ Commits | ✓ Commits | | Schema specified, validation fails | ✗ Rejects | ⚠ Warns, commits | | Schema specified but doesn't exist | ✗ Rejects | ✗ Rejects | | No schema specified, but schemas exist in ledger | ✗ Rejects | ⚠ Warns, commits | | No schema specified, no schemas exist | ✓ Commits | ✓ Commits | When a schema is specified but not found, it's always an error regardless of mode. The enforcement mode only affects validation failures. ### Configuration ## Transaction templates Transaction templates let you define reusable Numscript programs in your schema. Instead of sending raw Numscript with each request, your application references a template by name and provides variable values. ### Defining templates Templates are defined in the `transactions` field alongside the chart: ```json { "chart": { "world": {}, "users": { "$userId": { "wallet": {} } } }, "transactions": { "DEPOSIT": { "description": "Fund a user wallet", "script": "vars {\n account $user\n}\nsend [COIN 10] (\n source = @world\n destination = $user\n)" } } } ``` Each template has the following properties: | Property | Required | Description | | --- | --- | --- | | `script` | Yes | The Numscript program to execute | | `description` | No | Human-readable description of what the template does | | `runtime` | No | Which Numscript interpreter to use: `machine` (default) or `experimental-interpreter` | See [Selecting an Interpreter](/modules/numscript/interpreter) for the difference between the `machine` and `experimental-interpreter` runtimes, and which experimental features each Ledger release supports. ### Executing templates To execute a template, pass the template name and variables in the `script` field: Variables can be passed as: - **Strings**: `"user": "users:alice:wallet"` - **Monetary values**: `"amount": "USD/2 5000"` or `"amount": { "asset": "USD/2", "amount": 5000 }` The ledger executes your template with the provided values. The template name is recorded on the transaction and returned in the response, so you can trace which template was used for each transaction. When a schema exists with templates, transactions must reference a template. In strict mode, transactions without a template are rejected. In audit mode, a warning is logged but the transaction is allowed. ## Query templates Query templates let you define reusable, parameterized queries in your schema. Instead of constructing filter expressions in your application code, you define named queries that target a specific resource type (transactions, accounts, logs, or volumes) and accept typed variables at runtime. ### Defining query templates Query templates are defined in the `queries` field of your schema: ```json { "chart": { "..." : {} }, "transactions": {}, "queries": { "RECENT_USER_TRANSACTIONS": { "description": "List transactions for a specific user", "resource": "transactions", "vars": { "userId": { "type": "account" } }, "body": { "$match": { "destination": ":userId:" } }, "params": { "pageSize": 25, "sort": "id:desc" } } } } ``` Each query template has the following properties: | Property | Required | Description | | --- | --- | --- | | `resource` | Yes | The resource type to query: `transactions`, `accounts`, `logs`, or `volumes` | | `body` | No | A filter expression using the same syntax as [filtering queries](/modules/ledger/working-with/filtering-queries). Variables are referenced with `:varName:` syntax. | | `vars` | No | Variable declarations with types. Each variable specifies a `type` (e.g., `account`, `string`) and an optional `default` value. | | `params` | No | Default pagination and sorting parameters (`pageSize`, `sort`, `endTime`, `startTime`, `expand`) | | `description` | No | Human-readable description of what the query does | For `volumes` resources, `params` also supports `groupBy` (integer) and `insertionDate` (boolean). ### Running a query template Run a query template with the `POST /v2/{ledger}/queries/{id}/run` endpoint: The `schemaVersion` query parameter is required and tells the ledger which schema version contains the query template. The request body accepts: | Field | Description | | --- | --- | | `vars` | Values for the declared variables in the query template | | `params` | Override the template's default pagination and sorting | | `cursor` | Pagination cursor for fetching subsequent pages | The response is a standard cursor response with an additional `resource` field indicating the type of results returned: ```json { "resource": "transactions", "cursor": { "hasMore": true, "next": "...", "pageSize": 25, "data": [...] } } ``` ### Variable substitution Variables declared in `vars` are substituted into the filter `body` at runtime. Reference variables using `:varName:` syntax in the filter expression: ```json { "vars": { "src": { "type": "account" }, "dst": { "type": "account" } }, "body": { "$and": [ { "$match": { "source": ":src:" } }, { "$match": { "destination": ":dst:" } } ] } } ``` When running the query, pass the variable values: ```json { "vars": { "src": "world", "dst": "users:alice" } } ``` ## Example: Payment platform A complete schema for a payment platform: ```json { "chart": { "world": {}, "platform": { ".self": {}, "fees": {}, "float": {} }, "merchants": { "$merchantId": { ".pattern": "^mch_[a-zA-Z0-9]{16}$", ".self": {}, ".metadata": { "type": { "default": "merchant" } }, "pending": {}, "available": {} } }, "customers": { "$customerId": { ".pattern": "^cus_[a-zA-Z0-9]{16}$", ".metadata": { "type": { "default": "customer" } }, "wallet": {} } }, "orders": { "$orderId": { ".pattern": "^ord_[a-zA-Z0-9]{16}$", "capture": {}, "refunds": { "$refundId": { ".pattern": "^ref_[a-zA-Z0-9]{16}$" } } } } }, "transactions": {}, "queries": {} } ``` **Valid accounts:** - `platform:fees` - `merchants:mch_abc123def456ghij:available` - `customers:cus_xyz789abc123defg:wallet` - `orders:ord_123abc456def789g:refunds:ref_abc123def456ghij` **Rejected accounts:** - `merchants:acme` — doesn't match `mch_` prefix pattern - `customers:cus_abc:savings` — `savings` not defined in chart - `payments:xyz` — `payments` not in chart --- ## Open Banking Source: https://docs.formance.com/modules/payments/connectors/open-banking Open Banking connectors link Formance to open-banking aggregators, enabling end users to securely connect their bank accounts and share account data and balances with your product. All three connectors ship in the Community edition. ## Supported Providers - **[Plaid](/modules/payments/connectors/open-banking/plaid)** - **[Tink](/modules/payments/connectors/open-banking/tink)** - **[Powens](/modules/payments/connectors/open-banking/powens)** ## Key Concepts - **[Payment Service Users (PSUs)](/modules/payments/payment-service-users)** — represent end-users who connect their bank accounts - **Connections** — link a PSU to their bank account(s) through a provider - **Webview** — the provider's frontend interface where users authenticate with their bank --- ## Payments Source: https://docs.formance.com/modules/payments/payments A **payment** in the Payments module is a single-asset balance movement on a single account leg, surfaced by a connector after the upstream provider returns it. Deposits, withdrawals, internal transfers, settlement events, refunds, fee payments — they all land here. Exchange-flavoured flows that move two assets atomically (stablecoin redemption, FX swaps) live on [Conversions](/modules/payments/conversions), and trading orders against an order book live on [Orders](/modules/payments/orders). ## Payment model | Field | Description | | --- | --- | | `id` | Platform-assigned UUID, derived from `(connectorID, reference, type)`. | | `reference` | The provider's payment ID. Unique within the connector. | | `connectorID`, `provider` | The connector this payment was fetched from. | | `createdAt` | Provider-side creation date. | | `type` | `PAY-IN`, `PAYOUT`, `TRANSFER`, or `OTHER`. | | `status` | `PENDING`, `SUCCEEDED`, `FAILED`, `CANCELLED`, `EXPIRED`, `OTHER`, `UNKNOWN`. | | `amount`, `initialAmount` | Integer amounts at the asset's precision. Initial = first-seen value before adjustments; amount = current. | | `asset` | UMN asset code (`USD/2`, `EUR/2`, `BTC/8`). | | `scheme` | Payment rail — `SEPA`, `ACH`, `CARD_VISA`, `OTHER` (crypto rails always collapse to `OTHER`). | | `sourceAccountID`, `destinationAccountID` | Resolved Payments module account IDs when the upstream payload identifies wallets, otherwise null. External addresses (e.g. crypto withdrawals) surface in metadata instead. | | `metadata` | Connector-specific keys under `com..spec/` or `com.formance.connectors..`. | | `adjustments` | Append-only list of every observed state change — the audit trail. | | `raw` | Full upstream payload, available on the detail endpoint (not inlined in the list response for payload-size reasons). | ## Listing payments ### Formance Console Open the **Payments** tab in the sidebar. The Payments view lists every payment synced from every installed connector with filters for connector, status, type, scheme, asset, and date range. Click a row for the full payment detail, including the adjustment history and the upstream raw payload. ### API `v3ListPayments` accepts a `$match` query body for server-side filtering — significantly more efficient than fetching the full set and filtering client-side, especially against large connectors: ```bash curl -s "$STACK/api/payments/v3/payments?pageSize=15" \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{"$match": {"connectorID": "'$CONNECTOR_ID'", "status": "SUCCEEDED"}}' | jq ``` A single payment by id: " }} /> The detail endpoint inlines `adjustments` and `raw`, neither of which appear in the list response. ## Querying by reference or metadata `v3GetPayment` only accepts the platform-assigned `id`. To find a payment by the provider's reference, or by a `com..spec/` metadata key, use the list endpoint with a `$match` body: ### By reference ```json { "$match": { "reference": "tx_4f3a8e9d1c" } } ``` ### By metadata key ```json { "$match": { "metadata[com.routable.spec/payment_initiation_reference]": "payout-acmecorp-20260506-172725" } } ``` Querying with `$match` is significantly more efficient than fetching all payments and filtering client-side. It reduces data transfer, reduces server load, and avoids 504 timeouts on large connectors. ## Payment types The `type` field collapses the provider's lifecycle vocabulary into four buckets: | Type | Meaning | | --- | --- | | `PAY-IN` | Funds arriving at an internal account (deposits, customer payments, settlement credits, rewards). | | `PAYOUT` | Funds leaving an internal account to an external destination (bank withdrawals, vendor payouts). | | `TRANSFER` | Funds moving between two internal accounts at the same provider (sweep, sub-account routing). | | `OTHER` | Provider-specific event that doesn't fit the above (chain-level events on Coinbase Prime, staking transitions, fees). | The per-connector reference pages (Stripe, Mangopay, Coinbase Prime, …) document the mapping from each provider's upstream event vocabulary to the four-type model. ## Status mapping The `status` field collapses each provider's status enum into a uniform Payments vocabulary: - `PENDING` — created, accepted, in-flight at the provider. - `SUCCEEDED` — terminal success. - `FAILED` — terminal failure. - `CANCELLED` — user-cancelled or merchant-voided. - `EXPIRED` — provider expired the payment before it settled. - `OTHER` — provider-specific non-terminal status the connector chose not to collapse. - `UNKNOWN` — provider returned a status the connector doesn't recognize. The exact upstream-status → Payments-status mapping is documented on each connector's reference page. Coinbase Prime's table is in [Coinbase Prime → Payments](/modules/payments/connectors/exchange/coinbaseprime#status-mapping); the same pattern applies to every other connector. ## Adjustments Every observed state change lands on the payment's adjustment list, including the new status, the timestamp, and (when the connector kept it) the raw upstream payload that triggered the change. Use it as the audit trail for back-office UIs and reconciliation runs. Adjustments aren't inlined in `v3ListPayments` — fetch a payment by id with `v3GetPayment` to see them. ## Payment reference enrichment Some connectors enrich payment metadata with PSP-specific reference fields they extract from upstream payloads — end-to-end IDs, mandate IDs, clearing-system references, creditor reference info, and similar. These land under the `com.formance.connectors..` metadata namespace. The Banking Bridge connector applies this pattern most extensively. See [Operations → Connector reliability](/modules/payments/operations) for the cross-cutting behavior. --- ## Orders Source: https://docs.formance.com/modules/payments/orders Orders capture exchange trading activity — instructions to swap one asset for another at a price. Payments exposes them as their own resource, distinct from [Payments](/modules/payments/payments) (which move a single asset on a single leg) and [Conversions](/modules/payments/conversions) (atomic two-asset swaps that don't go through an order book). --- ## Conversions Source: https://docs.formance.com/modules/payments/conversions Conversions capture atomic two-asset swaps — moving a balance from one asset to another at a venue, with distinct source and destination amounts. Stablecoin redemption (USDC ↔ USD), wrapped-asset swaps, and any other direct asset-to-asset exchange that bypasses the order book belong here, separate from both [Payments](/modules/payments/payments) and [Orders](/modules/payments/orders). --- ## Events Publishers Source: https://docs.formance.com/modules/ledger/advanced/events-publishers ## Events The following event types are available: ## Publishing to HTTP An HTTP publisher is available out of the box since 1.3. It allows you to push ledger events to an HTTP server of your choice, in a webhook-like fashion. It can be enabled by setting the [configuration variables](/deploy/self-hosted/config/settings) `PUBLISHER_HTTP_ENABLED` and `PUBLISHER_TOPIC_MAPPING`. For HTTP publishing on Formance Cloud, refer to the [Webhooks module](/modules/webhooks). The event payload is formatted as follows: ## Publishing to Kafka An integration with Kafka is available out of the box since 1.3. It allows you to push ledger events to a Kafka broker of your choice, bringing easy and reliable extensibility. It can be enabled by setting the [configuration variables](/deploy/self-hosted/config/settings) `PUBLISHER_KAFKA_ENABLED`, `PUBLISHER_KAFKA_BROKER` and `PUBLISHER_TOPIC_MAPPING`. Other [variables](/deploy/self-hosted/config/settings) are available for further authentication and configuration. Access to Kafka publishing is not yet available in Formance Cloud. The event payload format is the same as for HTTP publishing: --- ## Build a connector Source: https://docs.formance.com/modules/payments/connectors/build-connector ## Goal This page is the contributor guide for adding a new connector to [`formancehq/payments`](https://github.com/formancehq/payments). If your team uses a payment service provider Formance does not integrate with, the project welcomes a pull request that wires it in. Merged connectors ship in the Community edition binary. The platform side stays generic: the framework schedules the connector's fetchers and routes initiation workflows. The contributor writes the provider-specific glue — auth, HTTP plumbing, format conversion, and metadata mapping. ## Scope This guide covers **CE PSP connectors** — Community edition payment service providers that fetch accounts, balances, payments, and (where the upstream supports it) initiate transfers and payouts. Stripe, Bankingcircle, Mangopay, Wise are the kind of integrations to follow. Out of scope here: - **Enterprise connectors.** EE plugins live under `ee/plugins/` and ship in a separate binary; the source isn't open and external contributions don't apply. - **Open Banking aggregators.** Plaid, Tink, and Powens implement an additional surface for end-user link sessions and consent renewals on top of the PSP plugin contract. Their pattern is involved enough to warrant a dedicated guide and is not covered here. ## What you'll add to the repo A new folder at `internal/connectors/plugins/public//` with: - A plugin entry point that registers the connector at process start. - A `Config` struct describing the install-time fields readers fill in. - A workflow declaration listing the periodic fetch tasks. - A `[]models.Capability` slice declaring what the plugin supports. - One Go file per capability (`accounts.go`, `payments.go`, `webhooks.go`, …) implementing the `PSPPlugin` methods you opted into. - Table-driven unit tests next to each file. - A `MAPPING.md` cataloguing the upstream-to-Formance field mapping. When the PR merges, the connector appears on the [Capabilities](/modules/payments/capabilities) matrix on the next docs build. Upstream regenerates `docs/other/connector-capabilities.json` on every commit, and the docs site reads that file at build time. ## Where the code lives ``` formancehq/payments └── internal/ ├── connectors/plugins/ │ ├── public/ # CE connectors — your new folder goes here │ │ ├── stripe/ │ │ ├── bankingcircle/ │ │ ├── mangopay/ │ │ └── … │ ├── registry/ # Plugin registration │ ├── sharedconfig/ # Re-used config primitives (PollingPeriod, …) │ ├── base_plugin.go # Embedded zero-impl base │ └── plugin.go # Plugin top-level type └── models/ ├── plugin.go # Plugin interface ├── plugin_psp.go # PSPPlugin sub-interface └── capabilities.go # CAPABILITY_* constants ``` Every connector folder is named after the lowercase provider id (`stripe`, `bankingcircle`, `mangopay`). ## The Plugin interface The methods you implement are on `PSPPlugin` plus a small set of lifecycle hooks. The full surface is in [`internal/models/plugin_psp.go`](https://github.com/formancehq/payments/blob/main/internal/models/plugin_psp.go): ```go type PSPPlugin interface { FetchNextAccounts(ctx, FetchNextAccountsRequest) (FetchNextAccountsResponse, error) FetchNextPayments(ctx, FetchNextPaymentsRequest) (FetchNextPaymentsResponse, error) FetchNextBalances(ctx, FetchNextBalancesRequest) (FetchNextBalancesResponse, error) FetchNextExternalAccounts(ctx, …) (…) FetchNextOthers(ctx, …) (…) FetchNextOrders(ctx, …) (…) // Payments 3.3.0+ FetchNextConversions(ctx, …) (…) // Payments 3.3.0+ CreateBankAccount(ctx, …) (…) CreateTransfer(ctx, …) (…) ReverseTransfer(ctx, …) (…) PollTransferStatus(ctx, …) (…) CreatePayout(ctx, …) (…) ReversePayout(ctx, …) (…) PollPayoutStatus(ctx, …) (…) } ``` Plus the lifecycle methods on the parent `Plugin` interface: ```go Name() string Config() PluginInternalConfig Install(context.Context, InstallRequest) (InstallResponse, error) Uninstall(context.Context, UninstallRequest) (UninstallResponse, error) CreateWebhooks(context.Context, CreateWebhooksRequest) (CreateWebhooksResponse, error) TrimWebhook(context.Context, TrimWebhookRequest) (TrimWebhookResponse, error) VerifyWebhook(context.Context, VerifyWebhookRequest) (VerifyWebhookResponse, error) TranslateWebhook(context.Context, TranslateWebhookRequest) (TranslateWebhookResponse, error) ``` You only implement what your provider supports. The [`basePlugin`](https://github.com/formancehq/payments/blob/main/internal/connectors/plugins/base_plugin.go) type returns `ErrNotImplemented` for every method by default — embed it in your struct and override the methods that match the capabilities you'll declare. The platform reads the `[]models.Capability` slice you register at startup and only routes calls to the methods you opted into; methods you didn't override are never invoked. ## Anatomy of a connector folder Stripe is a good reference for a fully-featured CE PSP — accounts, balances, external accounts, payments, transfer + payout initiation, and webhooks. Its folder: ```text internal/connectors/plugins/public/stripe/ ├── plugin.go # struct + New + init() registration ├── config.go # Config struct + JSON unmarshal/validate ├── workflow.go # ConnectorTasksTree declaration ├── capabilities.go # []models.Capability slice ├── accounts.go # FetchNextAccounts implementation ├── balances.go ├── external_accounts.go ├── payments.go ├── webhooks.go # Create/Translate/Trim/Verify ├── client/ # HTTP wrapper around the upstream API └── *_test.go # Per-file table-driven unit tests ``` ### `plugin.go` — entry point and registration ```go const ProviderName = "stripe" func init() { registry.RegisterPlugin( ProviderName, models.PluginTypePSP, func(_ models.ConnectorID, name string, logger logging.Logger, rm json.RawMessage) (models.Plugin, error) { return New(name, logger, rm, nil) }, capabilities, Config{}, PAGE_SIZE, ) } type Plugin struct { models.Plugin // embed the base for the zero-impl methods name string logger logging.Logger client client.Client config Config } func New(name string, logger logging.Logger, rawConfig json.RawMessage, backend stripe.Backend) (*Plugin, error) { config, err := unmarshalAndValidateConfig(rawConfig) if err != nil { return nil, err } c, err := client.New(ProviderName, logger, backend, config.APIKey) if err != nil { return nil, err } return &Plugin{ Plugin: plugins.NewBasePlugin(), name: name, logger: logger, client: c, config: config, }, nil } func (p *Plugin) Name() string { return p.name } func (p *Plugin) Config() models.PluginInternalConfig { return p.config } func (p *Plugin) Install(_ context.Context, _ models.InstallRequest) (models.InstallResponse, error) { return models.InstallResponse{Workflow: workflow()}, nil } ``` `init()` runs at process start. The linker pulls every package under `public/` in via a blanket import in [`registry/plugins.go`](https://github.com/formancehq/payments/blob/main/internal/connectors/plugins/registry/plugins.go), so adding a new connector folder is enough — no central list to edit. `Install` returns the connector's **workflow tree** — see below. ### `config.go` — typed config and validation Define a `Config` struct with `json` tags and `validate` tags (go-playground validator). The registry reads the tag set to extract the field list for the OpenAPI spec, so a new field appears in the public docs on the next regeneration without a docs-side edit. ```go type Config struct { APIKey string `json:"apiKey" validate:"required"` PollingPeriod sharedconfig.PollingPeriod `json:"pollingPeriod"` } const PAGE_SIZE = 100 // upstream cap func unmarshalAndValidateConfig(payload json.RawMessage) (Config, error) { // … } ``` Re-use [`sharedconfig.PollingPeriod`](https://github.com/formancehq/payments/blob/main/internal/connectors/plugins/sharedconfig) for the polling cadence so the floor (`MinimumPollingPeriod`) stays uniform across connectors. Pick the same minimum every other connector uses (`20m`) unless your provider rate-limits at a coarser cadence. ### `workflow.go` — task tree The workflow declares which capabilities run **periodically** at install time and how they nest: ```go func workflow() models.ConnectorTasksTree { return []models.ConnectorTaskTree{ { TaskType: models.TASK_FETCH_ACCOUNTS, Name: "fetch_accounts", Periodically: true, NextTasks: []models.ConnectorTaskTree{ { TaskType: models.TASK_FETCH_BALANCES, Periodically: false }, { TaskType: models.TASK_FETCH_PAYMENTS, Periodically: true }, { TaskType: models.TASK_FETCH_EXTERNAL_ACCOUNTS, Periodically: true }, }, }, { TaskType: models.TASK_CREATE_WEBHOOKS, Periodically: false }, } } ``` Nested non-periodic tasks (here `fetch_balances`) get triggered by their parent at the same cadence — useful when the parent already pulled the data the child needs. ### `capabilities.go` — declared capability set ```go var capabilities = []models.Capability{ models.CAPABILITY_FETCH_ACCOUNTS, models.CAPABILITY_FETCH_BALANCES, models.CAPABILITY_FETCH_EXTERNAL_ACCOUNTS, models.CAPABILITY_FETCH_PAYMENTS, models.CAPABILITY_CREATE_TRANSFER, models.CAPABILITY_CREATE_PAYOUT, models.CAPABILITY_CREATE_WEBHOOKS, models.CAPABILITY_TRANSLATE_WEBHOOKS, } ``` This slice is the source of truth for the [Capabilities matrix](/modules/payments/capabilities) on the docs site — upstream serializes it into [`docs/other/connector-capabilities.json`](https://github.com/formancehq/payments/blob/main/docs/other/connector-capabilities.json) on every commit, and the docs build reads that file. Declare only what you'll actually implement. ## Implementing a capability Each `FetchNext*` method takes a `FromPayload`, an opaque `State`, and a `PageSize`, and returns a slice of `PSP*` objects plus the next `State` and a `HasMore` flag. The platform persists `State` between cycles, so the connector keeps no in-memory cursors: ```go func (p *Plugin) FetchNextAccounts(ctx context.Context, req models.FetchNextAccountsRequest) (models.FetchNextAccountsResponse, error) { if p.client == nil { return models.FetchNextAccountsResponse{}, plugins.ErrNotYetInstalled } return p.fetchNextAccounts(ctx, req) } ``` The body lives in `accounts.go`: ```go type accountsState struct { LastSeenAt time.Time `json:"lastSeenAt"` } func (p *Plugin) fetchNextAccounts(ctx context.Context, req models.FetchNextAccountsRequest) (models.FetchNextAccountsResponse, error) { var state accountsState if len(req.State) > 0 { if err := json.Unmarshal(req.State, &state); err != nil { return models.FetchNextAccountsResponse{}, err } } page, hasMore, err := p.client.ListAccounts(ctx, state.LastSeenAt, req.PageSize) if err != nil { return models.FetchNextAccountsResponse{}, err } out := make([]models.PSPAccount, 0, len(page)) for _, a := range page { out = append(out, models.PSPAccount{ Reference: a.ID, CreatedAt: a.CreatedAt, Name: &a.Name, DefaultAsset: pointer.For(formatAsset(a.Currency)), Metadata: map[string]string{ "com.stripe.spec/type": a.Type, }, Raw: json.RawMessage(a.Raw), }) } if len(out) > 0 { state.LastSeenAt = out[len(out)-1].CreatedAt } nextState, _ := json.Marshal(state) return models.FetchNextAccountsResponse{Accounts: out, NewState: nextState, HasMore: hasMore}, nil } ``` Key invariants: - **The cursor lives in `State`.** The plugin reads it, the platform persists whatever you return as `NewState`. Crash mid-cycle and the worker resumes from the last persisted checkpoint. - **`HasMore=true`** tells the platform to call you again immediately for the next page; `HasMore=false` ends the cycle and the schedule re-triggers after `PollingPeriod`. - **Metadata goes under a provider-specific namespace** — `com..spec/` — so downstream consumers can find it without colliding with other connectors. - **Raw upstream payloads land on `.Raw`** so consumers needing fidelity beyond the typed surface can reach for them. Available on every `PSP*` resource. `CreateTransfer` and `CreatePayout` follow the same request/response pattern but synchronously return either a `Payment` (when the upstream settles immediately) or a `PollingTransferID` — in the latter case the platform schedules `PollTransferStatus` until a terminal `Payment` surfaces. ## Webhooks If the upstream provider supports webhooks, implement the four-method contract: - `CreateWebhooks` — POST a webhook config upstream pointing at the platform's ingress URL. Persist the upstream config IDs in `CreateWebhooksResponse.Configs` so the platform can delete them on uninstall. - `TrimWebhook` — split an upstream batched-events payload into one `PSPWebhook` per event. - `VerifyWebhook` — verify the upstream signature and return the idempotency key the platform should use to dedupe replays. - `TranslateWebhook` — convert a verified upstream payload into a `WebhookResponse` carrying the affected `PSPAccount` / `PSPPayment` / `PSPBalance`. Declare both `CAPABILITY_CREATE_WEBHOOKS` and `CAPABILITY_TRANSLATE_WEBHOOKS` in `capabilities.go` so the platform schedules the create-webhook task at install and routes inbound webhooks back to your plugin. ## Configuration in the OpenAPI spec The registry reads the `Config` struct's reflection metadata and emits the field list into [`openapi/v3/v3-connectors-config.yaml`](https://github.com/formancehq/payments/blob/main/openapi/v3/v3-connectors-config.yaml) during the upstream OpenAPI regeneration. The docs build reads that file via the `payments.connectors..fields` data key, and the per-connector page renders the table from it. A new field upstream surfaces in the public docs on the next content build with no docs-side edit. The `validate:"required"` tag is what the docs surface as the `Required: yes` column. Omit it for optional fields. ## Testing Each connector ships with table-driven unit tests next to each capability file (`accounts_test.go`, `payments_test.go`, …) using `mockgen`-generated mocks of the HTTP client. See [`stripe/accounts_test.go`](https://github.com/formancehq/payments/blob/main/internal/connectors/plugins/public/stripe/accounts_test.go) for the pattern. For end-to-end smoke testing against the real upstream API, point the [connector workbench](https://github.com/formancehq/payments/tree/main/cmd/connectors/workbench) at your plugin and a sandbox key — it boots a single-plugin Temporal worker and runs the install workflow without the full platform stack. ## Mapping documentation Every connector carries a `MAPPING.md` next to the code that catalogues: - Upstream resource → Formance resource field mapping. - Status enum collapses (upstream values → Payments status). - `com..spec/` metadata key inventory. - Known gaps and not-yet-implemented capabilities. The mapping doc is the reference both for connector reviewers and for the docs writer who turns it into the per-connector page at `apps/docs/content/pages/modules/payments/connectors/psp/.mdx` in the [docs repo](https://github.com/formancehq/docs). ## CE references to read alongside this guide The shipped CE connectors are the best living references — pick one with a capability set close to your target: - [`stripe/`](https://github.com/formancehq/payments/tree/main/internal/connectors/plugins/public/stripe) — full set: accounts, balances, external accounts, payments, transfer + payout initiation, webhooks. - [`bankingcircle/`](https://github.com/formancehq/payments/tree/main/internal/connectors/plugins/public/bankingcircle) — bank-rails connector with bank-account creation, transfers, payouts; mTLS + OAuth2 auth. - [`mangopay/`](https://github.com/formancehq/payments/tree/main/internal/connectors/plugins/public/mangopay) — multi-currency e-wallet provider with the full read + write surface and webhook ingest. - [`wise/`](https://github.com/formancehq/payments/tree/main/internal/connectors/plugins/public/wise) — multi-profile multi-currency, quote-then-fund payout flow. - [`atlar/`](https://github.com/formancehq/payments/tree/main/internal/connectors/plugins/public/atlar) — read-only bank-rails connector (smaller surface, useful as a starting point if your provider is observation-only). When in doubt, copy the closest match into a new folder, rename, and adapt — the framework is consistent across connectors so the boilerplate ports cleanly. --- ## Performance model Source: https://docs.formance.com/modules/ledger/advanced/performance-model When building an application on top of Formance Ledger that is built for scale, it is important to understand the performance characteristics of the ledger. This document summarizes the cost model for each type of ledger operation. ## Costs breakdown | Request pattern (Ledger v1) | Complexity model | | --- | --- | | Transaction commit | O(N) + W | | Aggregation • balances | O(Log(N)*M) | | Point query • accounts | O(1) | | Range query • accounts | O(Log(N)) | | Point query • transactions | O(1) | | Range query • transactions | O(Log(N)) | The complexity model is based on the in-process cost, and doesn't take into account the networking cost. ## Data volume In the [architecting for scale](/modules/ledger/advanced/architecting-for-scale) section, we outline different strategies to handle a growing volume of ledger data, with a focus on efficient usage of multiple ledgers. ## Large Integer Values Formance supports storing very large integer values, which is particularly useful when working with cryptocurrencies or other high-precision numerical data. ### Storage Capacity The Formance Ledger can store integers up to **131,072 digits** in length. This is more than sufficient for handling cryptocurrency amounts like ETH or other use cases requiring high numerical precision. ### JavaScript Integration Considerations While the Formance API and database can handle very large integers, you may encounter display or processing limitations when working with these numbers in JavaScript-based applications or tools. This is due to JavaScript's inherent number precision limitations. If you're building integrations that handle large integers, you should either: - Use `BigInt` in JavaScript when available - Process the numbers as strings ### API String Representation To handle large integers in various tools and SDKs, Formance provides an option to receive large integer values as strings instead of numbers in the API responses, using the `Formance-Bigint-As-String: true` request header. The underlying data storage remains the same — this is purely a presentation layer option to ensure compatibility with different tools and frameworks. ## Benchmarking Formance Ledger releases are tested against our load testing [suite](https://github.com/formancehq/ledger/tree/main/test/performance). The suite can be run locally by following the instructions in the repository. If you venture beyond common patterns and ledger architectures, you can customize the suite to fit your use case and push the ledger into its least favorable conditions cost-wise. --- ## Verifying integrity Source: https://docs.formance.com/modules/ledger/advanced/verifying-integrity With `HASH_LOGS` enabled, the Ledger is tamper-evident: it maintains a hash chain over its history that a change made outside the Ledger will break. Nothing checks that chain on its own, so detection is a job you run. This page covers how to run it in each `HASH_LOGS` mode, and why a chain check alone does not prove that history is unchanged. How you run it depends on which `HASH_LOGS` mode the ledger uses — `SYNC` and `ASYNC` build **different structures**, so they give you different things to verify against. Everything here requires `HASH_LOGS` set to `SYNC` or `ASYNC`. A ledger created with `HASH_LOGS: DISABLED` has no hash chain, so there is nothing to recompute and no way to detect tampering from the ledger's own data — [reconciliation](/modules/reconciliation) against outside records is the only check available to it. Features are immutable after creation, so this cannot be retrofitted. Note that importing a hash-less export into a `SYNC` ledger is **not** a way around it. The target compares the hash it computes against the hash carried in the export; with nothing there to compare, every log fails with `ErrInvalidHash` even though nothing was tampered with. The same is true of an export taken from an `ASYNC` ledger. ## What each mode gives you to verify `SYNC` and `ASYNC` are not two speeds of the same mechanism. | | `SYNC` | `ASYNC` | | --- | --- | --- | | Where the hash lives | `hash` column on every log | separate `logs_blocks` table | | What is chained | each log to the previous log | each block to the previous block | | Granularity of a mismatch | the exact log | the block's log range | | Built by | a trigger, at insert time | the ledger worker, on a schedule | | Write cost | a per-ledger advisory lock serializes inserts | none on the write path | | Verifiable through the API | yes, via import | no, requires database access | With `SYNC`, a trigger computes each log's SHA-256 over the previous log's hash plus the current payload, so every entry is chained to the one before it. The insert takes a per-ledger advisory lock — that lock, not the hashing, is what limits write throughput. With `ASYNC`, logs are inserted **without a hash**. A background worker later groups them into blocks and writes one hash per block into `logs_blocks`, covering a range of log IDs and chaining to the previous block's hash. Nothing is on the write path, so throughput is much higher, and mismatches localize to a block rather than to a single log. Because `ASYNC` never populates the per-log `hash` column, the export-and-import check below **does not work on an `ASYNC` ledger**. Choose the mode with the verification workflow you want in mind — `HASH_LOGS` cannot be changed after the ledger is created. See [Data isolation with buckets](/modules/ledger/working-with/data-isolation-buckets#hash_logs) for how to set it, and [the self-hosted deployment guide](/deploy/self-hosted/modules/ledger#async-block-hasher) for the worker configuration `ASYNC` requires. ## What the chain proves, and what it doesn't The chain only covers changes made to the database outside the Ledger. There are two ways that happens, and they need different checks. **Someone edits a row and leaves the hashes after it alone.** Recomputing the chain catches this, and tells you which log broke. This is the accident and the careless insider — an `UPDATE` run against the database, and every hash after it stops matching. The checks below cover it. **Someone edits a row and recomputes every hash or block after it.** Recomputing the chain does not catch this. The chain is internally consistent, and every value you compare it against came out of the same database. A check sourced entirely from the database it is checking proves consistency, not authenticity. To cover the second case, keep an anchor outside an attacker's reach. **Anchor hashes outside the database.** On a schedule, copy the newest hash — the last log's `hash` under `SYNC`, the newest block's `(to_id, hash)` under `ASYNC` — somewhere the database operator cannot rewrite: a write-once bucket, an append-only store, or a signed copy sent to an auditor. A recomputed chain is then checked against a value that did not come from the database. Do this in both modes. An anchor also saves work later. You can trust history up to that point without rechecking everything before it. ## Detecting tampering is a job you schedule Here is how to run the recomputation in each mode. Neither belongs on the write path. ### For `SYNC` ledgers: export and re-import The Ledger's import path already recomputes and verifies a per-log chain, so you do not have to implement the check yourself. It runs entirely through the API. Export every log from the ledger you want to verify. The ledger name comes from the `--ledger` flag, and the stream goes to stdout unless you pass `--file`. ```bash fctl ledger export --ledger=my-ledger > my-ledger.export ``` The equivalent API call is `POST /v2/{ledger}/logs/export`, which returns the logs as an `application/octet-stream`. The verification happens on the **target** side, and only when the target has `HASH_LOGS` set to `SYNC`. An `ASYNC` or `DISABLED` target skips the hash comparison entirely and the import will succeed without telling you anything. ```bash fctl ledger create my-ledger-verify --features HASH_LOGS=SYNC ``` Unspecified features keep their defaults, so this only pins `HASH_LOGS`. Import takes the ledger name and the file path as positional arguments. ```bash fctl ledger import my-ledger-verify my-ledger.export ``` The equivalent API call is `POST /v2/{ledger}/logs/import`. If a large import is interrupted, `fctl ledger import --resume-from-last-log` picks up from the last log the target accepted rather than starting over. For each log, the target recomputes the hash from its own chain and compares it against the hash carried in the export. On a mismatch the import stops with an `ErrInvalidHash` naming the expected hash, the hash it got, and the log ID — which is the exact point where the history diverges. A clean import means the chain is internally consistent. To make it mean *unchanged*, compare the last log's hash against your external anchor as a final step; the export carries the source's own hashes, so on its own this check cannot detect a chain that was rewritten wholesale. Two import constraints affect how you run this: - **Log IDs must be strictly monotonically increasing.** Import rejects any log whose ID is less than or equal to the highest ID already in the target, with `log already exists`. Import into a ledger created for the check, not one already holding data. - **A concurrent write aborts the import.** Each log is imported in its own transaction, and a serialization conflict fails the import with `concurrent transaction occurred, cannot import the ledger` rather than producing a half-verified result. Nothing else should be writing to the target while the check runs. The cost of this check scales with the size of the export, so on a large ledger, run full passes on a schedule rather than continuously. ### For `ASYNC` ledgers: verify the block chain An `ASYNC` ledger has no per-log hashes to compare, so there is nothing for import to check. What it has instead is `logs_blocks`, where each row covers a log range and chains to the row before it. Verifying means recomputing a block's hash from the logs in its range and confirming it reproduces the stored value, then confirming each block's `previous` reference lines up with the block before it. `logs_blocks` is not exposed through the Ledger API today. Verifying an `ASYNC` ledger's chain means querying the bucket schema directly, which makes it an operator task rather than something an application can do for itself. If API-level verification matters to you, that is an argument for `SYNC`. Because blocks are already checkpoints, you do not have to re-verify all of history on every pass. Verify only the blocks added since your last run, and re-verify older ranges on a rolling schedule. A mismatch narrows the tampering to that block's log range, so the block size you configure on the worker trades checkpoint density against how precisely you can localize a problem. The same limit applies as with `SYNC`: recomputing blocks from the same database proves the block chain is self-consistent, and only a block hash you anchored externally tells you it has not been rebuilt. ### Reconcile against source-of-truth systems Hash verification is a check on the shape of the history, not on its contents — it does not tell you the numbers are correct. Comparing ledger balances against outside records — bank statements, PSP reports — is what surfaces real errors and fraud. [Reconciliation](/modules/reconciliation) covers that check. It also covers the case the hash chain structurally cannot: a rewritten-but-consistent history still has to reconcile against records held in someone else's system. Along with external anchoring, this is the reason reconciliation is not optional. ## Recovering from a detected problem When you find tampering, do not edit the history to repair it. Rewriting past entries destroys the property that let you detect the problem in the first place. The check tells you where the chain diverges — the exact log under `SYNC`, the block's range under `ASYNC` — so you know how far back you have to go. You have two ways to rebuild a clean ledger: - **Restore the database.** Restore Postgres to a point before the divergence, then replay the legitimate activity that occurred after it. - **Rebuild from a clean export.** Import the clean prefix of an export, up to but not including the divergence, into a fresh ledger, then re-post the legitimate transactions that came after. Once the chain is continuous again, correct any resulting balances by appending compensating transactions, which is the Ledger's built-in way to reverse and adjust while keeping a complete record of what happened. ## Where to start Pick the mode for the verification you actually intend to run, since you cannot change it later: - **`SYNC`** when you want verification you can drive entirely through the API, and per-log precision when something is wrong. Accept the per-ledger insert lock. - **`ASYNC`** when write throughput rules the decision. Budget for an operator-side job against `logs_blocks`, and for localizing a mismatch to a block rather than a log. Then run all three checks, because each one covers what the others cannot: recompute the chain on a schedule to catch an edit that left the hashes after it alone, anchor hashes outside the database to catch one that recomputed them, and reconcile against source-of-truth systems to catch numbers that are wrong regardless of what the chain says. --- ## Numscript Source: https://docs.formance.com/modules/numscript Numscript is a Domain-Specific Language (DSL) designed to help you model complex financial transactions, replacing complex and error-prone custom code with easy-to-read, declarative scripts. ## Financial transactions We define a financial transaction as a series of discrete value movements between abstract accounts. Each movement represents transfer of value from one account to another, with an associated amount and asset denomination. Assets being transferred can represent any kind of value, from traditional currencies like USD or JPY to custom tokens or commodities. Accounts involved in a transaction can represent anything, from a bank account to a voucher, a virtual wallet or an order that has yet to be paid out. You can read more about the structure of programs and transactions [here](/modules/numscript/program-structure). ## Design principles ### Readability The intent of a Numscript program should always be clear and easy to understand. Numscript programs should be readable by both developers and non-technical financial users, providing a shared, executable definition of money movements. ### Correctness Monetary computations in Numscript should always yield correct results, avoiding common currency rounding errors and accidental money creation or destruction. Execution is atomic, ensuring that either all modeled transactions are committed or none. ### Finiteness Numscript programs are deterministic, always terminating with a predictable output. This ensures that the behavior of Numscript programs can be reliably predicted and controlled. These principles are the guiding light behind Numscript, and they are reflected in the design of the language itself. By using Numscript, you can model complex financial transactions in a way that is clear, accurate, and predictable. ## Example Here is a simple transaction example of what a Numscript transaction can look like. We use multiple `send` statements, moving USD through a series of accounts, and splitting the final amount between a driver, a charity, and platform fees.
```numscript send [USD/2 599] ( source = @world destination = @payments:001 ) send [USD/2 599] ( source = @payments:001 destination = @rides:0234 ) send [USD/2 599] ( source = @rides:0234 destination = { 85% to @drivers:042 remaining to { 10% to @charity remaining to @platform:fees } } ) ```
Executed by the Numscript interpreter, the above script will result in the following transaction: ```json { "postings": [ { "source": "world", "destination": "payments:001", "amount": 599, "asset": "USD/2" }, { "source": "payments:001", "destination": "rides:0234", "amount": 599, "asset": "USD/2" }, { "source": "rides:0234", "destination": "drivers:042", "amount": 510, "asset": "USD/2" }, { "source": "rides:0234", "destination": "charity", "amount": 9, "asset": "USD/2" }, { "source": "rides:0234", "destination": "platform:fees", "amount": 80, "asset": "USD/2" } ] } ``` A VSCode extension for Numscript is available [here](https://marketplace.visualstudio.com/items?itemName=formance.formance-vscode&ssr=false#overview). --- ## Program Structure Source: https://docs.formance.com/modules/numscript/program-structure The goal of a Numscript program is to output financial transactions, based on the rules and constraints defined in the script, along with an initial state of the system of accounts provided as input. Once a transaction is computed by the Numscript interpreter, it then becomes the role of the system leveraging Numscript to concretize it against the accounts involved. ## Program A Numscript program is composed of the following elements: * Variables declaration * This block is optional and allows you to define variables that can be used in the program, instead of hardcoding values. * Statements * Send statements * These statements are the core of a Numscript program and define the movement of value between accounts. * Metadata statements * These statements are optional and allow you to define metadata for accounts or the transaction itself. The grammar of a Numscript program can be found [here](https://github.com/formancehq/numscript/blob/main/Numscript.g4). --- ## Selecting an Interpreter Source: https://docs.formance.com/modules/numscript/interpreter Since Ledger v2.2, two versions of the Numscript interpreter are available: the original, built directly into our Ledger service, and a new portable and embeddable interpreter that can run independently or enhance other ledgering systems. The original interpreter is enabled by default but is no longer evolving, while the new version will receive ongoing updates and must be manually enabled. To enable the new interpreter: The new interpreter supports additional language features not available in the original, including `oneof`, account interpolation, asset colors, and more. See the [Numscript reference](/modules/numscript/reference) for the full list and how to enable each one. ## Selecting the interpreter per transaction Each transaction picks its interpreter through the `runtime` field on the [create-transaction](/modules/ledger/core-concepts/transactions) request body. - `machine` (default) — the original interpreter built into the Ledger. - `experimental-interpreter` — the new interpreter. **Required for any experimental feature.** Using `runtime: experimental-interpreter` also requires the Ledger to run with the new interpreter enabled (the `ledger.experimental-features` and `ledger.experimental-numscript` settings above), and each experimental feature must additionally be enabled via its own `#![feature(...)]` flag. ## Experimental feature availability Which experimental features you can use depends on the Numscript interpreter version bundled in your Ledger release. The reference pages reflect this automatically — a feature only appears once your selected Stack/Ledger version bundles an interpreter recent enough to support it. | Feature | Flag | Interpreter | Available from | | --- | --- | --- | --- | | [OneOf](/modules/numscript/reference/oneof) | `experimental-oneof` | 0.0.15 | Ledger 2.3 / Stack v3.1 | | [Account interpolation](/modules/numscript/reference/account-interpolation) | `experimental-account-interpolation` | 0.0.15 | Ledger 2.3 / Stack v3.1 | | [Mid-script function calls](/modules/numscript/reference/mid-script-functions) | `experimental-mid-script-function-call` | 0.0.15 | Ledger 2.3 / Stack v3.1 | | [Overdraft function](/modules/numscript/reference/overdraft) | `experimental-overdraft-function` | 0.0.15 | Ledger 2.3 / Stack v3.1 | | [get_asset](/modules/numscript/reference/get-asset) | `experimental-get-asset-function` | 0.0.16 | Ledger 2.3 / Stack v3.1 | | [get_amount](/modules/numscript/reference/get-amount) | `experimental-get-amount-function` | 0.0.16 | Ledger 2.3 / Stack v3.1 | | [Asset colors](/modules/numscript/reference/asset-colors) | `experimental-asset-colors` | 0.0.17 | Ledger 2.3 / Stack v3.1 | For the callable functions specifically (with live per-function availability against your selection), see [Functions](/modules/numscript/reference/functions). The [Numscript playground](https://playground.numscript.org/?template=simple-send) uses the new interpreter. Experimental features can be toggled individually in the playground's feature flags selector. Ledger 3.0 ships a single Numscript interpreter — the portable one — so there is no interpreter to choose: no `runtime` field on a transaction request and no `ledger.experimental-*` server setting to turn on. Every experimental feature is available, and the server restricts none of them. Each script opts in to the ones it uses by declaring a `#![feature("...")]` pragma before the rest of the script, so a feature is enabled per script rather than per request or per cluster. Declaring `#![feature("experimental-oneof")]`, for example, makes `oneof` available to that script alone. | Feature | Pragma | What it does | | --- | --- | --- | | [OneOf](/modules/numscript/reference/oneof) | `experimental-oneof` | Conditional routing based on availability | | [Account interpolation](/modules/numscript/reference/account-interpolation) | `experimental-account-interpolation` | Dynamic addresses such as `@escrow:$order_id` | | [Mid-script function calls](/modules/numscript/reference/mid-script-functions) | `experimental-mid-script-function-call` | Query balances during execution | | [Overdraft function](/modules/numscript/reference/overdraft) | `experimental-overdraft-function` | Dynamic overdraft calculation | | [get_asset](/modules/numscript/reference/get-asset) | `experimental-get-asset-function` | Extract the asset from a monetary value | | [get_amount](/modules/numscript/reference/get-amount) | `experimental-get-amount-function` | Extract the amount from a monetary value | | [Asset colors](/modules/numscript/reference/asset-colors) | `experimental-asset-colors` | Track fund origins with colored assets | To submit a script, see [Numscript transactions](/modules/ledger/transactions/numscript). The [Numscript playground](https://playground.numscript.org/?template=simple-send) runs the same interpreter. Its feature flags selector is a playground control that stands in for the pragmas above; against Ledger 3.0, declare the pragma in the script itself. --- ## Send Source: https://docs.formance.com/modules/numscript/reference/send In Numscript, a [posting](/modules/ledger/core-concepts/transactions#postings) models the movement of an amount of an asset from one account to another. Postings are wrapped in a [transaction](/modules/ledger/core-concepts/transactions) to ensure that they are applied atomically. In Numscript, a [posting](/modules/ledger/transactions/create-with-postings) models the movement of an amount of an asset from one account to another. Postings are wrapped in a [transaction](/modules/ledger/transactions) to ensure that they are applied atomically. ## Defining a posting A posting is defined as follows:
```numscript send [COIN 100] ( source = @world destination = @users:001 ) ```
Square brackets describe the asset being moved using Formance's [Unambiguous Monetary Notation (UMN)](/modules/numscript/monetary-notation). It is composed of the asset type optionally followed by a scaling value, and the amount. In this case, the asset type is `COIN` with no scaling value, and the amount is `100`. ## Postings with balance You can omit the amount of an asset in a posting and replace it with a `*`. This is used when you want to move all of the assets of the accounts in the source block to those in the destination block.
```numscript send [USD/2 *] ( source = @order:1234 destination = { 10% to @platform:fees remaining to @merchant:5678 } ) ```
Here, we are moving all of the `USD/2` asset from the `@order:1234` account. 10% of the `@order:1234` balance is moved to the `@platform:fees` account, and the remaining 90% is moved to the `@merchant:5678` account. --- ## Payment Initiation Source: https://docs.formance.com/modules/payments/transfer-initiation A **payment initiation** is your stack's instruction to a connector to move funds — either between two internal accounts at the provider (`TRANSFER`) or out to an external account such as a bank account (`PAYOUT`). The initiation is a Payments-side record with its own lifecycle that the platform resolves against the connector's `CreateTransfer` / `CreatePayout` workflow, then against the provider's asynchronous status updates, until it links to a settled [`Payment`](/modules/payments/payments). This page documents the v3 surface (`/api/payments/v3/payment-initiations`). The legacy v2 transfer-initiation endpoints are still served for backward compatibility on stacks that pin Payments < 3.0 — they use the same state machine and a near-identical shape. ## Lifecycle A payment initiation moves through six states: ```text WAITING_FOR_VALIDATION ─── approve ──▶ PROCESSING ──▶ PROCESSED │ │ │ ├── failure ──▶ FAILED ──▶ retry ──▶ PROCESSING │ │ └── reject ──▶ REJECTED └── reverse ──▶ REVERSE_PROCESSING ──▶ REVERSED ``` - **WAITING_FOR_VALIDATION** — initial state when the create request passes `validated: false`. The platform parks the initiation; nothing is sent to the connector yet. - **PROCESSING** — either the platform sent the create request straight through (`validated: true` on create, or `approve` on a parked one), or it's retrying a previously failed one. The connector's `CreateTransfer` / `CreatePayout` workflow is in flight. - **PROCESSED** — the connector returned a settled payment. The initiation is linked to a `Payment` via `v3ListPaymentInitiationRelatedPayments`. - **FAILED** — the connector returned a terminal error. The full error body is on the latest [adjustment](#adjustments). Retryable per §Retrying. - **REJECTED** — the initiation was rejected from `WAITING_FOR_VALIDATION`. Terminal. - **REVERSE_PROCESSING** / **REVERSED** — a `v3ReversePaymentInitiation` call is in flight or has been confirmed by the connector. Listing initiations carries the current status; the full transition log is on `v3ListPaymentInitiationAdjustments`. ## Listing payment initiations ## Reading a single initiation " }} /> The `id` is the platform-assigned UUID returned by `v3InitiatePayment`. The `reference` is the caller-supplied idempotency key — typically echoed downstream on the resulting `Payment` so you can resolve one from the other (see [Correlation](#correlation)). ## Creating a payment initiation ", type: "PAYOUT", amount: 50000, asset: "USD/2", sourceAccountID: "", destinationAccountID: "", description: "Vendor payment", metadata: { "invoice_id": "inv_123" }, validated: true }} /> **Required fields**: | Field | Type | Purpose | | --- | --- | --- | | `reference` | string | Caller-supplied unique key. Sent to the connector as the upstream `Idempotency-Key` when the connector supports one; retries with the same reference return the original initiation. | | `connectorID` | string | The connector instance to route through. Find it in the response of `v3ListConnectors` or in the Console under Connectors. | | `type` | enum | `TRANSFER` (internal → internal) or `PAYOUT` (internal → external). | | `amount` | integer | Minor units at the asset's precision. `100` of `USD/2` = $1.00. | | `asset` | string | Asset code with precision suffix (`USD/2`, `EUR/2`, `BTC/8`). Must match the source account's currency. | | `sourceAccountID` | string | Internal account ID at the connector. | | `destinationAccountID` | string | Internal account for `TRANSFER`, external for `PAYOUT`. | **Optional**: - `description` — propagated to the upstream payment when the connector supports a memo / description field. - `metadata` — opaque map. Connectors that accept upstream metadata forward it where they can; otherwise it stays Payments-side only. - `validated` — when `true`, skip the `WAITING_FOR_VALIDATION` step and go straight to `PROCESSING`. Default `false`. - `scheduledAt` — RFC3339 timestamp. Defers the connector call until that wall-clock time (used for cut-off-aware ACH or wire submission windows). ### Connector-specific metadata Some connectors accept extra metadata keys to control rail selection, delivery method, or operator context. These live under `com..spec/` and are documented on the per-connector reference pages. Routable, for example, uses: | Metadata key | Purpose | | --- | --- | | `com.routable.spec/type` | Routable payable rail (`ach`, `wire`, `check`, …). | | `com.routable.spec/delivery_method` | Specific delivery option compatible with `type`. | | `com.routable.spec/acting_team_member` | Routable team member initiating the payable (required by Routable's API). | See the [Routable reference](/modules/payments/connectors/psp/routable#initiating-payouts-and-transfers) for the full list. ## Approving or rejecting When the initiation is created with `validated: false`, the platform parks it in `WAITING_FOR_VALIDATION` until your reviewer decides: " }} /> " }} /> Approve transitions the initiation to `PROCESSING` and dispatches it to the connector. Reject moves it to `REJECTED` terminally — the connector is never called. The approval step is the platform's hook for a four-eyes / RBAC gate. Approval API access can be restricted independently of creation access via [RBAC](/manage/identity/rbac). ## Retrying a failed initiation " }} /> `v3RetryPaymentInitiation` is valid only when the initiation is in `FAILED`. It re-dispatches to the connector with the same `reference`, which means a connector that uses the reference as an upstream idempotency key will either return the failed payment as-is (no retry actually attempted) or accept a fresh attempt — connector-dependent. See each connector's reference page for the per-provider retry semantics. ## Reversing a settled initiation " }} body={{ reference: "rev--001", description: "Reverse mistaken payout", metadata: {} }} /> Available on connectors that declare `ReverseTransfer` / `ReversePayout`. The reversal carries its own reference and goes through `REVERSE_PROCESSING` → `REVERSED` (or `FAILED`). The list of reversal attempts is accessible via the same adjustment endpoint. ## Adjustments Every state transition lands on the initiation's adjustment list: " }} /> Adjustments carry the timestamp, the new status, and the connector's raw response when relevant — including the upstream error body on failed transitions. This is the audit trail you'd surface in a back-office UI. ## Correlation Once the connector returns a settled payment, the initiation links to it in two directions: ```bash curl -s "$STACK/api/payments/v3/payment-initiations/$PI_ID/payments" \ -H "Authorization: Bearer $TOKEN" | jq '.cursor.data[] | {reference, status}' ``` Returned by `v3ListPaymentInitiationRelatedPayments`. There can be more than one — a retry that succeeded leaves both the original failed payment and the successful one linked. Most connectors that initiated a payment carry the originating initiation reference under their `com..spec/` namespace. The exact key varies by connector — Routable uses `com.routable.spec/payment_initiation_reference`, Stripe uses `com.stripe.spec/transfer_initiation_ref`, etc. See the per-connector reference for the canonical key. ```bash curl -s "$STACK/api/payments/v3/payments" \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{"$match": {"connectorID": "'$CONNECTOR_ID'"}}' \ | jq '.cursor.data[] | { reference, pi_ref: .metadata."com.routable.spec/payment_initiation_reference" }' ``` ## Deleting " }} /> Only valid in `WAITING_FOR_VALIDATION`. Once an initiation has been dispatched to the connector, it stays in the platform indefinitely so the linked payment(s) and adjustment log remain queryable. ## Connector support matrix Initiation requires `CAPABILITY_CREATE_TRANSFER` or `CAPABILITY_CREATE_PAYOUT` on the target connector. See [Capabilities](/modules/payments/capabilities) for the live matrix — the columns light up per connector and per Payments version. Reversal is available on connectors that additionally implement `ReverseTransfer` or `ReversePayout`; that subset is smaller. --- ## Sources Source: https://docs.formance.com/modules/numscript/reference/sources There are several options when it comes to deciding _where_ the money should come from. The `send` statement gives you the following possibilities: ## Single source The simplest way of sending a monetary value is from a single source. Here, we draw `COIN 100` from the `world` account:
```numscript send [COIN 100] ( source = @world destination = @users:001 ) ```
## Ordered sources Using an ordered source block, you can define several accounts to draw from sequentially until the desired monetary value is reached.
```numscript send [COIN 100] ( source = { @users:001:wallet @payments:001 } destination = @orders:001 ) ```
In the example above, if the balance of `COIN` on the account `users:001:wallet` is 30, another 70 will be drawn from the `payments:001` account. Ordered sources can also be capped with a monetary value, so that no more than the specified amount is drawn from them:
```numscript send [COIN 100] ( source = { max [COIN 10] from @users:001:wallet @payments:001 } destination = @orders:001 ) ```
## Portioned sources In addition to sequential accounts, source blocks can also use fractions to split the expense across multiple accounts. In any case, the summed total of fractions in a block needs to be equal to 1 and the `remaining` keyword can be used to reach that total.
```numscript send [COIN 100] ( source = { 10/100 from @platform:marketing remaining from @users:001:wallet } destination = @orders:001 ) ```
Out of convenience, percentage notation is also available:
```numscript send [COIN 100] ( source = { 10% from @platform:marketing remaining from @users:001:wallet } destination = @orders:001 ) ```
## Nested sources Source blocks can be nested with a combination of recursive ordered / portioned specifications:
```numscript send [COIN 100] ( source = { 50% from { max [COIN 10] from @users:001:wallet @users:001:chest } remaining from @payments:001 } destination = @orders:001 ) ```
--- ## Destinations Source: https://docs.formance.com/modules/numscript/reference/destinations As with sources, there are several options when it comes to deciding where the funds in a financial transaction should go. The `send` statement provides the following ways of defining destinations: ## Single destination
```numscript send [COIN 100] ( source = @world destination = @users:001 ) ```
## Allocation destinations Similar to portioned sources, destinations can be defined as a sequence of fractions that splits the monetary value across multiple accounts. In any case, the summed total of fractions in a block needs to be equal to 1 and the `remaining` keyword can be used to reach that total:
```numscript send [COIN 100] ( source = @world destination = { 90/100 to @users:001 remaining to @fees } ) ```
Out of convenience, percentage notation is also available:
```numscript send [COIN 100] ( source = @world destination = { 90% to @users:001 remaining to @fees } ) ```
### Kept destinations Instead of transferring all funds to new accounts, you can keep part of the amount in the source account with the `kept` keyword. It stands in for a destination account inside a block, and pairs naturally with `remaining` to keep whatever is left after the explicit allocations:
```numscript send [COIN 100] ( source = @world destination = { 50% to @users:001 remaining kept } ) ```
Here 50% of the amount goes to `users:001` and the remaining 50% stays in `world`. This is useful when you only want to transfer a portion of the funds. ### Ordered destinations with maximum caps Ordered destinations route funds to multiple accounts in sequence, sending up to a maximum cap to each one before moving on. Any leftover goes to the final `remaining` destination:
```numscript send [COIN 100] ( source = @world destination = { max [COIN 20] to @users:001 max [COIN 50] to @users:002 remaining to @users:003 } ) ```
This sends COIN 20 to `users:001`, COIN 50 to `users:002`, and the remaining COIN 30 to `users:003`. ## Nested destinations Finally, as with sources, destination blocks can be nested:
```numscript send [COIN 100] ( source = @world destination = { 80% to @users:001 20% to { 70% to @platform 15% to @taxes remaining to @charity } } ) ```
--- ## Rounding Source: https://docs.formance.com/modules/numscript/reference/rounding There is no support for floating point or decimal numbers in Numscript, which will always make sure non integer values resulting from monetary computations are balanced. Practically, this means appropriately distributing the non integer allocation remainder to accounts. Numscript works by flooring any computed amount and subsequently spreading the remaining amount as fairly as possible starting from top to bottom. In the example below:
```numscript send [COIN 99] ( source = @world destination = { 50% to @rider 50% to @taxes } ) ```
The `@rider` account will receive `COIN 50` and the `@taxes` account `COIN 49`. The opposite can be achieved by reversing the order of destinations:
```numscript send [COIN 99] ( source = @world destination = { 50% to @taxes 50% to @rider } ) ```
In a more complex example below, we are splitting 99 into 5 which would result in 19.8 allocated to each account. Numscript will first allocate 19 to every account, then attempt to distribute the remaining 4 evenly starting from `@a`:
```numscript send [COIN 99] ( source = @world destination = { 1/5 to @a 1/5 to @b 1/5 to @c 1/5 to @d 1/5 to @e } ) ```
Which will resolve into the following postings: ```json [ { "source": "world", "destination": "a", "amount": 20, "asset": "COIN" }, { "source": "world", "destination": "b", "amount": 20, "asset": "COIN" }, { "source": "world", "destination": "c", "amount": 20, "asset": "COIN" }, { "source": "world", "destination": "d", "amount": 20, "asset": "COIN" }, { "source": "world", "destination": "e", "amount": 19, "asset": "COIN" } ] ``` ## Fixed fees and allocation order When combining fixed amounts with percentage-based allocations, the order of destinations matters due to the multi-pass resolution mechanism. ### The problem Consider a transaction splitting funds between a payment provider (fixed fee + percentage), a franchise fee (percentage), and a store (remaining):
```numscript send [AUD/2 1999] ( source = @world destination = { 7/1999 to @payment_provider 0.6% to @payment_provider 0.5% to @franchise_fee remaining to @store } ) ```
You might expect `@payment_provider` to receive exactly 7 cents as a fixed fee, but it actually receives **8 cents**. This happens because of the two-pass allocation mechanism. ### How multi-pass allocation works **First pass** - Numscript allocates whole amounts: - `7/1999 * 1999 = 7` → `@payment_provider` - `0.6% * 1999 = 11.994` → floors to `11` for `@payment_provider` (keeps 0.994 aside) - `0.5% * 1999 = 9.995` → floors to `9` for `@franchise_fee` (keeps 0.995 aside) - `remaining = 1999 - 7 - 11 - 9 - ceil(0.994 + 0.995) = 1970` → `@store` **Second pass** - Numscript distributes the remaining fragments: - Total distributed: `1999 - 2 = 1997` - Remaining to distribute: `2 cents` - Distribution is **top to bottom**: first position (`@payment_provider`) gets +1, second position (`@payment_provider` again) gets +1 Result: `@payment_provider` receives `7 + 1 = 8` cents instead of the expected 7. ### The solution Move the fixed fee **after** the percentage allocations. Since percentage computations generate at most 1 cent fragment each, placing them first ensures they absorb the remainder:
```numscript send [AUD/2 1999] ( source = @world destination = { 0.6% to @payment_provider 0.5% to @franchise_fee 7/1999 to @payment_provider remaining to @store } ) ```
Now the percentages receive any extra cents, and the fixed fee remains exactly 7 cents. When mixing fixed amounts and percentages, place percentage-based allocations **before** fixed amounts to ensure fixed fees remain exact. --- ## Save Source: https://docs.formance.com/modules/numscript/reference/save It is sometimes helpful to prevent an account from going below a certain threshold balance. The `save` directive allows you to specify a minimum balance for an account, which is deducted from the account's available balance for the transaction.
```numscript // Keep the closing balance of @merchants:1234 at or above [USD/2 100] save [USD/2 100] from @merchants:1234 send [USD/2 500] ( source = @merchants:1234 destination = @payouts:T1891G ) ```
In this transaction example, even if the account `@merchants:1234` has an initial balance of `[USD/2 500]`, the transaction will fail as the account post-transaction balance would otherwise be less than `[USD/2 100]`. If an additional source of funds is provided, the account behaves as if its balance is `[USD/2 100]` less than it actually is:
```numscript // Keep the closing balance of @merchants:1234 at or above [USD/2 100] save [USD/2 100] from @merchants:1234 send [USD/2 500] ( source = { @merchants:1234 @world } destination = @payouts:T1891G ) ```
```json [ { "source": "merchants:1234", "destination": "payouts:T1891G", "amount": 400, "asset": "USD/2" }, { "source": "world", "destination": "payouts:T1891G", "amount": 100, "asset": "USD/2" } ] ``` ## Insufficient funds error When the requested amount exceeds the available balance (after applying `save`), the transaction fails with an `INSUFFICIENT_FUND` error. **Example:** Account has `[GBP/2 120]`, save `[GBP/2 100]`, available = `[GBP/2 20]`
```numscript save [GBP/2 100] from @my_account send [GBP/2 30] ( source = @my_account destination = @world ) ```
This fails because you're trying to send 30 but only 20 is available: ```json { "errorCode": "INSUFFICIENT_FUND", "errorMessage": "running numscript: script execution failed: account(s) @my_account had/have insufficient funds" } ``` ## Using save with send [ASSET *] When using `send [ASSET *]` (send entire balance) with `save`, the behavior depends on the account balance: ### Balance greater than save amount If the balance exceeds the saved amount, the transaction sends `balance - saved_amount`:
```numscript // Account balance: [GBP/2 120] save [GBP/2 100] from @my_account send [GBP/2 *] ( source = @my_account destination = @world ) ```
Result: A transaction of `[GBP/2 20]` is created (120 - 100 = 20). ### Balance less than or equal to save amount If the balance is less than or equal to the saved amount, a transaction with **amount 0** is created:
```numscript // Account balance: [GBP/2 80] save [GBP/2 100] from @my_account send [GBP/2 *] ( source = @my_account destination = @world ) ```
Result: ```json { "postings": [ { "amount": 0, "asset": "GBP/2", "destination": "world", "source": "my_account" } ] } ``` A transaction with 0 amount is still created and recorded in the ledger. This can be useful for audit purposes but may need to be filtered out in reporting. ## Multiple source accounts with save When using multiple sources with `save`, funds are taken from the saved account up to its available limit, then completed from other sources:
```numscript // @account_a balance: [USD/2 150] save [USD/2 100] from @account_a send [USD/2 80] ( source = { @account_a @account_b } destination = @destination ) ```
Result: - `@account_a` contributes `[USD/2 50]` (its available balance: 150 - 100) - `@account_b` contributes `[USD/2 30]` (remainder needed) --- ## Overdraft Source: https://docs.formance.com/modules/numscript/reference/overdraft The `overdraft` directive lets you instruct the Numscript interpreter that an account's post-transaction balance is allowed to be less than zero. ## Unbounded The overdraft directive allows the account to go below zero without any limit, by using the `unbounded` keyword:
```numscript send [USD/2 100] ( source = @foo allowing unbounded overdraft destination = @bar ) ```
## Bounded The exact amount below zero to which the account is allowed to go can be specified by using the `up to` keyword with a monetary value:
```numscript send [USD/2 100] ( source = @foo allowing overdraft up to [USD/2 50] destination = @bar ) ```
## The overdraft() function The `overdraft()` function is an experimental feature. While it is expected to remain available either in its current form or through an equivalent mechanism in future versions, you need to enable experimental features to use it. When working with asset accounts that have negative balances, you can use the `overdraft()` function to safely check balances and perform transactions atomically. ### Overview The `overdraft()` function returns the positive amount of overdraft when an account's balance is negative, or `0` if the balance is positive. This is particularly useful when working with asset accounts, which by design have negative balances in Formance. ### Usage Here's how to use the overdraft function:
```numscript #![feature("experimental-overdraft-function", "experimental-mid-script-function-call")] vars { monetary $acc_overdraft = overdraft(@account, USD/2) } // Use $acc_overdraft in your transaction logic send $acc_overdraft ( source = @world destination = @account ) ```
In this example: - If `@account` has a balance of `[USD/2 -50]`, `$acc_overdraft` will be `[USD/2 50]` - If `@account` has a balance of `[USD/2 100]`, `$acc_overdraft` will be `[USD/2 0]` ### Enabling the experimental feature To use `overdraft()`, you need to enable two experimental features: 1. The experimental rewrite feature 2. The `experimental-overdraft-function` flag See the [Numscript embedding documentation](/modules/numscript) for instructions on enabling these features. ### Alternative: Using balance() If you cannot use the experimental features, note that the standard `balance()` function only works with non-negative balances and will fail if the account balance is negative.
```numscript #![feature("experimental-mid-script-function-call")] vars { // This will fail if @account has a negative balance monetary $bal = balance(@account, USD/2) } ```
For accounts that may have negative balances (like asset accounts), use the `overdraft()` function instead. ## Setting Overdrafts via API In addition to specifying overdrafts in Numscript, you can also configure overdraft limits directly via the API when creating transactions. ### Using the transaction endpoint When sending a transaction via the API, you can specify overdraft allowances in the script's `vars` section: ```json { "script": { "vars": { "amount": "USD/2 1000" }, "plain": "send $amount ( source = @customers:123 allowing unbounded overdraft destination = @merchants:456 )" } } ``` ### Bounded overdrafts via API For bounded overdrafts, include the limit in the Numscript: ```json { "script": { "vars": {}, "plain": "send [USD/2 1000] ( source = @customers:123 allowing overdraft up to [USD/2 500] destination = @merchants:456 )" } } ``` When using the API, the overdraft behavior is the same as when using Numscript directly. The API is simply a transport mechanism for the Numscript execution. ## Overdraft Validation with Backdated Transactions When inserting backdated transactions with overdrafts, the ledger validates the final state rather than intermediate states. See [bi-temporality](/modules/ledger/working-with/bi-temporality#backdated-transaction-validation) for details on how backdated transactions interact with overdraft limits. --- ## Account Pools Source: https://docs.formance.com/modules/payments/cash-pools A **cash pool** is a collection of payment accounts from one or more payment service providers that you want to manage as a single unit. Cash pools aggregate balances across multiple accounts, providing a unified view of your funds for financial reporting, treasury management, and reconciliation. Cash pools can be created in two ways: - **Static pools**: Explicitly specify which accounts to include. Membership is fixed until you edit the pool. - **Dynamic pools**: Define a query that matches accounts at evaluation time. New accounts that match the query — including accounts discovered on a fresh connector cycle after the pool was created — are automatically included in the pool's next balance query. Accounts removed from the connector (uninstalled, or now non-matching) drop out the same way. Dynamic pools resolve membership at every balance read, so a treasury dashboard pointed at a dynamic pool reflects the current set of matching accounts in real time without manual pool maintenance. ## Why cash pools? When managing funds across multiple financial institutions, you often need to view and analyze balances from different accounts together. For example, you might have: - A Stripe account for card payments - A PayPal account for alternative payments - A bank account for direct transfers Rather than querying each account individually, cash pools let you: - **View aggregated balances** across all accounts in real-time - **Track historical balances** at specific points in time - **Simplify financial reporting** with consolidated balance views - **Enable reconciliation** against your internal ledger - **Monitor liquidity** across multiple financial institutions - **Automatically include new accounts** that match your criteria (dynamic pools) ## Pool structure A cash pool contains: - **ID**: Unique UUID identifier - **Name**: Human-readable name (must be unique) - **Created at**: Timestamp when the pool was created - **Pool accounts**: Array of account IDs included in the pool (for static pools) - **Query**: JSON query defining account matching criteria (for dynamic pools) ## Creating cash pools Cash pools are created via the Payments API using one of two approaches: ### Static pools Specify exact accounts to include in the pool: ### Dynamic pools Define criteria that automatically match accounts: Static pools require explicit account IDs, while dynamic pools use JSON query syntax. The two approaches are mutually exclusive - you cannot specify both `accountIDs` and `query` in the same request. ## Query syntax Dynamic pools use the same query syntax as the [List Accounts API](/stack-api-reference/paymentsv3/v3-list-accounts). Supported query fields include: - **connector_id**: Match accounts from specific connectors - **default_asset**: Filter by currency/asset - **type**: Account type (e.g., `INTERNAL`, `EXTERNAL`) - **name**: Account name matching - **psu_id**: Payment service user ID - **metadata.\{key\}**: Custom metadata fields ### Query examples **All EUR accounts across connectors:** ```json { "name": "EUR Accounts", "query": "{\"$match\": {\"default_asset\": \"EUR\"}}" } ``` **Stripe accounts only:** ```json { "name": "Stripe Pool", "query": "{\"$match\": {\"connector_id\": \"stripe:connector-123\"}}" } ``` **Internal accounts with specific metadata:** ```json { "name": "Business Accounts", "query": "{\"$match\": {\"type\": \"INTERNAL\", \"metadata.category\": \"business\"}}" } ``` ## Static vs. Dynamic pools | Feature | Static Pools | Dynamic Pools | |---------|-------------|-------------------| | **Account membership** | Static - explicitly defined | Dynamic - automatically updated | | **New accounts** | Must be manually added | Automatically included if they match criteria | | **Use case** | Known, fixed set of accounts | Accounts that change frequently or match patterns | | **Management** | Requires manual updates | Self-maintaining | | **Account operations** | Can add/remove individual accounts | Cannot modify - accounts determined by query | ### When to use static pools - **Fixed account sets**: When you have a specific, unchanging group of accounts - **Mixed criteria**: When accounts don't follow a consistent pattern - **Fine-grained control**: When you need to include/exclude specific accounts manually ### When to use dynamic pools - **Dynamic environments**: When new accounts are frequently created - **Connector-based grouping**: All accounts from a specific payment provider - **Currency segregation**: Separate pools for different currencies - **Automated workflows**: When pools need to be configured declaratively via API Dynamic pools do not support manual account addition or removal. To modify membership, update the pool's query criteria. ## Using cash pools ### Balance queries Pool balance endpoints aggregate balances from all accounts in the pool by asset: **Latest aggregated balances:** " }} noFctl /> **Historical balances at a specific timestamp:** " }} query={{ at: "2024-01-15T23:59:59Z" }} noFctl /> Balance aggregation logic: 1. **Static pools**: Fetches balances from the specified account IDs 2. **Dynamic pools**: Runs the query to find matching accounts, then fetches their balances 3. Groups by asset/currency 4. Sums amounts for each asset 5. Returns array of aggregated balances Dynamic pools resolve their account membership dynamically at query time, ensuring balances always reflect the current set of accounts matching your criteria.

Using pools with Reconciliation

Cash pools can be used as balance sources in `ledger_vs_pool_drift` and `source_parity` [control templates](/modules/reconciliation/controls). Rules can compare a pool with Ledger or with another pool, use a per-asset tolerance, and read each source at an independent point in time. ```json { "templateKind": "source_parity", "templateSpec": { "left": { "kind": "ledger", "ledger": "main", "query": { "$match": { "address": "control:provider" } } }, "right": { "kind": "payments_pool", "poolID": "" }, "tolerance": { "USD/2": 0 } } } ``` See [Getting Started with Reconciliation](/modules/reconciliation/getting-started) for the complete rule and evaluation workflow. Cash pool IDs are used in the `paymentsPoolID` field of [reconciliation policies](/modules/reconciliation/concepts#policies): ```json { "name": "string", "ledgerName": "string", "ledgerQuery": "object", "paymentsPoolID": "uuid-string" } ``` The reconciliation service compares ledger account balances against the aggregated cash pool balances. See [Getting Started with Reconciliation](/modules/reconciliation/getting-started) for a complete workflow. --- ## Variables Source: https://docs.formance.com/modules/numscript/reference/variables Hardcoded values are great for quick prototyping and iteration, but chances are that you will at some point need to inject some variables into your Numscript files. Here is an example using definitions of all the supported variable types:
```numscript vars { monetary $price account $trade portion $commission asset $pair number $id string $reference } send $price ( source = @world destination = { $commission to @platform remaining to $trade } ) set_tx_meta("asset", $asset) set_tx_meta("id", $id) set_tx_meta("reference", $reference) ```
Injections of variables can be done at execution, by using the `POST /{ledger}/transactions` endpoint. The `script.vars` field in the request body is used to inject variable values: ```json { "script": { "vars": { "price": "USD/2 100", "trade": "trades:108391999", "commission": "15%", "pair": "EUR/2", "id": "108391999", "reference": "USD/EUR:108391999" } } } ``` Variable names must be lowercase and start with at least one letter or `_`. They can contain letters, digits, and `_`. ## Account type This type represents account names. They must start with at least one letter or `_`. They can contain letters, digits, `_`, and `:`. ## Asset type This type represents asset names. They can contain uppercase letters, digits and `/`. ## Monetary type This type represents a positive integer amount associated with an asset. ```json "USD/2 100" ``` It is possible to pull the balance of an account and inject it in a monetary variable by using the `balance(_account_, _asset_)` statement. The balance pulled needs to be non-negative, or the script will fail to execute. Here is an example:
```numscript #![feature("experimental-mid-script-function-call")] vars { monetary $initial = balance(@A, USD/2) } send [USD/2 100] ( source = { @A @C } destination = { max $initial to @B remaining to @D } ) ```
## Portion type This type represents portions of monetary values. They can be expressed in 2 different ways: * As a percentage: `15%` * As a fraction: `15/100` Their computed values must be between 0 and 1 (inclusive). Variables can also be pulled from account metadata, as described in the [metadata](/modules/numscript/reference/metadata) section. --- ## Metadata Source: https://docs.formance.com/modules/numscript/reference/metadata Numscript transactions can interact with metadata, both on transactions and accounts. ## Account metadata ### Reading metadata to initialize a variable Structured account metadata can be injected in Numscript variables during initialization. In the example below, we inject the monetary value stored under the metadata key `"coupon_value"` from the `coupon` account:
```numscript #![feature("experimental-mid-script-function-call")] vars { account $coupon account $wallet monetary $value = meta($coupon, "coupon_value") } send $value ( source = $coupon destination = $wallet ) ```
Metadata injected into variables needs to be typed, and its type is read from the value of the key `type` of the object stored under the metadata key. Its value is read from the value of the key `value`. Here are all the available types: ```json { "amount": { "type": "number", "value": 1000 } } ``` ```json { "reference": { "type": "string", "value": "82HHON80ILP" } } ``` ```json { "currency": { "type": "asset", "value": "USD/2" } } ``` ```json { "coupon_value": { "type": "monetary", "value": { "amount": 1000, "asset": "USD/2" } } } ``` ```json { "merchant": { "type": "account", "value": "platform:merchant" } } ``` ```json { "commission": { "type": "portion", "value": "15.5%" } } ``` ### Writing account metadata during a transaction Metadata can be written to an account using the `set_account_meta(_account_, "key", _value_)` statement. The statement takes a string-type key and a value which can be of any type, either as a variable or a literal. ## Writing metadata to a transaction Metadata can be written to a transaction using the `set_tx_meta("key", _value_)` statement. The statement takes a string-type key and a value which can be of any type, either as a variable or a literal.
```numscript set_tx_meta("order_fee", [USD/2 100]) set_tx_meta("tax", 20/100) set_tx_meta("collection_account", @platform:commission) set_tx_meta("commission", $commission) ```
## Wrap-up example
```numscript #![feature("experimental-mid-script-function-call")] vars { account $order account $merchant = meta($order, "merchant") monetary $fee portion $commission string $ref } send $fee ( source = @orders:1234 destination = @platform:fees ) send [USD/2 *] ( source = @orders:1234 destination = { $commission to @platform:fees remaining to $merchant } ) set_account_meta($order, "reference", $ref) set_tx_meta("order_fee", $fee) set_tx_meta("tax", 20/100) set_tx_meta("commission", $commission) ```
```json { "script": { "vars": { "order": "orders:186HH78UH", "fee": { "amount": 1000, "asset": "USD/2" }, "commission": "15.5%", "reference": "108IUYGI" } } } ``` --- ## Unambiguous Monetary Notation Source: https://docs.formance.com/modules/numscript/monetary-notation The Formance Platform uses a unified, safe-by-design way of representing monetary values across all its services and components. We call this representation the _Unambiguous Monetary Notation_, or UMN for short. While you can use any `[A-Z]{1,16}(\/\d{1,6})` asset in your ledger transactions, it is encouraged to always use UMN, especially if you're dealing with any of the standardized [ISO-4217](https://en.wikipedia.org/wiki/ISO_4217) currencies. ## Specification A UMN value is represented as: ```text [ASSET/SCALE AMOUNT] ``` Where: * `ASSET` is a string of 1 to 16 uppercase letters, representing the currency code of the asset, either standardized or fictional. * `SCALE` represents the negative power of ten by which the amount is multiplied to obtain the decimal value in the given asset * `AMOUNT` is an unsigned integer. As an example `[USD/2 30]` is equivalent to `USD 30*1E-2`, i.e `USD 0.30`, i.e 30 USD cents. For values where the amount already represents the amount of said asset, a scale of zero should not be represented, e.g. `[JPY 100]`. ## Precision The UMN specification does not enforce a specific precision of the amount, beyond the fact that it must be represented as an unsigned integer. Decisions on the precision of the amount are left to the implementation when implemented by a third party. Internally, Formance Stack components all use arbitrary precision unsigned integers to represent amounts. ## Examples | UMN | Human Readable | ISO-4217 code | | --- | --- | --- | | `[USD/2 30]` | `$0.30` | `USD` | | `[JPY 100]` | `¥100` | `JPY` | | `[BTC/8 100000000]` | `1 BTC` | `BTC` | | `[GBP/2 100]` | `£1.00` | `GBP` | | `[EUR/2 100]` | `€1.00` | `EUR` | | `[INR/2 100]` | `₹1.00` | `INR` | | `[CNY/2 100]` | `¥1.00` | `CNY` | | `[CAD/2 100]` | `CA$1.00` | `CAD` | While `USD/2` is a reasonable notation for most USD-handling use-cases, nothing prevents you from using `USD/4` or `USD/6` if you need to represent smaller amounts and subdivisions of USD in your system. The same applies to other currencies, e.g. `JPY/2` or `JPY/4` for Japanese Yen and while such a coin is not in circulation, it is still a valid notation when these amounts are used in a context where they will end up being floored or ceiled to the nearest whole unit later down the line. ## Rationale The reason behind this recommendation is that using non explicitly scaled currencies like `USD` is inherently ambiguous, with interpretation of the scale left as an exercise to the reader. If you receive from a payment processor an API response as follows: ```json { "amount": 100, "currency": "USD" } ``` Without more context, it is unfortunately impossible to tell whether the amount is in cents, or in dollars. While best practices dictate that the amount should be denominated in the smallest unit of the currency, this is not always the case as this interpretation is not standardized across payments services providers. Some services will inevitably use different formats and encoding rules, resulting in situations where both `100`, and `100.30` are happily parsed, leaving the door open to catastrophic consequences. As you start to scale your business and deal with multiple and specialized payment services providers, the risk of different formats making their way to your internal representation increases along with the risk of misinterpreting the amount. As Formance components are designed to be used in a variety of contexts and find themselves dealing with a variety of formats from different providers, we decided to explicitly specify the scale of the amount in the notation, making UMN really hard to misinterpret by design. --- ## CLI Source: https://docs.formance.com/modules/numscript/cli ## Install You can install the `numscript` CLI in one of the following ways: #### Using curl For Mac and Unix: ```sh curl -sSf https://raw.githubusercontent.com/formancehq/numscript/main/install.sh | bash ``` #### Using golang toolchain ```sh go install github.com/formancehq/numscript/cmd/numscript@latest ``` ## Check You can use the `numscript check` command to run static analysis on a numscript program. The static analysis includes parsing errors, wrong variables or types usage, as well as more advanced checks on numscript constructs. You can use it this way: ```shellscript numscript check my-file.num ``` The command will exit with an error status code if there is at least one error or warning. ## Run Available from numscript@0.0.19 You can use the CLI to run local scripts (mostly intended for local prototyping). For example, given this script: ```numscript my-script.num vars { monetary $amt } send $amt ( source = @alice destination = @world ) ``` And this inputs file (which has to have the same name as the numscript file, plus the `.inputs.json` suffix): ```json my-script.num.inputs.json { "$schema": "https://raw.githubusercontent.com/formancehq/numscript/refs/heads/main/v1.inputs.schema.json", "variables": { "amt": "USD/2 100" }, "balances": [ { "account": "alice", "asset": "USD/2", "amount": 9999 } ] } ``` This format is available from numscript@0.0.25. If you're on an older version, use the [legacy inputs schema](https://raw.githubusercontent.com/formancehq/numscript/refs/heads/main/inputs.schema.json) instead — it uses a map instead of an array for `balances` (and `metadata`). You can run the file using: ```shellscript numscript run my-script.num ``` You'll see the postings: ``` Postings: | Source | Destination | Asset | Amount | | alice | world | USD/2 | 100 | ``` ## Test Available from numscript@0.0.19 You can use the `numscript test` command to check that the specs given in a [numscript specs format](/modules/numscript/specs) file are valid for a given numscript. For example: ```shellscript numscript test src/domain/numscript ``` This looks through the `src/domain/numscript` folder for every `.num` that has a matching `.num.specs.json` specs file. --- ## Functions Source: https://docs.formance.com/modules/numscript/reference/functions Built-in functions callable from Numscript. Which functions you can use depends on the interpreter version bundled in your Ledger release — see [Selecting an Interpreter](/modules/numscript/interpreter) for the full availability matrix. The entries below reflect your current selection: each experimental function shows whether it is available for the interpreter version you have selected. ## Statements These are available on both the original (`machine`) and experimental interpreters. ### `set_tx_meta("key", value)` Writes metadata to the transaction. The key is a string; the value can be any type, as a literal or a variable. [Details](/modules/numscript/reference/metadata) **Available from:** all versions (`machine` and `experimental-interpreter`). ### `set_account_meta(account, "key", value)` Writes metadata to an account during the transaction. [Details](/modules/numscript/reference/metadata) **Available from:** all versions (`machine` and `experimental-interpreter`). ## Functions in `vars` These functions are called when initializing variables in the `vars` block. They require the experimental interpreter (`runtime: experimental-interpreter`) and the `experimental-mid-script-function-call` flag. ### `meta(account, "key")` Reads structured account metadata into a typed variable. [Details](/modules/numscript/reference/metadata) **Available from:** interpreter `0.0.15` · Ledger 2.3 · Stack v3.1 · flag `experimental-mid-script-function-call` Not available in your selected version (requires interpreter 0.0.15+).}> Available in your selected version. ### `balance(account, asset)` Returns the balance of an account for an asset. Fails on negative balances — use `overdraft()` for accounts that may go negative. [Details](/modules/numscript/reference/mid-script-functions) **Available from:** interpreter `0.0.15` · Ledger 2.3 · Stack v3.1 · flag `experimental-mid-script-function-call` Not available in your selected version (requires interpreter 0.0.15+).}> Available in your selected version. ### `overdraft(account, asset)` Returns the positive overdraft amount of an account, or zero if the balance is non-negative. [Details](/modules/numscript/reference/overdraft) **Available from:** interpreter `0.0.15` · Ledger 2.3 · Stack v3.1 · flags `experimental-overdraft-function`, `experimental-mid-script-function-call` Not available in your selected version (requires interpreter 0.0.15+).}> Available in your selected version. ### `get_asset(monetary)` Returns the asset part of a monetary value. [Details](/modules/numscript/reference/get-asset) **Available from:** interpreter `0.0.16` · Ledger 2.3 · Stack v3.1 · flag `experimental-get-asset-function` Not available in your selected version (requires interpreter 0.0.16+).}> Available in your selected version. ### `get_amount(monetary)` Returns the numeric amount part of a monetary value. [Details](/modules/numscript/reference/get-amount) **Available from:** interpreter `0.0.16` · Ledger 2.3 · Stack v3.1 · flag `experimental-get-amount-function` Not available in your selected version (requires interpreter 0.0.16+).}> Available in your selected version. --- ## Numscript specs format Source: https://docs.formance.com/modules/numscript/specs The Numscript specs format is a conventional way to express unit tests about Numscript, using JSON. It can be used to define assertions over the results of a numscript run, given certain inputs. You can execute the tests using the [`numscript test`](/modules/numscript/cli#test) command. A JSON schema is available [online](https://raw.githubusercontent.com/formancehq/numscript/refs/heads/main/v1.specs.schema.json), so that you can have autocomplete and diagnostics in your editor. In many editors, such as VS Code, you can enable it by adding it to the JSON like this: ```json { "$schema": "https://raw.githubusercontent.com/formancehq/numscript/refs/heads/main/v1.specs.schema.json" } ``` This format is available from numscript@0.0.25. If you're on an older version, use the [legacy specs schema](https://raw.githubusercontent.com/formancehq/numscript/refs/heads/main/specs.schema.json) instead — it uses maps instead of arrays for `balances` and `metadata`. Here's the schema (using typescript notation): ```typescript type Specs = { balances?: Balances; variables?: Vars; metadata?: AccountsMetadata; featureFlags?: Array; testCases: Array }; type TestCase = { balances?: Balances; variables?: Vars; metadata?: AccountsMetadata; it: string; "expect.error.missingFunds"?: boolean; "expect.error.negativeAmount"?: boolean; "expect.postings"?: Array; "expect.txMetadata"?: TxMetadata; "expect.metadata"?: SetAccountsMetadata; "expect.endBalances"?: Balances; "expect.endBalances.include"?: Balances; "expect.movements"?: Movements; } ``` ```typescript type Balances = Array; type BalanceRow = { account: string; asset: string; amount: number; // Optional, for setups using asset colors or scoped accounts color?: string; scope?: string; }; type Vars = { [name: string]: string }; type AccountsMetadata = Array; type AccountMetadataRow = { account: string; key: string; value: string; scope?: string; }; // Used by `expect.metadata`. The value is the metadata's rendered form (see // the note under `expect.txMetadata`), same shape as the `metadata` precondition. type SetAccountsMetadata = Array; type SetAccountMetadataRow = { account: string; key: string; value: string; scope?: string; }; type TxMetadata = Array; type TxMetadataRow = { key: string; value: string; }; type Posting = { source: string; destination: string; asset: string; amount: number; sourceScope?: string; destinationScope?: string; color?: string; }; type Movements = Array; type Movement = { source: string; destination: string; asset: string; amount: number; sourceScope?: string; destinationScope?: string; color?: string; }; ``` ### Example Say we have the following numscript:
```numscript vars { monetary $cap account $source account $destination } send [EUR/2 *] ( source = max $cap from $source destination = $destination ) ```
And we want to test that we never send more than `$cap`. We can express the relevant test cases in the following way: ```json { "$schema": "https://raw.githubusercontent.com/formancehq/numscript/refs/heads/main/v1.specs.schema.json", "variables": { "source": "alice", "destination": "bob" }, "balances": [ { "account": "alice", "asset": "EUR/2", "amount": 500 } ], "testCases": [ { "it": "sends all the available balance when it doesn't exceed the cap and @alice has enough balance", "variables": { "cap": "EUR/2 9999" }, "expect.postings": [ { "source": "alice", "destination": "bob", "amount": 500, "asset": "EUR/2" } ] }, { "it": "caps the sent amt to $cap when lower than available balance", "variables": { "cap": "EUR/2 10" }, "expect.postings": [ { "source": "alice", "destination": "bob", "amount": 10, "asset": "EUR/2" } ] } ] } ``` ## Preconditions The inputs of each test cases. You can set the preconditions top-level (in the outer object), and/or in each `testCase` . The preconditions in a testCase will be merged to the top-level preconditions (with the precedence being given to the inner preconditions). Each entry in `balances` and `metadata` is keyed by its `account` (plus `asset` for balances, `key` for metadata, and optionally `color` / `scope`) — merging replaces entries with a matching key and keeps the rest. For example, in the following specs: ```json { "balances": [ { "account": "alice", "asset": "EUR/2", "amount": 100 }, { "account": "alice", "asset": "USD/2", "amount": 100 }, { "account": "bob", "asset": "EUR/2", "amount": -2 } ], "testCases": [ { "it": "example specs", "balances": [ { "account": "alice", "asset": "EUR/2", "amount": 999 } ] } ] } ``` The inner preconditions will only override `@alice` 's `EUR/2` balance, resulting in: ```json [ { "account": "alice", "asset": "EUR/2", "amount": 999 }, { "account": "alice", "asset": "USD/2", "amount": 100 }, { "account": "bob", "asset": "EUR/2", "amount": -2 } ] ``` ### `variables` The (stringified) value of each variable ```json { "variables": { "amount": "USD/2 100" } } ``` ### `balances` The initial accounts' balances. ```json { "balances": [ { "account": "alice", "asset": "USD/2", "amount": 200 }, { "account": "bob", "asset": "USD/2", "amount": -42 } ] } ``` ### `metadata` The initial accounts' metadata. ```json { "metadata": [ { "account": "alice", "key": "id", "value": "1234" } ] } ``` ## Assertions Assertions are only run if explicitly defined. The recommended assertion to use by default are `expect.postings` or `expect.error.missingFunds`, but there are also a few weaker assertion that might be useful when the exact postings are an implementation detail of your business logic. ### `expect.error.missingFunds` Assert that the script failed because of missing funds. Even if this is set to true, the test will still fail if the script outputs a different error. Defaults to `false`. Note: this was called `expect.error` in earlier releases ### `expect.error.negativeAmount` Assert that the script failed because of a send statement using a negative amount. Defaults to `false`. ### `expect.postings` Assert against the exact postings emitted by the script. To assert that there are no postings, you can use the empty array. To assert that no postings are produced because of a failure due to missing funds, you can use the `expect.error.missingFunds` assertion instead. ```json { "expect.postings": [ { "source": "world", "destination": "user:001", "asset": "EUR/2", "amount": 100 } ] } ``` ### `expect.txMetadata` Assert against the transaction meta emitted by the script (using `set_tx_meta`). It's an array of entries, one per metadata key. Each `value` is the metadata's rendered form — a monetary renders as `"USD/2 100"`, a portion as `"1/2"`, an account as its bare name, and so on — rather than a type-tagged object. Because the wire form is untyped, values of different types that render alike (a string `"42"` and the number `42`) are indistinguishable here. ```json { "expect.txMetadata": [ { "key": "senderAccount", "value": "user:5829" } ] } ``` An unreleased build briefly represented this value as a type-tagged object, e.g. `{"type": "monetary", "asset": "USD/2", "amount": "100"}` instead of `"USD/2 100"`. That representation never reached a stable release and has been reverted — if a specs file still uses it, rewrite each `value` to its rendered string form. ### `expect.metadata` Assert against the accounts metadata at the end of script execution (using `set_account_meta`). It's an array of entries, each keyed by `account` and `key`, with the same rendered `value` shape as `expect.txMetadata` (including the same short-lived tagged-object format, now reverted). ```json { "expect.metadata": [ { "account": "alice", "key": "id", "value": "1234" } ] } ``` Note that it takes into account the values defined with the `metadata` precondition as well. ### `expect.endBalances` Assert against the balances at the end of the script For example: ```json { "expect.endBalances": [ { "account": "alice", "asset": "EUR/2", "amount": 100 } ] } ``` means that `@alice` has `[EUR/2 100]` balance after the script is applied. You might consider using this assertion when you only care about the end balance of an account, for example if you need to bring an account to a certain value (not less, not more), so you don't care about how the postings are composed exactly. Note: this was called `expect.volumes` in earlier releases ### `expect.endBalances.include` A weaker version of `expect.endBalances` that allows defining a subset of the balances we assert against. For example, the following: ```json { "expect.endBalances.include": [ { "account": "alice", "asset": "EUR/2", "amount": 100 } ] } ``` passes even if there are more accounts in the involved balances, and if `alice` emit postings involving other currencies. ### `expect.movements` Assert against the resulting movements. A movement is an array entry from a source account, to a destination account, for a given asset and amount. For example, this assertion: ```json { "expect.movements": [ { "source": "alice", "destination": "bob", "asset": "EUR/2", "amount": 100 } ] } ``` means that `@alice` sent `[EUR/2 100]` to `@bob` You might consider using this assertion when you care about the movements graph from-to accounts, and you don't care about the order of the postings or the way they are split. ## Focus mode You can select a subset of test to run by using the `focus` and `skip` modifiers on a test case definition. They are only meant to be used while developing, and will produce an error status code so that they aren't committed by mistake thus producing false positive tests. ### `focus` If at least a test has a `focus` modifier, all the tests without the `focus` modifier will be skipped. ```json { "testCases": [ { "it": "only run this test!", "focus": true, "expect.postings": // .. }, { "it": "this test is skipped", "expect.postings": // .. } ] } ``` ### `skip` If a test is marked with the `skip` modifier, it will not be run. ```json { "testCases": [ { "it": "skip this test", "skip": true, "expect.postings": // .. } ] } ``` --- ## Payment Service Users Source: https://docs.formance.com/modules/payments/payment-service-users A Payment Service User (PSU) represents an end-user in the Formance Payments system. PSUs serve as a way to associate accounts, payments, and connections with specific individuals or entities across different connector types. ## PSU Properties | Field | Description | |-------|-------------| | **ID** | Unique identifier for the PSU | | **Name** | User's full name (encrypted at rest) | | **Contact Details** | Email, phone number, locale (encrypted at rest) | | **Address** | Street name, street number, city, region, postal code, country (encrypted at rest) | | **Bank Account IDs** | Associated traditional bank accounts | | **Metadata** | Additional custom key-value pairs | | **Created At** | Timestamp of PSU creation | All personally identifiable information (name, contact details, address) is encrypted at rest to ensure compliance with data protection regulations. ## Connections A connection represents the link between a PSU and their bank account(s) through a provider. Key characteristics: - Usually, a user has one connection per bank - There can be multiple accounts in a single connection - The same account could appear in different connections (e.g. joint accounts, guardianship) ## Working with PSUs For a complete walkthrough of creating PSUs, establishing connections, and accessing account data, see the [Open Banking Getting Started guide](/modules/payments/connectors/open-banking/getting-started). --- ## oneof Source: https://docs.formance.com/modules/numscript/reference/oneof Requires flag: `experimental-oneof` Select the first source or destination that can satisfy the transaction. Unlike ordered sources (which split funds across accounts), `oneof` picks exactly one branch. ## Source Try each account in order. The first one with sufficient balance handles the entire amount: ```numscript #![feature("experimental-oneof")] send [USD/2 10000] ( source = oneof { @users:1234:main @users:1234:savings @world } destination = @merchants:5678 ) ``` If `@users:1234:main` has 10000 or more, it's used entirely. If not, `@users:1234:savings` is tried. Falls through to `@world` only if neither user account can cover the full amount. This differs from ordered sources (`{ @a @b @c }`) which would drain `@a` first, then take the remainder from `@b`, then `@c`. With `oneof`, it's all-or-nothing per branch. ## Destination ```numscript #![feature("experimental-oneof")] send [USD/2 10000] ( source = @revenue destination = oneof { max [USD/2 5000] to @payouts:priority remaining to @payouts:standard } ) ``` The first matching constraint is used. `max` caps the amount for that branch; `remaining` catches everything else. --- ## Payments Source: https://docs.formance.com/modules/payments For the mental model of how Payments, Ledger, and Reconciliation work together, see [How the Modules Fit Together](/getting-started/modules-fit-together). ## Getting started Learn the core concepts you need to know to use the Payments service. Find which capabilities does each provider support. Install the Formance connector for your payment service provider. Connect to Open Banking providers like Plaid, Tink, and Powens for secure bank account access. --- ## Account Interpolation Source: https://docs.formance.com/modules/numscript/reference/account-interpolation Requires flag: `experimental-account-interpolation` Build account addresses dynamically by inserting variables into the address: ```numscript #![feature("experimental-account-interpolation")] vars { string $user_id string $type } send [USD/2 1000] ( source = @world destination = @users:$user_id:wallets:$type ) ``` With `$user_id = "42"` and `$type = "main"`, the destination resolves to `@users:42:wallets:main`. ## Supported variable types Account, string, and number variables can be interpolated. Each is coerced to a string and inserted into the address. ```numscript #![feature("experimental-account-interpolation")] vars { number $id account $org } send [USD/2 500] ( source = @world destination = @orgs:$org:members:$id:balance ) ``` ## Use cases - Route to user-specific accounts without hardcoding addresses - Build hierarchical account structures from transaction parameters - Create dynamic chart-of-accounts patterns like `@orders:$order_id:payments:$payment_id` --- ## get_asset Source: https://docs.formance.com/modules/numscript/reference/get-asset Requires flag: `experimental-get-asset-function` Returns the asset part of a monetary value. ```numscript #![feature("experimental-mid-script-function-call", "experimental-get-asset-function")] vars { monetary $payment asset $asset = get_asset($payment) } send $payment ( source = @world destination = @users:1234 ) set_tx_meta("currency", $asset) ``` With `$payment = [EUR/2 5000]`, `$asset` resolves to `EUR/2`. ## Use cases - Store the asset of a dynamic monetary variable in transaction metadata - Route transactions differently based on currency - Validate asset type before processing --- ## get_amount Source: https://docs.formance.com/modules/numscript/reference/get-amount Requires flag: `experimental-get-amount-function` Returns the numeric amount part of a monetary value. ```numscript #![feature("experimental-mid-script-function-call", "experimental-get-amount-function")] vars { monetary $payment number $amount = get_amount($payment) } send $payment ( source = @world destination = @users:1234 ) set_tx_meta("amount", $amount) ``` With `$payment = [EUR/2 5000]`, `$amount` resolves to `5000`. ## Use cases - Store the amount of a dynamic monetary variable in transaction metadata - Compute fees or splits based on the amount - Log transaction amounts independently of the asset --- ## Mid-script Function Calls Source: https://docs.formance.com/modules/numscript/reference/mid-script-functions Requires flag: `experimental-mid-script-function-call` Call functions like `balance()` directly in variable declarations, and use arithmetic expressions in the `vars` block. ## Function calls in vars ```numscript #![feature("experimental-mid-script-function-call")] vars { monetary $current = balance(@users:1234, USD/2) } send $current ( source = @users:1234 destination = @savings:1234 ) ``` Without this feature, `balance()` can only be used inline. With it, you can capture the result in a variable and reference it multiple times. ## Arithmetic expressions ```numscript #![feature("experimental-mid-script-function-call")] vars { portion $fee = 80% monetary $total = balance(@users:1234, USD/2) } send $total ( source = @users:1234 destination = { $fee to @platform:fees remaining to @merchants:5678 } ) set_tx_meta("fee", $fee) set_tx_meta("total", $total) ``` --- ## Asset Colors Source: https://docs.formance.com/modules/numscript/reference/asset-colors Requires flag: `experimental-asset-colors` Restrict which funds can be sourced from an account based on a color tag. Colors track the origin or purpose of funds — useful for compliance, earmarking, or regulatory constraints. ## Syntax Use `\` after a source account to restrict by color: ```numscript #![feature("experimental-asset-colors")] send [COIN 100] ( source = @treasury \ "GRANT" destination = @programs:education ) ``` Only funds tagged with the `"GRANT"` color in `@treasury` are used. If the account has 200 COIN total but only 80 are tagged `"GRANT"`, this transaction fails with insufficient funds. ## Ordered color fallback Combine with ordered sources to try different colors in priority: ```numscript #![feature("experimental-asset-colors")] send [COIN 100] ( source = { @treasury \ "GRANT" @treasury \ "DONATION" @treasury } destination = @programs:education ) ``` Try grant-colored funds first, then donation-colored, then uncolored funds. The final entry is not an "any color" fallback: a source with no color restriction reads the uncolored bucket only, so `@treasury` there is the same source as `@treasury \ ""`. No single source draws across every color — to reach more buckets, list each one as its own ordered entry. ## With overdraft Color restrictions can be combined with overdraft: ```numscript #![feature("experimental-asset-colors")] send [COIN 100] ( source = @treasury \ "GRANT" allowing unbounded overdraft destination = @programs:education ) ``` ## Use cases - Earmark funds for specific purposes (grants, donations, operating) - Regulatory compliance — track fund origins through the ledger - Prevent mixing of restricted and unrestricted funds --- ## Getting Started with Open Banking Source: https://docs.formance.com/modules/payments/connectors/open-banking/getting-started This guide walks you through implementing Open Banking with Formance, from creating users to accessing bank account data. - An Open Banking connector installed ([Plaid](/modules/payments/connectors/open-banking/plaid), [Tink](/modules/payments/connectors/open-banking/tink), or [Powens](/modules/payments/connectors/open-banking/powens)) ## Overview The Open Banking workflow consists of these key steps: 1. **Create a Payment Service User (PSU)** - Represents your end user 2. **Forward PSU to Connector** - Register the user with your Open Banking provider 3. **Create Authentication Link** - Generate a secure URL for bank connection 4. **User Authentication** - User connects their bank account via the provider's interface 5. **Access Account Data** - Retrieve accounts, balances, and transactions ## Implementation ### Step 1: Create a Payment Service User Create a [Payment Service User (PSU)](/modules/payments/payment-service-users) to represent the end user who will connect their bank account. The fields shown below are exhaustive. Some providers may require only a subset of these fields, while others may require all of them. Check the connector docs for your specific provider's requirements. **Response:** ```json { "data": "5968b0e2-06da-4552-8ad0-c484706bd2d7" } ``` Save the returned PSU ID for subsequent steps. ### Step 2: Forward PSU to Connector Register the PSU with your Open Banking connector to prepare them for authentication. ", connectorID: "" }} noFctl /> **Parameters:** - `psuID`: The PSU ID from Step 1 - `connectorID`: Your Open Banking connector ID (find this in your Formance Console under Connectors) **Response:** ``` 204 No Content ``` ### Step 3: Create Authentication Link Generate a secure authentication URL for the user to connect their bank account. ", connectorID: "" }} noFctl body={{ applicationName: "Your App Name", clientRedirectURL: "https://yourapp.com/banking" }} /> Check the connector docs for your provider's redirect URL requirements and restrictions. **Response:** ```json { "attemptID": "xyz789", "link": "https://secure.plaid.com/hl/authentication-link" } ``` ### Step 4: User Authentication Flow Direct the user to the authentication link to connect their bank account. **Your Application:** Redirect the user to the `link` from Step 3 See [Frontend Integration Guidelines](#frontend-integration-guidelines) for detailed implementation guidance across different platforms. **User Experience (Provider Interface):** The user will: 1. Select their bank from the provider's interface and be redirected to the bank's interface 2. Enter credentials or complete OAuth flow with their bank 3. Grant permissions for account access 4. See confirmation that the connection was successful **Return to Your Application:** - The provider automatically redirects the user back to your `clientRedirectURL` - Check the authentication status by requesting the link attempt: ", connectorID: "", attemptID: "" }} noFctl /> **Response:** ```json { "data": { "id": "adc80553-02df-4d42-ad88-44099af38580", "psuID": "7ab143dd-e686-4fbe-a64d-11f67b40985d", "connectorID": "eyJQcm92aWRlciI6InBvd2VucyIsIlJlZmVyZW5jZSI6ImMxMTMyYjg0LTdmYTEtNDRhZS1hZmRjLTBjMWZjMjIyYTIyYSJ9", "createdAt": "2025-09-25T15:05:04.316284Z", "status": "completed", "clientRedirectURL": "https://console.v3.staging.formance.cloud/knonmzexcoal/vayn?region=staging.formance.cloud", "error": null } } ``` - Use the `status` field to determine if the authentication was successful ### Step 5: Access Connected Accounts Once the connection is established, you can access the user's account data. **List accounts for a specific PSU:** **List all accounts (no filter):** **Get specific account:** " }} noFctl /> **Get account balances:** " }} noFctl /> ## Connection Management ### List User Connections View all connections for a specific user and connector: ", connectorID: "" }} noFctl /> **Response:** ```json { "cursor": { "pageSize": 15, "hasMore": false, "data": [ { "connectionID": "conn_456def789", "connectorID": "plaid_prod_001", "createdAt": "2024-01-15T10:30:00Z", "dataUpdatedAt": "2024-01-15T10:35:00Z", "status": "ACTIVE", "error": null, "metadata": {} } ] } } ``` ### Monitor Connection Status Set up webhook handlers to receive notifications about connection status changes. You'll receive events when users complete authentication, when new data is synced, or when connections are lost. ### Refresh Stale Connections If a connection becomes stale, generate a new authentication link: ", connectorID: "", connectionID: "" }} noFctl body={{ applicationName: "Your App Name", clientRedirectURL: "https://yourapp.com/banking" }} /> ### Delete Operations **Delete a specific connection:** ", connectorID: "", connectionID: "" }} noFctl /> **Delete entire user:** " }} noFctl /> Deletion operations are permanent and cannot be undone. All related data (accounts, transactions, connections) will be permanently removed. ## Frontend Integration Guidelines ### Browser Integration For web applications: - **Use full page redirects** - Do not use iframes as they're not compatible with all bank redirections - Follow provider-specific browser integration guidelines ### Mobile Integration For mobile applications: - **Android**: Use Chrome Custom Tabs for the authentication flow - **iOS**: Use SFSafariViewController for the authentication flow - Avoid using in-app webviews as they may not support all authentication flows ### Provider-Specific Guidelines Each Open Banking provider has specific integration requirements: - **[Plaid](https://plaid.com/docs/)**: Standard OAuth implementation with redirect URL restrictions - **[Powens](https://docs.powens.com/api-reference/overview/webview#browser-integration)**: Specific webview browser integration guidelines - **[Tink](https://docs.tink.com/resources/transactions/optimize-your-transactions-integration)**: Platform-specific optimization guides for Android and iOS --- ## Plaid Source: https://docs.formance.com/modules/payments/connectors/open-banking/plaid The Plaid connector links Formance to Plaid's bank-aggregation platform. It drives the [Open Banking PSU flow](/modules/payments/connectors/open-banking/getting-started): create a Payment Service User, forward them to Plaid, hand them a Plaid Link session, and sync the resulting accounts, balances, and transactions back through Payments. Available from Payments 3.2.0. The Plaid connector requires Payments **3.2.0 or higher**. Your stack pins an older version — upgrade to use it. ## Prerequisites You need a Plaid account and a `clientID` + `clientSecret` pair. The key needs access to Auth, Transactions, and Identity at minimum (set products on the Plaid dashboard before issuing the key). Plaid signs every webhook with a JWT verified against Plaid's public key — no shared secret to configure. ## Installation {"fctl payments connectors install plaid config.json"} ### Configuration fields `isSandbox: true` flips to Plaid's sandbox host (`https://sandbox.plaid.com`); production uses `https://production.plaid.com`. No separate "development" environment. ## Capabilities - `FETCH_ACCOUNTS` — depository, credit, and loan accounts per PSU connection. - `FETCH_BALANCES` — available + current balance via Plaid's `/accounts/balance/get`. - `FETCH_EXTERNAL_ACCOUNTS` — counterparty accounts identified by Plaid's `Transfer` product when present. - `FETCH_PAYMENTS` — transactions via Plaid's incremental `/transactions/sync` cursor. - `CREATE_WEBHOOKS` + `TRANSLATE_WEBHOOKS` — webhooks per Item (Plaid's connection primitive), provisioned on link. The connector doesn't initiate transfers or payouts — Plaid's `Transfer` product is read-only here. Use a PSP connector (Stripe, Increase) for outbound flows. ## Account model Every Payments internal account is one Plaid `Account` on a linked Item (depository, credit, or loan). The `reference` is the Plaid `account_id`; `name` is the Plaid account name; `defaultAsset` is the account's ISO currency at standard precision. Each account is PSU-scoped — `psuID` carries the Payments PSU and `openBankingConnectionID` carries the Plaid Item. EXTERNAL accounts come from Plaid's `Transfer` product when present. See [Accounts](/modules/payments/accounts) for the cross-connector model. ## Linking a user Plaid's auth ceremony runs through Plaid Link. The Payments surface: 1. **Create a PSU** — `v3CreatePaymentServiceUser` returns a PSU ID. 2. **Forward to Plaid** — `v3ForwardPaymentServiceUserToProvider` calls `/link/token/create` and stores the `link_token` as PSU metadata. 3. **Create a Link session** — `v3CreateLinkForPaymentServiceUser` returns a public Link URL + `attemptID`. 4. **Frontend redirect** — the user picks a bank, authenticates, and lands on your `clientRedirectURL`. 5. **Item-creation webhook** — the connector exchanges the `public_token` for an `access_token` and stores it on the connection. Step-by-step walkthrough on the [Open Banking Getting Started guide](/modules/payments/connectors/open-banking/getting-started). ### Redirect URL requirements - **HTTPS** in production. - **Registered** in the Plaid dashboard under **Developers → API → Allowed redirect URIs** before issuing a Link token targeting it. The connector doesn't auto-register; mismatches surface as `INVALID_OAUTH_STATE_PARAMETER` at the Link step. - Mobile apps must use **Plaid's universal-link** patterns ([iOS](https://plaid.com/docs/link/ios/), [Android](https://plaid.com/docs/link/android/)). The connector returns the URL unchanged — universal-link routing is a frontend concern. ## Asset model Multi-currency, formatted to UMN at ISO 4217 precision (`USD/2`, `EUR/2`, `GBP/2`). Amounts arrive as decimal strings; the connector applies major-to-minor scaling. ## Status mapping Plaid transactions are non-stateful — Plaid surfaces them only after they post, so the connector emits each as `SUCCEEDED`. The `pending` flag maps to `PENDING` until Plaid clears it and the row's `pending_transaction_id` is replaced by a permanent `transaction_id`; at that point the connector swaps the reference and moves the row to `SUCCEEDED`. ## Metadata keys Under `com.plaid.spec/`: - **Account**: `account_id`, `mask` (account-number last4), `name`, `official_name`, `subtype` (`checking` / `savings` / `credit card` / …), `verification_status`. - **External account**: `account_number_last4`, `routing_number`, `wire_routing_number` (where available). - **Payment**: `transaction_id`, `pending_transaction_id` (when `pending=true`), `category`, `category_id`, `merchant_name`, `personal_finance_category.primary`, `payment_channel`, `iso_currency_code`. PSU-level metadata also carries `user_token` and `link_token` — connector internals, not editable. ## Workflow tree ```text FetchAccounts (periodic) — per PSU/Item ├── FetchBalances (FromPayload — no extra API call) └── FetchPayments (periodic) — per PSU/Item, /transactions/sync cursor FetchExternalAccounts (periodic) CreateWebhooks — one hook per Item, provisioned automatically on link ``` ## Pagination and recovery `/transactions/sync` is cursor-based and idempotent. The connector persists the latest cursor per Item in platform-managed `State`; restarts resume from the last committed cursor (Plaid's at-least-once semantics aside). ## Known gaps - **Identity** and **Income** products aren't surfaced — only Auth, Transactions, and (when present) Transfer. - **Investment accounts** are surfaced as accounts but their holdings are not — only depository balances and transactions land. - **Webhooks** failing signature verification are dropped silently with a single-line log. Look for `plaid: webhook signature failed verification`. --- ## Tink Source: https://docs.formance.com/modules/payments/connectors/open-banking/tink The Tink connector links Formance to Tink's European bank-aggregation platform. It drives the [Open Banking PSU flow](/modules/payments/connectors/open-banking/getting-started) across Tink's markets: create a Payment Service User, forward them to Tink, hand them a Tink Link session, and sync the resulting accounts, balances, and transactions back through Payments. Available from Payments 3.2.0. The Tink connector requires Payments **3.2.0 or higher**. Your stack pins an older version — upgrade to use it. ## Prerequisites You need a Tink account and a `clientID` + `clientSecret` pair. Tink uses OAuth2 client-credentials; the connector exchanges the credentials for a short-lived bearer token and refreshes automatically. Covered EU markets: AT, BE, DE, DK, EE, ES, FI, FR, GB, IE, IT, LV, LT, NL, NO, PL, PT, SE. Tink enforces market restriction at the Link session. ## Installation {"fctl payments connectors install tink config.json"} ### Configuration fields `endpoint` is `https://api.tink.com`. No separate sandbox endpoint — Tink uses test users on the same host. ## Capabilities - `FETCH_ACCOUNTS` — depository accounts per PSU connection. - `FETCH_BALANCES` — available + booked balance per account. - `FETCH_EXTERNAL_ACCOUNTS` — counterparty accounts when Tink Payments identifies them. - `FETCH_PAYMENTS` — transactions, paginated via `pageToken`. - `CREATE_WEBHOOKS` + `TRANSLATE_WEBHOOKS` — webhooks per PSU user for account-data-refreshed and consent-expiry events. The connector doesn't initiate transfers or payouts — Tink Payments (the outbound product) isn't wired here. ## Account model Every Payments internal account is one Tink `Account` (a depository account on a linked connection). The `reference` is the Tink account ID; `name` is the account name; `defaultAsset` is null — Tink's account payload doesn't surface currency at the account level reliably, so balance assets are inferred per-cycle via `FETCH_BALANCES`. Accounts are PSU-scoped — `psuID` tracks the PSU and `openBankingConnectionID` is set when Tink's webhook payload carries the connection ID. EXTERNAL accounts are emitted when Tink Payments identifies them. See [Accounts](/modules/payments/accounts) for the cross-connector model. ## Linking a user Tink's auth ceremony runs through Tink Link: 1. **Create a PSU** — `v3CreatePaymentServiceUser` returns a PSU ID. 2. **Forward to Tink** — `v3ForwardPaymentServiceUserToProvider` creates a Tink user and stores the `user_id` as PSU metadata. 3. **Create a Link session** — `v3CreateLinkForPaymentServiceUser` returns a `https://link.tink.com/1.0/transactions/...` URL + `attemptID`. 4. **Frontend redirect** — the user picks a bank, completes SCA, and lands on your `clientRedirectURL` with an authorization code on the query string. 5. **Refresh-finished webhook** — the connector exchanges the code for a `refresh_token` and binds the connection. Step-by-step walkthrough on the [Open Banking Getting Started guide](/modules/payments/connectors/open-banking/getting-started). ### Redirect URL requirements - **HTTPS** in production. - **Registered** under your Tink client's allowed redirect URIs. - Mobile apps follow Tink's [Android](https://docs.tink.com/resources/transactions/optimize-your-transactions-integration) and [iOS](https://docs.tink.com/resources/transactions/optimize-your-transactions-integration) optimization guides for the in-app browser flow. ## Asset model Multi-currency, formatted to UMN at ISO 4217 precision. Amounts are already in minor units. ## Status mapping Tink transactions arrive in three lifecycle states: | Tink `status` | Payment `status` | | --- | --- | | `PENDING`, `UNDEFINED` | `PENDING` | | `BOOKED` | `SUCCEEDED` | | anything else | `UNKNOWN` | When a `PENDING` transaction matures to `BOOKED`, Tink replaces the provisional ID with a permanent one — the connector swaps the reference and moves the row to `SUCCEEDED`. ## Metadata keys Under `com.tink.spec/`: - **Account**: `account_id`, `name`, `iban`, `bic`, `account_number`, `holder_name`, `type` (`CHECKING` / `SAVINGS` / `CREDIT_CARD` / …), `flags`. - **External account**: `counterparty_name`, `counterparty_account_number`, `counterparty_iban`. - **Payment**: `transaction_id`, `provider_transaction_id`, `merchant_category_code`, `description`, `payee_message`, `payer_message`, `transaction_code`. PSU-level metadata carries `user_id` — the Tink user the connector created. ## Workflow tree ```text FetchAccounts (periodic) — per PSU ├── FetchBalances (FromPayload — no extra API call) └── FetchPayments (periodic) — per PSU/account, pageToken cursor FetchExternalAccounts (periodic) CreateWebhooks — provisioned automatically on link ``` ## Pagination and recovery `/data/v2/transactions` uses `pageToken`-based pagination. The connector persists the cursor per PSU/account in platform-managed `State`; restarts resume from the last committed cursor. ## Known gaps - **Tink Payments** (outbound initiation) is not wired. - **Identity** and **Income** products aren't surfaced. - **Consent expiry**: connections expire after the PSD2 SCA window (~90 days). Tink emits `consent_expired`; the connector translates it to a `UserConnectionPendingDisconnect` → `UserConnectionDisconnected` pair. Trigger a fresh link via `v3UpdateLinkForPaymentServiceUser` to renew. --- ## Reference Source: https://docs.formance.com/modules/numscript/reference ## Numscript Reference Language reference for Numscript — the transaction scripting language used by the Formance Ledger. --- ## Powens Source: https://docs.formance.com/modules/payments/connectors/open-banking/powens The Powens connector links Formance to Powens' European bank-aggregation platform. It drives the [Open Banking PSU flow](/modules/payments/connectors/open-banking/getting-started) across Powens' markets: create a Payment Service User, forward them to Powens, hand them a Powens Webview session, and sync the resulting accounts, balances, and transactions back through Payments. Available from Payments 3.2.0. The Powens connector requires Payments **3.2.0 or higher**. Your stack pins an older version — upgrade to use it. ## Prerequisites You need a Powens domain (`*.biapi.pro`) and a `clientID` + `clientSecret` + `configurationToken` triplet. Each PSU is authenticated by a user-scoped permanent access token derived from the configurationToken at registration; the connector mints and stores the token per PSU automatically. `maxConnectionsPerLink` caps the number of bank connections per Webview session — Powens lets you raise the cap; pick the depth of integration you want. ## Installation {"fctl payments connectors install powens config.json"} ### Configuration fields `domain` is the tenant-scoped subdomain Powens issued (e.g. `acme.biapi.pro`). `endpoint` is the API root for that domain, typically `https:///2.0`. ## Capabilities - `FETCH_ACCOUNTS` — depository, credit, and loan accounts per PSU connection. - `FETCH_BALANCES` — available + booked balance per account. - `FETCH_EXTERNAL_ACCOUNTS` — counterparty accounts when Powens identifies them. - `FETCH_PAYMENTS` — transactions, paginated via `limit` + `offset`. - `CREATE_WEBHOOKS` + `TRANSLATE_WEBHOOKS` — subscribes to `USER_SYNCED`, `CONNECTION_SYNCED`, `CONNECTION_DELETED`, …. The connector doesn't initiate transfers or payouts — Powens' outbound product is not wired here. ## Account model Every Payments internal account is one Powens `BankAccount` from a linked connection. The `reference` is the account `ID` stringified; `name` is the `original_name`; `defaultAsset` is `currency.id` at the precision Powens publishes. Accounts are PSU-scoped — `psuID` and `openBankingConnectionID` track the link. When Powens reports an `error` on the account, it lands in metadata. EXTERNAL accounts are emitted when Powens identifies them. See [Accounts](/modules/payments/accounts) for the cross-connector model. ## Linking a user Powens' auth ceremony runs through Powens Webview: 1. **Create a PSU** — `v3CreatePaymentServiceUser` returns a PSU ID. 2. **Forward to Powens** — `v3ForwardPaymentServiceUserToProvider` creates a Powens user with the `configurationToken`, mints a permanent access token, and stores both as PSU metadata. 3. **Create a Webview session** — `v3CreateLinkForPaymentServiceUser` returns a `https://webview.powens.com/...` URL + `attemptID`. 4. **Frontend redirect** — the user picks a bank, completes SCA, and lands on your `clientRedirectURL`. 5. **`USER_SYNCED` webhook** — the connector ingests it and starts syncing the linked connections. Step-by-step walkthrough on the [Open Banking Getting Started guide](/modules/payments/connectors/open-banking/getting-started). ### Redirect URL requirements - **HTTPS** in production. - **Registered** under the Powens client's allowed redirect URIs. - Mobile apps must follow Powens' [Webview browser-integration guide](https://docs.powens.com/api-reference/overview/webview#browser-integration) — in-app webviews aren't supported by every bank in the network. ## Asset model Multi-currency, formatted to UMN at ISO 4217 precision. Amounts arrive as decimal strings; the connector applies major-to-minor scaling. ## Status mapping Powens transactions are non-stateful; they appear after they post: | Powens `coming` flag | Payment `status` | | --- | --- | | `true` (provisional / pending settlement) | `PENDING` | | `false` (posted) | `SUCCEEDED` | When a `PENDING` transaction matures, Powens replaces the provisional ID with a permanent one and the connector swaps the reference. ## Metadata keys Under `com.powens.spec/`: - **Account**: `account_id`, `iban`, `bic`, `account_number`, `name`, `type` (`checking` / `savings` / `card` / `loan` / …), `usage` (`PRIV` / `ORGA`). - **External account**: `counterparty_id`, `account_number`, `iban`, `name`. - **Payment**: `transaction_id`, `category_id`, `description`, `original_wording`, `simplified_wording`, `last_update`, `type` (`transfer` / `card` / `bank` / …). PSU-level metadata carries `user_id` (the Powens user) and `expires_in` (token remaining lifetime). ## Workflow tree ```text FetchAccounts (periodic) — per PSU ├── FetchBalances (FromPayload — no extra API call) └── FetchPayments (periodic) — per PSU/account, offset/limit cursor FetchExternalAccounts (periodic) CreateWebhooks — provisioned at install ``` ## Pagination and recovery `limit` + `offset`. Watermarks persist per PSU/account in platform-managed `State`; restarts resume from the last committed offset. ## Known gaps - **Outbound initiation** is not wired. - **Investment accounts** surface as accounts but their positions don't — only depository balances and transactions land. - **Consent expiry**: connections expire after the PSD2 SCA window. Powens emits `CONNECTION_PENDING_DISCONNECT`; the connector translates it to `UserConnectionPendingDisconnect` — trigger a fresh Webview via `v3UpdateLinkForPaymentServiceUser` to renew. --- ## Connectors Source: https://docs.formance.com/modules/payments/connectors Formance Payments supports three categories of connectors for integrating with financial service providers. **PSP Connectors** integrate with traditional payment service providers like Stripe, Adyen, and Wise — syncing payments, accounts, and balances into Formance. **Exchange Connectors** integrate with crypto and asset exchanges like Coinbase Prime and Fireblocks — syncing wallets, balances, trading orders, and conversions alongside payments. **Open Banking** connectors link to aggregators like Plaid, Tink, and Powens — enabling end users to securely connect their bank accounts and share account data. ## Edition Connectors split across the two Formance editions. The standard PSP connectors and every Open Banking connector ship in the **Community edition**. The Exchange connectors (Coinbase Prime, Fireblocks, Bitstamp) and the two EE PSPs (Routable, Banking Bridge) are part of the **Enterprise edition** and require an EE license to install. The [Capabilities](/modules/payments/capabilities) page lists every connector's edition alongside its supported capabilities — the matrix is derived from upstream code at build time, so it stays in sync with what actually ships in each Payments minor. --- ## How it Works Source: https://docs.formance.com/modules/payments/connectors/generic/how-it-works The Generic Connector for Formance Payments provides a way to connect your Formance Stack with Financial Service Providers that are not natively supported by Formance. Formance Payments interacts with the remote Financial Service Provider through the Generic Connector as follows: ```mermaid sequenceDiagram Payments ->> Generic Connector: Poll the data activate Generic Connector Generic Connector -->> Payments: Ok par Account list Generic Connector ->> PSP: Request account list PSP -->> Generic Connector: Account list loop for each account Generic Connector ->> PSP: Request account balance PSP -->> Generic Connector: Account balance end Generic Connector -->> Payments: Account list Generic Connector ->> PSP: Request account transactions PSP -->> Generic Connector: Account transactions Generic Connector -->> Payments: Transactions list and Beneficiary list Generic Connector ->> PSP: Request beneficiary list PSP -->> Generic Connector: Beneficiary list Generic Connector -->> Payments: Beneficiary list end deactivate Generic Connector ``` The Generic Connector is in charge of polling the data from the Financial Service Provider and then sending it to Formance Payments. It polls the following data: - The list of accounts available in the Financial Service Provider and their associated balances and transactions - The list of beneficiaries available in the Financial Service Provider for payouts ## Integration with the Financial Service Provider The Generic Connector interacts with the Financial Service Provider by sending requests formatted according to the contract defined in [`generic-openapi.yaml`](https://github.com/formancehq/payments/blob/main/internal/connectors/plugins/public/generic/client/generic-openapi.yaml) and expecting responses formatted according to the same contract. As a consequence, it is necessary to create a service on your side that will interact with the Financial Service Provider and expose the data in the expected format. ### Endpoints your service must expose The connector calls six endpoints on the service URL you configured at install time. Every endpoint is mounted at the root and authenticates via the `Authorization: Bearer ` header. | Method | Path | Purpose | Payments capability | | --- | --- | --- | --- | | `GET` | `/accounts?pageSize=&page=&sort=&createdAtFrom=` | List internal accounts. | `FETCH_ACCOUNTS` | | `GET` | `/accounts/{accountId}/balances` | Return point-in-time balance(s) for an internal account. | `FETCH_BALANCES` | | `GET` | `/beneficiaries?pageSize=&page=&sort=&createdAtFrom=` | List external accounts (beneficiaries). | `FETCH_EXTERNAL_ACCOUNTS` | | `GET` | `/transactions?pageSize=&page=&sort=&updatedAtFrom=` | List transactions updated after `updatedAtFrom`. | `FETCH_PAYMENTS` | | `POST` | `/payouts` | Initiate a payout to a beneficiary. Idempotent via `idempotencyKey` on the body. | `CREATE_PAYOUT` | | `POST` | `/transfers` | Initiate a transfer between two internal accounts. Idempotent via `idempotencyKey` on the body. | `CREATE_TRANSFER` | ### Required behaviors - **Pagination** — the list endpoints (`/accounts`, `/beneficiaries`, `/transactions`) accept `pageSize` and `page` query parameters. `pageSize` defaults to 100; honour it. The connector advances `page` until the response returns fewer than `pageSize` items, at which point it considers the cycle complete. - **Incremental sync** — `/transactions` carries an `updatedAtFrom` query parameter that the connector seeds with the latest watermark from the previous cycle. Return only transactions updated at or after that timestamp. The connector dedupes by the `id` field on the response, so re-emitting the boundary row is safe (and recommended — it covers clock-skew edges). - **Idempotency** — `POST /payouts` and `POST /transfers` both carry an `idempotencyKey` on the request body. If your service sees the same key twice, return the original response (HTTP 201) rather than initiating a duplicate. - **Error envelope** — failures return any non-`2xx` status with a JSON body shaped as `{"errorCode": "...", "errorMessage": "..."}`. The connector surfaces both fields in the resulting Payment adjustment. The full OpenAPI document for the contract lives at [`internal/connectors/plugins/public/generic/client/generic-openapi.yaml`](https://github.com/formancehq/payments/blob/main/internal/connectors/plugins/public/generic/client/generic-openapi.yaml). That's the source of truth — fields you don't see on this page are still defined there (request bodies, response schemas, enum values). The typical deployment of the Generic Connector is as follows: ```mermaid graph LR Payments["Payments Core"] connector["Generic Connector"] service["Integration Service"] psp["Financial Service Provider"] subgraph Formance Payments Payments <--> connector end subgraph Your infrastructure connector <-- HTTP --> service end service <-- PSP specific protocol --> psp ``` ## Polling mechanism The Generic Connector uses a state-based approach for polling payment data efficiently. ### How it works 1. The connector stores the timestamp of the last successful data retrieval 2. In subsequent polls, it uses this timestamp to fetch only new or updated data 3. The `UpdatedAtFrom` query parameter is passed in API calls to your service 4. This parameter indicates that only transactions from that specific point in time should be returned ### Data storage and updates The system doesn't fetch all data fresh every time it polls. Instead: - Each batch of data received is stored - The internal state is updated with the latest timestamp of the data received - In the next polling cycle, the updated timestamp is used to fetch only new or changed data ### Benefits This polling method offers several advantages: - **Reduced data transfer**: Only fetches new or updated information - **Minimized load**: Reduces strain on both Formance and your API - **No duplicates**: Ensures the database stays up-to-date without duplicating existing data When setting up the Generic Connector, ensure that your API can handle and respond correctly to the `UpdatedAtFrom` query parameter. This allows the system to efficiently retrieve only the necessary data during each polling cycle. ## Authentication When instantiating the Generic Connector, you will need to pass an API key that will be used to authenticate the requests to your service. The Generic Connector will send requests with the API key in the `Authorization` header so that your service can authenticate the requests. Example: ``` Authorization: Bearer ``` --- ## Adyen Source: https://docs.formance.com/modules/payments/connectors/psp/adyen The Adyen connector syncs merchant accounts from Adyen's Management API and ingests Adyen's webhook stream. It does not poll for payments or balances — Adyen's canonical transaction surface is the webhook feed, which the connector subscribes to at install time. ## Prerequisites You need an Adyen Management API key with read access to `MerchantAccount` for the company. Webhooks use Basic Auth credentials (`webhookUsername` / `webhookPassword`) set on the webhook config in Adyen's Customer Area and sent as `Authorization: Basic ` on every delivery. ## Installation {"fctl payments connectors install adyen config.json"} ### Configuration fields `liveEndpointPrefix` only applies in production — copy it from Adyen's Customer Area under **Developers → API URLs** (the company-specific fragment between `https://` and `-checkout-live.adyenpayments.com`). Leave it empty for the test environment. ## Capabilities - `FETCH_ACCOUNTS` — merchant accounts via `GET /Management/v3/merchants`. - `CREATE_WEBHOOKS` — provisions a "Standard webhook" on the merchant at install, configured with the Basic Auth credentials from the config. - `TRANSLATE_WEBHOOKS` — converts `AUTHORISATION`, `CAPTURE`, `REFUND`, `CHARGEBACK`, `CANCELLATION`, etc. to `PSPPayment`s. Payments, balances, and external accounts are deliberately not polled — Adyen's canonical transaction surface is webhooks. Ensure the webhook is reachable (HTTPS, valid certificate) and the Basic Auth credentials match the config. ## Account model Every Payments internal account is one Adyen **merchant account** from `GET /Management/v3/merchants`. The `reference` is the merchant ID; `name` is the display name; `defaultAsset` is null (Adyen merchants are multi-currency). No EXTERNAL accounts are emitted — Adyen "destination accounts" are not pulled (see [Known gaps](#known-gaps)). See [Accounts](/modules/payments/accounts) for the cross-connector model. ## Asset model Currencies are uppercase ISO 4217 (`USD`, `EUR`, `JPY`), formatted to UMN at standard precision. Amounts are already in minor units — no scaling. ## Status mapping Webhook events combine `success: true|false` with `eventCode`: | Adyen `eventCode` + `success` | Payment `status` | | --- | --- | | `AUTHORISATION` + `true` | `PENDING` (until capture) | | `CAPTURE` + `true`, `REFUND` + `true` | `SUCCEEDED` | | `AUTHORISATION` + `false`, `REFUND_FAILED`, `CHARGEBACK` | `FAILED` | | `CANCELLATION` + `true` | `CANCELLED` | Full event-code → type/status mapping in [`webhooks.go`](https://github.com/formancehq/payments/blob/main/internal/connectors/plugins/public/adyen/webhooks.go). ## Metadata keys Under `com.adyen.spec/`: - **Account**: `merchant_account_code`, `description`, `data_centers`. - **Payment**: `event_code`, `psp_reference`, `merchant_reference`, `payment_method`, `payment_method_variant`, `acquirer_code`. ## Workflow tree ```text FetchAccounts (periodic — merchant accounts only) CreateWebhooks (one-shot at install) ``` ## Known gaps - **Balances and payouts** are not exposed — Adyen's payout flow is bank-side, outside the Management API. - **External accounts** (Adyen "destination accounts") are not pulled. - **Transfer / Payout Initiation** is not implemented; outbound flows live in Adyen's Customer Area. --- ## Atlar Source: https://docs.formance.com/modules/payments/connectors/psp/atlar The Atlar connector polls an Atlar workspace and surfaces accounts, counterparties (external accounts), and transactions. Atlar's transactional model is bank-rail flat (SEPA, BACS, ACH), so the connector uses one shared shape across rails. ## Prerequisites You need an Atlar workspace and a pair of HMAC credentials (`accessKey` + `secret`). Atlar signs every request with HMAC-SHA256; the connector signs internally — you only supply the credentials. ## Installation {"fctl payments connectors install atlar config.json"} ### Configuration fields `baseUrl` defaults to `https://api.atlar.com`; override only when Atlar provisions you against a dedicated sandbox or staging URL. ## Capabilities - `FETCH_ACCOUNTS` — internal accounts via `/v1/accounts`. - `FETCH_EXTERNAL_ACCOUNTS` — counterparties via `/v1/counterparties`. - `FETCH_PAYMENTS` — `/v1/transactions`, classified as PAY-IN / PAYOUT / TRANSFER by direction. - `FETCH_OTHERS` — `external-payments` (incoming SCT/SDD instructions awaiting reconciliation), surfaced under the `external-payment` collection on `v3ListPaymentsOther`. Outbound transfer/payout initiation, bank-account creation, and webhooks are not exposed on the Atlar API surface this connector reaches. ## Account model Every Payments internal account is one Atlar `/v1/accounts` row. The `reference` is the Atlar account `ID`; `name` is the account name; `defaultAsset` is the account currency at ISO 4217 precision. EXTERNAL accounts come from `/v1/counterparties`. Metadata stamps the underlying `bank/id`, `bank/name`, `bank/bic`, IBAN and other identifiers per market+type, plus `alias`, `owner/name`, and the `fictive` flag. See [Accounts](/modules/payments/accounts) for the cross-connector model. ## Asset model Uppercase ISO 4217 (`EUR`, `GBP`, `USD`), formatted to UMN at standard precision. Amounts are already in minor units — no scaling. ## Status mapping | Atlar transaction `status` | Payment `status` | | --- | --- | | `INITIATED`, `PENDING`, `SUBMITTED`, `INFLIGHT` | `PENDING` | | `EXECUTED`, `REGISTERED`, `RECONCILED` | `SUCCEEDED` | | `FAILED`, `REJECTED` | `FAILED` | | `CANCELLED` | `CANCELLED` | | anything else | `UNKNOWN` | ## Metadata keys Under `com.atlar.spec/`: - **Account**: `account_id`, `bank_iban`, `bank_bic`, `account_type`, `currency_code`. - **External account**: `counterparty_id`, `name`, `account_number`, `routing_number`, `bic`, `country`. - **Payment**: `transaction_id`, `transaction_type`, `bank_reference`, `remittance_information_unstructured`, `direction`. ## Workflow tree ```text FetchAccounts (periodic) └── FetchPayments (periodic) — per account FetchExternalAccounts (periodic) FetchOthers (periodic) — external-payments awaiting reconciliation ``` ## Known gaps - **Balances** are not exposed — Atlar's `/v1/accounts` payload doesn't carry a real-time balance. - **Outbound initiation** (`CreateTransfer` / `CreatePayout`) is not implemented; Atlar's payment-initiation API requires payment-plan scaffolding without a Payments-side model. - **Webhooks** are not wired; reconciliation runs on the polling cycle. --- ## Banking Circle Source: https://docs.formance.com/modules/payments/connectors/psp/bankingcircle The Banking Circle connector polls a Banking Circle tenant and surfaces settlement accounts, balances, and transactions. It also initiates transfers and payouts, and registers new beneficiaries on Banking Circle's side. ## Prerequisites You need a Banking Circle tenant and a mutual-TLS-capable API user. Authentication combines a username/password pair (for OAuth2) with a client certificate (`userCertificate` + `userCertificateKey`) on every call. The cert is PEM-encoded; supply certificate and key as multi-line strings in the config. Banking Circle exposes distinct base URLs for the authorization endpoint (`https://authorizationsandbox.bankingcircleconnect.com` in sandbox) and the data endpoint (`https://sandbox.bankingcircleconnect.com`). Both are required. ## Installation {"fctl payments connectors install bankingcircle config.json"} ### Configuration fields ## Capabilities - `FETCH_ACCOUNTS` — settlement and customer accounts via `GET /api/v1/accounts`. - `FETCH_BALANCES` — available + intraday balance per account. - `FETCH_PAYMENTS` — transactions, classified PAY-IN / PAYOUT / TRANSFER by direction. - `CREATE_BANK_ACCOUNT` — registers a beneficiary from a `v3CreateBankAccount` call. - `CREATE_TRANSFER` — `POST /api/v1/payments/singles` between two Banking Circle accounts. - `CREATE_PAYOUT` — `POST /api/v1/payments/singles` to a registered beneficiary. Webhooks are not exposed on the API surface this connector reaches; refresh runs on the polling cycle. ## Account model Every Payments internal account is one Banking Circle account from `/api/v1/accounts` — typically a settlement or customer ledger. The `reference` is the Banking Circle `accountId`; `name` is the description; `defaultAsset` is the account's `currency`. EXTERNAL accounts aren't pulled, but `v3CreateBankAccount` (via `CREATE_BANK_ACCOUNT`) registers a beneficiary on Banking Circle and links the result back. See [Accounts](/modules/payments/accounts) for the cross-connector model. ## Asset model Uppercase ISO 4217 (`EUR`, `GBP`, `USD`, …), formatted to UMN at standard precision. Amounts arrive as decimal strings; the connector applies major-to-minor scaling. ## Status mapping | Banking Circle transaction `status` | Payment `status` | | --- | --- | | `Pending`, `Processed` (booked but not settled) | `PENDING` | | `Settled`, `Completed` | `SUCCEEDED` | | `Failed`, `Rejected` | `FAILED` | | `Cancelled` | `CANCELLED` | `Create*` initiations schedule `PollTransferStatus` / `PollPayoutStatus` against the singles-payment endpoint until terminal. ## Metadata keys Under `com.bankingcircle.spec/`: - **Account**: `account_id`, `iban`, `bic`, `currency_code`, `account_type`. - **Payment**: `transaction_id`, `transaction_reference`, `end_to_end_id`, `payment_method`, `clearing_system`. - **External account**: `beneficiary_id`, `country`, `iban`, `bic`, `routing_number`. ## Workflow tree ```text FetchAccounts (periodic) ├── FetchBalances (periodic) — per account └── FetchPayments (periodic) — per account CreateBankAccount / CreateTransfer / CreatePayout (event-driven) └── PollTransferStatus / PollPayoutStatus until terminal ``` ## Pagination and recovery Banking Circle transactions are `pageNumber` + `pageSize`. The connector persists the latest watermark per account in platform-managed `State` and re-fetches one cycle back on restart, deduping by `PSPPayment.Reference`. ## Known gaps - **Webhooks** are not wired. - **Reversals** (`ReverseTransfer` / `ReversePayout`) are not implemented; corrections live in Banking Circle's UI. - **External accounts via FETCH** are not pulled — beneficiaries are managed via `CreateBankAccount` and persisted Payments-side. --- ## Coinbase Prime Source: https://docs.formance.com/modules/payments/connectors/exchange/coinbaseprime The Coinbase Prime connector polls a Coinbase Prime portfolio and surfaces its wallets, balances, payments, trading orders, and conversions to the Payments service. It is read-only. ## Prerequisites The Coinbase Prime connector requires Payments module **3.2.0 or higher** for Accounts, Balances, and Payments. Your stack pins an older version — upgrade to use it. Your stack supports Accounts, Balances, and Payments via Coinbase Prime. **Orders and Conversions** require Payments module 3.3.0 or higher. You need a Coinbase Prime account and an API key with the minimum permissions for the capabilities you use. To create an API key in the Coinbase Prime console: Go to **Settings**, then **API**. Click **Create API Key** and select the portfolio you want to connect. Choose an expiration date as needed. If you set an expiration date, you will need to update the connector configuration with new credentials when the key expires. Set **Read** permissions — the connector is read-only and polls Accounts, Balances, Payments, Orders, and Conversions. Configure IP restrictions if your environment requires them. Save the **API key**, **API secret**, and **passphrase** — all three are required for the connector configuration. ## Installation {"fctl payments connectors install coinbaseprime config.json"} With `config.json` containing: ```json { "apiKey": "string", "apiSecret": "string", "name": "string", "passphrase": "string", "pollingPeriod": "30m", "portfolioId": "string" } ``` ### Configuration fields | Field | Required | Default | Description | |---|---|---|---| | `apiKey` | yes | — | Coinbase Prime API key. | | `apiSecret` | yes | — | Secret key from API-key creation. | | `name` | yes | — | Unique name for this connector instance. | | `passphrase` | yes | — | Passphrase from API-key creation. | | `pollingPeriod` | no | `30m` | Sync cadence. | | `portfolioId` | yes | — | Portfolio to connect. One connector per portfolio — install several to track more than one. | ## Capabilities - `FETCH_ACCOUNTS` — portfolio wallets. - `FETCH_BALANCES` — per-wallet balances. - `FETCH_PAYMENTS` — deposits, withdrawals, internal transfers, rewards, and staking flows. - `FETCH_ORDERS` — spot trading orders with full fill lifecycle. Requires Payments 3.3.0+. - `FETCH_CONVERSIONS` — atomic two-asset swaps (e.g. stablecoin redemption). Requires Payments 3.3.0+. Outbound initiation, webhooks, and external-account creation are not implemented — see [Known gaps](#known-gaps). ## Account model Every Payments internal account is one Coinbase Prime wallet in the configured portfolio. Coinbase Prime distinguishes wallet types — trading, vault, onchain, custody (`QC`), and `WALLET_TYPE_OTHER` — and only **trading wallets** settle orders. Order resolution is therefore restricted to trading wallets so each asset symbol resolves unambiguously. The `reference` is the `walletID`; `defaultAsset` is the wallet's symbol (e.g. `BTC/8`, `USD/2`). No EXTERNAL accounts are emitted — counterparty addresses surface in Payment metadata only. See [Accounts](/modules/payments/accounts) for the cross-connector model. ## Payments Coinbase Prime transactions land on the [Payments](/modules/payments/payments) stream, **excluding** rows of type `CONVERSION` (which feed [Conversions](#conversions)). Each upstream transaction lands on exactly one stream. ### Transaction type to Payment type | Coinbase transaction type | Payment `type` | | --- | --- | | `DEPOSIT`, `COINBASE_DEPOSIT`, `COINBASE_REFUND`, `REWARD`, `DEPOSIT_ADJUSTMENT`, `CLAIM_REWARDS` | `PAY-IN` | | `WITHDRAWAL`, `SWEEP_WITHDRAWAL`, `PROXY_WITHDRAWAL`, `BILLING_WITHDRAWAL`, `WITHDRAWAL_ADJUSTMENT`, `SLASH` | `PAYOUT` | | `INTERNAL_DEPOSIT`, `INTERNAL_WITHDRAWAL`, `SWEEP_DEPOSIT`, `PROXY_DEPOSIT`, `STAKE`, `RESTAKE`, `PORTFOLIO_STAKE`, `UNSTAKE`, `PORTFOLIO_UNSTAKE` | `TRANSFER` | | Chain-level events (`KEY_REGISTRATION`, `DELEGATION`, `VOTE_AUTHORIZE`, `ONCHAIN_TRANSACTION`, …) and any unrecognized type | `OTHER` | ### Status mapping Successful transactions → `SUCCEEDED`; failures and rejections → `FAILED`; mid-flight → `PENDING`. Cancellations and expirations map distinctly for dashboarding. | Coinbase transaction status | Payment `status` | | --- | --- | | `TRANSACTION_PENDING`, `TRANSACTION_CREATED`, `TRANSACTION_REQUESTED`, `TRANSACTION_APPROVED`, `TRANSACTION_GASSING`, `TRANSACTION_GASSED`, `TRANSACTION_PROVISIONED`, `TRANSACTION_PLANNED`, `TRANSACTION_PROCESSING`, `TRANSACTION_RESTORED`, `TRANSACTION_IMPORT_PENDING`, `TRANSACTION_DELAYED`, `TRANSACTION_BROADCASTING`, `TRANSACTION_CONSTRUCTED` | `PENDING` | | `TRANSACTION_DONE`, `TRANSACTION_IMPORTED` | `SUCCEEDED` | | `TRANSACTION_CANCELLED` | `CANCELLED` | | `TRANSACTION_EXPIRED` | `EXPIRED` | | `TRANSACTION_FAILED`, `TRANSACTION_REJECTED` | `FAILED` | | `OTHER_TRANSACTION_STATUS` | `OTHER` | | anything else | `UNKNOWN` | ### Amount and asset Amounts use the precision from Coinbase Prime's asset catalogue (`GetAssets` provides `decimal_precision` per symbol). Common assets: `BTC/8`, `ETH/18`, `USDC/6`, `USD/2`. Always trust the `asset` field on the Payment rather than assuming a precision. Transactions referencing an asset not in the catalogue are skipped (logged) and don't produce a Payment. ### Account resolution Each leg resolves independently: 1. If Coinbase populates `transfer_from.value` (or `transfer_to.value`) with `type == WALLET`, that wallet ID becomes the leg's account reference. 2. Otherwise the connector falls back to the transaction's `wallet_id`: `PAY-IN` → destination, `PAYOUT` → source. 3. External blockchain addresses (e.g. self-custody withdrawal) land in Payment metadata as `source_address` / `deposit_address`, never as account references. ### Metadata Under `com.formance.connectors.coinbaseprime.`: - **Always present**: `type`, `status`. - **Present when populated**: `wallet_id`, `portfolio_id`, `network`, `external_tx_id`, `source_address`, `deposit_address`, `completed_at`, `blockchain_ids`. - **Present when any fee is non-zero**: `fees`, `network_fees` (only non-zero), `fee_symbol`. ### Example response A 1.5 ETH withdrawal from a trading wallet to an external address: ```json { "id": "", "connectorID": "", "provider": "coinbaseprime", "reference": "tx_4f3a8e9d1c", "createdAt": "2026-04-30T08:14:22Z", "type": "PAYOUT", "amount": 1500000000000000000, "initialAmount": 1500000000000000000, "asset": "ETH/18", "scheme": "OTHER", "status": "SUCCEEDED", "sourceAccountID": "", "destinationAccountID": null, "metadata": { "com.formance.connectors.coinbaseprime.type": "WITHDRAWAL", "com.formance.connectors.coinbaseprime.status": "TRANSACTION_DONE", "com.formance.connectors.coinbaseprime.wallet_id": "wlt_eth_abc123", "com.formance.connectors.coinbaseprime.deposit_address": "0xabc1234567890def...", "com.formance.connectors.coinbaseprime.fees": "0.0021", "com.formance.connectors.coinbaseprime.fee_symbol": "ETH" } } ``` `destinationAccountID` is `null` because the destination is an external blockchain address, preserved in `metadata.deposit_address`. ## Orders Orders require Payments module 3.3.0 or higher. Coinbase Prime trading orders land on the [Orders](/modules/payments/orders) stream — one record per Coinbase order with an append-only `adjustments` list capturing each fill or status change. ### Direction and account legs | Direction | `sourceAsset` | `destinationAsset` | `sourceAccountID` | `destinationAccountID` | | --- | --- | --- | --- | --- | | `BUY` | quote | base | quote-currency trading wallet | base-currency trading wallet | | `SELL` | base | quote | base-currency trading wallet | quote-currency trading wallet | Both legs are restricted to **trading** wallets — vault, onchain, and custody are ineligible. If the trading wallet for a symbol hasn't been synced yet, the order page retries on the next cycle (no row is dropped). ### Status mapping Coinbase Prime statuses map directly to Payments statuses, with one exception: `PARTIALLY_FILLED` is computed locally — Coinbase doesn't expose it. An order becomes `PARTIALLY_FILLED` when the upstream status is `OPEN` and the filled quantity sits strictly between zero and the ordered quantity. | Coinbase status | Order `status` | | --- | --- | | `PENDING` | `PENDING` | | `OPEN` with no fills | `OPEN` | | `OPEN` with `0 < filled_quantity < base_quantity` | `PARTIALLY_FILLED` (computed) | | `FILLED` | `FILLED` | | `CANCELLED` | `CANCELLED` | | `EXPIRED` | `EXPIRED` | | `FAILED` | `FAILED` | ### Quantities, amounts, and prices | Coinbase field | Order field | Precision | | --- | --- | --- | | `base_quantity` | `baseQuantityOrdered` | base asset | | `filled_quantity` | `baseQuantityFilled` | base asset | | `filled_value` | `quoteAmount` | quote asset | | `commission` | `fee` (with `feeAsset` = quote) | quote asset | Coinbase Prime returns price strings derived from `float64`, so values like `"1825.6099999998417653"` carry float-encoding noise past roughly 10 decimals. The connector picks a **dynamic precision** equal to the maximum number of decimals across `limit_price`, `stop_price`, and `average_filled_price`, capped at 10, and parses prices with a truncating decimal parser. `priceAsset` may therefore declare a higher precision than `quoteAsset` (e.g. `quoteAsset = USD/2`, `priceAsset = USD/6`). ### Metadata Under `com.formance.connectors.coinbaseprime.`: `product_id`, `portfolio_id`, `client_order_id`, `quote_value`, `filled_value`, `order_total`, `exchange_fee`, `net_average_filled_price`, `historical_pov`, `quote_currency`, `price_asset`, `base_wallet_id`, `quote_wallet_id`, optional `post_only`. Commission breakdown adds `commission_total`, `commission_client`, `commission_venue`, `commission_ces`, `commission_financing`, `commission_regulatory`, `commission_clearing` when Coinbase returns the detail block. ### Example response A partially-then-fully-filled `BUY` of 0.5 BTC on `BTC-USD` at a 50,000 USD limit: ```json { "id": "", "reference": "ord_9c7e1a4b3d", "direction": "BUY", "sourceAsset": "USD/2", "destinationAsset": "BTC/8", "type": "LIMIT", "status": "FILLED", "timeInForce": "GOOD_UNTIL_CANCELLED", "baseQuantityOrdered": 50000000, "baseQuantityFilled": 50000000, "limitPrice": 5000000, "averageFillPrice": 4998750, "quoteAmount": 2499375, "quoteAsset": "USD/2", "priceAsset": "USD/2", "fee": 1250, "feeAsset": "USD/2", "adjustments": [ { "createdAt": "2026-04-30T09:00:05Z", "status": "PENDING", "baseQuantityFilled": 0 }, { "createdAt": "2026-04-30T09:00:30Z", "status": "OPEN", "baseQuantityFilled": 0 }, { "createdAt": "2026-04-30T09:08:45Z", "status": "PARTIALLY_FILLED", "baseQuantityFilled": 22500000, "fee": 562 }, { "createdAt": "2026-04-30T09:12:30Z", "status": "FILLED", "baseQuantityFilled": 50000000, "fee": 1250 } ] } ``` ## Conversions Conversions require Payments module 3.3.0 or higher. The connector queries Coinbase Prime's transactions feed filtered to type `CONVERSION` and emits one [Conversion](/modules/payments/conversions) per row. Coinbase Prime exposes a single `amount` field per conversion, so `sourceAmount` and `destinationAmount` are populated **1:1** (parsed at each side's precision) — correct for stablecoin redemption (USDC ↔ USD) where nominal value is identical. ### Asset and account resolution | Coinbase field | Conversion field | | --- | --- | | `symbol` | `sourceAsset` | | `destination_symbol` | `destinationAsset` | | `transfer_from.value` | `sourceAccountID` | | `transfer_to.value` | `destinationAccountID` | Rows with a missing symbol or one absent from the catalogue are skipped (logged). Account references are used directly without gating on `transfer_*.type`; either may be empty without blocking emission. ### Status mapping Conversions have a tighter lifecycle than Payments — no `CANCELLED` or `EXPIRED`. | Coinbase transaction status | Conversion `status` | | --- | --- | | `TRANSACTION_DONE`, `TRANSACTION_IMPORTED` | `COMPLETED` | | `TRANSACTION_FAILED`, `TRANSACTION_REJECTED`, `TRANSACTION_CANCELLED` | `FAILED` | | anything else | `PENDING` | ### Fees `fees` is parsed at the precision of `fee_symbol`, falling back to the source symbol when absent. If the fee currency isn't in the catalogue, only the fee is dropped — the conversion itself is still emitted. ### Metadata Under `com.formance.connectors.coinbaseprime.`: `transaction_id`, `type` (always present); `portfolio_id` (when populated). ### Example response A 10,000 USDC → USD conversion (no fee on stablecoin redemption): ```json { "id": "", "reference": "tx_d2b4a17e9c", "sourceAsset": "USDC/6", "destinationAsset": "USD/2", "sourceAmount": 10000000000, "destinationAmount": 1000000, "fee": null, "feeAsset": null, "status": "COMPLETED", "sourceAccountID": "", "destinationAccountID": "" } ``` ## Troubleshooting Orders and Conversions resolve their account legs against wallets pulled by Accounts. On a fresh install, the first Accounts cycle must complete before Order and Conversion pages succeed — they retry until then. Verify the trading wallet has synced: ```bash fctl payments accounts list --connector-id $CONNECTOR_ID | grep TRADING ``` If still missing after several cycles, check connector logs for an unsupported asset or a Coinbase API error. `PENDING` covers everything other than `TRANSACTION_DONE` / `TRANSACTION_IMPORTED` (success) or the failure family. Inspect the conversion's metadata for the exact upstream status: ```bash curl -s "$STACK/api/payments/v3/conversions/$CONVERSION_ID" \ -H "Authorization: Bearer $TOKEN" | jq '.data.metadata' ``` Cross-reference `transaction_id` against the Coinbase Prime portal. Amounts use the precision from Coinbase Prime's `GetAssets` (plus a small fiat fallback). Transactions referencing an asset not in the catalogue are **silently skipped** and logged — no Payment, Order, or Conversion is produced. Look for `unsupported currency` in connector logs. The asset starts flowing once Coinbase publishes precision metadata; re-trigger an Accounts sync to refresh the catalogue. Each capability schedule exposes its run history via the schedule APIs. See [Monitoring connector schedules](/modules/payments/operations#monitoring-connector-schedules). ## Known gaps - **Outbound initiation** — Coinbase Prime exposes withdrawal APIs, but the connector is read-only. `CreateTransfer` and `CreatePayout` are not wired. - **Webhooks** — not consumed; refresh runs on the polling cycle. - **Historical Orders and Conversions** — read from the activity feed forward from the first install cycle; pre-install activity is not back-filled. - **Cross-portfolio aggregation** — one connector per `portfolioID`. Install several to track multiple portfolios. --- ## Column Source: https://docs.formance.com/modules/payments/connectors/psp/column The Column connector polls a Column bank-as-a-service tenant and surfaces accounts, balances, counterparties, and payments. It covers the full US-bank rail set (ACH credit/debit, wire, book transfer) for both observation and outbound initiation, plus webhook ingest. ## Prerequisites You need a Column tenant and an API key with read access to accounts and transfers, plus write access for the rails you initiate (`ach.create`, `wire.create`, `book.create`, `counterparty.create`). ## Installation {"fctl payments connectors install column config.json"} ### Configuration fields `endpoint` defaults to `https://api.column.com`. Sandbox uses the same host with sandbox-scoped keys — no separate base URL. ## Capabilities - `FETCH_ACCOUNTS` — bank accounts via `GET /bank-accounts`. - `FETCH_BALANCES` — available + held + locked balance per account. - `FETCH_EXTERNAL_ACCOUNTS` — counterparties via `GET /counterparties`. - `FETCH_PAYMENTS` — ACH, wires, book transfers, and incoming credits, unified into `PSPPayment`. - `CREATE_BANK_ACCOUNT` — registers a counterparty via `v3CreateBankAccount`. - `CREATE_TRANSFER` — book transfer between two Column accounts. - `CREATE_PAYOUT` — ACH credit (`POST /transfers/ach`) or wire (`POST /transfers/wire`); rail selected via `com.column.spec/rail`. - `CREATE_WEBHOOKS` + `TRANSLATE_WEBHOOKS` — provisions an endpoint at install and ingests `ach.*`, `wire.*`, `book.*` events. ## Account model Every Payments internal account is one Column bank account from `GET /bank-accounts`. The `reference` is the Column account ID; `name` is the description; `defaultAsset` is `USD/2` (Column is USD-only on the bank-product side). Metadata stamps `type`, `bic`, `default_account_number`, `routing_number`, `is_overdraftable`, `owners`. EXTERNAL accounts come from `/counterparties`; `v3CreateBankAccount` registers new ones. See [Accounts](/modules/payments/accounts) for the cross-connector model. ## Asset model USD-only on the bank-product side — every balance and payment is `USD/2`. Amounts are already in minor units. ## Status mapping Column statuses vary per rail; the connector collapses them: | Column status (ACH / wire / book) | Payment `status` | | --- | --- | | `initiated`, `pending`, `submitted`, `manual_review`, `holding` | `PENDING` | | `completed`, `settled`, `posted` | `SUCCEEDED` | | `returned`, `failed`, `rejected` | `FAILED` | | `canceled` | `CANCELLED` | ACH credits land `settled` after the daily ACH return window — the connector folds the window into the `PENDING → SUCCEEDED` transition. ## Metadata keys Under `com.column.spec/`: - **Account**: `bank_account_id`, `account_number_last4`, `routing_number`, `account_type`. - **Counterparty (external account)**: `counterparty_id`, `account_number_last4`, `routing_number`, `account_type`, `name`. - **Payment**: `transfer_id`, `rail` (`ach` / `wire` / `book`), `direction`, `network_response_code`, `idempotency_key`, `description`. Outbound initiation requires the rail on the PaymentInitiation metadata: ```json { "metadata": { "com.column.spec/rail": "ach", "com.column.spec/sec_code": "PPD" } } ``` ## Workflow tree ```text FetchAccounts (periodic) ├── FetchBalances (periodic) — per account └── FetchPayments (periodic) — per account, per rail FetchExternalAccounts (periodic) CreateBankAccount / CreateTransfer / CreatePayout (event-driven) └── PollTransferStatus / PollPayoutStatus until terminal CreateWebhooks (one-shot at install) ``` ## Pagination and recovery Cursor-based pagination via `starting_after`. The connector persists cursors per stream in platform-managed `State` and resumes from the last committed cursor on restart. ## Known gaps - **Reverse transfer** is not implemented; ACH returns surface as new payment rows rather than reversal events. - **International wires** are supported only insofar as Column's API accepts the destination — no extra Payments-side metadata mapping for SWIFT BIC / IBAN. --- ## Currencycloud Source: https://docs.formance.com/modules/payments/connectors/psp/currencycloud The Currencycloud connector polls a Currencycloud account and surfaces sub-accounts, multi-currency balances, beneficiaries, and transactions (including FX conversions). It also initiates transfers between sub-accounts and payouts to registered beneficiaries. ## Prerequisites You need a Currencycloud account and a `loginID` + `apiKey` pair. Currencycloud uses a session token derived from those credentials; the connector manages the session and re-authenticates on expiry. ## Installation {"fctl payments connectors install currencycloud config.json"} ### Configuration fields `endpoint` is `https://devapi.currencycloud.com/v2/` for demo/sandbox, `https://api.currencycloud.com/v2/` for production. ## Capabilities - `FETCH_ACCOUNTS` — sub-accounts via `GET /accounts/find`. - `FETCH_BALANCES` — per-account, per-currency balances. - `FETCH_EXTERNAL_ACCOUNTS` — beneficiaries via `GET /beneficiaries/find`. - `FETCH_PAYMENTS` — transactions and conversions, classified PAY-IN / PAYOUT / TRANSFER. - `CREATE_TRANSFER` — `POST /transfers/create` between two sub-accounts. - `CREATE_PAYOUT` — `POST /payments/create` to a registered beneficiary. `CREATE_BANK_ACCOUNT` is not implemented; manage beneficiaries through Currencycloud directly. Webhooks are not wired. ## Account model Every Payments internal account is one Currencycloud sub-account (`POST /v2/accounts/find`). The `reference` is the account `id`; `name` is `account_name`; `defaultAsset` is null — sub-accounts are multi-currency, with one balance row per asset via `FETCH_BALANCES`. EXTERNAL accounts come from `/beneficiaries/find`. See [Accounts](/modules/payments/accounts) for the cross-connector model. ## Asset model Multi-currency, formatted to UMN at ISO 4217 precision. Amounts arrive as decimal strings; the connector applies major-to-minor scaling. ## Status mapping | Currencycloud `status` | Payment `status` | | --- | --- | | `pending`, `awaiting_authorization`, `submitted` | `PENDING` | | `completed`, `released`, `settled` | `SUCCEEDED` | | `failed`, `deleted` (reversed) | `FAILED` | | `cancelled`, `cancellation_requested` | `CANCELLED` | `CREATE_TRANSFER` and `CREATE_PAYOUT` schedule `PollTransferStatus` / `PollPayoutStatus` against `/transfers/{id}` or `/payments/{id}` until terminal. ## Metadata keys Under `com.currencycloud.spec/`: - **Account**: `account_id`, `account_name`, `legal_entity_type`, `your_reference`, `status`. - **External account**: `beneficiary_id`, `beneficiary_country`, `currency`, `account_number`, `iban`, `bic_swift`, `routing_code_value_1`. - **Payment**: `transaction_id`, `payment_id`, `transfer_id`, `conversion_id` (for FX legs), `payment_type` (`regular` / `priority`), `reason`. ## Workflow tree ```text FetchAccounts (periodic) ├── FetchBalances (periodic) — per account └── FetchPayments (periodic) — per account FetchExternalAccounts (periodic) CreateTransfer / CreatePayout (event-driven) └── PollTransferStatus / PollPayoutStatus until terminal ``` ## Pagination and recovery 1-indexed `page` + `per_page`. The connector persists the watermark per stream in platform-managed `State`; restarts resume from the last committed page boundary. ## Known gaps - **Webhooks** are not implemented. - **Beneficiary creation** through the Payments module is not wired — beneficiaries must be created upstream before payouts can target them. - **FX conversions** surface as `PSPPayment`s with `com.currencycloud.spec/conversion_id`, not as separate [Conversion](/modules/payments/conversions) entries (the Conversion resource is exchange-flavoured and gated to 3.3.0+). --- ## Fireblocks Source: https://docs.formance.com/modules/payments/connectors/exchange/fireblocks The Fireblocks connector polls a Fireblocks workspace and surfaces vault accounts, balances, and transactions as read-only streams. ## Prerequisites The Fireblocks connector requires Payments module **3.2.0 or higher**. Your stack pins an older version — upgrade to use it. You need a Fireblocks account with an API key carrying the minimum permissions for the capabilities you use. ## Installation {"fctl payments connectors install fireblocks config.json"} With `config.json` containing: ```json { "apiKey": "string", "endpoint": "string", "name": "string", "pollingPeriod": "30m", "privateKey": "string" } ``` ### Configuration fields | Field | Required | Default | Description | |---|---|---|---| | `apiKey` | yes | — | API key from the Fireblocks workspace. | | `endpoint` | yes | — | Fireblocks API base URL — depends on your workspace environment (see [Endpoints](#endpoints)). | | `name` | yes | — | Unique name for this connector instance. | | `pollingPeriod` | no | `30m` | Sync cadence. | | `privateKey` | yes | — | Fireblocks API secret. Signs the JWT for the `Authorization` header. Single-line string with literal `\n` characters. | Convert a PEM file to the expected single-line format with: ```bash cat fireblocks_private.pem | awk '{printf "%s\\n", $0}' ``` ### Endpoints | Environment | URL | |---|---| | US Sandbox | `https://sandbox-api.fireblocks.io` | | US Mainnet/Testnet | `https://api.fireblocks.io` | | EU Mainnet/Testnet | `https://eu-api.fireblocks.io` | | EU2 Mainnet/Testnet | `https://eu2-api.fireblocks.io` | ## Capabilities Read-only — no payouts, transfers, or webhook deliveries. - **FetchAccounts** — vault accounts. - **FetchBalances** — per-asset balances per vault. - **FetchPayments** — transactions, including withdrawals, transfers, staking/unstaking, and minting/burning. ## Account model Every Payments internal account is one Fireblocks **vault account** (the multi-asset container that holds wallets across chains). The `reference` is the vault `id`; `name` is the vault name; `defaultAsset` is null — vaults are multi-asset, with balances per asset via `FETCH_BALANCES` (same-symbol holdings across chains collapse into one row; see [Multi-chain aggregation](#multi-chain-aggregation)). Vault-specific fields land under [account metadata](#account-metadata). No EXTERNAL accounts are emitted — Fireblocks counterparties live in the workspace UI. See [Accounts](/modules/payments/accounts) for the cross-connector model. ## Asset model The canonical asset string is derived from Fireblocks' `displaySymbol`, sanitized where needed, and suffixed with `/precision` (`USDT/6`, `ETH/18`). The Fireblocks `legacyId` (`USDT_ERC20`, `ETH_TEST5`, …) is preserved on every payment under `com.fireblocks.spec/legacy_id`. ### Multi-chain aggregation Same-symbol holdings across chains in one vault collapse into a single balance row. A vault holding `USDT_ERC20` and `USDT_TRX` produces one `USDT/6` row with the summed amount. Per-chain identity stays observable at the payment level via `com.fireblocks.spec/blockchain_id`, `legacy_id`, and `contract_address`. ### Testnet segregation Assets on testnet blockchains (per `Blockchain.onchain.test`) carry a `_TEST` suffix and `com.fireblocks.spec/testnet=true` (e.g. `ETH_TEST/18`, `SOL_TEST/9`). Mainnet and testnet holdings of the same symbol never collide. ### Asset classes the connector skips Only `NATIVE`, `FT`, and `FIAT` are ingested. `NFT`, `SFT`, `VIRTUAL`, and deprecated assets don't appear in Accounts, Balances, or Payments. ### Sanitization rules When `displaySymbol` can't be used verbatim: - Lowercase → uppercase (`xDAI` → `XDAI`). - Digit prefix stripped until first letter (`1INCH` → `INCH`). - Non-`[A-Z0-9]` characters dropped. - Base capped at 17 characters; testnet assets additionally carry `_TEST`. The initial poll pins `after=1`, bypassing Fireblocks' default 90-day window so the full history is backfilled. Long histories may take several cycles to surface entirely, with new rows appearing each cycle. ## Metadata keys Fireblocks fields land on the `metadata` of the Payments Account or Payment under `com.fireblocks.spec/`. The connector stamps them during mapping so chain, contract, and classification stay visible after the canonical asset string is computed. ### Account metadata Stamped on a Payments Account when the `VaultAccount` field is set or `true`. | Key | Source | |---|---| | `com.fireblocks.spec/customer_ref_id` | `VaultAccount.customerRefId` | | `com.fireblocks.spec/hidden_on_ui` | `VaultAccount.hiddenOnUI` | | `com.fireblocks.spec/auto_fuel` | `VaultAccount.autoFuel` | ### Payment metadata Two groups of keys land on each Payment: #### Cached asset metadata Copied from the connector's in-process Asset cache (loaded at install, refreshed via TTL). Stamped on **every** Payment so consumers retain the chain, contract, and classification even after the asset string is canonicalized away from `legacyId`. | Key | Source | |---|---| | `com.fireblocks.spec/legacy_id` | `Asset.legacyId` | | `com.fireblocks.spec/asset_uuid` | `Asset.id` | | `com.fireblocks.spec/display_name` | `Asset.displayName` | | `com.fireblocks.spec/display_symbol` | `Asset.displaySymbol` | | `com.fireblocks.spec/blockchain_id` | `Asset.blockchainId` | | `com.fireblocks.spec/asset_class` | `Asset.assetClass` (`NATIVE`, `FT`, `FIAT`) | | `com.fireblocks.spec/contract_address` | `Asset.onchain.address` | | `com.fireblocks.spec/token_standard` | `Asset.onchain.standards` (comma-joined) | | `com.fireblocks.spec/verified` | `Asset.metadata.verified` (only when `true`) | | `com.fireblocks.spec/features` | `Asset.metadata.features` (comma-joined) | | `com.fireblocks.spec/testnet` | derived from `Blockchain.onchain.test` (only when `true`) | #### Transaction-level metadata Read directly from the Fireblocks `Transaction`. Present **only when the upstream field is populated**, so absence is meaningful (no `tx_hash` ⇒ off-chain transfer, no `note` ⇒ unannotated, …). | Key | Source | |---|---| | `com.fireblocks.spec/tx_hash` | `Transaction.txHash` | | `com.fireblocks.spec/network_fee` | `Transaction.feeInfo.networkFee` | | `com.fireblocks.spec/note` | `Transaction.note` | | `com.fireblocks.spec/sub_status` | `Transaction.subStatus` | | `com.fireblocks.spec/destination_ids` | `Transaction.destinations[].id` (multi-destination transactions only) | The full Fireblocks `Transaction` body is preserved verbatim on `/v3/payments/{paymentID}.adjustments[].raw`. The list endpoint doesn't inline `adjustments` — fetch by ID for the raw payload. Vault payloads are inlined on both the list and detail account endpoints via the account's `raw` field. ## Rate limits Per endpoint, per transaction, per minute — depending on your Fireblocks contract. See the [Fireblocks rate-limiting docs](https://developers.fireblocks.com/reference/rate-limiting). ## Known gaps - **Outbound initiation** — Fireblocks supports transaction creation upstream; the connector is read-only. `CreateTransfer` and `CreatePayout` are not wired. - **Webhooks** — not consumed; refresh runs on the polling cycle. - **External accounts** — `FETCH_EXTERNAL_ACCOUNTS` is not implemented. Counterparties live in the workspace UI. - **NFT, SFT, VIRTUAL** — filtered out; only `NATIVE`, `FT`, `FIAT` are ingested (see [Asset model](#asset-model)). --- ## Increase Source: https://docs.formance.com/modules/payments/connectors/psp/increase The Increase connector polls an Increase account and surfaces US bank accounts, balances, counterparties, and payments (transfers, ACH, wires, checks). It covers the full US-bank rail set for outbound initiation plus real-time webhook ingest. The Increase connector requires Payments module **3.1.0 or higher**. Your stack pins an older version — upgrade to use it. ## Prerequisites You need an Increase account and an API key with read access to accounts, balances, transfers, and inbound credits, plus write access for the outbound rails you initiate. For webhooks, the connector creates an `EventSubscription` at install signed with the `webhookSharedSecret` you supply; Increase signs every delivery HMAC-SHA256 and the connector verifies before translating. ## Installation {"fctl payments connectors install increase config.json"} ### Configuration fields `endpoint` is `https://api.increase.com` for production, `https://sandbox.increase.com` for sandbox. ## Capabilities - `FETCH_ACCOUNTS` — internal accounts via `GET /accounts`. - `FETCH_BALANCES` — available + pending balance per account. - `FETCH_EXTERNAL_ACCOUNTS` — counterparties via `GET /account_numbers` and `GET /external_accounts`. - `FETCH_PAYMENTS` — transactions, classified per direction and rail. - `CREATE_BANK_ACCOUNT` — registers an external account on Increase. - `CREATE_TRANSFER` — `POST /account_transfers` between two Increase accounts. - `CREATE_PAYOUT` — ACH (`POST /ach_transfers`), wire (`POST /wire_transfers`), or check (`POST /check_transfers`); rail selected via `com.increase.spec/rail`. - `CREATE_WEBHOOKS` + `TRANSLATE_WEBHOOKS` — `EventSubscription` ingesting `account.*`, `transaction.*`, `*_transfer.*`. ## Account model Every Payments internal account is one Increase account from `GET /accounts`. The `reference` is the account ID; `name` is the account name; `defaultAsset` is `USD/2` (Increase is USD-only). EXTERNAL accounts come from `/account_numbers` and `/external_accounts`; `v3CreateBankAccount` registers new ones. See [Accounts](/modules/payments/accounts) for the cross-connector model. ## Asset model USD-only — every balance and payment is `USD/2`. Amounts are already in minor units. ## Status mapping | Increase `status` (per rail) | Payment `status` | | --- | --- | | `pending_approval`, `pending_submission`, `submitted`, `pending_mailing` (check) | `PENDING` | | `complete`, `posted`, `mailed` | `SUCCEEDED` | | `returned`, `rejected`, `failed` | `FAILED` | | `canceled` | `CANCELLED` | ACH credits stay `PENDING` until the return window closes — the connector folds it into the `PENDING → SUCCEEDED` transition. ## Metadata keys Under `com.increase.spec/`: - **Account**: `account_id`, `account_number`, `routing_number`, `account_type`, `bank` (Increase's underlying program bank). - **External account**: `account_number_id` or `external_account_id`, `account_number_last4`, `routing_number`, `funding`. - **Payment**: `transaction_id` or `*_transfer_id`, `rail` (`ach` / `wire` / `check` / `book`), `description`, `idempotency_key`, `network_response_code`. Outbound initiation requires the rail on the PaymentInitiation metadata: ```json { "metadata": { "com.increase.spec/rail": "ach", "com.increase.spec/standard_entry_class_code": "PPD" } } ``` ## Workflow tree ```text FetchAccounts (periodic) ├── FetchBalances (periodic) — per account └── FetchPayments (periodic) — per account, per rail FetchExternalAccounts (periodic) CreateBankAccount / CreateTransfer / CreatePayout (event-driven) └── PollTransferStatus / PollPayoutStatus until terminal CreateWebhooks (one-shot at install) ``` ## Pagination and recovery Cursor-based (`cursor` + `limit`). The connector persists the latest cursor per stream and resumes on restart; the engine dedupes by `PSPPayment.Reference` since Increase replays events during webhook backfills. ## Known gaps - **Reverse transfer** is not implemented; ACH returns appear as new payment rows linked via metadata. - **Real-time payments (RTP / FedNow)** are not exposed — connector covers ACH, wire, and check. --- ## Mangopay Source: https://docs.formance.com/modules/payments/connectors/psp/mangopay The Mangopay connector polls a Mangopay client and surfaces multi-currency e-wallets, balances, bank accounts (external), and transactional events. It also initiates transfers and payouts, registers bank accounts, and ingests webhooks. ## Prerequisites You need a Mangopay client and an API key pair. Mangopay uses HTTP Basic Auth (`clientID:apiKey`); the connector sets the header on every request. ## Installation {"fctl payments connectors install mangopay config.json"} ### Configuration fields `endpoint` is `https://api.mangopay.com` for production, `https://api.sandbox.mangopay.com` for sandbox. ## Capabilities - `FETCH_ACCOUNTS` — e-wallets, grouped by `UserId`. - `FETCH_BALANCES` — per-wallet balance via Mangopay's `Balance` resource. - `FETCH_EXTERNAL_ACCOUNTS` — bank accounts attached to each user. - `FETCH_PAYMENTS` — payins, transfers, payouts, refunds, and disputes, unified into `PSPPayment`. - `FETCH_OTHERS` — KYC documents and mandates, surfaced under `mangopay-*` on `v3ListPaymentsOther`. - `CREATE_BANK_ACCOUNT` — IBAN / US / GB / CA / OTHER variants. - `CREATE_TRANSFER` — wallet-to-wallet via `POST /transfers`. - `CREATE_PAYOUT` — wallet-to-bank-account via `POST /payouts/bankwire`. - `CREATE_WEBHOOKS` + `TRANSLATE_WEBHOOKS` — one hook per consumed `EventType`. ## Account model Every Payments internal account is one Mangopay **e-wallet** (single-currency, scoped to a `UserId`). The `reference` is the wallet ID; `defaultAsset` is the wallet's currency at ISO 4217 precision. Accounts are emitted per user (one fetch cycle per user from the workflow's parent payload). EXTERNAL accounts come from `/users/{UserId}/bankaccounts`; `v3CreateBankAccount` registers new ones (IBAN / US / GB / CA / OTHER). See [Accounts](/modules/payments/accounts) for the cross-connector model. ## Asset model Multi-currency (EUR / USD / GBP, …), formatted to UMN at ISO 4217 precision. Amounts are already in minor units — no scaling. ## Status mapping | Mangopay `Status` (Transactions) | Payment `status` | | --- | --- | | `CREATED`, `IN_PROGRESS` | `PENDING` | | `SUCCEEDED` | `SUCCEEDED` | | `FAILED` | `FAILED` | | `CANCELLED` | `CANCELLED` | `CREATE_*` initiations schedule `PollTransferStatus` / `PollPayoutStatus` against the relevant endpoint until terminal. ## Metadata keys Under `com.mangopay.spec/`: - **Account**: `wallet_id`, `description`, `owners` (comma-joined `UserId` list), `currency`. - **External account**: `bank_account_id`, `type` (`IBAN` / `US` / `GB` / `CA` / `OTHER`), `iban`, `bic`, `owner_name`, `owner_address`. - **Payment**: `transaction_id`, `transaction_type` (`PAYIN` / `PAYOUT` / `TRANSFER` / `REFUND`), `nature` (`REGULAR` / `REPUDIATION` / `REFUND` / `SETTLEMENT`), `result_code`, `result_message`. Bank-account creation uses `com.mangopay.spec/bank_account_type` on the `v3CreateBankAccount` body to select the variant (default `IBAN`). ## Workflow tree ```text FetchAccounts (periodic) — paginated over UserList → WalletList ├── FetchBalances (FromPayload — no extra API call) └── FetchPayments (periodic) — per wallet FetchExternalAccounts (periodic) — per user FetchOthers (periodic) — KYC docs, mandates CreateBankAccount / CreateTransfer / CreatePayout (event-driven) └── PollTransferStatus / PollPayoutStatus until terminal CreateWebhooks (one-shot at install) — one hook per EventType ``` ## Pagination and recovery 1-indexed `Page` + `Per_Page` (max 100). The connector persists the watermark per stream in platform-managed `State`. Webhook deliveries dedupe by `(EventType, ResourceId)` against `PSPPayment.Reference`. ## Known gaps - **Pre-authorizations**: not surfaced; only successful captures land as `PSPPayment`s. - **Disputes**: arrive via webhooks as `PSPPayment` rows tagged `transaction_type=DISPUTE`; the full lifecycle lives in Mangopay's dashboard. - **KYC**: documents surface via `FETCH_OTHERS` only — submission happens in Mangopay's UI. --- ## Modulr Source: https://docs.formance.com/modules/payments/connectors/psp/modulr The Modulr connector polls a Modulr customer and surfaces UK/EU bank accounts, balances, beneficiaries, and transactions. It also initiates transfers between Modulr accounts and payouts to registered beneficiaries. ## Prerequisites You need a Modulr customer and an API key + secret pair. Modulr signs every request with HMAC-SHA1; the connector builds the `Authorization` header automatically. ## Installation {"fctl payments connectors install modulr config.json"} ### Configuration fields `endpoint` is `https://api-sandbox.modulrfinance.com/api-sandbox` for sandbox, `https://api.modulrfinance.com/api-live` for production. ## Capabilities - `FETCH_ACCOUNTS` — bank accounts via `GET /accounts`. - `FETCH_BALANCES` — available + reserved per account. - `FETCH_EXTERNAL_ACCOUNTS` — beneficiaries via `GET /beneficiaries`. - `FETCH_PAYMENTS` — transactions, classified PAY-IN / PAYOUT / TRANSFER. - `CREATE_TRANSFER` — `POST /payments` between two Modulr accounts on the same rail. - `CREATE_PAYOUT` — `POST /payments` to a beneficiary (Faster Payments for GBP, SEPA for EUR). `CREATE_BANK_ACCOUNT` is not implemented. Webhooks are not wired. ## Account model Every Payments internal account is one Modulr account from `GET /accounts`. The `reference` is the account ID; `name` is the account name; `defaultAsset` is the account's `currency` (Modulr supports GBP and EUR). EXTERNAL accounts come from `/beneficiaries`. See [Accounts](/modules/payments/accounts) for the cross-connector model. ## Asset model GBP and EUR at the account level — `GBP/2` / `EUR/2`. Amounts arrive as decimal strings; the connector applies major-to-minor scaling. ## Status mapping | Modulr `status` | Payment `status` | | --- | --- | | `SUBMITTED`, `VALIDATED`, `SCHEDULED`, `PROCESSING` | `PENDING` | | `PROCESSED`, `CONFIRMED` | `SUCCEEDED` | | `ER_INVALID`, `ER_EXTSYS`, `ER_GENERAL` | `FAILED` | | `CANCELLED`, `RECALLED` | `CANCELLED` | `CREATE_TRANSFER` and `CREATE_PAYOUT` schedule `PollTransferStatus` / `PollPayoutStatus` against `/payments/{id}` until terminal. ## Metadata keys Under `com.modulr.spec/`: - **Account**: `account_id`, `account_number`, `sort_code` (UK), `iban` (EU), `bic`, `customer_id`. - **External account**: `beneficiary_id`, `account_number`, `sort_code`, `iban`, `bic`, `name`. - **Payment**: `transaction_id`, `payment_id`, `reference`, `end_to_end_id`, `payment_method` (`PAYOUT` / `SEPA_CT` / `FPS_OUT` / …), `direction`. ## Workflow tree ```text FetchAccounts (periodic) ├── FetchBalances (FromPayload — no extra API call) └── FetchPayments (periodic) — per account FetchExternalAccounts (periodic) CreateTransfer / CreatePayout (event-driven) └── PollTransferStatus / PollPayoutStatus until terminal ``` ## Pagination and recovery 0-indexed `page` + `size` (max 100). Watermarks persist in platform-managed `State`; restarts resume from the last committed page boundary. ## Known gaps - **Webhooks** are not implemented — subscription provisioning is not wired. - **CHAPS** (high-value GBP rail) is not exposed; outbound covers Faster Payments and SEPA only. - **Beneficiary creation** through the Payments module is not wired; manage beneficiaries in the Modulr portal. --- ## Moneycorp Source: https://docs.formance.com/modules/payments/connectors/psp/moneycorp The Moneycorp connector polls a Moneycorp account and surfaces multi-currency wallets, balances, beneficiaries, and transactions. It also initiates transfers between Moneycorp accounts and payouts to registered beneficiaries. ## Prerequisites You need a Moneycorp account and a `clientID` + `apiKey` pair. Moneycorp uses OAuth2 client-credentials; the connector exchanges the credentials for a short-lived bearer token and refreshes automatically. ## Installation {"fctl payments connectors install moneycorp config.json"} ### Configuration fields `endpoint` defaults to `https://api.moneycorp.com`. Sandbox: `https://sandbox-corpapi.moneycorp.com`. ## Capabilities - `FETCH_ACCOUNTS` — wallets per ledger. - `FETCH_BALANCES` — per-account balance. - `FETCH_EXTERNAL_ACCOUNTS` — beneficiaries. - `FETCH_PAYMENTS` — transactions and FX conversions. - `CREATE_TRANSFER` — wallet-to-wallet. - `CREATE_PAYOUT` — payment to a registered beneficiary. `CREATE_BANK_ACCOUNT` is not implemented; manage beneficiaries through Moneycorp. Webhooks are not wired. ## Account model Every Payments internal account is one Moneycorp wallet under the configured client/ledger. The `reference` is the Moneycorp account ID (numeric, used as the cursor since Moneycorp doesn't expose creation dates); `name` is the account name; `defaultAsset` is null — wallets are multi-currency, with one balance row per asset via `FETCH_BALANCES`. EXTERNAL accounts come from Moneycorp's beneficiaries endpoint. See [Accounts](/modules/payments/accounts) for the cross-connector model. ## Asset model Multi-currency, formatted to UMN at ISO 4217 precision. Amounts arrive as decimal strings; the connector applies major-to-minor scaling. ## Status mapping | Moneycorp `status` | Payment `status` | | --- | --- | | `pending`, `processing` | `PENDING` | | `completed`, `released` | `SUCCEEDED` | | `failed`, `cancelled-due-to-failure` | `FAILED` | | `cancelled` | `CANCELLED` | `CREATE_TRANSFER` and `CREATE_PAYOUT` schedule `PollTransferStatus` / `PollPayoutStatus` until terminal. ## Metadata keys Under `com.moneycorp.spec/`: - **Account**: `account_id`, `account_name`, `account_type`, `client_reference`. - **External account**: `beneficiary_id`, `country`, `currency`, `bank_account_number`, `iban`, `bic`, `routing_code_type1`. - **Payment**: `transaction_id`, `payment_id`, `payment_type`, `currency`, `client_reference`, `reason`. ## Workflow tree ```text FetchAccounts (periodic) ├── FetchBalances (periodic) — per account └── FetchPayments (periodic) — per account FetchExternalAccounts (periodic) CreateTransfer / CreatePayout (event-driven) └── PollTransferStatus / PollPayoutStatus until terminal ``` ## Pagination and recovery 1-indexed `pageNumber` + `pageSize`. Watermarks persist in platform-managed `State`; restarts resume from the last committed page boundary. ## Known gaps - **Webhooks** are not implemented. - **FX conversions** surface as `PSPPayment`s with `com.moneycorp.spec/payment_type=fx`, not as separate [Conversion](/modules/payments/conversions) entries. - **Beneficiary creation** through the Payments module is not wired. --- ## Bitstamp Source: https://docs.formance.com/modules/payments/connectors/exchange/bitstamp The Bitstamp connector polls a Bitstamp account and surfaces currency wallets, balances, payments, trading orders, and conversions. It is read-only and spot-only. Bitstamp API keys scope to a single account — Main or one named sub-account — so **install one connector instance per Bitstamp account** you need to reconcile. ## Prerequisites You need a Bitstamp account and an API key with the minimum permissions for the capabilities you use. Bitstamp uses HMAC-SHA256 v2 signing; the connector signs internally — you only supply the key and secret. ## Installation {"fctl payments connectors install bitstamp config.json"} With `config.json` containing: ```json { "apiKey": "string", "apiSecret": "string", "endpoint": "https://www.bitstamp.net", "name": "string", "pollingPeriod": "30m" } ``` ### Configuration fields | Field | Required | Default | Description | |---|---|---|---| | `apiKey` | yes | — | Bitstamp API key. Sent in `X-Auth` as `BITSTAMP `. | | `apiSecret` | yes | — | HMAC-SHA256 signing secret. Never logged. | | `endpoint` | no | `https://www.bitstamp.net` | API root. Override only for non-production. | | `name` | yes | — | Unique name for this connector instance (e.g. `bitstamp-main`, `bitstamp-treasury` when running one per scope). | | `pollingPeriod` | no | `30m` | Sync cadence (min `20m`). Drives every capability. | The config is deliberately minimal — the API key scopes the connection, and Bitstamp's API exposes no portable way to fan out across scopes. ## Capabilities - **FetchAccounts** — currency wallets in scope via `POST /api/v2/account_balances/`. - **FetchBalances** — re-reads `account_balances/` per cycle. - **FetchPayments** — `user_transactions/` on a single `since_id` watermark. - **FetchOrders** — open-orders snapshot reconciled against `order_status/` per tracked id. - **FetchConversions** — `user_transactions/` rows with `type=36` (instant buy/sell). Payouts, transfers, webhooks, and bank-account creation are not implemented; Bitstamp's API surface for those flows is uneven. ## Account model Every Payments internal account is one currency in the Bitstamp scope — one account per `(connector install, currency)`. The `reference` is the currency ticker (`USD`, `EUR`, `BTC`); the connector-level `name` (e.g. `bitstamp-main`) disambiguates the scope. `defaultAsset` is `TICKER/precision` from the `currencies` cache. No EXTERNAL accounts are emitted. See [Accounts](/modules/payments/accounts) for the cross-connector model. Bitstamp returns every currency the account *could* hold. Rows with `Available`, `Total`, and `Reserved` all zero are skipped — emitting hundreds of empty accounts pollutes the catalogue without informing anyone. Bitstamp doesn't expose per-currency creation dates, so `CreatedAt` defaults to `BitstampGenesis = 2011-08-02 UTC` (the platform's launch). The sentinel is stable across reinstalls. ## Asset model The canonical asset is the uppercased currency ticker with precision suffix from the `currencies` cache — `USD/2`, `EUR/2`, `BTC/8`, `USDT/6`. The cache loads at install and refreshes on a TTL; assets not in the cache are logged and skipped rather than emitted with a guessed precision. ## Workflow tree ```text FetchAccounts (periodic) └── FetchOrders (periodic, derives tracked markets from accounts) FetchBalances (periodic root) FetchPayments (periodic root) FetchConversions (periodic root) ``` `FetchOrders` nests under `FetchAccounts` because it derives tradeable markets from account metadata. Balances, Payments, and Conversions are independent roots — their Bitstamp endpoints are account-global at the API-key level, so no parent context is needed. ## Payments A Payment is one row from `user_transactions/`, polled on a single inclusive `since_id` watermark. Trade rows (`type=2`) feed [Orders](#orders); instant-buy/sell rows (`type=36`) feed [Conversions](#conversions); everything else (deposits, withdrawals, settled activity, sub-account transfer legs of types 14 / 33 / 35) lands as a Payment. The watermark is inclusive — the last row of cycle N reappears as the first of N+1, deduped downstream by `PSPPayment.Reference`. End-of-pagination keeps the watermark; we never reset. Sub-account transfer rows (types 14 / 33 / 35) are mapped defensively — signed PAY-IN / PAYOUT legs sharing a `transfer_pair_id`. A Main-account API key does not actually surface them on `user_transactions/`. Customers needing transfer reconciliation install one connector per sub-account; the pair-id correlation works once both legs' keys are integrated. ## Orders Bitstamp doesn't expose an "orders since X" endpoint. The connector reconciles a live snapshot every cycle: 1. `GetOpenOrders` returns currently-open orders. 2. New IDs are seeded into `trackedOrders` state with their first-sight `LimitPrice`. 3. `GetOrderStatus` is called per id (snapshot ∪ tracked) for fills, fees, datetime, and market. 4. The order maps to a `PSPOrder` with adjustments aggregating each observed state change. 5. Tracked entries drop on terminal status (`FILLED` / `CANCELLED`). 6. Tracked entries also drop after `FirstSeenAt + 25d`, emitting `com.bitstamp.spec/retention_expired = true`. Bitstamp retains `order_status/` rows for 30 days; the 5-day margin avoids losing the terminal state. `Trade` primitives in `user_transactions/` (`type=2` rows with a parent `order_id`) aggregate under their parent rather than being emitted as standalone Orders — one `PSPOrder` per Bitstamp order, fills attached. ## Conversions `user_transactions/` returns two primitives that both look like "buys" and "sells" in the web UI: | Wire | Has `order_id`? | Lifecycle | Formance model | |---|---|---|---| | `type=2` (Trade — order fill) | yes | order-book — In Queue → Open → Finished / Cancelled | `PSPOrder` | | `type=36` (Instant buy/sell) | no | atomic — settled in one round-trip | `PSPConversion` | Conversions share the `user_transactions/` stream with payments but hold their own watermark — the two cursors advance independently. Asset class plays no role in classification: Bitstamp tags every crypto (BTC, USDC, EURC, …) as `currency.type = "crypto"` with no stablecoin tag. A `type=36` BTC↔EUR row and a `type=36` USDC↔EUR row are the same primitive; consumers wanting "market exposure" vs "stable-value swap" semantics apply their own allow-list against `SourceAsset` / `DestinationAsset`. ## Install-time enrichment Four reference datasets load in parallel at install, refreshed via TTL cache: - `markets` — every trading pair, used to resolve order quote/base currencies. - `my_markets` — pairs the key has actually traded (gates Order details). - `fees/trading` — per-market trading fees, surfaced on Order metadata. - `fees/withdrawal` — per-currency withdrawal fees, surfaced on withdrawal-request payments. Permission-gated endpoints feed a process-lifetime `derivSkip` cache: the first `403`-style response for a key without `my_markets` scope logs once at Info, then subsequent attempts go silent. Keeps logs readable on read-only keys without trading scope. ## Metadata keys Under `com.bitstamp.spec/`. Full list in the connector's [`MAPPINGS.md`](https://github.com/formancehq/payments/blob/main/ee/plugins/bitstamp/MAPPINGS.md); highlights: - **Account**: `currency_type`, `currency_decimals`, `withdrawal_fee?`, `is_crypto?`. - **Payment**: `tx_type`, `bank_transaction_id?`, `transfer_pair_id?`, `transfer_direction?` (set on the defensive sub-account transfer legs). - **Order**: `order_subtype` (`LIMIT` / `MARKET` / `INSTANT` / `STOP_LIMIT`), `order_status_datetime`, `client_order_id?`, `historical?`, `retention_expired?`. - **Conversion**: `from_amount_raw`, `to_amount_raw`, `fee_market`. ## Pagination and recovery `FetchPayments` and `FetchConversions` each persist a `LastTransactionID` watermark and advance only after the cycle completes — a mid-cycle worker crash replays the same page on restart, with downstream dedupe absorbing the overlap. `FetchOrders` checkpoints `LastSeenEventIDPerMarket` plus `HasMoreCurrentMarket` so a partial paginated walk resumes from the same market on the next cycle. ## Known gaps - **Historical orders** — orders placed and filled before install, or older than 30 days, fall outside Bitstamp's `order_status/` retention and aren't back-filled. Per-fill rows exist in `user_transactions/` as `type=2` with `order_id`; `MAPPINGS.md §9` documents the aggregation approach. - **No programmatic sub-account discovery** — Bitstamp's API doesn't expose a "list my scopes" call. Deploy one connector per account scope, named via `name`. --- ## Qonto Source: https://docs.formance.com/modules/payments/connectors/psp/qonto The Qonto connector polls a Qonto organization and surfaces business bank accounts, balances, beneficiaries, and transactions. It is **read-only**: Qonto's outbound transfer and webhook surfaces require three-legged OAuth2, which the framework does not drive. ## Prerequisites You need a Qonto organization and an API key pair. Qonto authenticates with `login:secret-key` in the `Authorization` header; staging additionally requires a `stagingToken` header. ## Installation {"fctl payments connectors install qonto config.json"} ### Configuration fields Production: `https://thirdparty.qonto.com`. Staging: `https://thirdparty.staging.qonto.co` with a `stagingToken`. ## Capabilities - `FETCH_ACCOUNTS` — bank accounts via `GET /v2/organizations`. - `FETCH_BALANCES` — authorized + balance per account. - `FETCH_EXTERNAL_ACCOUNTS` — beneficiaries. - `FETCH_PAYMENTS` — `GET /v2/transactions`, classified per direction. Qonto exposes outbound transfer initiation and webhooks upstream, but both require three-legged OAuth2 that the framework doesn't yet drive — the connector advertises only the read capabilities. ## Account model Every Payments internal account is one Qonto bank account under the organization. `GET /v2/organizations` returns the whole tree at once — there's no per-account endpoint. The `reference` is the bank-account ID; `name` is the account name; `defaultAsset` is `EUR/2` (Qonto is EUR-only). EXTERNAL accounts come from Qonto beneficiaries. See [Accounts](/modules/payments/accounts) for the cross-connector model. ## Asset model EUR-only — every balance and payment is `EUR/2`. Amounts are already in minor units. ## Status mapping | Qonto transaction `status` | Payment `status` | | --- | --- | | `pending` | `PENDING` | | `completed` | `SUCCEEDED` | | `declined` | `FAILED` | | `reversed` | `CANCELLED` | | anything else | `UNKNOWN` | ## Metadata keys Under `com.qonto.spec/`: - **Account**: `iban`, `bic`, `currency`, `organization_slug`. - **External account**: `beneficiary_id`, `iban`, `bic`, `bank_name`, `trusted` (boolean). - **Payment**: `transaction_id`, `operation_type` (`card` / `transfer` / `direct_debit` / `cheque` / `swift_income` / …), `side` (`credit` / `debit`), `reference`, `note`. ## Workflow tree ```text FetchAccounts (periodic) ├── FetchBalances (FromPayload — no extra API call) └── FetchPayments (periodic) — per account FetchExternalAccounts (periodic) ``` ## Pagination and recovery 1-indexed `current_page` + `per_page` (max 100). The connector persists the watermark per stream in platform-managed `State`. ## Known gaps - **Outbound initiation** is not implemented — Qonto's `CreateTransfer` API requires 3-legged OAuth2. - **Webhooks** are not implemented for the same reason. - **Multi-organization tenants**: one connector install covers one organization. --- ## Stripe Source: https://docs.formance.com/modules/payments/connectors/psp/stripe The Stripe connector polls a Stripe account and surfaces `acct_*` balances, charges, transfers, payouts, payments, and refunds. It initiates transfers and payouts via v3 PaymentInitiation, and ingests Stripe webhooks for real-time updates. ## Prerequisites You need a Stripe account and a restricted API key scoped to charges, transfers, payouts, balance, balance transactions, and external accounts. Webhook creation additionally requires `webhook_endpoints:write`. ## Installation {"fctl payments connectors install stripe config.json"} ### Configuration fields The connector uses the official `stripe-go/v80` SDK with the `apiKey` as a Bearer token. The webhook secret is not in the config — Stripe assigns one when the connector creates the endpoint at install, and the platform stores it out-of-band. ## Capabilities - `FETCH_ACCOUNTS` — connected accounts plus the platform's own `acct_*`, paginated via `starting_after`. - `FETCH_BALANCES` — available + pending balance per currency, via `GET /v1/balance` per connected account. - `FETCH_EXTERNAL_ACCOUNTS` — external bank accounts and debit cards per connected account. - `FETCH_PAYMENTS` — charges, refunds, transfers, payouts, and balance transactions, unified into `PSPPayment`. - `CREATE_TRANSFER` — `POST /v1/transfers` between platform-controlled accounts. - `CREATE_PAYOUT` — `POST /v1/payouts` to an attached external account. - `CREATE_WEBHOOKS` — provisions a `WebhookEndpoint` at install for the events the connector consumes. - `TRANSLATE_WEBHOOKS` — converts `charge.*`, `payout.*`, `transfer.*`, `balance.available`, etc. to Payments events. `CREATE_BANK_ACCOUNT` is not implemented — Stripe expects account holders to attach externals via the dashboard or Stripe-issued links. ## Account model Every Payments internal account is one Stripe `acct_*` (a connected account or the platform's own root). The `reference` is the Stripe account ID, except for the platform's own which uses the legacy literal `root` for backwards compatibility; `name` is the display name; `defaultAsset` is the `default_currency` for single-currency accounts, null on multi-currency. EXTERNAL accounts come from each connected account's external bank accounts and debit cards. See [Accounts](/modules/payments/accounts) for the cross-connector model. ## Asset model Stripe returns lowercase ISO 4217 (`usd`, `eur`, `jpy`), formatted to UMN at standard precision (`USD/2`, `EUR/2`, `JPY/0`). Amounts are already in minor units — no scaling. ## Status mapping | Stripe `status` / event family | Payment `status` | | --- | --- | | `pending`, `in_transit`, `paid` (payout pre-settlement) | `PENDING` | | `succeeded`, `paid` (charge), `completed` | `SUCCEEDED` | | `failed`, `requires_payment_method` (terminal) | `FAILED` | | `canceled` | `CANCELLED` | | anything else | `UNKNOWN` | Refunds land as separate `PAY-IN` rows against the original payment's account, linked via `com.stripe.spec/refund_of`. ## Metadata keys Under `com.stripe.spec/`: - **Account**: `type` (`standard`, `express`, `custom`), `country`, `default_currency`, `details_submitted`, `payouts_enabled`, `charges_enabled`. - **Payment**: `payment_method`, `payment_method_type` (`card`, `sepa_debit`, `us_bank_account`, …), `application_fee_amount`, `transfer_group`, `refund_of` (when applicable). - **External account**: `routing_number`, `last4`, `bank_name`, `fingerprint`. ## Workflow tree ```text FetchAccounts (periodic) ├── FetchBalances (FromPayload — no extra API call beyond /v1/balance per account) ├── FetchPayments (periodic) — Charges → Refunds → Transfers → Payouts → Balance Transactions └── FetchExternalAccounts (periodic) — per connected account CreateWebhooks (one-shot at install) ``` ## Pagination and recovery Cursor-based via `starting_after` per resource. The connector persists the latest cursor per stream in platform-managed `State`; restarts resume from the last committed cursor (Stripe's at-least-once event semantics aside). The engine dedupes by `PSPPayment.Reference`. ## Known gaps - **Bank-account creation** is not implemented; account holders attach externals via the Stripe dashboard or Financial Connections. - **Reverse transfer / reverse payout** is not wired; refunds appear as separate `PAY-IN` rows linked via metadata. - **Multi-currency accounts**: balance per `(account, currency)`. Stripe doesn't expose a "default asset" on multi-currency accounts, so `defaultAsset` is null in that case. --- ## Kraken Pro Source: https://docs.formance.com/modules/payments/connectors/exchange/krakenpro The Kraken Pro connector polls a Kraken Pro account and surfaces its asset wallets, balances, payments, trading orders, and conversions. It is read-only and spot-only. Kraken Pro API keys scope to a single account, so **install one connector instance per Kraken Pro account** you need to get data from. ## Prerequisites You need a Kraken Pro account and an API key. Kraken signs requests with HMAC-SHA512; the connector signs internally — you supply only the key and secret. The API key must carry at least the following scopes: - **Funds** → Query - **Orders & trades** → Query closed orders & trades - **Data** → Query ledger entries ## Installation {"fctl payments connectors install krakenpro config.json"} With `config.json` containing: ```json { "apiKey": "string", "apiSecret": "string", "endpoint": "https://api.kraken.com", "name": "string", "pollingPeriod": "30m" } ``` ### Configuration fields | Field | Required | Default | Description | |---|---|---|---| | `apiKey` | yes | — | Kraken Pro API key. | | `apiSecret` | yes | — | Kraken Pro private key. | | `endpoint` | yes | — | Kraken Pro API base URL, e.g. `https://api.kraken.com`. | | `name` | yes | — | Unique name for this connector instance (e.g. `krakenpro-main`, `krakenpro-treasury` when running one per account). | | `pollingPeriod` | no | `30m` | Sync cadence (min `20m`). Drives every capability. | The config is deliberately minimal — the API key scopes the connection to a single Kraken Pro account, and Kraken's API exposes no portable way to fan out across accounts. ## Capabilities The Kraken Pro connector supports the following read-only capabilities: - **FetchAccounts** — one account per asset variant present via `POST /0/private/BalanceEx`. - **FetchBalances** — derived from the same `BalanceEx` call, no extra hop. - **FetchPayments** — deposits, withdrawals, transfers, staking, rewards, and adjustments from `POST /0/private/Ledgers`. - **FetchOrders** — closed/historical orders from `POST /0/private/ClosedOrders`. - **FetchConversions** — off-orderbook swaps from `POST /0/private/Ledgers`, grouped by `refid`. ## Account model Each Payments account maps to one Kraken asset wallet. Kraken keeps a separate wallet per asset variant — the spot balance, staked balance, rewards balance, and so on are distinct — and each becomes its own account. The `reference` is Kraken's asset code (`XXBT`, `XBT.M`, `ZUSD`, `ADA.S`), a `wallet_type` metadata key records the variant (`spot`, `staked`, `rewards`, `yield`, `earn`, `parachain`, `tokenised`, `hold`, `margin`), and `defaultAsset` is the normalized `TICKER/precision` (see [Asset model](#asset-model)). Only assets the account holds or has held are surfaced. See the generic [Accounts](/modules/payments/accounts) page for the cross-connector model. Kraken doesn't expose a creation date per asset wallet, so `CreatedAt` is set to a fixed placeholder — Kraken's launch date, `2011-08-01T00:00:00Z`. ## Asset model Kraken uses its own asset codes (`XXBT`, `ZUSD`, `XETH`). The connector normalizes them to standard tickers — `XXBT` → `BTC`, `ZUSD` → `USD`, `XETH` → `ETH` — so assets appear under the symbols you expect. Precision follows Kraken's internal precision for each asset, which is finer than common market conventions (e.g. `BTC/10` rather than `BTC/8`, `USD/4` rather than `USD/2`). This keeps every amount exact — no value is rounded or truncated on the way in. ## Status mapping Payments come from Kraken's Ledgers endpoint, which only writes an entry on settlement — there is no pending state at this layer, so every Payment is `SUCCEEDED`. Order status is derived from Kraken's `status` enum combined with the filled-vs-ordered volume: | Kraken `status` | `vol_exec` vs `vol` | Order `status` | |---|---|---| | `closed` | exec ≥ vol | `FILLED` | | `closed` | 0 < exec < vol | `PARTIALLY_FILLED` | | `closed` | 0 | `CANCELLED` | | `canceled` | exec > 0 | `PARTIALLY_FILLED` | | `canceled` | 0 | `CANCELLED` | | `expired` | — | `EXPIRED` | ## Payments A Payment is one row from `/0/private/Ledgers`. Kraken's ledger `type` enum maps as follows: | Kraken `type` | Payment `type` | |---|---| | `deposit` | `PAYIN` | | `withdrawal` | `PAYOUT` | | `transfer`, `custodytransfer` | `TRANSFER` | | `staking`, `reward`, `dividend`, `credit`, `nft_rebate` | `PAYIN` | | `nftcreatorfee` | `PAYOUT` | | `adjustment`, `rollover`, `settled`, `reserve`, `ic_settlement`, … | `OTHER` | | `trade`, `eqtrade` | skipped — handled by [Orders](#orders) | | `conversion`, `sale`, `marginconversion`, `margin_conversion` | skipped — handled by [Conversions](#conversions) | The Payment `reference` is the ledger entry id (not `refid`, which groups multi-leg events). The row's `fee` is recorded in metadata but not subtracted from the amount. Unknown future `type` values fall back to `OTHER` with a warning log. ## Orders Only closed orders are registered — an order is surfaced once it has been filled, cancelled, or expired. In-flight orders that are still open or partially filled are not tracked while they remain active. ## Conversions Conversions share the `/0/private/Ledgers` stream with payments but classify a distinct type set — `conversion`, `sale`, `marginconversion`, `margin_conversion` (plus derivatives variants for exhaustiveness; spot-only accounts see only the first two). A conversion is a **pair** of ledger rows sharing one `refid`: a negative-amount leg (source asset) and a positive-amount leg (destination asset). `SourceAmount` / `DestinationAmount` are gross; `fee` is the sum across both legs. ## Metadata keys All keys are namespaced `com.krakenpro.spec/`: - **Account**: `wallet_type` (`spot` / `staked` / `rewards` / …). - **Payment / Conversion**: `refid`, `kraken_type`, `subtype`, `aclass`, `balance_after`; `fee` (payments); `source_ledger_id`, `destination_ledger_id` (conversions). - **Order**: `pair`, `ws_name`, `ordertype`, `price_asset`, `fills` (comma-separated fill txids), `cl_ord_id?`. ## Known gaps - Orders appear only once they're closed (filled, cancelled, or expired) — orders that are still open or partially filled aren't shown while active. - A Kraken Pro API key is scoped to a single account, so reconciling multiple accounts means installing one connector per account. --- ## Wise Source: https://docs.formance.com/modules/payments/connectors/psp/wise The Wise connector polls a Wise Business profile and surfaces multi-currency balances, recipients, and transactions. It initiates transfers between your balances and payouts to recipients across the Wise rail network, and ingests Wise webhooks. ## Prerequisites You need a Wise Business profile and an API token with read access to balances, recipients, and transfers, plus write access for the outbound rails you initiate. The connector creates a webhook subscription at install and verifies deliveries against the `webhookPublicKey` you supply — mis-signed events are rejected. ## Installation {"fctl payments connectors install wise config.json"} ### Configuration fields ## Capabilities - `FETCH_ACCOUNTS` — multi-currency balances per profile via `GET /v4/profiles/{profileId}/balances`. - `FETCH_BALANCES` — derived from the Accounts payload. - `FETCH_EXTERNAL_ACCOUNTS` — recipients via `GET /v1/accounts`. - `FETCH_PAYMENTS` — transfers with full lifecycle. - `FETCH_OTHERS` — quotes and requirements, surfaced under `wise-*` on `v3ListPaymentsOther`. - `CREATE_TRANSFER` — `POST /v1/transfers` between your own balances (after a quote). - `CREATE_PAYOUT` — `POST /v1/transfers` to a registered recipient (after a quote). - `CREATE_WEBHOOKS` + `TRANSLATE_WEBHOOKS` — subscribes to `transfers#state-change` and related events. `CREATE_BANK_ACCOUNT` is not implemented; manage recipients via the Wise dashboard or `POST /v1/accounts`. ## Account model Every Payments internal account is one Wise **balance** (single-currency, scoped to a profile). The `reference` is the balance ID; `name` is the balance name; `defaultAsset` is the balance currency at ISO 4217 precision. Accounts are emitted per profile. EXTERNAL accounts come from `/v1/accounts` (Wise recipients). See [Accounts](/modules/payments/accounts) for the cross-connector model. ## Asset model Multi-currency, formatted to UMN at ISO 4217 precision (`USD/2`, `EUR/2`, `JPY/0`, …). Amounts arrive as decimal strings; the connector applies major-to-minor scaling. ## Status mapping | Wise transfer `status` | Payment `status` | | --- | --- | | `incoming_payment_waiting`, `processing`, `funds_converted`, `outgoing_payment_sent` | `PENDING` | | `outgoing_payment_received` (terminal for the rail), `funds_refunded` (for refund leg) | `SUCCEEDED` | | `bounced_back`, `charged_back`, `failed` | `FAILED` | | `cancelled` | `CANCELLED` | `outgoing_payment_sent` is non-terminal — the connector keeps the transfer in `PENDING` until destination-side receipt is confirmed. ## Metadata keys Under `com.wise.spec/`: - **Account**: `balance_id`, `profile_id`, `type` (`STANDARD` / `SAVINGS`), `currency`. - **External account / recipient**: `account_holder_name`, `currency`, `country`, `bank_name`, `iban`, `swift_code`, `legal_entity_type`. - **Payment**: `transfer_id`, `source_currency`, `target_currency`, `quote_id`, `source_amount`, `target_amount`, `rate`, `fee`, `reference`. ## Workflow tree ```text FetchAccounts (periodic) — per profile ├── FetchBalances (FromPayload — no extra API call) └── FetchPayments (periodic) — per profile FetchExternalAccounts (periodic) — per profile FetchOthers (periodic) — quotes, requirements CreateTransfer / CreatePayout (event-driven) ├── Create quote (POST /v3/quotes) ├── Create transfer (POST /v1/transfers) ├── Fund transfer (POST /v3/transfers/{id}/payments) └── PollTransferStatus / PollPayoutStatus until terminal CreateWebhooks (one-shot at install) ``` ## Pagination and recovery `offset` + `limit` (max 100). Watermarks persist per profile / per stream in platform-managed `State`. Webhook deliveries dedupe at the engine level by `PSPPayment.Reference`. ## Known gaps - **Quote regeneration** is handled inside `CreateTransfer`, but if the rate moves enough between quote and funding the transfer can fail with `quote_expired` — the PaymentInitiation lands in `FAILED`; retry to issue a fresh quote. - **Profile multi-tenant**: one connector install covers every profile attached to the token (personal + business). Filter on `metadata.profile_id` to slice. --- ## Wallets Source: https://docs.formance.com/modules/wallets Wallets is a fully managed, white-label wallet service to materialize and spend users' funds. It comes with built-in support for multi-currency balances and temporary holds capabilities (and upcoming support for reserved funds and expirable fungibles). It is built on top of the Formance Ledger service and is designed to provide an easy way to add wallet capabilities to your application without having to worry about the underlying transaction structure, providing an opinionated model implementation. ## Use-cases Wallets can be a great fit if you are building: * A closed-loop economy on top of your marketplace, where funds paid out can be used at checkout * Savings on processing fees, by keeping some transactions internal to your system * A better repeat purchase rate, by implementing flows like refund to wallet ## Using Wallets vs Ledger An honest question that might come to mind is: what's the difference between using Wallets and using Ledger directly? The answer is that the Ledger is a low-level service that provides a generic way to store and retrieve transactions, while Wallets is a higher-level service that provides out-of-the-box capabilities that would require otherwise non-trivial effort to implement on top of the Ledger. Note that you can start using Wallets now, and eject at any time to use the Ledger directly if you start outgrowing it, at the cost of having to implement some of the Wallets features yourself. --- ## Prerequisites Source: https://docs.formance.com/modules/wallets/prerequisites For this tutorial, you will need to have a Formance Cloud Sandbox provisioned. Formance Cloud Sandbox is a free, fully functional, trial environment for Formance Enterprise Edition that you can use to learn about Formance Stack and to develop and test your applications. You can create a Formance Cloud Sandbox by following the instructions in the [Getting Started guide](/getting-started/quickstart). --- ## Basic wallet operations Source: https://docs.formance.com/modules/wallets/basic-operations In this section, you'll use Wallets to model basic RPG game mechanics. You'll manage the wallet of Eryldor, an elf mage living through countless adventures! ## An introduction to Wallets At its core, a **wallet** is a container of assets related to one entity. It can be used to store and manage any kind of assets, from money to items, and even more complex objects. In the context of Cones of Dunshire, a wallet is used to store the money, currencies, and points of a character. A wallet is composed of **balances**. A balance is a mechanism to logically separate assets according to nature, usage, lifecycle or any other criteria. A balance can be seen as a sub-wallet, and can be used to store and manage a specific set of assets. A balance can expire. When a balance expires, its assets cannot be used anymore. In Cones of Dunshire, Eryldor has a wallet with two balances: one for coins and experience points, and another for badges, which are earned in tournaments and expire when the tournament season ends. Here is the structure of Eryldor's wallet. ![Eryldor's wallet structure](/assets/images/wallets/eryldor-wallet.png) ## Creating Eryldor's wallet Before starting, make sure you have followed the [prerequisites](/getting-started/quickstart) and that you're logged in with the `fctl login` command. First you'll create Eryldor's wallet. You'll use the `fctl wallet create` command to create a wallet named `eryldor`. {"fctl wallets create eryldor"} You'll get a response like this: ``` You are about to create a wallet. Do you want to continue [Y/n]: y SUCCESS Wallet created successfully with ID: 9d21fd84-xxxx-yyyy-zzzz-250aaa73b374 ``` Wallets come with a default balance named `main`. You can use it to store and manage assets without creating additional balances. Here, you'll use the `main` balance to store and manage the coins and experience points of Eryldor. ### Create the `season-2024` balance Now you'll create a balance named `season-2024` to store the badges of Eryldor. This balance will expire at the end of the tournament season 2024. Here it will expire on January 1st, 2025. Feel free to adapt the expiration date to your needs. " }} body={{ name: "season-2024", expiresAt: "2025-01-01T00:00:00Z" }}> {"fctl wallets balances create season-2024 --name eryldor --expires-at \"2025-01-01T00:00:00Z\""} You'll get a response like this: ``` SUCCESS Balance created successfully with name: season-2024 ``` ### Inspecting Eryldor's wallet You can inspect the wallet in the Formance Console. First open the console. ```bash fctl ui ``` Then navigate to the Wallets section and select the `eryldor` wallet. You'll see the `main` balance and the `season-2024` balance as shown in the screenshot below. A common use case for balances is to model vouchers, coupons, or any kind of assets that have a limited lifecycle. For example, you can create a balance for each voucher type, and set the expiration date to the end of the voucher validity period. ## Crediting Eryldor with coins and experience points After tremendous adventures, Eryldor managed to complete their first quest! As a well-deserved reward, they earned 10 COIN and 150 experience points. You'll use the `fctl wallets credit` command to credit Eryldor's wallet with these assets. When you credit a wallet, you must specify where the funds come from. Here, you'll specify that the funds come from the account `world`. `world` is a special account that represents the external world, and is used to introduce or remove assets from the system. Note that you didn't specify a balance. When you don't specify a balance, the `main` balance is used by default as it is the primary balance of the wallet. **Add the coins** " }} body={{ amount: { asset: "COIN", amount: 10 }, sources: [{ type: "account", identifier: "world" }] }}> {"fctl wallets credit 10 COIN --name eryldor --source account=world"} **Add the experience points** " }} body={{ amount: { asset: "XP", amount: 150 }, sources: [{ type: "account", identifier: "world" }] }}> {"fctl wallets credit 150 XP --name eryldor --source account=world"} If you specify a source different from `world`, you'll get an `INSUFFICIENT_FUND` error. This is because you can only credit a wallet with assets coming from either the external world or another wallet. ### Inspecting Eryldor's wallet Let's inspect Eryldor's wallet in the Formance Console. You'll see that the `main` balance contains 10 COIN and 150 XP, as shown in the screenshot below. ## Debiting Eryldor's wallet Eryldor is now ready to buy a new spellbook! They found a rare spellbook for 5 COIN! It is a very good deal, so they decide to buy it. You'll use the `fctl wallets debit` command to debit Eryldor's wallet with 5 COIN. " }} body={{ amount: { asset: "COIN", amount: 5 }, destination: { type: "account", identifier: "world" } }}> {"fctl wallets debit 5 COIN --name eryldor --destination account=world"} You'll get a response like this: ``` SUCCESS Wallet debited successfully! ``` As we did for the credit, you must specify the destination of the funds. Here, you specify that the funds go to the account `world`. `world` being a special account that represents the external world, it can also be used to remove assets from the system. ### Inspecting Eryldor's wallet Let's inspect Eryldor's wallet in the Formance Console. You'll see that the `main` balance contains 5 COIN and 150 XP, as shown in the screenshot below. ## Earning badges Eryldor is a very skilled mage and they are participating in the tournament season 2024. Thanks to their new spellbook, they managed to win the first tournament and earned a badge! Badges are earned in tournaments and expire at the end of the tournament season. You'll use the `fctl wallets credit` command to credit Eryldor's wallet with a badge. It's a special asset that you'll represent with the `BADGE` currency. The process is the same as for the coins and experience points, with the difference that you'll specify the `season-2024` balance. " }} body={{ amount: { asset: "BADGE", amount: 1 }, balance: "season-2024", sources: [{ type: "account", identifier: "world" }] }}> {"fctl wallets credit 1 BADGE --name eryldor --balance season-2024 --source account=world"} You'll get a response like this: ``` SUCCESS Wallet credited successfully! ``` ### Inspecting Eryldor's wallet Let's inspect Eryldor's wallet in the Formance Console. You'll see that the `season-2024` balance contains 1 BADGE, as shown in the screenshot below. ## Next steps In this section, you've learned how to create a wallet, create balances, credit and debit a wallet, and credit a balance. You've also learned how to inspect a wallet in the Formance Console. In the next session, you'll learn how to use the Wallets to model a hold, a mechanism to freeze assets for a specific purpose, such as a deposit or a reservation. --- ## Managing holds Source: https://docs.formance.com/modules/wallets/managing-holds ## What is a hold? A **hold** is a temporary reservation of funds. It's a way to ensure that funds are available when you need them. When you place a hold on a wallet, the funds are not available for spending, but they are still part of the wallet's balance. After being placed, a hold can be **confirmed**, or it can be **canceled**. When a hold is confirmed, the funds are no longer available for spending and are transferred to the recipient's wallet. When a hold is canceled, the funds are released back to the wallet's balance. ## Placing a hold While reading their brand new spellbook [acquired earlier](/modules/wallets/basic-operations), Eryldor found the spell they were looking for. However, they need to buy an extra component to cast it. As it is a rare component, the only way they can get it is by participating in an auction. Fortunately, at the auction-house, Eryldor found the component they need. They placed a bid of 2 COIN, and the auctioneer placed a hold on Eryldor's wallet to ensure that the funds are available when the auction ends, if Eryldor wins. You'll act as the auctioneer and place a hold on Eryldor's wallet. With `fctl`, you place a hold using the `wallets debit` command, as if you were debiting the wallet. To create a hold rather than a debit, you use the `--pending` flag. " }} body={{ amount: { asset: "COIN", amount: 2 }, pending: true }}> {"fctl wallets debit 2 COIN --pending --name eryldor"} You should see the following output: ```bash You are about to debit a wallets. Do you want to continue [Y/n]: y SUCCESS Wallet debited successfully with hold id '1f0dbe17-4be4-4db0-ab85-72404c059e73'! ``` ### Inspecting the hold Let's inspect Eryldor's wallet in the Formance Console. Open the Formance Console and navigate to the Wallets page. You should see a new hold on Eryldor's wallet. ``` fctl ui ``` Originally, assuming you've followed the previous tutorials, Eryldor's wallet had a balance of 5 COIN. After placing the hold, the wallet's balance became 3 COIN, and the hold was created with an amount of 2 COIN. Now, should Eryldor win the auction? You choose! ## Confirming the hold After the auction ends, Eryldor won the bid. The auctioneer will take the money from the hold placed earlier on Eryldor's wallet. To confirm the hold, you use the `fctl wallet hold confirm` command. " }} body={{ final: true }}> {"fctl wallets holds confirm --final"} ## Canceling the hold If Eryldor didn't win the auction, the hold can be canceled to release the funds back to the wallet's balance. To cancel the hold, you use the `fctl wallet hold void` command. " }}> {"fctl wallets holds void "} ## Partially confirming the hold In some cases, you may want to confirm only a portion of the hold. For example, Eryldor is ready to pay up to 4 coins for the auction. The auctioneer placed a hold of 4 coins on Eryldor's wallet. However, Eryldor won the auction with a bid of 3 coins. The auctioneer can confirm only 3 coins and cancel the remaining 1 coin. First, you'll put a hold of 4 coins on Eryldor's wallet. " }} body={{ amount: { asset: "COIN", amount: 4 }, pending: true }}> {"fctl wallets debit 4 COIN --pending --name eryldor"} You should see the following output: ```bash You are about to debit a wallets. Do you want to continue [Y/n]: y SUCCESS Wallet debited successfully with hold id '3af573f9-4c92-490b-9919-4848bf51fa33'! ``` If your wallet doesn't have enough funds, you can top it up using: ```bash fctl wallets credit "" COIN --name eryldor --source account=world ``` Now, you'll confirm only 3 coins of the hold. " }} body={{ amount: 3, final: true }}> {"fctl wallets holds confirm --amount 3 --final"} You should see the following output: ```bash SUCCESS Hold '3af573f9-4c92-490b-9919-4848bf51fa33' confirmed! ``` ## Summary In this section, you learned how to manage holds with Formance Wallets. You placed a hold on a wallet, confirmed the hold, canceled the hold, and partially confirmed the hold. You also learned how to inspect a wallet's holds in the Formance Console. --- ## Creating Wallets Source: https://docs.formance.com/modules/wallets/creating Create a wallet using the API. The response returns a blank wallet object you can store the reference of in your system. {"fctl wallets create my-wallet"} **Response:** ```json { "id": "e21494fe-dbd1-4323-8f2c-28c3bafb96d1", "balances": {}, "metadata": {} } ``` At this point, the wallet will start to exist on the underlying ledger, but it will not have any balances. ## Choosing a Wallet Strategy There are multiple strategies you can adopt when creating wallets: you can choose to create a wallet for a specific user, or you can create a wallet for a specific resource in your system. ## Using metadata Wallets carry a `metadata` field that can be used to store any information you want to associate with the wallet. This is useful when you want to e.g. attach a reference to a resource in your system to the wallet. {"fctl wallets create user-wallet --metadata user_id=1234"} --- ## Adding funds Source: https://docs.formance.com/modules/wallets/adding-funds Add funds to a wallet by issuing a credit via the API. The credit is applied to the wallet's balance immediately. All amounts use [Universal Monetary Notation](/modules/numscript/monetary-notation) — `USD/2` means US dollars with 2 decimal places, so `100` = $1.00. ## Credit a wallet The simplest credit — add $1.00 (100 in USD/2) from the default `world` source: " }} body={{ amount: { asset: "USD/2", amount: 100 }, sources: [{ type: "account", identifier: "world" }] }}> {"fctl wallets credit 100 USD/2 --name my-wallet --source account=world"} ## Funding from a specific account You can fund a wallet from any ledger account — not just `world`. Set the `sources` field to reference the account: " }} body={{ amount: { asset: "USD/2", amount: 100 }, sources: [{ type: "account", identifier: "orders:1234" }] }}> {"fctl wallets credit 100 USD/2 --name my-wallet --source account=orders:1234"} The source account must have sufficient funds. If it doesn't, the credit will fail with an `INSUFFICIENT_FUND` error. ## Funding from another wallet You can also fund a wallet from another wallet by using the `wallet` source type: " }} body={{ amount: { asset: "USD/2", amount: 100 }, sources: [{ type: "wallet", identifier: "" }] }}> {"fctl wallets credit 100 USD/2 --name my-wallet --source wallet="} ## Funding a specific balance By default, credits go to the `main` balance. To credit a named balance (e.g. a voucher or seasonal balance), include the `balance` field: " }} body={{ amount: { asset: "COIN", amount: 50 }, balance: "season-2024", sources: [{ type: "account", identifier: "world" }] }}> {"fctl wallets credit 50 COIN --name my-wallet --balance season-2024 --source account=world"} --- ## Spending funds Source: https://docs.formance.com/modules/wallets/spending-funds Spend funds from a wallet by issuing a debit. The debit reduces the wallet's balance immediately. ## Basic debit Debit $1.00 from a wallet. By default, funds go to the `world` account (representing the outside world): " }} body={{ amount: { asset: "USD/2", amount: 100 } }}> {"fctl wallets debit 100 USD/2 --name my-wallet"} ## Setting the destination Send debited funds to a specific ledger account instead of `world`: " }} body={{ amount: { asset: "USD/2", amount: 100 }, destination: { type: "account", identifier: "orders:1234" } }}> {"fctl wallets debit 100 USD/2 --name my-wallet --destination account=orders:1234"} ## Sending to another wallet You can also send funds directly to another wallet: " }} body={{ amount: { asset: "USD/2", amount: 100 }, destination: { type: "wallet", identifier: "" } }}> {"fctl wallets debit 100 USD/2 --name my-wallet --destination wallet="} The wallet must have sufficient funds. If it doesn't, the debit fails with an `INSUFFICIENT_FUND` error. --- ## Holding and confirming Source: https://docs.formance.com/modules/wallets/holding-confirm In some cases you want to reserve funds before confirming the debit — for example, holding funds while an order is being shipped, then confirming once it's delivered. ## Create a hold Issue a debit with `pending: true` to create a hold. The funds are reserved but not yet spent: " }} body={{ amount: { asset: "USD/2", amount: 100 }, pending: true }}> {"fctl wallets debit 100 USD/2 --pending --name my-wallet"} The response includes a `hold` field with the hold ID. Use this ID to confirm or void the hold. ## Confirm the hold Once the event is confirmed (e.g. order shipped), confirm the hold to finalize the debit: " }} body={{ final: true }}> {"fctl wallets holds confirm --final"} ## Void the hold If the event doesn't happen (e.g. order cancelled), void the hold to release the funds back to the wallet: " }}> {"fctl wallets holds void "} ## Partial confirmation You can confirm only part of a hold. For example, if you held $1.00 but only need to charge $0.75: " }} body={{ amount: 75, final: true }}> {"fctl wallets holds confirm --amount 75 --final"} The remaining $0.25 is automatically released back to the wallet when `final: true` is set. Without `final: true`, partial confirmations leave the hold open — you can confirm additional amounts later until you finalize it. --- ## Flows Source: https://docs.formance.com/modules/flows Flows is an integrated service that lets you quickly set up end-to-end money flows, without the headache of piecing together APIs and weaving together complex system interconnections. With a unified compatibility model, you can easily move value between different ledgers, wallets, and payment processors. Plus, Formance Flows takes care of translating and interpreting the transactions for you. On top of that, Formance Flows comes with flexible workflow capabilities, so you can create complex flows that account for delays or external events, as well as retry and fallback options. ## Examples Learn how to initiate a Stripe Connect transfer, sourcing funds from a Formance ledger account. Learn how to automate the deposit of funds on a user wallet once a payment is received. Learn how to move funds between accounts existing on different ledgers using Flows. --- ## Workflows definition Source: https://docs.formance.com/modules/flows/definition Within the Flows service, a _Workflow_ refers to a template that defines an ordered sequence of actions, called _stages_, to be executed as a Workflow _Instance_. ## Workflow Definition Syntax A workflow definition is a YAML file, with the following structure: ```yaml --- name: "my-workflow" stages: - send: # ... - wait_event: event: deposit.confirmed - delay: seconds: 7d - send: # ... ``` ## Available stages Currently, the following stages are supported: | Stage | Description | |------------|-----------------------------------------------------------------------------------------------------------------| | `send` | Transfers value from a source to a destination. This stage is compatible with ledgers, payments, and wallets. | | `wait_event` | Makes the workflow instance wait for an event to be emitted by an external system. | | `delay` | Makes the workflow instance wait for a given amount of time. | To learn more about each stage, please refer to the dedicated documentation page in the following section. ## Using variables Variables can be used in the workflow definition, by using the `${}` syntax. Variables will then be expanded according to the values passed at workflow execution time. ```yaml --- stages: - wait_event: event: "${event}" ``` ## Lifecycle Workflows have a lifecycle of their own, and need to be created before they can be executed as instances - follow on to the next section to learn more and create your first workflow. --- ## Workflows execution Source: https://docs.formance.com/modules/flows/execution ## Creating a Workflow Before running a workflow, we need a definition file. Let's begin by creating a simple one with the YAML below. ```yaml --- name: "my-workflow" stages: - send: source: account: id: "world" ledger: "flows-demo-001" destination: account: id: "deposits:${depositID}" ledger: "flows-demo-001" amount: amount: 100 asset: "JPY" ``` Let's save this file as `my-workflow.yaml`. We can now create the workflow using the following command: Executing the above command will save this workflow, and return an ID that can be used to execute it as a workflow instance. The output of the above command should look like this: ``` [SUCCESS] Workflow created with ID: e6415ff5-1d83-4853-998a-cac09ae1513c ``` For the sake of learning the available commands, let's verify that the workflow was successfully saved by listing all our created workflows: ## Executing a Workflow Alright; we have now created our first workflow, but nothing has happened yet within the ledger `flows-demo-001` that we used in the workflow definition. Let's jump straight to the fun part and execute our workflow as a workflow instance. We can do so using the following command: " }} body={{ variables: { depositID: "1234" } }}> {"fctl orchestration workflows run \\\n --variable depositID=1234"} ## Checking a Workflow instance status Workflow instances are long-lived. Their current state of execution and termination can be checked using the following command: ## Debugging a Workflow instance If you're having trouble understanding what a workflow instance is currently doing, you can use the following command to get a detailed view of its current internal state: ```shell fctl orchestration instances describe dff3791d-b82c-4ed5-bf35-e954872cd2af ``` ### Using the API for debugging For more detailed debugging, you can use the API endpoints directly: **Get instance history:** **Get specific stage history:** These endpoints provide detailed information about: - The sequence of events that occurred during execution - Input and output values at each stage - Error messages and stack traces for failed stages - Timestamps for each operation ## Terminating a Workflow instance There are cases where you might want to terminate a workflow instance, e.g. when you want to recreate it after making changes to definition or restart it after a failure with different variables. If you want to do so, you can simply use the following command: --- ## Triggers Source: https://docs.formance.com/modules/flows/triggers A trigger is a way to fire a workflow from a payment event. The trigger is linked to a unique workflow that will be executed only if its filter condition is satisfied. If so, it will forward values from the event to the workflow in a set of configured variables corresponding to the ones expected in the workflow. ## Creating a trigger To create a trigger, use the [Create Trigger](/stack-api-reference/orchestrationv2/create-trigger) endpoint: {"fctl orchestration triggers create SAVED_PAYMENT efxxxxx-xxxx-yyyy-dddd-d236abzzzzzz \\\n --filter 'event.type == \"PAY-IN\" && event.provider == \"ADYEN\" && hasPrefix(event.rawData.merchantReference, \"test\") == true' \\\n --vars paymentID=event.id \\\n --vars amount=event.amount \\\n --vars asset=event.asset \\\n --vars merchantID=\"'001'\" \\\n --vars userID=\"'003'\" \\\n --vars merchantReference=event.rawData.merchantReference"} ### Filter syntax The syntax for the filter is based on the [expr-lang expression language](https://expr-lang.org/docs/language-definition). Example filter: ``` event.type == "PAY-IN" && event.provider == "PROVIDERID" && hasPrefix(event.rawData.merchantReference, "test") == true ``` ## Testing a trigger Before deploying a trigger to production, you can test it to verify that the filter matches correctly and that variables are extracted as expected. Use the [Test Trigger](/stack-api-reference/orchestrationv2/test-trigger) endpoint. The payload should be the payment event you want to test against: " }} body={{ id: "dummyValue", type: "PAY-IN", asset: "EUR/2", amount: 4199, scheme: "visa", status: "SUCCEEDED", rawData: { amount: { value: 4199, currency: "EUR" }, reason: "012789:0000:03/2030", success: "true", eventCode: "AUTHORISATION", eventDate: "2023-12-15T15:22:32+01:00", operations: ["CANCEL", "CAPTURE", "REFUND"], pspReference: "XXXXXXXX", paymentMethod: "visa", additionalData: { authCode: "789789", expiryDate: "03/2030", cardSummary: "0000" }, merchantReference: "XXXX", merchantAccountCode: "XXXXX" }, metadata: {}, provider: "PROVIDERID", createdAt: "2023-12-15T15:22:32+01:00", reference: "XXXXXX", connectorId: "connectorID", initialAmount: 4199 }} noFctl /> The response shows whether the filter matched and the extracted variable values: ```json { "data": { "filter": { "match": true }, "variables": { "amount": { "value": "4199" }, "asset": { "value": "EUR/2" }, "merchantID": { "value": "001" }, "paymentID": { "value": "dummyValue" }, "userID": { "value": "003" } } } } ``` ## Evaluating metadata in triggers You can use `link()` and `get()` functions to retrieve metadata from related accounts in your trigger variables and filters. ### Accessing account metadata To retrieve metadata from a payment's associated account: ```json { "event": "SAVED_PAYMENT", "workflowID": "xxx", "vars": { "myVar": "get(link(event, \"destination_account\").metadata, \"foo\")" } } ``` This example retrieves the `foo` metadata field from the destination account linked to the payment event. ## Webhooks You can create webhooks to get notified of Flows events, whether they succeed or fail: | Event | Description | |-------|-------------| | `STARTED_WORKFLOW` | Workflow instance has started | | `SUCCEEDED_WORKFLOW` | Workflow instance completed successfully | | `FAILED_WORKFLOW` | Workflow instance failed | | `STARTED_WORKFLOW_STAGE` | A workflow stage has started | | `SUCCEEDED_WORKFLOW_STAGE` | A workflow stage completed successfully | | `FAILED_WORKFLOW_STAGE` | A workflow stage failed | | `SUCCEEDED_TRIGGER` | Trigger fired successfully | | `FAILED_TRIGGER` | Trigger failed to fire | You can find the complete list of available events in the [Formance events repository](https://github.com/formancehq/stack/blob/main/libs/events/generated/all.json). --- ## Banking Bridge Source: https://docs.formance.com/modules/payments/connectors/psp/bankingbridge The Banking Bridge connector polls a Banking Bridge workspace and surfaces accounts, balances, and payments as read-only streams. It applies the Payments module's most extensive per-payment enrichment: end-to-end IDs, mandate IDs, clearing-system references, creditor info, and remittance fields all land under `com.formance.connectors.bankingbridge.*` on every payment. Available from Payments 3.3.0. Source: [`ee/plugins/bankingbridge`](https://github.com/formancehq/payments/tree/main/ee/plugins/bankingbridge). The Banking Bridge connector requires Payments **3.3.0 or higher**. Your stack pins an older version — upgrade to use it. ## Prerequisites You need a Banking Bridge workspace and a `clientID` + `clientSecret` pair. Banking Bridge uses OAuth2 client-credentials against a separate `authEndpoint`; the connector exchanges the credentials for a short-lived bearer token and refreshes automatically. ## Installation {"fctl payments connectors install bankingbridge config.json"} With `config.json` containing: ```json { "name": "string", "clientID": "string", "clientSecret": "string", "endpoint": "https://api.bankingbridge.example", "authEndpoint": "https://auth.bankingbridge.example" } ``` ### Configuration fields | Field | Required | Default | Description | |---|---|---|---| | `name` | yes | — | A unique name for this connector instance. | | `clientID` | yes | — | Banking Bridge OAuth2 client ID. | | `clientSecret` | yes | — | OAuth2 client secret. Never logged. | | `endpoint` | yes | — | Data API base URL. | | `authEndpoint` | yes | — | OAuth2 token endpoint. Separate from the data endpoint so the connector can reach a dedicated auth host. | ## Capabilities - `FETCH_ACCOUNTS` — internal accounts via `GET /accounts`, paginated by an opaque cursor + `LastSeenImportedAt` watermark. - `FETCH_BALANCES` — per-account balance, independent periodic root. - `FETCH_PAYMENTS` — `GET /transactions`, each row carrying the full enrichment set under `com.formance.connectors.bankingbridge.*`. Payouts, transfers, bank-account creation, and webhooks are not implemented. Banking Bridge is a read-only observation surface. ## Account model Every Payments internal account is one Banking Bridge `/accounts` row (a bank account observed via one of the aggregated providers). The `reference` is the Banking Bridge account reference; `name` is the upstream name; `defaultAsset` comes from Banking Bridge's `defaultAsset` field (already UMN-formatted); `createdAt` is the upstream `ImportedAt`. No EXTERNAL accounts are emitted. See [Accounts](/modules/payments/accounts) for the cross-connector model. ## Asset model Uppercase ISO 4217, formatted to UMN at standard precision. Amounts are already in minor units — no scaling. ## Status mapping | Banking Bridge transaction status | Payment `status` | | --- | --- | | `BOOK`, `BOOKED`, `INFO` (terminal observation) | `SUCCEEDED` | | `PDNG`, `PENDING` | `PENDING` | | `RJCT`, `REJECTED` | `FAILED` | | anything else | `UNKNOWN` | ## Scheme mapping Banking Bridge tags each transaction with an ISO 20022 triplet (`Domain.Family.SubFamily`). The connector collapses it into the Payments module's `PaymentScheme` + `PaymentType` via [`schemes.go`](https://github.com/formancehq/payments/blob/main/ee/plugins/bankingbridge/schemes.go): - `PMNT.ICRD` / `PMNT.MCRD` (cards) → `CARD_*`, PAY-IN or PAYOUT per `IssuedX` vs `ReceivedX`. - `PMNT.ICDT` / `PMNT.RCDT` (credit transfers) → `SEPA` / `WIRE` per sub-family. - `PMNT.IDDT` / `PMNT.RDDT` (direct debits) → `SEPA_DEBIT`. - `PMNT.IRCT` / `PMNT.RRCT` (real-time SCT) → `SEPA_INSTANT`. - Unknown / `MCOP` / `MDOP` → `UNKNOWN`, classified `OTHER`. ## Metadata keys Banking Bridge fields land under `com.formance.connectors.bankingbridge.*` (not `com.bankingbridge.spec/`) — Banking Bridge is the canonical example of the [payment-reference enrichment pattern](/modules/payments/operations). Operationally useful keys: - **Account**: `iban`, `bic`, `currency_code`, `account_type`, `provider_name`, `provider_id`, `branch_code`, `imported_at`. - **Payment** (always emitted when populated by the provider): - End-to-end identifiers: `end_to_end_id`, `instruction_id`, `transaction_id`, `mandate_id`, `clearing_system_reference`. - Counterparty info: `debtor_name`, `debtor_iban`, `debtor_bic`, `creditor_name`, `creditor_iban`, `creditor_bic`. - Creditor reference: `creditor_reference_type`, `creditor_reference_value`, `creditor_reference_issuer`. - Remittance: `remittance_information_unstructured`, `remittance_information_structured`. - Bank-side fields: `bank_transaction_code` (the raw `Domain.Family.SubFamily` triplet), `value_date`, `booking_date`, `provider_name`, `provider_id`. ## Workflow tree ```text FetchAccounts (periodic root) FetchPayments (periodic root) FetchBalances (periodic root) ``` Each root holds its own cursor + `LastSeenImportedAt` watermark and advances independently — Banking Bridge's `/transactions` endpoint is workspace-global, not per-account. ## Pagination and recovery Each capability persists `(cursor, lastSeenImportedAt)` in platform-managed `State`. The `cursor` is opaque; `lastSeenImportedAt` re-seeds the next cycle's lower bound after the current page set is exhausted. Restarts resume from the last committed `State`; the engine dedupes by `PSPPayment.Reference`. ## Known gaps - **Outbound initiation** (transfers, payouts, bank-account creation) is not implemented. - **Webhooks** are not wired. - **Reversals** are not modelled as reversal events — reversed transactions appear as new rows with the `RRTN` / `XRTN` sub-family code. - **Multi-workspace tenants**: one connector install covers one Banking Bridge workspace. --- ## Send Statement Source: https://docs.formance.com/modules/flows/stages/send The `send` statement is the main and most powerful stage of the Workflow definition. It moves funds between ledger accounts, wallets, and payment service providers (PSPs), handling the underlying ledger transactions automatically. A `send` stage has three attributes: - `source` — where the funds come from - `destination` — where the funds go - `amount` — how much, and which asset Both `source` and `destination` may reference a ledger account, a wallet, or a payment. Skeleton: ```yaml --- stages: - send: source: # ... destination: # ... amount: amount: 100 asset: "EUR/2" ``` ## Compatibility matrix | | ↘ Ledger account | ↘ Wallet | ↘ Payment | | --- | --- | --- | --- | | ↗ **Ledger account** | ✅ | ✅ | ✅ | | ↗ **Wallet** | ✅ | ✅ | ✅ | | ↗ **Payment** | ✅ | ✅ | ❌ | Payment destinations work with **any PSP connector configured in the Payments service** — Stripe, Wise, Modulr, Banking Circle, Currency Cloud, Mangopay, and the rest. Validation is delegated to Payments, so new connectors light up here automatically. | | ↘ Ledger account | ↘ Wallet | ↘ Payment | | --- | --- | --- | --- | | ↗ **Ledger account** | ✅ | ✅ | Partial support **[1]** | | ↗ **Wallet** | ✅ | ✅ | Partial support **[1]** | | ↗ **Payment** | ✅ | ✅ | ❌ | **[1]** Payment destinations are currently supported only for the `stripe` connector. Support for the rest will be added in a later release. ## Source types ### Ledger account Reference a ledger account by `id` (the address) and `ledger` (the ledger name): ```yaml source: account: id: "users:42" ledger: "flows-demo-001" ``` For cross-ledger flows you can also customize the intermediate "bridge" account via `throughAccount` and allow it to go negative via `allowOverdraft`: | Field | Type | Required | Default | Description | | --- | --- | --- | --- | --- | | `id` | string | yes | — | Ledger account address | | `ledger` | string | yes | — | Ledger name | | `throughAccount` | string | no | `"world"` | Intermediate account for cross-ledger or payment flows | | `allowOverdraft` | bool | no | `false` | Allow unbounded overdraft on the source account | ```yaml source: account: id: "users:123" ledger: "main" throughAccount: "liabilities:pending" allowOverdraft: true ``` ### Wallet Reference a wallet by either `id` or `name`. `balance` selects a specific balance (defaults to `main`): ```yaml source: wallet: id: "22d5de50-b5ef-407d-9a03-9e4fc36356f8" balance: "main" # optional ``` ### Payment Reference an existing payment (a payin) by `id`: ```yaml source: payment: id: "22d5de50-b5ef-407d-9a03-9e4fc36356f8" ``` The payment is "ingested" — moved into a ledger as an intermediate step — before being transferred to the destination. By default this uses an internal orchestration ledger; you can override the ingestion target: | Field | Type | Required | Default | Description | | --- | --- | --- | --- | --- | | `id` | string | yes | — | Payment ID from the Payments service | | `ledger` | string | no | `"orchestration-000-internal"` | Ledger for payment ingestion | | `holdingAccount` | string | no | `"payment:{id}"` | Account where ingested funds are held | | `throughAccount` | string | no | `"world"` | Source account for the ingestion transaction | | `allowOverdraft` | bool | no | `false` | Allow unbounded overdraft on `throughAccount` | ```yaml source: payment: id: "${paymentID}" ledger: "main" holdingAccount: "assets:stripe:held" throughAccount: "assets:stripe:incoming" allowOverdraft: true ``` See [Payment ingestion](#payment-ingestion) below for the full picture. ## Destination types ### Ledger account ```yaml destination: account: id: "users:42" ledger: "flows-demo-001" ``` Same `throughAccount` / `allowOverdraft` options as ledger-account sources — they govern the intermediate account used on cross-ledger transfers: ```yaml destination: account: id: "merchants:456" ledger: "main" throughAccount: "assets:incoming" allowOverdraft: true ``` ### Wallet ```yaml destination: wallet: id: "22d5de50-b5ef-407d-9a03-9e4fc36356f8" ``` ### Payment Unlike source payments, destination payments aren't referenced by ID (the payment doesn't exist yet). Instead, you tell the stage which PSP connector to use: ```yaml destination: payment: psp: "stripe" ``` The `send` statement creates a payment initiation against the PSP named in `psp`. Only `stripe` is supported at this version — other connectors will be added in a later release. ```yaml destination: payment: psp: "stripe" type: "PAYOUT" sourceAccount: "${sourceAccountID}" metadata: "stripeConnectID" ``` The full field set: | Field | Type | Required | Default | Description | | --- | --- | --- | --- | --- | | `psp` | string | yes | — | PSP connector name (`"stripe"`, `"wise"`, `"modulr"`, …) | | `type` | string | no | `"TRANSFER"` | `"TRANSFER"` or `"PAYOUT"` | | `metadata` | string | no | — | Account-metadata key carrying the destination PSP account ID | | `sourceAccount` | string | no | — | Explicit PSP source account ID for the transfer | | `connectorID` | string | no | — | Specific connector ID (when multiple connectors of the same type exist) | | `waitingValidation` | bool | no | `false` | Whether to wait for manual validation before settling | The ledger or wallet debit happens **before** the PSP transfer is initiated. On PSP failure the workflow is left in a consistent state — funds remain debited from the source until the workflow handles the failure explicitly. ## Cross-ledger transfers When source and destination live on different ledgers, the `throughAccount` field on each side controls the intermediate "bridge" account used. ### Same ledger (direct transfer) ```yaml send: source: account: { id: "users:123", ledger: "main" } destination: account: { id: "merchants:456", ledger: "main" } ``` A single transaction posts on `main` from `users:123` to `merchants:456`. ### Different ledgers (bridge transfer) ```yaml send: source: account: id: "users:sender" ledger: "ledger1" throughAccount: "bridge:outbound" destination: account: id: "merchants:receiver" ledger: "ledger2" throughAccount: "bridge:inbound" allowOverdraft: true ``` Two transactions post: 1. On `ledger1`: `users:sender` → `bridge:outbound` 2. On `ledger2`: `bridge:inbound` → `merchants:receiver` `bridge:inbound` may need `allowOverdraft: true` if it isn't pre-funded. ## The `allowOverdraft` field By default the Ledger requires source accounts to have sufficient funds. The special `"world"` account has unbounded overdraft and is used by default for bridge transactions — which is why the cross-ledger example above works without explicit overdraft when the default `world` bridge is in play. When you point `throughAccount` at a custom account, you may need `allowOverdraft: true` to let the account go negative. The generated Numscript carries an `allowing unbounded overdraft` clause. | Flow | Transaction | Overdraft applied to | | --- | --- | --- | | Account → Payment | `source.id → throughAccount` | `source.id` | | Payment → Account | `throughAccount → holdingAccount` | `throughAccount` | | Account → Wallet (cross-ledger) | `source.id → throughAccount` | `source.id` | | Wallet → Account (cross-ledger) | `throughAccount → destination.id` | `throughAccount` | | Account → Account (1st tx) | `source.id → sourceThroughAccount` | `source.id` | | Account → Account (2nd tx) | `destThroughAccount → destination.id` | `destThroughAccount` | ### Example: liability tracking for payouts ```yaml send: source: account: id: "users:${userID}" ledger: "main" throughAccount: "liabilities:payouts-pending" allowOverdraft: true destination: payment: psp: "stripe" type: "PAYOUT" ``` Posts as `users:{userID}` → `liabilities:payouts-pending` instead of `users:{userID}` → `world`, so the in-flight payout shows up on your liabilities account until the PSP confirms settlement. ## Payment ingestion When the source is a payment, the funds are ingested into a ledger before being transferred to the destination. By default this happens on an internal orchestration ledger: ``` 1. world → payment:{paymentID} on orchestration-000-internal 2. payment:{paymentID} → world on orchestration-000-internal (with metadata) 3. world → destination on the destination ledger ``` To ingest directly into your own ledger and account scheme: ```yaml source: payment: id: "${paymentID}" ledger: "main" holdingAccount: "assets:stripe:held" throughAccount: "assets:stripe:incoming" allowOverdraft: true ``` Produces: ``` 1. assets:stripe:incoming → assets:stripe:held on main 2. assets:stripe:held → destination on main ``` ## Complete examples ```yaml name: "payout-with-tracking" stages: - send: source: account: id: "users:${userID}" ledger: "main" throughAccount: "liabilities:payouts-pending" allowOverdraft: true destination: payment: psp: "${psp}" type: "PAYOUT" sourceAccount: "${sourceAccountID}" amount: amount: "${amount}" asset: "${asset}" ``` ```yaml name: "payin-custom-ingestion" stages: - send: source: payment: id: "${paymentID}" ledger: "main" holdingAccount: "assets:stripe:pending" throughAccount: "assets:stripe:bridge" allowOverdraft: true destination: account: id: "revenue:${merchantID}" ledger: "main" amount: amount: "${amount}" asset: "${asset}" ``` ```yaml name: "cross-ledger-transfer" stages: - send: source: account: id: "users:${userID}" ledger: "users-ledger" throughAccount: "bridge:to-merchants" destination: account: id: "merchants:${merchantID}" ledger: "merchants-ledger" throughAccount: "bridge:from-users" allowOverdraft: true amount: amount: "${amount}" asset: "${asset}" ``` --- ## Waiting for events Source: https://docs.formance.com/modules/flows/stages/wait-event Using the `wait_event` stage, you can wait for an event to be submitted to a running workflow instance before continuing. This is useful for workflows that require user input, such as a manual approval of a transaction. ```yaml --- stages: - wait_event: event: "payout.confirmed" ``` Events can be then submitted to running workflow instances using the API or fctl: --- ## Waiting for a delay Source: https://docs.formance.com/modules/flows/stages/delay The `delay` stage is used to wait for a specific duration or until a specific date: ```yaml --- stages: - delay: duration: 60s ``` ```yaml --- stages: - delay: # until: 2023-06-01T00:00:00Z ``` --- ## Ledger to Ledger Source: https://docs.formance.com/modules/flows/examples/ledger-to-ledger In the Formance Ledger world, ledgers are logically separated from one another. Sometimes you need to transfer value from one ledger to another. Fortunately for us, Flows provides a way to do this that takes care of the underlying details for us. ## Workflow definition ```yaml --- stages: # let's first provision a user account on our first ledger - send: source: account: id: "world" ledger: "flows-demo-001" destination: account: id: "users:42" ledger: "flows-demo-001" amount: asset: "USD/2" amount: 100 # now let's transfer that value to our second ledger - send: source: account: id: "users:42" ledger: "flows-demo-001" destination: account: id: "users:42" # note that we're sending to a different ledger ledger: "flows-demo-002" amount: asset: "USD/2" amount: 100 ``` ## Running the workflow Let's save the above workflow to a file called `ledger-to-ledger.yaml` and create it: Now run the workflow: " }} /> We can now check the status of the workflow instance: " }} /> --- ## Payment to Wallet Source: https://docs.formance.com/modules/flows/examples/payment-to-wallet - At least one payment object synced from the connector ## Workflow definition ```yaml --- name: "payment-to-wallet-demo" stages: - send: source: payment: id: "${paymentID}" destination: account: id: "deposits:${depositID}:pending" ledger: "flows-demo-001" amount: amount: 100 asset: "USD/2" - wait_event: event: "deposit.confirmed" - send: source: account: id: "deposits:${depositID}:pending" ledger: "flows-demo-001" destination: wallet: id: "${walletID}" amount: amount: 100 asset: "USD/2" ``` Let's save this file as `payment_to_wallet.yaml`. We can now create this workflow with the following command: Now that we have a workflow, we can run a workflow instance with the following command: " }} body={{ variables: { paymentID: "ch_1G4Z4p2eZvKYlo2C4q0Z0Z0Z", walletID: "wallet_1G4Z4p2eZvKYlo2C4q0Z0Z0Z", depositID: "deposit_1G4Z4p2eZvKYlo2C4q0Z0Z0Z" } }}> {"fctl orchestration workflows run \\\n--variable paymentID=ch_1G4Z4p2eZvKYlo2C4q0Z0Z0Z \\\n--variable walletID=wallet_1G4Z4p2eZvKYlo2C4q0Z0Z0Z \\\n--variable depositID=deposit_1G4Z4p2eZvKYlo2C4q0Z0Z0Z"} Note that we're passing in the payment ID, wallet ID, and deposit ID as variables. These variables are used in the workflow definition to reference the payment, wallet, and deposit objects. You'll need to replace these values with the IDs of the payment and wallet you prepared in the prerequisites. --- ## Routable Source: https://docs.formance.com/modules/payments/connectors/psp/routable The Routable connector polls a Routable workspace and surfaces settings accounts, companies (counterparties), payables, and receivables. It also initiates outbound payables in response to Formance `CreateTransfer` and `CreatePayout` workflows. ## Prerequisites You need a Routable account and an API key. Decide upfront which **team member** will own payables initiated through Formance — Routable requires an `acting_team_member` on every payable. Set it per-connector via config, or per-request via metadata; see [Initiating payouts and transfers](#initiating-payouts-and-transfers). ## Installation {"fctl payments connectors install routable config.json"} ### Configuration fields | Field | Required | Default | Purpose | |---|---|---|---| | `apiKey` | yes | — | Routable bearer token. Sent as `Authorization: Bearer ` on every request. | | `endpoint` | no | `https://api.routable.com` | API root. Use `https://api.sandbox.routable.com` for the Routable sandbox. | | `actingTeamMember` | no | `""` | Default Routable team member ID for payable creation. Optional at the connector level — callers can override per-request via the `com.routable.spec/acting_team_member` metadata key. If neither is set, payable creation fails with a clear validation error before any HTTP call. | | `pollingPeriod` | no | `30m` | Sync cadence (minimum 20m). | The connector validates the API key at install with a `GET /v1/settings/accounts?page=1&page_size=1` probe — bad keys fail install rather than the first `FETCH_ACCOUNTS` cycle. ## Capabilities - `FETCH_ACCOUNTS` — settings accounts via `GET /v1/settings/accounts`. - `FETCH_BALANCES` — settings-account `available_amount` via `GET /v1/settings/accounts/{id}`. - `FETCH_EXTERNAL_ACCOUNTS` — companies (counterparties) via `GET /v1/companies`. - `FETCH_PAYMENTS` — payables and receivables via `GET /v1/payables` and `GET /v1/receivables`. - `CREATE_TRANSFER` and `CREATE_PAYOUT` — both `POST /v1/payables` (transfers are payables with the `TRANSFER` type). Webhooks, bank-account creation, and reversals are not implemented. Routable caps API throughput at roughly 1.5 payouts per second. The connector declares this via `PluginWithPayoutThrottle`, so the platform throttles `CreatePayout` / `CreateTransfer` through a dedicated Temporal task queue. See [Operations → Connector reliability](/modules/payments/operations). ## Account model Every Payments internal account is one Routable settings account from `GET /v1/settings/accounts`. The `reference` is the settings-account ID; `defaultAsset` is the account currency. EXTERNAL accounts come from `/v1/companies` (the payee surface), refreshed on a 24-hour cadence to respect upstream rate limits. See [Accounts](/modules/payments/accounts) for the cross-connector model. ## Asset model Routable returns ISO currency codes (`USD`, `EUR`, `KWD`). Formatted to UMN via `currency.FormatAsset` — `USD/2`, `EUR/2`, `KWD/3`. Unsupported currencies are skipped (logged) without dropping the rest of the page. Amounts arrive as decimal strings and convert to integer minor units with half-up rounding. ## Status mapping Routable payable statuses → Payments module payment statuses: | Routable `status` | Payment `status` | | --- | --- | | `draft`, `ready_to_send`, `pending`, `scheduled`, `initiated`, `processing`, `in_transit`, `awaiting_delivery` | `PENDING` | | `completed`, `paid`, `externally_paid`, `delivered` | `SUCCEEDED` | | `failed`, `returned`, `nsf` | `FAILED` | | `stopped`, `canceled`, `cancelled`, `voided` | `CANCELLED` | | `expired` | `EXPIRED` | | anything else (or empty) | `UNKNOWN` | Comparison is case-insensitive. Payment `scheme` is mapped from `delivery_method`: `ach_*` → `ACH`, everything else → `OTHER`. ## Metadata keys Under `com.routable.spec/`. Full list in the connector's `MAPPINGS.md`; highlights: ### Account metadata (internal accounts) `object`, `type`, `is_valid`, `currency_code`, plus `type_details.account_type`, `type_details.bank_name`, `type_details.account_number`, `type_details.routing_number`. ### External-account metadata (companies) `object`, `type`, `status`, `country_code`, `is_vendor`, `is_customer`, `is_archived`, `external_id`, `business_name`, `display_name`, plus the structured `registered_address.*` keys. ### Payment metadata For payables (PAYOUT) and receivables (PAYIN): `type`, `delivery_method`, `status`, `external_id`, `memo`, `reference`. Two correlation aliases also land on every Formance-initiated Payment: - `com.routable.spec/payment_initiation_reference` — the originating PaymentInitiation reference (absent on payables created in Routable's UI). - `com.routable.spec/payable_id` — the Routable payable UUID (mirrors `Payment.Reference`). See [Correlating an initiation with the synced payment](#correlating-an-initiation-with-the-synced-payment). ## Initiating payouts and transfers `CreateTransfer` and `CreatePayout` translate the Formance `PSPPaymentInitiation` into a `POST /v1/payables`. Most fields come from the structured initiation; a few Routable knobs are exposed via metadata: | Metadata key | Required | Default | Maps to | Purpose | |---|---|---|---|---| | `com.routable.spec/type` | no | `ach` | `type` | Payable rail (`ach`, `wire`, `check`, `international`, `external`, `vendor_choice`). | | `com.routable.spec/delivery_method` | no | `ach_standard` | `delivery_method` | Specific delivery option (`ach_standard`, `ach_same_day`, `wire`, `check`, …). Must be compatible with `type`. | | `com.routable.spec/acting_team_member` | conditional | connector config | `acting_team_member` | Routable team member ID initiating the payable. Required at request time — from either the connector config or this key. | | `com.routable.spec/external_id` | no | `""` | `external_id` | Caller-supplied external reference (idempotent lookup key on Routable's side). | | `com.routable.spec/line_item_description` | no | `PSPPaymentInitiation.Description`, then `"Payment "` | `line_items[0].description` | Description on the auto-generated single-line item. Routable v1 requires a non-empty value. | The Formance `PaymentInitiation.Reference` is sent as the `Idempotency-Key` header — Routable returns the original payable on retries with the same key, which Formance's create-then-poll workflow relies on. ### Async response handling `POST /v1/payables` answers in two shapes: | Routable response | Behavior | |---|---| | `202 Accepted` (async) | `PollPayoutStatus` / `PollTransferStatus` scheduled against `GET /v1/payables/{id}`; the first successful poll links the PaymentInitiation to a Payment and ends the loop. | | `201 Created` with a **terminal** status (`completed`, `failed`, `cancelled`, `expired`) | Payment returned immediately. | | `201 Created` with a **non-terminal** status | Polling round scheduled; first poll returns the Payment. | Once linked, further transitions (PENDING → PROCESSING → SUCCEEDED) flow through the periodic `FETCH_PAYMENTS` schedule. ### Correlating an initiation with the synced payment Initiating a payable through Formance produces two rows: - A `PaymentInitiation` keyed by the reference you supplied (e.g. `payout-acmecorp-20260506-172725`). - A `PSPPayment` keyed by Routable's payable UUID. They're linked at the engine level. To resolve one from the other: ```bash curl "$STACK/api/payments/v3/payment-initiations/$PI_ID/payments" \ -H "Authorization: Bearer $TOKEN" | jq '.cursor.data[] | {reference, status}' ``` Formance-initiated Payments carry the originating reference under `com.routable.spec/payment_initiation_reference` (and its Routable alias `com.routable.spec/external_id`). Payments created in Routable's UI don't have these keys. ```bash curl -s "$STACK/api/payments/v3/payments" \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{"$match": {"connectorID": "'$CONNECTOR_ID'"}}' \ | jq '.cursor.data[] | { payment_ref: .reference, pi_ref: .metadata."com.routable.spec/payment_initiation_reference", payable_id: .metadata."com.routable.spec/payable_id" }' ``` ## Pagination and recovery List endpoints are 1-indexed `page` + `page_size` (cap 100). The connector checkpoints pagination opaquely between cycles, so a mid-cycle worker crash resumes deterministically — no row double-billed or dropped. The payments fetcher walks payables, then receivables, then advances its cycle watermark. The watermark stays **immutable for the full duration of a cycle** to avoid the page-2-tighter-than-page-1 race that would drop rows whose `status_changed_at` lands between page boundaries. Routable's `status_changed_at.gte` filter is inclusive, so cycle-boundary rows re-emit each cycle. The engine dedupes by `PSPPayment.Reference` — wasted bandwidth, never a correctness issue. ## Known gaps - **Webhooks** — Routable exposes a webhook product upstream; the connector uses polling. Subscription shape in upstream `MAPPINGS.md §6.4`. - **Bank-account creation** — companies and delivery methods are pulled, not created. Manage new companies in Routable's UI. - **Reversals** — `ReverseTransfer` and `ReversePayout` are not wired. - **Throughput cap** — Routable caps API throughput at ~1.5 payouts/s. The platform throttles outbound workflows via a dedicated task queue; bursts queue under load rather than fail. --- ## Ledger to Stripe Payout Source: https://docs.formance.com/modules/flows/examples/ledger-to-payout In this example, we're going to create a workflow that will transfer funds from a ledger account to a Stripe Connect account by leveraging the `send` stage. To initiate a transfer from a ledger account to a Stripe Connect account, you must have set the `formanceAccountID` metadata key on the ledger account metadata to identify the formance account ID related to the Stripe Connect account to initiate the transfer to. ## What's happening in this example? Before we dive into the workflow definition, let's take a look at what's happening in this example. As you may imagine, when we say "transfer funds from a ledger account to a Stripe Connect account", we don't mean that funds are actually being teleported from our own ledger account to our Stripe Connect account balance. Instead, we'll be doing two things: 1. Recycle the funds we previously introduced in the ledger by transferring them to the `@world` account 2. Transfer the funds from our main Stripe balance to the target Stripe Connect account ```mermaid graph TD A["payouts:1234"] -->|"100 USD/2"| B["world"] C["stripe main balance"] -->|"100 USD/2"| D["stripe connect account (acct_xxx)"] A -.->|"flows orchestration"| D ``` *Ledger transaction (top flow) and Stripe Connect transfer (bottom flow) orchestrated by Flows* The Flows service will be taking care of the sequencing for us. - A ledger account provisioned with funds - A `formanceAccountID` metadata key set on the account, corresponding to the Stripe Connect account to transfer funds to ## Workflow definition Here's the workflow definition we'll be using in this example: ```yaml --- name: "ledger-to-stripe-payout-demo" stages: - send: source: account: id: "payouts:1234" ledger: "flows-demo-001" destination: payment: psp: "stripe" amount: amount: "${amount.amount}" asset: "${amount.asset}" ``` --- ## Reconciliation Source: https://docs.formance.com/modules/reconciliation For the mental model of how Reconciliation, Ledger, and Payments work together, see [How the Modules Fit Together](/getting-started/modules-fit-together). Formance Reconciliation continuously checks that your financial state matches the controls you define. It can compare Ledger balances with external cash, verify relationships inside a ledger, and watch account groups for unexpected balances. When a control fails, Reconciliation preserves the numbers behind the failure, opens a stable alert, and records how your team handled it. This turns reconciliation from a one-off balance check into an operating process: ```mermaid flowchart LR Observe["Observescheduled controls"] --> Detect["Detecta failed check"] Detect --> Alert["Alertthe right team"] Alert --> Evidence["Reviewfrozen evidence"] Evidence --> Resolve["Resolveor accept"] ``` ## What you can control - **External backing:** confirm that cash held by a payment provider matches the corresponding Ledger position. - **Ledger integrity:** prove that two or more groups of Ledger accounts remain in balance. - **Balance thresholds:** detect when a treasury, suspense, reserve, or operational account group moves outside an expected range. - **Source parity:** compare any two supported balance sources, including ledger-to-ledger, ledger-to-pool, and pool-to-pool checks. Controls run on demand or on a cron schedule. Each run produces a persisted evaluation with a `PASS`, `FAIL`, or `ERROR` result. Failures open one alert per affected asset, so different currencies can be investigated and closed independently. ## From detection to closure An alert remains a single operational case within its reconciliation period. Your team can acknowledge it, temporarily snooze notifications, mark it fixed after corrective bookings, or formally accept the discrepancy with an author and required note. A later passing evaluation closes it automatically. Every evaluation and manual transition is retained in the alert timeline. This lets finance and compliance teams answer not only *whether* a control failed, but also when it began, what values were observed, who handled it, and why it was closed. Reconciliation checks aggregate financial state. It does not match individual Ledger postings to individual bank or PSP statement lines. ## Explore Reconciliation Learn how rules, evaluations, alerts, evidence, and reconciliation periods fit together. Create a Ledger integrity rule, evaluate it, and inspect the result. Compare the four control templates and configure their balance sources and tolerances. Acknowledge, snooze, resolve, or accept a discrepancy without losing its history. Formance Reconciliation compares balances between your Formance Ledger and [cash pools](/modules/payments/cash-pools) to verify financial consistency and identify discrepancies that need investigation. You create a policy that selects Ledger accounts and a cash pool. When you run the policy, Reconciliation reads both sides at the requested timestamps and reports whether their per-asset balances match. ```mermaid flowchart LR Pool["Cash poolexternal funds"] --> Compare["Reconciliationpolicy"] Ledger["Ledger accountsrecorded position"] --> Compare Compare --> Result["OK or NOT_OKwith drift by asset"] ``` Regular checks help you detect discrepancies, prepare financial reports, and retain a history of the balances observed. Start with [Concepts](/modules/reconciliation/concepts), then follow [Getting Started](/modules/reconciliation/getting-started) to create and run a policy. This version provides synchronous, on-demand policy runs. Rules, scheduled evaluations, managed alerts, and resolution workflows require Reconciliation 2.4.0 or later. --- ## Concepts Source: https://docs.formance.com/modules/reconciliation/concepts Reconciliation organizes continuous financial controls into five client-facing concepts: **rules**, **evaluations**, **alerts**, **evidence**, and **resolutions**. ```mermaid flowchart LR Rule["Rulewhat and when to check"] --> Evaluation["Evaluationone execution"] Evaluation -- FAIL --> Alert["Alertone case per asset and period"] Evaluation -- PASS --> Proof["Green proof"] Alert --> Timeline["Evidence andevent timeline"] Alert --> Resolution["Auto-resolve,fix, or accept"] ``` ## Rules A **rule** defines the business control. It contains: - a control template and its configuration; - an on-demand or cron **schedule**, controlling how often it runs; - a severity, and a **period length** for grouping its results; - optional key-value labels copied to alerts and webhook event payloads; - an enabled state. Templates keep the public configuration focused on financial intent. You choose a supported comparison and provide its sources, assets, signs, thresholds, or tolerances; the underlying expression is generated and validated by Reconciliation. Labels are metadata for downstream consumers; Reconciliation does not interpret them or route work by itself. For example, `{ "team": "treasury", "environment": "production" }` lets your webhook consumer send the resulting alert to the treasury queue while retaining the same tags in its audit record. Rules are enabled by default. Disabling a rule stops scheduled and on-demand evaluations without deleting its existing evidence or alerts; see [Pause or resume a rule](/modules/reconciliation/getting-started#6-pause-or-resume-a-rule) for the operational effects. See [Control Templates](/modules/reconciliation/controls) for the available controls. ## Evaluations An **evaluation** is one execution of a rule. It is always persisted and has one of three results: | Result | Meaning | | --- | --- | | `PASS` | Every checked asset satisfied the control. | | `FAIL` | At least one asset fell outside the control. | | `ERROR` | Reconciliation could not complete the check, for example because a source was unavailable. | For each asset, a passing result stores a compact proof containing the observed balances and predicate inputs needed to verify the result after a rule edit. A failing result stores a fuller evidence breakdown, including the observed values, calculated difference, configured tolerance, and the generated expression used to explain the check. Amount comparisons use arbitrary-precision integers, so large financial values remain exact. ### Point-in-time reads Each balance source is read at an explicit point in time. By default, Reconciliation subtracts a 30-second **safety margin** from `at`. For example, an evaluation requested at `10:00:00` reads the sources at `09:59:30`. The margin avoids evaluating the newest edge of the data, where a Ledger write, connector poll, or balance update may still be reaching one source but not another. It reduces false discrepancies caused by ingestion timing; it does not change or delay the underlying financial activity. Omit `safetyMargin` to use the 30-second default. Set it explicitly to `"0s"` to read exactly at `at`; this explicit zero is preserved on scheduled rules and is not replaced by the default. It is useful for deterministic tests, demos, and historical evaluations where you control the data and need the exact requested instant. You can also override the effective timestamp for an individual source when systems settle on different cycles. After resolving the balances, Reconciliation evaluates immutable per-asset snapshots without reading the sources again. The resolved timestamps are stored in `pitPerSource` on the evaluation. Together with the frozen proof or evidence, this records the financial state that produced the verdict. If you omit `at`, Reconciliation asks Payments for its latest known balance. Payments returns the balance but not the time at which that snapshot became effective, so Reconciliation can record only when it fetched the value. Replaying a historical query at that fetch time may therefore return a different snapshot. For a reproducible run, provide a past `at` or an explicit Payments timestamp in `sourcePITs`; regardless of the read mode, the values stored in the evaluation remain unchanged. ## Alerts A failed evaluation opens an **alert** for each failing fingerprint. In the current templates, the fingerprint is the asset, such as `asset:USD/2`. A USD discrepancy and a EUR discrepancy therefore become separate cases that can resolve independently. Alerts have three statuses: ```mermaid stateDiagram-v2 [*] --> OPEN: first failure OPEN --> ACKNOWLEDGED: operator acknowledges OPEN --> RESOLVED: passes, fixed, or accepted ACKNOWLEDGED --> OPEN: later failure resurfaces ACKNOWLEDGED --> RESOLVED: passes, fixed, or accepted RESOLVED --> OPEN: fails again in the same period ``` The alert row represents the current state. Its event timeline records every evaluation that touched it and every acknowledgement, snooze, resolution, acceptance, or reopen. ## Reconciliation periods Each rule groups its results into **reconciliation periods**. The `periodType` field sets how long a period is: `daily`, `weekly`, `monthly`, or `continuous` for no boundary at all. The `cadence` field sets how long a period is: `daily`, `weekly`, `monthly`, or `continuous` for no boundary at all. `periodType` sets the length of a period, not how often the rule runs — that is its [schedule](#rules), a separate field. A rule with an hourly schedule and a `monthly` `periodType` runs hundreds of times in July, and those runs are grouped into one period, `2026-07`, rather than into hundreds of separate ones. `cadence` sets the length of a period, not how often the rule runs — that is its [schedule](#rules), a separate field. A rule with an hourly schedule and a `monthly` cadence runs hundreds of times in July, and those runs are grouped into one period, `2026-07`, rather than into hundreds of separate ones. Grouping uses the effective read timestamp rather than the moment the rule fired, so the [safety margin](#point-in-time-reads) decides which side of a boundary a run falls on. With the 30-second default, a run at `2026-07-01T00:00:00Z` reads at `2026-06-30T23:59:30Z` and is therefore grouped under `2026-06`. **Renamed in 2.5.0.** This field was called `cadence` in earlier versions. Nothing breaks: both names are accepted when creating a rule, and both are returned, so existing integrations keep working untouched. Sending both is rejected with a `400` unless the two values are equal — rather than silently picking one. `cadence` is deprecated and will be removed in a future major version, so prefer `periodType` in new code. | Period type | Period | Example period id | | --- | --- | --- | | `continuous` | One ongoing case per asset, with no period boundary | `continuous` | | `daily` | Each UTC day, certified separately | `2026-07-21` | | `weekly` | Each ISO week | `2026-W30` | | `monthly` | Each calendar month, for period-end certification | `2026-07` | Within a period, the same asset reuses its alert. A failure in a new daily, weekly, or monthly period creates a new alert, preserving earlier periods as independent records. This is what the period length buys you: an hourly rule that keeps failing all month produces one `monthly` alert with a rising `occurrenceCount`, not hundreds of separate cases. ### How period boundaries are drawn Boundaries are fixed UTC calendar buckets. They are not configurable, and there is no way to define a period by hand. | | Boundary | | --- | --- | | `daily` | The UTC calendar day, midnight to midnight. Always exactly 24 hours: UTC has no daylight saving, so a period is never 23 or 25 hours long. | | `weekly` | The **ISO 8601** week, which starts on **Monday**, not Sunday. The year in the id is the ISO year, which for a few days around 1 January differs from the calendar year — a run on 1 January 2027 can file under `2026-W53`. | | `monthly` | The Gregorian calendar month, so the length follows the calendar: 28, 29, 30, or 31 days. Nothing assumes a fixed 30. | Three things this does **not** support today: - **A local or per-tenant timezone.** A `daily` period is a UTC day, not your business day. For a team at UTC-5, period `2026-03-15` runs from 19:00 on 14 March to 19:00 on 15 March local time. - **A period starting anywhere other than midnight.** There is no business-day offset. - **A fiscal calendar.** No 4-4-5, no 13-period year, and no fiscal year that starts on a date other than 1 January. `schedule.tz` does **not** move period boundaries. It sets the timezone the cron expression fires in — that is, when the rule *runs*. A rule with `tz: "America/New_York"` runs on New York time and still files its results into UTC periods. To choose the period a result lands in — backfilling a close, or replaying a past date — evaluate on demand and pass `at`. The period is derived from that instant, less the [safety margin](#point-in-time-reads), so `at` selects the bucket. It must be in the past. Scheduled runs derive their own instant and have no equivalent override. A period is operationally green when it has no `OPEN` or `ACKNOWLEDGED` alerts. Resolved and accepted cases remain available for audit. ## Resolutions An alert can close in three ways: | Resolution | When to use it | Recorded context | | --- | --- | --- | | `auto` | A later evaluation passes. | System attribution and timestamp. | | `fixed_by_booking` | Your team made a corrective booking. | Author, optional note, and optional transaction references. | | `accepted_by_business` | The discrepancy is understood and approved without a correction. | Author, required note, and a frozen evidence snapshot. | If the control fails again within the same period, the alert reopens and its current resolution is cleared. The previous resolution remains in the append-only timeline. ## Data discrepancies and execution errors Financial failures create asset alerts. Failures to execute a rule create a separate `engine.error` alert so operational problems do not look like balance discrepancies. Use the alert labels and fingerprint to route these two classes to different teams. Reconciliation compares Formance Ledger balances against external payment-system data grouped into [cash pools](/modules/payments/cash-pools). ## Policies A **policy** identifies the Ledger account set and cash pool to compare. It contains a name, Ledger name, Ledger query, and payments pool ID. Both sets are resolved when the policy runs, so query-based account and pool membership can change over time. ## Reconciliations A **reconciliation** is one synchronous policy run. You provide separate past timestamps for Ledger and Payments to account for processing or settlement delays. The result stores: - `status`: `OK` or `NOT_OK`; - `ledgerBalances` and `paymentsBalances` by asset; - `driftBalances` for any difference; - the timestamps used for both sides. Completed results are immediately available from the reconciliation list and detail endpoints. There is no in-progress state. ## Balance availability Cash-pool balances are historical observations produced by connector ingestion. If you request a timestamp beyond the latest known historical balance window, the point-in-time response can be empty even though a latest balance exists. Choose reconciliation timestamps covered by your connector's ingested balance history. Shorter polling periods reduce the gap between external balance observations. --- ## Getting Started Source: https://docs.formance.com/modules/reconciliation/getting-started This guide creates a control that verifies two groups of Ledger accounts remain in balance, runs it once, and shows how to inspect a discrepancy. - Reconciliation 2.4.0 or later and Ledger 2.4.11 or later - A Ledger containing the accounts you want to check - API access with `reconciliation:read` and `reconciliation:write` - Account metadata or address patterns that identify both sides of the control - The Webhooks module enabled and a [webhook endpoint configured](/modules/webhooks) if alert notifications should be delivered outside Formance All amounts in this guide use Ledger's smallest units. For `USD/2`, `100` represents USD 1.00. ## 1. Define the financial relationship Assume your `main` ledger contains: - asset accounts tagged with `reconciliation.category=held`; - obligation accounts tagged with `reconciliation.category=obligation`. The control should verify, per asset, that the two groups net to zero. Each term has a sign so both account sets can be compared regardless of how their balances are represented. ## 2. Create the rule Create a `ledger_invariant` rule with a strict zero tolerance for USD: The response includes the rule `id`, its enabled state, and an `explanationCEL` field describing the generated financial check. Save the rule ID for the next request. Start on demand while validating account selection and sign conventions. After a few correct runs, update the rule to a cron schedule. ## 3. Evaluate the rule Run the rule at a known historical instant: " }} noFctl body={{ at: "2026-07-20T23:59:59Z", safetyMargin: "0s" }} /> This example uses `"0s"`, so every source is read exactly at `2026-07-20T23:59:59Z`. If `safetyMargin` were omitted, the 30-second default would make the effective read time `2026-07-20T23:59:29Z` instead. In production, the margin avoids the newest edge of the data, where a Ledger write or connector balance update may still be in flight. Use `"0s"` for deterministic tests, demos, or historical runs where the data is already settled and the exact instant matters. The evaluation returns one outcome for every configured asset. A passing outcome contains a compact proof: ```json { "data": { "id": "", "ruleID": "", "result": "PASS", "pitPerSource": { "ledger:main#0": "2026-07-20T23:59:59Z", "ledger:main#1": "2026-07-20T23:59:59Z" }, "evidence": [ { "fingerprint": "asset:USD/2", "passed": true, "proof": { "positive": "250000", "negative": "-250000", "tolerance": "0" } } ] } } ``` If the groups differ, `result` is `FAIL`. The failing entry contains the full balance breakdown and an alert is opened for `asset:USD/2`. ## 4. Inspect the alert List active alerts for the rule using the query builder: "}},{"$match":{"status":"OPEN"}}]}'} noFctl /> The alert identifies the asset, current evidence, first and last observation times, occurrence count, and reconciliation period. Its `lastEvaluationID` links back to the evaluation that most recently changed it. For a complete chronology, list the alert's events: " }} noFctl /> ## 5. Move to a schedule Once the rule selects the right data, update it to run every hour. Its `daily` period length is unaffected; running more often does not create more periods, because the hourly runs are still grouped by UTC day: " }} noFctl body={{ schedule: { kind: "cron", expr: "0 * * * *", tz: "UTC", safetyMargin: "30s" } }} /> For a tick scheduled at `10:00:00`, the 30-second margin reads every source at `09:59:30`. This gives recent Ledger writes and Payments ingestion a short window to settle before comparison. It does not wait 30 seconds before running; it moves the financial observation time 30 seconds into the past. If you omit `safetyMargin`, scheduled evaluations use the 30-second default. To disable the margin, save `safetyMargin: "0s"`; the explicit zero is preserved and every scheduled run reads at its exact occurrence time. For a `daily` rule, alerts are grouped by the UTC day of the effective read timestamp. Use a `continuous` `periodType` for one ongoing operational case, or `weekly` and `monthly` for period-based certification. Use a `continuous` cadence for one ongoing operational case, or `weekly` and `monthly` for period-based certification. ## 6. Pause or resume a rule Rules are enabled by default. Disable a rule when a control is temporarily not applicable or its sources are under maintenance: " }} noFctl body={{ enabled: false }} /> Disabling a rule: - stops scheduled evaluations and cancels pending jobs for the previous rule revision; - rejects on-demand evaluation requests while the rule is disabled; - preserves existing evaluations, alerts, and alert timelines; - leaves active alerts active—they do not become resolved merely because the rule stopped running. Resume the rule with: " }} noFctl body={{ enabled: true }} /> For a cron rule, Reconciliation calculates the next future occurrence when it is re-enabled. It does not create evaluations for the interval during which the rule was disabled. ## 7. Delete a rule Rule deletion is available through the API; no manual PostgreSQL operation is required: " }} noFctl /> A successful deletion returns `204 No Content`. It permanently removes the rule and cascades to all of its evaluations, alerts, and alert-event timelines. Deletion removes the audit history associated with the rule. Disable the rule instead when you need to stop future evaluations but retain prior evidence and resolution records. Export any required records before deleting. ## Next steps - Compare this template with the other [control templates](/modules/reconciliation/controls). - Learn how to [acknowledge, snooze, resolve, and accept alerts](/modules/reconciliation/alerts). - Use a [cash pool](/modules/payments/cash-pools) to reconcile external provider balances. This guide creates a policy, runs a synchronous ledger-to-cash-pool reconciliation, and interprets the result. - A [cash pool](/modules/payments/cash-pools) containing the payment accounts to reconcile - API access with `reconciliation:read` and `reconciliation:write`

1. Create a policy

" }} /> Save the returned policy `id`.

2. Run the policy

" }} body={{ reconciledAtLedger: "2026-07-20T23:59:59Z", reconciledAtPayments: "2026-07-20T23:59:59Z" }} /> Both timestamps must be in the past. They can differ when Ledger and the payment provider reflect the same business event at different times.

3. Interpret the result

- `OK` means the balances matched under the policy's sign convention. - `NOT_OK` means the response contains a non-zero `driftBalances` value or another comparison error. Use `listReconciliations` and `getReconciliation` in the [Stack API reference](/stack-api-reference) to retrieve previous runs. Legacy policies remain supported in later versions. New implementations should use rules and evaluations to gain tolerances, scheduling, alert management, and auditable resolution. --- ## Control Templates Source: https://docs.formance.com/modules/reconciliation/controls Control templates describe the relationship Reconciliation should protect. Choose the template closest to the business question, then configure its sources and per-asset tolerance. | Template | Business question | Sources | | --- | --- | --- | | `ledger_vs_pool_drift` | Does cash held externally back the position recorded in Ledger? | One Ledger account set and one cash pool | | `ledger_invariant` | Do several Ledger account groups maintain the required net relationship? | N Ledger account sets as signed terms (at least one, with no upper limit; meaningful net relationships usually use two or more) | | `account_threshold` | Is a Ledger account group within its allowed balance range? | One Ledger account set | | `source_parity` | Do two independent balance sources agree? | Any two Ledger account sets or cash pools | All four templates evaluate one outcome per asset. If two assets fail, Reconciliation opens two alerts so they can be handled independently. ## Template catalog Use `ledger_vs_pool_drift` for the classic backing check between a dynamic Ledger account set and a [cash pool](/modules/payments/cash-pools). ```json { "templateKind": "ledger_vs_pool_drift", "templateSpec": { "ledger": "main", "ledgerQuery": { "$match": { "metadata[reconciliation.pool]": "stripe" } }, "paymentsPoolID": "", "ledgerSign": -1, "tolerance": { "USD/2": 0, "EUR/2": 50 } } } ``` The control checks this relationship for every asset found on either side: ```text absolute(ledgerSign × ledger balance + pool balance) ≤ tolerance ``` Choose the sign from the way your Ledger represents the external position: - use `-1` when the Ledger and pool balances are both naturally positive; - use `+1`, the default, when the Ledger balance is the negative counterpart of positive external cash. Tolerance defaults to zero for assets not listed. This means a newly observed asset is still checked strictly instead of being silently ignored. When migrating a legacy policy, omit `ledgerSign` to preserve its `ledger + pool = 0` convention. Review the first evaluation's raw and signed Ledger values before scheduling the rule. Use `ledger_invariant` when signed groups of accounts inside Ledger must preserve a financial identity. Typical examples include customer assets versus obligations, safeguarded funds versus customer entitlements, or a control account versus its sub-ledger accounts. ```json { "templateKind": "ledger_invariant", "templateSpec": { "terms": [ { "ledger": "main", "query": { "$match": { "metadata[funds.role]": "held" } }, "sign": 1 }, { "ledger": "main", "query": { "$match": { "metadata[funds.role]": "obligation" } }, "sign": -1 } ], "tolerance": { "USD/2": 0, "EUR/2": 0 } } } ``` Each term selects a Ledger account set and applies a sign of `+1` or `-1`. Reconciliation requires at least one term and does not impose an upper limit, although a meaningful net relationship normally uses two or more: ```text absolute(Σ signᵢ × balanceᵢ) ≤ tolerance ``` A one-term invariant is valid, but it reduces to checking whether that signed balance is within tolerance and therefore overlaps with `account_threshold`. For every asset listed in `tolerance`, Reconciliation checks the signed sum against that tolerance. Assets not listed are outside this rule's scope. Use metadata queries when membership is expected to evolve. New accounts matching a term are included automatically at the next evaluation. Use `account_threshold` to ensure the aggregate balance of an account set remains above a minimum, below a maximum, or inside a range. ```json { "templateKind": "account_threshold", "templateSpec": { "ledger": "main", "query": { "$match": { "metadata[treasury.role]": "operating" } }, "mode": "aggregate", "bounds": { "USD/2": { "min": 100000, "max": 5000000 }, "EUR/2": { "min": 50000 } } } } ``` Each asset needs at least one bound. If both are present, `min` must be less than or equal to `max`. The current release supports `aggregate` mode: all accounts matching the query are summed before the bounds are checked. To watch one account, make the query select only that address. Per-account fan-out is not part of this release. Use `source_parity` when two independently maintained balance sources should agree. Either side can be: - a Ledger account set, identified by a Ledger and query; - a Payments cash pool, identified by its pool ID. ```json { "templateKind": "source_parity", "templateSpec": { "left": { "kind": "ledger", "ledger": "main", "query": { "$match": { "address": "control:stripe" } } }, "right": { "kind": "payments_pool", "poolID": "" }, "scope": "aggregate", "tolerance": { "USD/2": 0, "EUR/2": 50 } } } ``` The template checks `absolute(left - right) ≤ tolerance` for the union of assets observed on both sides. A missing asset is treated as zero, ensuring that a one-sided balance is detected. `aggregate` is the only supported scope in this release. Both Ledger-to-Ledger and pool-to-pool comparisons are supported; each source is still read independently. ## Set tolerances deliberately Tolerances use integer asset units and must be non-negative. They are useful when systems observe the same economic event at slightly different times or when a known rounding convention creates small residuals. Prefer the smallest tolerance justified by the business process: 1. identify the expected timing or rounding difference; 2. express the maximum accepted amount per asset; 3. keep evidence and alerting enabled above that amount; 4. review tolerances when settlement behavior changes. A tolerance prevents a difference from opening an alert. It is not the same as [accepting an alert](/modules/reconciliation/alerts#accept-a-known-discrepancy), which records a human decision after evidence exists. ## Align sources with different timestamps Multi-source templates support an independent point in time for every source. First evaluate the rule once and inspect `pitPerSource` to discover its stable keys, such as: ```json { "ledger:main#0": "2026-07-20T23:59:59Z", "pool:01J...#0": "2026-07-20T23:30:00Z" } ``` You can then replay or align the sources explicitly: ```json { "at": "2026-07-20T23:59:59Z", "sourcePITs": { "ledger:main#0": "2026-07-20T23:59:59Z", "pool:01J...#0": "2026-07-20T23:30:00Z" } } ``` Values in `sourcePITs` are already effective timestamps, so the evaluation's safety margin is not subtracted from them. Unknown source keys and future timestamps are rejected. ## Choose the account sets carefully A control is only as meaningful as its source selection. Before enabling a schedule: - run the underlying Ledger and cash-pool queries independently; - confirm that every intended account is included and no unrelated account matches; - verify sign conventions with a known-good balance; - test each configured asset; - run a known failing scenario and inspect its evidence. Ledger metadata queries at a historical point in time require Ledger 2.4.11 or later, which is the minimum supported by this Reconciliation release. --- ## Webhooks Source: https://docs.formance.com/modules/webhooks Webhooks converts events produced by Formance modules into outbound HTTP `POST` requests. Use it when your application must react to a Ledger transaction, a payment update, a workflow transition, or another Stack event without polling the source API. A webhook is an asynchronous notification, not a remote procedure call. The source operation does not wait for your endpoint, and a successful source operation does not mean that your application has already processed its webhook. ## How Webhooks fits into the Stack ```mermaid flowchart LR Producer["Ledger, Payments,Orchestration, Reconciliation"] -->|"publishes an event"| Broker["Stack event broker"] Broker -->|"matching event type"| Webhooks["Webhooks"] Webhooks -->|"signed HTTP POST"| Endpoint["Your endpoint"] Endpoint -->|"2xx or error"| Webhooks ``` Three terms describe this flow: | Term | Meaning | | --- | --- | | **Event** | An immutable message published by a Formance module, such as `ledger.committed_transactions`. | | **Delivery** | The work required to send one event to one webhook configuration. If three configurations subscribe to the same event, Webhooks creates three independent deliveries. | | **Attempt** | One HTTP request for a delivery. A delivery can have several attempts when the endpoint times out or returns a retryable response. | Webhooks 2.5.0 uses a durable delivery model. It stores matching deliveries before acknowledging the event broker, sends them through a separate dispatcher, records every HTTP attempt, and exposes delivery inspection and replay APIs. This separates event ingestion from endpoint availability: a slow or unavailable endpoint does not block Webhooks from accepting later events into its delivery queue. See [Delivery lifecycle and guarantees](/modules/webhooks/deliveries) for the exact retry policy, state transitions, duplicate and ordering semantics, retention, and replay behavior. ## Create a webhook configuration A configuration binds an endpoint to an explicit list of event types. Event type matching is exact and case-insensitive; Webhooks stores identifiers in lowercase. {"fctl webhooks create \"https://example.com/webhooks/formance\" \"ledger.committed_transactions\" \"payments.saved_payment\""} If you omit `secret`, Webhooks generates a 24-byte secret and returns it base64-encoded. Store it in your secrets manager: your endpoint needs it to verify every delivery. Use an HTTPS endpoint in production. Event subscriptions do not currently support wildcards such as `ledger.*`; list every event type that the endpoint handles. New configurations are active immediately. You can deactivate a configuration to stop new deliveries, reactivate it later, update its endpoint or event types, rotate its signing secret, send a test request, or delete it. ## What your endpoint receives For every attempt, Webhooks sends: - an HTTP `POST` request with `Content-Type: application/json`; - a normalized Formance event envelope and its module-specific `payload`; - a stable delivery identifier and, when provided by the source module, an idempotency key; - an HMAC-SHA256 signature and a fresh attempt timestamp; - a `formance-webhook-test` flag that distinguishes test requests from live events. Webhooks waits up to 30 seconds for the endpoint response. A `2xx` response marks the delivery as successful. Other responses are either retried or marked as permanently failed according to the [delivery policy](/modules/webhooks/deliveries#response-classification). Retry classification and retry windows differ in earlier Webhooks releases. Upgrade to Webhooks 2.5.0 before relying on the bounded retry contract documented for the durable delivery model. Return a `2xx` response only after your application has durably accepted the event. A common pattern is to verify the signature, reject stale requests, write the delivery to an internal queue or inbox table, and then respond. Perform slower business processing asynchronously. ## Build a reliable receiver Your endpoint should implement four controls: 1. Verify the signature against the exact raw request body before parsing JSON. 2. Reject timestamps outside a short tolerance window to limit replay attacks. 3. Deduplicate deliveries by `formance-webhook-id` before applying business effects. 4. Process events without assuming that they arrive in source order. The [Receiving webhooks](/modules/webhooks/receiving) guide documents the headers, signature input, verification code, idempotency strategy, secret rotation, and test deliveries. ## Explore Webhooks Verify requests, prevent duplicate effects, and design a production endpoint. Understand persistence, retries, delivery states, replay, retention, and operational limits. Browse the Stack v3.2.8 event types, envelope, and payload shapes. Configure the broker that carries events between Stack modules. --- ## Alerts and Evidence Source: https://docs.formance.com/modules/reconciliation/alerts An alert is the operational case created when one asset fails a reconciliation rule. It brings together the latest evidence, current status, alert severity, reconciliation period, and a complete transition history. ## Read evidence from a passing check A passing check does not open an alert. Its evidence is stored on the evaluation returned by `POST /rules/{ruleID}/evaluate` and available later from the evaluation endpoints. Each checked asset appears in `data.evidence` with `passed: true` and a compact, self-contained `proof`. It contains the observed balances and the predicate inputs needed to verify the historical result, even if the rule is edited later. The proof shape depends on the template: `ledger_vs_pool_drift` stores the raw Ledger and cash-pool balances with the sign and tolerance used by the check. In this example, the residual is `100 × 1 + (-95) = 5`, which passes a tolerance of `5`. ```json { "fingerprint": "asset:USD/2", "passed": true, "proof": { "ledger": "100", "ledgerSign": "1", "pool": "-95", "tolerance": "5" } } ``` `ledger_invariant` stores the total positive and negative contributions after each term's configured sign is applied, together with the tolerance used to check their residual. ```json { "fingerprint": "asset:USD/2", "passed": true, "proof": { "positive": "350", "negative": "-350", "tolerance": "0" } } ``` `account_threshold` stores the aggregate balance and its configured bounds. An unset `min` or `max` is omitted from the proof. ```json { "fingerprint": "asset:USD/2", "passed": true, "proof": { "balance": "500", "min": "400", "max": "600" } } ``` `source_parity` stores both observed balances and the tolerance used to compare them. In this example, the absolute difference is `30`, which passes a tolerance of `30`. ```json { "fingerprint": "asset:USD/2", "passed": true, "proof": { "left": "100", "right": "130", "tolerance": "30" } } ``` Proof amounts are integer strings in the asset's smallest unit; see [Unambiguous Monetary Notation](/modules/numscript/monetary-notation). Comparisons remain exact for arbitrarily large balances. PASS proofs omit generated expressions and derived fields, while retaining the inputs needed to verify the predicate; use `pitPerSource` for the source timestamps. ## Investigate an alert Start with the alert's current state: ```json { "id": "", "ruleID": "", "fingerprint": "asset:USD/2", "periodID": "2026-07", "status": "OPEN", "severity": "high", "firstSeenAt": "2026-07-20T00:00:02Z", "lastSeenAt": "2026-07-21T00:00:02Z", "occurrenceCount": 2, "lastEvaluationID": "", "evidence": { "asset": "USD/2", "leftBalance": "250000", "rightBalance": "249500", "difference": "500", "tolerance": 0 }, "labels": { "team": "treasury" } } ``` Review four things before acting: 1. **Rule and period:** confirm which control and financial period the case belongs to. 2. **Evidence:** compare the observed values, difference, and tolerance. 3. **Source timestamps:** inspect the linked evaluation's `pitPerSource` to verify both sides were read at the intended instants. 4. **Timeline:** review earlier failures and manual decisions on the same case. Alert lists are ordered by `lastSeenAt` descending, then by `id` descending when timestamps match. This stable order prevents alerts from moving across page boundaries while following a cursor. Retrieve the append-only timeline with: " }} noFctl /> The timeline is cursor-paginated and ordered most recent first. It includes `fail`, `pass`, `ack`, `resolve`, `accept`, `snooze`, and `unsnooze` events. A failure whose previous status was `RESOLVED` is marked as a reopen. If the alert does not exist, the endpoint returns `404` rather than an empty timeline. ### Build a chronological view Fetch every page, following `cursor.next` while `cursor.hasMore` is true. Concatenate the returned `cursor.data` arrays, then reverse the complete list because the API returns newest events first: ```typescript async function loadAlertTimeline( stackUrl: string, alertId: string, token: string, ) { const events = [] let next: string | undefined do { const url = new URL( `/api/reconciliation/alerts/${alertId}/events`, stackUrl, ) url.searchParams.set("pageSize", "100") if (next) url.searchParams.set("cursor", next) const response = await fetch(url, { headers: { Authorization: `Bearer ${token}` }, }) if (!response.ok) throw new Error(`Timeline request failed: ${response.status}`) const body = await response.json() events.push(...body.cursor.data) next = body.cursor.hasMore ? body.cursor.next : undefined } while (next) return events.reverse() } ``` Each entry carries `prevStatus` and `newStatus`, so the timeline can show the state transition without inferring it from neighboring rows. Use `isReopen: true` to highlight a failure that followed a resolution. | Event type | What to show | | --- | --- | | `fail` | The discrepancy evidence in `payload`; follow `evaluationID` for that evaluation's source timestamps. | | `pass` | Automatic resolution; follow `evaluationID` for the passing proof and `pitPerSource`. | | `ack` | The operator, timestamp, and note in `payload`. | | `resolve` | The `fixed_by_booking` resolution, including its author, note, and transaction references. | | `accept` | The `accepted_by_business` resolution and its frozen evidence snapshot. | | `snooze` / `unsnooze` | When notification muting began, changed, or ended. | Do not reconstruct history from the current alert row. Its `ack`, `resolution`, and `snooze` fields describe only the current state. The events endpoint is the source of truth for prior transitions and resolutions, including those cleared when an alert reopened. ## Acknowledge ownership Acknowledge an alert when someone has started investigating it: " }} noFctl body={{ by: "analyst@example.com", note: "Checking provider settlement files" }} /> Acknowledgement changes `OPEN` to `ACKNOWLEDGED`. It does not make the period green and does not stop later evaluations. If another failure arrives, the alert returns to `OPEN` so the prior acknowledgement cannot hide new evidence. ## Snooze notifications Use a snooze for a known, time-bounded situation such as a provider maintenance window or an in-flight migration: " }} noFctl body={{ by: "analyst@example.com", until: "2026-07-21T18:00:00Z", note: "Provider settlement replay in progress" }} /> A snooze mutes webhook notifications until `until`; it does not change the alert status. Evaluations continue, failures remain in the timeline, and the alert continues to count against the period's green status. The first failure at or after the snooze expires clears it and notifies once. You can also lift it early: " }} noFctl body={{ by: "analyst@example.com" }} /> Reconciliation automatically suppresses webhook delivery for repeated failures whose evidence is materially unchanged. The event is still recorded and `occurrenceCount` still increases. Changed evidence, reopens, and manual transitions continue to notify. ## Record a corrective booking After correcting the financial state, close the alert and optionally link the Ledger transactions that made the correction: " }} noFctl body={{ by: "analyst@example.com", note: "Booked the missing provider fee", transactionRefs: [""] }} /> This records a `fixed_by_booking` resolution. A later failing evaluation within the same period reopens the same alert; the earlier resolution remains in the [event timeline](#build-a-chronological-view). An alert also closes automatically when a later evaluation passes. Automatic closure is recorded as an `auto` resolution. ## Accept a known discrepancy Sometimes the observed difference is valid but should remain visible—for example, a confirmed settlement timing difference at period close. Accept it with an explicit business decision: " }} noFctl body={{ by: "controller@example.com", note: "Confirmed settlement lag; cash arrived in the next banking window" }} /> Acceptance requires an author and non-empty note. Reconciliation freezes the alert's current evidence into the `accepted_by_business` resolution so later evidence cannot rewrite what was approved. Acceptance closes the operational case; it does not change Ledger or external balances. Apply your organization's approval policy before treating an accepted alert as reconciled. ## Use webhook events Reconciliation publishes alert transitions through the Formance event bus for delivery by the [Webhooks module](/modules/webhooks). Subscribe only to the transitions your workflow needs: | Event | Meaning | | --- | --- | | `reconciliation.opened_alert` | A new asset and period failed. | | `reconciliation.updated_alert` | An active discrepancy changed materially. | | `reconciliation.acknowledged_alert` | An operator took ownership. | | `reconciliation.resolved_alert` | The control passed or an operator recorded a fix. | | `reconciliation.accepted_alert` | An authorized user accepted the discrepancy. | | `reconciliation.reopened_alert` | A resolved case failed again in the same period. | | `reconciliation.snoozed_alert` | Notifications were muted until a future time. | | `reconciliation.unsnoozed_alert` | A mute was lifted early. | Each payload contains the current alert and the event that caused the transition. Use the alert-event ID as the downstream idempotency key: webhook delivery can be retried and should not be treated as exactly once. ## Separate financial and operational failures If a source read or evaluation cannot complete, Reconciliation stores an `ERROR` evaluation and opens an alert with fingerprint `engine.error` and label `kind=engine.error`. Route this signal to the platform team rather than treating it as a confirmed financial discrepancy. Resolve the source or execution problem, then run the rule again. A successful evaluation closes the operational error path and produces the normal financial verdict. --- ## Receiving Webhooks Source: https://docs.formance.com/modules/webhooks/receiving A production webhook endpoint must authenticate the sender, preserve each delivery until it can be processed, and remain safe when the same delivery arrives more than once. ## Request contract Webhooks sends an HTTP `POST` with a JSON body and the following headers: | Header | Description | | --- | --- | | `formance-webhook-id` | Identifier for one delivery chain. It remains stable across automatic retries and, where replay is available, manual replay. | | `formance-webhook-timestamp` | Unix timestamp generated for this HTTP attempt. Each retry receives a new timestamp. | | `formance-webhook-signature` | One or more versioned signatures. The current format is `v1,`. | | `formance-webhook-idempotency-key` | Source idempotency key, when the module that produced the event supplied one. Do not assume it is always present. | | `formance-webhook-test` | `true` for a request sent by the configuration test endpoint, otherwise `false`. | | `Content-Type` | `application/json`. | Webhooks signs the raw request body, including whitespace and field ordering. See the [event reference](/modules/webhooks/events) for the Stack v3.2 envelope and payload catalog. ## Verify the signature For signature version `v1`, Webhooks computes HMAC-SHA256 over: ```text {formance-webhook-id}.{formance-webhook-timestamp}.{raw-request-body} ``` The result is base64-encoded and sent as `v1,`. The verification helper compares signatures in constant time, but it does not enforce timestamp freshness. Apply your own tolerance after parsing the timestamp. ```go verify-webhook.go package example "fmt" "io" "net/http" "strconv" "time" "github.com/formancehq/webhooks/pkg/security" ) const signatureTolerance = 5 * time.Minute func VerifyWebhook(r *http.Request, secret string) ([]byte, error) { id := r.Header.Get("formance-webhook-id") timestampHeader := r.Header.Get("formance-webhook-timestamp") signatures := r.Header.Get("formance-webhook-signature") if id == "" || timestampHeader == "" || signatures == "" { return nil, fmt.Errorf("missing webhook signature headers") } timestamp, err := strconv.ParseInt(timestampHeader, 10, 64) if err != nil { return nil, fmt.Errorf("invalid webhook timestamp: %w", err) } attemptTime := time.Unix(timestamp, 0) if age := time.Since(attemptTime); age < -signatureTolerance || age > signatureTolerance { return nil, fmt.Errorf("webhook timestamp outside tolerance") } body, err := io.ReadAll(r.Body) if err != nil { return nil, fmt.Errorf("read webhook body: %w", err) } verified, err := security.Verify(signatures, id, timestamp, secret, body) if err != nil { return nil, fmt.Errorf("verify webhook signature: %w", err) } if !verified { return nil, fmt.Errorf("invalid webhook signature") } return body, nil } ``` Do not parse and re-serialize the JSON before verification. Any byte-level change produces a different signature. Keep your server clock synchronized so a legitimate request is not rejected by the timestamp check. ## Acknowledge after durable acceptance Webhooks treats every `2xx` response as success. Return success after the event is stored in a system that survives process failure, such as a database-backed inbox or durable queue. If you respond before storing the event and then crash, Webhooks has no reason to retry it. A typical receiver follows this sequence: ```mermaid sequenceDiagram participant W as Webhooks participant E as Endpoint participant I as Inbox table or queue participant P as Business processor W->>E: Signed POST E->>E: Verify signature and timestamp E->>I: INSERT delivery ID and raw body I-->>E: Committed E-->>W: 204 No Content I->>P: Process asynchronously ``` If validation or durable acceptance fails, return a non-`2xx` status that reflects whether another attempt could succeed. Webhooks retries `408`, `429`, `5xx`, transport errors, and timeouts. Other `4xx` responses are terminal. Earlier releases use a different retry policy. Do not use a `4xx` response to control retries without checking the behavior of your selected Webhooks version. ## Make processing idempotent Webhook delivery can be duplicated at the HTTP boundary. A worker can lose its database connection after your endpoint accepted a request but before Webhooks recorded the successful attempt. Recovery can then send the event again. Create a unique constraint on `formance-webhook-id`, then record the identifier and the business result in one transaction. If the identifier already exists, return the same successful response without applying the effect again. The delivery ID remains stable across automatic retries. Manual replay also preserves the delivery ID. Also make the business operation idempotent from stable domain identifiers in the payload when possible. Use `formance-webhook-idempotency-key` as additional context when present, but do not rely on it exclusively because producers can leave it empty. ## Do not depend on delivery order Webhooks dispatches several deliveries concurrently. A retry can also allow a later event to arrive before an earlier one. If order matters, use the event's resource identifiers and timestamps to reconcile state, or fetch the current resource from its source API before applying a transition. ## Test an endpoint Use the configuration test operation to exercise network access, signature verification, and response handling: Test requests send `{"data":"test"}`, set `formance-webhook-test: true`, and are not retried or stored as durable deliveries. They test network access, signing, and the immediate endpoint response; they do not prove that a specific module event subscription or production payload handler is correct. Validate live event handling in a non-production Stack before enabling business effects. ## Rotate a secret Secret rotation takes effect immediately; Webhooks does not provide a sender-managed dual-secret grace period. Deploy the new secret to your receiver in coordination with the rotation. The dispatcher reads the current configuration before each attempt, so queued attempts use the new secret after rotation. A request that was already in flight can still carry the previous signature. During a controlled transition, let your receiver temporarily accept both secrets, rotate the Webhooks configuration, then remove the previous secret after in-flight requests have completed. --- ## Delivery Lifecycle and Guarantees Source: https://docs.formance.com/modules/webhooks/deliveries Webhooks 2.5.0 uses one durable delivery model that separates broker ingestion from outbound HTTP attempts. This page defines what the pipeline guarantees, where duplicates can occur, how retries are scheduled, and how operators inspect and replay deliveries. The worker does not expose a runtime pipeline selector. Delivery inspection and replay APIs use the same persisted model as automatic dispatch. ## Upgrade from the attempts model New installations start directly with durable deliveries. Existing installations that still have outstanding retries in the former `attempts` table require a coordinated upgrade: 1. Stop every old Webhooks worker and wait for it to terminate. 2. Apply the new database migrations. 3. Run `webhooks backfill-deliveries` until it completes. 4. Deploy the new Webhooks version and recreate its workers. The backfill command is resumable and idempotent. It exists only as an upgrade adapter: the runtime does not read or write the old attempts queue, and running old and new workers together is not supported. Operator-managed installations coordinate this sequence during the upgrade. ## Delivery pipeline ```mermaid flowchart LR Broker["Event broker"] --> Consumer["Webhooks consumer"] Consumer -->|"transactional insert"| Deliveries["deliveries table"] Deliveries -->|"commit succeeds"| Ack["Broker ACK"] Deliveries --> Dispatcher["Concurrent dispatcher"] Dispatcher --> Endpoint["Configured endpoint"] Endpoint --> Attempt["Append-only attempt record"] Attempt --> Deliveries ``` For each broker event, Webhooks selects active configurations that subscribe to its exact event type. In one database transaction, it creates one `pending` delivery per matching configuration. The pair `(event ID, configuration ID)` is unique, so broker redelivery cannot enqueue the same delivery twice. Webhooks acknowledges the broker message only after this transaction commits. The dispatcher then claims due rows with database locks, marks them `delivering`, and performs outbound requests independently of broker consumption. Multiple workers can claim work without intentionally selecting the same row. ## Durable guarantee summary | Property | Contract | | --- | --- | | **Acceptance** | A matching delivery is stored before the broker event is acknowledged. If persistence fails, the broker can redeliver the event. | | **Broker deduplication** | Repeated broker delivery of the same event does not create another delivery for the same configuration. | | **HTTP delivery** | At-least-once attempts until the endpoint returns `2xx` or the delivery reaches a terminal condition. Successful processing by the receiver is not guaranteed. | | **Duplicates** | Possible. A request can reach the endpoint before Webhooks loses the response or fails to commit the attempt result. Receivers must deduplicate. | | **Ordering** | Not guaranteed across events or configurations. The dispatcher is concurrent and retries can overtake earlier deliveries. | | **Timeout** | Each HTTP attempt has a 30-second timeout. | | **Retry budget** | At most 15 attempts and at most 10 hours per retry generation with default settings. The first limit reached terminates the generation. | | **Manual recovery** | Failed deliveries can receive a fresh retry generation through the replay API. Pending deliveries can be expedited. | This is not an exactly-once protocol. Exactly-once business effects require idempotency in the receiving application. ## Durable delivery states | Status | Meaning | | --- | --- | | `pending` | Stored and waiting for its first or next attempt. `nextAttemptAt` indicates when it becomes eligible. | | `delivering` | Claimed by a dispatcher worker and currently in flight. | | `succeeded` | The endpoint returned a `2xx` response. This is terminal. | | `failed` | The endpoint returned a permanent error, or the retry count or elapsed retry window was exhausted. | | `cancelled` | Delivery stopped because its configuration was deactivated or deleted. | Every HTTP call creates an append-only attempt record with the endpoint, attempt number, replay generation, outcome, status code, duration, response excerpt, error, and timestamp. If a worker stops while a delivery is `delivering`, claims older than five minutes are recovered: active configurations return to `pending`; inactive or deleted configurations become `cancelled`. ## Response classification Webhooks classifies the final response from the endpoint as follows: | Result | Delivery action | | --- | --- | | `2xx` | Mark `succeeded`; no more automatic attempts. | | `408 Request Timeout` | Retry. | | `429 Too Many Requests` | Retry and honor a valid `Retry-After` when it requests a longer delay. | | Other `4xx` | Mark `failed` immediately. These errors normally require a configuration or application change. | | `5xx` | Retry. A valid `Retry-After` can extend the delay. | | Network error or 30-second timeout | Retry. The attempt records status code `0`. | The response body does not affect classification. Webhooks reads at most 64 KiB of it for diagnostics. Return `429` with `Retry-After` when your service is healthy but temporarily rate-limited. Return another `4xx` only when retrying the same request cannot succeed without a change, because Webhooks treats it as terminal. ## Retry schedule The default policy uses exponential backoff without jitter. The first request is immediate, then delays double from one minute until they reach the one-hour cap: | Attempt | Approximate time from first attempt | | --- | --- | | 1 | Immediately | | 2 | 1 minute | | 3 | 3 minutes | | 4 | 7 minutes | | 5 | 15 minutes | | 6 | 31 minutes | | 7 | 1 hour 3 minutes | | 8 | 2 hours 3 minutes | | 9–15 | Once per hour, with attempt 15 at approximately 9 hours 3 minutes | Automatic retries stop when either limit is reached: - 15 total attempts in the current generation; - 10 elapsed hours from the first attempt in the current generation. For retryable responses, Webhooks uses `Retry-After` when it is valid and longer than the computed backoff. It accepts delay-seconds or an HTTP date, caps an endpoint-supplied delay at six hours, and never allows it to extend the 10-hour retry window. Webhooks does not open a per-endpoint circuit breaker or automatically disable a configuration after repeated failures. Attempt and time caps bound each delivery; operators must monitor terminal failures and queue growth. ## Configuration changes during delivery Deactivating or deleting a configuration cancels its pending deliveries. An in-flight attempt can finish, but any non-successful result becomes `cancelled` instead of returning to the queue. Reactivating a configuration does not resurrect cancelled deliveries; replay also rejects cancelled deliveries. Updating an endpoint affects later attempts because the dispatcher reads the current configuration before sending. Rotating the secret similarly causes later attempts to use the new secret. ## Inspect deliveries List deliveries by configuration, status, or creation window. The list excludes payloads and uses an opaque cursor; request one delivery to retrieve its payload. ```sh curl --get "$STACK_URL/api/webhooks/deliveries" \ --header "Authorization: Bearer $ACCESS_TOKEN" \ --data-urlencode "configId=$CONFIG_ID" \ --data-urlencode "status=failed" \ --data-urlencode "createdAtFrom=2026-08-01T00:00:00Z" \ --data-urlencode "pageSize=100" ``` Valid statuses are `pending`, `delivering`, `succeeded`, `failed`, and `cancelled`. `pageSize` defaults to 100 and cannot exceed 1,000. Retrieve a delivery and its attempt history: ```sh curl "$STACK_URL/api/webhooks/deliveries/$DELIVERY_ID" \ --header "Authorization: Bearer $ACCESS_TOKEN" curl "$STACK_URL/api/webhooks/deliveries/$DELIVERY_ID/attempts?pageSize=100" \ --header "Authorization: Bearer $ACCESS_TOKEN" ``` Use `lastStatusCode`, `lastError`, `nextAttemptAt`, and the attempts list to distinguish an endpoint rejection from a timeout, retry backlog, or exhausted retry budget. ## Replay one delivery Only `failed` and `pending` deliveries belonging to an active configuration are eligible for replay. A replay request requires an `Idempotency-Key` header. ```sh curl --request POST "$STACK_URL/api/webhooks/deliveries/$DELIVERY_ID/replay" \ --header "Authorization: Bearer $ACCESS_TOKEN" \ --header "Idempotency-Key: replay-$DELIVERY_ID-20260805" ``` For a `failed` delivery, replay increments `replayGeneration`, resets its attempt count and 10-hour retry window, and queues it immediately. For a `pending` delivery, replay only moves the next attempt to now; it does not reset the current budget. `succeeded`, `delivering`, and `cancelled` deliveries are not eligible for replay. The same idempotency key and request return the original replay result for 24 hours. Reusing the key for a different replay returns a conflict. ## Replay a bounded set Bulk replay operates synchronously on one page of at most 1,000 deliveries. A request must include `createdAtFrom`; its creation-time window must be positive and cannot exceed 90 days. ```sh curl --request POST "$STACK_URL/api/webhooks/deliveries/replay" \ --header "Authorization: Bearer $ACCESS_TOKEN" \ --header "Content-Type: application/json" \ --header "Idempotency-Key: incident-20260805-page-1" \ --data '{ "createdAtFrom": "2026-08-05T08:00:00Z", "createdAtTo": "2026-08-05T10:00:00Z", "statuses": ["failed", "pending"], "configIds": ["'"$CONFIG_ID"'"], "pageSize": 1000 }' ``` The response separates `replayed` failed deliveries, `expedited` pending deliveries, and `skipped` rows. When `hasMore` is true, submit another request with `nextCursor` and a new `Idempotency-Key`. Keep the original filters unchanged; the cursor binds to the time window, statuses, and configuration IDs. ## Retention and observability With default settings, Webhooks retains succeeded deliveries for 30 days and failed or cancelled deliveries for 90 days. Attempt history is deleted with its delivery. Retention runs hourly. These values are runtime settings for self-hosted deployments and can be changed or disabled. Webhooks exports OpenTelemetry traces and the following delivery metrics when metrics export is configured: | Metric | Purpose | | --- | --- | | `webhooks_delivery_attempts_total` | Attempts by outcome and HTTP status class. | | `webhooks_delivery_duration_seconds` | Outbound request duration. | | `webhooks_retry_queue_depth` | Pending delivery count, capped at 1,000,000. | | `webhooks_replayed_deliveries_total` | Deliveries replayed or expedited manually. | | `webhooks_delivery_transitions_total` | Durable delivery state transitions. | | `webhooks_delivery_claims_recovered_total` | Stale in-flight claims recovered after worker interruption. | Alert on a growing retry queue, sustained `5xx` or timeout rates, terminal failures, and old `pending` deliveries. A delivery can be safely considered complete only when it is `succeeded`, or when your operational process has accepted its terminal failure. --- ## Event Reference Source: https://docs.formance.com/modules/webhooks/events This reference lists the webhook events and JSON payload formats published with [Formance Stack v3.2.8](https://github.com/formancehq/stack/releases/download/v3.2.8/all.json). It uses Ledger events v2.0.0, Orchestration events v2.0.0, Payments events v3.0.0, and Reconciliation events v2.4.0 from that release artifact. ## Event envelope Every delivery uses the same top-level envelope. Webhooks builds the subscription identifier by lowercasing the producer message's `app` and `type` fields and joining them with a dot. It also writes that normalized identifier back to the outgoing `type` field. For example, `app: ledger` and producer type `COMMITTED_TRANSACTIONS` become `type: ledger.committed_transactions` in the webhook body. The `payload` object changes with the event type. This example shows a complete Ledger committed-transactions message: ```json { "idempotency_key": "order-2026-0042", "app": "ledger", "version": "v2", "date": "2026-08-05T10:30:00Z", "type": "ledger.committed_transactions", "payload": { "ledger": "main", "transactions": [ { "postings": [ { "source": "world", "destination": "users:001", "amount": 1000, "asset": "USD/2" } ], "metadata": { "order": "ORD-2026-0042" }, "id": 42, "timestamp": "2026-08-05T10:30:00Z", "reverted": false } ] } } ``` | Field | Type | Required | Description | | --- | --- | --- | --- | | `idempotency_key` | string | Yes | Idempotency key supplied by the producing module. The field is present but can be an empty string. | | `version` | string | Yes | Version of the module event contract. | | `date` | string (`date-time`) | Yes | Time at which the event was published. | | `app` | string | Yes | Module that published the event. | | `type` | string | Yes | Normalized lowercase `.` identifier shown below. | | `payload` | object | Yes | Event-specific data. | The Stack release artifact describes producer envelopes and event payloads. Webhooks normalizes the outgoing envelope as shown above, while the nested `payload` remains governed by the event's JSON Schema. The tables below summarize its top-level shape. Use the linked v3.2.8 artifact for required payload fields, nested objects, array items, formats, and enum values when generating validators or types. ## Ledger events | Webhook event type | Payload shape | | --- | --- | | `ledger.committed_transactions` | `ledger`: string; `transactions`: array of objects | | `ledger.deleted_metadata` | `ledger`: string; `targetType`: string; `targetId`: string; `key`: string | | `ledger.reverted_transaction` | `ledger`: string; `revertedTransaction`: object; `revertTransaction`: object | | `ledger.saved_metadata` | `ledger`: string; `targetType`: string; `targetId`: string; `metadata`: object | ## Orchestration events | Webhook event type | Payload shape | | --- | --- | | `orchestration.failed_trigger` | `id`: string; `triggerID`: string; `error`: string | | `orchestration.failed_workflow` | `id`: string; `instanceID`: string; `error`: string | | `orchestration.failed_workflow_stage` | `id`: string; `instanceID`: string; `number`: integer; `error`: string | | `orchestration.started_workflow` | `id`: string; `instanceID`: string | | `orchestration.started_workflow_stage` | `id`: string; `instanceID`: string; `number`: integer | | `orchestration.succeeded_trigger` | `id`: string; `triggerID`: string | | `orchestration.succeeded_workflow` | `id`: string; `instanceID`: string | | `orchestration.succeeded_workflow_stage` | `id`: string; `instanceID`: string; `number`: integer | ## Payments events | Webhook event type | Payload shape | | --- | --- | | `payments.connector_reset` | `createdAt`: string; `connectorID`: string | | `payments.deleted_pool` | `id`: string; `createdAt`: string | | `payments.saved_account` | `id`: string; `provider`: string; `connectorID`: string; `createdAt`: string; `reference`: string; `type`: string; `rawData`: object; `defaultAsset`: string; `name`: string; `metadata`: object | | `payments.saved_balance` | `accountID`: string; `connectorID`: string; `provider`: string; `createdAt`: string; `lastUpdatedAt`: string; `asset`: string; `balance`: number | | `payments.saved_bank_account` | `id`: string; `createdAt`: string; `name`: string; `accountNumber`: string; `iban`: string; `swiftBicCode`: string; `country`: string; `metadata`: object; `relatedAccounts`: array | | `payments.saved_payment` | `id`: string; `connectorID`: string; `provider`: string; `reference`: string; `createdAt`: string; `type`: string; `status`: string; `scheme`: string; `asset`: string; `amount`: number; `initialAmount`: number; account IDs, links, metadata, and raw data | | `payments.saved_payment_initiation` | `id`: string; `connectorID`: string; `provider`: string; `reference`: string; `createdAt`: string; `scheduledAt`: string; `description`: string; `type`: string; `amount`: number; `asset`: string; account IDs and metadata | | `payments.saved_payment_initiation_adjustment` | `id`: string; `paymentInitiationID`: string; `status`: string; `amount`: number; `asset`: string; `error`: string; `metadata`: object | | `payments.saved_payment_initiation_related_payment` | `paymentInitiationID`: string; `paymentID`: string | | `payments.saved_pool` | `id`: string; `name`: string; `createdAt`: string; `accountIDs`: array of strings | ## Reconciliation events | Webhook event type | Payload shape | | --- | --- | | `reconciliation.accepted_alert` | `alert`: object; `event`: object | | `reconciliation.acknowledged_alert` | `alert`: object; `event`: object | | `reconciliation.opened_alert` | `alert`: object; `event`: object | | `reconciliation.reopened_alert` | `alert`: object; `event`: object | | `reconciliation.resolved_alert` | `alert`: object; `event`: object | | `reconciliation.snoozed_alert` | `alert`: object; `event`: object | | `reconciliation.unsnoozed_alert` | `alert`: object; `event`: object | | `reconciliation.updated_alert` | `alert`: object; `event`: object | The `alert` object contains the alert state and evidence. The `event` object records the transition, including its previous and new status, actor metadata, notification decision, and timestamps. Refer to the release artifact for the complete nested schemas and enum values. --- ## Architecture Source: https://docs.formance.com/manage/architecture The Formance Platform embraces a cloud native architecture, enabling seamless deployments in various environments such as Formance Cloud, on-prem, or hybrid approaches. Formance Cloud offers minimal effort deployments with sane defaults and rolled-in support. Alternatively, on-prem and hybrid options provide a great level of flexibility — which has proven to be beneficial for deployments existing within the infrastructure of regulated financial institutions and platforms. ## Planes The Formance Platform is composed of two main planes: the control plane, and the data plane. The control plane is responsible for managing the platform and provisioning its components, while the data plane is responsible for delivering the actual core functions of the platform. **Trivia:** One non-intuitive thing to note is: _data_ in the control plane / data plane terminology refers not only to data storage, but to the storage and compute of services providing core functionality of the platform. ## Control plane The control plane server is responsible for managing the platform and its components, while the control plane client is responsible for provisioning the data plane according to the desired state, and for facilitating communication between the data plane and the control plane server. ### Components ## Data plane The data plane is composed of a number of components that work together to provide the core functionality of the platform. These components are either considered core services that deliver a specific piece of functionality to the platform (identified below as group "product"), or support services that provide horizontal support to the platform (identified below as group "system"). ### Components --- ## Single sign-on (SSO) for organizations Source: https://docs.formance.com/manage/identity/sso You can enable SSO on a per-organization basis using your existing identity provider. Supported providers are OIDC, Microsoft Entra ID (formerly Azure AD), Google, and GitHub. ## Prerequisites - A Formance Cloud organization and access to `fctl` - A client application created in your identity provider (Client ID and Client Secret) When creating the app in your IdP, set the redirect URI to your membership endpoint callback: `/api/authorize/callback`. See step 1 to determine your membership URI. ## Determine your membership URI Use the same membership URI you pass to `fctl login`. ```bash fctl login --membership-uri https://membership.BASE_URL/api ``` The SSO redirect URI to register in your IdP is therefore: ```text https://membership.BASE_URL/api/authorize/callback ``` After you create the SSO configuration, the exact redirect URI is also displayed in the UI. ## Configure the authentication provider The command format is: ```text fctl cloud organizations authentication-provider configure \ [--oidc-issuer ] [--microsoft-tenant ] ``` - **type**: one of `oidc`, `microsoft`, `github`, `google` - **name**: human-friendly provider name shown to users - **client-id / client-secret**: values from your IdP app - **--oidc-issuer**: required for `oidc` (e.g., `https://accounts.example.com`) - **--microsoft-tenant**: required for `microsoft` (tenant ID or verified domain) ```bash fctl cloud organizations authentication-provider configure \ oidc "My OIDC" "" "" \ --oidc-issuer https://accounts.example.com ``` ```bash fctl cloud organizations authentication-provider configure \ microsoft "My Entra" "" "" \ --microsoft-tenant "" ``` ```bash fctl cloud organizations authentication-provider configure \ google "Google" "" "" ``` ```bash fctl cloud organizations authentication-provider configure \ github "GitHub" "" "" ``` ### Microsoft Entra ID To configure Microsoft Entra ID as your identity provider, use the `oidc` type with the Microsoft issuer URL. The `--oidc-issuer` must follow this format: ```text https://login.microsoftonline.com//v2.0 ``` Replace `` with your Entra tenant ID. ```bash fctl cloud organizations authentication-provider configure \ oidc "msentra" "" "" \ --oidc-issuer "https://login.microsoftonline.com//v2.0" ``` Users must have an email address configured on their Entra identity to sign in via SSO. Ensure the redirect URI in your IdP exactly matches `/api/authorize/callback`. Mismatches (scheme, host, path, or trailing slash) will cause sign-in failures. ## Verify SSO Use the email domain associated with your IdP, if auto-login by domain is enabled. From the Formance Cloud portal, choose the newly configured provider and complete the sign-in flow. You should land back in the portal authenticated to your organization. If not, confirm the redirect URI and client credentials in your IdP and re-run the configure command if needed. ## Reference ```bash fctl cloud organizations authentication-provider configure -h ``` ```text Configure the authorization provider for the organization Usage: fctl cloud organizations authentication-provider configure [flags] Flags: -h, --help help for configure --microsoft-tenant string Microsoft tenant ID (used when type is 'microsoft') (default "tenant") --oidc-issuer string OIDC issuer URL (used when type is 'oidc') Global Flags: -c, --config string Path to configuration file -d, --debug Enable debug mode --insecure-tls Allow insecure TLS connections --organization string Selected organization (not required if only one organization is present) -o, --output string Output format (plain, json) -p, --profile string Configuration profile to use --stack string Specific stack (not required if only one stack is present) --telemetry Enable telemetry ``` --- ## Access Control Source: https://docs.formance.com/manage/identity/rbac --- ## Invite users Source: https://docs.formance.com/manage/identity/inviting-team --- ## Audit Logs Source: https://docs.formance.com/manage/identity/audit-logs The Formance Platform ships with an Audit Log feature that streams every administrative action and every API request against your stacks to a destination of your choice — typically a SIEM such as Splunk, Datadog, or Elastic, or any HTTPS endpoint that accepts JSON POSTs. This page describes what's captured, how forwarding works, and what the events look like on the wire. To enable Audit Log forwarding, contact your account team. ## What's captured Two independent streams are available, and you can subscribe to either or both. ### Lifecycle events Administrative activity at the organization and stack level: invitations, user and permission changes, stack create / update / delete / upgrade, region changes, module enable / disable, and similar. Volume is proportional to administrative activity, not to your API traffic, so it tends to be light. The following lifecycle event types are emitted: ### Module HTTP audit Every non-streaming API call against your module services (ledger, payments, wallets, reconciliation, orchestration, and so on). Each event captures: Volume scales one-to-one with your API call rate. ## How forwarding works The Formance Platform pushes events to an HTTPS endpoint you provide. There is no queue for you to subscribe to and no agent to deploy — events arrive at your endpoint as POST requests as they happen, with the headers you've configured. To set up forwarding, share with your account team: - The HTTPS URL where events should be delivered. - Any authentication headers to include on the POST. For Splunk HEC this is typically `Authorization: Splunk `. - Which stream(s) you want — lifecycle, module, or both. - The stacks the integration should cover, if you have multiple. - Whether you want events batched. By default each event is delivered as its own POST; we can batch into bulk POSTs on request. Provisioning is operated by Formance. ## Event format Both streams share a common envelope: ```json { "date": "", "app": "gateway | membership", "version": "v2", "type": "", "payload": { ... } } ``` The shape of `payload` depends on the stream and event type. ### Module audit event ```json { "date": "2026-04-30T09:23:41.123Z", "app": "gateway", "version": "v2", "type": "AUDIT", "payload": { "id": "550e8400-e29b-41d4-a716-446655440000", "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736", "actor": { "claims": { "sub": "...", "email": "...", "iss": "..." }, "organization_id": "...", "stack_id": "...", "ip_address": "..." }, "http": { "request": { "method": "POST", "path": "/api/ledger/v2/main/transactions", "host": "...", "header": { "Content-Type": ["application/json"] }, "body": "..." }, "response": { "status_code": 200, "headers": { "Content-Type": ["application/json"] }, "body": "..." } } } } ``` When the request's JWT can't be validated, `actor.claims` is omitted and `actor.token_validation_error` carries the reason instead. ### Lifecycle event Example — a stack creation: ```json { "date": "2026-04-30T09:15:00Z", "app": "membership", "version": "v2", "type": "stacks.created", "payload": { "ownerId": "...", "organizationId": "...", "subject": "...", "data": { "id": "...", "name": "prod-stack", "regionId": "..." } } } ``` The structure of `payload.data` varies by event type. The full list of lifecycle payload shapes is available on request from your account team. ## What's stripped before delivery Module audit events are intentionally complete. Auditors and security teams need to be able to reconstruct exactly what happened, so headers and bodies are captured verbatim — with three exceptions, applied at the source: - The `Authorization` request header is removed. - The response body for `/api/auth/oauth/token` is cleared. - Request and response bodies for streaming endpoints (Content-Type `application/vnd.formance*-stream` or `application/octet-stream`) are not captured. Everything else is preserved as it appeared on the wire. Module audit events will contain whatever your API calls contain. For ledger that means transaction amounts, account references, and any metadata you've attached. For payments, counterparty details and amounts. Once events are delivered to your SIEM, they live under your access controls, retention rules, and downstream pipelines — treat the audit stream as an extension of the data plane it observes, and apply the same data governance you'd apply to any system handling financial or personal information. ## Delivery semantics Events are delivered at least once. If your endpoint returns a non-2xx response or is unreachable, the Formance Platform retries. During extended outages on your side, events may be queued internally up to a bounded retention window; sustained outages can cause queued events to be lost. Treat the destination receiver as durable storage rather than relying on the Formance Platform as a long-term buffer. There is no API to replay historical events from before forwarding was set up. If you need historical data, request it during onboarding. ## Inspection from the command line For ad-hoc inspection of lifecycle events, `fctl` provides direct access: ```sh fctl cloud org history "" fctl stack history "" ``` This is intended for occasional review rather than continuous ingestion. For SIEM integration, use forwarding as described above. ## FAQ **Which destinations are supported?** Any HTTPS endpoint that accepts JSON POSTs. Splunk HEC, Datadog, Elastic, and custom endpoints are all in production with existing customers. **Can we filter events at the source?** Not today — the configured stream is forwarded in full. Filter at your destination. **Are the two streams delivered on the same connection?** They share an envelope but come from independent forwarders. Distinguish on the `app` field (`gateway` for module audit, `membership` for lifecycle) and on `type`. **Is the schema stable?** Yes. The current schema version is `v2`. Changes will be coordinated with your account team. **What's the latency from event to delivery?** Module audit events are forwarded as the request completes. Lifecycle events are forwarded as they occur. End-to-end latency to your endpoint is typically sub-second under normal conditions. --- ## Event Streaming Source: https://docs.formance.com/manage/events/streaming The Formance Platform can stream all platform events to a message broker, giving you a real-time feed of everything happening across your stack. This enables you to build reactive integrations, power audit pipelines, or feed events into your own data infrastructure. ## Overview When a broker is configured, every service in the stack publishes events to it. Events are organized into topics in the format: ``` {stackName}-{module} ``` The following services produce events: | Role | Services | |------|----------| | Producers | Ledger, Gateway, Payments | | Consumers | Orchestration, Webhooks | Webhooks are built on top of the event stream — they subscribe to broker topics and forward matching events to your configured endpoints. See [Webhooks](/modules/webhooks) for more details. ## Supported brokers The platform supports two message brokers: | Broker | Requirements | |--------|-------------| | [NATS](https://nats.io/) | Version 2.6+ with Jetstream enabled | | [Kafka](https://kafka.apache.org/) | Standard Kafka cluster | ## Configuration The broker is configured at the stack level via the `broker.dsn` setting. ```yaml apiVersion: formance.com/v1beta1 kind: Settings metadata: name: formance-dev-broker spec: key: broker.dsn stacks: - "formance-dev" value: nats://nats.formance-system.svc:4222?replicas=3 ``` ```yaml apiVersion: formance.com/v1beta1 kind: Settings metadata: name: formance-dev-broker spec: key: broker.dsn stacks: - "formance-dev" value: kafka://kafka.formance-system.svc:9092 ``` For detailed broker setup instructions, see [Message Broker infrastructure](/deploy/self-hosted/infrastructure/message-broker). --- ## SDKs Source: https://docs.formance.com/manage/sdks The Formance Platform comes with a set of client libraries that you can use to connect your app, while ensuring you are always using the API in a consistent and up-to-date manner. ## Supported platforms --- ## Release Policy Source: https://docs.formance.com/release-policy This page describes the release policy for the Formance Platform and its components. This policy is subject to change; the latest version is the one available on this website. The Formance Platform is released as a Stack: a coherent, reproducible, and supported set of components. ## Stack releases A Stack release is a versioned compatibility line backed by an exact manifest of component versions. The Stack version exposed at runtime identifies the compatibility line, for example `v3.2`. The exact composition of the Stack is pinned in the Helm Chart and versioned through chart revisions. A deployment never resolves component versions dynamically: the installed component versions are the versions pinned by the applied chart. The Stack version keeps three segments, but it does not follow strict SemVer: | Segment | Meaning | | --- | --- | | Major | Breaks the Stack compatibility contract | | Minor | Introduces a new Stack compatibility line | | Patch | Publishes a new compatible chart and manifest revision in the same Stack line | For example, `v3.2.x` represents the `v3.2` compatibility line. A new chart revision for that line can update one or more compatible components while the runtime Stack line remains `v3.2`. ## Compatibility contract All components embedded in the same Stack line must be compatible with: - the Ledger version embedded by that line; - the other embedded components; - the public APIs exposed by the Stack line; - the migrations and schemas supported by the Stack line; - the operational constraints of the target deployment. A component version can only be integrated into an existing Stack line if it respects that contract. If it requires an incompatible API change, migration, Ledger dependency, or operational requirement, it must be released through a new Stack compatibility line or documented as an explicit exception. ## Component release regimes Components keep their own SemVer versioning. Within a component major version, releases are expected to remain compatible with that component's public contract. The Stack does not mechanically propagate component minor or patch versions into the Stack version. For components other than Ledger, compatible minor and patch bumps can be integrated into an existing Stack line as Stack patch revisions. A new Stack compatibility line is required when one of those components moves to a new major version, or otherwise introduces a change that breaks the Stack compatibility contract. Whether that new Stack line is a Stack minor or a Stack major depends on the impact on the Stack contract: a Stack minor introduces a new compatibility line, while a Stack major breaks compatibility for existing users. ### Ledger Ledger is pinned for each supported Stack line. Ledger holds client data and carries high-impact migrations, so each Stack line pins an exact Ledger version. Fixes can be backported to the supported Ledger line, then integrated by publishing a new compatible chart revision. An incompatible Ledger migration requires a new Stack compatibility line. ### Rolling within major components The following components are released as rolling within major components: - `transaction-plane` - `payments` - `wallets` - `flows` - `reconciliation` - `webhooks` - `auth` - `gateway` These components keep their own SemVer, but compatible features and fixes can be integrated into an existing Stack line as long as they remain compatible with the component major version and the Stack compatibility contract. A compatible bump of one of these components produces a Stack patch revision. This means a Stack patch can include a new feature from a rolling component, provided the feature does not break the Stack line. ## Helm Chart and Versions CRD The Helm Chart is the technical source of truth for the exact composition of a Stack release. For a published chart revision, the manifest is immutable. Any component version change requires a new chart revision and a change visible in Git. The `Versions` CRD (`versions.formance.com`) exposes the installed Stack compatibility line and the component versions embedded by the applied chart. It must make support and diagnostics able to map a Stack line to concrete component versions, for example: ```yaml stack: v3.2 modules: ledger: 2.3.4 transaction-plane: 1.4.2 payments: 1.12.3 wallets: 1.7.0 flows: 1.9.1 reconciliation: 1.3.5 webhooks: 1.5.0 auth: 1.8.2 gateway: 1.6.4 ``` ## Changelog The Stack changelog is part of the release contract. Each Stack release must list the component deltas since the previous chart revision in the same line, including: - features; - fixes; - security fixes; - migrations; - configuration changes; - known risks or required operator actions. A release is not considered complete if this information is not available. ## Security and support Security fixes are integrated by publishing new chart revisions for the relevant supported Stack lines. When a fix is compatible with an existing line, the Stack compatibility line remains the same and only the chart revision changes. The Formance Platform supports the last two Stack compatibility lines, commonly referred to as `N` and `N-1`. These lines continue to receive compatible fixes, especially security fixes. If a security fix cannot be applied compatibly to a supported line, Formance may provide a compatible backport, require an upgrade to a more recent Stack line, or document a support exception. --- ## MCP Source: https://docs.formance.com/mcp The Formance MCP server exposes selected Formance capabilities to MCP-compatible clients such as coding assistants and local agent runtimes. It lets an agent inspect financial data and validate Numscript without giving it write access to the stack. MCP support requires Stack v3.2 or later and `fctl` v3.4.0 or later. ## Enable MCP on a stack Check your local `fctl` version before enabling MCP: ```bash fctl version ``` Enable the MCP module on the target stack with `fctl`: ```bash fctl stack module enable mcp ``` Run this command from an authenticated `fctl` context that targets the organization and stack where MCP should be enabled. ## Run MCP over stdio MCP clients typically start the server as a local process and communicate with it over standard input and output. With Formance, `fctl` can serve as the MCP process: ```bash fctl stack mcp serve --transport=stdio --organization=YOUR_ORGANIZATION_ID --stack=YOUR_STACK_ID ``` `--organization` and `--stack` are only optional when your `fctl` context has exactly one organization and one stack. Otherwise `fctl` exits with `organization not specified` (or `stack not specified`), so pass both explicitly — as above, or as global flags before the subcommand (`fctl --organization=… --stack=… stack mcp serve --transport=stdio`). Your MCP client configuration should pass the target organization and stack to `fctl`. The exact configuration format depends on the client: ```json { "mcpServers": { "formance": { "command": "fctl", "args": [ "--organization=YOUR_ORGANIZATION_ID", "--stack=YOUR_STACK_ID", "stack", "mcp", "serve", "--transport=stdio" ] } } } ``` ```toml [mcp_servers.formance] command = "fctl" args = [ "--organization=YOUR_ORGANIZATION_ID", "--stack=YOUR_STACK_ID", "stack", "mcp", "serve", "--transport=stdio" ] ``` Replace the `--organization` and `--stack` values with your own environment. ## Verify the server responds A stdio MCP server has no interactive output: run it on its own and it appears to hang while it waits for JSON-RPC on standard input. That is expected — it is not a failure. To confirm the server is healthy without an MCP client, pipe an `initialize` request into it: ```bash echo '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"test","version":"1.0"}}}' \ | fctl stack mcp serve --transport=stdio --organization=YOUR_ORGANIZATION_ID --stack=YOUR_STACK_ID ``` A healthy server replies with a single JSON line advertising its capabilities: ```json {"jsonrpc":"2.0","id":1,"result":{"capabilities":{"logging":{},"tools":{"listChanged":true}},"protocolVersion":"2024-11-05","serverInfo":{"name":"stack-mcp","version":"v0.1.0"}}} ``` If you see an `organization not specified` or `stack not specified` error instead, add the flags shown above. If the command hangs with no reply, check that the `mcp` module is enabled on the stack and that your `fctl` context is authenticated. ## Available capabilities The MCP server is read-only for Formance data. It is intended for exploration, diagnostics, and assisted development workflows. | Area | Capabilities | |------|--------------| | Ledger | Read ledgers, accounts, balances, volumes, transactions, and related metadata. | | Payments | Read payments data and inspect payment-related resources. | | Reconciliation | Read reconciliation data to investigate matching and reconciliation state. | | Numscript | Validate Numscript before running it against a ledger workflow. | MCP tools can expose sensitive financial data to the client that starts them. Only configure the Formance MCP server in trusted clients and environments. ## Typical workflow 1. Enable the `mcp` module on a Stack v3.2 or later stack. 2. Configure your MCP client to start `fctl stack mcp serve --transport=stdio`. 3. Ask the client to inspect Ledger, Payments, or Reconciliation state. 4. Use Numscript validation before promoting a script into an operational workflow. --- ## Membership API Source: https://docs.formance.com/membership-api The Membership API manages your Formance Cloud resources: organizations, stacks, users, invitations, regions, and access policies. --- ## Stack API Reference Source: https://docs.formance.com/stack-api-reference The Formance Platform exposes a unified REST API across all modules. All endpoints use JSON for request and response bodies, and authentication is handled via OAuth2 bearer tokens. ```bash ``` This reference is generated from a Stack release's published specification, and no v4 release has published one. Rather than serve you the **v3.2** specification — 135 endpoints for modules a v4 stack does not compose, and Ledger at `v2` when v4 pins `3.0.0` — the endpoint list is withheld at this version. For the modules a v4 stack does compose: - **Ledger 3.0** — the [Ledger documentation](/modules/ledger/get-started) covers the `/v3/` surface, and [HTTP API](/modules/ledger/reference/http-api) locates the contract for the release you run. - **Auth, Gateway and Search** — v4 composes these unchanged, and they are documented in the latest Stack API reference. That link deliberately leaves the v4 preview; an ordinary link could not, because version context persists across unversioned pages off LATEST. The endpoint list returns here once a v4 stack release publishes its specification. ---