# Architecture & protocol reference

The reference Claude and developers consult when touching indexer handlers or
explaining data. Protocol facts below were derived from the Midnight contract
sources and validated against live Base logs when this repo was built.

## Data flow

```
chain (Base 8453, mainnet soon)
  │  eth_getLogs / realtime blocks (reorg-aware)
  ▼
Ponder indexer  apps/indexer — one process
  │  handlers write tables (ponder.schema.ts) into a per-deploy Postgres schema
  │  stable views schema `midnight` flips atomically when a deploy finishes backfilling
  ▼
Hono API (same process)  /sql (typed SQL + SSE live queries) · /health · /ready
  │  browser holds ONE SSE connection; server pings per indexed block;
  │  invalidated queries refetch automatically
  ▼
Next.js web  apps/web — useExplorerQuery + LiveBridge re-render components live
  │
  └─ Next.js SERVER (same app, src/lib/server) reads /sql too, for machines:
       Markdown twins (`/<page>.md`, `Accept: text/markdown`) · REST `/api/v1`
       + OpenAPI · MCP `/mcp` · `/go?q=` resolver · JSON-LD in page heads ·
       `/docs/*.md`, `/llms.txt`. One facts function per page feeds all of
       them; every response carries a per-chain "indexed through" stamp.
       Browser pages never consume these routes (see docs/api.md).
```

USD is a display concern only: the browser fetches DeFiLlama prices
(`src/lib/prices.ts`, 5-min cache). The indexer stays purely
on-chain-deterministic — handlers must never call external APIs (they replay
on every re-index).

DeFiLlama is the explorer's **only** price feed. It lists tokens that trade
somewhere, which leaves out the per-market collateral vault shares
(`cbBTC-USDC-collat` and friends) — receipts for a deposit into a Morpho
variable market, with no market of their own. Those are priced from what a
share redeems for rather than fetched from anywhere:
`src/lib/vault-prices.ts` multiplies the indexer's on-chain assets-per-share
(`vault.totalAssets / totalSupply`, read hourly) by the DeFiLlama price of the
underlying asset, and `mergePrices` layers the result under the market prices.

Order of preference for any number on screen: **indexed table → DeFiLlama →
Tenor API**. The Tenor API (`api.tenor.finance/graphql`) is a last resort for
what neither can answer, and only two callers qualify — oracle feed metadata
and history (one oracle per request), and the token-icon CDN with its
vault-collateral map. It is slow and rate-limited, so every call carries a
timeout, a low retry count, and degrades to "—".

Position-level aggregates are indexed, not fetched: `/users` rolls the
`position` and `positionCollateral` tables into per-account borrower and
lender rows in the browser (`src/lib/users.ts`). Amounts there are **units** —
1 unit = 1 loan-token smallest-unit at settlement — so a borrower's total is
what they owe at maturity and a lender's is face value at maturity. What a
lender paid for that credit is not indexed, so there is no accrued-interest
figure; tracking a per-position cost basis is a backlog item.

**Whose app built each position** — Tenor, Morpho, Coinbase or direct — is
derived in the browser too (`src/lib/book-routes.ts`), with no indexer state
of its own. A live Tenor MigrationRatifier policy on the position's market
family, on its side of the book (a borrow renewal or exit to Blue, a lend
renewal or exit to a vault), makes the whole position Tenor's. Otherwise the
takes that built it decide, pro rata to the debt or credit each one created:
the TAKER's route is the router ladder (`routeSql`, as on /tools/routers);
the MAKER's comes from its offer — a Tenor ratifier, then the lend offer's
settlement callback (the take's `payer`: a Tenor callback, or a
factory-deployed BlueBuyCallback, which is Morpho's app), then how the offer
reached the Mempool (`broadcastRouteProtocol`, the /mempool Route rule: Bundler3 or a smart wallet is the
Tenor app, MidnightBundlesV2 Morpho's, the Mempool itself direct), with the
Coinbase account tag between the Tenor and Morpho rungs. The Take event's
ratifier is read from its activity row's `argsJson`, and the broadcast is
found by two LATERAL probes — offer hash, then (maker, group) for rate offers
whose hash the taker changed. Collateral follows its market's borrow side.
`/tools/deposits-borrows` ("Deposits & Borrows") splits the book by route —
a By route table, then every wallet with a route filter — reading the same
open positions as `/users` with the attribution switched on
(`useUsers({ attribution: true })`; `/users` leaves it off). `/address`
labels each position.

On the deposit side: **lend fixed** is credit on Midnight; **lend quotes** is
liquidity committed to quote loans at fixed rates — live lend offers waiting
for a borrower, whether the money is parked in a settlement callback (a
BlueBuyCallback's deposit on Morpho Blue, a Tenor vault callback) or held in
the maker's wallet; **deposits** = lend fixed + lend quotes + collateral.

The tool's chart is the same book OVER TIME (`src/lib/book-history.ts`):
borrows, lend fixed, lend quotes, collateral or deposits at the end of each
day — or each month, its last day — stacked by route. History comes from `positionChange` and
`collateralChange`: two grouped-subquery joins keep each position's last
write of each UTC day (~8k and ~5k rows on 2026-10-28, ~0.6s each), and the
browser carries each balance forward to its next write, through one
difference array per route. Each position keeps its CURRENT attribution over
its whole past, and every day is priced at today's prices, as on
`/tools/routers`. Lend quotes come from the indexer's end-of-day snapshots
(below), today's bar being the live figure; Deposits adds the three up day by
day. Collateral counts every balance, whether or not the account still owes,
in the chart and the table alike.

**Lend quotes** are the one figure here the indexer keeps state for, because
the browser cannot find live offers: ~1.6M broadcast offers on Base, no index
on expiry, and live ones reaching back to the first weeks (year-long vault
offers are posted once). `lendOfferGroup` holds one row per consumption group
with the lend offers that could still be its largest live one (shared
`lend-offers.ts`); `liveIdx` makes "still pending" an index range. A group
offers its largest live cap minus `consumed[maker][group]`; groups paid from
one BlueBuyCallback position are capped together at what it holds (shared
`lendQuotes`, the one sizing rule for the live figures and their history).
Attribution (`src/lib/pending-liquidity.ts`): a BlueBuyCallback is Morpho's,
a Tenor callback (the vault ones) Tenor's, else the broadcast route —
Bundler3 or a smart wallet Tenor, MidnightBundlesV2 Morpho, the Mempool
direct. A wallet-funded quote counts at its cap; the maker's balance is not
indexed.

Their HISTORY is a snapshot, since nothing else keeps it: the first
StatSnapshot tick of each UTC day (`src/lib/lend-quotes.ts`) sizes the groups
live at that moment, the BlueBuyCallback deposits as last read, into
`lendQuoteSnapshot` — labelled the day that just ended, guarded by
`lendQuoteDay` so it is taken once. Rows keep the facts the route rule reads,
not the route (a registry question the web answers), and fold the makers
out: the rule compares the broadcast's sender with the maker and the `to`,
so a row keeps those comparisons instead of the addresses — ~20 rows a day on
Base where per-maker rows would be ~450. A re-index replays it over all of
history; no RPC of its own.

**No migrations, by design.** Deploys index into a fresh schema named
`mn_<git-sha8>` and repoint views when ready (`--views-schema midnight`).
Editing `ponder.schema.ts` or any handler = full re-index on next deploy
(minutes; RPC responses are cached in Postgres). A weekly scheduled task
prunes old schemas.

## The two contracts

- **Midnight** — the protocol. Markets, positions, takes, liquidations, fees.
- **Mempool** (`periphery/Log.sol`) — a broadcast channel: its fallback ALWAYS
  emits `Data(msg.data)` (≤1MB). Offers live off-chain; this is how makers
  publish them. See `docs/decoder.md` for the wire format.

## Protocol model (the facts handlers encode)

- **1 unit = 1 zero-coupon obligation = 1 smallest-unit of loanToken at
  settlement.** Everywhere except `take`, units↔assets is 1:1 (repay,
  withdraw, liquidate). Only `take` applies a price < 1 — the discount IS the
  interest.
- **Buyer = lender, seller = borrower.** In a Take: buyer gains credit (or
  burns own debt); seller gains debt (or burns own credit). A user never holds
  credit and debt simultaneously in one market.
- **Ticks encode prices, not rates**: `price(tick) = 1e18 / (1 + 1.005^(3372 − tick))`,
  WAD, monotone. We use the vendored on-chain lookup table
  (`packages/shared/src/ticks-table.ts`) — never reimplement the exponential.
  Implied APR (display convention): `(1e18 − price)/price / ttm × secondsPerYear`.
- **marketId** = full CREATE2 hash: the Market struct is SSTORE2'd by the
  midnight contract at salt 0; `computeMarketId` in shared reproduces it
  (validated against real `Take.id_`).
- **A "market" is not maturity-defined**: the renewal-stable identity is the
  **Tenor market id** — `keccak256(abi.encode(chainId, midnight, loanToken,
  collateralParams, rcfThreshold, enterGate, liquidatorGate))`, i.e. the
  Market struct minus maturity (`TenorMarketIdLib.sol` in tenor-contracts).
  Many Midnight markets (one per maturity) share one Tenor market.
  `computeTenorMarketId` in shared reproduces it; the web groups client-side
  via `useTenorMarkets` (home "Largest markets", /markets) — each maturity
  row drills into its Midnight marketId page. `useTenorMarketGroups` is the
  same grouping without the live totals, for filters, labels and icon
  lookups: it reads only the market columns fixed at creation, so it is not
  refetched per block.
- **Vault market ids** name a vault-wrapped Tenor market without hashing:
  `vaultToTenorMarketId` packs the vault's address into the high 20 bytes over
  12 zero bytes (the migration ratifier keys vault lend routes this way). They
  commit to no chain; `tenorMarketIdToVault` in shared reads one back, and the
  vault registry supplies the chain (market-id decoder, search).
- **offerHash** = `keccak256(abi.encode(offer))` (validated against real
  `Take.offerHash`). Note: this is NOT the EIP-712 hash ratifiers sign.
- **Offer death**: consumed (`consumed[maker][group]` ≥ maxUnits/maxAssets —
  via fills or manual `SetConsumed`, which is absolute + monotone), expiry
  window, ratifier de-authorization, or market fee rising above the offer's
  cap. `consumed` is keyed (maker, group) GLOBALLY — one group can span
  markets; don't scope it by market.
- **Maturity** is a timestamp, not an event: after it, no new debt
  (`take` reverts on debt increase), post-maturity liquidations become
  unconditional (LIF ramps to max over 60 min), and lenders withdraw
  first-come-first-served from `withdrawable` (funded only by
  repayments/liquidations).
- **lossFactor** (scale `uint128.max`, NOT WAD) only moves on bad-debt
  liquidations; lender losses apply lazily via `UpdatePosition`, which the
  contract emits BEFORE any Take it affects in the same tx — so handlers
  processing logs in order need no accrual math.

## Handler invariants (checked in review)

- Every table PK starts with `chainId`; every query filters by it. This is
  what makes Ponder's default multichain ordering safe.
- Handlers never throw — tagged failures + raw bytes instead.
- Markets referenced before their MarketCreated is indexed (partial-window
  local dev) are chain-seeded via `ensureMarket` (`toMarket` + `marketState`
  reads, cached); balances/aggregates floor at zero (`max0`) since on-chain
  values can't be negative — both are no-ops in full-history production.
- Take per-side splits (`buyerCreditIncrease = max(0, units − buyerDebt)`,
  `sellerCreditDecrease = min(units, sellerCredit)`) depend on OUR tracked
  positions; the handler cross-checks against the event's `totalUnitsDelta`
  and warns on drift.
- Useful protocol invariant for sanity checks:
  `market.totalUnits == Σ position.debt + market.withdrawable`.
- Tenor vault totals (`vault`, `vault_snapshot`) are the one table pair that is
  READ, not derived from logs: an ERC-4626's `totalAssets()` grows with yield
  accrual, which emits nothing. The snapshot block handler reads
  `totalAssets`/`totalSupply` for every vault in the registry
  (`packages/shared/src/vaults.ts`) at the tick's block — still deterministic
  and replayable (block-pinned reads, cached in `ponder_sync`), never an
  external API. A vault that can't be read skips that hour instead of throwing.
- Vault holders (`vault_holder`) are the hybrid: the share token's ERC-20
  `Transfer` is the trigger, but the stored balance is `balanceOf(holder)` READ
  from the vault at the event's block (one multicall for both sides), not the
  running sum of deltas — exact wherever the indexer's window starts, immune
  to same-block pairs and self-transfers. Only a failed read falls back to the
  delta (floored at zero); the next transfer re-reads and corrects it.
- Vault activity (`vault_event`, /tools/vault-activity) is plain log decoding:
  one row per VaultV2 `Deposit`/`Withdraw`/`Allocate`/`Deallocate`/
  `ForceDeallocate`/`AccrueInterest` and per holder-to-holder share `Transfer`
  (mints and burns are the Deposit/Withdraw beside them; an `AccrueInterest`
  that accrued nothing is skipped). No reads, so it is exact from the first
  block the indexer sees.

## Decision record (why it's built this way)

- **Ponder over SQD**: single process, 3 config files, zero codegen/migration
  rituals, first-party SSE live queries. SQD would have meant a
  codegen+migration workflow, a deprecated GraphQL server, no production
  live-update story, and its managed cloud can't run in our AWS.
- **All-ECS over Amplify**: Amplify buffers SSE (kills live queries), capped
  at Next 15, and would split deploys into two systems. One ALB with path
  routing gives same-origin `/sql/*` (no CORS needed) on one hostname.
- **No public GraphQL**: the web app only ever queries `/sql`; Ponder's
  GraphQL middleware (and its GraphiQL UI + introspection) was removed from
  the API as unused public surface.
- **Terraform over CDK/SST**: house convention across the tenor repos (same
  `modules/` + tfvars layout as tenor-api), applied from a laptop. CI keeps
  GitHub-OIDC auth (ahead of house — no long-lived keys) but only builds
  images and registers task-definition revisions; it never touches
  infrastructure. (Started as CDK; ported before ever deploying.)
- **Per-deploy schemas over migrations**: schema changes become "edit one
  file, merge" — the only schema-change workflow a non-developer can't break.
- **DeFiLlama in the browser over indexer-side prices**: keeps handlers
  deterministic and replayable; USD is presentation.