@ekubo/indexer

Service for indexing Ekubo events into a Postgres database.

The indexer focuses on producing an always-consistent realtime view of Ekubo events, using the Starkstream service to get a stream of relevant data.

Events are not transformed by the indexer, simply cataloged for later use such as in materialized views or complex analytical queries.

It can take days to sync a fresh database with all the networks, so it's recommended to start from a backup of our production database.

Nightly dumps are published by .github/workflows/pg-dump.yaml—grab the most recent db-backup-<timestamp>.dump artifact from the Actions tab, then import it into your Postgres instance:

pg_restore --clean --if-exists --no-owner \
  --dbname postgres://user:pass@host:5432/dbname \
  db-backup-20240101T000000Z.dump

During restore you may see warnings or errors about the DigitalOcean doadmin role or the pg_cron extension; those are expected and can be ignored if your target database lacks the same privileges/extensions.

Join the Discord and ask in the #devs channel if you need support.

Nightly backups run through .github/workflows/pg-dump.yaml, which connects to the production database using repository secrets, runs pg_dump -Fc, and uploads the resulting db-backup-<timestamp>.dump as a GitHub Actions artifact (retained for 7 days, named db-backup-<run_id>). These artifacts let you bootstrap a new node quickly without waiting for a multi-day sync—grab the latest run from the Actions tab when you need a fresh snapshot.

Build the runtime image once. Bun executes the TypeScript sources directly, so no separate build step is required:

docker build -t ekubo-indexer .

CI publishes the same image to GitHub Container Registry under ghcr.io/ekuboprotocol/indexer:<git-sha> so other environments can pull the exact build:

docker pull ghcr.io/ekuboprotocol/indexer:<git-sha>

The resulting image can execute any of the TypeScript entrypoints. Run the network-specific indexer entrypoint directly:

docker run --rm \
  -e NETWORK=mainnet \
  ekubo-indexer bun src/starknet.ts

Override the command to reuse the same image for auxiliary scripts such as migrations. The default entrypoint is already bun, so point it to the desired TypeScript file:

docker run --rm ekubo-indexer scripts/migrate.ts

Match the examples in .do/app.yaml to run other helpers, e.g.:

docker run --rm ekubo-indexer src/price-sync/index.ts

Token metadata generation and database synchronization are owned by the EkuboProtocol/default-tokens repository. The indexer image does not fetch or write token metadata.

The price-sync process runs every configured job as an independent recurring loop. A job declares the chains it may write prices for plus a three-character source identifier; startup fails if two jobs claim the same chain and source. Most jobs price one chain, but a job may price several when one upstream request covers them all. TOKEN_PRICE_SYNC_INTERVAL_MS controls the default cadence in milliseconds (default: 60000). CoinGecko jobs use COINGECKO_TOKEN_PRICE_SYNC_INTERVAL_SECONDS; set it to a positive number and provide COINGECKO_API_KEY to enable them. Zero or an unset value disables those jobs.

Two properties keep CoinGecko request volume flat rather than growing with erc20_tokens:

• Native currency prices for every chain come from a single cgn job. Chains sharing a CoinGecko coin ID cost one request between them, not one apiece.
• The per-chain cg1 jobs request only the tokens CoinGecko has actually priced. Everything else is re-probed on a slow rotation (a full pass per day), so a chain with thousands of unlisted tokens does not pay for them every cycle. This state is held in the worker process, so a restart replays one full sweep before settling back down.

Per-source confidence lives in erc20_token_price_sources, seeded once by the migration — the worker never writes that table, so policy adjustments (made in a later migration) survive deploys. Freshness is per observation instead: each price row carries a valid_until, stamped by the job as three of its own sync intervals (one-minute floor), except Chainlink, which uses the feed's heartbeat window anchored at the round's updatedAt. Rows predating the column fall back to five minutes past their timestamp. The latest-price cache prefers the quoter (the interface derives price-impact loss from it), then Chainlink, then CoinGecko, then SushiSwap; it averages sources tied at the highest confidence, records the result under the synthetic AVG source, and reconciles expiration once per second so a lower-confidence source is promoted when the leader goes stale.

Chainlink token/USD feeds supplement those sources over EVM RPC. Set CHAINLINK_TOKEN_PRICE_SYNC_INTERVAL_SECONDS to a positive number and provide CHAINLINK_TOKEN_PRICE_CONFIG as a JSON object keyed by chain ID. Each chain declares fallback RPC URLs and a Chainlink Reference Data Directory catalogUrl. The catalog supplies token/USD proxy addresses and heartbeats; feeds are matched only when an eligible indexed token and catalog base asset have a unique symbol. Discovery prefers standard reference-price proxies, falls back to the underlying proxy associated with a shared SVR feed on networks that only publish that variant, and also supports primary tokenized-price feeds. Secondary SVR proxies, ambiguous symbols, and hidden, deprecating, or non-USD feeds are skipped. An optional feeds array can override or supplement discovery for exceptional mappings.

Catalogs are refreshed hourly by default, controlled by CHAINLINK_FEED_CATALOG_REFRESH_INTERVAL_SECONDS, and the last successful response remains usable during a catalog outage. All discovered feed reads for a chain are aggregated into one on-chain Multicall3 call, which the RPC provider accounts for as a single eth_call; multicallAddress can override the standard 0xcA11...CA11 deployment for a chain. Stale, incomplete, non-positive, and superseded rounds are skipped. The RPC-reported chain ID is also checked before reading feeds.

{
  "1": {
    "rpcUrls": ["https://eth-mainnet.example/v1/API_KEY"],
    "catalogUrl": "https://reference-data-directory.vercel.app/feeds-mainnet.json"
  }
}

Chainlink jobs are disabled when the interval is zero/unset or the config is empty. Valid observations are stored under the cl1 source using the feed round's updatedAt timestamp, and unchanged rounds are not inserted repeatedly. One failing feed does not prevent fresh observations from other configured feeds on that chain.

• Local: bun run migrate or bun scripts/migrate.ts (both invoke scripts/migrate.ts).
• Docker: docker run --rm --env-file .env ekubo-indexer scripts/migrate.ts.
• DigitalOcean: the .do/app.yaml run-migrations pre-deploy job automatically applies migrations before rolling out new workers, ensuring the Postgres schema is up-to-date.

Migration files live under migrations/ and execute in order via scripts/migrate.ts.

The DigitalOcean Apps spec in .do/app.yaml documents the full production stack:

• Workers for each network (e.g.: starknet-sepolia, starknet-mainnet, eth-sepolia, eth-mainnet) that run the corresponding network entrypoint (bun src/starknet.ts or bun src/evm.ts) with the appropriate NETWORK value, pulling the published Docker image (ghcr.io/ekuboprotocol/indexer:${IMAGE_TAG}).
• Managed Postgres (indexer-db-nyc1) wired in via the PG_CONNECTION_STRING env var along with secrets such as DNA_TOKEN.
• A run-migrations pre-deploy job and the long-running src/price-sync/index.ts process. Each price source/chain job has an independent timer, with separately configured CoinGecko and Chainlink cadences. The app spec discovers Chainlink feeds for eligible tokens on Ethereum, Base, Arbitrum, and Robinhood through Chainlink's multi-network catalogs and the existing Alchemy API key secret.

Use this file as a base to recreate the stack in a new DigitalOcean App Platform project or as a reference for configuring similar infrastructure elsewhere.

This log records indexer deployments that:

• require manual intervention beyond running scripts/migrate.ts (e.g., backfilling data, reseeding state, or pausing workers), or
• introduce schema changes, even when the standard migration workflow can apply them automatically. Schema-only updates may not mandate manual steps but can still break downstream consumers that rely on the previous structure, so they belong here as well.

Per-source confidence now lives in erc20_token_price_sources, seeded by the migration and never written by the worker, with the quoter ranked highest. erc20_tokens_usd_prices gains a nullable valid_until that each observation carries (legacy rows are treated as valid for five minutes past their timestamp). The compact erc20_tokens_latest_price_by_source cache tracks those expirations, while the physical, primary-keyed erc20_tokens_latest_price table stores the fresh maximum-confidence aggregate for fast quoter reads. The price worker reconciles expirations once per second, promoting a lower-confidence source when needed. The all_pool_states_view definition remains unchanged. Apply migrations before deploying the updated price-sync worker. Consumers selecting every column from erc20_tokens_latest_price must account for its new confidence and valid_until columns and the synthetic AVG source on tied values; consumers of erc20_tokens_usd_prices gain a nullable column. No manual backfill is required.

This release also adds the cl1 Chainlink price source, which reads token/USD reference feeds over EVM RPC and is seeded above the aggregator APIs (the quoter ranks highest). It is inert until CHAINLINK_TOKEN_PRICE_SYNC_INTERVAL_SECONDS and CHAINLINK_TOKEN_PRICE_CONFIG are set, so no manual intervention is required to deploy without it.

Adds two indexes on pool_keys — (chain_id, token1) and (chain_id, pool_extension) — so the API's new /poolKeys discovery route can filter by a single token (either side) or by extension without scanning a chain's whole pool set. Schema-only change: run the migration; no manual backfill is required and no existing structure changes.

erc20_tokens now has a nullable, non-negative integer circulating_supply column. Like total_supply, the value is stored in the token's indivisible units so consumers must divide by 10 ^ token_decimals before multiplying by a per-token USD price. Apply this migration before deploying the default-tokens database sync that writes the new field. Existing rows remain NULL until that sync finds a supply source; no manual backfill is required.

hourly_volume_by_token now tracks the Ve33 portion of its total fees in a dedicated ve33_fees column. The 24-hour pool stats views expose ve33_fees0_24h and ve33_fees1_24h as components of the existing inclusive fees0_24h and fees1_24h totals. The migration backfills the breakdown from indexed PoolFeesAccounted events and keeps both totals reorg-safe. Apply the indexer migration before deploying API code that selects the new columns; no manual backfill is required.

erc20_tokens_usd_prices now has a covering index on (chain_id, token_address, timestamp DESC) that includes the price value and source. Apply migrations before deploying the token price-history API to keep its bounded chart queries index-only. No backfill or manual intervention is required beyond running the migration.

The per_pool_per_tick_liquidity triggers now retain transient rows until both the net liquidity delta and total liquidity are zero. This prevents order-dependent corruption when position updates are cascade-deleted during a reorg. The migration atomically rebuilds every tick aggregate from canonical position_updates; no manual backfill is required beyond running the migration.

pool_keys now has an index on (chain_id, token0, token1) so API queries can find every configuration for a token pair without scanning all pool keys. Apply migrations before deploying the optimized pair-events API query. No backfill or manual intervention is required beyond running the migration.

EVM V3 PoolFeesAccounted events now contribute to hourly_volume_by_token.fees, which feeds the API's fee totals and APRs. The migration backfills existing Ve33 fee events and keeps the hourly aggregates correct when events are inserted or removed during a reorg. These events now require a pool_key_id, so an unresolved pool fails indexing instead of being silently omitted from fee stats. Apply migrations before deploying the updated EVM indexer; no manual backfill is expected.

The DigitalOcean sync-tokens scheduled job and scripts/sync-tokens.ts were removed. Before deploying this indexer version, configure the EkuboProtocol/default-tokens update and database-sync workflow secrets, run the token-list update once, and run the separate database sync once. Future token sources, generated metadata, provenance, bridge mappings, and hosted logos are audited in that repository.

EVM V3 VoteWeightApplied events now store the stake's selected fee in ve33_vote_weight_applied.voted_swap_fee. The migration backfills existing rows to 0, then removes the column default so new rows must provide the value. Apply migrations before deploying the updated EVM indexer; no manual backfill is required.

EVM V3 Ve33 events now write to ve33_stake_changed, ve33_vote_weight_applied, ve33_pool_fees_accounted, ve33_pool_fees_claimed, ve33_emissions_scheduled, ve33_pool_emissions_accrued, and ve33_rewards_claimed. Sepolia also indexes VeToken and FreeVe33Positions ERC721 transfers when VE_TOKEN_V3_ADDRESS and VE33_POSITIONS_V3_ADDRESS are configured. Apply migrations before deploying consumers that read these tables.

Ve33 pool quote state is now maintained in ve33_pool_states and exposed through all_pool_states_view via ve33_* columns plus is_ve33_pool. Apply migrations before deploying consumers that select from the view; no manual backfill is required beyond the migration.

all_pool_states_view no longer filters rows by supported pool extension state markers, so any pool with a pool_states row now appears in the view. Downstream consumers that assumed the view contained only quoter-supported pools should add their own filtering before deploy; no backfill or manual intervention is required beyond running migrations.

all_pool_states_view now includes pool_tvl_usd, computed from pool_tvl plus erc20_tokens_latest_price for both pool tokens. The column is NULL when either side lacks a latest USD price. Apply migrations before deploying any consumer that selects from this view.

EVM V3 auction events now write to auction_completed, auction_funds_added, auction_boost_started, and auction_creator_proceeds_collected.

Boosted fees now write to boosted_fees_events, boosted_fees_donate_rate_deltas, and boosted_fees_donated, while all_pool_states_view now exposes the boosted fee donate rates plus the last donated time and future deltas. Run migrations before deploying any consumers that read the view or expect boosted-fee schedules.

The indexer_cursor table now includes a fork_counter column that increments whenever the indexer deletes blocks during reorg handling. Downstream services can use it to detect reorgs even when the cursor position is unchanged. Run the migrations before deploying consumers that query indexer_cursor.

Incentives campaigns now require a single core_address and support optional locker/licensee filters (allowed_lockers). Run the migrations before computing rewards, and refresh incentives.campaign_rewards_overview_materialized after deploy so the updated filtering is reflected in dashboards.

The tvl_usd column has been removed from all_pool_states_view to keep the view lightweight. Update any consumers that read this column before deploying, then run the standard migrations; no backfill or manual work is required.

Pools now persist the raw PoolConfig word plus its decoded attributes. The pool_keys table gains pool_config, pool_config_type, stableswap_center_tick, and stableswap_amplification, and tick_spacing can be null for stableswap pools. all_pool_states_view also surfaces these new columns so downstream quoters can tell which pool type they are handling. Starknet pools continue to expose pool_config = NULL because their fee encoding is incompatible with the EVM packer. No manual work is required besides running the migrations, but any consumer that relied on tick_spacing always being non-null should be updated before ingesting stableswap data.

all_pool_states_view now joins limit_order_pool_states, exposes is_limit_order_pool, and allows pools with the limit-order extension to appear in the view. Apply migrations before deploying any component that reads this view; no manual backfills are required.

We had to reindex from the beginning due to a bug in inserting TWAMM proceeds withdrawal events. We also added some columns to the TWAMM order updates and TWAMM collect proceeds tables to improve correctness.