Overview

Acuity Index is a configurable event indexer for Substrate-based blockchains. It is primarily intended for dapps to query directly as an event indexer, although other consumers can use it too. It connects to a node over WebSocket RPC, decodes runtime events, stores queryable index entries in a local sled database, and exposes the indexed data through its own WebSockets API.

The project is intentionally config-driven:

• chain-specific indexing rules live in TOML instead of generated Rust types
• event payloads are decoded generically
• the on-disk index is built around explicit query keys
• returned events can include GRANDPA proofs so light clients can verify correctness
• operators can update accepted index specs without restarting the public service

Funding

Acuity Index was originally called Hybrid and was funded by two (1, 2) Web3 Foundation grants and a Kusama Treasury referendum. A second funding referendum failed. Treasury funding for a Kusama Forum based on Acuity Index has been secured.

Who This Book Is For

This book serves three overlapping audiences:

• operators running acuity-index against a live chain
• application developers integrating with the WebSockets API
• contributors working on the Rust codebase and benchmarks

Problem

Dapps need to read and write blockchain state, consume events, and often work with off-chain content such as IPFS. Current state can be read on-chain or off-chain, but historic state and events are usually consumed off-chain.

Writing data to events is often much cheaper than writing it to state, but that creates a retrieval problem for dapps: the chain emits the data, while users and applications still need a fast, trustworthy way to search and subscribe to it.

Why Plain RPC Is Not Enough

Typical dapps depend on centralized RPC providers. That creates several issues:

Incorrect Or Missing Data

A dapp generally trusts the RPC provider to return complete and correct results. A malicious or faulty provider can omit data or return misleading answers.

Event Query Limits

Substrate RPC does not provide rich event search. Other ecosystems often impose limits such as:

• maximum blocks scanned per query
• earliest scannable block
• maximum execution time
• indexing disabled on free tiers

Slow Queries

Scanning logs repeatedly is slower and more resource-intensive than querying a purpose-built index.

Unavailability And Policy Risk

Centralized providers can be unavailable due to outages, geoblocking, payment requirements, or policy decisions.

Tracking And Privacy Loss

Providers may correlate IP addresses, identities, and query history.

Pressure Toward Centralized Dapp Backends

Because raw RPC is a poor fit for search, teams are incentivized to build their own centralized indexing backend, which weakens the decentralization story.

More Expensive Chain Design

If developers cannot reliably search events, they may push more data into chain state just to make retrieval easier, increasing fees and block-space pressure.

Limited Extensibility

Chains and dapps may need richer indexing beyond events, including full-text, classification, or domain-specific search over linked off-chain data.

Acuity Index exists to solve the event indexing part of this problem with a configurable, chain-aware, high-performance utility.

Solution

Acuity Index runs alongside a Substrate node and builds queryable indexes over runtime events. Instead of forcing every dapp to rescan chain history, the indexer stores event references keyed by explicit query keys such as account IDs, object IDs, or composite identifiers.

Core Idea

At its simplest, Acuity Index maintains an index of (block_number, event_index) for each configured key. Clients can then:

• query matching events efficiently
• subscribe to live updates for a key
• optionally request decoded event payloads
• optionally request finalized proof material for returned blocks

Why This Helps

High Performance

Acuity Index uses sled to maintain an embedded local index. That is better suited to repeated key-based queries than ad hoc log scanning.

Config-Driven Operation

Per-chain indexing rules live in TOML, not generated Rust code. Operators can adapt behavior by editing index specs instead of recompiling the binary.

Well-Defined Query Surface

Every index node exposes the same public WebSockets API for a given binary version, making it easier for clients to switch providers.

Verification-Friendly Responses

When the indexer runs in finalized mode, GetEvents can include finalized block headers and System.Events storage proofs so clients can verify returned event material against the chain state root.

Extensible Design

The event index is the current focus, but the architecture leaves room for richer indexing layers in the future.

Features

User-Facing Features

• Config-driven indexing with support for scalar, composite, and multi-value keys
• Explicit pallet and event mappings in TOML
• Index specification checkpoints via spec_change_blocks for controlled reindexing
• Hot reload for accepted index-spec changes
• Concurrent backfill and live head catch-up
• WebSocket queries and subscriptions
• Event variant indexing via index_variant
• Optional finalized event proofs for GetEvents
• Prometheus / OpenMetrics metrics export
• Resume-safe persistence of indexed spans
• Synthetic devnet, integration tests, and end-to-end benchmarking

Operational Features

• chain identity protection through persisted genesis_hash
• recoverable RPC reconnect loop with exponential backoff
• public WebSocket server stays up across accepted spec reloads and transient upstream failures
• configurable connection and subscription limits
• bounded subscription control paths and backpressure handling

Installation

Requirements

• Rust stable as pinned by rust-toolchain.toml
• a Substrate node with WebSocket RPC enabled
• archival historical state on that node via --state-pruning archive-canonical
• polkadot-omni-node if you want to run integrations tests or indexing benchmark
• just for the documented developer command surface

Install The Binary

From crates.io:

cargo install acuity-index

For local development you can also run it directly without installing:

cargo run -- run ./mychain.toml

Build From Source

Common entry points:

just build
just test
just build-release

Install Documentation Tooling

If you want to build this book locally, install mdbook:

cargo install mdbook

Then use:

just book-build
just book-serve

Quickstart

1. Generate A Starter Spec

acuity-index generate-index-spec ./mychain.toml --url wss://mynode:443

This inspects live runtime metadata and writes a starter TOML file.

2. Review And Edit The Spec

At minimum, verify:

• name
• genesis_hash
• default_url
• declared [keys]
• [[pallets]] and [[pallets.events]] entries you want indexed

Remove keys you do not need and add composite keys where appropriate.

3. Run The Indexer

acuity-index run ./mychain.toml

Common overrides:

acuity-index run ./mychain.toml --url wss://mynode:443 --queue-depth 4 --port 8172

4. Query The Service

See the WebSockets API for the protocol and methods.

By default, connect to:

ws://localhost:8172

Minimal request:

{"id":1,"type":"Status"}

5. Iterate Safely

When run <INDEX_SPEC> points to a file, Acuity Index watches that file for accepted changes. Valid changes restart only the RPC/indexer loop. The public WebSocket server and optional metrics listener stay up.

Rejected changes do not kill the running process.

Configuration

Acuity Index uses two configuration layers:

• an index specification TOML that describes chain identity and indexing rules
• an optional runtime options TOML passed via --options-config

Runtime precedence is:

CLI flags > --options-config file > built-in defaults

Index Specification

Each chain is described by an index specification passed as <INDEX_SPEC>.

name = "mychain"
genesis_hash = "abc123..."
default_url = "wss://my-node:443"
index_variant = false
spec_change_blocks = [0]

[keys]
account_id = "bytes32"
item_id = "bytes32"
revision_id = "u32"
item_revision = { fields = ["bytes32", "u32"] }

[[pallets]]
name = "MyPallet"

[[pallets.events]]
name = "SomeEvent"

[[pallets.events.params]]
field = "who"
key = "account_id"

[[pallets.events.params]]
fields = ["item_id", "revision_id"]
key = "item_revision"

Important Fields

• name: logical name for the spec and default local storage path
• genesis_hash: chain identity guard; a database is tied to one chain only
• default_url: fallback node URL when CLI and runtime options do not override it
• index_variant: whether to store pallet and variant lookups in the variant tree
• spec_change_blocks: revision boundaries for historical reindex behavior

spec_change_blocks must:

• not be empty
• start with 0
• be strictly increasing

Declared Keys

Scalar key kinds:

• bytes32
• u32
• u64
• u128
• string
• bool

Composite keys are declared structurally:

[keys]
item_revision = { fields = ["bytes32", "u32"] }

Event parameter mappings use:

• field = "..." for scalar keys
• fields = ["...", "..."] for composite keys
• multi = true for one-to-many scalar extraction when the event field contains multiple values

Composite key values are ordered and binary encoded, so field order matters.

Runtime Options Config

Deployment-specific settings can live in a separate TOML file:

url = "wss://mynode:443"
db_path = "/var/lib/acuity-index/mychain"
db_mode = "low_space"
db_cache_capacity = "1024 MiB"
queue_depth = 4
finalized = false
port = 8172
metrics_port = 9000
max_connections = 1024
max_total_subscriptions = 65536
max_subscriptions_per_connection = 128
subscription_buffer_size = 256
subscription_control_buffer_size = 1024
idle_timeout_secs = 300
max_events_limit = 1000

Supported fields:

index_variant belongs in the index spec, not the runtime options file.

Hot Reload Notes

• index-spec changes can restart the indexer when accepted
• changes to name or genesis_hash are rejected
• options-config changes can also be reloaded
• some WebSocket settings are applied live, while others require indexer restart according to runtime validation logic

CLI Reference

Main Command

acuity-index [OPTIONS] <COMMAND>

Global Options

These options are accepted by acuity-index for every command:

Commands

run

acuity-index run [OPTIONS] <INDEX_SPEC>

Run the indexer for the specified chain.

Arguments

Options

purge-index

acuity-index purge-index [OPTIONS] <INDEX_SPEC>

Delete the index database for the specified chain.

Arguments

Options

generate-index-spec

acuity-index generate-index-spec [OPTIONS] --url <URL> <INDEX_SPEC>

Generate a starter index spec TOML from live node metadata.

Arguments

Options

WebSockets API

Connect to ws://localhost:8172 by default.

For Internet-facing deployment guidance and the current security review, see Security And Deployment.

The public API is a JSON-RPC 2.0-over-WebSocket protocol with two message classes:

• request/response messages, which carry a numeric or string id
• server notifications, which never carry an id

Protocol Overview

Requests

Every client request must conform to the JSON-RPC 2.0 specification:

• jsonrpc: must be "2.0"
• id: unsigned integer or string chosen by the client
• method: the RPC method name
• params: optional method parameters

Example:

{"jsonrpc":"2.0","id":1,"method":"acuity_indexStatus","params":{}}

Responses

Every successful request receives a JSON-RPC 2.0 response with the same id.

• jsonrpc: "2.0"
• id: matches the request id
• result: the method return value

Example:

{
  "jsonrpc": "2.0",
  "id": 1,
  "result": {
    "spans": [
      {"start": 1, "end": 1000}
    ]
  }
}

Error Responses

Failed requests receive a JSON-RPC 2.0 error response with the same id.

• jsonrpc: "2.0"
• id: matches the request id (or null if the id could not be parsed)
• error: error object with:
  • code: integer error code
  • message: human-readable description
  • data: optional additional information

Example:

{
  "jsonrpc": "2.0",
  "id": 1,
  "error": {
    "code": -32602,
    "message": "Invalid params",
    "data": {
      "reason": "invalid_key"
    }
  }
}

Notifications

Notifications are pushed by the server for active subscriptions. They use a consistent envelope:

• jsonrpc: "2.0"
• method: "acuity_subscription"
• params:
  • subscription: the subscription ID string
  • result: the notification payload, which always includes a type field

Example:

{
  "jsonrpc": "2.0",
  "method": "acuity_subscription",
  "params": {
    "subscription": "sub_123",
    "result": {
      "type": "event",
      "key": {"type": "Custom", "value": {"name": "ref_index", "kind": "u32", "value": 42}},
      "event": {
        "blockNumber": 50,
        "eventIndex": 3,
        "timestamp": 1717171717000,
        "event": {
          "specVersion": 1234,
          "palletName": "Referenda",
          "eventName": "Submitted",
          "palletIndex": 42,
          "variantIndex": 0,
          "eventIndex": 3,
          "fields": {"index": 42}
        }
      }
    }
  }
}

Methods

acuity_indexStatus

Request:

{"jsonrpc":"2.0","id":1,"method":"acuity_indexStatus","params":{}}

Response payload:

• spans: array of indexed block spans
• each span has:
  • start: first indexed block
  • end: last indexed block

Example:

{
  "jsonrpc": "2.0",
  "id": 1,
  "result": {
    "spans": [
      {"start": 1, "end": 1000}
    ]
  }
}

acuity_getEventMetadata

Request:

{"jsonrpc":"2.0","id":2,"method":"acuity_getEventMetadata","params":{}}

Response payload:

• pallets: array of pallets
• each pallet has:
  • index: pallet event index
  • name: pallet name
  • events: array of event variants
• each event variant has:
  • index: variant index
  • name: variant name

Example:

{
  "jsonrpc": "2.0",
  "id": 2,
  "result": {
    "pallets": [
      {
        "index": 42,
        "name": "Referenda",
        "events": [
          {"index": 0, "name": "Submitted"}
        ]
      }
    ]
  }
}

acuity_getEvents

Request parameters:

• key: query key
• limit: optional u16, default 100
• before: optional event cursor

Request:

{
  "jsonrpc": "2.0",
  "id": 3,
  "method": "acuity_getEvents",
  "params": {
    "key": {"type": "Custom", "value": {"name": "ref_index", "kind": "u32", "value": 42}},
    "limit": 100,
    "before": null
  }
}

Composite custom keys use an ordered array of typed values:

{
  "jsonrpc": "2.0",
  "id": 3,
  "method": "acuity_getEvents",
  "params": {
    "key": {
      "type": "Custom",
      "value": {
        "name": "item_revision",
        "kind": "composite",
        "value": [
          {"kind": "bytes32", "value": "0xabc123..."},
          {"kind": "u32", "value": 7}
        ]
      }
    }
  }
}

Response payload:

• key: the queried key
• events: matching hydrated events, newest first
• proofs: proof availability and items
• page: pagination cursor information

proofs object:

• available: boolean indicating whether proofs are included
• reason: stable machine-readable reason such as included, rpc_proof_unavailable, or finalized_proofs_unavailable
• message: human-readable explanation
• items: array of proof objects (present when available is true)

Each proof item contains:

• blockNumber: u32
• blockHash: hex-encoded block hash
• header: serialized block header JSON
• storageKey: hex-encoded System.Events storage key
• storageValue: hex-encoded SCALE-encoded System.Events bytes for that block
• storageProof: array of hex-encoded trie proof nodes

page object:

• nextCursor: cursor for the next page, or null if no more results
• hasMore: boolean indicating additional pages exist

Each event in events contains:

• blockNumber: u32
• eventIndex: zero-based u32 ordinal within the block
• timestamp: block timestamp in milliseconds since Unix epoch, from Timestamp::Now when available; 0 for blocks where that storage value is unavailable
• event: decoded event JSON

Decoded event JSON currently contains:

• specVersion
• palletName
• eventName
• palletIndex
• variantIndex
• eventIndex: zero-based u32 ordinal within the block
• fields

Example:

{
  "jsonrpc": "2.0",
  "id": 3,
  "result": {
    "key": {"type": "Custom", "value": {"name": "ref_index", "kind": "u32", "value": 42}},
    "events": [
      {
        "blockNumber": 50,
        "eventIndex": 3,
        "timestamp": 1717171717000,
        "event": {
          "specVersion": 1234,
          "palletName": "Referenda",
          "eventName": "Submitted",
          "palletIndex": 42,
          "variantIndex": 0,
          "eventIndex": 3,
          "fields": {
            "index": 42
          }
        }
      }
    ],
    "proofs": {
      "available": true,
      "reason": "included",
      "message": "Finalized event proofs included.",
      "items": [
        {
          "blockNumber": 50,
          "blockHash": "0xabc123...",
          "header": {
            "parent_hash": "0x...",
            "number": 50,
            "state_root": "0x...",
            "extrinsics_root": "0x...",
            "digest": {"logs": []}
          },
          "storageKey": "0x26aa394eea5630e07c48ae0c9558cef780d41e5e16056765bc8461851072c9d7",
          "storageValue": "0x...",
          "storageProof": ["0x..."]
        }
      ]
    },
    "page": {
      "nextCursor": {"blockNumber": 49, "eventIndex": 7},
      "hasMore": true
    }
  }
}

Example when proofs are unavailable (indexer not in finalized mode):

{
  "jsonrpc": "2.0",
  "id": 3,
  "result": {
    "key": {"type": "Custom", "value": {"name": "ref_index", "kind": "u32", "value": 42}},
    "events": [{
      "blockNumber": 50,
      "eventIndex": 3,
      "timestamp": 1717171717000,
      "event": {
        "specVersion": 1234,
        "palletName": "Referenda",
        "eventName": "Submitted",
        "palletIndex": 42,
        "variantIndex": 0,
        "eventIndex": 3,
        "fields": {
          "index": 42
        }
      }
    }],
    "proofs": {
      "available": false,
      "reason": "finalized_proofs_unavailable",
      "message": "Finalized proofs are only available when the indexer is running with finalized indexing."
    },
    "page": {
      "nextCursor": null,
      "hasMore": false
    }
  }
}

acuity_subscribeStatus

Request:

{"jsonrpc":"2.0","id":5,"method":"acuity_subscribeStatus","params":{}}

Response payload: subscription ID string

Example:

{
  "jsonrpc": "2.0",
  "id": 5,
  "result": "sub_abc123"
}

acuity_unsubscribeStatus

Request:

{"jsonrpc":"2.0","id":6,"method":"acuity_unsubscribeStatus","params":{"subscription":"sub_abc123"}}

Response payload: true

Example:

{
  "jsonrpc": "2.0",
  "id": 6,
  "result": true
}

acuity_subscribeEvents

Request:

{
  "jsonrpc": "2.0",
  "id": 7,
  "method": "acuity_subscribeEvents",
  "params": {
    "key": {"type": "Custom", "value": {"name": "account_id", "kind": "bytes32", "value": "0xabc..."}}
  }
}

Response payload: subscription ID string

Example:

{
  "jsonrpc": "2.0",
  "id": 7,
  "result": "sub_def456"
}

acuity_unsubscribeEvents

Request:

{"jsonrpc":"2.0","id":8,"method":"acuity_unsubscribeEvents","params":{"subscription":"sub_def456"}}

Response payload: true

Example:

{
  "jsonrpc": "2.0",
  "id": 8,
  "result": true
}

Notifications

All subscription notifications use the method acuity_subscription and include the subscription ID in params.subscription. The params.result object always contains a type field indicating the notification type.

Status notifications

Sent to status subscribers whenever the persisted indexed span advances.

params.result.type: "status"

Payload (in params.result):

• spans: same array returned by acuity_indexStatus

Example:

{
  "jsonrpc": "2.0",
  "method": "acuity_subscription",
  "params": {
    "subscription": "sub_abc123",
    "result": {
      "type": "status",
      "spans": [
        {"start": 1, "end": 1001}
      ]
    }
  }
}

Event notifications

Sent to event subscribers whenever a matching event is indexed.

params.result.type: "event"

Payload (in params.result):

• key: subscribed key that matched
• event: matching hydrated event object

Example:

{
  "jsonrpc": "2.0",
  "method": "acuity_subscription",
  "params": {
    "subscription": "sub_def456",
    "result": {
      "type": "event",
      "key": {"type": "Custom", "value": {"name": "item_id", "kind": "bytes32", "value": "0xabc..."}},
      "event": {
        "blockNumber": 50,
        "eventIndex": 3,
        "timestamp": 1717171717000,
        "event": {
          "specVersion": 1234,
          "palletName": "Referenda",
          "eventName": "Submitted",
          "palletIndex": 42,
          "variantIndex": 0,
          "eventIndex": 3,
          "fields": {
            "index": 42
          }
        }
      }
    }
  }
}

Subscription termination notifications

Sent best-effort before the server drops a subscriber because it cannot keep up.

params.result.type: "terminated"

Payload (in params.result):

• reason: currently backpressure
• message: human-readable explanation

Example:

{
  "jsonrpc": "2.0",
  "method": "acuity_subscription",
  "params": {
    "subscription": "sub_def456",
    "result": {
      "type": "terminated",
      "reason": "backpressure",
      "message": "subscriber disconnected due to backpressure"
    }
  }
}

This notification is best-effort only. If the subscriber queue is already full, the termination notice may not be delivered before the connection is dropped.

Errors

The server uses standard JSON-RPC 2.0 error codes:

Application-specific error reasons are provided in error.data.reason:

• subscription_limit: connection exceeds the per-connection subscription cap or the global total subscription cap
• temporarily_unavailable: an RPC-backed request (acuity_getEventMetadata or acuity_getEvents) is made while the node connection is down

Example when no request id could be recovered:

{
  "jsonrpc": "2.0",
  "id": null,
  "error": {
    "code": -32700,
    "message": "Parse error"
  }
}

Invalid custom key payloads are returned as -32602 responses, including:

• custom key names longer than 128 bytes
• custom string values longer than 1024 bytes
• composite keys with more than 64 elements
• composite keys nested deeper than 8 composite levels
• custom values whose encoded key payload exceeds 16384 bytes

Operational failure modes that may also appear as connection drops or protocol-level failures include:

• oversized WebSocket frame or message rejected during protocol handling
• subscription control queue saturation
• connection idle timeout

The server sends periodic WebSocket ping frames as a heartbeat. By default this is every 120 seconds, and when an idle timeout is configured the heartbeat interval is reduced to at most half of idle_timeout_secs. Incoming ping/pong frames count as connection activity.

During node outages, local requests such as acuity_indexStatus continue to work. acuity_getEventMetadata and acuity_getEvents require live RPC access and return -32001 (upstream unavailable) until the node connection is re-established.

Pagination Semantics

acuity_getEvents returns matches in descending order by (blockNumber, eventIndex).

• default limit is 100
• limit is clamped to 1..=max_events_limit (default 1000, configurable via --max-events-limit; invalid zero-valued configs are rejected at startup)
• before is exclusive

Example cursor:

{"blockNumber": 50, "eventIndex": 3}

This returns only events strictly older than (50, 3).

Use the page object in the response for cursor-based pagination:

{
  "page": {
    "nextCursor": {"blockNumber": 49, "eventIndex": 7},
    "hasMore": true
  }
}

Pass nextCursor as the before parameter to fetch the next page.

Key Format

Keys are JSON objects with a type discriminant.

Supported key shapes:

{"type":"Variant","value":[0,3]}
{"type":"Custom","value":{"name":"para_id","kind":"u32","value":1000}}
{"type":"Custom","value":{"name":"account_id","kind":"bytes32","value":"0x1234..."}}
{"type":"Custom","value":{"name":"published","kind":"bool","value":true}}
{"type":"Custom","value":{"name":"revision","kind":"u128","value":"42"}}
{"type":"Custom","value":{"name":"slug","kind":"string","value":"hello"}}
{"type":"Custom","value":{"name":"balance","kind":"u64","value":"42"}}

Variant

• array of [pallet_index, variant_index]

Custom

Fields:

• name: key name
• kind: scalar type
• value: scalar value encoded according to kind

Supported scalar kinds:

• bytes32: 32-byte hex string, 0x prefix accepted
• u32: JSON number
• u64: JSON integer or string on input, serialized as string on output
• u128: JSON integer or string on input, serialized as string on output
• string: JSON string
• bool: JSON boolean

Composite keys are encoded recursively and must satisfy these protocol-level limits:

• max composite elements per composite value: 64
• max composite nesting depth: 8
• max encoded custom value size: 16384 bytes

Storage Notes

Index entries store event refs locally in sled.

Hydrated event payloads are fetched from the node on demand for:

• acuity_getEvents responses
• event subscription notifications

Hydration also reads Timestamp::Now for each returned block so every hydrated event includes a top-level timestamp field. For rare blocks where Timestamp::Now is unavailable, the API returns timestamp: 0 instead of failing the request.

The API always returns hydrated events for these surfaces, so live node access is required to serve them.

Delivery and Backpressure

Subscription delivery uses bounded internal queues.

• slow subscribers are removed instead of being buffered indefinitely
• the server attempts to send a terminated notification before removal
• if that best-effort notification cannot be queued, the client may observe only the disconnect

Connection Limits

The server applies these connection-level limits (defaults; all configurable via CLI flags or --options-config):

• max concurrent WebSocket connections: 1024 (--max-connections)
• max total subscriptions across all connections: 65536 (--max-total-subscriptions)
• max subscriptions per connection: 128 (--max-subscriptions-per-connection)
• subscription notification buffer size: 256 (--subscription-buffer-size)
• subscription control channel buffer: 1024 (--subscription-control-buffer-size)
• idle timeout: 300s (--idle-timeout-secs, 0 disables the timeout)
• max events per query: 1000 (--max-events-limit)

If the global connection cap is exhausted, new upgrade attempts are rejected with HTTP 503 Service Unavailable.

If the total subscription cap is reached, new subscription requests are rejected with a -32602 error response with data.reason: "subscription_limit". Because the subscription was never established, no termination notification is sent for that initial rejection.

Protocol-level limits (not configurable at runtime):

• max WebSocket message size: 256 KiB
• max WebSocket frame size: 64 KiB
• max custom key name length: 128 bytes
• max custom string key length: 1024 bytes
• max composite elements per composite value: 64
• max composite nesting depth: 8
• max encoded custom value size: 16384 bytes

Recipes

Run Against A Live Node

acuity-index run ./mychain.toml --url wss://mynode:443

Generate A Starter Spec

acuity-index generate-index-spec ./mychain.toml --url wss://mynode:443

Purge An Existing Index

acuity-index purge-index ./mychain.toml

Start The Synthetic Node

just synthetic-node

Seed Synthetic Smoke Data

just seed-smoke

Run Ignored Integration Tests

just test-integration

Build The Book

just book-build

Serve The Book Locally

just book-serve

Security And Deployment

This chapter summarizes the current deployment posture of the Internet-facing surfaces in this repository, the hardening already implemented, and the main remaining requirements before the indexer is exposed directly to dapp clients.

It is a companion to Architecture and WebSockets API, and should reflect the current code layout under src/main.rs, src/indexer.rs, src/event_hydration.rs, src/runtime_state.rs, src/protocol.rs, and src/metrics.rs.

Scope

Reviewed components:

• src/main.rs
• src/indexer.rs
• src/event_hydration.rs
• src/runtime_state.rs
• src/protocol.rs
• src/metrics.rs
• dependency surface via cargo audit

Primary attack surface:

• public WebSocket listener on 0.0.0.0:<port>
• JSON-RPC request parsing for:
  • acuity_indexStatus
  • acuity_getEventMetadata
  • acuity_getEvents
  • acuity_subscribeStatus
  • acuity_unsubscribeStatus
  • acuity_subscribeEvents
  • acuity_unsubscribeEvents
• subscription dispatch path between connection handlers and the indexer/runtime subscription registry
• optional OpenMetrics HTTP listener on 0.0.0.0:<metrics_port>
• upstream node RPC trust boundary used for event hydration and proof retrieval

Current Hardening

The following protections are implemented in the current server.

Resource-exhaustion controls

• Bounded subscription control queue

  • The connection-to-dispatcher control path uses a bounded Tokio channel.
  • Saturation is treated as a disconnect/error path rather than unbounded buffering.

• Bounded per-subscriber notification queues

  • Each connection gets a bounded notification channel for subscription pushes.
  • Slow subscribers are removed instead of being buffered indefinitely.
  • The server sends a best-effort terminated notification with reason backpressure before removal.

• WebSocket frame and message size limits

  • max WebSocket message size: 256 KiB
  • max WebSocket frame size: 64 KiB
  • Oversized payloads are rejected during protocol handling.

• Bounded custom key input sizes

  • custom key name limit: 128 bytes
  • custom string value limit: 1024 bytes
  • composite custom values are limited to:
    • 64 elements
    • 8 nesting levels
    • 16384 encoded bytes
  • Invalid or oversized key payloads are rejected before subscription registration or index scans.

• Global connection cap

  • Concurrent WebSocket connections are capped at 1024 by default.
  • When the cap is exhausted, new upgrade attempts are rejected with HTTP 503 Service Unavailable.

• Subscription caps

  • A single connection may hold at most 128 subscriptions by default.
  • There is also a global total subscription cap of 65536 by default across all connections.
  • Duplicate subscription attempts do not create extra server-side registrations.

• Query result cap

  • acuity_getEvents applies a configurable per-request limit clamp.
  • Default maximum events per query: 1000.
  • Client-provided limit values are clamped into 1..=max_events_limit.

• Idle timeout

  • Idle connections are closed after 300 seconds by default.
  • Setting idle_timeout_secs = 0 disables this timeout.

• Heartbeat pings

  • The server sends WebSocket ping frames periodically.
  • The interval is bounded by the idle-timeout configuration (idle_timeout_secs / 2, capped at 120 seconds).
  • Incoming ping/pong traffic counts as connection activity.

Crash-resistance and recovery improvements

• Recoverable upstream RPC failures no longer require a full process crash.

  • The supervisor loop reconnects with exponential backoff.
  • The current span is saved before the indexer task returns.
  • Existing WebSocket clients remain connected while local-only requests continue to work.

• RPC-backed requests degrade to temporary unavailability.

  • acuity_getEventMetadata and acuity_getEvents return JSON-RPC -32001 with data.reason = "temporarily_unavailable" while the node RPC is down.
  • acuity_indexStatus remains available from local sled state.

• Malformed persisted records are handled defensively.

  • Malformed span values/keys are skipped with logging during reads.
  • Malformed persisted event index records are skipped with logging rather than crashing the process.

• Poisoned subscription/runtime locks are recovered.

  • Shared runtime mutexes use a recovery path that logs and continues rather than panicking immediately.

• Startup and reconnect behavior are explicit.

  • Genesis-hash mismatch remains a fail-fast startup/runtime error to prevent cross-chain data mixing.
  • State-pruning misconfiguration remains a fatal operator error.

Exposure segmentation

• Metrics are served on a separate HTTP listener and port.
  • This keeps the observability surface distinct from the public WebSockets API.
  • It does not make the metrics endpoint safe for public exposure by itself.

Residual Risks

The most important remaining deployment requirements are below.

1. Meter every query and require payment

Before direct public exposure, the indexer needs per-query metering and payment enforcement so the index provider can bill usage.

Impact:

• Every query should be counted and accounted for.
• Expensive request paths should be covered by the metering model.
• Payment checks should gate access before event hydration and proof retrieval are served.
• Direct dapp access should remain tied to the provider’s billing policy.

Assessment:

• This is the primary remaining product requirement before treating the service as Internet-facing.

2. No in-process TLS

The service speaks plain WebSocket (ws) and plain HTTP for metrics.

Impact:

• Confidentiality and integrity depend on external deployment infrastructure.
• Direct exposure without TLS termination allows traffic observation and tampering in transit.
• The metrics endpoint has the same issue when exposed directly.

Assessment:

• Safe deployment requires TLS termination and normal edge protections outside the process.

3. Public expensive endpoints remain available

acuity_getEventMetadata, acuity_getEvents, event hydration, proof retrieval, and live subscriptions are still publicly callable.

Impact:

• Attackers can still consume CPU, RPC bandwidth, storage I/O, and upstream node capacity within the configured limits.
• Hydrated event reads are not purely local; they depend on live node access.
• Finalized proof inclusion adds extra upstream work.
• Current caps reduce per-connection blast radius but do not provide fairness across many clients or many source IPs.

Assessment:

• The server is substantially more resilient than an unbounded implementation.
• It is still vulnerable to sustained abuse, especially distributed abuse, because there is no built-in rate limiting or admission control by identity/network.

4. Metrics endpoint can leak operational metadata

If enabled, the separate OpenMetrics endpoint exposes operational state such as:

• RPC connectivity
• reconnect counts
• current indexed span
• latest seen head
• WebSocket connection count
• status and event subscription counts
• database size
• block fetch/process/commit timing histograms

Impact:

• Anything that can reach the metrics port can observe internal health and capacity signals.
• These signals may help attackers tune abuse or infer operational state.

Assessment:

• Lower severity than unauthenticated data access, but still an important exposure.
• The metrics port should be treated as an internal observability interface, not part of the public API.

5. Dependency maintenance risk remains material

The project still depends on some crates with maintenance or ecosystem advisories.

Impact:

• Even without a confirmed application-level exploit, these dependencies increase long-term supply-chain and maintenance risk.
• Some affected crates are in production dependency paths; others land through dev/test or optional light-client-related paths.

Assessment:

• This is primarily a patch-management and dependency-upgrade concern.
• The sled dependency remains a notable long-term risk because multiple advisories continue to land through it.

6. Some fail-fast paths remain intentional

The production runtime has moved many remotely reachable failure paths away from unwrap()-style crashes, but some startup-only or invariant-enforcement failures still intentionally stop the process.

Impact:

• Misconfiguration such as wrong genesis hash or pruned historical state still results in process exit.
• This is mainly an operability/reliability concern rather than a direct remote-exploitation path.

Assessment:

• Reasonable for invariant protection.
• Operators should still treat startup config validation and deployment checks as part of the security boundary.

Cargo Audit

cargo audit -q reported the following advisories during review:

1. RUSTSEC-2024-0388

   • dependency chain includes derivative 2.2.0
   • status: unmaintained

2. RUSTSEC-2025-0057

   • dependency chain: sled -> fxhash 0.2.1
   • status: unmaintained

3. RUSTSEC-2024-0384

   • dependency chain: sled -> parking_lot 0.11.2 -> instant 0.1.13
   • status: unmaintained

4. RUSTSEC-2025-0161

   • dependency chain includes libsecp256k1 0.7.2
   • status: unmaintained

5. RUSTSEC-2024-0436

   • dependency chain includes paste 1.0.15
   • status: unmaintained

Assessment:

• The sled advisories continue to indicate maintenance risk in the storage stack.
• Additional advisories now also land through Substrate / cryptography dependency chains.
• These findings do not by themselves prove a directly exploitable application bug in the WebSocket service, but they should be tracked and revisited during dependency updates.

Recommended Next Steps

Highest-value remaining work:

1. Add per-query metering for all public request paths.
2. Require cryptocurrency payment to the index provider before serving billable queries.
3. Define which request types are billable and enforce that policy before hydration or proof retrieval.
4. Continue to require TLS termination in documented deployment paths.
5. Keep the metrics endpoint internal-only by default in deployment examples.
6. Track or remediate cargo audit findings, especially long-term sled replacement/containment work and transitive cryptography dependency maintenance risk.
7. Continue expanding end-to-end tests around overload and backpressure behavior.

Deployment Guidance

For Internet exposure, deploy behind infrastructure that supports the billing and metering model:

• TLS termination
• request logging for query accounting
• per-query metering
• cryptocurrency payment enforcement at the access edge or application layer
• health checks
• internal-only exposure for the metrics port when enabled

Also assume the upstream Substrate node is part of the service envelope:

• keep RPC access restricted where possible
• run archival pruning settings required by the application
• monitor RPC availability separately from the public WebSocket service

Without metering and payment enforcement, the service should not be treated as ready for direct public dapp access.

Benchmarking

The repository includes an end-to-end benchmark harness for the synthetic runtime.

Run The Benchmark

just benchmark-indexing

This recipe:

1. builds the release binaries
2. starts a disposable synthetic node with timed blocks
3. bulk-seeds deterministic transactions
4. runs benchmark_synthetic_indexing
5. waits until seeded queries are observable through GetEvents
6. emits throughput and size metrics

Architecture

Acuity Index is a config-driven event indexer for Substrate chains. It decodes runtime events with subxt, derives query keys from TOML config, stores index entries in sled, and serves query access over WebSocket.

Main Components

• src/main.rs: CLI, config loading, database initialization, watcher setup, metrics listener, and reconnect supervisor loop
• src/indexer.rs: indexing pipeline, resume logic, live-head tailing, key derivation, notification fanout
• src/config.rs: index-spec and options-config parsing plus validation
• src/ws_api/: public API implementation and connection lifecycle
• src/protocol.rs: wire types, key encodings, tree layout, and WebSocket runtime limits
• src/runtime_state.rs: shared live process state across long-lived tasks
• src/event_hydration.rs: decoded-event hydration plus finalized System.Events proof fetching
• src/config_gen.rs: live metadata to starter spec generation
• src/metrics.rs: metrics registry and HTTP export
• src/synthetic_devnet.rs: synthetic local chain helpers and shared test types

Startup Sequence

The normal startup path is:

1. parse CLI args
2. load the required index spec
3. validate config and resolve runtime options
4. open sled
5. verify or initialize genesis_hash
6. start long-lived tasks such as WebSocket serving and spec watching
7. enter the RPC reconnect and indexing supervisor loop

Data Model

The sled database is organized into trees opened by Trees::open:

• root: database-level values such as genesis_hash
• span: indexed block spans for resume and reindex logic
• variant: event references keyed by pallet and variant indices
• index: custom and built-in query keys with (block_number, event_index) suffixes
• events: decoded event JSON keyed by (block_number, event_index)

Indexing Flow

High-level behavior:

• determine the starting head block
• load previously indexed spans
• resume an existing tail span or index the current head immediately
• run backward backfill and live-head tracking concurrently

If finalized mode is enabled, the same finalized-only setting governs whether API callers can receive proof material for GetEvents.

Concurrency Model

The indexer uses async queues for both descending backfill and ascending live head processing. queue_depth applies to both so the process can catch up on fast-moving chains instead of advancing one block at a time.

Because futures can complete out of order, the implementation keeps orphan maps until contiguous spans can be extended safely.

Invariants

• a database path belongs to exactly one chain identity
• accepted spec reloads should not take down the public service
• malformed persisted data should be handled defensively
• the synthetic test path should stay close to production architecture
• public API correctness is validated end to end, not only through unit tests

Contributing

Development Entry Points

The repository uses just as the main developer command surface:

just build
just test
just runtime-build
just synthetic-node
just seed-smoke
just test-integration
just benchmark-indexing

Code Areas

High-level module boundaries:

• src/main.rs: CLI, startup, supervisor loop, long-lived tasks
• src/indexer.rs: indexing pipeline and span tracking
• src/config.rs: TOML schema and runtime mapping resolution
• src/ws_api/: public WebSockets API
• src/protocol.rs: shared protocol types and database key layout
• src/runtime_state.rs: live shared runtime state
• src/metrics.rs: Prometheus/OpenMetrics integration
• src/config_gen.rs: starter index-spec generation from metadata
• src/synthetic_devnet.rs: synthetic local chain support
• runtime/: in-repo synthetic runtime used for integration testing and benchmarking

Suggested Workflow

1. run unit tests with just test
2. use the synthetic harness when your change affects observable runtime behavior
3. run ignored integration tests when changing indexing, API, or reconnect logic
4. benchmark with the synthetic harness when changing throughput-sensitive code
5. update the book when user-facing or operator-facing behavior changes

Testing Philosophy

Unit tests must be deterministic and must not rely on wall-clock timing. Behavior that depends on real node timing or networking belongs in integration tests instead.

Maintainer Release Process

acuity-index uses two separate release tools:

• cargo-release handles the crate release itself
• cargo-dist handles the GitHub Release artifacts
• GitHub Actions only reacts after a release tag has been pushed

In practice, that means the version bump and crates.io publish happen through standard Rust release tooling, while GitHub is used only to build and upload binary artifacts for the tagged release.

Prerequisites

The release machine must have:

• Rust stable
• cargo-release
• cargo-dist
• just
• polkadot-omni-node

The integration suite is part of the release gate, so the machine running the release must be able to build the in-repo synthetic runtime and run the local node-backed tests.

Release Gate

Before cargo-release is allowed to tag or publish, it runs:

cargo fmt --check
cargo clippy --all-targets --all-features -- -D warnings
cargo test
just test-integration

These checks are wired through scripts/release-checks.sh and are also exposed as:

just release-checks

If any step fails, the release stops before the version is bumped, before a tag is created, and before anything is published.

Updating The Changelog

Before running the release command, update the canonical repository changelog in CHANGELOG.md and commit it. The GitHub release workflow publishes the matching version section from that file as the release notes.

You can preview exactly what GitHub will publish with:

just release-notes
just release-notes v0.8.0

That helper reads CHANGELOG.md and extracts the section whose heading starts with ## v<version>.

The book no longer owns the release history; if you want a book-visible entry point, keep book/src/changelog.md as a short pointer back to CHANGELOG.md.

Creating A Release

Run one of:

just release patch
just release minor
just release major

Equivalent direct commands are:

cargo release patch --execute
cargo release minor --execute
cargo release major --execute

cargo-release is configured to:

1. require the main branch
2. run the release gate
3. bump the crate version in Cargo.toml
4. create a release commit with the message chore(release): <version>
5. create a git tag named v<version>
6. publish the crate to crates.io
7. push the release commit and tag

Dry runs are still available via plain cargo release <level> without --execute.

GitHub Artifacts

After cargo-release pushes the release tag, GitHub Actions runs the cargo-dist workflow that was generated from the dist configuration.

cargo-dist is not the thing that decides whether the release is allowed to happen. The integration tests and other release checks run locally before the tag exists.

The GitHub workflow only:

1. reads the pushed version tag
2. plans the dist build for that tag
3. builds the configured release artifacts
4. extracts the matching tagged section from CHANGELOG.md
5. creates or updates the GitHub Release with those notes
6. uploads the archives and checksum files

The current dist configuration builds release artifacts for:

• aarch64-apple-darwin
• aarch64-unknown-linux-gnu
• x86_64-apple-darwin
• x86_64-unknown-linux-gnu
• x86_64-pc-windows-msvc

Dist Configuration

cargo-dist is configured in dist-workspace.toml. That file tells dist what artifacts to build and how the release workflow should behave.

The repo uses a dedicated dist profile in Cargo.toml:

[profile.dist]
inherits = "release"
lto = "thin"

That keeps distribution builds optimized while avoiding the heavier fat LTO setting used by the normal release profile.

Recommended Maintainer Flow

For a normal patch release:

git switch main
git pull --ff-only
just release-checks
just release patch

After the tag is pushed:

• crates.io gets the new crate release from cargo-release
• GitHub Releases gets binary artifacts from cargo-dist

Publicise The Release

Once the release is published, share it in the places most likely to reach acuity-index users and contributors:

• the Polkadot forum or another main Polkadot/Substrate developer discussion channel
• the Polkadot Discord, Matrix, or Telegram developer channels
• a short X/Twitter post linking to the GitHub release

For larger releases, consider also posting to Hacker News (Show HN) and targeted Reddit communities. If the release is especially substantial, a short blog or book note can help with long-tail discovery and search visibility.

Regenerating Dist CI

If you change dist-workspace.toml or other dist-related release settings, regenerate the GitHub Actions workflow with:

dist generate --mode ci

This updates the generated workflow under .github/workflows/ so it matches the current dist configuration.

The generated workflow is intentionally tag-driven only. Do not move the integration suite into GitHub Actions; those checks are meant to stay in the local release gate.