# Observability \[Logs, metrics, and indexing status] ## Logs Ponder produces logs to help you understand and debug your application. ![Terminal logs gif](/logs-014.gif) ### Log level There are two ways to configure the minimum log level. If specified, the environment variable takes precedence over the CLI flag. * Set the `PONDER_LOG_LEVEL` environment variable * Use the `--log-level `, `-v` (debug) or `-vv` (trace) CLI option ```js [.env.local] PONDER_LOG_LEVEL=trace ``` ```bash [Terminal] ponder dev --log-level warn # or, use the shortcut flag for debug ponder dev -v ``` #### Levels | Log level | Example | | :--------------- | :------------------------------------------------------------------------- | | `silent` | | | `error` | Errors thrown in user code and other errors that will likely cause a crash | | `warn` | Malformed RPC data, reorgs, and other errors that will be retried | | `info` (default) | Indexing progress and key lifecycle events | | `debug` | Internal updates | | `trace` | Database query logs, RPC request logs | #### User logs Logs produced by your code (e.g. `console.log` statements in `ponder.config.ts` or indexing functions) will always be written to the console. Note that Ponder *does* catch **errors** thrown by your code and emits an `error` log including the original error message and stack trace. ### Log format Use the `--log-format ` CLI option to set the log format. #### Pretty (default) ```bash [Terminal] ponder start --log-format pretty ``` ```bash [Output] 12:12:15.391 INFO Indexed block chain=mainnet number=23569900 event_count=14 (23ms) 12:12:16.159 INFO Indexed block chain=polygon number=77633630 event_count=0 (1ms) 12:12:16.174 INFO Indexed block chain=optimism number=142386579 event_count=1 (4ms) 12:12:16.226 INFO Indexed block chain=base number=36791294 event_count=9 (14ms) 12:12:18.068 INFO Indexed block chain=optimism number=142386580 event_count=2 (8ms) 12:12:18.125 INFO Indexed block chain=polygon number=77633631 event_count=0 (1ms) 12:12:18.188 INFO Indexed block chain=base number=36791295 event_count=10 (16ms) 12:12:20.021 INFO Indexed block chain=optimism number=142386581 event_count=0 (4ms) ``` #### JSON ```bash [Terminal] ponder start --log-format json ``` The JSON log format emits newline-delimited JSON objects with required properties `level`, `time`, and `msg`. Most logs also include a `duration` property and other properties depending on the context. ```json [Output] {"level":30,"time":1760372079306,"msg":"Indexed block","chain":"mainnet","chain_id":1,"number":23569912,"event_count":17,"duration":27.752416999996058} {"level":30,"time":1760372080106,"msg":"Indexed block","chain":"polygon","chain_id":137,"number":77633702,"event_count":0,"duration":3.4684160000033444} {"level":30,"time":1760372080122,"msg":"Indexed block","chain":"optimism","chain_id":10,"number":142386651,"event_count":0,"duration":2.3179999999993015} {"level":30,"time":1760372080314,"msg":"Indexed block","chain":"base","chain_id":8453,"number":36791366,"event_count":10,"duration":18.320999999996275} {"level":30,"time":1760372082131,"msg":"Indexed block","chain":"optimism","chain_id":10,"number":142386652,"event_count":0,"duration":3.074124999999185} {"level":30,"time":1760372082258,"msg":"Indexed block","chain":"polygon","chain_id":137,"number":77633703,"event_count":0,"duration":1.7850829999952111} {"level":30,"time":1760372082328,"msg":"Indexed block","chain":"base","chain_id":8453,"number":36791367,"event_count":4,"duration":9.394625000000815} {"level":30,"time":1760372084153,"msg":"Indexed block","chain":"optimism","chain_id":10,"number":142386653,"event_count":0,"duration":2.679999999993015} ``` ### Terminal UI The dynamic terminal UI displays a useful summary of chain connection status, indexing function duration, and backfill progress. ```bash [Output] Chains │ Chain │ Status │ Block │ RPC (req/s) │ ├──────────┼────────┼───────────┼─────────────┤ │ optimism │ live │ 142388578 │ 1.6 │ │ polygon │ live │ 77635629 │ 17.2 │ │ base │ live │ 36793293 │ 4.2 │ │ mainnet │ live │ 23570232 │ 7.5 │ Indexing (live) │ Event │ Count │ Duration (ms) │ ├───────────────┼───────┼───────────────┤ │ WETH:Deposit │ 107 │ 1.554 │ API endpoints Live at http://localhost:42069 ``` The terminal UI is disabled by default for `ponder start`. Use the `--disable-ui` CLI option to disable the UI for `ponder dev`. ```bash [Terminal] ponder dev --disable-ui ``` ## Metrics Ponder apps publish Prometheus metrics at the `/metrics` path. :::warning Metrics are not part of the public API, so these are subject to change without notice. Do not rely on these metrics for anything important (yet). ::: | name | description | type | | :------------------------------------- | :-------------------------------------------------------------------- | --------- | | ponder\_indexing\_total\_seconds | Total number of seconds required for indexing | gauge | | ponder\_indexing\_completed\_seconds | Number of seconds that have been completed | gauge | | ponder\_indexing\_completed\_events | Number of events that have been processed | gauge | | ponder\_indexing\_completed\_timestamp | Timestamp through which all events have been completed | gauge | | ponder\_indexing\_function\_duration | Duration of indexing function execution | histogram | | ponder\_indexing\_function\_error\_total | Total number of errors encountered during indexing function execution | counter | | ponder\_historical\_start\_timestamp | Unix timestamp (ms) when the historical sync service started | gauge | | ponder\_historical\_total\_blocks | Number of blocks required for the historical sync | gauge | | ponder\_historical\_cached\_blocks | Number of blocks that were found in the cache for the historical sync | gauge | | ponder\_historical\_completed\_blocks | Number of blocks that have been processed for the historical sync | gauge | | ponder\_realtime\_is\_connected | Boolean (0 or 1) indicating if the realtime sync service is connected | gauge | | ponder\_realtime\_latest\_block\_number | Block number of the latest synced block | gauge | | ponder\_realtime\_latest\_block\_timestamp | Block timestamp of the latest synced block | gauge | | ponder\_realtime\_reorg\_total | Count of how many re-orgs have occurred | counter | | ponder\_database\_method\_duration | Duration of database operations | histogram | | ponder\_database\_method\_error\_total | Total number of errors encountered during database operations | counter | | ponder\_http\_server\_active\_requests | Number of active HTTP server requests | gauge | | ponder\_http\_server\_request\_duration\_ms | Duration of HTTP responses served by the server | histogram | | ponder\_http\_server\_request\_size\_bytes | Size of HTTP requests received by the server | histogram | | ponder\_http\_server\_response\_size\_bytes | Size of HTTP responses served by the server | histogram | | ponder\_rpc\_request\_duration | Duration of RPC requests | histogram | | ponder\_rpc\_request\_error\_total | Total number of failed RPC requests | counter | | ponder\_postgres\_pool\_connections | Gauge of current connections for PostgreSQL pools | gauge | | ponder\_postgres\_query\_queue\_size | Current size of the query queue for PostgreSQL | gauge | | ponder\_postgres\_query\_total | Total number of queries processed by PostgreSQL | counter | ## Indexing status To check the indexing status of your app, use the `/status` endpoint or the `_meta` field in the GraphQL API. ### Usage Use the indexing status to quickly confirm that Ponder is working as expected. You can also poll the status to confirm that a specific block number has been ingested by Ponder before refetching a query client-side (for example, in a form submit handler). #### HTTP ```bash [Request] curl http://localhost:42069/status ``` ```json [Response] { "mainnet": { "id": 1, "block": { "number": 20293450, "timestamp": 1720823759 } }, "base": { "id": 8453, "block": { "number": 17017206, "timestamp": 1720823759 } } } ``` #### GraphQL ```graphql [Query] query { _meta { status } } ``` ```json [Result] { "_meta": { "status": { "mainnet": { "id": 1, "block": { "number": 20293464, "timestamp": 1720823939 } }, "base": { "id": 8453, "block": null } } } } ``` ### API The response object contains a property for each chain in your app with the following fields. | field | type | description | | :-------- | :----------------------------------------------: | :-------------------------------------------------------------------------- | | **id** | `number` | The chain ID. | | **block** | `{ number: number; timestamp: number; } \| null` | The most recently indexed block, or `null` if the backfill is not complete. |