# Observability

Debug InstantML with its health endpoints, Prometheus-compatible metrics, and structured request logs — all designed so support can trace incidents without leaking user payloads.

## Check health endpoints

| Endpoint | Meaning |
| --- | --- |
| `GET /health` | Process is alive. |
| `GET /healthz` | Alias for health checks. |
| `GET /readyz` | ClickHouse stores are reachable and the control projection has loaded. |
| `GET /metrics` | Prometheus text metrics. |
| `GET /openapi.json` | Compact OpenAPI route index for the active service plane. |

`/readyz` fails until startup replay has completed. In hosted mode, later control refresh failures can keep serving the last-known-good projection while surfacing degraded state in `/readyz` and `/metrics`.

## Correlate requests with request IDs

Every response should include `x-request-id`. Keep this value with bug reports, screenshots, and support tickets.

Hosted Rust origin logs include:

- Request ID.
- Observed Cloudflare Ray ID when present.
- Method.
- Path without query string.
- Status.
- Latency.
- Service plane.
- Coarse user-agent family.

> **Note:** Cloudflare Ray IDs are edge correlation hints, not unique application request IDs. Pair them with timestamp, host, path, status, and `x-request-id`.

## Know what logs can and cannot contain

Server-error logs include stable fields such as status, code, retryability, and a safe summary. Workflow logs cover metric ingestion, console-log ingestion, artifact upload/download, imports, readiness, startup, and worker cleanup.

Logs may include stable product IDs:

- `org_id`
- `project_id`
- `run_id`
- `artifact_id`
- `import_id`

Logs must not include:

- Bearer tokens.
- Session IDs.
- API-key plaintext.
- Cookies.
- User emails.
- Query strings.
- Request bodies.
- Project or run names.
- Metric keys or values.
- Console messages.
- Artifact filenames.
- Object-storage bucket names, keys, or signed URLs.

## Configure hosted logging

Recommended hosted defaults:

```text
INSTANTML_LOG_FORMAT=json
RUST_LOG=instantml_rust_server=info,tower_http=info
INSTANTML_SLOW_REQUEST_MS=1000
```

Use debug logging only for a short incident window and confirm that provider errors and request payloads remain sanitized.

## Recognize a waking warehouse

When a tenant warehouse is waking or temporarily unavailable, the API can return HTTP `503` with:

```json
{
  "error": "data warehouse unavailable",
  "code": "warehouse_unavailable"
}
```

The dashboard treats that as "API reachable, data warehouse starting" rather than a generic offline state.

## Next steps

- [Health and observability API](/docs/api/health-observability.md)
- [Errors and limits](/docs/api/errors-and-limits.md)
- [Troubleshooting](/docs/troubleshooting.md)
- [Service planes](/docs/architecture/service-planes.md)

## Agent navigation

- [Docs index](/llms.txt)
- [Full docs bundle](/llms-full.txt)

### Get Started

- [Overview](/docs/index.md)
- [Quickstart](/docs/quickstart.md)
- [Core Concepts](/docs/concepts/core-concepts.md)
- [Pricing](/docs/pricing.md)
- [Benchmarks](/docs/benchmarks.md)
- [Examples](/docs/guides/examples.md)
- [Troubleshooting](/docs/troubleshooting.md)

### SDK

- [Installation Auth](/docs/sdk/installation-auth.md)
- [Logging](/docs/sdk/logging.md)
- [Tracing](/docs/sdk/tracing.md)
- [Metrics Steps](/docs/sdk/metrics-steps.md)
- [Config Tags Notes](/docs/sdk/config-tags-notes.md)
- [Artifacts Checkpoints](/docs/sdk/artifacts-checkpoints.md)
- [Rich Objects](/docs/sdk/rich-objects.md)
- [Distributed Training](/docs/sdk/distributed-training.md)
- [Console System Integrations](/docs/sdk/console-system-integrations.md)
- [Reliability](/docs/sdk/reliability.md)
- [Cli Login](/docs/sdk/cli-login.md)
- [Querying Data](/docs/sdk/querying-data.md)
- [Agent Mcp](/docs/sdk/agent-mcp.md)
- [Examples Patterns](/docs/sdk/examples-patterns.md)

### Integrations

- [Overview](/docs/integrations/overview.md)
- [Pytorch Lightning](/docs/integrations/pytorch-lightning.md)
- [Huggingface Transformers](/docs/integrations/huggingface-transformers.md)
- [Keras](/docs/integrations/keras.md)
- [Wandb](/docs/integrations/wandb.md)

### Dashboard

- [Tour](/docs/dashboard/tour.md)
- [Organizations Workspaces](/docs/dashboard/organizations-workspaces.md)
- [Runs Workspace](/docs/dashboard/runs-workspace.md)
- [Metrics Charts](/docs/dashboard/metrics-charts.md)
- [Run Detail](/docs/dashboard/run-detail.md)
- [Traces](/docs/dashboard/traces.md)
- [Compare Runs](/docs/dashboard/compare-runs.md)
- [Research Dashboards](/docs/dashboard/research-dashboards.md)
- [Artifacts Files](/docs/dashboard/artifacts-files.md)
- [Run Health](/docs/dashboard/alerts.md)
- [Datasets](/docs/dashboard/datasets.md)
- [Checkpoints](/docs/dashboard/checkpoints.md)
- [Reports](/docs/dashboard/reports.md)
- [Settings Api Keys](/docs/dashboard/settings-api-keys.md)
- [Api Tab](/docs/dashboard/api-tab.md)
- [Onboarding Team Billing](/docs/dashboard/onboarding-team-billing.md)

### Data

- [Imports](/docs/guides/imports.md)
- [W&B alternative](/docs/guides/wandb-alternative.md)
- [InstantML vs MLflow](/docs/guides/instantml-vs-mlflow.md)
- [W&B import guide](/docs/guides/wandb-import-guide.md)
- [W&B and Neptune imports](/docs/guides/wandb-neptune-imports.md)
- [Export Usage Limits](/docs/guides/export-usage-limits.md)
- [Pricing Limits Billing](/docs/guides/pricing-limits-billing.md)
- [Auth Billing Storage](/docs/guides/auth-billing-storage.md)
- [Customer Owned Clickhouse](/docs/guides/customer-owned-clickhouse.md)
- [Observability](/docs/guides/observability.md) (current page)

### API

**Practical API guides**

- [Authentication](/docs/api/authentication.md)
- [Errors And Limits](/docs/api/errors-and-limits.md)
- [Health Observability](/docs/api/health-observability.md)
- [Projects Runs](/docs/api/projects-runs.md)
- [Metrics Series](/docs/api/metrics-series.md)
- [Attributes Objects](/docs/api/attributes-objects.md)
- [Artifacts](/docs/api/artifacts.md)
- [Reports](/docs/api/reports.md)
- [Iframe Embeds](/docs/api/iframe-embeds.md)
- [Import Export Usage](/docs/api/import-export-usage.md)
- [Dashboard Control State](/docs/api/dashboard-control-state.md)
**Architecture**

- [System Overview](/docs/architecture/system-overview.md)
- [Service Planes](/docs/architecture/service-planes.md)
- [Storage Model](/docs/architecture/storage-model.md)
- [Google Clickhouse](/docs/architecture/google-clickhouse.md)
- [Auth Tenancy](/docs/architecture/auth-tenancy.md)
- [Schema Reference](/docs/architecture/schema-reference.md)

### API Reference

- [API Reference](/docs/api-reference.md)