# System overview InstantML has four moving parts: SDKs write training-loop events, a Rust API validates and stores them, ClickHouse holds operational records and metric history, and the dashboard reads those records back for run comparison. > **Info:** A **workspace** is the top-level tenant boundary. The API and billing surfaces call the same boundary an **organization** (`org_id`, org-scoped API keys). ## Primary product path | Producer | API boundary | Storage path | | --- | --- | --- | | Python SDK / uploader | Rust API | ClickHouse operational records | | Dashboard (Next.js) | Rust API | ClickHouse metric tables | | SDK and dashboard artifact flows | Rust API | Artifact byte storage | The Rust API is the single backend boundary. It owns the route contract for SDK ingestion, dashboard reads, imports, exports, usage, billing gates, and artifact metadata. Neither the browser nor the SDK ever talks to ClickHouse or object storage directly. ## Component responsibilities | Component | Responsibility | | --- | --- | | Python SDK | Create runs, log metrics, console logs, attributes, artifacts, rich objects, checkpoints, and source context. | | Uploader | Drain process-isolated spool events from disk and replay them through the API. | | Rust API | Validate requests, enforce auth/scopes/limits, store records, stream artifact downloads, and expose dashboard/API routes. | | Control-plane Postgres | Store account records: users, organizations, memberships, sessions, API keys, billing, invitations, tenant routes, dashboard preferences, and saved workspace views. Single-binary local mode keeps these records in ClickHouse instead. | | ClickHouse operational layer | Store low-volume append-only product records (projects, runs, attributes, artifacts, reports, imports). | | ClickHouse metric layer | Store raw scalar and per-rank metric points and maintained metric summaries. | | Artifact store | Store artifact bytes in private object storage (hosted) or on local disk (self-hosted single binary). | | Dashboard and docs (Next.js) | Browse, chart, compare, annotate, export, administer workspaces, and serve this documentation. | ## Hosted topology The hosted API is one public HTTPS URL backed by two service roles. | Layer | Role | | --- | --- | | Browser / dashboard and Python SDK / uploader | Send all product traffic to the public HTTPS API. | | Public HTTPS API | Routes each request to the control or data service. | | Control service | Reads and writes the control-plane Postgres database for account, billing, and tenant-route records. | | Data service | Reads and writes tenant ClickHouse databases and private artifact buckets. | The control service handles auth, workspaces, sessions, API keys, billing, seats, invitations, tenant-route metadata, dashboard preferences, and saved workspace views. The data service handles projects, runs, metrics, console logs, attributes, rich objects, artifacts, reports, imports, export, and usage. Tenant run data and metrics live in InstantML-owned ClickHouse on Google Cloud; see [Service planes](/docs/architecture/service-planes.md) for the full route split. ## Design constraints you can rely on - List endpoints are bounded; no endpoint returns unbounded history. - Raw metric history is fetched through dedicated series endpoints with explicit limits. - Metric summaries are maintained during ingestion, so run tables never scan full metric history. - Artifact byte uploads and downloads are separate paths from scalar metric ingestion. - Every API key resolves to exactly one workspace, with an optional project scope. - Exports are explicit, bounded, and report truncation. ## Next steps - [Service planes](/docs/architecture/service-planes.md) - [Storage model](/docs/architecture/storage-model.md) - [Auth and tenancy](/docs/architecture/auth-tenancy.md) - [Schema reference](/docs/architecture/schema-reference.md) - [Benchmarks](/docs/benchmarks.md) ## Agent navigation - [Docs index](/llms.txt) - [Full docs bundle](/llms-full.txt) ### Get Started - [Overview](/docs/index.md) - [Quickstart](/docs/quickstart.md) - [Core Concepts](/docs/concepts/core-concepts.md) - [Pricing](/docs/pricing.md) - [Benchmarks](/docs/benchmarks.md) - [Examples](/docs/guides/examples.md) - [Troubleshooting](/docs/troubleshooting.md) ### SDK - [Installation Auth](/docs/sdk/installation-auth.md) - [Logging](/docs/sdk/logging.md) - [Tracing](/docs/sdk/tracing.md) - [Metrics Steps](/docs/sdk/metrics-steps.md) - [Config Tags Notes](/docs/sdk/config-tags-notes.md) - [Artifacts Checkpoints](/docs/sdk/artifacts-checkpoints.md) - [Rich Objects](/docs/sdk/rich-objects.md) - [Distributed Training](/docs/sdk/distributed-training.md) - [Console System Integrations](/docs/sdk/console-system-integrations.md) - [Reliability](/docs/sdk/reliability.md) - [Cli Login](/docs/sdk/cli-login.md) - [Querying Data](/docs/sdk/querying-data.md) - [Agent Mcp](/docs/sdk/agent-mcp.md) - [Examples Patterns](/docs/sdk/examples-patterns.md) ### Integrations - [Overview](/docs/integrations/overview.md) - [Pytorch Lightning](/docs/integrations/pytorch-lightning.md) - [Huggingface Transformers](/docs/integrations/huggingface-transformers.md) - [Keras](/docs/integrations/keras.md) - [Wandb](/docs/integrations/wandb.md) ### Dashboard - [Tour](/docs/dashboard/tour.md) - [Organizations Workspaces](/docs/dashboard/organizations-workspaces.md) - [Runs Workspace](/docs/dashboard/runs-workspace.md) - [Metrics Charts](/docs/dashboard/metrics-charts.md) - [Run Detail](/docs/dashboard/run-detail.md) - [Traces](/docs/dashboard/traces.md) - [Compare Runs](/docs/dashboard/compare-runs.md) - [Research Dashboards](/docs/dashboard/research-dashboards.md) - [Artifacts Files](/docs/dashboard/artifacts-files.md) - [Run Health](/docs/dashboard/alerts.md) - [Datasets](/docs/dashboard/datasets.md) - [Checkpoints](/docs/dashboard/checkpoints.md) - [Reports](/docs/dashboard/reports.md) - [Settings Api Keys](/docs/dashboard/settings-api-keys.md) - [Api Tab](/docs/dashboard/api-tab.md) - [Onboarding Team Billing](/docs/dashboard/onboarding-team-billing.md) ### Data - [Imports](/docs/guides/imports.md) - [W&B alternative](/docs/guides/wandb-alternative.md) - [InstantML vs MLflow](/docs/guides/instantml-vs-mlflow.md) - [W&B import guide](/docs/guides/wandb-import-guide.md) - [W&B and Neptune imports](/docs/guides/wandb-neptune-imports.md) - [Export Usage Limits](/docs/guides/export-usage-limits.md) - [Pricing Limits Billing](/docs/guides/pricing-limits-billing.md) - [Auth Billing Storage](/docs/guides/auth-billing-storage.md) - [Customer Owned Clickhouse](/docs/guides/customer-owned-clickhouse.md) - [Observability](/docs/guides/observability.md) ### API **Practical API guides** - [Authentication](/docs/api/authentication.md) - [Errors And Limits](/docs/api/errors-and-limits.md) - [Health Observability](/docs/api/health-observability.md) - [Projects Runs](/docs/api/projects-runs.md) - [Metrics Series](/docs/api/metrics-series.md) - [Attributes Objects](/docs/api/attributes-objects.md) - [Artifacts](/docs/api/artifacts.md) - [Reports](/docs/api/reports.md) - [Iframe Embeds](/docs/api/iframe-embeds.md) - [Import Export Usage](/docs/api/import-export-usage.md) - [Dashboard Control State](/docs/api/dashboard-control-state.md) **Architecture** - [System Overview](/docs/architecture/system-overview.md) (current page) - [Service Planes](/docs/architecture/service-planes.md) - [Storage Model](/docs/architecture/storage-model.md) - [Google Clickhouse](/docs/architecture/google-clickhouse.md) - [Auth Tenancy](/docs/architecture/auth-tenancy.md) - [Schema Reference](/docs/architecture/schema-reference.md) ### API Reference - [API Reference](/docs/api-reference.md)