# Distributed training

Log per-rank metrics from distributed jobs so the Distributed dashboard tab can show reducers, rank coverage, heatmaps, and outliers instead of one flattened scalar.

## Log per-rank metrics

Each call records one rank's metric batch for one global step.

```python
run.log_rank_metrics(
    {"train/loss": 0.12, "optimizer/grad_norm": 3.4},
    step=100,
    rank=0,
    world_size=8,
    local_rank=0,
    weight=1024,
)
```

| Argument | Meaning |
| --- | --- |
| `metrics` | Scalar metric dictionary for one rank. |
| `step` | Global step shared by all ranks. |
| `rank` | Zero-based global rank. |
| `world_size` | Expected number of ranks, capped at 512. |
| `local_rank` | Optional zero-based rank within a node. |
| `weight` | Optional sample count or work weight for weighted reducers. Defaults to `1.0`. |

> **Note:** Rank metric rows count toward the same monthly metric-point usage guardrail as ordinary scalar metric rows.

## Inspect ranks in the Distributed tab

The Distributed tab reads a run-scoped summary and renders:

- Mean and weighted mean.
- Min, max, range, and standard deviation.
- p05, p50, and p95 percentiles.
- Rank coverage per step.
- Heatmap cells.
- Outlier rows by z-score and delta from mean.

Use these views to answer questions like:

- Did one rank stop reporting?
- Is a rank consistently slower or noisier?
- Did a data shard or process produce outlier losses?
- Are weighted reducers different from unweighted reducers?

## Use spool mode for rank metrics

Process-isolated spool mode writes rank metric calls as JSONL-style request events and replays them with the event ID as `Idempotency-Key`.

```python
import instantml as im

run = im.init(
    project="distributed-job",
    upload_mode="spool",
    spool_dir=".instantml/spool",
)

run.log_rank_metrics(
    {"train/loss": loss},
    step=step,
    rank=rank,
    world_size=world_size,
    weight=batch_size,
)
```

This keeps post-init HTTP calls out of the training process while preserving retry safety for compatible servers.

## Choose which keys to log per rank

Log rank-aware metrics only for the keys that help debug distributed behavior. For example, log ordinary `train/loss` and `eval/accuracy` from rank 0 or a trainer process, then log rank-aware `train/loss`, `optimizer/grad_norm`, and `system/gpu_memory_mb` from all ranks.

That keeps scalar comparison fast while still giving the Distributed tab enough information to detect coverage and outlier problems.

## Next steps

- [Log metrics and steps](/docs/sdk/metrics-steps.md)
- [Buffering, offline mode, and reliability](/docs/sdk/reliability.md)
- [SDK patterns by workflow](/docs/sdk/examples-patterns.md)
- [Usage limits and billing](/docs/guides/pricing-limits-billing.md)

## Agent navigation

- [Docs index](/llms.txt)
- [Full docs bundle](/llms-full.txt)

### Get Started

- [Overview](/docs/index.md)
- [Quickstart](/docs/quickstart.md)
- [Core Concepts](/docs/concepts/core-concepts.md)
- [Pricing](/docs/pricing.md)
- [Benchmarks](/docs/benchmarks.md)
- [Examples](/docs/guides/examples.md)
- [Troubleshooting](/docs/troubleshooting.md)

### SDK

- [Installation Auth](/docs/sdk/installation-auth.md)
- [Logging](/docs/sdk/logging.md)
- [Tracing](/docs/sdk/tracing.md)
- [Metrics Steps](/docs/sdk/metrics-steps.md)
- [Config Tags Notes](/docs/sdk/config-tags-notes.md)
- [Artifacts Checkpoints](/docs/sdk/artifacts-checkpoints.md)
- [Rich Objects](/docs/sdk/rich-objects.md)
- [Distributed Training](/docs/sdk/distributed-training.md) (current page)
- [Console System Integrations](/docs/sdk/console-system-integrations.md)
- [Reliability](/docs/sdk/reliability.md)
- [Cli Login](/docs/sdk/cli-login.md)
- [Querying Data](/docs/sdk/querying-data.md)
- [Agent Mcp](/docs/sdk/agent-mcp.md)
- [Examples Patterns](/docs/sdk/examples-patterns.md)

### Integrations

- [Overview](/docs/integrations/overview.md)
- [Pytorch Lightning](/docs/integrations/pytorch-lightning.md)
- [Huggingface Transformers](/docs/integrations/huggingface-transformers.md)
- [Keras](/docs/integrations/keras.md)
- [Wandb](/docs/integrations/wandb.md)

### Dashboard

- [Tour](/docs/dashboard/tour.md)
- [Organizations Workspaces](/docs/dashboard/organizations-workspaces.md)
- [Runs Workspace](/docs/dashboard/runs-workspace.md)
- [Metrics Charts](/docs/dashboard/metrics-charts.md)
- [Run Detail](/docs/dashboard/run-detail.md)
- [Traces](/docs/dashboard/traces.md)
- [Compare Runs](/docs/dashboard/compare-runs.md)
- [Research Dashboards](/docs/dashboard/research-dashboards.md)
- [Artifacts Files](/docs/dashboard/artifacts-files.md)
- [Run Health](/docs/dashboard/alerts.md)
- [Datasets](/docs/dashboard/datasets.md)
- [Checkpoints](/docs/dashboard/checkpoints.md)
- [Reports](/docs/dashboard/reports.md)
- [Settings Api Keys](/docs/dashboard/settings-api-keys.md)
- [Api Tab](/docs/dashboard/api-tab.md)
- [Onboarding Team Billing](/docs/dashboard/onboarding-team-billing.md)

### Data

- [Imports](/docs/guides/imports.md)
- [W&B alternative](/docs/guides/wandb-alternative.md)
- [InstantML vs MLflow](/docs/guides/instantml-vs-mlflow.md)
- [W&B import guide](/docs/guides/wandb-import-guide.md)
- [W&B and Neptune imports](/docs/guides/wandb-neptune-imports.md)
- [Export Usage Limits](/docs/guides/export-usage-limits.md)
- [Pricing Limits Billing](/docs/guides/pricing-limits-billing.md)
- [Auth Billing Storage](/docs/guides/auth-billing-storage.md)
- [Customer Owned Clickhouse](/docs/guides/customer-owned-clickhouse.md)
- [Observability](/docs/guides/observability.md)

### API

**Practical API guides**

- [Authentication](/docs/api/authentication.md)
- [Errors And Limits](/docs/api/errors-and-limits.md)
- [Health Observability](/docs/api/health-observability.md)
- [Projects Runs](/docs/api/projects-runs.md)
- [Metrics Series](/docs/api/metrics-series.md)
- [Attributes Objects](/docs/api/attributes-objects.md)
- [Artifacts](/docs/api/artifacts.md)
- [Reports](/docs/api/reports.md)
- [Iframe Embeds](/docs/api/iframe-embeds.md)
- [Import Export Usage](/docs/api/import-export-usage.md)
- [Dashboard Control State](/docs/api/dashboard-control-state.md)
**Architecture**

- [System Overview](/docs/architecture/system-overview.md)
- [Service Planes](/docs/architecture/service-planes.md)
- [Storage Model](/docs/architecture/storage-model.md)
- [Google Clickhouse](/docs/architecture/google-clickhouse.md)
- [Auth Tenancy](/docs/architecture/auth-tenancy.md)
- [Schema Reference](/docs/architecture/schema-reference.md)

### API Reference

- [API Reference](/docs/api-reference.md)