Coppelius training on the DGX Spark.

Prompt probe

What the model says next

A predict invocation of a saved checkpoint on a context whose text ends with the prompt at rows 255, 511 and 1023; each row gives the model's most likely next token after a different stretch of preceding text. Probes run between training runs: the GPU runs one invocation at a time.

Loss

Cross-entropy at every AdamW step

Mean over the 1,024 rows of each step's context; the band is the 10th–90th percentile across rows. The first step of an invocation is scored on a context the model has not trained on (dots), so its fall is learning rather than memorization. Dashed: ln 12,288 = 9.416, a model that knows nothing.

Invocation inspector

One invocation, every field

Device timing

On the GPU's own clock

Every submission's pre-launch and fence semaphore releases carry GB10 timestamps. Submission 333 is the forward pass, 334–343 the ten AdamW updates, 344 the forward pass after them; the rest move the 692 MB checkpoint in and out.

Optimizer internals

The token embedding, read back from the device

L2 norms of the first 1,024 scalars of the embedding's last-step gradient, its parameters and AdamW's first moment after each invocation, and a count of non-finite values in those windows.

GPU telemetry

nvidia-smi, per minute

Sampled every second; line: the minute's average, band: its range. The GB10 shares the host's memory, so it reports no frame-buffer figure.

Host telemetry

The Grace CPU running the AArch64 host

Channel health

Every invocation's exit, notifier and numerics

One cell per planned invocation: exit 0 with a clear error notifier, an all-zero host status block and a finite loss; a failure or a non-finite loss; pending.

Checkpoint lineage

Every invocation, newest first

Each invocation resumes the checkpoint the one before wrote. Predictions are the argmax at rows 255, 511 and 1023 on the state the invocation started from; truth is the real next token.