How Alpha learning systems are expected to work

Alpha is not meant to be a framework with one blessed model hidden behind a configuration file. It is meant to be a laboratory in which a learning system is a checked program: the model, learning rule, data path, optimizer, schedule, memory plan, target requirements, and evidence contract can all be changed deliberately.

That distinction matters in the agentic era. An agent should be able to propose an unfamiliar mechanism, run the smallest experiment that could disprove it, inspect exactly what reached the machine, and retain the result. It should not have to translate every idea into the assumptions of PyTorch, CUDA, a Transformer class, or a vendor kernel before it can learn whether the idea works.

What is real now, and what remains a direction

The current evidence is recorded in the repository goal. The larger intended surface is described in the execution PRD. Their roles differ: the goal says what has been demonstrated; the PRD says what the compiler is being built to make possible.

The experiment loop

The expected loop is:

question
  -> typed learning hypothesis
  -> checked model and learning program
  -> target-independent operators
  -> selected realizations and schedule
  -> proven resource and capability requirements
  -> native CPU/GPU artifact
  -> physical run and telemetry
  -> checkpoint, identities, and receipt
  -> compare, reject, refine, or retain

The output is therefore not just a weight file. A useful result binds together:

• the exact source and compiler identities;
• model shape, initialization, RNG, tokenizer, dataset, and cursor identities;
• the selected learning transformation and optimizer state;
• the operator realizations, placement, schedule, and memory plan;
• the target profile and emitted artifact identities;
• finite losses, advancing counters, resource observations, and device evidence; and
• a checkpoint that validates, reloads, and continues.

That receipt lets another agent distinguish a promising idea from an accidental fallback, a stale artifact, a host-side calculation, or a run that merely launched.

What the compiler is meant to keep separate

Many machine-learning stacks collapse several decisions into a tensor program. Alpha instead treats them as related but different layers.

Model semantics

The model says what state exists and what mathematical relationship produces an output. A Transformer can be expressed here, but Transformer is not the definition of a model.

Learning semantics

The learning program says how parameters or state change. The design calls for reverse-, forward-, and mixed-mode differentiation; custom VJP/JVP rules; implicit differentiation; local and multi-phase learning; predictive-coding and equilibrium procedures; perturbation and population methods; and nested learners. These are roadmap categories, not a claim that each path is physically mature today.

Operators

The foundational abstraction is a typed operator, not matrix multiplication. Maps, folds, scans, recurrence, state-space transitions, graphs, event streams, differential-equation solvers, stochastic operators, and alternative accumulation algebras can all be first-class. GEMM and attention become library constructions over that layer.

Realizations

A realization is one concrete way to execute an operator on a target. A dense operator might have a scalar reference realization, an SM86 realization, and a deliberately strange multiplication-light experiment. Selecting one must not silently change the model's declared meaning.

Schedule and memory

Tiling, fusion, recomputation, asynchronous copies, shared-memory allocation, and placement introduce requirements. Alpha's intended contract makes those requirements explicit and asks the target to prove both that it supports them and that the plan fits. Runtime shape drift or unsupported hardware should fail before submission.

Evidence

Compilation, execution, and quality are different questions. A program can check yet lack a native lowering. An ELF can be emitted yet fail on a physical device. A training run can complete yet learn nothing useful. Alpha keeps those states separate so an agent cannot promote one into another by rhetoric.

The conditional-explosion problem

Model-training code commonly accumulates configuration branches until the program is difficult to reason about:

if precision is fp16 ...
if sequence length exceeds this kernel's limit ...
if gradient accumulation is enabled ...
if this GPU has enough shared memory ...
if this instruction exists on this device ...
if the batch no longer fits ...
if the fast attention path is available ...
else silently use another implementation ...

Those decisions are not independent. Changing one “hyperparameter” can change tensor extents, activation memory, the number of optimizer updates per token, accumulation order, numerical stability, kernel launch geometry, register pressure, shared-memory use, checkpoint shape, and which hardware realization is legal. A batch-size change that looks local in a configuration file can therefore produce an out-of-memory failure or an unsupported kernel only after the run reaches a particular device.

Hardware multiplies the cases. VRAM, instruction sets, supported numeric formats, warp geometry, register files, shared-memory limits, alignment rules, driver protocols, and synchronization facilities differ. In a runtime-branching stack, the actual training program is effectively assembled while it is already running. Many combinations are rarely exercised, and an agent changing one value cannot easily know which hidden path changed with it.

Alpha is trying to move as much of that decision surface as possible out of the training loop and into compilation.

Configuration should produce a program, not steer one

The intended flow treats a model configuration, learning configuration, target profile, and resource budget as inputs to the compiler:

semantic model + learning rule + declared experiment
  + closed hyperparameters
  + target capabilities and limits
  -> specialized shapes and schemas
  -> selected operator realizations
  -> schedule and liveness
  -> exact static memory plan
  -> target fit proof or compile-time rejection
  -> one target-specific training artifact

Closed choices can be normalized away. A precision, head count, width, vocabulary, context length, microbatch count, accumulation count, optimizer choice, or target feature that is fixed for an experiment should not remain as an if tested in every step. It should specialize the artifact. Proofs and configuration-only evidence can then be erased, leaving only the machine work required by that exact experiment.

This does not mean one universal binary must run every configuration on every device. It means one semantic learning system can be compiled into several explicit artifacts, each valid for a declared configuration and target. An RTX 3070 artifact and a different-device artifact may use different tiling, memory layouts, or operator realizations without duplicating the model's meaning or hiding a fallback inside the runtime.

What should happen when one hyperparameter changes

Suppose an agent changes context length, batch size, model width, precision, learning rate, or gradient accumulation.

The desired response is mechanical:

1. Recompute the derived shapes, schemas, numerical contracts, and update semantics affected by that value.
2. Re-plan lifetimes and exact peak memory rather than waiting for an allocator to fail.
3. Re-evaluate each selected realization's target capabilities and resources.
4. Recompile only the invalidated dependency cone; reuse interfaces whose semantic identity did not change.
5. Either emit a new artifact with a new receipt, or reject the configuration with the exact unsatisfied requirement.

For example, a larger batch may be accepted with a different static arena, accepted only with a different explicit realization or microbatch plan, or refused because it exceeds the target's declared memory. “Try it and discover an OOM halfway through” is not the intended control system.

A learning-rate change is different from a shape change, but it is not consequence-free. It may alter optimizer validation, token-interval normalization, numerical contracts, scheduler state, and checkpoint identity. Alpha's goal is to derive those consequences from one typed learning program instead of asking several configuration files and runtime branches to remain synchronized.

Selection is not fallback

Compile-time selection may consider more than one legal realization. That is not the same as runtime fallback.

The selected realization must be named in the plan, prove its capability requirements, fit its resource budget, and appear in the artifact receipt. If no realization is valid, compilation fails. The compiler must not catch the failure and quietly send the operation to a host implementation or a slower kernel with different semantics.

This is especially important for agentic iteration. The agent needs to know whether its hypothesis improved or whether the system merely chose a different path.

What conditionals remain at runtime

Alpha is not trying to remove all control flow. Some decisions depend on information that genuinely exists only while the program runs:

• data-dependent model decisions;
• finite loops over runtime input;
• observed loss, convergence, or bounded stopping state;
• device completion, fences, and errors;
• checkpoint or input validation; and
• explicitly modeled adaptive policies.

Those conditions should remain visible in typed control flow or a declared state machine. The goal is to eliminate configuration dispatch from hot execution, not to pretend dynamic computation has no branches.

If an adaptive choice can change shapes, memory, or target requirements, the compiled artifact must carry a proven envelope covering every allowed outcome. Otherwise the alternatives should be separate compiled artifacts.

Current implementation boundary

This architecture is partly real and partly the direction of travel.

The current tree explicitly models AdamW configuration and validation, promoted batch/token geometry and physical tensor-memory failures, and target-specific resource gates such as the SM86 block-reduction gate. The compiler PRD requires post-elaboration types to carry exact shapes, layouts, address spaces, numerical contracts, target requirements, resource use, and synchronization state; it also requires targets to prove both capability satisfaction and resource fit.

That does not yet mean every hyperparameter is a type-level constant, every cross-device choice is automatically solved, or every invalid experiment is rejected before execution. The complete hyperparameter-to-artifact specialization system is a core Alpha objective. Each promoted path still needs compiler and physical evidence.

Rapid experimentation without hidden escape hatches

The point of the type system is not to prevent experiments. It is to make rapid experiments comparable.

An agent should be able to change one axis at a time:

• replace attention with recurrence, sparse routing, a state-space operator, or an original mechanism;
• replace backpropagation with a local, implicit, perturbative, or multi-phase rule;
• alter precision, layout, tiling, accumulation law, or synchronization;
• train on a smaller, structured, synthetic, or curriculum-shaped dataset;
• target a CPU reference path, SM86, or a future architecture; and
• define a new success measure beyond next-token likelihood.

The compiler then exposes the consequences: which types no longer line up, which quantities are duplicated, which effects appeared, which target capability is missing, which resource bound no longer fits, and which evidence gate has not been crossed.

Fast experimentation does not mean skipping those questions. It means answering them mechanically instead of rediscovering them after a long run.

The billionth-scale question

“Could attention be useful at one billionth (10^-9) of today's scale?” is intentionally an extreme research question. Alpha does not currently establish that it can. The value of the question is that it separates several assumptions that are often bundled together:

1. Does the useful mechanism require today's parameter count?
2. Does it require today's sequence length or vocabulary?
3. Does it require dense global attention at every layer?
4. Does it require next-token string generation as the objective?
5. Does it require web-scale pretraining, or can structure and curriculum replace some volume?
6. Does the task need a general conversational model at all?

A good Alpha experiment would make the scale, operator, objective, dataset, compute budget, and evaluation contract explicit; compile the smallest viable version; and compare it against named baselines. A failed result is still useful if the receipt makes the failure reproducible.

Maybe less data is enough

Alpha's philosophy does not assert that enormous datasets are unnecessary. It refuses to make enormous datasets a precondition of asking the question.

Smaller-data experiments may explore stronger priors, typed state, synthetic curricula, active selection, online adaptation, local objectives, better interfaces between code and learned decisions, or tasks whose output space is constrained in advance. Each can reduce one burden while increasing another. The compiler cannot prove empirical quality; it can make the trade explicit and preserve the experiment faithfully.

What success looks like

A learning-system experiment succeeds at several distinct levels:

Alpha's wager is that agents will explore more boldly when every layer is replaceable, but every claim has to survive this ladder.

Read next

• How the language and compiler work
• The philosophy behind Alpha
• Evidence and status
• What Alpha is good and bad at