Catalogue-guided search for learning systems#

Alpha's living catalogueUnderstand · analysisThe living catalogueHow a growing library of typed knowledge could let agents assemble and test learning systems from humanity's reusable ideas. asks how agents could inherit more of humanity's reusable technical knowledge. This page asks the next question:

If the catalogue existed, how could an agent use it to propose genuinely different learning systems, reject incoherent combinations early, and spend physical experiments on a small, diverse portfolio rather than on whichever idea sounded best first?

The proposed answer is catalogue-guided search: literature supplies versioned mechanisms and evidence; agents propose and repair candidate programs; deterministic checks reject known contradictions; a replaceable learned evaluator helps allocate a bounded conceptual-search budget; and Alpha compiles only the qualified finalists into reproducible experiments.

This is a research architecture, not a result. No broad search advantage, new learning algorithm, or Jev advantage has yet been demonstrated.

Relationship to the source proposal#

This page expands the 18 September 2026 team proposal, Catalogue-Guided Search for Learning Systems. The PDF established the system boundary, candidate representation, diversity-preserving beam, Jev adapter, associative-memory example, evaluation plan, and legal cautions. This page adds:

• a wider map of prior work;
• a sharper statement of what may be novel in the combination;
• a compiler-centered answer to hyperparameter and hardware conditionals;
• explicit hypotheses, controls, measurements, and falsification rules;
• a staged implementation plan connected to Alpha's current catalogue; and
• stronger criticism of the catalogue, search policy, judge, and scientific claims.

The core loop#

primary sources
      ↓
versioned technical catalogue
      ↓
retrieved components + assumptions + negative evidence
      ↓
agent proposals ↔ deterministic contract checks
      ↓
diverse conceptual frontier ↔ bounded learned judgments
      ↓
small experimental portfolio
      ↓
Alpha specialization, compilation, execution, measurement
      ↓
results, failures, counterexamples, and improved components
      └─────────────────────────────── back to the catalogue

The loop separates three achievements that are easy to conflate:

1. Catalogue progress means recovering a mechanism faithfully enough that another researcher can inspect its definition, assumptions, provenance, and unresolved gaps.
2. Search progress means allocating attention and experiment budget better than the controls.
3. Scientific progress means a checked theoretical result, a controlled empirical improvement, a useful negative result, or another defensible result beyond the nearest prior work.

A million catalogue rows do not imply a good search policy. A million judged candidates do not imply one valid experiment. A successful experiment does not establish that the search process, rather than luck or a hidden implementation choice, caused the result.

Why this is more than neural architecture search#

Most automated machine-learning systems search inside a vocabulary chosen in advance: layers, cells, optimizers, schedules, or numerical hyperparameters. That can be valuable, but it leaves the deepest assumptions fixed.

AutoML-Zero showed that evolutionary search can recover complete learning algorithms from basic mathematical operations rather than only arranging expert-designed neural layers. FunSearch paired an LLM with a program evaluator to search function space, and AlphaEvolve extended evaluator-driven evolutionary code search across mathematics and computing. These are strong precedents for generating programs and using automated feedback.

Catalogue-guided search makes a different bet: the search vocabulary itself should be historically broad, source-grounded, typed, versioned, and able to improve. A candidate might change not only an attention block but also its representation, state transition, credit assignment, memory topology, numerical method, control policy, physical realization, or learning objective. It may draw from adaptive filtering, control, information theory, numerical analysis, symbolic learning, neuroscience, or a newly proposed operator.

The system should therefore search over learning programs, not a single architecture grammar.

The five layers#

1. A source-grounded technical catalogue#

The capture target is not just papers, named architectures, or equations. It is every ingredient needed to understand and reconstruct a mechanism:

• the exact source artifact and version;
• the relevant region, formula, procedure, or code;
• local symbol definitions and notation;
• inputs, outputs, state, reset behavior, update order, and randomness;
• assumptions, domains of validity, units, numerical conditions, and failure modes;
• theoretical support, empirical support, counterexamples, corrections, and retractions;
• known implementations and their differences; and
• fields that remain unresolved.

Existing knowledge systems show pieces of this shape. The NIST Digital Library of Mathematical Functions combines curated definitions, equations, references, software links, and mathematical search. Mathlib demonstrates how a community can grow a unified formal library with dependent types and automation. OpenML links tasks, data, flows, parameters, runs, and results so experiments can be inspected and compared. Wikifunctions demonstrates an openly editable, machine-accessible library of functions.

None of these is the proposed system, but each exposes a necessary lesson: reusable knowledge needs stable identity, explicit structure, provenance, governance, and tools for correction. A large corpus without those properties is retrieval material, not yet a scientific component library.

2. A contract-bearing candidate language#

A catalogue record is not automatically executable. A source may omit a reset condition, rely on an unstated unit convention, or describe a biological observation without defining an algorithm.

The explorer therefore works on a separate, immutable candidate representation:

Program = (
  graph,          # components and control/data flow
  contracts,      # interfaces and semantic constraints
  holes,          # explicitly unresolved responsibilities
  state,          # parameters and persistent/episode-local state
  protocol,       # learning and evaluation rules
  evidence        # pinned catalogue revisions and proposal lineage
)

Every component contract should state, where relevant:

• value and index types;
• shapes, dimensions, units, coordinate frames, and layouts;
• ownership, effects, causality, mutability, and lifetime;
• sources of randomness and seed scope;
• state initialization, update, and reset rules;
• learning signal and credit path;
• numerical regime, approximation, and stability assumptions;
• target capabilities and resource bounds; and
• proof, test, oracle, measurement, or source evidence.

A hole is first-class information. “The address rule is not yet learnable,” “the estimator assumes independent samples,” or “this realization has no qualified derivative” is more useful than invented certainty.

Candidate identity must include canonical program structure, pinned component revisions, assumptions, target profile, and protocol. Two candidates are merged only when their identities match or an explicit equivalence has been established. Textual similarity, equal tensor shapes, or a judge's confidence is not equivalence.

3. A broad, diversity-preserving explorer#

The explorer proposes legal edits to partial programs:

• replace a mechanism;
• change a representation;
• add or remove persistent state;
• alter credit assignment or update order;
• rewire control flow;
• specialize for a target;
• combine mechanisms from different research branches; or
• introduce a new operator with an explicit contract and origin.

Some changes must happen together. A new representation may require a different update law; a protected memory direction may require a compatible address learner. The explorer therefore permits short, bounded lookahead rather than rejecting every incomplete intermediate node.

MAP-Elites motivates retaining high-performing solutions across chosen dimensions of variation instead of collapsing immediately to one winner. POET shows why stepping stones and transfers between concurrently explored paths can matter. The initial Alpha policy can borrow those principles without claiming their results transfer automatically.

A practical frontier should mix:

• judge-guided selections;
• underrepresented structural regions;
• seeded random candidates;
• intentionally incomplete candidates whose holes are valuable to resolve; and
• a few low-ranked controls that reveal evaluator blind spots.

The search tree is only a view. Storage is a graph because several edit sequences can reach the same candidate, and a previously weak branch may become useful when the catalogue, task, or target changes.

4. Qualification and physical experiments in Alpha#

Conceptual exploration tolerates explicit holes. Experimental validation does not.

Before a candidate consumes serious hardware, Alpha should require:

• resolved public contracts;
• type, quantity, effect, shape, and target-capability checks;
• independent numerical or semantic oracles where available;
• derivative and update-law qualification where learning needs them;
• a frozen task, dataset split, seeds, metrics, resource budget, and stop rule;
• an exact compiler, source, component, and hardware identity; and
• a continuation-safe checkpoint contract.

This boundary is where Alpha contributes something that ordinary code-generating search lacks. The compiler is not asked whether an idea is scientifically good. It is asked whether the executable experiment means what its contract says, whether the selected target can realize it, and whether the resulting artifact carries enough identity to reproduce and compare it.

Counterexample-guided inductive synthesis offers a useful pattern: generate a candidate, validate it mechanically, and turn a failure into a counterexample that constrains the next proposal. Catalogue-guided search generalizes that pattern beyond fully decidable specifications. Proof and deterministic checks eliminate what they can; experiments resolve questions that remain empirical.

5. Evidence returns to the library#

Each run should produce more than a score. It should return:

• exact program and component identities;
• compilation and qualification receipts;
• hardware and runtime measurements;
• learning curves and checkpoint lineage;
• failures and the phase that produced them;
• evidence for and against the hypothesis;
• discovered sensitivities and domains of validity;
• candidate counterexamples; and
• reusable components or contracts that survived review.

Results never overwrite their predecessors. They extend, contradict, qualify, or supersede them. This allows the catalogue to preserve negative knowledge and changing evidence instead of presenting only the surviving winners.

What the learned evaluator is allowed to do#

The source proposal uses Jev for bounded, typed judgments over supplied alternatives. TypeSafe describes Jev as an early-access system for “typed probabilistic decisions” rather than string generation in its launch post. That is a provider claim, not independent evidence of calibration or suitability for this research.

The evaluator may help answer questions such as:

• Which continuation is most worth specifying under the remaining conceptual budget?
• Which stated interaction is most likely to defeat the intended mechanism?
• Which unresolved responsibility should be expanded next?
• Does the supplied evidence justify further investigation?

It may not establish:

• that a candidate is correct;
• that a mechanism is novel;
• that a learning system will train;
• that a probability is calibrated to scientific success;
• that a failed check can be ignored; or
• that repeated model agreement is independent corroboration.

Every decision packet should be rendered from structured contracts, not promotional prose. Store the exact input, output, candidate order, rubric, model identity, timing, validation state, and retry lineage. Randomize irrelevant ordering and names in sensitivity tests. Compare Jev with a compact LLM evaluator and with no learned evaluator.

This adapter must remain replaceable. The current TypeSafe customer agreement restricts some uses of the service and its outputs; the applicable order and written permission must be checked before automated research, publication of benchmark results, or any workflow that could be interpreted as developing a similar system. Console access is not permission to scrape an undocumented endpoint.

Removing conditional training logic by specialization#

Contemporary training programs often become forests of runtime conditionals:

if optimizer == ...
if precision == ...
if device supports ...
if sequence length exceeds ...
if checkpoint format is ...
if this hyperparameter activates that feature ...

The conditionals are not merely untidy. They multiply the number of partially tested programs. A one-line hyperparameter change can select a different kernel, layout, numerical path, memory footprint, or distributed protocol and fail only after hours on one device class.

Catalogue-guided Alpha should move as much of that choice as possible into typed compile-time specialization:

1. A candidate declares its model, learning rule, hyperparameters, state, protocol, and target profile as data with types and constraints.
2. Catalogue components publish capability and resource contracts.
3. The compiler resolves compatible components, layouts, derivatives, schedules, and realizations for that exact candidate and target.
4. Constraint failures stop before training: unsupported precision, invalid state size, missing derivative, impossible buffer plan, or incompatible reset semantics.
5. The emitted artifact contains only the selected path plus a manifest of every choice that shaped it.

This does not abolish all branching. Decisions that genuinely depend on runtime observations—adaptive stopping, data-dependent routing, fault handling—remain explicit program behavior. The aim is to remove configuration ambiguity and untracked hardware dispatch from the training loop, not to pretend dynamic systems are static.

The search space then contains distinct, inspectable experimental programs rather than one mega-program whose behavior depends on a hidden web of flags. The compiler can compare target-specific resource consequences before a candidate runs, while the catalogue keeps the semantic mechanism separate from each physical realization.

A worked example: memory without collateral damage#

Suppose a streaming learner must revise one association while preserving unrelated ones. A baseline uses an associative matrix with a corrective update. Algebra shows that overlap between keys controls how much a write changes another query.

The catalogue can retrieve nearby work on fast weights, associative memory, constrained updates, address formation, partitioning, and learned write control. The agent then proposes several branches:

A judge may prioritize which contract to complete. It cannot answer the experiment. Alpha must separately test fixed-vector operator behavior, whether the address learner can discover useful keys, held-out generalization, preservation at matched revision success, and full resource cost.

This example illustrates the full method:

• derive a limitation rather than asking an agent for free-form novelty;
• retrieve close prior art before naming a new mechanism;
• expose unresolved learning rules as holes;
• keep multiple causal hypotheses alive;
• require matched controls; and
• promote only after physical evidence.

What may actually be novel#

None of the ingredients is unprecedented on its own. Evolutionary program search, automated evaluators, program synthesis, learned libraries, experiment databases, formal libraries, quality-diversity search, and compiler specialization all have substantial histories.

The potentially novel contribution is their specific composition:

1. a historically broad, source-local catalogue of learning mechanisms and negative evidence;
2. a contract graph that keeps unresolved assumptions visible;
3. agent-generated edits over whole learning programs rather than only neural architectures;
4. diversity-preserving conceptual search with a replaceable, bounded judge;
5. proof- and compiler-based rejection before expensive execution;
6. target-specific artifacts with explicit physical identity; and
7. a closed loop in which experimental evidence improves both the catalogue and future search.

That composition is still only a proposal. Novelty must be established against the closest located systems, not inferred from the absence of a keyword match.

Research hypotheses#

The architecture decomposes into hypotheses that can fail independently.

H1 — Structured retrieval improves proposals#

Given the same proposer, task, and token budget, source-grounded component contracts produce more valid, distinct, and prior-art-aware candidates than ordinary paper passages or no retrieval.

H2 — Diversity preservation improves the final portfolio#

At the same conceptual and experimental budgets, a diversity-preserving frontier produces a better measured quality–resource portfolio than greedy selection.

H3 — A learned judge allocates conceptual search usefully#

At the same candidate pool and budget, Jev or another judge improves the final experimentally measured portfolio over seeded random selection and a compact-LLM baseline—not merely its own ratings.

H4 — Compile-time qualification saves failed experiments#

Typed contracts and target specialization reject a material fraction of candidates that would otherwise fail late, without excluding a disproportionate share of viable unconventional systems.

H5 — The loop compounds#

After accepted components, counterexamples, and evidence are returned to the catalogue, later search rounds reach equally strong portfolios with less proposal, repair, or experiment cost.

The project succeeds only if some of these survive controlled tests. A valuable catalogue can outlive a failed evaluator hypothesis. A useful compiler gate can outlive a failed search policy.

Experimental design#

Separate the variables#

Run both matched-wall-time and matched-spend comparisons. Neither implies the other. Count retrieval, proposal, validation, retries, failed compiles, training, and analysis—not only successful evaluator calls.

Keep random search serious#

Random search is not a token baseline. Random Search and Reproducibility for Neural Architecture Search found it competitive with leading NAS methods in the tested settings, and Evaluating the Search Phase of Neural Architecture Search found several search policies similar on average to random selection. The Alpha study should use seeded, diversity-aware random controls with the same validity filters and budget accounting as the proposed policy.

Use staged budgets#

Start with a frozen candidate pool so evaluator ranking can be studied without generator feedback. Then run adaptive searches at increasing scales—such as 100, 1,000, and 10,000 unique assessed candidates—while keeping the number of final physical experiments fixed.

Use early, cheap tests only when their relationship to the final objective is measured. Hyperband provides a principled precedent for allocating limited resources across many configurations, while Bayesian optimization for machine-learning experiments shows how variable experiment cost can enter the allocation policy. Neither removes the need to validate that a low-fidelity Alpha probe predicts full training.

Measure the whole portfolio#

Report:

• valid, distinct, and executable candidates;
• structural and mechanistic diversity;
• unresolved-contract burden;
• source support and nearest-prior-art recovery;
• proposer, retriever, checker, judge, compiler, and runner cost;
• late failure rate and failure phase;
• judge calibration, order sensitivity, and wording sensitivity;
• final task quality, memory, latency, energy or compute, and training cost;
• seed variation and unsuccessful runs; and
• whether the selected portfolio dominates or merely reshuffles the controls.

Do not compress these into one magic score. The intended output is a portfolio: a credible reference, one or more promising compositions, and a structurally different exploratory candidate.

The judge can become the problem#

Searching harder against an imperfect learned evaluator can select candidates that exploit its preferences. Reward-model overoptimization demonstrates the general failure mode: optimizing a proxy far enough can reduce the underlying objective.

Controls should therefore include:

• mechanically rendered contracts with promotional names removed;
• randomized candidate order and identifiers;
• semantically irrelevant paraphrases;
• hidden duplicate candidates;
• low-ranked and randomly selected experimental probes;
• a locked final evaluation never shown to the judge;
• separate scores for promise, evidence, missing information, interaction risk, and implementation uncertainty; and
• an audit of whether larger search budgets improve only judge ratings.

If the judge's advantage disappears under paraphrase, ordering changes, or physical evaluation, the result is a judge-specific prompt optimizer—not a better learning-system explorer.

The catalogue can become the problem#

A larger catalogue can also make search worse.

False authority#

A structured record can look more certain than its source. Sealed revisions mean immutable interpretations, not true interpretations. Source text, transcription, inference, implementation, and independent verification must remain distinct.

Ontology lock-in#

If every mechanism must fit one early schema, the catalogue will make unfamiliar ideas invisible. Keep a small relational core, validated extensible fields, explicit unknowns, and versioned mappings. Retrieval similarity must not silently merge concepts.

Survivorship and availability bias#

The easiest papers to acquire and parse are not necessarily the most important. Record discovery coverage, access gaps, language and era bias, extraction audit coverage, and negative results separately.

Combinatorial explosion#

More components create more incompatible compositions. Types and checks can remove impossible combinations, but they cannot make arbitrary search cheap. The project must measure proposal yield per unit cost and learn which abstractions reduce search depth without erasing productive irregularity.

Stale abstractions#

Components that were useful once can harden yesterday's assumptions into tomorrow's search prior. Deprecation, supersession, counterexamples, alternate definitions, and periodic rediscovery from sources are essential.

No universal search advantage#

The No Free Lunch theorems are a reminder that search performance depends on assumptions about the problem distribution. Catalogue-guided search does not escape that fact; it makes its priors inspectable.

Its expected advantage comes from structure believed to exist in real scientific work:

• useful mechanisms recur across tasks;
• interfaces and assumptions rule out many combinations;
• evidence and counterexamples improve allocation;
• related tasks share abstractions;
• hardware and numerical constraints can be checked; and
• failed experiments can inform later ones.

If the target tasks do not share that structure—or if the catalogue represents it badly—the added machinery may lose to simple search.

What this approach should be good at#

• Problems with many reusable mechanisms but no single accepted architecture grammar.
• Expensive experiments where cheap rejection and careful allocation matter.
• Cross-domain combinations whose assumptions can be stated explicitly.
• Work that benefits from preserving alternatives, negative evidence, and exact lineage.
• Systems whose hardware, numerical, or state constraints often fail late in ordinary frameworks.
• Long-running research programs where each result should improve the next search.

What it should be bad at#

• Problems with no informative evaluator until the full experiment is complete.
• Domains where key tacit knowledge cannot yet be represented or tested.
• One-off tasks where catalogue and contract overhead exceeds experiment cost.
• Open-ended goals with no defensible task or measurement boundary.
• Mechanisms whose behavior depends on inaccessible proprietary data or hardware.
• Claims of universal novelty, completeness, or superiority.
• Safety-critical deployment without domain-specific validation far beyond compiler correctness.

Security, rights, and scientific governance#

The ingestion and search system will process hostile inputs: papers, archives, LaTeX, repositories, model outputs, and generated code. Parsing must be isolated; shell escape and automatic code execution must be prohibited; redirects, file writes, and network access must be constrained; credentials must never enter evidence packets.

Source rights and licences survive ingestion. A searchable citation is not permission to redistribute a work or send it to a third-party model. Candidate lineage must preserve licences and provider-processing restrictions.

Governance must distinguish:

• source-attested content;
• project interpretations;
• agent-generated hypotheses;
• formally checked results;
• independent reproductions; and
• physical measurements bound to exact artifacts.

Disagreement is not a database error. Competing interpretations and contradictory results should coexist until a scoped decision or stronger evidence resolves them.

A buildable first version#

Alpha does not need an exhaustive history of science before testing the loop.

Phase A — Make one source-to-search path trustworthy#

1. Select a small, historically varied set of learning mechanisms.
2. Preserve exact artifacts, regions, equations, procedures, assumptions, and gaps.
3. Audit extraction against original rendered sources.
4. Convert selected revisions into candidate contracts without changing the source record.
5. Expose retrieval by responsibility, interface, assumptions, and failure mode.

Phase B — Build a reproducible explorer ledger#

Record search runs, candidates, multi-parent edit edges, holes, checks, judge calls, selections, experiments, and decisions in an independent SQLite ledger. Freeze the catalogue snapshot, protocol, budgets, policies, and seeds for every run. Unknown is neither pass nor fail.

Phase C — Qualify the policy before training#

Use a frozen pool to test deduplication, lineage, deterministic checks, diversity retention, judge response validation, order sensitivity, and replay. Compare random, compact-LLM, and Jev selection only after provider permission is clear.

Phase D — Execute a deliberately small portfolio#

Choose one bounded learning task and a portfolio containing controls, promising candidates, and structurally different exploration. Compile exact target artifacts. Separate operator tests from learned behavior and held-out generalization.

Phase E — Close the loop#

Return measurements, failures, counterexamples, and reviewed component improvements to the catalogue. Then test whether a second task becomes cheaper or better. That compounding test is more important than a flashy first winner.

How this connects to Alpha today#

The public catalogue browser is already a concrete starting point, but it is not the proposed historical research catalogue. Its 20 September 2026 snapshot reports 2,171 Alpha module identities, 135,542 structural declarations, and 12,051 direct import edges. It also reports zero populated contract records and only 14 evidence records.

Those numbers expose the exact gap:

• structural identity exists;
• source ownership and imports are browseable;
• the richer contract and evidence layers are mostly not populated; and
• no catalogue-guided search controller is present.

The next Alpha-native step is not to pour papers into the repository. It is to define a narrow interchange between an independently useful research catalogue and Alpha's compiler-owned symbol, contract, realization, and evidence identities. The research catalogue preserves source history. Alpha owns executable semantics and qualification. The explorer pins both.

Acceptance gates#

The first pilot is credible only when all of the following are true:

• real sources from different eras and formats are represented with inspectable anchors and visible omissions;
• extraction audits include material the parser failed to detect;
• candidate identities and search runs replay from frozen inputs;
• deterministic conflicts are rejected and unknown contracts remain visible;
• evaluator failures, malformed output, or missing permission cannot become approvals;
• random and compact-LLM baselines receive equal budgets and information;
• experimental criteria are frozen before final results are seen;
• Alpha produces exact native artifacts, continuation-safe checkpoints, and measured physical receipts;
• failed runs and negative results are reported; and
• any novelty or performance claim names its nearest prior art and precise scope.

Falsification rules#

The project should say in advance what would change its mind.

• If structured catalogue retrieval does not improve validity, diversity, or prior-art awareness over ordinary passages, keep the archive but reject the retrieval claim.
• If guided search does not improve the final physical portfolio over seeded random or compact-LLM search, do not make it the default policy.
• If larger conceptual budgets improve judge ratings but not locked experimental outcomes, treat the evaluator as overoptimized.
• If compile-time qualification rejects unconventional viable programs more often than it prevents late failures, redesign the contract boundary.
• If returned evidence does not reduce later search or experiment cost, the proposed compounding loop has not been shown.
• If provider terms prevent the intended evaluator study, replace the evaluator rather than bending the research around it.

Research lineage#

The standard of proof#

The motivating vision is ambitious: agents should be able to draw from a continually improving technical memory, combine ideas that present frameworks make awkward, and test alternatives rapidly without disguising assumptions or hardware failures.

The right response to that ambition is not a larger promise. It is a tighter loop:

Preserve the source. State the contract. Keep the holes. Search broadly. Check what can be checked. Spend experiments on a diverse portfolio. Bind every result to its artifact. Return what was learned.

If that loop produces better research per unit cost, the catalogue becomes an engine for invention. If it does not, its design makes the failure legible enough to improve—or to stop.