RFC-0042.1: Pluggable input source — `@@fsm` over owned buffers, borrowed slices, and callbacks

Status: Accepted; implemented for the Rust backend.
Author: Mark Truluck mark.truluck@cogiton.com
Created: 2026-06-17
Amends: RFC-0042 — Principle 9 (construction-driven execution) and §11.3 (streaming / incremental drive)
Motivated by: dogfooding @@fsm on framec’s own lexical leaf scanners (prolog_scanner, lex_number, lex_string, scan_identifier).

Summary

RFC-0042 fixes the v0.1 invocation model: an @@fsm is constructed with its full input and runs to completion at construction. On the Rust backend the recognizer owns its input — new(src: Vec<char>) — and always scans from offset 0.

That fits “recognize one complete thing.” It is the wrong shape for the other half of what @@fsm advertises — byte-level scanning and lexical analysis (§1) — where the host already holds the input and wants to recognize a prefix at a moving cursor, calling the recognizer once per token. Under the construction model the host must hand it buf[cursor..] as a fresh owned buffer every token, copying the tail each call — O(n²) input marshalling. The DFA does not rescan (each scan is forward-once, O(token)); the API forces the copy.

This RFC makes the recognizer generic over a pluggable input source and adds a reusable drive form:

the recognizer reads input through a tiny per-fsm trait (fsm_get/fsm_len), so the host owns the storage strategy — an owned buffer, a borrowed slice (zero-copy), or a host callback;
over(src) builds the recognizer without running; scan_at(start) re-seeds and re-runs from any offset, reusable across a whole input with no per-call copy.

Construction (new) and every observable field are unchanged (back-compat).

Motivation — the dogfood finding

Three of framec’s four regular leaf recognizers — number, string, identifier — are lexer methods, called once per token as the cursor advances. The generated recognizer owns its buffer and scans from 0, so using it at cursor N forces src[N..] to be copied each call: O(n²). The recognizer is fine; the IO model is the cost. The construct that advertises “lexical analysis” should be usable as a scan-at-cursor primitive. This RFC makes it one — and the four dogfood leaves now drive their recognizers over the host’s &[u8] zero-copy.

1. Pluggable input source

Each generated recognizer emits a small per-fsm trait:

pub trait <Name>Input { fn fsm_get(&self, i: usize) -> <Elem>; fn fsm_len(&self) -> usize; }

<Elem> is char (bytes/char alphabets) or String (token). All input reads in the DFA / Pike VM / anchor / word-boundary / Mode-C code go through fsm_get / fsm_len. The recognizer is generic: struct <Name><I: <Name>Input>. Emitted impls per fsm:

Source	Type	Use
owned	`Vec<char>` / `Vec<String>`; also `Vec<u8>` (bytes)	the recognizer owns the buffer; the back-compat `new` path
borrowed	`&[char]` / `&[String]`; also `&[u8]` (bytes)	zero-copy over the host’s buffer
callback	`<Name>Fn(closure, len)`, `closure: Fn(usize) -> Elem`	the host computes elements on demand (rope, mmap, windowed buffer)

For a bytes recognizer, &[u8] / Vec<u8> read each byte as its code point, so a byte scanner drives the recognizer over its raw &[u8] with no decode and no copy. The trait is named per fsm (<Name>Input) so multiple @@fsm in one module never collide.

Why a random-access accessor, not a forward-only next(). The DFA does greedy longest-match: it reads past the match end and backs up to the last accept (/abc|a/ on "abx" reads index 2 but matches to 1). A forward-only pull loses the over-read symbols. fsm_get(i) is random-access, which is what a backtrack-free longest-match recognizer needs.

2. Drive forms

new(src, params…) -> Self — construct and run (unchanged; back-compat — new(vec![...]) still infers the owned source).
over(src, params…) -> Self — build without running, for reuse.
scan_at(&mut self, start: usize) -> bool — reset the per-scan state (cursor = start; accepted / reject_position / return_value / matched / captures / Mode-C instances reset; domain vars re-evaluated from their initializers), then run(). Returns accepted; the host reads self.cursor / self.return_value / captures afterwards. The input and the construction parameters persist across calls — build once, scan many.

// zero-copy lexer primitive: build once over the host's bytes, scan many
let mut s = NumberFsm::over(&source[..]);
loop { if s.scan_at(cursor) { /* token [cursor, s.cursor) */ cursor = s.cursor; } }
// or a host callback
let mut s = NumberFsm::over(NumberFsmFn(|i| compute(i), len));

3. Mode C stays owned

A Mode-C inner recognizer is retained in a self field (for $state.label.return_value). Storing an inner that borrows self.input would be a self-referential borrow — rejected by Rust. So the inner is the owned form, constructed over a window materialized through the outer’s accessor: (cursor..end).map(|i| self.input.fsm_get(i)).collect::<Vec<…>>(). Same per-call-out cost as before (Mode C was never the O(n²) hot path), and it works uniformly whether the outer source is a buffer, slice, or callback.

4. Relationship to RFC-0042 §11.3 (streaming)

RFC-0042 §11.3 defers “streaming / incremental drive” — input arriving over time, the host pausing recognition between deliveries, the fsm carrying suspended automaton state. This RFC is deliberately not that. Here the whole input is present behind the accessor; scan_at runs to completion (accept/reject) on each call, exactly like construction, just from an offset over a host-owned source. fsm_get(i) assumes the element at i is available. The suspend/resume streaming half stays in §11.3.

5. Design alternatives considered

Borrowed-only scan_at(&buf, start) -> Outcome (the original draft of this RFC). Stateless and zero-copy, but reuses nothing and — for bytes/char, where the recognizer holds a decoded buffer — “borrow the host’s bytes” needs the &[u8] impl anyway. Subsumed by the generic source (&[u8] is one of the sources), which additionally gives reuse and callbacks.
Reusable but owns-one-copy (non-generic). Simplest; kills the O(n²) with a single decoded copy per scanner. Rejected as the only option because it cannot borrow the host’s &[u8] zero-copy and cannot take a callback — the generic source delivers all three over the same engine.
Generic with a dyn trait object. A vtable call per element for every source (including owned). The monomorphized generic is free for owned/slice and inlines the closure.

6. Amendments to RFC-0042

Principle 9 gains a clause: construction-driven execution remains the default; a recognizer may additionally be built with over over a pluggable input source and driven repeatedly with scan_at (this RFC). Suspended-state streaming stays deferred (§11.3).
§11.3 is narrowed: the single-shot, fully-present, host-owned-source half is specified here; the suspend/resume streaming half remains deferred.

7. Scope and status

Implemented for the Rust backend (fsm_rust.rs): the per-fsm <Name>Input trait + owned / borrowed / callback impls, the generic recognizer, over / scan_at, Mode-C-owned-via-accessor. The four dogfood leaves drive their recognizers over the host’s &[u8] zero-copy.
The other 16 backends define their own input-source idiom (interface / closure / block) as a follow-on; the contract (fsm_get/fsm_len, over/scan_at) is uniform.
Out of scope: true async streaming with suspended DFA state (§11.3).

RFC-0042.1: Pluggable input source — @@fsm over owned buffers, borrowed slices, and callbacks