References v3: function registry (Function3/factory) + first-pass FilesReferenceFinder3 - #214
Merged
Merged
Conversation
added 3 commits
July 29, 2026 18:33
…/last Function3 (csvpath/references/functions/function_3.py): base class for real, behavior-having reference functions, distinct from FunctionCall3 (the parsed name+arg shape the transformer builds). Declarative metadata per subclass (SUMMARY, ROLE, DATATYPES, ARG_TYPES, ARG_REQUIRED) instead of Args/ArgSet -- that machinery solves multi-arg overload problems these functions do not have, since references-v3 functions take at most 1 arg. check_valid() validates arg presence/ type generically from that metadata, recursing into a nested Function3 arg, mirroring matchable.py's parse-time structural-check pattern. describe() returns the metadata as a dict, feeding a future type-ahead registry layer. ReferenceFunctionFactory: name-keyed registry, conceptually similar to csvpath.matching.functions.function_factory.FunctionFactory per "requirements for functions.txt". build() compiles one FunctionCall3 into a validated Function3, recursing first if its arg is itself a FunctionCall3. build_chain() compiles a whole function chain and enforces "at most one pointer function per chain" -- a pointer nested inside another function's argument (already compiled by build()) does not count toward its own outer chain's budget. First3/Last3: the first two concrete functions, both POINTER role, arrival/registration order, no arg. Chosen as the smallest unambiguous starting set; Index3 held back pending a spec question about 0- vs 1-based indexing. 256 tests in tests/references/ now (26 new), full suite still at the known 11-failure (SFTP/S3) baseline.
David confirmed the spec's own "index(7) means the seventh file" wording was a documentation mistake -- indexing is 0-based, so index(7) is actually the eighth file. Fixed the wording in "creating references v3.txt" (untracked notes) to match. Index3: POINTER role, required int arg, registered in ReferenceFunctionFactory alongside first/last. 7 new tests (263 total in tests/references/), full suite still at the known 11-failure (SFTP/S3) baseline.
Discovered via direct testing (not assumed) that a literal dotted
filename like "zero.csv" cannot be written as a bare name_one path
segment: PATH_SEGMENT's charset excludes ".", and "." is the grammar's
own name_one/name_three separator, so the string doesn't just parse
wrong, it fails to parse past that point. v1/v2 solved this with a
lossy "swap . for _" convention (files_reference_finder_2.py's
_starts_with); David and I agreed that is worth avoiding in v3 (it
had a real collision risk, e.g. "test_data.csv" and "test.data.csv"
both normalizing to the same thing) in favor of using the grammar's
own STRING mechanism, which already allows a literal "." with zero
ambiguity.
Name3 (:name("...")): CONTEXT_SETTER role, str arg only for this pass
(the grammar also allows "*"/"@var"/regex there, but those need
machinery -- wildcard-as-arg semantics, runtime variable lookup --
this pass does not need). Used only as a name_one path segment, not in
name_three, matching the STRUCTURE table's division of labor.
FilesReferenceFinder3: first concrete finder, grounded in the real
on-disk manifest schema (one flat, append-only manifest.json per
named-file; "file_home" shared by every version of the same logical
file; arrival order is manifest array order, not sorted by "time" --
all confirmed against FileManager/FileRegistrar and a real fixture,
not assumed). name_one narrows to a file (literal segments, "*", or
:name("...")); name_three must reduce, via
ReferenceFunctionFactory.build_chain(), to exactly one pointer
function (:first()/:last()/:index(n)) that picks the version. Root
major as "*" (every named-file), the "#worksheet" marker, function-
valued path segments other than :name(), and a literal name_three body
are all explicitly out of scope for this pass and raise a clear
ReferenceException3 rather than silently doing the wrong thing.
30 new tests (286 total in tests/references/), including a direct
reproduction of the spec's own EXAMPLE SCENARIO. Full suite still at
the known 11-failure (SFTP/S3) baseline.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Function3base class (csvpath/references/functions/function_3.py): real, behavior-having reference functions, distinct fromFunctionCall3(the transformer's raw parsed name+arg shape). Declarative per-subclass metadata (SUMMARY,ROLE,DATATYPES,ARG_TYPES,ARG_REQUIRED) instead of portingArgs/ArgSet— that solves multi-arg-overload problems these ≤1-arg functions don't have.check_valid()validates generically from that metadata;describe()feeds a future type-ahead registry.ReferenceFunctionFactory: name-keyed registry.build()compiles aFunctionCall3into a validatedFunction3, recursing into a nested function arg first.build_chain()compiles a whole function chain and enforces "at most one pointer function per chain, per nesting level" — a pointer nested inside another function's own argument does not count toward its outer chain's budget.First3/Last3(POINTER, no arg),Index3(POINTER, required int arg, 0-based — the spec doc's own "index(7) means the seventh file" wording was a documentation mistake, corrected to "eighth file"),Name3(CONTEXT_SETTER, required str arg).Name3exists because of a real discovery made via direct testing: a literal dotted filename ("zero.csv") cannot be written as a barename_onepath segment —PATH_SEGMENT's charset excludes., and.is the grammar's ownname_one/name_threeseparator, so it fails to parse past that point, not just parses wrong. v1/v2 solved this with a lossy "swap.for_" convention; we used:name("...")instead (a quotedSTRINGalready handles a literal.with zero ambiguity, and avoids the collision risk of the underscore approach).FilesReferenceFinder3: first concrete finder, grounded in the real on-disk manifest schema (confirmed againstFileManager/FileRegistrarand a realmanifest.jsonfixture, not assumed).name_onenarrows to which file (literal segments,*, or:name("..."));name_threemust reduce, viabuild_chain(), to exactly one pointer function that narrows which version — matching the STRUCTURE table's division of labor. Deliberately out of scope for this pass (each raises a clearReferenceException3):root_major == "*", the#worksheetmarker, function-valuedname_onesegments other than:name(), and a literalname_threebody.Test plan
tests/references/— 286 tests, all passingFilesReferenceFinder3verified directly against the spec's own EXAMPLE SCENARIO in "creating references v3.txt" ($alpha.files.*.:last()resolves to the exact file the spec says it should)