Experimental Polars reasoning

gestaltdb.reasoning.adapters.polars.PolarsDatalogPrototype is an explicit, optional experiment for the existing binary, positive, named-entity Datalog profile. Install gestaltdb[polars]. The default remains DatalogEngine; there is no automatic backend selection, source extraction or publication. The adapter/compiler import boundary remains inward; only this optional adapter loads Polars and Arrow, on demand.

from gestaltdb.reasoning import Atom, Fact, Program, ReasoningLimits, Rule, Variable
from gestaltdb.reasoning.adapters.polars import PolarsDatalogPrototype
from gestaltdb.reasoning.protocols import InputContext

x, y, z = Variable("x"), Variable("y"), Variable("z")
program = Program([Rule("transitive", [Atom(x, "p", y), Atom(y, "p", z)], Atom(x, "p", z))])
run = PolarsDatalogPrototype().evaluate(
    program, [Fact("a", "p", "b"), Fact("b", "p", "c")],
    context=InputContext("in-memory-example", "default", "none", "all", "identity"),
    limits=ReasoningLimits(), max_intermediate_rows=100_000,
)
assert Fact("a", "p", "c") in run.result
assert run.result.proofs is None
assert run.identity.executor_version == "polars-prototype-v1"
assert run.metrics["join_calls"] > 0

PolarsRun contains an ordinary complete EvaluationResult plus an exact CompilationIdentity and immutable telemetry. It is deliberately not the existing Evaluator protocol: witnesses, all-support provenance, incremental maintenance, authenticated source evaluation and persisted generations are not implemented. collect_proofs=True fails before input consumption. Evidence inputs affect logical truth as facts only; references are not returned as proofs. Do not silently substitute this prototype into claims/publication adapters.

Execution and semantic boundary

Input conversion is bounded Python work: deduplicate facts, intern full entity/ predicate/constant names with the RE-02 dictionary and transfer compact uint64 relations through Arrow. Final decoding iterates result rows in Python. These boundary costs are measured, not described as zero-copy KV decoding.

Between boundaries, Polars performs filters, repeated-variable equality checks, projection, exact equijoins, distinct, union and stable-minus-new anti-joins. There are no Python per-row callbacks or dictionary conversions inside these operators. Each body occurrence can serve as the designated delta relation; identical repeated atoms need only one pivot. A common unit column preserves ground-atom truth and disconnected-body joins. All new facts are visible in the next synchronous round, never midway through an active round. Results have exact set semantics and canonical external-ID ordering, independent of native row order.

Internal semi/anti-join primitives are existence/set operations, not exposed negation-as-failure semantics. Literals, arbitrary-arity execution, negative premises, aggregates and equality canonicalization remain unsupported. Safe Program construction and v1 digests are unchanged. Small finite grounding oracles, generated programs, late-delta/repeated/ground/disconnected bodies and analytical ontology closures test the selected semantics, not full OWL support.

The prototype rebuilds eager native expression/join plans per rule occurrence and round. It does not cache lazy plans, reorder body joins by cost, implement SCC scheduling or offer native proof propagation. Telemetry separates input, conversion, dispatch planning, per-occurrence expression construction, operator execution, join-count preflight, fixpoint and decoding. Nested timings overlap; do not sum fixpoint and operator/preflight durations as independent stages. planning_s measures Python dispatch construction; expression_s measures Python expression construction. Eager native query planning and kernel execution remain combined in operator_s/preflight_s; these are not native compiler instrumentation or isolated join-kernel timings.

Limits are intentionally different

ReasoningLimits.max_input_facts, max_rules, max_iterations and max_facts retain their input/output meaning. Duplicate evidence counts toward input bounds; failures return no successful partial run and never publish data.

max_candidate_rows instead counts conservative materialized/output row work: atom input upper bounds, join outputs, projections, filter-join input upper bounds and concatenations. This is not attempted-unification parity with the indexed Python engine, and can fail earlier even for the same successful logical closure. The benchmark reports actual counts separately; do not compare limited runs as speedups. Proof budgets have no effect because proof mode is rejected.

Before an equijoin, a native grouped multiplicity query counts matching outputs. max_intermediate_rows is checked before allocating that output. Key-count query state is bounded by input rows, not the potentially quadratic result. Native count overflow fails explicitly. Selection/concatenation use conservative row upper bounds. This protects obvious Cartesian explosions without claiming exact allocator, hash-table, dictionary, string-size, RSS or disk limits. No spill is implemented. A running native call is not cooperatively interrupted; use a separate worker for hard deadlines and external memory controls.

Context identities are compatibility keys supplied by an already validated caller, not authenticated database horizons. A new invocation rebuilds its state; different contexts/dictionaries never share caches implicitly. No PyRex scan API, storage dependency or JVM runtime is required by this prototype.

Measurements and decision

Extend the existing analytical comparison runner rather than introduce a second oracle. For example:

uv run --with pyarrow --with polars --with clingo python -m benchmarks.ontology --backends native polars clingo --sizes 16 128 --repeats 3 --threads 1 --temp-root /tmp/opencode --output /tmp/opencode/re03-comparison.json
uv run --with pyarrow --with polars --with clingo python -m benchmarks.ontology --backends native polars clingo --workloads hierarchy recursive duplicates --sizes 1024 --repeats 3 --threads 4 --temp-root /tmp/opencode --output /tmp/opencode/re03-sparse.json

The runner now records thread caps for Polars/OpenMP/BLAS, per-stage telemetry, runtime installed-file footprint and source digests. Every successful row checks the identical analytical closure and digest. native retains the historical benchmark name for indexed Python, not a compiled extension. Clingo remains a benchmark-only dependency without witness/resource-accounting equivalence.

Clean-source 14ade5a results (Python 3.13, three fresh workers per cell; indexed Python and Polars receive identical independent Evidence wrappers):

Workload	Indexed Python	Polars	Clingo
Dense N=128	534.73ms	46.44ms	54.29ms
Sparse recursive N=1024	90.37ms	40.40ms	33.03ms
Hierarchy N=1024	17.66ms	21.27ms	18.22ms
Duplicate evidence N=1024	16.42ms	21.12ms	25.69ms

These are median pipeline times including ontology compilation, initial fact conversion, full prototype conversion/fixpoint/decoding, but excluding separately measured package/runtime imports and fixture construction. Small N=16 cases were consistently slower with Polars. Peak full-worker RSS in the comparison was about 127 MiB for Polars versus 54 MiB indexed Python. Four threads did not materially improve the tested N=1024 sparse workloads; the one- and four-thread matrices were rerun sequentially on the same shared host (not a dedicated CPU-isolated machine).

This supports continuing the columnar experiment, not switching the default. No million-row, universal scaling, proof-mode or maintenance-performance claim is made. Full RE-06 backend selection still needs representative user workloads, source extraction, footprint, failure/cancellation and supported-mode evidence. The compact Python batch container has no independent fixpoint executor to time; its conversion/transfer costs are measured in the RE-02 report instead.