Storage Backends

GestaltDB separates graph logic from storage. GraphDB receives a key-value store instance and a serializer instance.

Backend Selection Summary

The current backend trade-offs are:

Backend	Best fit	Main trade-offs
LMDBStore	Mature embedded storage	Requires choosing a sufficient map_size
LevelDBStore	Simple local graphs	No native Arrow/Polars columnar fast path
PyRexStore	Large writes and ingestion	Optional dependency, RocksDB tuning knobs, opt-in transactions

Recommendations for the library as-is:

• Start with LevelDBStore for small applications, examples, and local testing when you do not need RocksDB’s native columnar ingestion path.

• Use PyRexStore for bulk loads and append-heavy workloads, especially with GraphDB.ingest_arrow or GraphDB.ingest_polars.

• Use LMDBStore when LMDB’s storage model is desirable and the database size is predictable enough to configure map_size safely.

• Keep disable_wal=False for durable RocksDB writes. Consider disable_wal=True only for disposable benchmark runs or reloadable bulk-load experiments.

• Benchmark with your actual serializer, index definitions, and graph shape. Backend differences can be hidden by Python object creation, JSON serialization, or index maintenance.

LMDB Backend

Use LMDBStore for a mature embedded backend with named sub-databases.

from gestaltdb.graphdb import GraphDB
from gestaltdb.kvstores import LMDBStore
from gestaltdb.serializers import PickleSerializer

store = LMDBStore(path="graph_lmdb", map_size=2**30)
graph_db = GraphDB(store, PickleSerializer())

LMDB keeps separate databases for nodes, edges, adjacency, typed adjacency, and sorted indexes. Increase map_size when loading large graphs.

LMDB is a good fit when you want a mature embedded backend and can provision the maximum database size ahead of time. It is not currently the optimized backend for native Arrow/Polars columnar ingestion; those paths are designed around PyRexStore when PyRex exposes write_columnar_batch.

LevelDB Backend

Use LevelDBStore when you want LevelDB through plyvel.

from gestaltdb.graphdb import GraphDB
from gestaltdb.kvstores import LevelDBStore
from gestaltdb.serializers import PickleSerializer

store = LevelDBStore(path="graph_leveldb")
graph_db = GraphDB(store, PickleSerializer())

plyvel requires compatible CPython wheels or local LevelDB build tooling. If installation fails on Python 3.14 or a free-threaded interpreter, create a Python 3.12 environment and install gestaltdb[leveldb] there.

LevelDB is the pragmatic default for modest local workloads. It supports the same GraphDB API and index semantics, but columnar ingestion falls back to Python bulk writes rather than RocksDB’s native columnar batch writer. In local benchmarks, LevelDB can be close to RocksDB when serialization dominates, but it does not get the same write-phase advantage on larger native columnar loads.

RocksDB Backend

Use PyRexStore for RocksDB through the optional pyrex-rocksdb package. This backend uses one physical RocksDB database with prefixed keys and exposes several RocksDB tuning knobs.

from gestaltdb.graphdb import GraphDB
from gestaltdb.kvstores import PyRexStore
from gestaltdb.serializers import PickleSerializer

store = PyRexStore(
    path="graph_rocksdb",
    parallelism=4,
    max_background_jobs=4,
    write_buffer_size=64 * 1024 * 1024,
    bloom_bits_per_key=10,
)
graph_db = GraphDB(store, PickleSerializer())

disable_wal=True can be useful for bulk-loading experiments, but it weakens durability and should not be used as a safe default.

When installed with pyrex-rocksdb>=0.4.1, PyRexStore can use PyRex’s native write_columnar_batch API through GraphDB.ingest_nodes_arrow and GraphDB.ingest_edges_arrow. Serialized node_value and edge_value columns backed by Arrow binary arrays are passed directly to PyRex’s native batch writer where possible. The native path still constructs RocksDB keys in GestaltDB, but avoids Python value materialization for Arrow-backed value columns.

For JSON-compatible payloads, GraphDB.ingest_nodes_polars_entities and GraphDB.ingest_edges_polars_entities can build serialized payload columns from structured Polars columns before using the same columnar write path:

import polars as pl

from gestaltdb.graphdb import GraphDB
from gestaltdb.kvstores import PyRexStore
from gestaltdb.serializers import JSONSerializer

graph_db = GraphDB(PyRexStore(path="graph_rocksdb"), JSONSerializer())
graph_db.ingest_nodes_polars_entities(
    pl.DataFrame({
        "node_id": ["n1"],
        "labels": [["Entity"]],
        "kind": ["drug"],
    }),
    property_columns=["kind"],
)

Other serializers use the same entity methods as a convenience API, but fall back to Python row-wise serialization. Columnar edge ingestion remains append-only.

RocksDB is the backend to choose when ingestion throughput matters. A targeted 100,000-node and 500,000-edge local benchmark using JSON payloads showed RocksDB with native columnar ingestion at 3.400 seconds total versus 3.948 seconds for LevelDB on the same Python JSON serialization path, a 1.16x total speedup and a 1.23x write-phase speedup. The same run showed larger gains from avoiding Python JSON serialization with Polars, so pair RocksDB with Polars/Arrow entity-column ingestion when your input data is already tabular.

Indexes

All backends implement sorted index primitives used by labels, relationship type catalogs, property lookups, and range scans. The high-level indexes maintained by GraphDB are:

• label indexes for Node.labels and GraphDB.nodes_by_label

• relationship type indexes for edge.properties["type"] and GraphDB.edges_by_type

• explicit node and edge property indexes

• composite label/property and type/property indexes

• scalar range indexes for indexed string and numeric properties

Property indexes are intentionally explicit. Register them only for predicates you expect to use frequently:

graph_db.create_node_property_index("name")
graph_db.create_edge_property_index("score")

graph_db.nodes_by_property("name", "Aspirin")
graph_db.edges_by_property_range("score", 0.8, None)

Cypher uses these indexes when possible for label/property scans and typed relationship predicates. Index definitions are persisted in backend metadata, so reopened databases continue maintaining the configured property indexes.

Columnar ingestion keeps label, relationship type, property, composite, and range indexes current for configured indexed properties.

Columnar ingestion also accepts index_mode to control when secondary indexes are updated:

IndexMaintenanceMode.MAINTAIN

Updates label, type, property, composite, and range indexes during ingestion. Use this for incremental writes or when index-backed queries must be valid as soon as ingestion returns.

IndexMaintenanceMode.DEFER

Writes canonical node and edge records plus traversal-critical typed adjacency records, marks affected secondary indexes stale, and requires an explicit GraphDB.rebuild_deferred_indexes() before indexed queries. Use this when the write window is more important than immediate index readiness.

IndexMaintenanceMode.DEFER_REBUILD

Defers secondary-index writes during ingestion and immediately rebuilds stale indexes before returning. This is the high-level default for GraphDB.ingest_arrow and GraphDB.ingest_polars.

On a targeted 100,000-node and 500,000-edge local benchmark with node property indexes on kind and group plus an edge property index on weight, deferred mode reduced the write phase from 12.269 seconds to 1.407 seconds (8.72x faster). The immediate rebuild took 12.973 seconds, so total ingest-plus-rebuild time was 14.380 seconds, which was 17.2% slower than inline maintenance for that subset. Use deferred indexing to move index work out of the critical write path; do not assume it always reduces total wall-clock time when a full rebuild runs immediately.

Backend Index Interface

Backend implementations expose lower-level sorted index methods such as put_index_entry, delete_index_entry, iter_index_prefix, and range index equivalents. Most users should prefer the GraphDB helpers above.

Transactions

GestaltDB exposes optional graph-level transactions for backends that can commit all graph records and indexes atomically:

with graph_db.transaction() as tx:
    tx.put_node(Node(node_id="n1"))
    tx.put_edge(Edge(edge_id="e1", source="n1", target="n2", properties={"type": "rel"}))

The context commits on clean exit and rolls back if an exception leaves the block. LMDBStore supports transactions directly because LMDB transactions span all named sub-databases in the environment.

PyRexStore supports transactions when opened in transaction-capable mode with pyrex-rocksdb>=0.4.1:

store = PyRexStore(path="graph_rocksdb", transactional=True)
graph_db = GraphDB(store, PickleSerializer())

The default PyRexStore mode remains the fastest non-transactional path and keeps native columnar ingestion available. Transactional PyRex writes still use RocksDB PyWriteBatch internally for bulk methods, but the current PyRex TransactionDB API does not expose the native columnar writer.

A focused local benchmark on this branch used 100,000 nodes, 500,000 edges, Pickle payloads, chunk size 10,000, four RocksDB background threads, a 64 MiB write buffer, and Bloom filters. The transaction-capable RocksDB backend was opened with PyRexStore(transactional=True) and used the same public ingestion API, not one giant user transaction.

RocksDB backend	Node write	Edge write	Native columnar
default, Arrow payloads	0.457 s	1.886 s	yes
transaction-capable, Arrow payloads	0.540 s	3.623 s	no
default, Polars payloads	0.429 s	1.828 s	yes
transaction-capable, Polars payloads	0.545 s	3.670 s	no

Use transaction-capable RocksDB when graph-level atomicity is required. Keep the default RocksDB backend for append-only bulk loads where native columnar ingestion is the priority.

The current LevelDBStore layout uses multiple physical LevelDB databases, so graph-level transactions are intentionally unsupported there.

Backend Selection Pattern

from pathlib import Path

from gestaltdb.graphdb import GraphDB
from gestaltdb.kvstores import LMDBStore, LevelDBStore, PyRexStore
from gestaltdb.serializers import PickleSerializer

def open_graph(path: str, backend: str = "lmdb") -> GraphDB:
    Path(path).parent.mkdir(parents=True, exist_ok=True)
    if backend == "lmdb":
        store = LMDBStore(path=path, map_size=2**30)
    elif backend == "leveldb":
        store = LevelDBStore(path=path)
    elif backend == "rocksdb":
        store = PyRexStore(path=path)
    else:
        raise ValueError(f"unknown backend: {backend}")
    return GraphDB(store, PickleSerializer())

Cleanup

Always close stores when a script or notebook cell is finished with them.

graph_db = GraphDB(LMDBStore(path="example_lmdb"), PickleSerializer())
try:
    graph_db.put_node(Node(node_id="n1"))
finally:
    graph_db.close()