Joseph T. French

Building an AI-Ready Financial Intelligence Platform on LadybugDB

I started on Neo4j because I knew Cypher. The prototype worked; the licensing didn't — RoboSystems is open source, meant to be forked into a customer's own AWS account, and Neo4j Enterprise would have put a toll on every deployment. I rebuilt on Kuzu; less than a week after I open-sourced, Apple acquired it and shut the project down. LadybugDB, a community fork with a maintainer committed to keeping it open source, was the remaining option and then the better one: DuckDB integration, columnar storage, Icebug, a single-file distribution model.

The first chapter was shredding the SEC XBRL corpus into a shared knowledge graph. That pipeline is still in production. What I needed next was a durable system of record so agents could operate on real books — and a graph that was a projection of that record, not the record itself.

The Problem

Every company that files annual and quarterly financial reports with the SEC does so in XBRL, a structured data format that is surprisingly difficult to work with at scale. There are more than ten thousand reporting companies, hundreds of thousands of filings going back to 2009, and hundreds of millions of individual data points. I wanted to shred all of that into a queryable knowledge graph, not just the raw numbers, but the semantic structure: what elements mean, how they relate to each other across companies, and how financial statements are constructed. And I wanted AI agents to be able to query it naturally through MCP tools without needing to write complex Cypher.

That is still the shared-repository problem, and LadybugDB still solves it: embeddable on a single EC2 instance, distributable as a file, scalable horizontally by launching read-only replicas that pull the latest snapshot from S3.

The entity-graph problem is different. Each customer gets a dedicated graph, but the books themselves live in PostgreSQL — a schema-per-tenant OLTP database with a full events ledger (including the accounting transaction ledger), a taxonomy system (accounting and reporting frameworks as data), and an information block envelope that carries semantically rich fact sets, either derived from the ledger or asserted.

QuickBooks is the first connector. The books sync down into the events ledger, and entries the platform originates — the closing entries from a period close, for instance — write back to QuickBooks as journal entries, so QB stays the customer's authoritative books. The round trip closes cleanly: a written-back entry comes down on the next sync and dedupes against its origin.

Reads are a GraphQL endpoint. Writes are named business operations, callable from the API, from MCP, or by internal agents through the same MCP tooling. One operations kernel, three transports.

The graph is not the source of truth. It is a projection of the enterprise's knowledge, rebuilt from OLTP, portable as a file, and allowed to go stale — as long as the agent can see how stale it is.

The Architecture

Two pipelines, the same last mile.

Entity graphs run PostgreSQL → DuckDB → LadybugDB. DuckDB's postgres_scanner reads the tenant schema directly on the Graph API instance and shapes each OLTP table into a graph-shaped staging table. Deduplication happens in DuckDB. The handoff to LadybugDB is an Arrow record-batch stream — DuckDB result vectors into COPY — with no intermediate file. Materialization is blue/green: build a WIP database beside the live one, then swap. OLTP writes mark the graph stale; a sensor rematerializes; an MCP tool tells the agent whether the graph is fresh or stale, why, and for how long.

The SEC pipeline is the same DuckDB → LadybugDB handoff, fed from parquet rather than Postgres. Six Dagster stages, nightly after filings stop around 9pm EST:

• Download — Discover and download XBRL filings from SEC EDGAR, partitioned by quarter and form type
• Process — Shred XBRL into parquet files using a custom processor that extracts entities, facts, elements, labels, structures, and associations
• Stage — Ingest parquet into DuckDB using glob patterns, with deduplication via GROUP BY and FIRST() aggregation
• Enrich — Generate knowledge artifacts using Icebug's graph analytics (PageRank, Core Decomposition, BFS) via zero-copy Arrow-to-CSR construction
• Materialize — COPY from DuckDB staging tables directly into LadybugDB, with batch processing for tables exceeding 20M rows
• Publish — Backup to S3 and distribute to a fleet of read-only shared replicas behind an ALB

A company like Uber files their 10-K on a Friday, and by Saturday morning it's been shredded, staged, enriched, materialized, and available for AI-powered analysis.

Why LadybugDB Works Here

DuckDB integration is first-class. For entity graphs, postgres_scanner plus Arrow COPY is how a live ledger becomes a graph without an ETL dump. For the SEC dataset, globbed parquet into DuckDB, then the same COPY, moves well over a hundred million rows at several million rows per minute. For large tables like Fact (which can exceed 100M rows), I use batched materialization with 20M-row chunks. I could not do this with a client-server graph database.

Embeddable means operationally simple — and portable. Each LadybugDB instance runs on a single EC2 node (ARM64 Graviton). The shared repository tier uses r7g.2xlarge instances (64GB RAM) for the shared master, with a fleet of read-only replicas behind an ALB on m7g.large instances. There's no cluster to manage, no replication protocol to debug. The replicas simply download the published .lbug file from S3 on startup. User graphs are dedicated; the file is the backup, the snapshot, and the thing you take with you. Download it and explore locally, or hit it remotely over MCP. That is what makes fork-and-deploy viable.

Columnar storage is the agent-facing shape. The OLTP schema is the right shape for writes and for typed GraphQL reads. The graph is the right shape for walking a calculation structure, seeing which facts sit in which statements, and asking what else an entity connects to. Agents figure that out in Cypher, then they call a named operation to change the books.

I retired the corpus-wide 384-dimension index on the SEC graph; HNSW is still there if a custom graph wants it.

Icebug: Graph Analytics That Changed Everything

The real breakthrough on the SEC corpus came when I integrated Icebug for offline graph analytics. The XBRL data contains millions of edges representing how financial elements relate to each other — calculation relationships (revenue minus expenses equals income) and presentation relationships (how statements are laid out).

I use Graph.fromCSR() to construct Icebug graphs directly from Arrow arrays exported by DuckDB, zero-copy with no per-element Python loops. From roughly 50M raw association rows (the same element relationships repeat across 100K filings), DuckDB deduplicates down to 3M unique edges, which are then fed into Icebug for three key analyses:

• PageRank identifies the most important financial elements across the entire corpus
• Core Decomposition reveals the structural backbone of financial reporting
• BFS from known roots (NetIncomeLoss, Assets, Net Cash Change) classifies every element by its primary financial statement

These analytics produce a confidence-scored classification for every concept in the graph. Combined with a small curated taxonomy of canonical concepts, that is enough to resolve a natural-language idea like revenue to the right XBRL element, then do a straightforward filtered traversal. The result: instead of requiring an LLM to generate complex Cypher on a schema with 14 node types and 21 relationship types, I can give it simple, tailored MCP tools that just provide the answer with filtering options.

Enriching the Graph After Construction

The biggest strides on the SEC graph still aren't in the raw data pipeline, they're in enriching the graph after it's built. The philosophy is simple: you can't classify what's in the graph until you can see the full picture. Trying to infer structural patterns during construction, when you're processing one filing at a time, doesn't work. You need the complete graph first.

So after shredding the XBRL, I spin up a temporary LadybugDB instance, load the filing contents into the graph, and run Cypher queries that identify what things actually are: roll-ups, roll-forwards, hierarchies, disclosure patterns. From those patterns, I create new Classification and FactSet nodes that make the graph dramatically easier for downstream tools to navigate. After exporting the enriched nodes to parquet, I then throw the temporary graph away. That's a pattern that only works with an embedded database. You can't justify standing up a Neo4j cluster for a short-lived enrichment job.

This is where the real value is. A graph full of raw XBRL data is useful. A graph where every structure has been classified, every element has been scored by importance, and every concept has been mapped to a canonical meaning is something an AI agent can actually work with reliably.

Beyond Shared Data: User Graphs, Connectors, and Agents

The SEC dataset proves the pattern. The long-term potential is in user graphs, and that is now a working system, not a plan.

RoboSystems uses the same infrastructure to give every customer their own dedicated LadybugDB instance. Accounting systems like QuickBooks connect through OAuth; the pipeline lands in the OLTP ledger, then DuckDB stages and LadybugDB materializes. Customers can still upload parquet, stage in DuckDB, and materialize into their own graph. Subgraphs — isolated databases on the same instance — remain available for workspaces, experiments, or AI memory.

The finding that changed how I build: agents strongly prefer the graph for understanding the context they're operating in, then act on the OLTP layer. They walk calculation structures, inspect mappings, check which facts sit in which statements, then they call a named operation — close a period, write a journal entry — through the same MCP tools a human would use.

I have now run three real month-end closes for my own companies' books entirely through MCP, on LadybugDB 0.18.1. The agent does the heavy lifting; I review rather than drive. The UI is mostly a view layer at this point. The magic is in the agentic workflows.

Because the graph is a projection, it can go stale. Blue/green materializations fire off staleness triggers. Before an agent trusts a number it saw in Cypher, it checks how fresh the graph is — source-system sync on one edge, OLTP-to-graph lag on the other. That is what makes a stale projection safe to give an agent.

The projection being a file matters here too. The same .lbug the replicas pull from S3 is the thing you download and explore locally. Remote MCP and local exploration are the same artifact.

The Distribution Model

RoboSystems is open source. The entire codebase — the SEC pipeline, the graph API, the MCP tools, the OLTP kernel — is designed to be forked and deployed into a customer's own AWS account. The infrastructure is defined in CloudFormation templates parameterized by a single graph.yml config file. GitHub Actions handles deployment. There's no managed service vendor lock-in.

This only works because LadybugDB is embeddable. If the graph engine required a separately managed cluster, the fork-and-deploy model would collapse under operational complexity. Instead, each tier is a single EC2 instance running the Graph API with LadybugDB embedded — the same Docker image and deployment pattern whether you're running a 5M-node entity graph or a 200M-node SEC replica, just on a bigger instance.

What I've Learned

Building on LadybugDB, a few things stand out:

The graph as a projection, not a system of record, is the architecture that made agents work. I spent a long time trying to make the graph durable enough to be the books. The books need a ledger; the agents need a graph. LadybugDB is extraordinary at the second job once you stop asking it to be the first.

The DuckDB-to-LadybugDB pipeline is still the killer feature — and postgres_scanner made it a live ledger pipeline, not just a parquet pipeline. Being able to stage, reshape, enrich, and deduplicate in DuckDB, then materialize directly into the graph without an intermediate serialization step, is what makes both the SEC corpus and a customer's QuickBooks books travel the same last mile. I couldn't do this with a client-server graph database.

Icebug completes the analytical graph. Raw structure plus PageRank, core decomposition, and BFS-derived classifications — together with a small curated taxonomy — is what let me build SEC tools that just work, instead of tools that sometimes work.

The agent wants two stores, and it wants to know the lag between them. Graph for context, OLTP for action, one freshness check so it doesn't lie to itself. That is the design I would start with if I were beginning today.

Operational simplicity compounds. Every operational decision I didn't have to make — no cluster management, no replication protocol, no connection pooling theatre on the graph side — freed up time to build the OLTP kernel, the MCP surface, and the close workflow. Replicas are files. Blue/green is a file swap. The constraint of simplicity pushed better architecture.

I'm committed to supporting LadybugDB and calling it out at every opportunity because I believe this is a foundational open-source technology. The combination of an embedded columnar graph engine with native DuckDB integration and Icebug analytics is genuinely unique — and it's what makes an agentic close on real books possible.