dataforge

A resident in-memory data/math engine for the Wippy runtime. Each open dataset becomes its own process (a process.lua actor with a stable PID + inbox) that loads the data once into its own memory and answers every query in-process — you talk to the dataset over messages and the data never leaves it. It loads CSV / Parquet / HDF5 / Excel from a filesystem, computes exact streaming statistics, runs k-means, exposes a composable query DSL, and renders charts via a separate visualization WASM. Wippy agents and the Keeper MCP can drive it.

Architecture

• The dataset is the actor. df.open(path) spawns a dataforge:dataset process holding {schema, exact per-column stats, a bounded reservoir} in its own memory; every op is computed inside that process — no per-op data crossing; df.close() ends it. (funcs are stateless and an inline pool has no stable identity, so the data-holder must be a process.)
• dataforge:parse is a stateless one-shot function.wasm (Rust) that reads the file and returns the bounded {schema, stats, reservoir} — called once at load, holds nothing.

SDK

local df = require("dataforge")
df.open("/data/iris.csv")              -- CSV, .parquet, .h5, or .xlsx
df.schema()                            -- columns + inferred types
df.describe()                          -- count/null/min/max/mean/std (exact, streaming)
df.sample(n)                           -- first n rows
df.histogram("petal_length", bins)     -- equal-width buckets
df.correlation({ "a", "b" })           -- Pearson correlation matrix
df.query():filter("a", ">", 3)         -- composable pipeline, one batched call
         :select({ "a", "b" }):limit(10):collect()
df.kmeans({ k = 3, cols = { "a", "b" } })
df.cluster_plot({ k = 3, cols = { "a", "b" } })  -- k-means -> viz SVG
df.close()

The loaded dataset persists across calls until the next open or a close.

Components

• dataforge:dataset — the dataset IS this process.lua actor (stable PID + inbox); holds the data in Lua and computes every op in-process.
• dataforge:parse — stateless one-shot function.wasm parser (CSV/Parquet/HDF5/xlsx → bounded schema+stats+reservoir); called once at load.
• dataforge:viz — stateless function.wasm rendering SVG scatter plots.
• dataforge:dataforge — the Lua SDK.
• dataforge:tool — a meta.type: tool entry exposing the full action surface to agents and the Keeper MCP.

Formats

• CSV — streaming / out-of-core (validated on a 2M-row file).
• Parquet — streaming / out-of-core (row-group batches; a custom WASI ChunkReader). A 2GB-decoded file opens in ~tens of MB.
• HDF5 (.h5/.hdf5) — out-of-core; opened by path (seeked, never whole-loaded) and read one column at a time. Top-level datasets are read as columns.
• Excel (.xlsx) — streaming / out-of-core; the first sheet is read cell-by-cell (seeked over WASI, never whole-loaded), its first non-empty row is the header. Peak memory is the shared-strings table + reservoir, independent of row count.

Statistics (describe) are exact over all rows; sample/histogram/correlation/ query/kmeans operate on a bounded reservoir sample and flag sampled for large inputs.

Requirements

Runs on a Wippy runtime with WASI preview2 filesystem/clocks support for Rust-std components. See the source repository for the command protocol and architecture docs.