Dataframe — Columnar Data in U
U's Dataframe is not a library — it is a compile-time template tag that builds typed, columnar query plans. The compiler sees the entire pipeline and optimizes it: predicate pushdown, projection pruning, stage fusion, and GPU kernel emission. The same .on() / .filter() / .map() / .reduce() chain that works on lists works on Dataframes, with the modifier system controlling execution strategy.
Construction
Dataframes are built from columns, CSV, JSON, or the backtick template:
// From columns:
df = Dataframe.from_columns({
"name": ["Alice", "Bob", "Carol", "Dave"],
"age": [30, 25, 35, 28],
"salary": [70000, 55000, 90000, 62000]
})
// From CSV (async):
df = Dataframe.from_csv("data.csv") +A
// From JSON rows:
df = Dataframe.from_json([
{"name": "Alice", "age": 30},
{"name": "Bob", "age": 25}
])
// Backtick template with interpolation:
min_age: I = 25
cols = ["name", "salary"]
df2 = Dataframe`select {{cols}} from {{df}} where age > {{min_age}}`
The backtick form produces a Formats.Dataframe — a typed query plan, not a string. The compiler's __validate__ dunder checks column names, types, and predicates at compile time. Interpolated variables ({{cols}}, {{min_age}}) are type-checked against the schema.
Schema
Every Dataframe carries a compile-time schema. The compiler knows column names and types:
d UserRow
name: S
age: I
salary: I
active: B = true
df = Dataframe[UserRow].from_csv("users.csv")
// Compiler knows: df has columns name:S, age:I, salary:I, active:B
// df["email"] → compile error: 'email' is not a column in UserRow
When no schema type is given, the Dataframe infers column types from the data (first row for CSV, all rows for JSON).
Pipeline Operations
Dataframes use the same .on() primitives as lists. Every operation returns a new Dataframe (immutable by default):
Selection
df.select(["name", "salary"]) // keep only these columns
df.drop(["active"]) // remove columns
df.head(10) // first 10 rows
df.tail(5) // last 5 rows
df.slice(100, 200) // rows 100–199
df.sample(50) // random 50 rows
Filtering
df.filter(row => row["age"] > 30)
df.where("salary", ">", 60000)
df.filter(row => row["name"].starts_with("A") & row["active"])
Transforms
df.with_column("bonus", row => row["salary"] * 0.1)
df.rename("salary", "annual_pay")
df.sort("age", desc: true)
df.unique("name")
df.cast("age", "N") // I → N
Aggregation
df.sum("salary") // → I
df.mean("age") // → N
df.min("salary") // → I
df.max("salary") // → I
df.count() // → I
// GroupBy:
df.groupby(["active"])
.agg("salary", "mean") // mean salary per group
.agg("age", "max") // max age per group
.collect()
Joins
orders = Dataframe.from_csv("orders.csv")
users = Dataframe.from_csv("users.csv")
joined = users.join(orders, on: "user_id", how: "inner")
cross = users.cross_join(products)
Column Expressions
Direct column access returns a Column — a typed array with element-wise operations:
col = df["salary"] // Column[I]
col.sum() // → I
col.mean() // → N
col * 1.1 // → Column[N], element-wise
col > 60000 // → Column[B], boolean mask
// Column arithmetic:
df.with_column("tax", df["salary"] * 0.22)
df.with_column("net", df["salary"] - df["salary"] * 0.22)
Modifiers
The modifier system controls how the pipeline executes:
+W — Lazy Evaluation
// Nothing executes until .collect():
result = df
.filter(row => row["age"] > 30)
.select(["name", "salary"])
.sort("salary", desc: true)
.head(10)
+W // lazy: build plan only
// Execute the plan:
result.collect()
// The compiler applies:
// 1. Predicate pushdown: filter before sort
// 2. Projection pruning: only read name, salary, age columns
// 3. Limit pushdown: stop after 10 rows
+V — Vectorized / GPU
// CPU vectorized (SIMD):
total = df["salary"].sum() +V
// GPU compute shader:
df.with_column("norm", df["value"] / df["value"].max()) +V
// → compiler emits WGSL:
// @compute @workgroup_size(64)
// fn main(@builtin(global_invocation_id) id: vec3<u32>) {
// outp[id.x] = inp[id.x] / max_val;
// }
// Fused GPU pipeline:
result = df
.filter(row => row["price"] > 100)
.with_column("margin", row => row["price"] * 0.15)
.sum("margin")
+V
// → one GPU kernel: filter + multiply + reduce
+R(GPU) — GPU-Resident Data
// Load data to GPU once:
gpu_df = df +R(GPU)
// All operations stay on GPU — no CPU↔GPU transfer:
filtered = gpu_df.filter(row => row["value"] > threshold) +V
transformed = filtered.with_column("scaled", col => col * factor) +V
total = transformed.sum("scaled") +V
// One transfer at the end:
result = total.to_cpu()
Printing
Dataframes print as aligned tables:
print(df.head(3))
// ┌──────┬─────┬────────┐
// │ name │ age │ salary │
// ├──────┼─────┼────────┤
// │ Alice│ 30 │ 70000 │
// │ Bob │ 25 │ 55000 │
// │ Carol│ 35 │ 90000 │
// └──────┴─────┴────────┘
The __string__ dunder on Dataframe produces the table format. __pack__ produces a Tree (JSON-serializable). Dataframe.from_tree(tree) round-trips.
Tensor Integration
Dataframe columns can be extracted as Tensors for numerical computation:
// Column → Tensor:
ages = df["age"].to_tensor() // Tensor[I], shape [n]
salaries = df["salary"].to_tensor() // Tensor[I], shape [n]
// Tensor operations:
correlation = ages.dot(salaries) / (ages.norm() * salaries.norm())
// Tensor → Column:
df.with_column("predicted", Tensor.matmul(weights, features).to_column())
Compile-Time Validation
The __validate__ dunder runs at compile time on Dataframe backtick templates:
Dataframe`select name, agee from {{users}}`
// ^^^^
// Compile error: column 'agee' not in schema UserRow
// Did you mean: 'age'?
Dataframe`select name from {{users}} where salary > "high"`
// ^^^^^^
// Compile error: cannot compare salary (I) with "high" (S)
This is the same mechanism that makes SQL catch injection and HTML catch XSS — the template tag's __validate__ sees the full expression and the schema at compile time.
Implementation Notes
Storage: Columns are contiguous typed arrays ([I], [N], [S], [B]) with 64-byte aligned allocation for SIMD. The Dataframe struct holds a {S: Column} map plus schema metadata.
Memory: Dataframes are +R (heap, refcounted) by default. Column data is shared between Dataframes that select subsets (zero-copy slicing via offset + length).
Serialization: __pack__ produces {"columns": {"name": [...], "age": [...]}, "schema": {...}}. CSV and JSON I/O use streaming parsers that allocate columns incrementally.
Pipeline fusion: When the compiler sees df.filter(p).map(f).reduce(r), it generates a single fused loop instead of three passes. With +V, it generates a single GPU kernel. With +W, it builds a pull-based iterator that processes one chunk at a time.
GPU dispatch: Column operations with +V emit WGSL compute shaders via the same codegen/wgsl.py emitter used for [I].map(). The +R(GPU) modifier keeps columns in WebGPU storage buffers between pipeline stages, avoiding CPU↔GPU round-trips.