Arrow and worker queries

Query before rendering

Existing rows and dataset() stay synchronous. The optional query layer moves filtering and aggregation to a worker-backed source, then gives React a bounded result. It does not turn every source row into a React object or chart mark.

The worker explorer uses deterministic Arrow IPC, category totals, metrics, a 25-row detail page and bounded scatter marks. Its first matching IDs are not a representative sample.

Install peers and own the worker

First build and pack the unpublished library. Install the verified optional peer pair only when using the DuckDB adapter:

pnpm add @duckdb/duckdb-wasm@1.32.0 apache-arrow@17.0.0

Use worker and Wasm files from the same installed DuckDB release. The gallery uses Vite asset imports and single-threaded MVP/EH bundles without cross-origin isolation; other bundlers need equivalent handling.

Copy and adapt the tested gallery initializer for complete startup. It selects compatible assets, fetches Wasm on the host with HTTP/abort checks, supplies a temporary blob URL, races worker errors and an initialization deadline, and releases resources on setup failure. A failed Wasm fetch inside the DuckDB 1.32.0 worker can otherwise leave instantiate() pending.

After startup succeeds, this wrapper transfers the initialized database and worker to the adapter:

import type { AsyncDuckDB } from '@duckdb/duckdb-wasm';
import { createDuckDBBackend } from '@quartile/react/duckdb';

// The application has already initialized this database and worker.
export function openBackend(database: AsyncDuckDB, worker: Worker) {
  return createDuckDBBackend({
    create: async () => ({
      database,
      dispose: async () => {
        try { await database.terminate(); }
        finally { worker.terminate(); }
      },
    }),
  });
}

Serve Wasm as application/wasm. The application's CSP must allow worker/Wasm execution and the temporary blob URL used during initialization. The gallery serves assets from its own origin; no library-owned CDN or remote database is required. Initialize outside rendering and dispose resources created after an effect has been canceled.

Ingest once

import { tableFromArrays } from 'apache-arrow';

const backend = await openBackend(database, worker); // Initialized by the application.
const source = await backend.fromArrow(tableFromArrays({
  region: ['Europe', 'Asia', 'Europe'],
  revenue: new Float64Array([120, 80, 60]),
}), {
  id: 'orders',
  fields: { revenue: { label: 'Revenue', format: 'currency', currency: 'USD' } },
});

fromArrow accepts an Arrow Table or Uint8Array IPC bytes, copies them into a private relation, and leaves the input caller-owned. IPC decoding, re-encoding and ingestion can allocate additional buffers; this is not a zero-copy API. The schema comes from Arrow's physical types rather than a sample of returned rows.

Link aggregates and exact detail pages

import { Selection } from '@quartile/react';
import { QueryBarList, QueryDataTable, QueryKPI,
  type QuerySource } from '@quartile/react/query';

export function OrderViews({ source }: { source: QuerySource }) {
  return <Selection>
    <QueryKPI data={source} value="revenue" label="Revenue" />
    <QueryKPI data={source} aggregate="count" label="Matching orders" />
    <QueryBarList data={source} category="region" value="revenue" select />
    <QueryDataTable data={source} pageSize={25} defaultSort="-revenue"
      columns={[{ field: 'region' }, { field: 'revenue' }]} />
  </Selection>;
}

QueryBarList supports sum/count, returns at most 20 categories by default and reports truncation. QueryKPI supports sum, count, mean, min and max over matching raw records. QueryDataTable sends sorting and paging to the worker, displays the matching total and resets to the first page when another view changes the filter. Its cell bars and shares describe the current page.

Build a bounded custom chart

import { ScatterPlot } from '@quartile/react';
import { QueryResultView, useDataQuery,
  type QuerySource } from '@quartile/react/query';

export function RequestScatter({ source }: { source: QuerySource }) {
  const query = useDataQuery(source, {
    kind: 'rows',
    fields: ['id', 'latency', 'bytes', 'service'],
    orderBy: [{ field: 'id', direction: 'asc' }],
    window: { offset: 0, limit: 2000 },
  }, { id: 'request-scatter' });

  return <QueryResultView query={query}>
    {(data, result) => <>
      <p>{data.rows.length} points of {result.totalRows} matching requests</p>
      <ScatterPlot data={data} id={query.sourceId} x="bytes" y="latency"
        color="service" select="service" typedSelection renderer="canvas"
        aria-label="Latency against response size" />
    </>}
  </QueryResultView>;
}

useDataQuery excludes its own publisher's predicates and applies all others before grouping or limiting. Pass its returned sourceId to the selecting chart. Keep the chart inside QueryResultView: this prevents already-consumed filters being reapplied to a bounded or aggregated result, while retaining publication and highlighting.

typedSelection preserves physical field identities in custom ScatterPlot or DataTable results, including ISO-looking nominal strings and nulls. BarList prepared accepts one numeric row per typed category without regrouping; QueryBarList configures this automatically. Display scalar query results as static KPI values. Counting grouped rows would count groups.

The hook returns idle/loading/ready/error, hides old results when inputs change, and ignores superseded successes and failures. retry() repeats the request. durationMs includes backend queueing, execution, transfer and decoding; it is not engine CPU time.

Plans, bounds and result counts

The adapter caps row/aggregate output at 10,000 and histogram output at 1,000 bins, rejecting unbounded requests. complete means the result covers everything from offset zero. Requests and results carry IDs and immutable source revisions; mismatches are rejected.

Physical types and matching

• Equality/inclusion follow the Arrow physical type. Date-looking strings in nominal fields stay strings. Temporal values accept epoch milliseconds, Date or ISO date strings; date-only strings use UTC midnight here. Synchronous row predicates retain their local-date convention.
• Null equality and inclusion are explicit. Empty inclusion matches nothing. Ranges include both ends and support null open endpoints.
• Numeric aggregates require numeric fields. Empty sum is zero; empty mean/min/max is null. Overflowing nonfinite aggregate results reject the query; raw float rows retain their physical values. Numeric-looking strings and booleans are not numeric measures.
• Count without a field counts records; count with a field excludes null and empty strings. Histogram bins include their lower edge, exclude the upper edge, and include the final edge.
• Text order is binary lexical, nulls sort last, and raw pages break remaining ties by original input order. Ordinary in-memory DataTable sorting is locale-aware.

Supported inputs are integers up to 32 bits, Float32/Float64, booleans, strings and string dictionaries, and dates/timestamps at millisecond precision or coarser. Exact decimals, 64-bit input integers, nested types and sub-millisecond timestamps are rejected. Convert deliberately before import. Metadata overrides can change labels, formats, currency and units, but not physical types.

Dispose, cancel and define application boundaries

source.dispose() drops that source's private relation. backend.dispose() disposes imported sources and adapter-owned connections. Both are idempotent. A create factory owns cleanup; a supplied database stays caller-owned, and a supplied connection is borrowed for exclusive use while the adapter is active.

Aborted requests reject promptly and queued work is skipped. Active execution may continue in the worker, retaining its execution lock until cleanup settles. Stale-result suppression is not a guarantee of engine interruption. With an owned create factory, backend disposal immediately invokes the required database/worker termination without waiting for pending SQL. This releases owned relations and connections even after fatal worker errors leave request promises unresolved. Monitor worker errors and create a fresh backend to recover.

Borrowed database/connection disposal remains graceful: it waits for active work before dropping private relations and closing adapter-created connections. A fatal worker failure can leave this wait pending; the caller must terminate a failed borrowed worker. Individual source disposal also waits for active work. Use owned backend disposal to recover from a failed owned worker. Pass exactly one of create, database or connection to the backend constructor.

Plans contain no SQL, URLs or relation names. The adapter validates fields, quotes identifiers and binds values. The application still owns authorization and may only ingest data the current user can access. Generated dashboard specs currently use synchronous datasets; they do not create connections or query SQL.

Read Canvas and WebGL scatter and Performance for rendering and measurement boundaries. Public source guide.