How RisingWave Uses Apache DataFusion for Faster Iceberg Analytics

Building a high-performance analytical query engine requires a lot of common but costly work, including SQL planning, query optimization, execution, file reading, memory management, scheduling, and parallel processing. These capabilities are essential for data systems, but they are not usually what makes a database product unique. Apache DataFusion provides a reusable and embeddable foundation, similar to how LLVM provides shared compiler infrastructure. Data teams can start with an existing engine, customize it through APIs, and focus on building the features that make their products unique. DataFusion is a Rust-based analytical query engine built on Apache Arrow. Queries can enter through SQL, DataFrame APIs, custom front ends, or logical-plan builders. DataFusion converts them into logical plans, optimizes those plans, and then creates an ExecutionPlan. During execution, it processes data as asynchronous streams of Arrow RecordBatch values. Its pull-based, Volcano-style model uses partitions for parallel execution, exchanges for cross-core communication, sort-order metadata to avoid unnecessary sorting, memory pools for cooperative memory tracking, and Tokio for asynchronous scheduling.

DataFusion provides reusable query-engine components that projects can use on their own or combine with existing planners, catalogs, execution engines, and front ends. Apache Arrow makes data exchange and reuse easier, while Rust provides memory and thread safety. This allows data systems to build on a shared foundation and focus on the features that make them unique.

RisingWave follows this model. It continues to specialize in stream processing via its stream execution engine while using DataFusion as a batch engine for Iceberg tables. RisingWave can continuously ingest and transform live data into Iceberg tables, then use DataFusion to run supported analytical queries over those tables without requiring users to switch to a separate analytical system.

Why DataFusion changes database development

Every database has unique requirements, but many parts of query processing are common.

A time-series database, geospatial engine, stream processing engine, observability system, and lakehouse query engine may all need:

• Expression representations

• SQL functions

• Query optimizations

• Parallel execution plans

• File-format support

• Joins and aggregations

• Arrow interoperability

Building and maintaining all these components independently requires significant engineering effort. DataFusion provides them in an extensible form, allowing systems to reuse the query engine while implementing their own storage model, catalog, domain-specific functions, query language, or execution operators.

This creates a new development paradigm:

Build on common query-processing foundation, then focus on what makes the system specialized for its particular workload.

That is also why DataFusion is not a complete replacement for the systems that embed it. It provides the analytical foundation, but each system still decides how data is stored, how queries are planned and which workloads should use DataFusion.

Data systems using Apache DataFusion

Many data systems use Apache DataFusion, including:

• InfluxDB: Time series database

• LanceDB: Vector database

• GreptimeDB: Distributed time series database

• OpenObserve: Observability platform

• Ballista: Distributed SQL query engine

• Comet: Native query execution plugin for Spark

• R2 SQL: Distributed query engine

• ParadeDB: PostgreSQL for search and analytics

• SedonaDB: Analytical database with geospatial support

• Vortex: Columnar data format

• Spice AI: Query engine for AI apps and agents

• Restate: Durable applications platform

• Cube Store: Semantic layer platform for Cube

• Funnel: Data platform for marketing intelligence

The ecosystem of DataFusion-based systems is growing fast.

These projects use DataFusion in different ways. Some use it as the core of a distributed SQL engine, while others use it for time-series analytics, observability, search, geospatial processing, Spark acceleration, streaming, or file-format tools.

What they share is a common architectural idea:

Apache DataFusion handles common query-processing tasks, allowing each project to focus on the needs of its specific use case.

For RisingWave, those specialized requirements include streaming-native execution, incremental stateful computation, and a unified experience across live data streams and historical Iceberg data.

Why RisingWave adopted DataFusion

RisingWave was originally built for stream processing. Its native engine is optimized for workloads such as:

• Incremental computation

• Streaming joins

• Continuously maintained materialized views

• Stateful stream processing

• Low-latency updates and serving

But, a streaming lakehouse must do more than continuously write data to Apache Iceberg. Users also need to query historical Iceberg data efficiently. Without an integrated analytical path, the workflow becomes:

This means operating another query system, maintaining a separate execution environment, and switching between different SQL interfaces.

RisingWave introduced a DataFusion-powered query path for Iceberg tables in RisingWave 2.8. In RisingWave 3.0, DataFusion became the default engine for supported batch queries, while RisingWave’s native engine continued to handle streaming and unsupported workloads.

The goal was not just to add a faster batch engine. It was to clearly divide the responsibilities:

The big-picture architecture

RisingWave does not send the original SQL statement directly to DataFusion. It first handles the main database tasks:

1. Parse the SQL.

2. Resolve tables, columns, functions, and data types.

3. Build a RisingWave logical plan.

4. Optimize the logical plan.

5. Check whether DataFusion supports the plan.

6. Convert the plan into DataFusion’s format.

7. Let DataFusion optimize and execute it.

8. Convert the results back into RisingWave’s format.

RisingWave represents this decision through BatchPlanChoice, which can contain either a native RisingWave plan or a DataFusion plan.

The complete path can be summarized as:

Step 1: RisingWave parses and plans the query

When a query arrives, RisingWave handles it first. It resolves the tables, columns, functions, and expressions, then creates and optimizes its own logical plan. Only after that does RisingWave check whether DataFusion can run the query. This keeps one SQL and planning layer for both streaming and batch queries.

Step 2: RisingWave decides whether DataFusion can execute the plan

After optimization, RisingWave uses DataFusionExecuteChecker to review the full logical plan. It checks whether every part of the plan is supported and whether the query includes an Iceberg scan.

DataFusion can run the query only when all plan nodes are supported and the plan contains at least one Iceberg scan. Supported nodes include filters, projections, aggregations, joins, limits, Top-N, unions, window functions, deduplication, values, Project Set, Expand, and Iceberg scans.

Some operator variants are still unsupported, such as aggregations with grouping sets, Top-N with WITH TIES, and Top-N with a group key. If any part of the plan is unsupported, RisingWave uses its native engine instead. Queries without an Iceberg scan also stay on the native path.

Step 3: RisingWave converts its logical plan into a DataFusion plan

Once a query is eligible, RisingWave converts each node in the logical plan into the matching DataFusion node. If DataFusionPlanConverter finds an unsupported plan node or expression, it returns an error.

Step 4: RisingWave extends DataFusion where needed

Some RisingWave logical nodes, such as Expand and ProjectSet, do not have built-in equivalents in DataFusion. RisingWave represents them as DataFusion extension nodes and provides extension planners that convert them into physical operators.

During physical planning, DataFusion’s DefaultPhysicalPlanner handles the plan. RisingWave registers extension planners only for RisingWave-specific logical nodes such as Expand and ProjectSet. This allows RisingWave to use DataFusion’s execution framework while keeping its own operators and SQL behavior.

Step 5: RisingWave turns Iceberg scans into DataFusion table scans

When RisingWave finds an Iceberg scan, it creates an IcebergTableProvider containing the Iceberg connection settings, Arrow schema, and file-scan tasks.

This provider connects RisingWave with DataFusion. When DataFusion requests the data, it creates RisingWave’s custom IcebergScan operator.

Step 6: Iceberg files are divided into parallel scan tasks

The custom IcebergScan works as a DataFusion execution operator. RisingWave groups Iceberg file tasks based on the available parallelism and any query limit. Each group becomes a DataFusion execution partition.

The old path had to convert Arrow data into RisingWave’s format before processing it. At scale, this repeated conversion can use a lot of CPU. DataFusion works with Arrow directly, reducing this extra processing.

Step 7: DataFusion performs its own optimization and physical planning

RisingWave first plans and optimizes the query. It then converts the plan into DataFusion’s format. DataFusion analyzes the converted plan, applies its own optimizations, and creates the physical execution plan.

RisingWave handles SQL semantics and database-level planning, while DataFusion applies optimizations for its execution engine.

Step 8: RisingWave creates a managed DataFusion runtime

RisingWave creates a DataFusion SessionContext to run the query. It includes a bounded memory pool and a temporary directory for operators that support spilling, DataFusion’s default features, and RisingWave-specific extension planners.

RisingWave also passes its own settings to DataFusion:

• Batch parallelism sets the number of DataFusion partitions.

• Chunk size sets the DataFusion batch size.

• Hash-join settings configure DataFusion’s join optimization.

This keeps DataFusion connected to RisingWave’s resource and execution settings.

Step 9: DataFusion executes Arrow batches

DataFusion runs the physical plan and returns Arrow RecordBatch results. For each batch, RisingWave checks for cancellation, converts it into a DataChunk, matches it to the expected schema, and sends it to the client.

RisingWave still manages timeouts, cancellations, PostgreSQL result types, output formats, query metrics, and final response delivery.

The client uses the same RisingWave connection, no matter which engine runs the query.

Why RisingWave does not use DataFusion for every workload

DataFusion is now an important part of RisingWave’s batch architecture, but it is not suitable for every workload.

Streaming needs a different engine

RisingWave’s native engine is built for continuously changing data. Streaming joins maintain state as new events arrive, while materialized views update incrementally. The native engine also handles checkpoints, recovery, watermarks, and continuous consistency.

DataFusion is used for analytical batch queries, while RisingWave’s native engine continues to handle streaming joins, incremental computation, and materialized views.

The full batch plan must be supported

RisingWave currently chooses one engine for the entire query. It does not split a query between RisingWave and DataFusion.

If any operator or expression is unsupported, RisingWave runs the query with its native batch engine. This keeps execution predictable and correct, while DataFusion support expands over time.

RisingWave keeps one SQL layer

RisingWave controls SQL semantics and initial planning for both streaming and batch queries. This prevents differences in type handling, null behavior, decimal calculations, and function results.

Some data conversion between Arrow RecordBatch and RisingWave DataChunk is still required, but it helps provide a consistent SQL experience across live streams and historical Iceberg data.

Memory and planning still matter

DataFusion performs well when joins and aggregations fit in memory. Large intermediate results can still cause out-of-memory errors when spilling is not sufficient.

Query planning is also important. A poor join order can create large intermediate results and increase memory use. RisingWave therefore needs both fast DataFusion operators and effective logical planning and memory management.

New TPC-H 100GB results: July 2026

The latest benchmark was run on July 7, 2026, using RisingWave v3.0.1.

The setup was:

Completion rate

In the July benchmark:

• RisingWave’s previous native batch engine completed 16 of 22 queries.

• DataFusion completed 20 of 22 queries.

• DuckDB completed all 22 queries.

DataFusion ran out of memory on Q5 and Q18. RisingWave’s native engine ran out of memory on Q5, Q7, Q8, Q9, Q18, and Q21.

This means DataFusion completed four more queries than the native engine under the same RisingWave memory limit.

Overall performance

RisingWave’s native engine and DataFusion both completed 16 queries.

Across those queries:

• Native engine total time: 4,536.198 seconds

• DataFusion total time: 1,668.953 seconds

• DataFusion was 2.72× faster overall

• DataFusion was faster on 15 of the 16 queries

Q2 was the only exception:

• Native engine: 149 seconds

• DataFusion: 165 seconds

Several queries showed particularly large improvements:

These results align with the earlier observation that DataFusion performs especially well on scan-heavy, join-heavy, and aggregation-heavy workloads.

How the July results compare with previous runs

The same 100GB benchmark was also run on January 20 and February 13, 2026. For a fair comparison, only queries completed by both RisingWave’s native engine and DataFusion are included.

Two trends stand out. First, DataFusion completed more queries over time, increasing from 18 in January to 20 in February and July. The native engine improved from 14 to 15 and then 16 completed queries. Second, DataFusion’s biggest advantage came in July, when it was 2.72× faster overall, compared with 2.16× in January and 1.40× in February. The July results clearly show that DataFusion performed better than RisingWave’s previous native engine.

What the benchmark shows, and what it does not

The results show clear benefits from using DataFusion for Iceberg analytical queries:

• More queries complete within the memory limit.

• Most shared queries run faster.

• The biggest gains appear in scans, joins, and aggregations.

• Arrow-native execution reduces data-conversion costs.

But Q5 and Q18 still ran out of memory in the July DataFusion benchmark. These failures mainly point to plan-quality issues, such as inaccurate or missing cardinality estimates that can lead to poor join ordering and rapid growth in intermediate results. Therefore, improving join planning and statistics propagation should be the main priority, while spilling should remain a fallback mechanism rather than the primary solution.

Earlier 10GB tests also showed that DataFusion was not faster for every query. At that time, Top-N queries were unsupported, types such as DECIMAL and INTERVAL could slow down some queries, Q17 took more than five minutes, and Q20 was slower with DataFusion.

DataFusion should therefore be seen as an improving execution path, not as one engine that solves every workload.

Conclusion

Apache DataFusion has changed how data systems are built. Instead of creating every query-engine component from scratch, teams can use a shared and extensible foundation. DataFusion provides fast, Arrow-native analytical execution, while each system keeps control of its own storage, SQL behavior, optimization, and user experience.

RisingWave divides the work between two engines. It uses DataFusion for supported Iceberg analytical queries and its native engine for streaming workloads, unsupported batch queries, and operations that require RisingWave-specific behavior.

The goal of the DataFusion integration is not just to make Iceberg queries faster. It is to combine RisingWave’s streaming-native foundation with a fast analytical engine while keeping one consistent SQL experience across real-time and historical data.