Overview
Backfilling is the process of using historical data to build or correct the state of a data system. In broader data engineering, it may involve filling missing records, correcting stale or incorrect data, recovering from pipeline downtime, reprocessing data after a logic change, or initializing a new pipeline from data that already exists. Although it may sound like simply rerunning a job, safe backfilling requires careful control because it can overwrite newer data, create duplicates, consume substantial resources, and affect downstream tables, materialized views (MVs) and dashboards.
In RisingWave, backfilling is a fundamental part of creating materialized views on top of existing data. When a materialized view is created, RisingWave first processes the historical data available from its upstream sources and builds the initial query state. After this historical phase is complete, the materialized view transitions to incremental streaming and continuously updates as new changes arrive. RisingWave supports multiple backfilling strategies and optimizations, including snapshot backfill, arrangement backfill, locality backfilling, and serverless backfilling, to make this process more consistent, efficient, observable, and isolated from live streaming workloads.
What Is Backfilling?
In general data engineering terms, backfilling means going back and processing historical data that is missing, incorrect, stale, or required by a newly created pipeline. A team may need to backfill because a pipeline was unavailable for a period of time, a transformation contained a bug, source data was corrected, business logic changed, or a new destination needs to be populated with historical records. Depending on the scope, a backfill may cover a few specific records, a date range, selected partitions, or the entire historical dataset.
A backfill is not always limited to inserting missing rows. It may also involve replacing incorrect records, recalculating values, rebuilding a table, populating a new column, or recreating downstream state using updated logic. For that reason, backfills should be treated as controlled production changes rather than ordinary reruns. Safe backfills should have a clearly defined scope, isolated processing, idempotent writes, validation before publication, and a rollback path. The objective is to correct or rebuild historical data without exposing users to inconsistent, duplicated, or partially updated results.
Backfilling in RisingWave
In RisingWave, backfilling is the initialization phase that occurs when a materialized view is created on top of upstream data that already exists. Suppose an upstream table contains millions of historical orders and a new materialized view calculates total sales per customer. RisingWave must first read those existing records, apply the query, build the aggregation state, and materialize the current result. Only after that initial state is complete can the materialized view continue with normal incremental processing as new data arrives.
This process is important because a streaming query must be correct and complete from the moment it becomes available. Without backfilling, a newly created materialized view would include only future changes and ignore the historical data that existed before the view was created. Backfilling therefore connects historical state with live streaming state. RisingWave coordinates this transition using consistent snapshots, barriers, checkpoints, fragment-level progress, and persisted state so the system can process historical data while also preserving the changes that occur during initialization.
Backfilling Strategies
RisingWave supports different backfilling strategies for initializing materialized views. The strategy used depends on the structure of the query, the upstream sources, the required state, the data distribution, and the relationship between upstream and downstream parallelism. The three strategies discussed here are snapshot backfill, arrangement backfill, and no-shuffle backfill.
Snapshot Backfill
Snapshot backfill uses a consistent snapshot of an upstream table at a specific logical point. RisingWave scans the snapshot in batches, applies the materialized view query, writes the resulting state, and then transitions to incremental processing after the historical scan is complete. The snapshot provides a stable view of the upstream source even if new changes continue to arrive while the backfill is running.
The main advantage of snapshot backfill is that RisingWave does not need to replay every historical change one by one when the current table state is already available. Instead, it can load the existing state in bulk and then process only the changes that occurred after the snapshot boundary. This can make the creation of materialized views on large existing datasets more practical.
Snapshot backfill also improves isolation between the historical backfill phase and the normal streaming phase. It has been enabled by default in RisingWave since v2.8. During the backfilling process, progress can be checkpointed so that a failure does not necessarily require restarting the entire backfill from the beginning.
Arrangement Backfill
Arrangement backfill is used for more complex queries that require intermediate state, such as joins and aggregations. An arrangement is an internally organized representation of data, usually keyed according to the needs of a streaming operator. For example, when joining an orders table with a customers table on customer_id, RisingWave may organize both sides of the join by customer_id so that each order can be matched efficiently with the corresponding customer record.
During backfilling, RisingWave may need to scan historical records, build and maintain these arrangements, preserve join or aggregation state, and process updates that arrive while initialization is still in progress. This is more complex than a simple snapshot scan because the system must maintain consistency across multiple stateful operators and upstream relations.
Barrier alignment is important in this process. If different input paths progress at different speeds, RisingWave must coordinate them so that the operators reach a consistent logical point before the materialized view is considered fully initialized.
Arrangement backfill became the default backfill strategy in RisingWave v1.10, before snapshot backfill became the default starting with v2.8. The enable_arrangement_backfill configuration option is now deprecated and ignored for new streaming jobs. Arrangement backfill is always retained as the fallback backfill strategy when the preferred backfill path cannot be used.
No-Shuffle Backfill
No-shuffle backfill was an optimization designed to avoid unnecessary data redistribution when the upstream and downstream parallelism matched and their data distributions were already compatible. In a distributed system, data often needs to move across the network so that rows with the same key are processed by the same worker. This redistribution is known as a shuffle.
When the upstream and downstream layouts were already aligned, no-shuffle backfill allowed records from each upstream partition to be sent directly to the corresponding downstream partition. This reduced network traffic, serialization and deserialization work, buffering, and coordination overhead during backfilling.
No-shuffle backfill has been deprecated since RisingWave v3.0. Its original goal of reducing unnecessary data movement was important, but newer locality-aware approaches provide broader optimizations across more parts of the backfill pipeline.
Locality Backfilling
Locality backfilling is a premium feature that preserves data locality across the entire backfill pipeline. In a standard backfill path, data may be scanned from storage, redistributed across nodes, and then processed by joins, aggregations, window functions, or Group TopN operators. This repeated movement can increase network traffic, memory pressure, cache misses, remote I/O, and CPU usage, especially for large and complex queries with many CTEs, subqueries, or stateful operators.
Index selection and locality backfilling address different parts of this problem. Index selection improves the scan phase by allowing RisingWave to read data in an order that matches grouping, join, or partition keys. Locality backfilling extends this optimization beyond the scan itself. When enabled, the optimizer inserts LocalityProvider operators into the query plan to keep data clustered by key as it moves through joins, aggregations, window functions, and Group TopN. This reduces unnecessary shuffling, improves cache utilization, lowers random remote I/O, and increases the throughput of complex backfills.
Locality can also be improved through query and storage design. When creating an MV on another MV, the ordering key of the upstream relation should align with the grouping key of the downstream MV whenever possible. Similarly, for an MV built on a table populated through SINK INTO TABLE, the sink should write data in an order that matches the downstream MV’s partitioning or ordering keys. This helps RisingWave process related rows together, reducing random I/O and improving cache efficiency during backfilling.
Locality backfilling was added in RisingWave v2.7 and is disabled by default. It can be enabled for the session with:
SET enable_locality_backfill = true;
Once enabled, the optimizer automatically adds LocalityProvider operators to newly created query plans where they can improve backfill execution. Locality backfill is treated as a premium feature for complex queries. A query plan that uses more than five LocalityProvider operators is considered complex and activates the premium feature.
Locality backfilling improves execution efficiency, but it does not remove the need to manage resource pressure. If CPU utilization, cache misses, memory pressure, or remote I/O become high during a large backfill, it can be more efficient to break the query into several smaller materialized views and create them one by one.
Serverless Backfilling
Backfilling historical data can consume substantial CPU, memory, storage I/O, network bandwidth, and compaction capacity. By default, the backfill phase of creating a materialized view, sink, or index runs on the same compute nodes as regular streaming workloads. Large backfills can therefore compete with existing jobs for resources and increase latency, especially when a query contains large joins, many CTEs or subqueries, or a significant amount of historical state.
Serverless backfilling in RisingWave Cloud runs this historical initialization phase on dedicated temporary backfiller resources instead of the main streaming compute nodes. After a CREATE MATERIALIZED VIEW, CREATE SINK, or CREATE INDEX statement is validated and the streaming job is ready to be created, RisingWave Cloud provisions a separate resource group using the configured backfiller SKU and replica count. If validation fails, these resources are not provisioned.
The backfill runs in this temporary resource group without directly competing with existing streaming jobs for compute resources. Once the backfill task is complete, the job moves to the steady-state resource group of its parent database for normal streaming execution, and the temporary backfiller resources are removed automatically. If the cluster restarts during backfilling, the job resumes from its most recently completed checkpoint instead of starting again from the beginning.
Serverless backfilling has been available since RisingWave v2.8.0, is disabled by default, and is supported only in RisingWave Cloud. Large backfills can also increase write throughput and compaction pressure. Compaction resources must therefore be sized for the temporary backfill load rather than only for steady-state traffic. For this reason, Serverless Compaction is recommended so that compactor resources can scale automatically. Otherwise, the compactor should be scaled manually before running heavy backfill jobs to avoid compaction back pressure or write stalls.
Serverless backfilling can be enabled for subsequent materialized view, sink, and index creation statements in the current session:
SET enable_serverless_backfill = true;
The default value is false, and the setting applies only to new DDL operations issued after the command. It does not change already-running jobs.
It can also be enabled for an individual materialized view using the statement-level WITH option:
CREATE MATERIALIZED VIEW mv
WITH (cloud.serverless_backfill_enabled = true)
AS
SELECT ...;
Backfill progress can be monitored through:
SELECT * FROM rw_catalog.rw_ddl_progress;
Users can also use SHOW JOBS to inspect background DDL jobs and view the dedicated risingwave-backfill-* node series on the RisingWave Cloud Metrics page to monitor backfiller CPU and memory usage.
Best Practices for Backfilling in RisingWave
Backfilling can place significant pressure on compute, storage, upstream systems, and compaction resources. The following practices can help improve backfill performance and reduce its impact on live workloads.
Schedule large backfills carefully. Create large materialized views during lower-traffic periods when possible. Even with serverless backfilling, large historical scans can still affect shared storage, upstream sources, object-storage bandwidth, and compaction.
Use serverless backfilling for heavy jobs. In RisingWave Cloud, dedicated backfiller nodes isolate much of the CPU and memory pressure from the main streaming compute nodes, reducing the impact on live streaming workloads.
Monitor backfill progress and resource usage. Use system catalogs and diagnostic tools such as
rw_catalog.rw_fragment_backfill_progress,rw_catalog.rw_ddl_progress, andDESCRIBE FRAGMENTS. Also monitor CPU and memory usage, barrier latency, checkpoint duration, storage I/O, and compaction pressure.Use background DDL for long-running operations. When the client should not remain blocked during materialized view creation, enable background DDL:
SET BACKGROUND_DDL = true;
Control rate limits and parallelism. Lower backfill parallelism can reduce resource contention, while normal streaming parallelism can be increased after initialization is complete. Limiting concurrent streaming job creation can also prevent multiple large backfills from exhausting cluster resources at the same time.
Break complex queries into smaller materialized views. For queries with many CTEs, subqueries, joins, or aggregations, consider creating several smaller materialized views one by one. A single SQL statement with many stateful branches can behave like several expensive queries running simultaneously, increasing memory pressure, cache misses, CPU usage, and remote I/O.
Improve data locality. For materialized views built on other materialized views, align the upstream order key with the downstream grouping, join, or partition key where possible. This allows related rows to be read consecutively, improves cache efficiency, and reduces random remote I/O. Index selection and locality backfilling can further improve this process.
Control the backfill order for joins. For join-heavy MVs, use
backfill_orderwhen appropriate to backfill smaller dimension tables before a large fact table. This can reduce failed lookups, retractions, and update churn caused by processing fact-table rows before the required dimension state is available.Avoid unnecessary full rebuilds. If an expensive MV is repeatedly dropped and recreated, RisingWave must rebuild its historical state each time. Keep stable and expensive state in materialized views, and place frequently changing logic in lighter downstream views when possible.
Backfilling should be treated as a controlled production workflow. Define its scope, understand downstream dependencies, monitor its progress, validate the resulting data, and ensure sufficient compaction capacity is available.
Conclusion
Backfilling bridges historical initialization and continuous streaming, allowing newly created materialized views to begin with a complete state and remain up to date as new data arrives. With snapshot and arrangement backfill, locality-aware execution, serverless resource isolation, rate limits, parallelism controls, progress monitoring, and checkpoint-based recovery, RisingWave makes backfilling more efficient, observable, and resilient. Rather than treating historical data processing as a simple rerun, RisingWave makes it a first-class part of stream processing that can be carefully optimized and managed for production workloads at scale.
