Skip to content

scan_windows_via_exact assumes stored windows match the CURRENT config's shape/grid, but update_streaming_config doesn't purge old data #618

Description

@milindsrivastava1997

Problem

scan_windows_via_exact (asap-query-engine/src/engines/simple_engine/mod.rs, added in #616) walks the aggregation's window grid using window_size_ms/slide_interval_ms read from the current AggregationConfig (sc.get_aggregation_config(params.aggregation_id)), and probes the store with query_precomputed_output_exact for entries shaped exactly [window_start, window_start + window_size_ms) on that grid.

update_streaming_config (called by main.rs whenever the planner fires — a normal, periodic event, not an edge case) swaps the engine's and store's config by reference with no purge:

// stores/simple_map_store/per_key.rs
/// Replace the streaming config at runtime. Called by the applier task after
/// the planner fires. Stale precomputed data for dropped aggregations expires
/// naturally via the existing cleanup policy.
pub fn update_streaming_config(&self, new_config: StreamingConfig) {
    *self.streaming_config.write().unwrap() = Arc::new(new_config);
}

That comment only covers aggregation IDs that get dropped entirely. If an existing aggregation_id has its window_size_ms/slide_interval_ms changed in place (same ID, new shape), old data written under the previous shape/grid stays in the store — mixed with new data under the new shape — under the same aggregation_id, until count-based retention eventually evicts it (unrelated to shape).

scan_windows_via_exact has no way to know this happened. It always computes its probe shape/grid from whatever config is current right now, so any query whose time range still includes pre-reconfig data gets nothing back for that portion — silently, no error.

Why this is worse than a normal edge case

The old Store::query_precomputed_output tolerant scan never had this problem: it doesn't care about window_size_ms at all, it just returns whatever raw (start,end) entries exist via containment, regardless of shape or which config produced them. This bug is specific to the new exact-lookup grid-walk and hits the common case (any Tumbling query, any time after a reconfig) — not a rare shape.

Suggested fix directions (not decided)

  • Version the stored shape per epoch/aggregation so a query can look up the shape that was actually in effect for the time range it's reading, not just the current one.
  • Or: purge/reshape old data on reconfig (bigger behavior change, may not be desirable).
  • Or: detect "this time range might predate the current config" and fall back to the tolerant scan for it.

Scope

asap-query-engine/src/engines/simple_engine/mod.rs (scan_windows_via_exact), possibly asap-query-engine/src/stores/simple_map_store/*.rs (update_streaming_config) depending on which fix direction is chosen.

Found during review of #616.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions