Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 0 additions & 11 deletions apps/sim/background/table-run-dispatcher.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -23,15 +23,4 @@ describe('table-run-dispatcher task configuration', () => {
it('runs on a preset whose memory clears the observed plateau', () => {
expect(tableRunDispatcherTask.machine).toBe('small-2x')
})

/**
* `maxAttempts` alone does NOT cover `TASK_PROCESS_OOM_KILLED` — Trigger.dev
* retries an OOM only when `retry.outOfMemory.machine` names a larger preset.
* Every one of the four killed runs recorded `attempt_count = 1`, so the
* documented "retries and resumes from the persisted cursor" never happened.
*/
it('escalates to a larger machine on an out-of-memory kill', () => {
expect(tableRunDispatcherTask.retry?.outOfMemory?.machine).toBe('medium-1x')
expect(tableRunDispatcherTask.retry?.maxAttempts).toBe(3)
})
})
12 changes: 1 addition & 11 deletions apps/sim/background/table-run-dispatcher.ts
Original file line number Diff line number Diff line change
Expand Up @@ -17,16 +17,7 @@ export interface TableRunDispatcherPayload {
* dispatcher loop for the dispatch's entire lifetime — each iteration
* processes a window of cells via `batchTriggerAndWait`, which checkpoints
* the parent via CRIU during the wait so we don't pay compute while cells
* execute. The cursor is persisted in DB, so an attempt that starts after a
* crash resumes from it rather than replaying the dispatch.
*
* `maxAttempts` alone does NOT cover an OOM: Trigger.dev retries
* `TASK_PROCESS_OOM_KILLED` only when `retry.outOfMemory.machine` names a
* larger preset. Four runs were killed this way and every one recorded
* `attempt_count = 1` — no retry happened, and the dispatch row was left
* `dispatching` forever. The escalating preset is what makes the documented
* resume actually reachable; the cleanup sweep is the backstop for a dispatch
* whose holder dies without one.
Comment thread
TheodoreSpeaks marked this conversation as resolved.
* execute. The cursor is persisted in DB between windows.
*/
export const tableRunDispatcherTask = task({
id: 'table-run-dispatcher',
Expand All @@ -38,7 +29,6 @@ export const tableRunDispatcherTask = task({
* 0.03 for p90, so the larger preset is bought for its RAM.
*/
machine: 'small-2x',
retry: { maxAttempts: 3, outOfMemory: { machine: 'medium-1x' } },
queue: {
name: 'table-run-dispatcher',
concurrencyLimit: 8,
Expand Down
Loading