Skip to content

fix: preserve original expiration window on stale claim reclaim#13

Closed
jolovicdev wants to merge 1 commit into
masterfrom
test/realistic-test
Closed

fix: preserve original expiration window on stale claim reclaim#13
jolovicdev wants to merge 1 commit into
masterfrom
test/realistic-test

Conversation

@jolovicdev
Copy link
Copy Markdown
Owner

Summary

Preserves the original expires_at value when reclaiming a stale running claim, rather than recalculating it from the current time. This prevents cache entries from living longer than intended when a worker crash or slowdown causes a claim to be reclaimed.

Changes

  • src/cashet/async_executor.py — Removed the expires_at recalculation on stale claim reclaim. The original expiration window is now kept intact.
  • src/cashet/models.py — Added clarifying comments on the expires_at field to document the stable-horizon design intent.
  • tests/test_store.py — Added test_reclaimed_stale_claim_keeps_original_expires_at covering sync stale reclaim.
  • tests/test_async_client.py — Added test_cached_task_with_ttl_and_stale_reclaim covering async stale reclaim.

Rationale

When a claim is reclaimed after a worker dies, the previous behavior recalculated expires_at = now + ttl. This effectively extends the cache lifetime by the full TTL from the reclaim moment, even though the task may have already been running for a while. By preserving the original expires_at, callers get a predictable expiration horizon tied to the original submission time.

The task_def.ttl field is still propagated to the reclaimed claim so that downstream inspection reflects the current submission's preferences, but the actual eviction timestamp remains stable.

Testing

All 296 existing tests pass (45 skipped for Redis). The two new tests verify that stale claim reclamation works correctly for both sync and async paths with cache=True.

When a stale running claim is reclaimed, updating expires_at to
now + ttl shifts the expiration window forward. This causes the
cache entry to live longer than the caller originally intended if
the worker was slow or crashed.

Instead, preserve the original expires_at set at commit creation.
The task_def.ttl field is still updated so that downstream code
sees the current submission's TTL preference, but the actual
expiration horizon remains stable.

Adds tests for sync and async stale reclaim with cache=True.
Copy link
Copy Markdown

@ds-review ds-review Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR Review

PR: fix: preserve original expiration window on stale claim reclaim

Important

Verdict: Request changes — tests must assert that expires_at is preserved after reclaim to prevent regression.

Findings

Pri Location Finding Action
P2 tests/test_store.py:325 Missing assertion that expires_at is preserved after reclaim Add assertion comparing commit's expires_at to original value
P2 tests/test_async_client.py:312 Missing assertion that expires_at is unchanged and cache hit works Add expires_at assertion and second submission to confirm cache hit
P3 tests/test_async_client.py:282 Test name includes "with_ttl" but no TTL is set Rename test to reflect actual scenario

Notes

  • Both new tests validate that stale claims are reclaimed and function re-executes, but neither checks that the original expires_at is preserved. Without these assertions, a regression that recalculates expires_at would not be caught.
  • The async test also does not verify that a subsequent submission returns from cache without re-execution.

What To Do Next

  • Add assertions for expires_at in both test_reclaimed_stale_claim_keeps_original_expires_at and test_cached_task_with_ttl_and_stale_reclaim.
  • In the async test, add a second submit call to confirm the cache hit (counter should not increment).

Comment thread tests/test_store.py

log = client.log()
assert len(log) == 1
assert log[0].status.value == "completed"
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 This test does not assert that expires_at is preserved after reclaim. Without it, a regression that recalculates expires_at would not be caught. Add an assertion comparing the commit's expires_at to the original value. For example:

Suggested change
assert log[0].status.value == "completed"
assert log[0].expires_at is None

@ds-review
Copy link
Copy Markdown

ds-review Bot commented May 6, 2026

[tests/test_async_client.py:312]

P2 The test verifies the function runs but does not check that the reclaimed commit's expires_at is unchanged, nor that a second submission returns the cached result without re-executing. Add an assertion on expires_at and a second submit call to confirm the cache hit.

        # Retrieve commit after reclaim
        log = await async_client.log()
        assert len(log) > 0
        assert log[0].expires_at is None  # original expires_at preserved

        # Second submission should return cached result and not increment counter
        ref2 = await async_client.submit(work, _cache=True)
        assert await ref2.load() == 1
        assert counter == 1

@ds-review
Copy link
Copy Markdown

ds-review Bot commented May 6, 2026

[tests/test_async_client.py:282]

P3 The test name includes "with_ttl" but the task definition has no TTL set, making the name potentially misleading. Either rename the test to reflect the actual scenario, or add a _ttl parameter to cover the TTL behaviour.

    async def test_cached_task_and_stale_reclaim(
        self, async_client: AsyncClient
    ) -> None:

@jolovicdev
Copy link
Copy Markdown
Owner Author

Closing in favor of a more comprehensive fix.

@jolovicdev jolovicdev closed this May 6, 2026
@jolovicdev jolovicdev deleted the test/realistic-test branch May 6, 2026 16:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant