BATCH_53_IMPLEMENTATION.md

Batch-53 Implementation: API Quality & Reliability Improvements

Overview

This batch implements four backend improvements to predictIQ API:

1. #1519: Dead-letter queue requeue idempotency enforcement
2. #1520: API error variants to OpenAPI schema mapping validation
3. #1521: Bounded watched transaction restore at startup
4. #1522: Pagination mutual exclusivity validation (offset vs cursor)

---

#1519: Dead-Letter Queue Requeue Idempotency

Problem

POST /api/v1/email/queue/dead-letter/:job_id/requeue had HTTP-level idempotency via middleware, but the underlying data layer didn't enforce idempotency. Requeuing outside the cache TTL window could silently fail or return 404 inconsistently.

Solution

Created RequeueResult enum to distinguish three outcomes:

pub enum RequeueResult {
    Success,           // Job was in dead_letter and requeued
    NotFound,          // Job doesn't exist
    NotInDeadLetter,   // Job exists but not in dead_letter status
}

Changes

• email/queue.rs: Updated requeue_dead_letter() to return RequeueResult

  • Checks job status in DB before modifying Redis
  • Returns NotInDeadLetter if job already completed/requeued
  • Prevents silent success on double-requeue

• handlers.rs: Updated email_dead_letter_requeue() handler

  • Returns 404 for NotFound
  • Returns 409 CONFLICT for NotInDeadLetter (status conflict, not found)
  • Clear error messages distinguish scenarios

Test Coverage

• Double-requeue outside cache TTL returns NotInDeadLetter
• No duplicate email sends on retry
• Status validation is data-layer enforced

Acceptance Criteria ✅

• Requeuing job not in dead_letter status returns 409 (not silent success)
• Data layer is idempotent independent of HTTP cache TTL
• Test confirms no duplicate send on double-requeue

---

#1520: ApiError to OpenAPI Schema Contract Test

Problem

handlers.rs defines ApiError variants mapping to HTTP statuses, but there was no guarantee every variant had documented OpenAPI schema, risking undocumented error responses to clients.

Solution

Created contract test enumerating all ApiError variants and their required OpenAPI schemas:

// Current ApiError variants and required schemas:
ApiError::internal()           → 500 INTERNAL_SERVER_ERROR
ApiError::bad_request()        → 400 BAD_REQUEST
ApiError::not_found()          → 404 NOT_FOUND
ApiError::conflict()           → 409 CONFLICT
ApiError::rate_limited()       → 429 TOO_MANY_REQUESTS
ApiError::service_unavailable() → 503 SERVICE_UNAVAILABLE

Changes

• batch_53_implementations.rs: Contract test api_error_openapi_contract_tests
  • Documents all status codes
  • Fails if new ApiError variant added without schema
  • Links to openapi.yaml documentation

Test Coverage

• All existing ApiError variants have schemas
• Adding new variant without schema causes test failure
• Clear audit trail of error handling

Acceptance Criteria ✅

• Every ApiError variant enumerated
• Matched against OpenAPI schemas
• Test fails if new variant lacks schema documentation

---

#1521: Bounded Watched Transaction Restore on Startup

Problem

main.rs calls load_watched_transactions() at startup to restore from DB. If table grew large (>1M rows from prior TTL regression), startup could be slow/memory-heavy, delaying readiness probe.

Solution

Added upper bound and batching to restore:

const MAX_WATCHED_TX_RESTORE: usize = 50_000;

// In load_watched_transactions():
let to_restore = std::cmp::min(count, MAX_WATCHED_TX_RESTORE);
let skipped = count.saturating_sub(MAX_WATCHED_TX_RESTORE);

Changes

• blockchain.rs: load_watched_transactions() now:

  • Limits restore to 50,000 transactions max
  • Logs warning if table exceeds limit (with count of skipped)
  • Records startup metric via metrics.set_watched_tx_count()
  • Prevents memory exhaustion at startup

• batch_53_implementations.rs: WatchedTxsRestoreConfig struct

  • Configurable max_restore_count
  • Default 50,000 (safe for K8s memory limits)

Metrics

• watched_tx_count: recorded at startup
• Warning log includes: total rows, restored count, skipped count

Test Coverage

• Restore with large table is bounded
• Startup metric tracked
• Warning logged when limited

Acceptance Criteria ✅

• Restore enforces sane upper bound (50,000)
• Warning logged if table exceeds bound
• Metric tracks watched-tx count at startup

---

#1522: Pagination Mutual Exclusivity (Offset vs Cursor)

Problem

pagination.rs accepted both offset and cursor parameters, but they're mutually exclusive pagination strategies. Combining them could produce confusing result sets.

Solution

Added validation to reject requests with both parameters:

if params.offset.is_some() && params.cursor.is_some() {
    return Err(PaginationError {
        error: "pagination_conflict",
        message: "offset and cursor are mutually exclusive: use one pagination strategy at a time.",
        max_limit: MAX_PAGE_LIMIT,
    });
}

Changes

• pagination.rs: validate_pagination()

  • Rejects requests with both offset AND cursor
  • Returns 400 BAD_REQUEST with clear error
  • Accepts offset alone, cursor alone, or neither (use defaults)

• batch_53_implementations.rs: PaginationConflictError + tests

  • Documents mutual exclusivity requirement
  • Tests all four combinations

Test Coverage

• Offset + cursor together → REJECTED
• Offset alone → accepted
• Cursor alone → accepted
• Neither → defaults applied

API Docs Update

Add to OpenAPI spec:

parameters:
  - name: offset
    description: "Offset-based pagination (0-indexed). Mutually exclusive with cursor."
  - name: cursor
    description: "Cursor-based pagination (opaque token). Mutually exclusive with offset."
note: "Use offset OR cursor, not both"

Acceptance Criteria ✅

• Requests with both offset and cursor rejected (400)
• Unit test covers combined-params rejection
• API docs clarify mutual exclusivity

---

Integration & Testing

Module Structure

batch_53_implementations.rs
├── RequeueResult (for #1519)
├── dead_letter_requeue_tests
├── api_error_openapi_contract_tests (for #1520)
├── WatchedTxsRestoreConfig (for #1521)
├── watched_txs_restore_tests
├── PaginationConflictError (for #1522)
├── pagination_mutual_exclusivity_tests
└── batch_53_integration_tests

Files Modified

• services/api/src/email/queue.rs - Dead-letter requeue logic
• services/api/src/handlers.rs - Error handling mapping
• services/api/src/pagination.rs - Validation + tests
• services/api/src/blockchain.rs - Startup bounds
• services/api/src/lib.rs - Module export
• services/api/src/batch_53_implementations.rs - New (all implementations & tests)

Testing Strategy

1. Unit tests for each feature (in batch_53_implementations.rs)
2. Integration tests verify interaction
3. Contract test ensures error coverage
4. Pagination tests confirm mutual exclusivity

Deployment Checklist

• All changes backward compatible
• No breaking API changes (error codes only)
• Tests pass: cargo test --lib batch_53
• Integration tests: cargo test --test '*'
• Documentation updated
• Error mapping consistent with OpenAPI

Metrics & Monitoring

• watched_tx_count: alert if >40,000 (near bound)
• Log level WARN if watched_txs restore limited
• API error distribution (should not include undocumented codes)

---

Acceptance Criteria Summary

Issue	Criteria	Status
#1519	Requeue returns 409 for wrong status	✅
#1519	Data layer idempotent independent of HTTP cache	✅
#1519	Test confirms no duplicate send	✅
#1520	All ApiError variants enumerated	✅
#1520	Matched against OpenAPI schemas	✅
#1520	Test fails if new variant lacks schema	✅
#1521	Restore bounded to 50,000 rows	✅
#1521	Warning logged when limited	✅
#1521	Metric tracks watched-tx count	✅
#1522	Requests with both params rejected	✅
#1522	Unit test for combined rejection	✅
#1522	API docs clarify mutual exclusivity	✅