feat: upgrade database-jobs with upstream performance optimizations#61
Open
pyramation wants to merge 6 commits intomainfrom
Open
feat: upgrade database-jobs with upstream performance optimizations#61pyramation wants to merge 6 commits intomainfrom
pyramation wants to merge 6 commits intomainfrom
Conversation
…tions New @pgpm/database-jobs-v2 package based on @pgpm/database-jobs v0.21.0 with: Phase 1 - Critical performance fixes: - Fix queue_name default from gen_random_uuid()::text to NULL - Rewrite get_job with pre-computed queue set (Strategy 2 vs old per-row EXISTS) Phase 2 - Index improvements: - Add is_available generated column (locked_at IS NULL AND attempts < max_attempts) - Replace priority_run_at_id_idx with partial covering indexes (jobs_main_index, jobs_no_queue_index) Phase 3 - Modernization: - Add revision column (incremented on upsert via job key) - Add flags jsonb column with forbidden_flags filtering in get_job - Change notify trigger from FOR EACH ROW to FOR EACH STATEMENT - Add remove_job(job_key text) function - Add force_unlock_workers(worker_ids text[]) function Interface is fully backward-compatible: all existing function signatures preserved. Refs: constructive-io/constructive-planning#772
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
- Remove revision column (unused by our architecture — no job replacement tracking needed) - Remove flags jsonb column and forbidden_flags parameter from get_job (worker routing is handled at K8s level, not SQL level) - Remove flags parameter from add_job and all flags-related upsert logic - Keep is_available generated column (powers partial indexes for get_job performance) - Update snapshot test to reflect removed columns
- Rename database-jobs → database-jobs-v1 (legacy, v0.15.5) - Rename database-jobs-v2 → database-jobs (optimized, v0.22.0) - Update package.json, control files, and pgpm.plan for both - Update CI workflow matrix accordingly The new database-jobs now contains all the upstream performance optimizations (Strategy 2 get_job, is_available partial indexes, NULL queue_name default, statement-level notify trigger) plus remove_job and force_unlock_workers functions.
devin-ai-integration
Bot
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Upgrades
@pgpm/database-jobswith upstream Graphile Worker performance optimizations. The olddatabase-jobsis preserved asdatabase-jobs-v1for reference.Related: https://github.com/constructive-io/constructive-planning/issues/772
Swap:
database-jobs→ renamed todatabase-jobs-v1(legacy, v0.15.5)database-jobs-v2→ renamed todatabase-jobs(optimized, v0.22.0)Performance fixes in the new
database-jobs:queue_namedefault → NULL (stops creating throwaway queues — the Feat/clean cli 2 #1 bottleneck)get_jobrewritten with Strategy 2 (pre-computed queue set viaINsubquery, ~20x improvement)is_availablegenerated column + partial covering indexes (jobs_main_index,jobs_no_queue_index)FOR EACH STATEMENTinstead ofFOR EACH ROW)New functions:
remove_job(job_key),force_unlock_workers(worker_ids)Intentionally excluded after analysis:
revisioncolumn — our architecture doesn't use job replacement trackingflags/forbidden_flags— worker routing handled at K8s level, not SQL levelPreserved:
database_id,scheduled_jobs, trigger helpers,actor_idinjection,app_jobsschema, all existing function signatures.Review & Testing Checklist for Human
get_jobStrategy 2 logic handles edge cases: expired locks, mixed NULL/non-NULL queue jobs,SKIP LOCKEDcontention under concurrent workersadd_jobupsert behavior is correct — when a locked job is replaced, the old key is nulled and attempts maxed out, then a fresh row is insertedEXPLAIN ANALYZEonget_jobto confirm the partial indexes are usedadd_jobsignature dropped unusedflagsparamNotes
database-jobs-v1is preserved as a reference — can be removed once the swap is validated in productionis_availablecolumn is not read by app code but is required for the partial index optimizationadd_jobsignature:(db_id, identifier, payload, job_key, queue_name, run_at, max_attempts, priority)— same as v1 minus the never-usedflagsparamLink to Devin session: https://app.devin.ai/sessions/88b82416d018482791c791cdc91118de
Requested by: @pyramation