Skip to content

feat(cmdb-import): archive or soft-delete applications that left the export, revive returning ones - #1281

Open
WilcoLouwerse wants to merge 17 commits into
developmentfrom
feat/woo587-cmdb-import-archive-reconciliation
Open

feat(cmdb-import): archive or soft-delete applications that left the export, revive returning ones#1281
WilcoLouwerse wants to merge 17 commits into
developmentfrom
feat/woo587-cmdb-import-archive-reconciliation

Conversation

@WilcoLouwerse

@WilcoLouwerse WilcoLouwerse commented Oct 9, 2026 •
edited by atlassian Bot
Loading

Copy link
Copy Markdown

What

A repeated CMDB import now reconciles a municipality's applications with the newer TOPdesk export.

  • Archived application: an application whose APPID left the two CMDB sheets but is listed on the sheet "Gearchiveerde Applicaties" gets OpenRegister's archive state, on both the module and the municipality's usage.
  • Gone application: an application listed on no sheet at all is soft-deleted into OpenRegister's trash, where it can be restored for 30 days.
  • Returning application: an application that comes back is unarchived or restored and then updated as usual, so it is never duplicated.
  • Missing archive sheet: if the workbook has no archive sheet, missing applications are archived only, never deleted, and the report warns about it.
  • Report: four new outcomes: archived, unarchived, deleted and restored. Report rows carry only APPID, name and uuids, never person data.
  • Option: missingRecords is keep | archive, default archive. The CMDB import section shows the choice. remove stays reserved and answers 422.
  • Schemas: module (0.3.9) and usage (1.5.7) opt in to archiving through x-openregister-archive. Run Force Update after installing.
  • Quick filter: an "Archived" quick filter (_archived=true) on the Applications and Usage pages.
  • Other apps: OpenCatalogi and Portaliq list through OpenRegister's default view, so archived and deleted applications drop out of them without a change there. On OpenCatalogi's existing public routes, _archived=true finds the archived ones.
  • Spec: OpenSpec change cmdb-import-archive-reconciliation: ADDED REQ-CMDB-015..017, MODIFIED REQ-CMDB-012 and REQ-CMDB-014.

Jira: WOO-587 (sub-task of WOO-586). Plan: WOO-587 plan. Review and merge are tracked in WOO-590.

How the state of a stored object is read (design D9)

On the current OpenRegister beta (2.1.37, origin/beta ff4dad5c), a search cannot tell this import whether an object is archived or deleted. The cause is in MagicSearchHandler::convertRowToObjectEntity:

  • It does not hydrate _archived, so every search result looks live.
  • It parses the JSON _deleted column as a DateTime. That fails, so _includeDeleted=true never returns a deleted row.

The import therefore reads the state per lens instead:

  1. Live: a search without a lens.
  2. Archived: a search with _archived=true.
  3. Deleted: the trash, read once per import through MagicMapper::findDeletedAcrossAllMagicTables().

The unit-test doubles model that beta behaviour. Against the earlier service code, 7 tests fail on exactly the rig symptom; with the fix they pass.

Overlap with #1280 (WOO-588)

Both PRs touch CmdbImport.vue, openspec/specs/cmdb-export-import/spec.md and l10n. A local merge of both on development had two trivial conflicts (keep both sides). Whichever PR merges second needs development merged in.

Tests

Check Result
PHPUnit tests/Unit/Service, CmdbImportControllerTest, tests/Unit/Settings, tests/Unit/Fixtures ✅ 779 tests
CMDB suites on the local merge of #1280 and this PR ✅
jest ✅ 11 suites, 190 tests
vitest cmdbArchiveQuickFilter.spec.js ✅
full vitest ⚠️ 6 failures, the same 6 as on development c75dbb2; not touched here
phpcs / php lint / phpstan ✅ 0 errors
eslint / prettier / l10n / check:l10n-js / manifest / schema-l10n / vue-demi ✅
psalm ❓ could not run locally: psalm crashes while scanning vendor; CI is the evidence
full composer test:unit locally ⚠️ 20 errors in Repair/PortfolioReport tests (missing Doctrine/Symfony classes in the local vendor), outside this diff
Playwright tests/e2e/spec-coverage/cmdb-import.spec.ts updated, not run locally

⚠️ Heads-up on the Playwright test: its afterAll cleanup lists objects through REST with _includeDeleted=true. That runs into the same OpenRegister converter bug, so soft-deleted test objects may be left behind on the CI instance.

Rig smoke

The smoke ran on Nextcloud 35.0.1 with these beta releases:

App Version
OpenRegister 2.1.37
OpenCatalogi 2.1.18
Portaliq 0.2.9
Thematiq 1.2.26
stackiq this branch merged with #1280

It used only the anonymised fixtures in tests/fixtures/cmdb/.

Import Result
topdesk-export-anonymised.xlsx (keep) created 2
topdesk-archived-applications.xlsx archived 1, deleted 1; anonymous OpenCatalogi /api/applicatielandschap hides both, and _archived=true shows the archived one
topdesk-export-anonymised.xlsx again unarchived 1, restored 1, created 0 (no duplicate); every lens count is back to where it started

The three imports together took about 12 seconds, including the trash read.

🤖 Generated with Claude Code

WilcoLouwerse and others added 17 commits October 9, 2026 15:22
@WilcoLouwerse @claude
docs(openspec): add change cmdb-import-archive-reconciliation
c0de84c 
Proposal, design, tasks and the delta spec for archiving and soft-deleting
applications that left a newer CMDB export (REQ-CMDB-015..017, REQ-CMDB-012
and REQ-CMDB-014 modified).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@WilcoLouwerse @claude
feat(cmdb-import): let module and usage be archived and name the arch…
fd4aa7e 
…ive sheet

The register fragment opts module (0.3.9) and usage (1.5.7) into
OpenRegister's archive state. The import profile names the sheet
"Gearchiveerde Applicaties" and accepts missingRecords keep and archive,
readable before the packs are validated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@WilcoLouwerse @claude
feat(cmdb-import): read the APPIDs of the archive sheet
ed61ce7 
When the workbook has the sheet "Gearchiveerde Applicaties", the reader
loads it under the same bounds as the CMDB sheets, resolves only its APPID
column and returns the APPIDs next to the rows, with whether the sheet and
its key column were found.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@WilcoLouwerse @claude
feat(cmdb-import): archive or soft-delete applications that left the …
ed59f25 
…export

With missingRecords archive, the default, a completed import reconciles
the municipality's usages and the modules its import created: an APPID
that moved to the archive sheet is archived through OpenRegister's
ArchiveHandler, one on no sheet is soft-deleted, and without the archive
sheet missing applications are archived only, with a report warning. The
step runs under the register lock in its own progress phase, with an error
boundary per application. A module or usage matched among archived or
soft-deleted objects is unarchived or restored before it is updated, so a
returning application is never duplicated. The report counts archived,
unarchived, deleted and restored. Without the archive services an archiving
import is refused with 503 ARCHIVE_UNAVAILABLE before the file is read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@WilcoLouwerse @claude
feat(cmdb-import): accept missingRecords keep or archive, default arc…
2919497 
…hive

The endpoint passes the option to the import, names both accepted values
when it refuses a reserved one, and translates ARCHIVE_UNAVAILABLE. The
OpenAPI document and the Postman test describe the new values, outcomes and
error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@WilcoLouwerse @claude
feat(cmdb-import): offer the missing-records choice and an Archived q…
f2c533d 
…uick filter

The CMDB import section lets the admin archive or delete applications
missing from the export (the default) or keep them, with help text, and
shows the archived, unarchived, deleted and restored counts and outcomes.
A reconciliation entry has no row number and a deleted module no link, and
the progress line says when the import checks missing applications. The
Applications and Applications in use pages get an Archived quick filter
that lists the archive (_archived=true).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@WilcoLouwerse @claude
test(cmdb-import): add the archived-applications fixture and its roun…
0c37657 
…d trip

build-fixtures.py derives topdesk-archived-applications.xlsx, a later export
in which APPID 1234 moved to the archive sheet and APPID 2 is on no sheet.
The reader, the service and the hygiene test cover it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@WilcoLouwerse @claude
docs(cmdb-import): describe archiving and deleting on a repeat import
9cb2bfe 
Repeat imports, the section's new choice, the summary and row outcomes, the
archive sheet and the ARCHIVE_UNAVAILABLE and MISSING_RECORDS_UNSUPPORTED
errors, plus the feature overview and the changelog.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@WilcoLouwerse @claude
test(e2e): cover archiving, deleting and bringing back on a CMDB re-i…
7da3c82 
…mport

Imports the archived-applications variant after the anonymised export,
checks the archived and deleted counts, the Archived quick filter on the
Applications page and the unchanged contact persons, then imports the
original again and checks unarchived, restored and no duplicates. The
cleanup also looks through the archive and trash lenses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@WilcoLouwerse @claude
style(cmdb-import): wrap the ARCHIVE_UNAVAILABLE message under the li…
663b13c 
…ne limit

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@WilcoLouwerse @claude
docs(openspec): tick the completed tasks of cmdb-import-archive-recon…
8916cb7 
…ciliation

The Playwright test of task 7 is written but not run yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@WilcoLouwerse @claude
fix(cmdb-import): honour a cancel after the last row and during recon…
9993987 
…ciliation

The cancel check ran only at the top of each row, so a cancel pressed
during the last row or in the reconciling phase still archived and
soft-deleted every missing application while the endpoint reported the
cancel as accepted. The reconciliation now checks before it loads its
scope and before each application, stops there and marks the report
cancelled; what it already archived or deleted stays.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@WilcoLouwerse @claude
fix(cmdb-import): prefer a working match over an archived or deleted one
268cf14 
Matching across every state took the first hit. A module deleted by hand
before this change was invisible to the match, so the next import
created a second one with the same import key; when OpenRegister
returned the deleted one first, the import restored it next to the live
one. findOne() now takes a working object before an archived one, and an
archived one before one in the trash, for modules and usages alike.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@WilcoLouwerse @claude
docs(openspec): align the non-goals of cmdb-import-archive-reconcilia…
3878406 
…tion with D5

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@WilcoLouwerse @claude
fix(cmdb-import): read archived and deleted state by lens and from th…
344c3d6 
…e trash

OpenRegister's list search (MagicSearchHandler::convertRowToObjectEntity)
hydrates no archive marker and drops every soft-deleted row, whatever
_includeDeleted says, because it parses the JSON _deleted marker as a date.
A re-import therefore left a returning archived application archived and
created a second module for a soft-deleted one.

Look an object up working first, then with _archived=true, then in the
trash listing (MagicMapper::findDeletedAcrossAllMagicTables), read once per
import, and take its state from where it was found. A restored object is
unarchived when the object restoreObject() answers is archived, and the
update starts from that object. The reconciliation scope reads its usages
and modules the same way. The test doubles now model OpenRegister's search
results, so the existing round-trip tests fail on the old matching.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@WilcoLouwerse @claude
docs(openspec): record how cmdb-import-archive-reconciliation reads o…
986772a 
…bject state

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@WilcoLouwerse @claude
docs(openspec): name the trash read in the cmdb-import-archive-reconc…
141ea24 
…iliation spec and proposal

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

Quality Report — ConductionNL/stackiq @ 1095594

Check PHP Vue Security License Tests
lint ✅
phpcs ✅
phpmd ❌
psalm ✅
phpstan ✅
phpmetrics ✅
eslint ✅
stylelint ✅
build ✅
check-manifest ✅
check-vue-demi ✅
test-l10n ✅
format ✅
check-schema-l10n ✅
check-l10n-js ✅
composer ✅ ✅ 133/133
npm ✅ ✅ 807/807
app:check-code ⏭️
info.xml ✅
REUSE ❌
lockfile sync ✅
PHPUnit ✅ 1253/1253 · 20 skipped
Newman ⏭️
Playwright ⏭️ deferred: E2E runs locally and on the promotion path only. This pull request targets development, so the suite is asked once per promotion into beta and main rather than once per push per open pull request. Run it locally with npx playwright test, or from the Actions tab on a branch with no open pull request into development.
Hydra gates ✅

Quality workflow — 2026-10-09 15:20 UTC

Download the full PDF report from the workflow artifacts.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@WilcoLouwerse