Skip to content

Commit 67da6e3

Browse files
Merge pull request #1235 from ConductionNL/fix/cmdb-import-part-limit-read-sheets
fix(cmdb-import): read an export whose unread sheets are larger than the part limit, with a clearer message
2 parents 6e1c9b5 + 905cfaa commit 67da6e3

13 files changed

Lines changed: 798 additions & 67 deletions

File tree

‎docs/features/cmdb-import.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -288,7 +288,7 @@ and the section shows the reason and the error code.
288288
| `NO_SOURCE_SHEET` | Neither `Onbeh Applicaties CMDB` nor `Beheerde Applicaties CMDB` is in the workbook. | Check the sheet names; they must match exactly. |
289289
| `MISSING_COLUMN` | A present CMDB sheet has no `APPID` or `Applicatie Naam` column. The message names the sheet and the column. | Add the column to that sheet. |
290290
| `TOO_MANY_ROWS` | A CMDB sheet has more rows with data than the row limit (10,000 by default). The message names the sheet and the limit. | Split the export and import the parts one after the other. |
291-
| `WORKBOOK_TOO_LARGE` | Unpacked, the workbook is larger than the import reads: 50 MB in all, 10 MB for any one part (such as a sheet or the table of texts the sheets share), or more than 200,000 different texts, by default. An `.xlsx` is a compressed package, so a small file can unpack to far more. The message names the limit, and for a part also the part. | Remove sheets the import does not read, such as the archive sheet, or split the export. |
291+
| `WORKBOOK_TOO_LARGE` | Unpacked, the workbook is larger than the import reads: 50 MB in all, 10 MB for one sheet the import reads or for the table of texts the sheets share, or more than 200,000 different texts, by default. Sheets the import does not read may be larger than 10 MB. An `.xlsx` is a compressed package, so a file of a few MB can unpack to far more. The message names the limit, and for a sheet the sheet and its unpacked size. | For a sheet that is too large: split its rows over two copies of the export and import them one after the other. For a workbook that is too large in all: delete the sheets the import does not read from a copy and import that copy. |
292292
| `MUNICIPALITY_AMBIGUOUS` | More than one municipality has the typed name. The import does not guess which one. | Pick the municipality from the list instead of typing its name. |
293293
| `IMPORT_IN_PROGRESS` | Another CMDB import is running. Only one import runs at a time. | Wait until it has finished and try again. |
294294
| `FIELD_INVALID` | A form field of the request has a value the import does not accept, for example an `updateExisting` or `publish` that is neither `true` nor `false`. The message names the field. | Not reachable from the section; reported for API callers. |
@@ -324,7 +324,7 @@ every import:
324324
| `maxFileBytes` | `10485760` (10 MB) | the size of the uploaded file |
325325
| `maxRowsPerSheet` | `10000` | the rows with data on one CMDB sheet |
326326
| `maxUncompressedBytes` | `52428800` (50 MB) | the size of the workbook once unpacked, checked before a sheet is parsed |
327-
| `maxPartBytes` | `10485760` (10 MB) | the size of any one part of the workbook once unpacked, such as a sheet or the shared-strings table, checked before a sheet is parsed |
327+
| `maxPartBytes` | `10485760` (10 MB) | the size of one part the import parses once unpacked, such as a source sheet or the shared-strings table, checked before a sheet is parsed; a sheet the import does not read may be larger, and is read as an empty sheet |
328328
| `maxSharedStrings` | `200000` | the number of different texts in the workbook's shared-strings table, each formatted piece of a text with mixed formatting counted as one more, also in text written in a cell itself, counted before a sheet is parsed |
329329
| `maxReferencedStringBytes` | `67108864` (64 MB) | the shared text all cells of the workbook reference together, counted once per cell and once per CMDB sheet before a sheet is parsed; a long text that many cells repeat counts as often as it is repeated |
330330

‎l10n/en.js‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1110,7 +1110,6 @@ OC.L10N.register(
11101110
"Another CMDB import is running; try again when it has finished.": "Another CMDB import is running; try again when it has finished.",
11111111
"Several municipalities have this name; choose one from the list.": "Several municipalities have this name; choose one from the list.",
11121112
"Unpacked, the workbook is larger than {size}, the most the import reads.": "Unpacked, the workbook is larger than {size}, the most the import reads.",
1113-
"Remove sheets the import does not read, such as the archive sheet, or split the export, and try again. Nothing was imported.": "Remove sheets the import does not read, such as the archive sheet, or split the export, and try again. Nothing was imported.",
11141113
"The \"{schema}\" schema of the stackiq register is out of date.": "The \"{schema}\" schema of the stackiq register is out of date.",
11151114
"The stackiq register is out of date.": "The stackiq register is out of date.",
11161115
"It lacks properties the import matches on. Press Force Update at the top of this page to import the register configuration again, then try again. Nothing was imported.": "It lacks properties the import matches on. Press Force Update at the top of this page to import the register configuration again, then try again. Nothing was imported.",
@@ -1125,7 +1124,11 @@ OC.L10N.register(
11251124
"The key a repeated import matches this application on: topdesk:<municipality uuid>:<APPID>. Set by the CMDB import; only a Nextcloud admin can change it.": "The key a repeated import matches this application on: topdesk:<municipality uuid>:<APPID>. Set by the CMDB import; only a Nextcloud admin can change it.",
11261125
"conflict: the application with this import key is used by another organisation, so it is not changed": "conflict: the application with this import key is used by another organisation, so it is not changed",
11271126
"When on, a re-import overwrites the application's name, descriptions, application type, hosting model, BBN level, source fields and supplier, and the usage's status, phase-out date and business owner, with the values from the export. The usage's TIME classification and internal note are only set when the usage is created or the field is empty, so changes made in stackiq stay.": "When on, a re-import overwrites the application's name, descriptions, application type, hosting model, BBN level, source fields and supplier, and the usage's status, phase-out date and business owner, with the values from the export. The usage's TIME classification and internal note are only set when the usage is created or the field is empty, so changes made in stackiq stay.",
1128-
"Unpacked, the part {part} of the workbook is larger than {size}, the most the import reads of one part.": "Unpacked, the part {part} of the workbook is larger than {size}, the most the import reads of one part.",
1127+
"An .xlsx file is compressed, so a file of a few MB can be much larger once unpacked. Nothing was imported.": "An .xlsx file is compressed, so a file of a few MB can be much larger once unpacked. Nothing was imported.",
1128+
"Split the rows over two copies of the export and import them one after the other.": "Split the rows over two copies of the export and import them one after the other.",
1129+
"The sheet \"{sheet}\" is too large to import: unpacked it is {size}, and the import reads at most {limit} of one sheet.": "The sheet \"{sheet}\" is too large to import: unpacked it is {size}, and the import reads at most {limit} of one sheet.",
1130+
"The part {part} of the workbook is too large to import: unpacked it is {size}, and the import reads at most {limit} of one part.": "The part {part} of the workbook is too large to import: unpacked it is {size}, and the import reads at most {limit} of one part.",
1131+
"Delete the sheets the import does not read from a copy of the export (it reads only \"{first}\" and \"{second}\"), and import that copy.": "Delete the sheets the import does not read from a copy of the export (it reads only \"{first}\" and \"{second}\"), and import that copy.",
11291132
"The workbook holds more than {count} different texts, the most the import reads.": "The workbook holds more than {count} different texts, the most the import reads.",
11301133
"Together, the cells of the workbook reference more than {size} of shared text, the most the import reads.": "Together, the cells of the workbook reference more than {size} of shared text, the most the import reads."
11311134
},

‎l10n/en.json‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1109,7 +1109,6 @@
11091109
"Another CMDB import is running; try again when it has finished.": "Another CMDB import is running; try again when it has finished.",
11101110
"Several municipalities have this name; choose one from the list.": "Several municipalities have this name; choose one from the list.",
11111111
"Unpacked, the workbook is larger than {size}, the most the import reads.": "Unpacked, the workbook is larger than {size}, the most the import reads.",
1112-
"Remove sheets the import does not read, such as the archive sheet, or split the export, and try again. Nothing was imported.": "Remove sheets the import does not read, such as the archive sheet, or split the export, and try again. Nothing was imported.",
11131112
"The \"{schema}\" schema of the stackiq register is out of date.": "The \"{schema}\" schema of the stackiq register is out of date.",
11141113
"The stackiq register is out of date.": "The stackiq register is out of date.",
11151114
"It lacks properties the import matches on. Press Force Update at the top of this page to import the register configuration again, then try again. Nothing was imported.": "It lacks properties the import matches on. Press Force Update at the top of this page to import the register configuration again, then try again. Nothing was imported.",
@@ -1124,7 +1123,11 @@
11241123
"The key a repeated import matches this application on: topdesk:<municipality uuid>:<APPID>. Set by the CMDB import; only a Nextcloud admin can change it.": "The key a repeated import matches this application on: topdesk:<municipality uuid>:<APPID>. Set by the CMDB import; only a Nextcloud admin can change it.",
11251124
"conflict: the application with this import key is used by another organisation, so it is not changed": "conflict: the application with this import key is used by another organisation, so it is not changed",
11261125
"When on, a re-import overwrites the application's name, descriptions, application type, hosting model, BBN level, source fields and supplier, and the usage's status, phase-out date and business owner, with the values from the export. The usage's TIME classification and internal note are only set when the usage is created or the field is empty, so changes made in stackiq stay.": "When on, a re-import overwrites the application's name, descriptions, application type, hosting model, BBN level, source fields and supplier, and the usage's status, phase-out date and business owner, with the values from the export. The usage's TIME classification and internal note are only set when the usage is created or the field is empty, so changes made in stackiq stay.",
1127-
"Unpacked, the part {part} of the workbook is larger than {size}, the most the import reads of one part.": "Unpacked, the part {part} of the workbook is larger than {size}, the most the import reads of one part.",
1126+
"An .xlsx file is compressed, so a file of a few MB can be much larger once unpacked. Nothing was imported.": "An .xlsx file is compressed, so a file of a few MB can be much larger once unpacked. Nothing was imported.",
1127+
"Split the rows over two copies of the export and import them one after the other.": "Split the rows over two copies of the export and import them one after the other.",
1128+
"The sheet \"{sheet}\" is too large to import: unpacked it is {size}, and the import reads at most {limit} of one sheet.": "The sheet \"{sheet}\" is too large to import: unpacked it is {size}, and the import reads at most {limit} of one sheet.",
1129+
"The part {part} of the workbook is too large to import: unpacked it is {size}, and the import reads at most {limit} of one part.": "The part {part} of the workbook is too large to import: unpacked it is {size}, and the import reads at most {limit} of one part.",
1130+
"Delete the sheets the import does not read from a copy of the export (it reads only \"{first}\" and \"{second}\"), and import that copy.": "Delete the sheets the import does not read from a copy of the export (it reads only \"{first}\" and \"{second}\"), and import that copy.",
11281131
"The workbook holds more than {count} different texts, the most the import reads.": "The workbook holds more than {count} different texts, the most the import reads.",
11291132
"Together, the cells of the workbook reference more than {size} of shared text, the most the import reads.": "Together, the cells of the workbook reference more than {size} of shared text, the most the import reads."
11301133
}

‎l10n/nl.js‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1180,7 +1180,6 @@ OC.L10N.register(
11801180
"Another CMDB import is running; try again when it has finished.": "Er loopt al een andere CMDB-import; probeer het opnieuw wanneer die klaar is.",
11811181
"Several municipalities have this name; choose one from the list.": "Meerdere gemeenten hebben deze naam; kies er een uit de lijst.",
11821182
"Unpacked, the workbook is larger than {size}, the most the import reads.": "Uitgepakt is het werkboek groter dan {size}, het maximum dat de import leest.",
1183-
"Remove sheets the import does not read, such as the archive sheet, or split the export, and try again. Nothing was imported.": "Verwijder tabbladen die de import niet leest, zoals het archieftabblad, of splits de export, en probeer het opnieuw. Er is niets geïmporteerd.",
11841183
"The \"{schema}\" schema of the stackiq register is out of date.": "Het schema \"{schema}\" van het stackiq-register is verouderd.",
11851184
"The stackiq register is out of date.": "Het stackiq-register is verouderd.",
11861185
"It lacks properties the import matches on. Press Force Update at the top of this page to import the register configuration again, then try again. Nothing was imported.": "Het mist eigenschappen waarop de import records herkent. Klik bovenaan deze pagina op Force Update om de registerconfiguratie opnieuw te importeren en probeer het daarna opnieuw. Er is niets geïmporteerd.",
@@ -1195,7 +1194,11 @@ OC.L10N.register(
11951194
"The key a repeated import matches this application on: topdesk:<municipality uuid>:<APPID>. Set by the CMDB import; only a Nextcloud admin can change it.": "De sleutel waarop een herhaalde import deze applicatie koppelt: topdesk:<uuid van de gemeente>:<APPID>. Gezet door de CMDB-import; alleen een Nextcloud-beheerder kan deze wijzigen.",
11961195
"conflict: the application with this import key is used by another organisation, so it is not changed": "conflict: de applicatie met deze importsleutel wordt door een andere organisatie gebruikt en wordt daarom niet gewijzigd",
11971196
"When on, a re-import overwrites the application's name, descriptions, application type, hosting model, BBN level, source fields and supplier, and the usage's status, phase-out date and business owner, with the values from the export. The usage's TIME classification and internal note are only set when the usage is created or the field is empty, so changes made in stackiq stay.": "Staat dit aan, dan overschrijft een herhaalde import de naam, de omschrijvingen, het applicatietype, het hostingmodel, het BBN-niveau, de bronvelden en de leverancier van de applicatie, en de status, de uitfaseerdatum en de functioneel eigenaar van het gebruik, met de waarden uit de export. De TIME-classificatie en de interne notitie van het gebruik worden alleen gezet wanneer het gebruik wordt aangemaakt of het veld leeg is, zodat wijzigingen in stackiq blijven staan.",
1198-
"Unpacked, the part {part} of the workbook is larger than {size}, the most the import reads of one part.": "Uitgepakt is het onderdeel {part} van de werkmap groter dan {size}, het maximum dat de import van één onderdeel leest.",
1197+
"An .xlsx file is compressed, so a file of a few MB can be much larger once unpacked. Nothing was imported.": "Een .xlsx-bestand is gecomprimeerd: een bestand van een paar MB kan uitgepakt veel groter zijn. Er is niets geïmporteerd.",
1198+
"Split the rows over two copies of the export and import them one after the other.": "Verdeel de rijen over twee kopieën van de export en importeer die na elkaar.",
1199+
"The sheet \"{sheet}\" is too large to import: unpacked it is {size}, and the import reads at most {limit} of one sheet.": "Het tabblad \"{sheet}\" is te groot om te importeren: uitgepakt is het {size}, en de import leest van één tabblad hooguit {limit}.",
1200+
"The part {part} of the workbook is too large to import: unpacked it is {size}, and the import reads at most {limit} of one part.": "Het onderdeel {part} van de werkmap is te groot om te importeren: uitgepakt is het {size}, en de import leest van één onderdeel hooguit {limit}.",
1201+
"Delete the sheets the import does not read from a copy of the export (it reads only \"{first}\" and \"{second}\"), and import that copy.": "Verwijder in een kopie van de export de tabbladen die de import niet leest (hij leest alleen \"{first}\" en \"{second}\"), en importeer die kopie.",
11991202
"The workbook holds more than {count} different texts, the most the import reads.": "De werkmap bevat meer dan {count} verschillende teksten, het maximum dat de import leest.",
12001203
"Together, the cells of the workbook reference more than {size} of shared text, the most the import reads.": "Samen verwijzen de cellen van de werkmap naar meer dan {size} gedeelde tekst, het maximum dat de import leest."
12011204
},

0 commit comments

Comments
 (0)