Invoke-DbaDbDecryptObject - Add DataPages to decrypt without a dedicated admin connection - #10581
Invoke-DbaDbDecryptObject - Add DataPages to decrypt without a dedicated admin connection#10581howarthcd wants to merge 7 commits into
Conversation
…admin connection (do Invoke-DbaDbDecryptObject) Adds -NoDAC, which reads the encrypted definition straight from the raw data pages with DBCC PAGE instead of opening a dedicated admin connection and altering each object inside a rolled back transaction. Nothing is written to the database on this path. Omitting the switch keeps the original behaviour. The reader lives in four new private functions. Get-EncryptedObjectImageValue is the engine, seeking the sysobjvalues clustered index and falling back to a page scan; ConvertFrom-DbccPageDump, ConvertFrom-EncryptedObjectChunk and Get-EncryptedObjectKeystream are split out so they can be unit tested without an instance. Also fixed while here: - A dedicated admin connection this command opens is now closed even when the run fails. The instance loop body is wrapped in try/finally, because an instance allows only one and a leaked session blocked every later run. - Encrypted INSTEAD OF triggers on views are now found, and can only be decrypted with -NoDAC. The default method derives a known plaintext by rewriting the trigger as AFTER INSERT, which a view rejects. - Multi database runs no longer carry objects between databases. - Trigger discovery no longer costs one query per table, and IsEncrypted is added to the SMO init fields, which takes the test suite from 490s to 92s. - -EncodingType warns when bound with -NoDAC, because it is ignored there. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…nit tests (do Invoke-DbaDbDecryptObject) The unit tests reach the new private functions with & (Get-Module dbatools), which only works while exactly one dbatools module is loaded. Invoke-ManualPester imports dbatools.psd1 and dbatools.psm1, leaving a binary module and a script module both named dbatools, so Get-Module returned two objects. PowerShell joined their names and looked for a command called "dbatools dbatools", failing all ten tests that call a private function. The script module that carries the private functions is now resolved once in a Describe level BeforeAll and reused, and the tests throw a clear message if no such module is loaded rather than failing one by one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Nice piece of work. The reverse engineering in the PR description is the most useful thing I have read What I verifiedTest suite, both editions, against the lab:
Because the tests only ever run against one instance, I repeated the byte for byte check on every I also checked three things I expected to be problems and they were not, so for the record:
Please fix in this PR(You don't have to - I can also push those changes to this branch) Object names are interpolated into T-SQL unescaped, in the DAC path. This is not your bug — it is identical on An object name is allowed to contain a single quote, and a schema, object or parent name is allowed to
A name containing a quote therefore ends the literal early and whatever follows it runs as a further The
In the export block, Minor, take or leave
Questions
On the name of the switchNot something for you to decide, and not a blocker — this is for the maintainers, ultimately Chrissy. If the new path turns out to be strictly better — no write, no DAC, decrypts an The thing worth thinking about now is that This text was created by Claude and reviewed by Andreas Jordan. |
…ch the query text (do Invoke-DbaDbDecryptObject) The method that uses the dedicated admin connection interpolated the schema, object and parent names straight into T-SQL: the OBJECT_ID literal of the query that reads the secret, the OBJECT_ID literal of the known secret query, and the EXEC that runs the known plaintext, where the parent carried no brackets at all. An object name may contain a single quote, so a crafted name ended the literal early and whatever followed it ran as a further statement in the same batch, as sysadmin over the dedicated admin connection, while the command returned without error. Creating a procedure in one database was therefore enough to have a statement run as sysadmin the next time somebody decrypted that database. A name may equally contain a closing bracket, which ended the identifier early in the ALTER and left the rest of the name standing as statement text. The object id lookup that -NoDAC already did is now done for both methods, so each selects its rows by id and only ever splices a number. Where the ALTER genuinely needs the name, every closing bracket is doubled, the trigger's parent is written as [schema].[name] from the object rather than interpolated bare, and every single quote in the finished statement is doubled in one place before it goes inside the EXEC literal. Also fixed while here: - New-Item -Path $destinationFolder -ItemType Directory -Force:$Force bound to a parameter this command does not have, so the switch was always off and it would have thrown under StrictMode. It matches the same call in begin now. - $queryKnownPlain survived the object loop, so an object type that matched no branch of the switch would have been altered with the previous object's statement instead of reaching the check that reports a missing known plain. Two integration tests cover it, in their own database because the counts asserted against the shared fixture are exact. They were proven to fail against the previous code, where the dedicated admin connection path returned one of the three objects and created the table that the crafted name spells. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ound and refuse NoDAC on Azure (do Invoke-DbaDbDecryptObject) Finding the encrypted objects asks SMO to fetch IsEncrypted as part of enumerating the module collections, which takes the test suite from 490s to 92s. Those init fields belong to the connection and outlived the command on one the caller owns, so a caller who had chosen their own fields for StoredProcedure, View or UserDefinedFunction had them replaced for the rest of their session. Whatever is set is now captured before the change and put back in the finally that already closes a dedicated admin connection, so it is restored whether the run succeeds or fails. GetDefaultInitFields returns a copy rather than the live collection, which is what makes capturing it work. Measured on SQL Server 2025, reading IsEncrypted on 100 procedures on a fresh connection: 5955ms untouched, 0ms with the fields set, 6151ms after the restore, against a 5897ms control. So the restore undoes the effect rather than only reporting a restored value. A test covers it, using a field set that omits IsEncrypted so a missing restore leaves a visible difference. NoDAC is now refused on Azure SQL Database and Azure SQL Managed Instance, neither of which supports DBCC PAGE. MinimumVersion does not catch either, because both report a version this command is happy with, so the run reached the first DBCC and failed on something that did not name the real problem. Both the engine edition and the engine type are tested, because Connect-DbaInstance's AzureUnsupported tests only the engine type and would let a managed instance through. The refusal is scoped to NoDAC and sits ahead of the sysadmin check, so an Azure user is not told to reconnect as a sysadmin instead. It carries no test, because the branch cannot be reached without an Azure instance. Also in here: - The object ids wanted from a database are collected in one pass rather than by appending, matching the reader. The count is only the encrypted objects of one database, so this is for consistency rather than a measured cost. - Comments reviewed across the reader and the command. Two described an implementation outside this repository, which gives a reader here nothing to act on, and several described what the code used to do rather than what it does. The descent comparison now also records that its multi page case cannot be built, so nothing exercises that branch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…008 and record what read only databases can do (do Invoke-DbaDbDecryptObject) The NoDAC path asked Connect-DbaInstance for a minimum version of 9, on the grounds that sys.sysobjvalues arrived with SQL Server 2005. Tested against a real 2005 instance (9.00.5000.00), the command cannot work there and neither can the rest of the module: enumerating any SMO collection issues CONNECTIONPROPERTY and reading the databases asks for is_cdc_enabled, both of which arrived in 2008, so Get-DbaDatabase and Get-DbaLogin fail the same way. The default method fails one step sooner than NoDAC, at the RemoteDacConnectionsEnabled check, so 2005 is out of reach whichever method is chosen. The floor is now 10 and the help says so for the command rather than for the switch. Raising it does not refuse a 2005 instance, and the comment says why so that nobody later assumes it does. Connect-DbaInstance only applies MinimumVersion when it can read VersionMajor, and on 2005 that property comes back empty for the same reason everything else does, so both 9 and 10 allow the connection and the failure still happens inside SMO. SQL Server 2008 (10.0.5500.0) was tested and passes: the index seek route is taken, so the on disk index record decodes correctly there, and every storage shape comes back byte exact, in row, off row at 20 KB, a three level blob tree at 300 KB and a view holding non ASCII characters. The page allocations DMV does not exist before 2012, so this is also the first time the page chain walk has run against a version that genuinely needs it rather than one where ForceChainWalk made it, and the seek, the page list and the chain all agree. Both instances were provisioned for the exercise and no longer exist, so neither result is repeatable and neither is in CI. Also in here, a read only database is now documented as the second case that needs NoDAC, alongside an INSTEAD OF trigger on a view, and for the same underlying reason: the default method has to alter the object to obtain a known plaintext. A database snapshot and an availability group readable secondary can both be read with NoDAC and neither can be read without it. The snapshot half has a test, which covers an in row definition and one stored off row, and it also confirms that a snapshot reports the family GUID of its source, without which the keystream would be wrong. The readable secondary half was verified by hand on SQL Server 2019 and cannot be tested here, because no instance available provides one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Thank you for this. The injection in particular is a much better catch than a rubber stamp, and taking Everything under "Please fix in this PR" is done, along with both minor points. All three questions are What I fixedObject names are interpolated into T-SQL unescaped, in the DAC path. The object id lookup that Where the
I confirmed the same way you did, with a procedure whose name spells the end of the literal followed by After the fix, 3 of 3 byte exact on both methods and no table. There are two integration tests for I have kept the crafted name out of this comment for the same reason you did, but it is in the test
Fixed, to plain Minor, take or leave
Now put back. You called it a note rather than a defect, but it turned out to be cheap and provably What made it viable is that Measured on SQL Server 2025, reading
So the restore genuinely undoes the effect rather than just reporting a restored value. There is a test
A decision, and your instinct about where to look was right in one of the two places. The rule I applied was per page, per dump line or per blob link versus per object, per chunk or per
The general rule is now stated once in the reader's One more, found while making the escaping changes.
QuestionsWhich versions did you verify the on-disk format against? The same three as you: SQL Server 2019 (15.0.4440.1), 2022 (16.0.4262.2) and 2025 (17.0.4060.2), all I went looking for the older end you named, and found something worth raising beyond this PR. SQL Server
This is not specific to the new path. The default method fails on 2005 one step sooner: its first action It is worth setting this beside #9821, which is the nearest existing report and was closed as an That made One caveat goes with it, and the comment in the code carries the same note, because the change otherwise SQL Server 2008 is a different story, and it passes cleanly. On 10.0.5500.0, Express Edition 64-bit:
Two things in there are worth more than the pass itself. The seek route was taken on 2008, so the 20 So the on disk format is now verified at both ends of the range the module can reach: 2008 at the One caveat on those two results. The 2005 and 2008 instances were stood up for this exercise and are What happens on Azure SQL Database and Managed Instance? You were right, and it is now refused explicitly. Neither platform supports The check runs before anything else on that path: if ($server.DatabaseEngineEdition -eq "SqlManagedInstance") {
$azurePlatform = "Azure SQL Managed Instance"
} elseif ($server.DatabaseEngineType -eq "SqlAzureDatabase") {
$azurePlatform = "Azure SQL Database"
}Both properties are needed. It is scoped to Two things I want to be straight about rather than let the code imply otherwise:
Have you tried this against an AG readable secondary or a database snapshot? Both, and A readable secondary was tested on SQL Server 2019. That is the result I would hope for rather than a A database snapshot was tested on SQL Server 2022, holding one in row definition and one 20 KB The same run shows the contrast with the default method, which fails on a snapshot for the reason you So a read only database is a second case where On the name of the switchAgreed on both counts: worth settling now rather than deprecating later, and not mine to decide. The I am deliberately not proposing an alternative, because a name offered by the author of the PR tends to What I verifiedTest suite, against the lab:
That is the previous 40 plus two for the quoting of object names, one for the init fields and one for Repository wide compliance checks pass on both runs, no ScriptAnalyzer findings on the changed files, This text was created by Claude and reviewed by Chris Howarth. |
|
Thank you for this, and particularly for the parts where you went and measured something rather than I have re-verified everything rather than taking the summary on trust. Confirmed fixed
The separation in the escaping is the part I looked hardest at, since it is where a fix like this Your point about doubling in one place rather than by hand is the part that actually closes the hole, One more, and I think it needs fixing before mergeTwo objects can share a key in The map is keyed
Both key as Nothing warns. The object is returned under its own schema and name, with another object's body in Two things about scope, so this is not read as worse than it is. It needs a colliding pair to exist — For comparison, I am not going to prescribe a shape. The general point is only that a single string built from two SQL Server 2005This is worth more than a paragraph in a PR thread, and I agree it should not stay in one. SMO's own The caveat you attached to raising the floor to Accepted without verificationRecording these so it is clear what the evidence does and does not cover:
Nit, take or leave
On the name of the switchAgreed, and leaving it with the maintainers is the right call. Nothing further from me. This text was created by Claude and reviewed by Andreas Jordan. |
|
Picking the switch question back up, since you said you would happily rename once the maintainers chose The shapeReplace [ValidateSet("DAC", "DataPages")]
[string]$DecryptionMethod = (Get-DbatoolsConfigValue -FullName "commands.Invoke-DbaDbDecryptObject.decryptionmethod" -Fallback "DataPages")Set-DbatoolsConfig -FullName "commands.Invoke-DbaDbDecryptObject.decryptionmethod" -Value "DataPages" -Initialize -Validation string -Description "Which method Invoke-DbaDbDecryptObject uses to read an encrypted definition. DataPages reads the raw data pages and writes nothing. DAC uses a dedicated admin connection and briefly alters each object inside a rolled back transaction."This is an existing pattern rather than a new one.
Our suggestion: ship with DataPages as the defaultNot the conservative choice, deliberately. The new method is better on every axis a user cares about:
The evidence behind it is good. Byte exact on 2008, 2019, 2022 and 2025 in your testing, and I have There is precedent in the module for exactly this move. That said, this is Chrissy's call, not ours. What we would ask either way is that the endpoint gets One thing to settle before the default flipsAzure. This is the only case we can find where the flip could remove something rather than add it. Everything Two things that bite with a configuration backed default
The DocumentationIf the default changes, it needs saying somewhere a user reads before they hit it, not only in the We would also like the command to state at verbose level which method it actually used. With the method This text was created by Claude and reviewed by Andreas Jordan. |
… lookup keyed by schema and name (do Invoke-DbaDbDecryptObject) Both decrypt methods identified an object by an id looked up in a hashtable keyed "$schema.$name". Either half of that key may itself contain a dot: schema [a], object [b.c] and schema [a.b], object [c] both key as "a.b.c", so whichever row the lookup read last silently overwrote the other's id, and the object that lost the race was then decrypted with the other one's ciphertext under its own name and no warning. Andreas Jordan found this in review, with a fixture on SQL Server 2022 showing both methods returning 2 of 2 objects and one of the pair carrying the wrong definition. The id now comes straight from SMO's own ID property on each object, fetched as part of the same enumeration that already asks for IsEncrypted, rather than from a second query joined back to the objects by name. Confirmed against a live instance that SMO's ID matches sys.objects.object_id for a stored procedure, a view, a user-defined function and a trigger. ID is added to the init fields the command already forces for StoredProcedure, View and UserDefinedFunction, so fetching it costs no additional round trip; the query that built the old lookup map is gone entirely, along with the per-object name-based reads it fed. Because two colliding objects can no longer share a key, they cannot collide. Reproduced the exact fixture from the review against the previous code on both methods, then confirmed 2 of 2 byte exact, each with its own definition, once the id was read from SMO. Two integration tests cover it, in their own database because the counts asserted against the shared fixture are exact. Also checked: the review's other nit, that $savedInitFieldsProcedure and its two siblings were not reset between instances in the foreach, turned out to already be fixed, in the commit before this one - they are set to $null at the top of every instance iteration, ahead of the try that reassigns them. Verified against the current file rather than assumed from an earlier summary. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(do Invoke-DbaDbDecryptObject) Andreas Jordan pointed out that -NoDAC is a negative switch: a future configuration-backed default would read as "set NoDAC to true so that no DAC is used," and opting back out at the call site would be -NoDAC:$false, a double negative. He proposed replacing it with a -DecryptionMethod parameter carrying a ValidateSet and a configuration-backed default, following the pattern Add-DbaAgReplica already uses for ClusterType. Adopting the name change without the rest of that shape. The parameter is now [switch]$DataPages, same meaning and same default as -NoDAC: omitting it still uses the DAC method. Every mention in the help, the messages and the test suite is updated to match. The bigger design - a string parameter, a configuration setting, and a decision on which method the default should be - is left with the maintainers, since it changes runtime behaviour rather than a name and isn't something to build ahead of that decision. Full suite passes with the new name: 46/46, compliance 21/21, 0 parse/format/ analyzer/PS3 findings, no leftover databases or DAC sessions afterward. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Thank you for this - the collision was a real gap in what looked like a complete fix, and picking the switch name back up rather than letting it drift was the right call. Collision bug - fixed
Reproduced your exact fixture against the previous code on both methods (both returned the wrong definition for one of the pair), then confirmed 2 of 2 byte exact, each with its own definition, once the id came from SMO. Two new integration tests cover it, in their own database. Full suite: 46/46 passed, compliance 21/21, 0 parse/format/analyzer/PS3 findings, no leftover databases or DAC sessions afterward. The nit - already fixed before it was raisedChecked against the current file rather than assumed: Switch renamed to
|
(do Invoke-DbaDbDecryptObject)
Adds -NoDAC, which reads the encrypted definition straight from the raw data pages with DBCC PAGE instead of opening a dedicated admin connection and altering each object inside a rolled back transaction. Nothing is written to the database on this path. Omitting the switch keeps the original behaviour.
The reader lives in four new private functions. Get-EncryptedObjectImageValue is the engine, seeking the sysobjvalues clustered index and falling back to a page scan; ConvertFrom-DbccPageDump, ConvertFrom-EncryptedObjectChunk and Get-EncryptedObjectKeystream are split out so they can be unit tested without an instance.
Also fixed while here:
Type of Change
Invoke-ManualPester -Path <command> -ScriptAnalyzer -Compliance)Purpose
The existing command can only reach an encrypted definition through a dedicated admin connection, and it obtains the known plaintext it needs by altering every object inside a transaction that is rolled back. That rules the command out where a DAC is unavailable or where writing to the database, even transiently, is unacceptable, and an instance allows only one DAC at a time.
It also cannot decrypt an encrypted
INSTEAD OFtrigger defined on a view at all, because the known plaintext it builds rewrites the object as anAFTERtrigger and a view rejects that.Approach
-NoDACderives the RC4 key from public metadata rather than obtaining a known plaintext, so it needs no dedicated admin connection and writes nothing. The scheme is set out under Learning below.The ciphertext lives in
sys.sysobjvalues, which is DAC-only through T-SQL, so the raw pages are read withDBCC PAGE ... WITH TABLERESULTSand the family GUID withDBCC DBINFO WITH TABLERESULTS. Both need sysadmin, checked up front so the failure is a clear message rather than a permission error midway through reading pages.Rows are found by seeking the
sysobjvaluesclustered index, about five page reads whatever the size of the database, with a full page scan as the fallback and as the test oracle.Comment-based help was updated throughout, including a note that a view trigger requires
-NoDAC.Commands to test
The help examples cover the normal paths. Beyond those:
Learning
The part worth writing down is the obfuscation scheme itself.
WITH ENCRYPTIONis widely described as "not really encryption", but the actual construction does not appear to be written up anywhere, so it was reverse engineered for this change and is documented here in case it is useful to anyone else.SQL Server stores the module text in
sys.sysobjvalues.imageval, keyed on the object id withvalclass = 1. The bytes are the UCS-2 (UTF-16LE) source text XORed with an RC4 keystream. There is no secret: the RC4 key is a SHA1 over 22 bytes of metadata that any sysadmin can already read.Four details are load bearing, and each of them fails in a way that is quiet rather than obvious:
System.Guidlayout, where the first three fields are little endian, not the order the GUID prints in. Using the string order produces a valid looking key and complete garbage. The value isdbi_familyGUIDfromDBCC DBINFO, and it is a property of the database family rather than of the object.colIdis an input to the key, so a definition that spans more than onesysobjvaluesrow needs a separate keystream per row. Deriving one keystream for the whole object leaves the first chunk perfectly readable and everything after it mojibake, which reads like an encoding bug rather than a key bug. The chunks also have to be concatenated incolIdorder rather than in the order the rows were read.This is also why the existing method works at all. It never derives the key: it alters the object to a placeholder of exactly the same length inside a transaction that is rolled back, which yields a known plaintext and its matching ciphertext, and XORing the three values together recovers the original. That is a clever way around not knowing the key, but it costs a DAC, a write, and it cannot be applied to an object whose definition it is unable to legally rewrite, which is exactly the
INSTEAD OFtrigger on a view case. Deriving the key directly removes all three constraints.