Migrated Lucene indexes are reset to Corax at start-up when the index is not locked #5948

Describe the bug

ServiceControl 6.20.0 and 6.21.0 recommend the migration of RavenDB indexes from Corax to Lucene. The documentation said that the migrated index does not need a lock from version 6.20.0. That is not correct.

At each start-up, ServiceControl deploys its index definitions. These definitions have no search engine. RavenDB sees a different definition and builds a side-by-side ReplacementOf/<index> with the database default. For databases created before 6.20, the default is Corax. The migrated index is reset to Corax and rebuilt. On a large database, the rebuild can take days.

A customer unlocked a migrated MessagesViewIndexWithFullTextSearch on an audit instance and the index was reset to Corax.

Steps to reproduce

1. Create an error or audit database with ServiceControl 6.19 or earlier. The database default is Corax.
2. In RavenDB Studio, change the search engine of an index to Lucene on the Configuration tab. Wait for the swap. Do not lock the index.
3. Start ServiceControl 6.20.0 or 6.21.0.
4. The index is rebuilt as ReplacementOf/<index> with Corax and swapped in. The Lucene configuration is lost.

Relevant log output

Warn|ServiceControl.RavenDB.StartupChecks|The following RavenDB index(es) use the Corax search engine: audit/FailedAuditImportIndex, audit/ReplacementOf/MessagesViewIndexWithFullTextSearch, audit/SagaDetailsIndex. ...

Additional Information

• Affected versions: 6.20.0, 6.21.0
• Fixed on master by Keep the search engine configured on an index when deploying index definitions #5946, planned for 6.21.1
• Docs: Index locking is still required for ServiceControl 6.20.0 to 6.21.0 docs.particular.net#8521