[Bug]Toolkit db mv may delete the source database after a copy failure, causing data loss

[halibobo1205]

Bug Description

When Toolkit db mv copies database files, an IOException from an individual Files.copy call is printed but not propagated or recorded as an overall migration failure:

try {
  Files.copy(original, destination, StandardCopyOption.REPLACE_EXISTING);
} catch (IOException e) {
  spec.commandLine().getErr().println(e);
}

After copying, the command still attempts to delete the source directory. If deletion succeeds, it replaces the original path with a symbolic link to the destination:

if (FileUtils.deleteDir(p.original.toFile())) {
  Files.createSymbolicLink(p.original, p.destination);
}

If copying fails because the destination runs out of disk space or inodes, or because of another I/O error, the source database may be deleted and replaced with a symbolic link to an incomplete LevelDB/RocksDB database. The command may still print move db done. and exit with status 0.

The node may then fail to start and require recovery from a backup or a full resynchronization.

Environment

Network

N/A (offline maintenance tool)

Software Versions

OS: macOS 15 / Linux
JVM: Java 8 / Java 17
Git Commit: f8b05d40abc949fa588ab64d8fd8fd82845ebeed
Version: GreatVoyage-v4.8.2.1
Code: N/A

Expected Behavior

If any file copy fails, the command should:

• Preserve all source databases.
• Not create a symbolic link to an incomplete destination.
• Attempt to clean up incomplete destination data created by the migration.
• Not print move db done..
• Return a non-zero exit status.

Source directories should be deleted and replaced with symbolic links only after all databases have been copied successfully. A source deletion or symbolic-link creation failure should also return a non-zero status and provide recovery instructions.

Actual Behavior

A copy exception is printed and ignored. The command still attempts to delete the source directory and, if deletion succeeds, creates the symbolic link. Copy and finalization failures are not reliably propagated to the caller, so the command may still print move db done. and return 0.

Frequency

• Always (100%)
• Frequently (>50%)
• Sometimes (10-50%)
• Rarely (<10%)

The data-loss path is consistently reached when a file copy fails but the subsequent source deletion and symbolic-link creation succeed.

Steps to Reproduce

Use only a disposable database from a stopped node and an isolated test filesystem. Do not fill a production disk to reproduce this issue.

1. Configure the destination on an isolated test filesystem whose available space is smaller than the source database.
2. Ensure that the destination database path does not exist.
3. Run:

java -jar Toolkit.jar db mv -d output-directory -c config.conf
echo $?

4. Wait for the copy to fail because the destination filesystem runs out of space, then inspect the source path, destination directory, and exit status.

Based on the current code flow, if the subsequent deletion and symbolic-link creation succeed, the source path will become a symbolic link to an incomplete destination, while the command still reports success and returns 0.

Logs and Error Messages

The following is the expected error form for a full destination filesystem. It illustrates the current code flow and is not a captured reproduction log. Exact text depends on the operating system and filesystem.

java.nio.file.FileSystemException: <destination file>: No space left on device
move db done.
(exit code 0)

Additional Context (Optional)

Related Issues

N/A

Possible Solution

Use a two-phase migration:

1. Copy all databases first. If any copy fails, preserve every source database, attempt to remove incomplete destination data, and return a non-zero status.
2. Delete the source directories and create symbolic links only after all copies succeed.

A source deletion or symbolic-link creation failure should also return a non-zero status and provide recovery instructions.

[halibobo1205]

Implementation design

Problem addressed: The previous db mv implementation ignored per-file copy errors. It could then delete the source and link it to an incomplete destination while still printing move db done. and returning exit code 0.

Invariant: For a stopped node with a stable, non-overlapping source/destination layout, no source database is deleted or replaced until every selected database has been copied successfully. Handled validation, copy, and finalization failures return non-zero and report what remains on disk.

Flow:

1. Validate before writing. ConfigConverter performs database-independent checks during option parsing. Path validation runs later in call(), after all options have their final values. It rejects a missing source, a regular-file source, a source symlink, an existing destination—including a dangling symlink—or a destination equal to its source. Validation failures exit with code 2 before creating any destination.
2. Phase 1 — copy databases. Each source is walked recursively without following links. Directories are recreated, including empty directories; regular files are copied; symlinks, special entries, unreadable attributes, and traversal failures cause the copy to fail.
3. Handle copy failures. allMatch prevents later databases from starting after one database fails. Within a database, entries that observe hasError are skipped, while parallel copies already in progress may finish before cleanup begins. The command then attempts to remove every configured destination (all were validated absent at start, so anything present was created by this run) and exits with code 1. If cleanup succeeds, the command can be retried after correcting the I/O problem. If cleanup is incomplete, it reports the remaining paths for manual removal.
4. Phase 2 — replace sources with links. This phase starts only after every database copy succeeds. Each source is deleted and replaced with a symbolic link to its destination. Finalization continues through every selected database so that each result is reported. A deletion or link-creation failure keeps the complete destination, prints manual recovery instructions, and causes exit code 1.
5. Report success. move db done. is printed and exit code 0 is returned only after every database has been copied and every source has been replaced with its symbolic link.

Exit codes:

• 0: all copy and source-to-link operations completed successfully;
• 1: handled copy, cleanup, deletion, or link-creation failure;
• 2: configuration or path-validation failure.

Non-goals:

• Detecting a running node; stopping the node remains a documented precondition.
• Resumable copying; retries start from a clean destination after successful cleanup.
• Validating nested or overlapping path graphs beyond destination != source; the design assumes stable, non-overlapping paths.

Sequence Diagram

sequenceDiagram
    autonumber
    actor Op as Operator
    participant CLI as picocli
    participant CC as ConfigConverter
    participant MV as DbMove.call()
    participant CP as Copy phase
    participant LK as Link phase
    participant FS as File system

    Op->>CLI: db mv -d database-dir -c config-file
    CLI->>CC: parse configuration
    Note over CC: Keep usable name/path entries,<br/>require at least one,<br/>reject duplicate names
    alt configuration invalid
        CC-->>Op: error, exit 2
    end

    CLI->>MV: call() after options are final
    loop each selected database
        MV->>MV: construct and validate Property
        Note over MV: Reject missing/file/symlink source,<br/>existing destination including dangling link,<br/>or destination equal to source
    end
    alt path validation fails
        MV-->>Op: error, exit 2, nothing written
    end

    rect rgb(232, 240, 254)
        Note over CP,FS: Phase 1 — copy databases
        loop each database until one fails
            MV->>CP: copy(database)
            CP->>FS: create destination
            CP->>FS: recursively walk source
            loop entries processed in parallel
                CP->>CP: read attributes with NOFOLLOW_LINKS
                alt directory
                    CP->>FS: create directory
                else regular file
                    CP->>FS: copy file
                else unsupported entry or I/O failure
                    CP->>CP: record failure
                end
            end
            Note over CP: New work observing hasError is skipped,<br/>in-flight copies may finish
        end
    end

    alt copy failed
        MV->>FS: attempt to remove the configured destinations
        alt cleanup succeeded
            MV-->>Op: sources kept, exit 1,<br/>retry after correcting the failure
        else cleanup incomplete
            MV-->>Op: sources kept, report leftovers,<br/>manual cleanup required, exit 1
        end
    end

    rect rgb(254, 240, 232)
        Note over LK,FS: Phase 2 — replace sources with links
        loop each selected database
            MV->>LK: replaceSourceWithLink(database)
            LK->>FS: delete source
            alt deletion failed
                LK-->>Op: report source state,<br/>complete copy, and recovery instructions
            else source deleted
                LK->>FS: create symbolic link
                alt link creation failed
                    LK-->>Op: report complete copy<br/>and recovery instructions
                end
            end
        end
    end

    alt any finalization failed
        MV-->>Op: exit 1
    else all operations succeeded
        MV-->>Op: print "move db done.", exit 0
    end

Loading

Design guarantees

1. Phase 1 does not delete or replace any source database.
2. A copy failure prevents the entire link phase from starting.
3. A copy failure preserves every source and attempts to remove every configured destination.
4. A cleanup failure is reported explicitly instead of claiming that the command is directly retryable.
5. Complete destination copies are retained when finalization fails.
6. move db done. and exit code 0 occur only after the entire migration succeeds.

[halibobo1205]

This will be fixed by #6946.