Versioning CLI and Python API - #241
Conversation
The rel_path branch results in only including staged files in the snapshot. Complete snapshots of the remote folder are needed so unchanged files are not omitted in a snapshot.
Previous metadata tracking was only for files. This tracks paths (both files and directories) so we can preserve permissions for directory trees.
This moves the dsi_vcs into dsi as a package under utils so it will be installed and available. This also begins tests for the utility.
The rel_path branch results in only including staged files in the snapshot. Complete snapshots of the remote folder are needed so unchanged files are not omitted in a snapshot.
| txn_id = f"staging-{datetime.datetime.now(datetime.timezone.utc).strftime('%Y%m%d%H%M%S%f')}" | ||
|
|
||
| for raw in paths: | ||
| rel_path = raw |
There was a problem hiding this comment.
Shouldn't this path be relative like it is in add,delete, commit.
There was a problem hiding this comment.
Should probably do validity checks on the path like you do in cmd_add.
There was a problem hiding this comment.
Added validity checks in delete and remove. Every path is considered relative to the root directory. Probably in future, we can add feature that user can add from any subdirectory without specifying the full relative path from root.
There was a problem hiding this comment.
I think you still may want the same path normalization from cmd_add in cmd_delete (e.g., the os.path.relpath) so the staging keys are generated for both. cmd_add also seems to handle absolute paths.
There was a problem hiding this comment.
Added the path normalization. Also added checks to avoid adding outside files through ../../filename from root folder. Also, added recursive file remove and delete.
Fix VCS tests.
| _set_acl_darwin(path, acl_string) | ||
| elif sys.platform == "linux": | ||
| _set_acl_linux(path, acl_string) | ||
| print(f"Unsupported platform for ACL write: {sys.platform}") |
There was a problem hiding this comment.
Should this be in an else branch?
There was a problem hiding this comment.
Yes. Also, I'll add windows stuffs later.
4 participants
Adds Versioning with DSI.
A command line interface
dsi-vcs(rsync-based file version control system) module located indsi/utils/version_control/.A python API integrated with DSI
CLI Commands - Full reference for all 7 commands:
init: Initialize a versioning repository in a root folder.add: Add file(s) to the staging area for the next commit.remove: Remove file(s) from the staging area without touching the actual files.delete: Delete file(s) from the staging area for the next commit.commit: Commit a new version with the staged file(s) and an optional message describing the version.branch: Create a new branch with an optional starting point (commit hash).merge: Merge a branch into the current branch with an optional target commit hash.list-branch: List all branches in the versioning repository.switch: Switch to a different branch in the versioning repository.log: List most recent versions of the current branch.diff: Show differences between two versions. If no version is provided, show difference with the latest.restore: Restore a version with commit hash.clone: Clone a DSI-VCS repository.Python API: Complete
Versionclass reference with usage examplesHelper Functions: Metadata collection utilities from
vcs_metadata_helper.pyDatabase Functions and Schema: SQLite database operations and schema in
vcs_db.pyRepository Log: Tracks every version operations including contents in staging area.
Merkle Tree: Represents the entire directory for a individual version.
File Chunking: Default chunking method used FastCDC. File format specific chunking for the following file extensions
npy,csv,db,sqlite,sqlitle3,json,xml, andxlsx.