Log directory growing to untenable sizes

[CodyCBakerPhD]

I ran into a couple of logging things that seem off, wanted to check whether they're intentional before opening a PR...

On Engaging (as a part of DANDI Compute, which does a lot of automated DANDI download/upload operations) I noticed the ~/.local/state/dandi-cli/log/ directory fills up home disk space (nearly 100 GB!!)

My first instinct is to just relax the log level to error-only events instead of filling it with every last detail of run operations

But digging into it, I noticed two things:

1. DANDI_LOG_LEVEL doesn't seem to work from the CLI. It's honored when you use dandi as a library (it's read here in dandi/__init__.py), but main() in dandi/cli/command.py calls lgr.setLevel(logging.NOTSET) (and the same on the root logger) right after, which looks like it wipes out whatever the user sets. So the env var behaves differently depending on whether you're using the CLI or the Python API

2. Every CLI run drops a new DEBUG log file that never gets cleaned up. main() writes a fresh timestamped file to the log dir on each invocation, and the file handler is pinned to DEBUG no matter what --log-level you pass. There's no 'cache limit' nor pruning, and no way to turn it off, so it just keeps growing and growing

So is the "always log DEBUG to file regardless of --log-level" thing strictly intentional?

I would also recommend we set limits on how much this directory can grow

Any thoughts on what a sane default retention would be, like keep the last N files, or N days?

[yarikoptic]

Indeed we should introduce configuration (ATM env vars) for handling logs and potentially a helper command to clean them up based on some default, config and respecting provided options (e.g. might be worth removing logs for successful runs but keep some problematic logs or alike). I think generally it should be safe to prune older (over 30 days?) logs.

relates also to

• Add configuration system #1521

Re "helper" command, thinking of it , I think following git approach we might want to gain higher level dandi logs which would just list logs (similar to git remote and git branch, and con-duct ls), and also dandi logs remove which would interface desired cleanup .