Small tools for the messy first hour with field data.
Field data does not arrive as a dataset. It arrives as a hard drive, a shared folder, or a zip from a collaborator, and the first hour with it is always the same hour: what is actually in here, what is duplicated, what will break the moment it hits a different operating system, and can I prove later that this is what I received.
These are the checks that turn "here's a drive" into something you can start an analysis against. Nothing clever, just the things that cost an afternoon when they get skipped.
pip install -r requirements.txt
python intake_report.py /path/to/drop --manifest manifest.csv
pytest tests/ -qSample output:
4 files, 20 B
file types
.jpg 3 12 B
.csv 1 8 B
duplicate content: 1 group(s)
63c1dd951ffe... 2 copies
site_a/img_1.jpg
site_b/copy of img_1.jpg
filename issues: 3
unpadded_number 3
site_a/img_1.jpg
manifest hashes every file and optionally writes a CSV. Run it the day a drop arrives, before anyone reorganizes anything. It is the difference between "I think this is the original data" and knowing.
find_duplicates groups by content hash, not by name, so it catches the same photo saved as IMG_0042.JPG, IMG_0042 (1).JPG, and site3_final.jpg.
check_filenames flags names that cause trouble downstream:
| Flag | Why it matters |
|---|---|
illegal_chars |
breaks on Windows, needs quoting in shells |
reserved_name |
CON, PRN, AUX, COM1 and friends |
leading_space |
silently mangled by many tools |
unpadded_number |
sorts as 1, 10, 2 instead of 01, 02, 10 |
very_long |
path length limits |
The unpadded-number check earns its place. Files named site_1 through site_12 sort into the wrong order in almost every tool that touches them, and the resulting misalignment is quiet, plausible, and discovered three weeks later.
summarize_types reports counts and total bytes per extension, largest first, which is usually how you discover that 90% of a drop is one video nobody mentioned.
from fieldutils import manifest, find_duplicates, check_filenames, summarize_types
records = manifest("/path/to/drop", "manifest.csv")
dupes = find_duplicates(records) # {sha256: [paths]}
issues = check_filenames(records) # {issue_kind: [paths]}
types = summarize_types(records) # [(suffix, count, bytes)]The provenance discipline these tools support is formalized in research-defensibility-kit, where a recorded content hash for every input is a gate a claim has to pass.
MIT.
Aaron Price · Anima Audire, LLC · Profile · aaron.price.unl@gmail.com