NUNA/guidebook/05_trading_data_file_management.md

Trading Data File Management (Local Workflow)

This guide describes a clean, automated workflow for managing trading logs, raw exports, and final reports on your local machine. It also documents the included helper script: trading_data_manager.py.

Goals (what “good” looks like)

• Safe-by-default automation: preview changes first (dry-run); no surprise deletes.
• Analysis-ready reports: raw .csv exports are converted to .xlsx for Excel/Power BI.
• Low clutter: old runtime logs and raw exports are automatically cleaned up.
• Traceability: every run produces a log of what it planned/executed.

Recommended lifecycle and “what to keep”

1) Ingest (raw exports)

• File types: .csv from broker/platform exports
• Folder: raw_csv/
• Rule:
  • Convert to .xlsx into reports/
  • Then move the source .csv to trash/ (quarantine) instead of immediate deletion

2) Report (analysis-ready)

• File types: .xlsx
• Folder: reports/
• Rule:
  • Keep the newest report per day in reports/
  • Move older same-day reports into archive/ (so you can roll back if needed)

3) Archive (long-term history)

• File types: .xlsx
• Folder: archive/YYYY/MM/
• Rule:
  • Reports older than N days (default: 90) are moved into archive/

4) Logs (temporary)

• File types: .txt
• Folder: logs/
• Rule:
  • Logs older than N days (default: 14) are moved into trash/

5) Trash (safety quarantine)

• Folder: trash/
• Rule:
  • The tool moves files here instead of deleting them
  • Permanent deletion is a separate step (purge-trash) with a confirmation string

Folder structure

Under a single root directory (default: ./trading_data):

trading_data/
├─ logs/              # .txt runtime/debug logs
├─ raw_csv/           # CSV exports (broker/platform)
├─ reports/           # Final XLSX reports (latest per day)
├─ archive/           # Archived reports (organized by YYYY/MM)
├─ trash/             # Quarantine (moved here instead of delete)
└─ automation_logs/   # Logs written by the automation runs

Naming conventions (strongly recommended)

Good filenames make automation and auditing much easier:

• Raw exports: trades_<broker>_YYYY-MM-DD.csv
• Reports: report_YYYY-MM-DD.xlsx
• If you create multiple per day: report_YYYY-MM-DD_v2.xlsx (or include a timestamp)

If you don’t use a naming convention, the tool still works; it groups “per day” using the file’s local modified time.

Automation script: trading_data_manager.py

Install dependencies

python3 -m venv .venv
. .venv/bin/activate
pip install -r requirements.txt

Initialize folders + write an example config

python3 trading_data_manager.py init --write-example-config trading_data_config.example.json

Create your local config by copying:

cp trading_data_config.example.json trading_data_config.json

(trading_data_config.json is ignored by git.)

Daily run (dry-run first)

python3 trading_data_manager.py run

Apply the planned moves/conversions:

python3 trading_data_manager.py run --apply

Purge old items from trash (permanent delete)

Dry-run (prints required confirmation):

python3 trading_data_manager.py purge-trash

Apply (must match the printed confirmation string exactly):

python3 trading_data_manager.py purge-trash --confirm "PURGE <n> FILES" --apply

Scheduling (examples)

Linux (cron)

Run every day at 6:05am:

5 6 * * * cd /path/to/repo && /path/to/repo/.venv/bin/python trading_data_manager.py run --apply

Windows (Task Scheduler)

• Program: python
• Arguments: trading_data_manager.py run --apply
• Start in: repo folder path

Suggested defaults (tune as you learn)

• Logs: move to trash after 14 days
• Archive: move reports to archive after 90 days
• Trash purge: permanently delete trash after 30 days

These are configurable in trading_data_config.json.