Refactored `scripts/ci_validate_repo.py` to use `os.walk()` instead of `Path.rglob("*")` for repository traversal. Implemented in-place directory pruning in `scan_for_secrets` to avoid visiting excluded directories, significantly improving performance for large repositories.
- Added `import os`
- Refactored `validate_and_collect_files` to use `os.walk()`
- Refactored `scan_for_secrets` with `os.walk()` and directory pruning
- Verified performance gains and correctness with existing tests
Improved memory efficiency in `scripts/ci_validate_repo.py` by replacing
`read_bytes()` with chunked binary reading (64KB) for NUL byte detection.
This prevents loading entire source files into memory during CI runs.
- Consolidate file validation logic into a single loop
- Check file size before reading content to prevent loading large files into memory
- Add error handling for file reading
- Reduce I/O operations and loop iterations
This improves the efficiency of the CI validation script, especially for repositories with many files or when large files are accidentally introduced.
