Updated market_research.py and upgrade_repo.py to use 'gemini-flash-latest' as the default model, fixing 429/404 errors with the previous 'gemini-2.0-flash' default. Ran the scripts to generate fresh 'docs/market_research_report.md' and 'docs/upgrade_suggestions.md' using the provided API keys.
Parallelizes the Gemini and Jules AI analysis calls in `scripts/market_research.py` using `concurrent.futures`. This optimization allows both API requests to execute concurrently, reducing the total execution time of the script to the duration of the slower request rather than the sum of both.
Changes:
- Imported `concurrent.futures` in `scripts/market_research.py`.
- Replaced sequential calls to `analyze_with_gemini` and `analyze_with_jules` with `ThreadPoolExecutor` submission.
- Created `scripts/manage_cloudflare.py` to get/set security levels via API.
- Created `scripts/setup_cloudflare_tunnel.sh` to install `cloudflared`.
- Created `config/vault.json` with user-provided Zone/Account IDs for `lengkundee01.org`.
- Created `docs/CLOUDFLARE_GUIDE.md` with usage instructions.
- Updated `requirements.txt` to include `requests` as a core dependency.
- Added `yfinance` to fetch real market data.
- Enhanced `scripts/market_research.py` to use `yfinance` and support Jules AI.
- Enhanced `scripts/upgrade_repo.py` to support Jules AI for code suggestions.
- Updated `scripts/schedule_research.py` to load environment variables.
- Created `scripts/setup_research.sh` to automate setup.
- Updated `requirements.txt` and `README.md`.
Replaced `os.path.exists()` + `os.path.getmtime()` with a single `os.stat()` call in `scripts/web_dashboard.py` to reduce syscalls by 50% for cache checks. Also pre-calculated static file paths at module level to avoid redundant `abspath` and `join` calls on every request.
Impact:
- Reduces filesystem operations per request.
- Improves code cleanliness by centralizing path constants.
- Verified with existing tests and manual curl check.
- Added `scripts/market_research.py` to analyze market data using Gemini.
- Added `scripts/upgrade_repo.py` to suggest code upgrades based on research.
- Added `scripts/schedule_research.py` to orchestrate periodic research.
- Updated `config/startup_config.json` to include the scheduler.
- Added `schedule` and `google-generativeai` to `requirements.txt`.
- Added `--monitor` flag to `scripts/startup.sh`.
- Fixed circular nameref warning in `scripts/startup.sh`.
- Updated `startup.sh` to pass the monitor flag to `startup_orchestrator.py`.
- Verified system startup and web dashboard accessibility.
- Created `mt5/MQL5/Include/ZoloBridge.mqh` for shared, secure (HTTPS) bridge logic with proper JSON sanitization.
- Created `ExpertMACD_Enhanced.mq5`, `ExpertMAMA_Enhanced.mq5`, and `ExpertMAPSAR_Enhanced.mq5` using standard library and ZOLO bridge.
- Refactored `EXNESS_GenX_Trader.mq5` and `SMC_TrendBreakout_MTF_EA.mq5` to use `ZoloBridge.mqh` and updated default URL to `https://genx-fx.com/api/signal`.
- Updated `scripts/deploy_docker_hub.sh` and `scripts/update_vps.sh` to support `DOCKER_USERNAME` and `DOCKER_PASSWORD` environment variables.
- Added `scripts/fix_profiles.py` to copy defaults to empty profile directories.
- Added `SECRETS_TEMPLATE.md` for credential management guidance.
- Updated EA link properties to Forge URL.
Optimize `scripts/merge_best_prs.py` by using `concurrent.futures.ThreadPoolExecutor` to close duplicate PRs in parallel. This significantly reduces execution time by masking network latency for independent API calls.
Benchmarks show a reduction from ~1.31s to ~0.25s for closing 13 PRs with simulated 0.1s latency.
💡 What: Separated the /health endpoint from the main dashboard rendering logic. It now returns a lightweight JSON response.
🎯 Why: The previous implementation rendered the full Markdown dashboard for every health check, consuming unnecessary CPU and I/O resources during frequent polling.
📊 Impact: Reduces health check processing time from file reading + markdown parsing (~milliseconds) to a simple JSON return (~microseconds).
🔬 Measurement: Verified with new test script scripts/test_web_dashboard.py and updated render.yaml/app.yaml to use the new endpoint.
- Create `app.yaml` for Google App Engine Flexible environment.
- Update `scripts/deploy_cloud.py` to support `gcp` argument with specific instructions for project `infra-outrider-snqdt`.
- Update `mt5/MQL5/Experts/SMC_TrendBreakout_MTF_EA.mq5` default `GeminiModel` to `gemini-1.5-pro` and update comments.
- Consolidate file validation logic into a single loop
- Check file size before reading content to prevent loading large files into memory
- Add error handling for file reading
- Reduce I/O operations and loop iterations
This improves the efficiency of the CI validation script, especially for repositories with many files or when large files are accidentally introduced.
- Implemented a Flask-based web dashboard in `scripts/web_dashboard.py` to serve project status and documentation.
- Integrated the web dashboard into `config/startup_config.json` as a managed component.
- Updated `requirements.txt` with `flask` and `markdown` dependencies.
- Added `/health` endpoint for cloud platform health checks.
- Verified system integrity with existing validation and testing scripts.
Refactored `scripts/test_automation.py` to run independent integration tests in parallel using `concurrent.futures.ProcessPoolExecutor`.
* 💡 **What:** Used `ProcessPoolExecutor` to run tests concurrently.
* 🎯 **Why:** To improve test suite performance.
* 📊 **Measured Improvement:** ~30% faster (3.3s -> 2.25s).
* **Details:** Used processes instead of threads to ensure `contextlib.redirect_stdout` works correctly for each test without interference.
- Fix cron entry to use absolute paths instead of variables
- Add parents=True to mkdir for better error handling
- Change PowerShell scheduled task to use Limited privileges for security
- All tests still passing
Co-authored-by: Mouy-leng <199350297+Mouy-leng@users.noreply.github.com>
- AUTOMATION_FEATURES.md with comprehensive feature overview
- Integration test suite for verifying all scripts
- Logs directory README for documentation
- All tests passing successfully
Co-authored-by: Mouy-leng <199350297+Mouy-leng@users.noreply.github.com>
- Python orchestrator with JSON config, logging, and monitoring
- Windows batch script for simple automation
- PowerShell script with scheduled task creation
- Linux/WSL shell script with systemd/cron support
- Detailed documentation and quick start guides
- Example custom script template
- Configuration file with MT5 paths
- Updated .gitignore for logs and temp files
Co-authored-by: Mouy-leng <199350297+Mouy-leng@users.noreply.github.com>
