- Updated `docs/NOTEBOOK_LM_CONTEXT.txt` with user-provided NotebookLM URL and context directives.
- Enhanced `scripts/market_research.py` and `scripts/upgrade_repo.py`:
- Added proper Jules integration using the REST API structure `{"model": ..., "prompt": ...}` to match MQL5 implementation.
- Added warning suppression for deprecated `google.generativeai` warnings.
- Added robustness checks for API keys.
- Enhanced `scripts/schedule_research.py` with file logging (`logs/scheduler.log`) and next-run prediction.
- Implements `CPositionManager` class in `ManagePositions.mqh` for robust Trailing Stop and Break Even logic.
- Updates `SMC_TrendBreakout_MTF_EA.mq5` to use `CPositionManager` and exposes Scalping inputs.
- Adds preset files for M5, M15, M30 scalping in `mt5/MQL5/Presets/`.
- Documents strategy parameters in `docs/SCALPING_STRATEGY.md`.
- Adds `scripts/research_scalping.py` utility.
- Added `scripts/setup_forge_runner.sh` for automated Gitea runner registration.
- Optimized `SMC_TrendBreakout_MTF.mq5` by replacing manual loops with native `ArrayMaximum`/`ArrayMinimum`.
- Optimized `SMC_TrendBreakout_MTF_EA.mq5`:
- Replaced `CopyTime` with `iTime` for faster bar checks.
- Replaced non-existent `iDonchian` handle with standard MQL5 price range logic using `CopyHigh`/`CopyLow`.
- Removed redundant code and handles in `OnInit`/`OnDeinit`.
Co-authored-by: Mouy-leng <199350297+Mouy-leng@users.noreply.github.com>
- Moved `yfinance` and `pandas` imports inside `get_market_data`
- Moved `google.generativeai` import inside `analyze_with_gemini`
- Reduces startup time when these libraries are not immediately needed (e.g. missing API keys or dry runs)
- Keeps fallback simulation logic intact
Co-authored-by: Mouy-leng <199350297+Mouy-leng@users.noreply.github.com>
- Updated `scripts/market_research.py` to provide helpful hints on Jules NameResolutionError.
- Updated `docs/USER_NOTES.md` to document the schedule setup and current API status.
- Created `docs/upgrade_suggestions.md` to establish the upgrade context (NotebookLM) and track pending analysis.
- Verified scheduler execution (PID running).
- Note: Gemini API is currently rate-limited (429) and Jules API endpoint requires verification.
Co-authored-by: Mouy-leng <199350297+Mouy-leng@users.noreply.github.com>
Replaced per-branch git log calls with a single git for-each-ref command using the ahead-behind atom. This optimizes the script from O(N) to O(1) git subprocess calls.
Co-authored-by: Mouy-leng <199350297+Mouy-leng@users.noreply.github.com>
Replaced blocking `subprocess.run` calls with `asyncio.create_subprocess_exec` to ensure the Telegram bot event loop remains responsive during long-running deployment tasks.
Key changes:
- Switched to `asyncio.create_subprocess_exec` for command execution.
- implemented `asyncio.wait_for` to handle the 300s timeout.
- Added explicit process cleanup (kill/wait) on timeout.
- Manually decoded stdout/stderr since async subprocesses return bytes.
Verified with a reproduction script showing the event loop gap dropped from ~2.0s (blocking) to ~0.1s (non-blocking).
Co-authored-by: Mouy-leng <199350297+Mouy-leng@users.noreply.github.com>
Parallelizes the Gemini and Jules AI analysis calls in `scripts/market_research.py` using `concurrent.futures`. This optimization allows both API requests to execute concurrently, reducing the total execution time of the script to the duration of the slower request rather than the sum of both.
Changes:
- Imported `concurrent.futures` in `scripts/market_research.py`.
- Replaced sequential calls to `analyze_with_gemini` and `analyze_with_jules` with `ThreadPoolExecutor` submission.
- Created `scripts/manage_cloudflare.py` to get/set security levels via API.
- Created `scripts/setup_cloudflare_tunnel.sh` to install `cloudflared`.
- Created `config/vault.json` with user-provided Zone/Account IDs for `lengkundee01.org`.
- Created `docs/CLOUDFLARE_GUIDE.md` with usage instructions.
- Updated `requirements.txt` to include `requests` as a core dependency.
- Added `yfinance` to fetch real market data.
- Enhanced `scripts/market_research.py` to use `yfinance` and support Jules AI.
- Enhanced `scripts/upgrade_repo.py` to support Jules AI for code suggestions.
- Updated `scripts/schedule_research.py` to load environment variables.
- Created `scripts/setup_research.sh` to automate setup.
- Updated `requirements.txt` and `README.md`.
Replaced `os.path.exists()` + `os.path.getmtime()` with a single `os.stat()` call in `scripts/web_dashboard.py` to reduce syscalls by 50% for cache checks. Also pre-calculated static file paths at module level to avoid redundant `abspath` and `join` calls on every request.
Impact:
- Reduces filesystem operations per request.
- Improves code cleanliness by centralizing path constants.
- Verified with existing tests and manual curl check.
- Added `scripts/market_research.py` to analyze market data using Gemini.
- Added `scripts/upgrade_repo.py` to suggest code upgrades based on research.
- Added `scripts/schedule_research.py` to orchestrate periodic research.
- Updated `config/startup_config.json` to include the scheduler.
- Added `schedule` and `google-generativeai` to `requirements.txt`.
- Added `--monitor` flag to `scripts/startup.sh`.
- Fixed circular nameref warning in `scripts/startup.sh`.
- Updated `startup.sh` to pass the monitor flag to `startup_orchestrator.py`.
- Verified system startup and web dashboard accessibility.
- Created `mt5/MQL5/Include/ZoloBridge.mqh` for shared, secure (HTTPS) bridge logic with proper JSON sanitization.
- Created `ExpertMACD_Enhanced.mq5`, `ExpertMAMA_Enhanced.mq5`, and `ExpertMAPSAR_Enhanced.mq5` using standard library and ZOLO bridge.
- Refactored `EXNESS_GenX_Trader.mq5` and `SMC_TrendBreakout_MTF_EA.mq5` to use `ZoloBridge.mqh` and updated default URL to `https://genx-fx.com/api/signal`.
- Updated `scripts/deploy_docker_hub.sh` and `scripts/update_vps.sh` to support `DOCKER_USERNAME` and `DOCKER_PASSWORD` environment variables.
- Added `scripts/fix_profiles.py` to copy defaults to empty profile directories.
- Added `SECRETS_TEMPLATE.md` for credential management guidance.
- Updated EA link properties to Forge URL.
Optimize `scripts/merge_best_prs.py` by using `concurrent.futures.ThreadPoolExecutor` to close duplicate PRs in parallel. This significantly reduces execution time by masking network latency for independent API calls.
Benchmarks show a reduction from ~1.31s to ~0.25s for closing 13 PRs with simulated 0.1s latency.
💡 What: Separated the /health endpoint from the main dashboard rendering logic. It now returns a lightweight JSON response.
🎯 Why: The previous implementation rendered the full Markdown dashboard for every health check, consuming unnecessary CPU and I/O resources during frequent polling.
📊 Impact: Reduces health check processing time from file reading + markdown parsing (~milliseconds) to a simple JSON return (~microseconds).
🔬 Measurement: Verified with new test script scripts/test_web_dashboard.py and updated render.yaml/app.yaml to use the new endpoint.
- Create `app.yaml` for Google App Engine Flexible environment.
- Update `scripts/deploy_cloud.py` to support `gcp` argument with specific instructions for project `infra-outrider-snqdt`.
- Update `mt5/MQL5/Experts/SMC_TrendBreakout_MTF_EA.mq5` default `GeminiModel` to `gemini-1.5-pro` and update comments.
- Consolidate file validation logic into a single loop
- Check file size before reading content to prevent loading large files into memory
- Add error handling for file reading
- Reduce I/O operations and loop iterations
This improves the efficiency of the CI validation script, especially for repositories with many files or when large files are accidentally introduced.
- Implemented a Flask-based web dashboard in `scripts/web_dashboard.py` to serve project status and documentation.
- Integrated the web dashboard into `config/startup_config.json` as a managed component.
- Updated `requirements.txt` with `flask` and `markdown` dependencies.
- Added `/health` endpoint for cloud platform health checks.
- Verified system integrity with existing validation and testing scripts.
Refactored `scripts/test_automation.py` to run independent integration tests in parallel using `concurrent.futures.ProcessPoolExecutor`.
* 💡 **What:** Used `ProcessPoolExecutor` to run tests concurrently.
* 🎯 **Why:** To improve test suite performance.
* 📊 **Measured Improvement:** ~30% faster (3.3s -> 2.25s).
* **Details:** Used processes instead of threads to ensure `contextlib.redirect_stdout` works correctly for each test without interference.
