LTSM-Forex-Bot/AGENTS.md

LSTM Trading Bot - Agent Development Guide

Project Overview

This document provides guidelines for automated agents (Cursor, Copilot, Claude, etc.) and human contributors working on the LSTM-based multi-timeframe trading bot project.

1. Required Checks

Code Quality

• Linting: Run python -m py_compile on all Python files
• Type Checking: Ensure proper type hints throughout
• Import Validation: Verify all imports work correctly
• Configuration Validation: Test YAML configuration loading

Testing

• Unit Tests: Run tests for individual modules
• Integration Tests: Test data pipeline and model training
• Backtesting Validation: Verify strategy performance
• Import Tests: Ensure all modules can be imported

Dependency Management

• Requirements Check: Verify requirements.txt includes all dependencies
• Version Compatibility: Ensure PyTorch, pandas, etc. versions are compatible
• Optional Dependencies: Test functionality without optional packages

2. Project Structure

├── config/                    # Configuration files
│   └── config.yaml           # Main configuration
├── data/                     # Data storage (gitignored)
│   ├── raw/                 # Raw OHLCV data
│   └── cache/               # Processed features
├── notebooks/                # Jupyter notebooks
│   ├── 01_data_eda.ipynb    # Data exploration
│   ├── 02_train_optuna.ipynb # Model training
│   ├── 03_backtest.ipynb     # Backtesting
│   └── 04_live_trading.ipynb # Live trading
├── src/                      # Source code
│   ├── data/                # Data loading
│   │   └── loaders.py
│   ├── features/            # Feature engineering
│   │   ├── build_dataset.py
│   │   └── indicators.py
│   ├── models/              # LSTM architectures
│   │   └── lstm_fusion.py
│   ├── training/            # Training framework
│   │   ├── train.py
│   │   └── metrics.py
│   ├── backtest/            # Backtesting engine
│   │   ├── engine.py
│   │   └── strategy.py
│   ├── live/                # Live trading
│   │   ├── broker_base.py
│   │   ├── broker_alpaca.py
│   │   ├── streamer.py
│   │   └── executor.py
│   └── utils/               # Utilities
│       ├── config.py
│       ├── logging.py
│       ├── seeds.py
│       └── times.py
├── tests/                   # Test files
├── docs/                    # Documentation
│   └── agents/              # Agent-specific docs
└── requirements.txt          # Dependencies

3. Coding Conventions

Python Standards

• Language: Python 3.10+ with type hints
• Style: Follow PEP 8 with 4-space indentation
• Imports: Standard library first, then third-party, then local
• Documentation: Use Google-style docstrings
• Error Handling: Comprehensive try-catch with logging

Project-Specific Conventions

• Configuration: All parameters in config/config.yaml
• Logging: Use structured logging with src.utils.logging
• Error Handling: Graceful degradation with informative messages
• Testing: Unit tests for all public functions

File Naming

• Modules: snake_case.py
• Classes: PascalCase
• Functions: snake_case
• Constants: UPPER_SNAKE_CASE

4. Development Workflow

Agent Development Process

1. Read Memory: Check docs/agents/ledger.json for existing work
2. Understand Intent: Clarify requirements before implementation
3. Plan Changes: Create minimal, testable implementation plan
4. Implement: Write clean, documented code
5. Test: Validate functionality and edge cases
6. Document: Update relevant documentation
7. Log: Add entry to docs/agents/ledger.json

Human Contribution Process

1. Issue Creation: Create GitHub issue for proposed changes
2. Branch Creation: Create feature branch from main
3. Implementation: Follow coding conventions
4. Testing: Run all relevant tests
5. PR Creation: Submit pull request with description
6. Review: Address review comments
7. Merge: Merge after approval

5. Module-Specific Guidelines

Data Pipeline (src/data/, src/features/)

• Data Sources: Support CSV, Alpaca, Binance, OANDA
• Feature Engineering: Technical indicators, lagged features, calendar features
• Data Quality: Handle missing data, outliers, and anomalies
• Performance: Efficient processing for large datasets

Model Architecture (src/models/)

• LSTM Variants: Single LSTM, multi-LSTM fusion
• Attention Mechanisms: Multi-head attention implementation
• Regularization: Dropout, layer normalization
• Output Modes: Regression and classification

Training Framework (src/training/)

• Optimization: Optuna hyperparameter tuning
• Validation: Walk-forward cross-validation
• Metrics: Trading-specific performance measures
• Checkpointing: Model saving and loading

Backtesting (src/backtest/)

• Framework: Backtrader integration
• Risk Management: Position sizing, stop losses
• Performance: Comprehensive metrics and reporting
• Realism: Slippage, commission, latency modeling

Live Trading (src/live/)

• Broker Abstraction: Support multiple exchanges
• Streaming: Real-time market data
• Execution: Order management and risk controls
• Monitoring: Performance tracking and alerting

6. Configuration Management

Main Configuration (config/config.yaml)

• Data Settings: Symbols, timeframes, sources
• Model Parameters: Architecture, hyperparameters
• Training Settings: Batch size, epochs, optimization
• Risk Parameters: Position sizing, circuit breakers
• Live Trading: Broker settings, execution parameters

Environment Variables

# Broker API credentials
API_KEY_ALPACA=your_key
API_SECRET_ALPACA=your_secret
API_KEY_BINANCE=your_key
API_KEY_OANDA=your_key

# Optional: Alert webhooks
SLACK_WEBHOOK=your_webhook_url
DISCORD_WEBHOOK=your_webhook_url

7. Testing Strategy

Unit Tests

• Data Loaders: Test data loading from all sources
• Feature Engineering: Validate indicator calculations
• Model Components: Test LSTM layers and fusion strategies
• Utilities: Test configuration and logging

Integration Tests

• End-to-End Pipeline: Data loading → features → training → prediction
• Backtesting: Strategy execution with realistic conditions
• Live Trading: Paper trading execution and risk management

Performance Tests

• Training Speed: Model training time on different hardware
• Memory Usage: Memory consumption during processing
• Scalability: Performance with larger datasets

8. Deployment Guidelines

Google Colab

1. Upload project files to Colab environment
2. Install dependencies: !pip install -r requirements.txt
3. Run notebooks in sequence for complete workflow
4. Use GPU runtime for faster training

Linux Server

1. System Setup: Ubuntu/Debian with Python 3.10+
2. Dependencies: Install via pip install -r requirements.txt
3. Configuration: Set environment variables for API keys
4. Service Setup: Use systemd for 24/7 operation
5. Monitoring: Configure logging and alerting

Docker Deployment

FROM python:3.10-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .
CMD ["python", "main.py", "live", "run", "--mode", "paper"]

9. Risk Management

Position Sizing

• Fixed Percentage: Fixed % of portfolio per trade
• Kelly Criterion: Optimal position sizing based on win rate
• Volatility Adjusted: Position size based on market volatility

Circuit Breakers

• Drawdown Limits: Stop trading if portfolio drops too much
• Daily Loss Limits: Stop trading if daily loss exceeds threshold
• Position Limits: Maximum number of concurrent positions
• Time-based: Stop trading during high-risk periods

Monitoring

• Performance Tracking: Real-time P&L and risk metrics
• Alert System: Notifications for risk events
• Health Checks: System status and connectivity monitoring

10. Performance Optimization

Training Optimization

• GPU Acceleration: Use CUDA for faster training
• Batch Processing: Optimize batch sizes for hardware
• Mixed Precision: Use float16 for memory efficiency
• Model Parallelism: Distribute model across multiple GPUs

Inference Optimization

• Model Quantization: Reduce model size for faster inference
• Batch Inference: Process multiple predictions together
• Caching: Cache frequent calculations and features

Data Optimization

• Efficient Storage: Use Parquet for compressed data storage
• Lazy Loading: Load data only when needed
• Memory Mapping: Use memory-mapped files for large datasets

11. Troubleshooting

Common Issues

• Import Errors: Check Python path and dependencies
• Configuration Errors: Validate YAML syntax and required fields
• Data Issues: Check data format and missing values
• Model Errors: Verify tensor shapes and data types

Debugging Tools

• Logging: Comprehensive logging at all levels
• Profiling: Performance profiling for bottlenecks
• Visualization: Plot data and model outputs for inspection
• Interactive Debugging: Use IPython for step-through debugging

12. Future Enhancements

Model Improvements

• Transformer Architectures: Add attention-based models
• Ensemble Methods: Combine multiple model predictions
• Reinforcement Learning: Train using RL for better adaptation

Feature Enhancements

• Alternative Data: News, sentiment, macroeconomic indicators
• Advanced Indicators: More sophisticated technical analysis
• Regime Detection: Machine learning-based market regime classification

System Improvements

• Multi-Asset Trading: Handle multiple asset classes
• Portfolio Optimization: Modern portfolio theory integration
• Risk Parity: Equal risk contribution across positions

13. Contributing Guidelines

Code Contributions

1. Follow existing code style and conventions
2. Add tests for new functionality
3. Update documentation for API changes
4. Use meaningful commit messages

Issue Management

• Use clear, descriptive issue titles
• Provide detailed descriptions and reproduction steps
• Label issues appropriately (bug, enhancement, documentation)
• Reference related issues and PRs

Review Process

• All PRs require at least one review
• Address review comments promptly
• Ensure CI checks pass before merge
• Update changelog for significant changes

14. Maintenance

Regular Tasks

• Model Retraining: Update models with new data
• Performance Review: Analyze live vs backtested performance
• Risk Review: Adjust risk parameters based on performance
• Dependency Updates: Keep packages current and secure

Monitoring

• System Health: Monitor server and application health
• Performance Metrics: Track key performance indicators
• Error Rates: Monitor and address error patterns
• Resource Usage: Track CPU, memory, and disk usage

15. Support and Resources

Documentation

• README.md: Project overview and setup instructions
• API Documentation: Generated from docstrings
• Colab Notebooks: Interactive tutorials and examples
• Configuration Guide: Detailed parameter explanations

Community

• GitHub Issues: Bug reports and feature requests
• Discussions: General questions and discussions
• Wiki: Additional documentation and guides

Development Tools

• IDE Support: Cursor, VS Code with Python extensions
• Linting: Pylint, flake8 for code quality
• Formatting: Black for consistent code formatting
• Testing: Pytest for unit and integration tests

---

This guide ensures consistent development practices and high-quality code across all contributors and automated agents working on the LSTM trading bot project.