MQL5-Google-Onedrive/docs/GITLAB_CI_CD_SETUP.md

GitLab CI/CD Setup Guide

This guide explains how to set up GitLab CI/CD for the MQL5 Trading System repository using GitLab Environment Toolkit (GET).

Table of Contents

• Prerequisites
• GitLab Repository Setup
• CI/CD Pipeline Overview
• Environment Variables & Secrets
• GitLab Runner Setup
• GitLab Environment Toolkit (GET)
• Deployment Configuration
• Testing the Pipeline
• Troubleshooting

Prerequisites

Before setting up GitLab CI/CD, ensure you have:

1. GitLab Account - Create an account at GitLab.com
2. GitLab Repository - Either create a new repository or mirror from GitHub
3. GitLab Runner - Access to GitLab shared runners or your own runner
4. Docker (Optional) - For containerized builds and deployments
5. API Tokens - For deployment to cloud platforms

GitLab Repository Setup

Option 1: Mirror from GitHub

If you're using GitHub as the primary repository:

1. Create a new GitLab project:

   • Go to GitLab → New Project → Import Project
   • Select Repository by URL
   • Enter your GitHub repository URL: https://github.com/A6-9V/MQL5-Google-Onedrive.git
   • Choose visibility level (Private recommended)

2. Set up repository mirroring (Optional):

   • Go to Settings → Repository → Mirroring repositories
   • Add GitHub URL with credentials
   • Enable "Mirror only protected branches"
   • Set up push mirroring to keep GitHub updated

Option 2: Push Existing Repository

# Add GitLab as a remote
git remote add gitlab git@gitlab.com:your-username/mql5-google-onedrive.git

# Push to GitLab
git push gitlab main

# Optional: Push all branches
git push gitlab --all

CI/CD Pipeline Overview

The .gitlab-ci.yml file defines a multi-stage pipeline:

Pipeline Stages

1. Validate - Repository structure, scripts, and security checks
2. Build - Documentation and artifacts
3. Test - Automation tests
4. Package - MT5 files and Docker images
5. Deploy - Deploy to staging, production, and cloud platforms

Key Jobs

Job	Description	Trigger
validate:repository	Validates MQL5 files structure	Every push, MR
validate:scripts	Validates shell script syntax	Every push, MR
validate:secrets	Scans for accidentally committed secrets	Every push, MR
test:automation	Runs automation tests	Every push, MR
package:mt5	Creates MT5 source package	Main branch, tags, MR
package:docker	Builds Docker image	Main branch, tags
deploy:staging	Deploys to staging	Manual (main branch)
deploy:production	Deploys to production	Manual (tags only)
deploy:cloud	Deploys to cloud platforms	Manual

Environment Variables & Secrets

Required Variables

Go to Settings → CI/CD → Variables and add:

Basic Configuration

Variable	Value	Protected	Masked	Description
CI_REGISTRY	registry.gitlab.com	No	No	GitLab Container Registry
CI_REGISTRY_IMAGE	Auto-generated	No	No	Your project's registry path

Cloud Deployment (Optional)

Variable	Value	Protected	Masked	Description
RENDER_API_KEY	Your Render API key	Yes	Yes	For Render.com deployment
RAILWAY_TOKEN	Your Railway token	Yes	Yes	For Railway.app deployment
FLY_API_TOKEN	Your Fly.io token	Yes	Yes	For Fly.io deployment
DOCKER_USERNAME	Docker Hub username	No	No	For Docker Hub push
DOCKER_PASSWORD	Docker Hub token	Yes	Yes	For Docker Hub push

API & Bot Configuration

Variable	Value	Protected	Masked	Description
TELEGRAM_BOT_TOKEN	Bot token from BotFather	Yes	Yes	Telegram bot authentication
TELEGRAM_ALLOWED_USER_IDS	Comma-separated IDs	Yes	No	Authorized users
GEMINI_API_KEY	Google Gemini API key	Yes	Yes	AI trade filtering
JULES_API_KEY	Jules AI API key	Yes	Yes	Alternative AI provider
GITHUB_PAT	GitHub Personal Access Token	Yes	Yes	For GitHub integration

Cloudflare (Optional)

Variable	Value	Protected	Masked	Description
CLOUDFLARE_ZONE_ID	Your zone ID	Yes	No	Cloudflare zone
CLOUDFLARE_ACCOUNT_ID	Your account ID	Yes	No	Cloudflare account
CLOUDFLARE_API_TOKEN	API token	Yes	Yes	Cloudflare API access
DOMAIN_NAME	Your domain	No	No	Domain name

Setting Variables via GitLab CLI

If you have glab CLI installed:

# Install glab CLI
# macOS: brew install glab
# Linux: See https://gitlab.com/gitlab-org/cli

# Authenticate
glab auth login

# Set variables
glab variable set TELEGRAM_BOT_TOKEN "your_token_here" --masked --protected
glab variable set RENDER_API_KEY "your_key_here" --masked --protected
glab variable set FLY_API_TOKEN "your_token_here" --masked --protected

Using the Automated Script

Create a config/gitlab_vault.json file (copy from example):

cp config/vault.json.example config/gitlab_vault.json
# Edit gitlab_vault.json with your credentials

Then use the script to sync secrets:

# Create a script to set GitLab variables
bash scripts/set_gitlab_secrets.sh gitlab_vault

GitLab Runner Setup

Using Shared Runners

GitLab.com provides free shared runners. Enable them:

1. Go to Settings → CI/CD → Runners
2. Enable Shared runners
3. View available runners and their tags

Setting Up Your Own Runner

For better performance or specific requirements:

1. Install GitLab Runner

Ubuntu/Debian:

curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh" | sudo bash
sudo apt-get install gitlab-runner

macOS:

brew install gitlab-runner

Docker:

docker run -d --name gitlab-runner --restart always \
  -v /srv/gitlab-runner/config:/etc/gitlab-runner \
  -v /var/run/docker.sock:/var/run/docker.sock \
  gitlab/gitlab-runner:latest

2. Register the Runner

# Get registration token from Settings → CI/CD → Runners
sudo gitlab-runner register

# Follow prompts:
# - GitLab instance URL: https://gitlab.com
# - Registration token: [from GitLab]
# - Description: My MQL5 Runner
# - Tags: mql5,docker,python
# - Executor: docker
# - Default Docker image: python:3.12-slim

3. Start the Runner

sudo gitlab-runner start

4. Verify Runner

Check Settings → CI/CD → Runners to see your runner listed.

GitLab Environment Toolkit (GET)

GitLab Environment Toolkit (GET) is a collection of tools for deploying GitLab in various environments.

Installing GET

# Clone the GitLab Environment Toolkit
git clone https://gitlab.com/gitlab-org/gitlab-environment-toolkit.git
cd gitlab-environment-toolkit

# Install dependencies
terraform --version  # Should be >= 1.0
ansible --version    # Should be >= 2.9

GET for CI/CD Runners

If you want to deploy a scalable runner infrastructure:

1. Choose your platform (AWS, GCP, Azure)
2. Configure Terraform for your environment
3. Deploy runners using GET automation

Example for AWS:

cd gitlab-environment-toolkit/terraform/environments/aws
cp terraform.tfvars.example terraform.tfvars
# Edit terraform.tfvars with your settings

terraform init
terraform plan
terraform apply

GET Configuration Files

Create .get-config.yml in your repository root:

# GitLab Environment Toolkit Configuration
gitlab:
  version: "16.0"
  
runners:
  count: 2
  type: docker
  tags:
    - mql5
    - python
    - docker
  
deployment:
  staging:
    enabled: true
    url: "https://staging.example.com"
  production:
    enabled: true
    url: "https://production.example.com"

Deployment Configuration

Staging Deployment

Staging deployments are manual by default:

1. Go to CI/CD → Pipelines
2. Find your pipeline
3. Click the play button on deploy:staging

Production Deployment

Production deployments require a tag:

# Create a release tag
git tag -a v1.0.0 -m "Release v1.0.0"
git push origin v1.0.0

# In GitLab, manually trigger deploy:production

Cloud Platform Deployment

Configure cloud platform credentials, then:

1. Ensure variables are set (RENDER_API_KEY, etc.)
2. Manual trigger deploy:cloud job
3. Monitor deployment logs

Testing the Pipeline

Local Testing with GitLab Runner

Test your pipeline locally before pushing:

# Install gitlab-runner locally
# Then run:
gitlab-runner exec docker validate:repository
gitlab-runner exec docker test:automation
gitlab-runner exec docker package:mt5

Validate Configuration

# Install GitLab CI Linter
curl --header "PRIVATE-TOKEN: <your_access_token>" \
  "https://gitlab.com/api/v4/ci/lint" \
  --form "content=@.gitlab-ci.yml"

Or use the web interface:

• Go to CI/CD → Pipeline editor
• Paste your .gitlab-ci.yml content
• Click Validate

Running the Full Pipeline

1. Push changes to main branch or create a merge request
2. Go to CI/CD → Pipelines
3. View the running pipeline
4. Click on jobs to see logs

Troubleshooting

Common Issues

1. Runner Not Picking Up Jobs

Solution:

• Check runner status: sudo gitlab-runner status
• Verify runner tags match job tags
• Check runner is enabled in GitLab UI

2. Docker Permission Denied

Solution:

sudo usermod -aG docker gitlab-runner
sudo gitlab-runner restart

3. Secret Not Found

Solution:

• Verify variable name matches exactly (case-sensitive)
• Check variable is not protected when running on unprotected branch
• Ensure variable is marked as "masked" if it contains sensitive data

4. Package Upload Fails

Solution:

• Check artifact size limits: Settings → CI/CD → General pipelines
• Verify dist/ directory exists and has correct permissions
• Check job artifacts path matches created files

5. Docker Image Push Fails

Solution:

• Verify CI_REGISTRY_IMAGE variable is set
• Check container registry is enabled: Settings → Packages and registries
• Ensure Docker login credentials are correct

Debug Mode

Enable debug mode for more verbose logs:

variables:
  CI_DEBUG_TRACE: "true"

Or set per job:

job_name:
  variables:
    CI_DEBUG_TRACE: "true"
  script:
    - echo "Debug enabled"

Getting Help

• GitLab Documentation: https://docs.gitlab.com/ee/ci/
• GitLab Forum: https://forum.gitlab.com/
• GET Documentation: https://gitlab.com/gitlab-org/gitlab-environment-toolkit
• Support: Open an issue in your repository

Next Steps

After setting up GitLab CI/CD:

1. Review the pipeline - Check all jobs run successfully
2. Configure notifications - Set up Slack/email for pipeline status
3. Set up environments - Configure staging and production environments
4. Enable auto-deployment - For automatic deployments on main branch
5. Monitor performance - Use GitLab's CI/CD analytics

Related Documentation

• GitHub CI/CD Setup - GitHub Actions configuration
• Cloud Deployment Guide - Deploy to cloud platforms
• Secrets Management - Managing credentials securely
• Release Process - Creating releases

Security Best Practices

1. Never commit secrets - Use GitLab CI/CD variables
2. Enable protected branches - Require reviews for main
3. Use masked variables - Hide sensitive values in logs
4. Enable secret detection - GitLab can auto-detect secrets
5. Rotate credentials regularly - Update tokens periodically
6. Limit runner access - Use project-specific runners when possible
7. Enable dependency scanning - Automatically scan for vulnerabilities

---

Last Updated: 2026-02-14
Version: 1.0.0