Football Statistics Tracker 📊⚽

An end-to-end data engineering pipeline that collects, processes, and analyzes football match results, standings data, weather data, Reddit data and summarizes matchdays using Gemini from the top 5 European leagues. Used data sources include football-data.org API, Open-Meteo API, and PRAW (Reddit API), Maps...

Introduction

This project demonstrates a complete data pipeline for football (soccer) results, from data extraction to visualization. It implements some data engineering practices including data lakes, transformation layers, and Infrastructure as Code (IaC) with Terraform.

Features

• Automated Data Collection: Scheduled data fetching from multiple APIs using Google Cloud Functions
• Multi-layer Data Architecture: Raw data stored in GCS, processed data in BigQuery, and user-facing data in Firestore
• Weather Integration: Match statistics with weather data at match time
• Social Media (Reddit) Data: Reddit comments for fan sentiment
• Infrastructure as Code: Cloud Functions and Pub/Sub subscriptions and topics defined and deployed with Terraform

Architecture

The pipeline follows the following architecture:

1. Data Ingestion: Cloud Functions trigger on schedule to fetch data
2. Storage Layers: Raw data(json) → External BQ tables (Parquet) → Processed Data in BQ → Firestore
3. Validation: Very simple validation and Data qaulity with Dataplex
4. Summarization: Creation of short summaries in Markdown with Gemini 2.0 Flash
5. Visualization: Web app for insights

Data Sources

• Football-data.org: Match data, team data, and standings
• Open-Meteo API: Historical weather data
• Reddit (via PRAW): Fan comments and sentiment
• Maps SDK: Location of stadiums

Technology Stack

Project Structure

soccer-tracker-DE-project/
├── README.md
├── .gitignore
├── pyproject.toml
├── Github/workflows/                  # CI/CD in Github Actions
│   ├── cd.yml
│   └── ci.yml
├── terraform/                         # IaC definitions
│   ├── main.tf
│   ├── variables.tf
│   ├── pubsub.tf
│   └── cloud_functions.tf
├── cloud_functions/
│   ├── league_data/                   # League and Teams data extraction and load
│   ├── discord_utils/                 # Package for sending Discord notifications using webhooks
│   ├── match_data/                    # Match data extraction and load
│   ├── weather_data/                  # Weather data extraction and load
│   ├── reddit_data/                   # Reddit data extraction and load
│   ├── standings_data/                # Standings data extraction and load for each matchday
│   ├── data_validation/               # Data validation using Dataplex
│   ├── serving_layer/                 # Load data to firestore
│   └── generate_summaries/            # Generate match summaries with Gemini
├── soccer_tracker_ui/                 # Svelte web app in Firebase
│   ├── src/
│   │   ├── lib/                       # Reusable components
│   │   │   ├── components/            # UI components from [shadcn](https://next.shadcn-svelte.com/)
│   │   │   ├── firebase.ts            # Firebase/Firestore connection
│   │   │   └── stores/                # Svelte stores for state management
│   │   ├── routes/                    # Page components
│   ├── package.json                   # Dependencies and scripts
│   ├── svelte.config.js               # Svelte configuration
│   ├── vite.config.js                 # Vite bundler config
└── tests/                             # Test suite for Cloud Functions with Pytest

Additional Documentation

• Tests README
• Cloud Functions README
• Terraform README
• GH Actions README

Web app

The project includes a Svelte web app for visualizing match results, weather data, and match summaries.

App includes:

• Match Results
• Match summaries using an LLM (Gemini 2.0 Flash)
• Weather data during matches
• Comments from Reddit

I got the idea to make this project from this repo by digitalghost-dev