NFL data ELT tool.

Find a file

Erikson Arias 9522831938 fix(): schedule fails to update events with special names		2026-01-27 21:46:40 -05:00
dbt	feat(): allow tbd games to be exposed in marts modeling	2026-01-06 18:08:15 -05:00
nfl_data	fix(): schedule fails to update events with special names	2026-01-27 21:46:40 -05:00
.env.example	Add Infisical integration	2025-08-17 01:17:21 -04:00
.gitignore	chore(): organize hierarchy	2025-12-06 20:02:30 -05:00
.infisical.json	chore(): rotate infisical workspace id	2025-09-29 20:51:12 -04:00
app.py	feat(): add default year parameter to load-schedule	2026-01-06 13:03:10 -05:00
Dockerfile	feat(): create dockerfile	2025-12-30 17:08:29 -05:00
fantasy_scoring.md	Fix extension	2025-03-31 01:41:19 -04:00
README.md	feat(): add default year parameter to load-schedule	2026-01-06 13:03:10 -05:00
requirements.txt	feat(): add weather	2026-01-04 00:33:59 -05:00

README.md

NFL-DATA

NFL data extraction, load, and transformation from ESPN NFL API using Python and dbt.

Project setup

Set up and activate a virtual environment with the following command:

python -m venv venv

# Windows
./venv/Scripts/activate

# Linux
source venv/bin/activate

Install all the necessary packages using the following command:

python -m pip install -r requirements.txt

Environment variables

Secrets management is made possible using Infisical. Install the CLI to inject secrets.

Otherwise, use a .env to specify the required variables. See .env.example.

Command reference

NFL-DATA CLI

Infisical and secrets

Using Infisical for secrets management is recommended. Commands can be wrapped in an Infisical call:

infisical run --command "python app.py ..."

Load strategy

All commands have the --load-strategy or -ls option to specify how NFL-DATA will load new JSON data into the warehouse:

replace clears out all previously loaded JSON data before inserting.
add will load JSON data without any additional checks. Will introduce multiple entries for the same object.
skip will only load JSON data if no other data exists for a given object.
day_replace will replace JSON data if the same object was previously loaded today.

Commands

python app.py game-day # Game day run: pull today's schedule and load data for completed games.
python app.py load-season [year] # Load season data.
python app.py load-schedule --year year # Load schedule data. Year is optional, defaults to current year.
python app.py load-event [event-id] # Load event, drives, and roster data.
python app.py load-venue [venue-id|all] # Load venue data; will load all venues when venue-id is unspecified.
python app.py get-weather [event-id] # Load weather data by event. Loads all events with missing weather data when event-id is unspecified.

python app.py docs # Upload dbt static docs to S3.

dbt

To use dbt commands, first change directory into the dbt subdirectory.

cd dbt/

From here, all dbt commands can be accessed. Take a look at the dbt Command Reference page for details.

Running a full build will model newly loaded raw data since last full build:

dbt build