Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

Open In Colab Open In Kaggle

A result is only as trustworthy as the environment and the data behind it. This subchapter has two halves: the first packages reusable code into an installable, version-pinned project with uv, so an environment can be rebuilt identically on any machine; the second surveys the file formats you will meet, fetches a remote file once and verifies it, and builds the idempotent cache → hash → skip pattern that makes a data pipeline repeatable. At the end, the generated code trusts that a file on disk is the file you wanted, without checking.

Packaging with uv: The src Layout

For code you will reuse or share, promote it from a notebook to an installable package. uv init --lib scaffolds a src/ layout with a pyproject.toml, and uv.lock records the exact dependency versions.

uv init --lib mypackage       # create a src/ layout with pyproject.toml
cd mypackage
uv add numpy                  # add a dependency, updating uv.lock
uv run pytest                 # run the tests in the locked environment

The generated pyproject.toml records the package name, version, and the dependencies uv add pins:

[project]
name = "mypackage"
version = "0.1.0"
dependencies = [
    "numpy>=2.0",
]

Placing code under src/mypackage/ means tests import the installed package, not stray files in the working directory, so packaging mistakes surface immediately. uv.lock pins the full dependency graph, making the environment reproducible on any machine.

Going deeper: Ruff for linting and formatting

Ruff is a fast linter and formatter that replaces flake8, isort, and black.

uv tool install ruff
ruff check .        # lint
ruff format .       # format

Run it in an editor and in CI so style and simple errors never reach review.

Going deeper: static type checking

A type checker (mypy, or Astral’s newer ty) verifies the annotations you already write, catching a class of bugs before runtime.

uvx mypy src/       # check the package against its type hints

Type hints plus a checker turn “this function expects a float” from a comment into an enforced contract.

Going deeper: pre-commit and continuous integration

pre-commit runs checks before each commit; a CI workflow re-runs them on every push so nothing untested is merged.

# .github/workflows/ci.yml
on: [push, pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v5
      - run: uv run pytest

Local hooks catch problems fast, before a commit; CI is what actually enforces them, since a hook can be skipped locally but not in CI.

Going deeper: semantic versioning and coverage

Semantic versioning communicates the nature of a change through the version number MAJOR.MINOR.PATCH: bump MAJOR for breaking changes, MINOR for new features, PATCH for fixes. Coverage measures how much code the tests exercise.

uv run pytest --cov=mypackage      # report line coverage

High test coverage of meaningless tests proves nothing about correctness — coverage measures what ran, not what was checked.

From Reproducible Code to Reproducible Data

Packaging pins how your code runs: the same dependencies, the same versions, on any machine. The rest of this subchapter pins what it runs on — making sure a data file is fetched, verified, and cached the same way every time, so a result doesn’t quietly drift when a remote file changes underneath you.

A Tour of Formats

Match the format to the data. CSV is universal plain text, human-readable but untyped and bulky — fine for small tables and interchange. Unlike CSV, parquet is a columnar binary format: typed, compressed, and fast for large tables. For labelled n-dimensional arrays with metadata, netCDF is the standard single-file container. zarr stores the same array model as a chunked directory (or cloud object store), built for parallel and out-of-core access. When the data is a raster grid rather than a table or array, GeoTIFF holds it together with its coordinate reference system.

import numpy as np
import pandas as pd
from pathlib import Path

Path("_files").mkdir(exist_ok=True)

rng = np.random.default_rng(0)
n = 10_000
big = pd.DataFrame({
    "time": pd.date_range("2024-01-01", periods=n, freq="h"),
    "temp_celsius": (10 + rng.normal(0, 5, n)).round(2),
    "station": rng.choice(["BAS", "LUG", "JFJ"], n),
})

big.to_csv("_files/obs.csv", index=False)         # text, untyped
big.to_parquet("_files/obs.parquet")              # columnar binary, typed, compressed

print("csv bytes:    ", Path("_files/obs.csv").stat().st_size)
print("parquet bytes:", Path("_files/obs.parquet").stat().st_size)
print("parquet keeps dtypes:", pd.read_parquet("_files/obs.parquet").dtypes.to_dict())
csv bytes:     294209
parquet bytes: 116098
parquet keeps dtypes: {'time': dtype('<M8[us]'), 'temp_celsius': dtype('float64'), 'station': <StringDtype(na_value=nan)>}

import xarray as xr
import warnings
warnings.filterwarnings("ignore", message=".*Consolidated metadata.*")

ds = xr.Dataset(
    {"t2m": (("time", "lat", "lon"), rng.normal(5, 3, size=(12, 4, 6)))},
    coords={"time": pd.date_range("2024-01-01", periods=12, freq="MS"),
            "lat": np.linspace(46.0, 47.5, 4), "lon": np.linspace(6.5, 9.0, 6)},
)
ds.to_netcdf("_files/field.nc")               # one self-describing file
ds.to_zarr("_files/field.zarr", mode="w")     # a chunked directory store, cloud-friendly

print("netcdf is a file:    ", Path("_files/field.nc").is_file())
print("zarr is a directory: ", Path("_files/field.zarr").is_dir())
print("reopened netcdf shape:", xr.open_dataset("_files/field.nc")["t2m"].shape)
netcdf is a file:     True
zarr is a directory:  True
reopened netcdf shape: (12, 4, 6)

Fetching Data over HTTP

requests performs the HTTP GET underneath any download. The essential pattern checks the status and writes the bytes:

import requests
r = requests.get(url, timeout=30)
r.raise_for_status()                 # turn a 404/500 into an exception
Path("data.csv").write_bytes(r.content)

This works, but it re-downloads on every run and verifies nothing about which file arrived. Pinning and caching are the next step.

pooch: Download Once, Verify Always

pooch.retrieve wraps the request, caches the file, and checks its hash against a value you pin. Calling it again returns the cached path with no download.

import pooch
path = pooch.retrieve(
    url="https://example.org/data/temperature.csv",
    known_hash="sha256:1a2b3c...",      # pin the exact file
    path=pooch.os_cache("mlees"),        # OS-appropriate cache folder
)
data = pd.read_csv(path)

The pinned hash is what makes the pipeline reproducible: if the remote file ever changes, the check fails loudly instead of feeding you different data in silence. The call is shown rather than run so this book builds without network access; the cell below implements the same cache → hash → skip logic on a local file, using pooch’s real hashing.

The Idempotent Fetch: Cache, Hash, Skip

A robust fetch is idempotent — calling it repeatedly does the least work and always yields the same verified file. The logic is: if a cached copy exists and its hash matches, use it; otherwise (re)download and verify. Here a local file stands in for the remote source so the logic runs offline.

import shutil
import pooch

source = Path("_files/source_temperature.csv")   # stands in for a remote URL
pd.DataFrame({"date": pd.date_range("2024-06-01", periods=5, freq="D"),
              "temp_celsius": [18.2, 17.5, 19.1, 16.8, 18.0]}).to_csv(source, index=False)
known_hash = pooch.file_hash(source)              # real sha256, computed once
print("known sha256:", known_hash[:16], "...")

cache = Path("_files/cache/temperature.csv")

def fetch_verified(expected_hash):
    cache.parent.mkdir(exist_ok=True)
    if cache.exists() and pooch.file_hash(cache) == expected_hash:
        print("cache hit:  hash matches, skip download")
        return cache
    print("cache miss: download and verify")
    shutil.copy(source, cache)                    # stands in for the network download
    if pooch.file_hash(cache) != expected_hash:
        raise ValueError("hash mismatch after download")
    return cache

fetch_verified(known_hash)   # first call: miss -> download
fetch_verified(known_hash)   # second call: hit -> skip
known sha256: 428eeaa65c3c10f6 ...
cache miss: download and verify
cache hit:  hash matches, skip download
PosixPath('_files/cache/temperature.csv')
# point pandas at the verified cache path
data = pd.read_csv(cache, parse_dates=["date"])
print("shape:", data.shape, "| mean temp:", data["temp_celsius"].mean().round(2))
shape: (5, 2) | mean temp: 17.92

# xarray reads from the same verified cache, not just its own separate files
cached = pd.read_csv(cache, parse_dates=["date"])
ds_cached = xr.Dataset(
    {"temp_celsius": ("date", cached["temp_celsius"].values)},
    coords={"date": cached["date"].values},
)
print(ds_cached)
<xarray.Dataset> Size: 80B
Dimensions:       (date: 5)
Coordinates:
  * date          (date) datetime64[us] 40B 2024-06-01 2024-06-02 ... 2024-06-05
Data variables:
    temp_celsius  (date) float64 40B 18.2 17.5 19.1 16.8 18.0
Solution
from pathlib import Path
import pooch
ok = Path("_files/cache/temperature.csv").exists() and pooch.file_hash("_files/cache/temperature.csv") == known_hash
print(ok)

When generated code lies: trusting a file that is merely present

Asked to “download the data if it is not already there”, an assistant checks only whether the file exists. But a file can exist and still be wrong — truncated by an interrupted download, or left over from an older version.

def fetch_unverified(cache_path, source_path):
    # download only if the file is missing (as an assistant returned it)
    p = Path(cache_path)
    if not p.exists():
        shutil.copy(source_path, p)
    return pd.read_csv(p)

# a previous interrupted download left a TRUNCATED file in the cache
stale = Path("_files/cache_b/temperature.csv")
stale.parent.mkdir(exist_ok=True)
stale.write_text("date,temp_celsius\n2024-06-01,18.2\n")   # 1 of 5 rows

result = fetch_unverified(stale, source)
print("rows returned:", len(result), "(should be 5)")
rows returned: 1 (should be 5)

def fetch_verified_data(cache_path, source_path, expected_hash):
    p = Path(cache_path)
    if not (p.exists() and pooch.file_hash(p) == expected_hash):
        shutil.copy(source_path, p)               # re-fetch on miss OR hash mismatch
    if pooch.file_hash(p) != expected_hash:
        raise ValueError("hash mismatch: refusing corrupted data")
    return pd.read_csv(p)

result = fetch_verified_data(stale, source, known_hash)
print("rows returned:", len(result), "(now correct)")
rows returned: 5 (now correct)
Going deeper: multi-file registries

For several files, pooch.create centralises the base URL and a registry of names and hashes, fetched by name.

POOCH = pooch.create(
    path=pooch.os_cache("mlees"),
    base_url="https://example.org/data/",
    registry={
        "temperature.csv": "sha256:1a2b3c...",
        "discharge.nc": "sha256:4d5e6f...",
    },
)
path = POOCH.fetch("temperature.csv")   # cached + hash-verified
Going deeper: DOI-versioned data (Zenodo, figshare)

pooch can fetch directly from a DOI, which points to an immutable, archived version of a dataset.

doi = "doi:10.5281/zenodo.5739406"
path = pooch.retrieve(url=f"{doi}/wind_hourly.csv", known_hash="md5:cf059b...")

Zenodo records typically publish an MD5 checksum rather than SHA256; pooch accepts either algorithm pooch/hashlib supports. A DOI is the strongest pin available: the archive guarantees the bytes behind that identifier never change.

Going deeper: intake catalogs

intake describes datasets in a YAML catalog so that code refers to a dataset by name rather than by path or URL, separating what data you want from where it lives.

import intake
cat = intake.open_catalog("catalog.yml")
ds = cat["era5_t2m"].to_dask()

This keeps notebooks stable when storage locations change.

Going deeper: provenance and FAIR

FAIR data is Findable, Accessible, Interoperable, and Reusable. In practice: deposit data in an archive that issues a DOI (Zenodo, figshare), record the exact version and hash used, and keep the fetch code alongside the analysis. Provenance — a clear trail from raw input to final figure — is what lets someone else (including future you) reproduce a result rather than approximate it.

Summary

ConceptRule to remember
PackagingPromote reusable code to a src/ package with uv init --lib; pin dependencies with uv.lock.
FormatsCSV for small interchange, parquet for large tables, netCDF for labelled arrays, zarr for chunked or cloud arrays, GeoTIFF for rasters.
FetchingA bare requests.get re-downloads every time and verifies nothing.
PinningA hash says what a file is, not just what it is called.
The idempotent fetchcache → hash → skip; re-fetch on a miss or a hash mismatch.
Existence is not integrityA truncated or stale file passes an exists() check and corrupts everything downstream.

Resources

Machine Learning for Earth and Environmental Sciences
Bonus material
Machine Learning for Earth and Environmental Sciences
Exercises