6.3) (Exercise) Land Cover Classification

This notebook is converted from the source: https://www.coursera.org/learn/getting-started-with-tensor-flow2

Exercise Instruction¶

By the end of this notebook, you’ll be able to 😃😃😃

Construct CNNs that classifies EuroSAT images into one of its 10 classes;
Save and load trained models;
Explore ways to improve the model performance.

Land Cover Classification aims to automatically provide labels describing the represented physical land type or how a land area is used (e.g., residential, industrial).

Convolutional Neural Networks (CNNs), the state of-the-art image classification method in computer vision and machine learning, have been reported to be suitable for the classification of remotely sensed images.

However, the classification of remotely sensed images is a challenging task, particularly due to the lack of reliably labeled ground truth datasets.

The EuroSAT dataset provides large quantity of training data for this purpose. It consists of 27000 labelled Sentinel-2 satellite images of different land uses: residential, industrial, highway, river, forest, pasture, herbaceous vegetation, annual crop, permanent crop and sea/lake.

For a reference, see the following papers:

Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. Patrick Helber, Benjamin Bischke, Andreas Dengel, Damian Borth. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019.
Introducing EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification. Patrick Helber, Benjamin Bischke, Andreas Dengel. 2018 IEEE International Geoscience and Remote Sensing Symposium, 2018.

⚡⚡⚡ You can create your own code by following each question or complete the code with blanks.

# Run this cell first to import all required packages.
import torch
import torch.nn as nn
from collections import OrderedDict

import torchvision

import torchmetrics
from sklearn.model_selection import train_test_split

import os
import glob
import tarfile
import pooch
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
# If you would like to make further imports from torch or torchvision, add them here

Data Setup¶

Using the EuroSAT dataset which consists of 27000 images and labels might use too much memory, thus we use a smaller subset of the original dataset - 4000 training images and 1000 testing images with roughly equal numbers of each class.

# Run this cell to fetch and reassemble the dataset
chunk_suffixes = ["aa", "ab"]
chunk_hashes = {
    "aa": "ce9f73e68261a010956eeb73876f047f35a25986b037694cd08cc0a837418f3c",
    "ab": "b87e1a0b9da3e69579723103a4934d1b0bbb518d3d1a0d11f51b5ff75db56e25",
}
cache_dir = pooch.os_cache("mlees")
chunk_paths = [
    pooch.retrieve(
        url=f"https://raw.githubusercontent.com/gse-unil/2026_MLEES_book/main/data/part-III/data_cnn.zip.part{suffix}",
        known_hash=f"sha256:{chunk_hashes[suffix]}",
        fname=f"data_cnn.zip.part{suffix}",
        path=cache_dir,
    )
    for suffix in chunk_suffixes
]

data_cnn_archive = os.path.join(cache_dir, "data_cnn.zip")
if not os.path.exists(data_cnn_archive):
    with open(data_cnn_archive, "wb") as out_f:
        for chunk_path in chunk_paths:
            with open(chunk_path, "rb") as in_f:
                out_f.write(in_f.read())
    import zipfile
    with zipfile.ZipFile(data_cnn_archive) as z:
        z.extractall(cache_dir)

data_cnn_dir = os.path.join(cache_dir, "data_cnn")

# Import the Eurosat data
def load_eurosat_data():
    x_train = np.load(os.path.join(data_cnn_dir, 'x_train.npy'))
    y_train = np.load(os.path.join(data_cnn_dir, 'y_train.npy'))
    x_val  = np.load(os.path.join(data_cnn_dir, 'x_test.npy'))
    y_val  = np.load(os.path.join(data_cnn_dir, 'y_test.npy'))
    return (x_train, y_train), (x_val, y_val)

(X_train, y_train), (X_val_test, y_val_test) = load_eurosat_data()
X_val, X_test, y_val, y_test = train_test_split(X_val_test, y_val_test, test_size=0.5, random_state=42)

# Normalize data, move the channel dimension to PyTorch's expected position
# (channels first), and convert everything to tensors
def to_tensor(X, y):
    X = torch.tensor(X / 255.0, dtype=torch.float32).permute(0, 3, 1, 2)
    y = torch.tensor(y, dtype=torch.long).reshape(-1)  # (N, 1) -> (N,)
    return X, y

X_train, y_train = to_tensor(X_train, y_train)
X_val, y_val = to_tensor(X_val, y_val)
X_test, y_test = to_tensor(X_test, y_test)

Since PyTorch has no .compile()/.fit()/.evaluate(), we’ll define three small reusable helpers before building any models — one to train a model for a number of epochs (optionally with callbacks and a validation set), one to evaluate a model on a dataset, and a History class that mimics Keras’s history.history dict-of-lists so the rest of this notebook can stay close to the original.

class History:
    def __init__(self):
        self.history = {"loss": [], "accuracy": [], "val_loss": [], "val_accuracy": []}


def evaluate_model(model, X, y, batch_size=32):
    model.eval()
    loader = torch.utils.data.DataLoader(torch.utils.data.TensorDataset(X, y), batch_size=batch_size)
    losses = []
    accuracy.reset()
    with torch.no_grad():
        for X_batch, y_batch in loader:
            y_pred = model(X_batch)
            losses.append(loss_fn(y_pred, y_batch).item())
            accuracy.update(y_pred, y_batch)
    return [np.mean(losses), accuracy.compute().item()]


def train_model(model, X, y, epochs, validation_data=None, callbacks=None, verbose=1):
    callbacks = callbacks or []
    optimizer = optimizer_class(model.parameters())
    history = History()
    loader = torch.utils.data.DataLoader(torch.utils.data.TensorDataset(X, y), batch_size=32, shuffle=True)

    for epoch in range(epochs):
        model.train()
        train_losses = []
        accuracy.reset()
        for X_batch, y_batch in loader:
            optimizer.zero_grad()
            y_pred = model(X_batch)
            loss = loss_fn(y_pred, y_batch)
            loss.backward()
            optimizer.step()
            train_losses.append(loss.item())
            accuracy.update(y_pred, y_batch)

        logs = {"loss": np.mean(train_losses), "accuracy": accuracy.compute().item()}
        history.history["loss"].append(logs["loss"])
        history.history["accuracy"].append(logs["accuracy"])

        report = f"Epoch {epoch + 1}/{epochs} - loss: {logs['loss']:.4f} - accuracy: {logs['accuracy']:.4f}"
        if validation_data is not None:
            val_loss, val_accuracy = evaluate_model(model, *validation_data)
            logs["val_loss"], logs["val_accuracy"] = val_loss, val_accuracy
            history.history["val_loss"].append(val_loss)
            history.history["val_accuracy"].append(val_accuracy)
            report += f" - val_loss: {val_loss:.4f} - val_accuracy: {val_accuracy:.4f}"
        if verbose:
            print(report)

        stop_training = False
        for callback in callbacks:
            if callback.on_epoch_end(epoch, model, logs) == "stop":
                stop_training = True
        if stop_training:
            break

    return history

Q1 Build a CNN `model1` to classify Eurosat data.¶

Let’s construct a CNN called model1 using nn.Sequential, according to the following specifications:

The first layer should be a Conv2d layer with 16 filters, a 3x3 kernel size, and ‘same’ padding. Name this layer ‘conv_1’.
Follow it with a ReLU activation, named ‘relu_1’.
The third layer should also be a Conv2d layer with 8 filters, a 3x3 kernel size, and ‘same’ padding. Name this layer ‘conv_2’.
Follow it with a ReLU activation, named ‘relu_2’.
The fifth layer should be a MaxPool2d layer with a pooling window size of 8x8. Name this layer ‘pool_1’.
The sixth layer should be a Flatten layer, named ‘flatten’.
The seventh layer should be a Linear layer with 32 units, followed by a ReLU activation. Name these layers ‘dense_1’ and ‘relu_3’.
The final layer should be a Linear layer with 10 units and no activation (unlike Keras, nn.CrossEntropyLoss expects raw logits, not softmax probabilities). Name this layer ‘dense_2’.

Hint 1: Unlike keras.models.Sequential, nn.Sequential doesn’t have an .add() method — but if you pass it an OrderedDict of (name, layer) pairs, you get the same named-layer access (model1.conv_1, etc.) that Keras’s name= argument gives you.

Hint 2: nn.Conv2d needs an explicit number of input channels — 3 for the first layer (RGB), and conv_2’s input channels equal conv_1’s output channels.

Hint 3: As in the artificial-neural-networks and deep-computer-vision exercises, nn.LazyLinear can save you from computing the flattened size by hand.

# Assign value to input_shape variable (channels, height, width)
input_shape = ____.shape[___:]

# Build the model, one named layer at a time
model1 = nn.Sequential(OrderedDict([
    ("conv_1", nn.Conv2d(___, ___, kernel_size=___, padding="___")),
    ("relu_1", nn.___()),
    ("conv_2", nn.Conv2d(___, ___, kernel_size=___, padding="___")),
    ("relu_2", nn.___()),
    ("pool_1", nn.MaxPool2d((___, ___))),
    ("flatten", nn.___()),
    ("dense_1", nn.LazyLinear(___)),
    ("relu_3", nn.___()),
    ("dense_2", nn.LazyLinear(___)),
]))

# Run one batch through the model to materialize the LazyLinear layers
model1(X_train[:1])

❓❓❓ Do you have a model of the following structure?

(This screenshot is from the original Keras version of this exercise — print(model1) will show a differently formatted, but structurally equivalent, layer list.)

Q2 Set up `model1`’s loss function, optimizer, and an evaluation metric.¶

Use the Adam optimiser, cross entropy loss function, and a single accuracy metric.

loss_fn = nn.___() # Set the loss function
optimizer_class = torch.optim.___ # Set the optimizer class (train_model binds it to each model's parameters)
accuracy = torchmetrics.Accuracy(task="___", num_classes=___) # Set the evaluation metric

Q3 Evaluate the initial accuracy of `model1`: Is the initial accuracy of the model as you have expected?¶

# Calculate its initialised test accuracy
test_loss, test_acc = evaluate_model(____, ____, ____)
print('Test accuracy: {acc:0.3f}'.format(acc=test_acc))

❓❓❓ Does your model have a similar initial accuracy & why?

Q4 Train `model1` with 15 epochs, store the result in variable ‘history’;¶

Store the fitting result in a variable history;

epochs = ____

history = train_model(____, ____, ____,
                    epochs=____,
                    validation_data=(____, ____))

❓❓❓ Do you have similar print output?

(Again, expect a differently formatted but comparable progress report — there’s no Keras-style progress bar here.)

Q5 Evaluate the accuracy of fitted `model1`, plot training, validation set loss and accuracy, and print the model’s structure¶

Evaluate the fitted model
- What’s the test accuracy after training the model? Does it improve from the initialization?
Print the model’s structure

# Calculate the test accuracy
score = evaluate_model(____, ____, ____)
print('Test accuracy: {acc:0.3f}'.format(acc=score[1]))

❓❓❓ Do you have similar test accuracy?

# Plot training, validation set loss and accuracy
pd.DataFrame(____.____).plot(figsize=(8,5))
plt.show()

❓❓❓ Do you have a similar plot?

# Print the model's structure
print(____)

❓❓❓ Do you have a similar structure?

(This reference was generated with Keras’s plot_model utility, which has no PyTorch equivalent — print(model) gives you the same layer-by-layer information as text instead of a diagram.)

Q6 Create two callbacks to save the model weights at each epoch and the best validation accuracy epoch¶

checkpoint_every_epoch: a callback that saves the model weights every epoch during training;
checkpoint_best_only: a callback that saves only the weights with the highest validation accuracy.

Hint 1: Since train_model calls each callback’s on_epoch_end(epoch, model, logs) method after every epoch, a callback here is just a small class with that one method — logs is the same dict of "loss"/"accuracy"/"val_loss"/"val_accuracy" values printed for that epoch.

Hint 2: Use torch.save(model.state_dict(), path) to save weights — there’s no PyTorch equivalent of Keras’s ModelCheckpoint, so both callbacks save with plain torch.save().

# A callback that saves the model weights at the end of every epoch, into a
# directory called 'checkpoints_every_epoch', with filenames like
# 'checkpoint_000.pt', 'checkpoint_001.pt', etc. (the epoch number, formatted
# to three digits)
class CheckpointEveryEpoch:
    def __init__(self, dirpath):
        self.dirpath = dirpath
        os.makedirs(dirpath, exist_ok=True)

    def on_epoch_end(self, epoch, model, logs):
        path = os.path.join(self.___, f"checkpoint_{____:03d}.pt")
        torch.save(model.___(), ___)

checkpoint_every_epoch = CheckpointEveryEpoch(dirpath="checkpoints_every_epoch")

# A callback that saves the model weights only when they reach the highest
# validation accuracy so far, into a file called
# 'checkpoints_best_only/checkpoint.pt'
class CheckpointBestOnly:
    def __init__(self, filepath, monitor="val_accuracy"):
        self.filepath = filepath
        self.monitor = monitor
        self.best = -np.inf
        os.makedirs(os.path.dirname(filepath), exist_ok=True)

    def on_epoch_end(self, epoch, model, logs):
        if logs[___] > self.___:
            self.___ = logs[self.monitor]
            torch.save(model.___(), self.___)

checkpoint_best_only = CheckpointBestOnly(filepath=os.path.join("checkpoints_best_only", "checkpoint.pt"),
                                          monitor="___")

Q7 Build a CNN `model` with the same initial structure as `model1` and train for 15 epochs using the callbacks from Q6¶

Now, you will train the model using the two callbacks you created. If you created the callbacks correctly, two things should happen:

At the end of every epoch, the model weights are saved into a directory called checkpoints_every_epoch
At the end of every epoch, the model weights are saved into a directory called checkpoints_best_only only if those weights lead to the highest validation accuracy

You should then have two directories:

A directory called checkpoints_every_epoch containing filenames that include checkpoint_000.pt, checkpoint_001.pt, etc, with the numbers corresponding to the epoch
A directory called checkpoints_best_only containing a single checkpoint.pt file, containing only the weights leading to the highest validation accuracy

# Create a model with the same initial structure as model1 again from scratch
model = nn.Sequential(OrderedDict([
    ("conv_1", nn.Conv2d(3, 16, kernel_size=3, padding="same")),
    ("relu_1", nn.ReLU()),
    ("conv_2", nn.Conv2d(16, 8, kernel_size=3, padding="same")),
    ("relu_2", nn.ReLU()),
    ("pool_1", nn.MaxPool2d((8, 8))),
    ("flatten", nn.Flatten()),
    ("dense_1", nn.LazyLinear(32)),
    ("relu_3", nn.ReLU()),
    ("dense_2", nn.LazyLinear(10)),
]))
model(X_train[:1])

# Train model using the callbacks you just created with 15 epochs

callbacks = [____, ____]
history = train_model(____, ____, ____, epochs=15, validation_data=(X_val, y_val), ____=____)

❓❓❓ Do you have similar print output?

Q8 Create new models `model_last_epoch` and `model_best_epoch` with model1’s initial structure; load weights from the latest saved epoch and the saved epoch with the highest validation accuracy respectively¶

Now you will use the weights you just saved in a fresh model. You should load into two freshly instantiated model instances:

model_last_epoch should contain the weights from the latest saved epoch
model_best_epoch should contain the weights from the saved epoch with the highest validation accuracy

Hint: glob.glob("checkpoints_every_epoch/*.pt") gives you every saved checkpoint path; sorting that list gives you the latest one at the end (since the filenames are zero-padded).

# Create a new CNN with same structure as model1
model_last_epoch = nn.Sequential(OrderedDict([
    ("conv_1", nn.Conv2d(3, 16, kernel_size=3, padding="same")),
    ("relu_1", nn.ReLU()),
    ("conv_2", nn.Conv2d(16, 8, kernel_size=3, padding="same")),
    ("relu_2", nn.ReLU()),
    ("pool_1", nn.MaxPool2d((8, 8))),
    ("flatten", nn.Flatten()),
    ("dense_1", nn.LazyLinear(32)),
    ("relu_3", nn.ReLU()),
    ("dense_2", nn.LazyLinear(10)),
]))
model_last_epoch(X_train[:1])

# Load the weights from the last training epoch.
latest_checkpoint = sorted(glob.glob(____))[___]
model_last_epoch.___(torch.load(____, weights_only=True))

# Create a new CNN with same structure as model1
model_best_epoch = nn.Sequential(OrderedDict([
    ("conv_1", nn.Conv2d(3, 16, kernel_size=3, padding="same")),
    ("relu_1", nn.ReLU()),
    ("conv_2", nn.Conv2d(16, 8, kernel_size=3, padding="same")),
    ("relu_2", nn.ReLU()),
    ("pool_1", nn.MaxPool2d((8, 8))),
    ("flatten", nn.Flatten()),
    ("dense_1", nn.LazyLinear(32)),
    ("relu_3", nn.ReLU()),
    ("dense_2", nn.LazyLinear(10)),
]))
model_best_epoch(X_train[:1])

# Load the weights leading to the highest validation accuracy.
model_best_epoch.___(torch.load(____, weights_only=True))

# Verify the validation accuracy of the last and the best model.

score = evaluate_model(model_last_epoch, ____, ____)
print('Model validation accuracy with last epoch weights: {acc:0.3f}'.format(acc=score[1]))
print('')

score = evaluate_model(model_best_epoch, ____, ____)
print('Model validation accuracy with best epoch weights: {acc:0.3f}'.format(acc=score[1]))

❓❓❓ Are the saved models’ validation accuracy as expected?

Q9 Explore to improve the model performance by trying to reduce the bias: design and train a model2¶

For example, train a CNN called model2, which has the same structure as model1, but with more convolution units: conv_1 has 32 units, conv_2 has 64 units, dense_1 with 256 units

❓❓❓

Does the model performance improve?
What are other potential strategies to improve model performance by reducing the bias?

# Build model2 with the same structure as model1, but change the number of
# filters in conv_1 to 32 and conv_2 to 64, and dense_1 to 256 units.
model2 = nn.Sequential(OrderedDict([
    ("conv_1", nn.Conv2d(___, ___, kernel_size=3, padding="same")),
    ("relu_1", nn.ReLU()),
    ("conv_2", nn.Conv2d(___, ___, kernel_size=3, padding="same")),
    ("relu_2", nn.ReLU()),
    ("pool_1", nn.MaxPool2d((8, 8))),
    ("flatten", nn.Flatten()),
    ("dense_1", nn.LazyLinear(___)),
    ("relu_3", nn.ReLU()),
    ("dense_2", nn.LazyLinear(10)),
]))
model2(X_train[:1])

# Train the model and store the results in a variable called history
history = train_model(____, ____, ____, epochs=15, validation_data=(____, ____))

❓❓❓ Are your printed output similar to the following screenshot?

# Calculate the test accuracy
score = evaluate_model(____, ____, ____)
print('Test accuracy: {acc:0.3f}'.format(acc=score[1]))

❓❓❓ Does your model’s test accuracy improve and why?

Q10 Explore to improve the model performance by trying to reduce the variance: design and train a `model3`¶

For example, train a CNN called model3, which has the same structure as model2, add dropout layers one after the max pooling layer, one before the final dense layer with dropout rate of 0.2.

❓❓❓

Does the model performance improve?
What are other potential strategies to improve model performance by reducing the variance?

# Build model3 with the same structure as model2, but add two dropout layers -
# one after the max pooling layer, one before the final dense layer.
model3 = nn.Sequential(OrderedDict([
    ("conv_1", nn.Conv2d(32, 64, kernel_size=3, padding="same")),
    ("relu_1", nn.ReLU()),
    ("conv_2", nn.Conv2d(64, 64, kernel_size=3, padding="same")),
    ("relu_2", nn.ReLU()),
    ("pool_1", nn.MaxPool2d((8, 8))),
    ("dropout_1", nn.Dropout(___)),
    ("flatten", nn.Flatten()),
    ("dense_1", nn.LazyLinear(256)),
    ("relu_3", nn.ReLU()),
    ("dropout_2", nn.Dropout(___)),
    ("dense_2", nn.LazyLinear(10)),
]))
model3(X_train[:1])

# Train the model and store the results in a variable called history
history = train_model(____, ____, ____, epochs=15, validation_data=(____, ____))

❓❓❓ Are your printed output similar to the following screenshot?

# Calculate the test accuracy
score = evaluate_model(____, ____, ____)
print('Test accuracy: {acc:0.3f}'.format(acc=score[1]))

❓❓❓ Does your model’s test accuracy improve and why?

Q11 Explore to improve the model performance using transfer learning from a pretrained model¶

Use a pretrained VGG16 model from TorchVision.
- Does the model performance improve after the above explorations?
- How would you further improve the model performance?

Hint 1: torchvision.models.vgg16(weights=torchvision.models.VGG16_Weights.DEFAULT).features gives you just VGG16’s convolutional layers (no classifier head) — the PyTorch equivalent of Keras’s include_top=False.

Hint 2: Freeze every parameter in the pretrained feature extractor with param.requires_grad = False, so training only updates the new layers you add on top.

conv_base = torchvision.models.___(weights=torchvision.models.___.DEFAULT).___
# freeze the weights
for param in conv_base.___():
    param.requires_grad = ___

model4 = nn.Sequential(OrderedDict([
    ("features", conv_base),
    ("flatten", nn.Flatten()),
    ("dropout_1", nn.Dropout(0.2)),
    ("dense_1", nn.LazyLinear(256)),
    ("relu_1", nn.ReLU()),
    ("dropout_2", nn.Dropout(0.2)),
    ("dense_2", nn.LazyLinear(10)),
]))
model4(X_train[:1])

history = train_model(model4, X_train, y_train, epochs=15, validation_data=(X_val, y_val))

❓❓❓ Are your printed output similar to the following screenshot? It takes a really long time to train ...

# Calculate the test accuracy
score = evaluate_model(model4, X_test, y_test)
print('Test accuracy: {acc:0.3f}'.format(acc=score[1]))

✌✌✌ Congratulations! You have completed this exercise. Now you know how to train CNNs to classify remote sensing images.

Still, the accuracy reported from the Eurosat dataset paper - Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification is 98.57%. ⚡⚡⚡Have you found strategies to improve the performance to that level?

6.3) (Exercise) Land Cover Classification

Exercise Instruction¶

Data Setup¶

Q1 Build a CNN model1 to classify Eurosat data.¶

Q2 Set up model1’s loss function, optimizer, and an evaluation metric.¶

Q3 Evaluate the initial accuracy of model1: Is the initial accuracy of the model as you have expected?¶

Q4 Train model1 with 15 epochs, store the result in variable ‘history’;¶

Q5 Evaluate the accuracy of fitted model1, plot training, validation set loss and accuracy, and print the model’s structure¶