Model Selection

Model selection is the process of choosing the best model from a set of candidate models. This involves balancing model complexity to achieve the best generalization on new data.

1. The Validation Strategy

We cannot use the Test Set to choose our model, as that would "leak" information and lead to over-optimistic results. Instead, we split the training data further:

1. Training Set (e.g., 70-80%): Used to fit the parameters ($\boldsymbol{\theta}$) of the model.
2. Validation Set (e.g., 10-20%): Used to tune Hyperparameters (e.g., learning rate $\alpha$, polynomial degree $d$) and choose the best model architecture.
3. Test Set (e.g., 10%): Used ONLY once at the very end to estimate real-world performance on unseen data.

2. Hyperparameter Tuning

Hyperparameters are settings that are chosen before the learning process begins. They are not learned by Gradient Descent. We use the Validation Set to find their optimal values.

Example: Selecting the Learning Rate $\alpha$

Notice how Model 2 is the best choice because it has the lowest validation error, even if its training error is slightly higher than Model 1.

3. k-Fold Cross-Validation

When data is scarce, we use k-Fold Cross-Validation:

• Split training data into $k$ "folds".
• Train $k$ times, each time using $k-1$ folds for training and 1 fold for validation.
• Average the validation performance across all $k$ runs.

4. Visualizing the Selection Curve

The optimal model is located at the point where the validation error is at its minimum.