Understanding a Simple Health ML Classifier

This week I wanted to understand what actually happens inside a simple machine learning classification task in healthcare.

I used the Diabetic Retinopathy Debrecen dataset from UCI. It contains 1,151 examples. Each example has 19 numerical features extracted from a retinal image and one target label showing whether signs of diabetic retinopathy are present.

Understanding the data

The first thing I needed to understand was the structure of the data.

The dataset is simply a table. The feature columns are the information given to the model, usually called X, while the target column is the answer we want the model to learn, usually called y as seen in the code.

Splitting the data

I then split the data into training and test sets with the ratio 4:1.

X_train, X_test, y_train, y_test = train_test_split( 
  X,
  y, 
  test_size=0.2,
  random_state=42,
  stratify=y )

The training set is used to fit the model parameters, while the test set is held out and used to evaluate how well the trained model generalises to unseen data.

Training the model

I used logistic regression as a simple baseline model. Before training, I standardised the features because some had very different numerical scales, which can affect the model’s optimisation process.

model = make_pipeline(
 StandardScaler(), 
 LogisticRegression(max_iter=1000) )
model.fit(X_train, y_train.values.ravel())

Probability and prediction

The model first produces a probability for class 1. For example, 0.63 means the model estimates a 63% chance that the example belongs to the positive class. A threshold is then used to convert that probability into the final prediction, either 0or 1.

y_prob = model.predict_proba(X_test)[:, 1]

This returns the model's estimated probability for class 1 for each example in the test set

y_pred = (y_prob >= 0.5).astype(int)

In the above code block, I used a threshold of 0.5. Probabilities below 0.5 are classified as 0, while probabilities of 0.5 or above are classified as 1.

Evaluating the predictions

I compared the model's predictions with the actual test labels using a confusion matrix.

cm = confusion_matrix(y_test, y_pred)
tn, fp, fn, tp = cm.ravel().  # tn stands for true negatives, fp stands for false positives and so on ..

This makes it easier to see the types of errors the model is making, rather than only looking at overall accuracy which I will also calculate.

From these values, I calculated sensitivity and specificity:

sensitivity = tp / (tp + fn)
specificity = tn / (tn + fp)

Sensitivity measures how well the model identifies actual positive cases, while specificity measures how well it identifies actual negative cases.

Changing the threshold

On changing the threshold the following observations were made:

Lowering the threshold increased sensitivity because the model classified more examples as positive. At a threshold of 0.30, it detected about 95% of the actual positive cases, but specificity dropped to about 35%, meaning many negative cases were also classified as positive.

Raising the threshold had the opposite effect. At 0.70, specificity increased to about 96%, but sensitivity dropped to about 41%, meaning more actual positive cases were missed.

The model itself did not change during these tests. Only the threshold used to convert its probability estimates into class predictions was changed.

Key Take out

The main thing I learned is that a classifier first produces a probability estimate, and a threshold is then used to convert that probability into a class prediction.

Changing the threshold changes the balance between sensitivity and specificity. This is especially important in healthcare because false positives and false negatives can have different consequences.

Accuracy alone therefore does not fully describe how a model behaves. Metrics such as sensitivity and specificity help show what types of cases the model is correctly identifying or missing.

Code - The Jupyter notebook for this exercise is available on GitHub

References

Antal, B. & Hajdu, A. (2014). Diabetic Retinopathy Debrecen Dataset. UCI Machine Learning Repository. DOI: 10.24432/C5XP4P. The dataset . Accessed on 17th August, 2026.