# Classification on imbalanced data > 原文：[https://tensorflow.google.cn/tutorials/structured_data/imbalanced_data](https://tensorflow.google.cn/tutorials/structured_data/imbalanced_data) This tutorial demonstrates how to classify a highly imbalanced dataset in which the number of examples in one class greatly outnumbers the examples in another. You will work with the [Credit Card Fraud Detection](https://www.kaggle.com/mlg-ulb/creditcardfraud) dataset hosted on Kaggle. The aim is to detect a mere 492 fraudulent transactions from 284,807 transactions in total. You will use [Keras](https://tensorflow.google.cn/guide/keras/overview) to define the model and [class weights](https://tensorflow.google.cn/versions/r2.0/api_docs/python/tf/keras/Model) to help the model learn from the imbalanced data. . This tutorial contains complete code to: * Load a CSV file using Pandas. * Create train, validation, and test sets. * Define and train a model using Keras (including setting class weights). * Evaluate the model using various metrics (including precision and recall). * Try common techniques for dealing with imbalanced data like: * Class weighting * Oversampling ## Setup ```py import tensorflow as tf from tensorflow import keras import os import tempfile import matplotlib as mpl import matplotlib.pyplot as plt import numpy as np import pandas as pd import seaborn as sns import sklearn from sklearn.metrics import confusion_matrix from sklearn.model_selection import train_test_split from sklearn.preprocessing import StandardScaler ``` ```py mpl.rcParams['figure.figsize'] = (12, 10) colors = plt.rcParams['axes.prop_cycle'].by_key()['color'] ``` ## Data processing and exploration ### Download the Kaggle Credit Card Fraud data set Pandas is a Python library with many helpful utilities for loading and working with structured data and can be used to download CSVs into a dataframe. **Note:** This dataset has been collected and analysed during a research collaboration of Worldline and the [Machine Learning Group](http://mlg.ulb.ac.be) of ULB (Université Libre de Bruxelles) on big data mining and fraud detection. More details on current and past projects on related topics are available [here](https://www.researchgate.net/project/Fraud-detection-5) and the page of the [DefeatFraud](https://mlg.ulb.ac.be/wordpress/portfolio_page/defeatfraud-assessment-and-validation-of-deep-feature-engineering-and-learning-solutions-for-fraud-detection/) project ```py file = tf.keras.utils raw_df = pd.read_csv('https://storage.googleapis.com/download.tensorflow.org/data/creditcard.csv') raw_df.head() ``` ```py raw_df[['Time', 'V1', 'V2', 'V3', 'V4', 'V5', 'V26', 'V27', 'V28', 'Amount', 'Class']].describe() ``` ### Examine the class label imbalance Let's look at the dataset imbalance: ```py neg, pos = np.bincount(raw_df['Class']) total = neg + pos print('Examples:\n Total: {}\n Positive: {} ({:.2f}% of total)\n'.format( total, pos, 100 * pos / total)) ``` ```py Examples: Total: 284807 Positive: 492 (0.17% of total) ``` This shows the small fraction of positive samples. ### Clean, split and normalize the data The raw data has a few issues. First the `Time` and `Amount` columns are too variable to use directly. Drop the `Time` column (since it's not clear what it means) and take the log of the `Amount` column to reduce its range. ```py cleaned_df = raw_df.copy() # You don't want the `Time` column. cleaned_df.pop('Time') # The `Amount` column covers a huge range. Convert to log-space. eps = 0.001 # 0 => 0.1¢ cleaned_df['Log Ammount'] = np.log(cleaned_df.pop('Amount')+eps) ``` Split the dataset into train, validation, and test sets. The validation set is used during the model fitting to evaluate the loss and any metrics, however the model is not fit with this data. The test set is completely unused during the training phase and is only used at the end to evaluate how well the model generalizes to new data. This is especially important with imbalanced datasets where [overfitting](https://developers.google.cn/machine-learning/crash-course/generalization/peril-of-overfitting) is a significant concern from the lack of training data. ```py # Use a utility from sklearn to split and shuffle our dataset. train_df, test_df = train_test_split(cleaned_df, test_size=0.2) train_df, val_df = train_test_split(train_df, test_size=0.2) # Form np arrays of labels and features. train_labels = np.array(train_df.pop('Class')) bool_train_labels = train_labels != 0 val_labels = np.array(val_df.pop('Class')) test_labels = np.array(test_df.pop('Class')) train_features = np.array(train_df) val_features = np.array(val_df) test_features = np.array(test_df) ``` Normalize the input features using the sklearn StandardScaler. This will set the mean to 0 and standard deviation to 1. **Note:** The `StandardScaler` is only fit using the `train_features` to be sure the model is not peeking at the validation or test sets. ```py scaler = StandardScaler() train_features = scaler.fit_transform(train_features) val_features = scaler.transform(val_features) test_features = scaler.transform(test_features) train_features = np.clip(train_features, -5, 5) val_features = np.clip(val_features, -5, 5) test_features = np.clip(test_features, -5, 5) print('Training labels shape:', train_labels.shape) print('Validation labels shape:', val_labels.shape) print('Test labels shape:', test_labels.shape) print('Training features shape:', train_features.shape) print('Validation features shape:', val_features.shape) print('Test features shape:', test_features.shape) ``` ```py Training labels shape: (182276,) Validation labels shape: (45569,) Test labels shape: (56962,) Training features shape: (182276, 29) Validation features shape: (45569, 29) Test features shape: (56962, 29) ``` **Caution:** If you want to deploy a model, it's critical that you preserve the preprocessing calculations. The easiest way to implement them as layers, and attach them to your model before export. ### Look at the data distribution Next compare the distributions of the positive and negative examples over a few features. Good questions to ask yourself at this point are: * Do these distributions make sense? * Yes. You've normalized the input and these are mostly concentrated in the `+/- 2` range. * Can you see the difference between the distributions? * Yes the positive examples contain a much higher rate of extreme values. ```py pos_df = pd.DataFrame(train_features[ bool_train_labels], columns=train_df.columns) neg_df = pd.DataFrame(train_features[~bool_train_labels], columns=train_df.columns) sns.jointplot(pos_df['V5'], pos_df['V6'], kind='hex', xlim=(-5,5), ylim=(-5,5)) plt.suptitle("Positive distribution") sns.jointplot(neg_df['V5'], neg_df['V6'], kind='hex', xlim=(-5,5), ylim=(-5,5)) _ = plt.suptitle("Negative distribution") ``` ```py /home/kbuilder/.local/lib/python3.6/site-packages/seaborn/_decorators.py:43: FutureWarning: Pass the following variables as keyword args: x, y. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation. FutureWarning /home/kbuilder/.local/lib/python3.6/site-packages/seaborn/_decorators.py:43: FutureWarning: Pass the following variables as keyword args: x, y. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation. FutureWarning ``` ![png](img/72de951dda8439ddee9fe72a64a9ed2e.png) ![png](img/67c5e6fec9fd9b11f792765822fbb538.png) ## Define the model and metrics Define a function that creates a simple neural network with a densly connected hidden layer, a [dropout](https://developers.google.cn/machine-learning/glossary/#dropout_regularization) layer to reduce overfitting, and an output sigmoid layer that returns the probability of a transaction being fraudulent: ```py METRICS = [ keras.metrics.TruePositives(name='tp'), keras.metrics.FalsePositives(name='fp'), keras.metrics.TrueNegatives(name='tn'), keras.metrics.FalseNegatives(name='fn'), keras.metrics.BinaryAccuracy(name='accuracy'), keras.metrics.Precision(name='precision'), keras.metrics.Recall(name='recall'), keras.metrics.AUC(name='auc'), ] def make_model(metrics=METRICS, output_bias=None): if output_bias is not None: output_bias = tf.keras.initializers.Constant(output_bias) model = keras.Sequential([ keras.layers.Dense( 16, activation='relu', input_shape=(train_features.shape[-1],)), keras.layers.Dropout(0.5), keras.layers.Dense(1, activation='sigmoid', bias_initializer=output_bias), ]) model.compile( optimizer=keras.optimizers.Adam(lr=1e-3), loss=keras.losses.BinaryCrossentropy(), metrics=metrics) return model ``` ### Understanding useful metrics Notice that there are a few metrics defined above that can be computed by the model that will be helpful when evaluating the performance. * **False** negatives and **false** positives are samples that were **incorrectly** classified * **True** negatives and **true** positives are samples that were **correctly** classified * **Accuracy** is the percentage of examples correctly classified > $\frac{\text{true samples} }{\text{total samples} }$ * **Precision** is the percentage of **predicted** positives that were correctly classified > $\frac{\text{true positives} }{\text{true positives + false positives} }$ * **Recall** is the percentage of **actual** positives that were correctly classified > $\frac{\text{true positives} }{\text{true positives + false negatives} }$ * **AUC** refers to the Area Under the Curve of a Receiver Operating Characteristic curve (ROC-AUC). This metric is equal to the probability that a classifier will rank a random positive sample higher than a random negative sample. **Note:** Accuracy is not a helpful metric for this task. You can 99.8%+ accuracy on this task by predicting False all the time. Read more: * [True vs. False and Positive vs. Negative](https://developers.google.cn/machine-learning/crash-course/classification/true-false-positive-negative) * [Accuracy](https://developers.google.cn/machine-learning/crash-course/classification/accuracy) * [Precision and Recall](https://developers.google.cn/machine-learning/crash-course/classification/precision-and-recall) * [ROC-AUC](https://developers.google.cn/machine-learning/crash-course/classification/roc-and-auc) ## Baseline model ### Build the model Now create and train your model using the function that was defined earlier. Notice that the model is fit using a larger than default batch size of 2048, this is important to ensure that each batch has a decent chance of containing a few positive samples. If the batch size was too small, they would likely have no fraudulent transactions to learn from. **Note:** this model will not handle the class imbalance well. You will improve it later in this tutorial. ```py EPOCHS = 100 BATCH_SIZE = 2048 early_stopping = tf.keras.callbacks.EarlyStopping( monitor='val_auc', verbose=1, patience=10, mode='max', restore_best_weights=True) ``` ```py model = make_model() model.summary() ``` ```py Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= dense (Dense) (None, 16) 480 _________________________________________________________________ dropout (Dropout) (None, 16) 0 _________________________________________________________________ dense_1 (Dense) (None, 1) 17 ================================================================= Total params: 497 Trainable params: 497 Non-trainable params: 0 _________________________________________________________________ ``` Test run the model: ```py model.predict(train_features[:10]) ``` ```py array([[0.14155667], [0.08245954], [0.14218365], [0.09784216], [0.15547438], [0.04696527], [0.20200476], [0.16137381], [0.08730176], [0.04467227]], dtype=float32) ``` ### Optional: Set the correct initial bias. These initial guesses are not great. You know the dataset is imbalanced. Set the output layer's bias to reflect that (See: [A Recipe for Training Neural Networks: "init well"](http://karpathy.github.io/2019/04/25/recipe/#2-set-up-the-end-to-end-trainingevaluation-skeleton--get-dumb-baselines)). This can help with initial convergence. With the default bias initialization the loss should be about [`math.log(2) = 0.69314`](https://tensorflow.google.cn/api_docs/python/tf/math/log) ```py results = model.evaluate(train_features, train_labels, batch_size=BATCH_SIZE, verbose=0) print("Loss: {:0.4f}".format(results[0])) ``` ```py Loss: 0.1159 ``` The correct bias to set can be derived from: $$ p_0 = pos/(pos + neg) = 1/(1+e^{-b_0}) $$$$ b_0 = -log_e(1/p_0 - 1) $$$$ b_0 = log_e(pos/neg)$$ ```py initial_bias = np.log([pos/neg]) initial_bias ``` ```py array([-6.35935934]) ``` Set that as the initial bias, and the model will give much more reasonable initial guesses. It should be near: `pos/total = 0.0018` ```py model = make_model(output_bias=initial_bias) model.predict(train_features[:10]) ``` ```py array([[0.00172629], [0.00338373], [0.00397264], [0.00406079], [0.00449285], [0.00492807], [0.00254243], [0.00370109], [0.01771316], [0.0022068 ]], dtype=float32) ``` With this initialization the initial loss should be approximately: $$-p_0log(p_0)-(1-p_0)log(1-p_0) = 0.01317$$ ```py results = model.evaluate(train_features, train_labels, batch_size=BATCH_SIZE, verbose=0) print("Loss: {:0.4f}".format(results[0])) ``` ```py Loss: 0.0141 ``` This initial loss is about 50 times less than if would have been with naive initialization. This way the model doesn't need to spend the first few epochs just learning that positive examples are unlikely. This also makes it easier to read plots of the loss during training. ### Checkpoint the initial weights To make the various training runs more comparable, keep this initial model's weights in a checkpoint file, and load them into each model before training. ```py initial_weights = os.path.join(tempfile.mkdtemp(), 'initial_weights') model.save_weights(initial_weights) ``` ### Confirm that the bias fix helps Before moving on, confirm quick that the careful bias initialization actually helped. Train the model for 20 epochs, with and without this careful initialization, and compare the losses: ```py model = make_model() model.load_weights(initial_weights) model.layers[-1].bias.assign([0.0]) zero_bias_history = model.fit( train_features, train_labels, batch_size=BATCH_SIZE, epochs=20, validation_data=(val_features, val_labels), verbose=0) ``` ```py model = make_model() model.load_weights(initial_weights) careful_bias_history = model.fit( train_features, train_labels, batch_size=BATCH_SIZE, epochs=20, validation_data=(val_features, val_labels), verbose=0) ``` ```py def plot_loss(history, label, n): # Use a log scale to show the wide range of values. plt.semilogy(history.epoch, history.history['loss'], color=colors[n], label='Train '+label) plt.semilogy(history.epoch, history.history['val_loss'], color=colors[n], label='Val '+label, linestyle="--") plt.xlabel('Epoch') plt.ylabel('Loss') plt.legend() ``` ```py plot_loss(zero_bias_history, "Zero Bias", 0) plot_loss(careful_bias_history, "Careful Bias", 1) ``` ![png](img/9ec1c5121631fabdd2734d5a380b07fc.png) The above figure makes it clear: In terms of validation loss, on this problem, this careful initialization gives a clear advantage. ### Train the model ```py model = make_model() model.load_weights(initial_weights) baseline_history = model.fit( train_features, train_labels, batch_size=BATCH_SIZE, epochs=EPOCHS, callbacks=[early_stopping], validation_data=(val_features, val_labels)) ``` ```py Epoch 1/100 90/90 [==============================] - 1s 14ms/step - loss: 0.0127 - tp: 93.0000 - fp: 90.0000 - tn: 227362.0000 - fn: 300.0000 - accuracy: 0.9983 - precision: 0.5082 - recall: 0.2366 - auc: 0.8002 - val_loss: 0.0066 - val_tp: 8.0000 - val_fp: 4.0000 - val_tn: 45489.0000 - val_fn: 68.0000 - val_accuracy: 0.9984 - val_precision: 0.6667 - val_recall: 0.1053 - val_auc: 0.9394 Epoch 2/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0081 - tp: 99.0000 - fp: 25.0000 - tn: 181934.0000 - fn: 218.0000 - accuracy: 0.9987 - precision: 0.7984 - recall: 0.3123 - auc: 0.8446 - val_loss: 0.0044 - val_tp: 40.0000 - val_fp: 6.0000 - val_tn: 45487.0000 - val_fn: 36.0000 - val_accuracy: 0.9991 - val_precision: 0.8696 - val_recall: 0.5263 - val_auc: 0.9471 Epoch 3/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0067 - tp: 147.0000 - fp: 29.0000 - tn: 181930.0000 - fn: 170.0000 - accuracy: 0.9989 - precision: 0.8352 - recall: 0.4637 - auc: 0.8739 - val_loss: 0.0036 - val_tp: 45.0000 - val_fp: 6.0000 - val_tn: 45487.0000 - val_fn: 31.0000 - val_accuracy: 0.9992 - val_precision: 0.8824 - val_recall: 0.5921 - val_auc: 0.9472 Epoch 4/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0061 - tp: 151.0000 - fp: 32.0000 - tn: 181927.0000 - fn: 166.0000 - accuracy: 0.9989 - precision: 0.8251 - recall: 0.4763 - auc: 0.8867 - val_loss: 0.0031 - val_tp: 50.0000 - val_fp: 6.0000 - val_tn: 45487.0000 - val_fn: 26.0000 - val_accuracy: 0.9993 - val_precision: 0.8929 - val_recall: 0.6579 - val_auc: 0.9472 Epoch 5/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0059 - tp: 162.0000 - fp: 35.0000 - tn: 181924.0000 - fn: 155.0000 - accuracy: 0.9990 - precision: 0.8223 - recall: 0.5110 - auc: 0.8970 - val_loss: 0.0028 - val_tp: 53.0000 - val_fp: 6.0000 - val_tn: 45487.0000 - val_fn: 23.0000 - val_accuracy: 0.9994 - val_precision: 0.8983 - val_recall: 0.6974 - val_auc: 0.9538 Epoch 6/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0056 - tp: 167.0000 - fp: 34.0000 - tn: 181925.0000 - fn: 150.0000 - accuracy: 0.9990 - precision: 0.8308 - recall: 0.5268 - auc: 0.8942 - val_loss: 0.0026 - val_tp: 53.0000 - val_fp: 6.0000 - val_tn: 45487.0000 - val_fn: 23.0000 - val_accuracy: 0.9994 - val_precision: 0.8983 - val_recall: 0.6974 - val_auc: 0.9670 Epoch 7/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0051 - tp: 179.0000 - fp: 32.0000 - tn: 181927.0000 - fn: 138.0000 - accuracy: 0.9991 - precision: 0.8483 - recall: 0.5647 - auc: 0.9023 - val_loss: 0.0024 - val_tp: 58.0000 - val_fp: 6.0000 - val_tn: 45487.0000 - val_fn: 18.0000 - val_accuracy: 0.9995 - val_precision: 0.9062 - val_recall: 0.7632 - val_auc: 0.9669 Epoch 8/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0049 - tp: 181.0000 - fp: 29.0000 - tn: 181930.0000 - fn: 136.0000 - accuracy: 0.9991 - precision: 0.8619 - recall: 0.5710 - auc: 0.9040 - val_loss: 0.0023 - val_tp: 59.0000 - val_fp: 6.0000 - val_tn: 45487.0000 - val_fn: 17.0000 - val_accuracy: 0.9995 - val_precision: 0.9077 - val_recall: 0.7763 - val_auc: 0.9735 Epoch 9/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0050 - tp: 181.0000 - fp: 36.0000 - tn: 181923.0000 - fn: 136.0000 - accuracy: 0.9991 - precision: 0.8341 - recall: 0.5710 - auc: 0.9025 - val_loss: 0.0022 - val_tp: 56.0000 - val_fp: 6.0000 - val_tn: 45487.0000 - val_fn: 20.0000 - val_accuracy: 0.9994 - val_precision: 0.9032 - val_recall: 0.7368 - val_auc: 0.9735 Epoch 10/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0048 - tp: 179.0000 - fp: 32.0000 - tn: 181927.0000 - fn: 138.0000 - accuracy: 0.9991 - precision: 0.8483 - recall: 0.5647 - auc: 0.9041 - val_loss: 0.0021 - val_tp: 57.0000 - val_fp: 6.0000 - val_tn: 45487.0000 - val_fn: 19.0000 - val_accuracy: 0.9995 - val_precision: 0.9048 - val_recall: 0.7500 - val_auc: 0.9735 Epoch 11/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0044 - tp: 188.0000 - fp: 29.0000 - tn: 181930.0000 - fn: 129.0000 - accuracy: 0.9991 - precision: 0.8664 - recall: 0.5931 - auc: 0.9246 - val_loss: 0.0020 - val_tp: 57.0000 - val_fp: 6.0000 - val_tn: 45487.0000 - val_fn: 19.0000 - val_accuracy: 0.9995 - val_precision: 0.9048 - val_recall: 0.7500 - val_auc: 0.9735 Epoch 12/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0047 - tp: 168.0000 - fp: 34.0000 - tn: 181925.0000 - fn: 149.0000 - accuracy: 0.9990 - precision: 0.8317 - recall: 0.5300 - auc: 0.9184 - val_loss: 0.0019 - val_tp: 63.0000 - val_fp: 6.0000 - val_tn: 45487.0000 - val_fn: 13.0000 - val_accuracy: 0.9996 - val_precision: 0.9130 - val_recall: 0.8289 - val_auc: 0.9735 Epoch 13/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0045 - tp: 186.0000 - fp: 32.0000 - tn: 181927.0000 - fn: 131.0000 - accuracy: 0.9991 - precision: 0.8532 - recall: 0.5868 - auc: 0.9105 - val_loss: 0.0019 - val_tp: 63.0000 - val_fp: 6.0000 - val_tn: 45487.0000 - val_fn: 13.0000 - val_accuracy: 0.9996 - val_precision: 0.9130 - val_recall: 0.8289 - val_auc: 0.9735 Epoch 14/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0043 - tp: 199.0000 - fp: 37.0000 - tn: 181922.0000 - fn: 118.0000 - accuracy: 0.9991 - precision: 0.8432 - recall: 0.6278 - auc: 0.9217 - val_loss: 0.0019 - val_tp: 59.0000 - val_fp: 6.0000 - val_tn: 45487.0000 - val_fn: 17.0000 - val_accuracy: 0.9995 - val_precision: 0.9077 - val_recall: 0.7763 - val_auc: 0.9735 Epoch 15/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0045 - tp: 180.0000 - fp: 28.0000 - tn: 181931.0000 - fn: 137.0000 - accuracy: 0.9991 - precision: 0.8654 - recall: 0.5678 - auc: 0.9216 - val_loss: 0.0019 - val_tp: 63.0000 - val_fp: 6.0000 - val_tn: 45487.0000 - val_fn: 13.0000 - val_accuracy: 0.9996 - val_precision: 0.9130 - val_recall: 0.8289 - val_auc: 0.9801 Epoch 16/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0044 - tp: 188.0000 - fp: 41.0000 - tn: 181918.0000 - fn: 129.0000 - accuracy: 0.9991 - precision: 0.8210 - recall: 0.5931 - auc: 0.9200 - val_loss: 0.0019 - val_tp: 52.0000 - val_fp: 3.0000 - val_tn: 45490.0000 - val_fn: 24.0000 - val_accuracy: 0.9994 - val_precision: 0.9455 - val_recall: 0.6842 - val_auc: 0.9735 Epoch 17/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0045 - tp: 178.0000 - fp: 29.0000 - tn: 181930.0000 - fn: 139.0000 - accuracy: 0.9991 - precision: 0.8599 - recall: 0.5615 - auc: 0.9153 - val_loss: 0.0018 - val_tp: 66.0000 - val_fp: 6.0000 - val_tn: 45487.0000 - val_fn: 10.0000 - val_accuracy: 0.9996 - val_precision: 0.9167 - val_recall: 0.8684 - val_auc: 0.9801 Epoch 18/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0043 - tp: 186.0000 - fp: 33.0000 - tn: 181926.0000 - fn: 131.0000 - accuracy: 0.9991 - precision: 0.8493 - recall: 0.5868 - auc: 0.9248 - val_loss: 0.0018 - val_tp: 65.0000 - val_fp: 6.0000 - val_tn: 45487.0000 - val_fn: 11.0000 - val_accuracy: 0.9996 - val_precision: 0.9155 - val_recall: 0.8553 - val_auc: 0.9735 Epoch 19/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0044 - tp: 188.0000 - fp: 28.0000 - tn: 181931.0000 - fn: 129.0000 - accuracy: 0.9991 - precision: 0.8704 - recall: 0.5931 - auc: 0.9091 - val_loss: 0.0018 - val_tp: 66.0000 - val_fp: 6.0000 - val_tn: 45487.0000 - val_fn: 10.0000 - val_accuracy: 0.9996 - val_precision: 0.9167 - val_recall: 0.8684 - val_auc: 0.9801 Epoch 20/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0042 - tp: 189.0000 - fp: 30.0000 - tn: 181929.0000 - fn: 128.0000 - accuracy: 0.9991 - precision: 0.8630 - recall: 0.5962 - auc: 0.9249 - val_loss: 0.0018 - val_tp: 63.0000 - val_fp: 5.0000 - val_tn: 45488.0000 - val_fn: 13.0000 - val_accuracy: 0.9996 - val_precision: 0.9265 - val_recall: 0.8289 - val_auc: 0.9735 Epoch 21/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0042 - tp: 185.0000 - fp: 35.0000 - tn: 181924.0000 - fn: 132.0000 - accuracy: 0.9991 - precision: 0.8409 - recall: 0.5836 - auc: 0.9248 - val_loss: 0.0017 - val_tp: 66.0000 - val_fp: 6.0000 - val_tn: 45487.0000 - val_fn: 10.0000 - val_accuracy: 0.9996 - val_precision: 0.9167 - val_recall: 0.8684 - val_auc: 0.9801 Epoch 22/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0042 - tp: 188.0000 - fp: 31.0000 - tn: 181928.0000 - fn: 129.0000 - accuracy: 0.9991 - precision: 0.8584 - recall: 0.5931 - auc: 0.9249 - val_loss: 0.0017 - val_tp: 64.0000 - val_fp: 5.0000 - val_tn: 45488.0000 - val_fn: 12.0000 - val_accuracy: 0.9996 - val_precision: 0.9275 - val_recall: 0.8421 - val_auc: 0.9801 Epoch 23/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0045 - tp: 175.0000 - fp: 39.0000 - tn: 181920.0000 - fn: 142.0000 - accuracy: 0.9990 - precision: 0.8178 - recall: 0.5521 - auc: 0.9169 - val_loss: 0.0017 - val_tp: 65.0000 - val_fp: 6.0000 - val_tn: 45487.0000 - val_fn: 11.0000 - val_accuracy: 0.9996 - val_precision: 0.9155 - val_recall: 0.8553 - val_auc: 0.9801 Epoch 24/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0043 - tp: 188.0000 - fp: 31.0000 - tn: 181928.0000 - fn: 129.0000 - accuracy: 0.9991 - precision: 0.8584 - recall: 0.5931 - auc: 0.9122 - val_loss: 0.0017 - val_tp: 64.0000 - val_fp: 6.0000 - val_tn: 45487.0000 - val_fn: 12.0000 - val_accuracy: 0.9996 - val_precision: 0.9143 - val_recall: 0.8421 - val_auc: 0.9801 Epoch 25/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0043 - tp: 179.0000 - fp: 29.0000 - tn: 181930.0000 - fn: 138.0000 - accuracy: 0.9991 - precision: 0.8606 - recall: 0.5647 - auc: 0.9311 - val_loss: 0.0017 - val_tp: 64.0000 - val_fp: 6.0000 - val_tn: 45487.0000 - val_fn: 12.0000 - val_accuracy: 0.9996 - val_precision: 0.9143 - val_recall: 0.8421 - val_auc: 0.9801 Epoch 26/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0041 - tp: 189.0000 - fp: 32.0000 - tn: 181927.0000 - fn: 128.0000 - accuracy: 0.9991 - precision: 0.8552 - recall: 0.5962 - auc: 0.9218 - val_loss: 0.0017 - val_tp: 66.0000 - val_fp: 6.0000 - val_tn: 45487.0000 - val_fn: 10.0000 - val_accuracy: 0.9996 - val_precision: 0.9167 - val_recall: 0.8684 - val_auc: 0.9801 Epoch 27/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0040 - tp: 193.0000 - fp: 27.0000 - tn: 181932.0000 - fn: 124.0000 - accuracy: 0.9992 - precision: 0.8773 - recall: 0.6088 - auc: 0.9202 - val_loss: 0.0017 - val_tp: 66.0000 - val_fp: 6.0000 - val_tn: 45487.0000 - val_fn: 10.0000 - val_accuracy: 0.9996 - val_precision: 0.9167 - val_recall: 0.8684 - val_auc: 0.9801 Epoch 28/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0041 - tp: 189.0000 - fp: 31.0000 - tn: 181928.0000 - fn: 128.0000 - accuracy: 0.9991 - precision: 0.8591 - recall: 0.5962 - auc: 0.9187 - val_loss: 0.0017 - val_tp: 62.0000 - val_fp: 3.0000 - val_tn: 45490.0000 - val_fn: 14.0000 - val_accuracy: 0.9996 - val_precision: 0.9538 - val_recall: 0.8158 - val_auc: 0.9801 Epoch 29/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0041 - tp: 173.0000 - fp: 35.0000 - tn: 181924.0000 - fn: 144.0000 - accuracy: 0.9990 - precision: 0.8317 - recall: 0.5457 - auc: 0.9233 - val_loss: 0.0017 - val_tp: 64.0000 - val_fp: 4.0000 - val_tn: 45489.0000 - val_fn: 12.0000 - val_accuracy: 0.9996 - val_precision: 0.9412 - val_recall: 0.8421 - val_auc: 0.9801 Epoch 30/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0041 - tp: 193.0000 - fp: 36.0000 - tn: 181923.0000 - fn: 124.0000 - accuracy: 0.9991 - precision: 0.8428 - recall: 0.6088 - auc: 0.9218 - val_loss: 0.0017 - val_tp: 62.0000 - val_fp: 3.0000 - val_tn: 45490.0000 - val_fn: 14.0000 - val_accuracy: 0.9996 - val_precision: 0.9538 - val_recall: 0.8158 - val_auc: 0.9801 Epoch 31/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0044 - tp: 190.0000 - fp: 35.0000 - tn: 181924.0000 - fn: 127.0000 - accuracy: 0.9991 - precision: 0.8444 - recall: 0.5994 - auc: 0.9122 - val_loss: 0.0017 - val_tp: 60.0000 - val_fp: 3.0000 - val_tn: 45490.0000 - val_fn: 16.0000 - val_accuracy: 0.9996 - val_precision: 0.9524 - val_recall: 0.7895 - val_auc: 0.9801 Epoch 32/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0043 - tp: 184.0000 - fp: 33.0000 - tn: 181926.0000 - fn: 133.0000 - accuracy: 0.9991 - precision: 0.8479 - recall: 0.5804 - auc: 0.9186 - val_loss: 0.0017 - val_tp: 61.0000 - val_fp: 3.0000 - val_tn: 45490.0000 - val_fn: 15.0000 - val_accuracy: 0.9996 - val_precision: 0.9531 - val_recall: 0.8026 - val_auc: 0.9801 Epoch 33/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0041 - tp: 188.0000 - fp: 36.0000 - tn: 181923.0000 - fn: 129.0000 - accuracy: 0.9991 - precision: 0.8393 - recall: 0.5931 - auc: 0.9218 - val_loss: 0.0016 - val_tp: 62.0000 - val_fp: 3.0000 - val_tn: 45490.0000 - val_fn: 14.0000 - val_accuracy: 0.9996 - val_precision: 0.9538 - val_recall: 0.8158 - val_auc: 0.9801 Epoch 34/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0039 - tp: 196.0000 - fp: 37.0000 - tn: 181922.0000 - fn: 121.0000 - accuracy: 0.9991 - precision: 0.8412 - recall: 0.6183 - auc: 0.9297 - val_loss: 0.0017 - val_tp: 62.0000 - val_fp: 3.0000 - val_tn: 45490.0000 - val_fn: 14.0000 - val_accuracy: 0.9996 - val_precision: 0.9538 - val_recall: 0.8158 - val_auc: 0.9801 Epoch 35/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0040 - tp: 189.0000 - fp: 39.0000 - tn: 181920.0000 - fn: 128.0000 - accuracy: 0.9991 - precision: 0.8289 - recall: 0.5962 - auc: 0.9281 - val_loss: 0.0017 - val_tp: 60.0000 - val_fp: 3.0000 - val_tn: 45490.0000 - val_fn: 16.0000 - val_accuracy: 0.9996 - val_precision: 0.9524 - val_recall: 0.7895 - val_auc: 0.9801 Epoch 36/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0039 - tp: 197.0000 - fp: 29.0000 - tn: 181930.0000 - fn: 120.0000 - accuracy: 0.9992 - precision: 0.8717 - recall: 0.6215 - auc: 0.9203 - val_loss: 0.0016 - val_tp: 65.0000 - val_fp: 3.0000 - val_tn: 45490.0000 - val_fn: 11.0000 - val_accuracy: 0.9997 - val_precision: 0.9559 - val_recall: 0.8553 - val_auc: 0.9801 Epoch 37/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0039 - tp: 195.0000 - fp: 26.0000 - tn: 181933.0000 - fn: 122.0000 - accuracy: 0.9992 - precision: 0.8824 - recall: 0.6151 - auc: 0.9234 - val_loss: 0.0016 - val_tp: 62.0000 - val_fp: 3.0000 - val_tn: 45490.0000 - val_fn: 14.0000 - val_accuracy: 0.9996 - val_precision: 0.9538 - val_recall: 0.8158 - val_auc: 0.9801 Epoch 38/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0038 - tp: 197.0000 - fp: 40.0000 - tn: 181919.0000 - fn: 120.0000 - accuracy: 0.9991 - precision: 0.8312 - recall: 0.6215 - auc: 0.9329 - val_loss: 0.0017 - val_tp: 59.0000 - val_fp: 3.0000 - val_tn: 45490.0000 - val_fn: 17.0000 - val_accuracy: 0.9996 - val_precision: 0.9516 - val_recall: 0.7763 - val_auc: 0.9801 Epoch 39/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0040 - tp: 195.0000 - fp: 34.0000 - tn: 181925.0000 - fn: 122.0000 - accuracy: 0.9991 - precision: 0.8515 - recall: 0.6151 - auc: 0.9343 - val_loss: 0.0016 - val_tp: 62.0000 - val_fp: 3.0000 - val_tn: 45490.0000 - val_fn: 14.0000 - val_accuracy: 0.9996 - val_precision: 0.9538 - val_recall: 0.8158 - val_auc: 0.9801 Epoch 40/100 90/90 [==============================] - 1s 6ms/step - loss: 0.0039 - tp: 199.0000 - fp: 35.0000 - tn: 181924.0000 - fn: 118.0000 - accuracy: 0.9992 - precision: 0.8504 - recall: 0.6278 - auc: 0.9250 - val_loss: 0.0016 - val_tp: 60.0000 - val_fp: 3.0000 - val_tn: 45490.0000 - val_fn: 16.0000 - val_accuracy: 0.9996 - val_precision: 0.9524 - val_recall: 0.7895 - val_auc: 0.9801 Epoch 41/100 88/90 [============================>.] - ETA: 0s - loss: 0.0041 - tp: 185.0000 - fp: 35.0000 - tn: 179875.0000 - fn: 129.0000 - accuracy: 0.9991 - precision: 0.8409 - recall: 0.5892 - auc: 0.9305Restoring model weights from the end of the best epoch. 90/90 [==============================] - 1s 6ms/step - loss: 0.0041 - tp: 186.0000 - fp: 35.0000 - tn: 181924.0000 - fn: 131.0000 - accuracy: 0.9991 - precision: 0.8416 - recall: 0.5868 - auc: 0.9295 - val_loss: 0.0016 - val_tp: 65.0000 - val_fp: 5.0000 - val_tn: 45488.0000 - val_fn: 11.0000 - val_accuracy: 0.9996 - val_precision: 0.9286 - val_recall: 0.8553 - val_auc: 0.9801 Epoch 00041: early stopping ``` ### Check training history In this section, you will produce plots of your model's accuracy and loss on the training and validation set. These are useful to check for overfitting, which you can learn more about in this [tutorial](https://tensorflow.google.cn/tutorials/keras/overfit_and_underfit). Additionally, you can produce these plots for any of the metrics you created above. False negatives are included as an example. ```py def plot_metrics(history): metrics = ['loss', 'auc', 'precision', 'recall'] for n, metric in enumerate(metrics): name = metric.replace("_"," ").capitalize() plt.subplot(2,2,n+1) plt.plot(history.epoch, history.history[metric], color=colors[0], label='Train') plt.plot(history.epoch, history.history['val_'+metric], color=colors[0], linestyle="--", label='Val') plt.xlabel('Epoch') plt.ylabel(name) if metric == 'loss': plt.ylim([0, plt.ylim()[1]]) elif metric == 'auc': plt.ylim([0.8,1]) else: plt.ylim([0,1]) plt.legend() ``` ```py plot_metrics(baseline_history) ``` ![png](img/f021b204e92d0e77d8439a03a43bb21e.png) **Note:** That the validation curve generally performs better than the training curve. This is mainly caused by the fact that the dropout layer is not active when evaluating the model. ### Evaluate metrics You can use a [confusion matrix](https://developers.google.cn/machine-learning/glossary/#confusion_matrix) to summarize the actual vs. predicted labels where the X axis is the predicted label and the Y axis is the actual label. ```py train_predictions_baseline = model.predict(train_features, batch_size=BATCH_SIZE) test_predictions_baseline = model.predict(test_features, batch_size=BATCH_SIZE) ``` ```py def plot_cm(labels, predictions, p=0.5): cm = confusion_matrix(labels, predictions > p) plt.figure(figsize=(5,5)) sns.heatmap(cm, annot=True, fmt="d") plt.title('Confusion matrix @{:.2f}'.format(p)) plt.ylabel('Actual label') plt.xlabel('Predicted label') print('Legitimate Transactions Detected (True Negatives): ', cm[0][0]) print('Legitimate Transactions Incorrectly Detected (False Positives): ', cm[0][1]) print('Fraudulent Transactions Missed (False Negatives): ', cm[1][0]) print('Fraudulent Transactions Detected (True Positives): ', cm[1][1]) print('Total Fraudulent Transactions: ', np.sum(cm[1])) ``` Evaluate your model on the test dataset and display the results for the metrics you created above. ```py baseline_results = model.evaluate(test_features, test_labels, batch_size=BATCH_SIZE, verbose=0) for name, value in zip(model.metrics_names, baseline_results): print(name, ': ', value) print() plot_cm(test_labels, test_predictions_baseline) ``` ```py loss : 0.002797449706122279 tp : 68.0 fp : 3.0 tn : 56860.0 fn : 31.0 accuracy : 0.9994031190872192 precision : 0.9577465057373047 recall : 0.6868686676025391 auc : 0.949228823184967 Legitimate Transactions Detected (True Negatives): 56860 Legitimate Transactions Incorrectly Detected (False Positives): 3 Fraudulent Transactions Missed (False Negatives): 31 Fraudulent Transactions Detected (True Positives): 68 Total Fraudulent Transactions: 99 ``` ![png](img/85d63bf8a53bc6d25baa38c0e3e2dde0.png) If the model had predicted everything perfectly, this would be a [diagonal matrix](https://en.wikipedia.org/wiki/Diagonal_matrix) where values off the main diagonal, indicating incorrect predictions, would be zero. In this case the matrix shows that you have relatively few false positives, meaning that there were relatively few legitimate transactions that were incorrectly flagged. However, you would likely want to have even fewer false negatives despite the cost of increasing the number of false positives. This trade off may be preferable because false negatives would allow fraudulent transactions to go through, whereas false positives may cause an email to be sent to a customer to ask them to verify their card activity. ### Plot the ROC Now plot the [ROC](https://developers.google.cn/machine-learning/glossary#ROC). This plot is useful because it shows, at a glance, the range of performance the model can reach just by tuning the output threshold. ```py def plot_roc(name, labels, predictions, **kwargs): fp, tp, _ = sklearn.metrics.roc_curve(labels, predictions) plt.plot(100*fp, 100*tp, label=name, linewidth=2, **kwargs) plt.xlabel('False positives [%]') plt.ylabel('True positives [%]') plt.xlim([-0.5,20]) plt.ylim([80,100.5]) plt.grid(True) ax = plt.gca() ax.set_aspect('equal') ``` ```py plot_roc("Train Baseline", train_labels, train_predictions_baseline, color=colors[0]) plot_roc("Test Baseline", test_labels, test_predictions_baseline, color=colors[0], linestyle='--') plt.legend(loc='lower right') ``` ```py ``` ![png](img/74badb10696c0ffbba886121ce004be0.png) It looks like the precision is relatively high, but the recall and the area under the ROC curve (AUC) aren't as high as you might like. Classifiers often face challenges when trying to maximize both precision and recall, which is especially true when working with imbalanced datasets. It is important to consider the costs of different types of errors in the context of the problem you care about. In this example, a false negative (a fraudulent transaction is missed) may have a financial cost, while a false positive (a transaction is incorrectly flagged as fraudulent) may decrease user happiness. ## Class weights ### Calculate class weights The goal is to identify fraudulent transactions, but you don't have very many of those positive samples to work with, so you would want to have the classifier heavily weight the few examples that are available. You can do this by passing Keras weights for each class through a parameter. These will cause the model to "pay more attention" to examples from an under-represented class. ```py # Scaling by total/2 helps keep the loss to a similar magnitude. # The sum of the weights of all examples stays the same. weight_for_0 = (1 / neg)*(total)/2.0 weight_for_1 = (1 / pos)*(total)/2.0 class_weight = {0: weight_for_0, 1: weight_for_1} print('Weight for class 0: {:.2f}'.format(weight_for_0)) print('Weight for class 1: {:.2f}'.format(weight_for_1)) ``` ```py Weight for class 0: 0.50 Weight for class 1: 289.44 ``` ### Train a model with class weights Now try re-training and evaluating the model with class weights to see how that affects the predictions. **Note:** Using `class_weights` changes the range of the loss. This may affect the stability of the training depending on the optimizer. Optimizers whose step size is dependent on the magnitude of the gradient, like [`optimizers.SGD`](https://tensorflow.google.cn/api_docs/python/tf/keras/optimizers/SGD), may fail. The optimizer used here, [`optimizers.Adam`](https://tensorflow.google.cn/api_docs/python/tf/keras/optimizers/Adam), is unaffected by the scaling change. Also note that because of the weighting, the total losses are not comparable between the two models. ```py weighted_model = make_model() weighted_model.load_weights(initial_weights) weighted_history = weighted_model.fit( train_features, train_labels, batch_size=BATCH_SIZE, epochs=EPOCHS, callbacks=[early_stopping], validation_data=(val_features, val_labels), # The class weights go here class_weight=class_weight) ``` ```py Epoch 1/100 90/90 [==============================] - 1s 14ms/step - loss: 2.0102 - tp: 135.0000 - fp: 420.0000 - tn: 238402.0000 - fn: 281.0000 - accuracy: 0.9971 - precision: 0.2432 - recall: 0.3245 - auc: 0.8079 - val_loss: 0.0111 - val_tp: 45.0000 - val_fp: 51.0000 - val_tn: 45442.0000 - val_fn: 31.0000 - val_accuracy: 0.9982 - val_precision: 0.4688 - val_recall: 0.5921 - val_auc: 0.9314 Epoch 2/100 90/90 [==============================] - 1s 6ms/step - loss: 0.9256 - tp: 178.0000 - fp: 793.0000 - tn: 181166.0000 - fn: 139.0000 - accuracy: 0.9949 - precision: 0.1833 - recall: 0.5615 - auc: 0.8662 - val_loss: 0.0164 - val_tp: 66.0000 - val_fp: 108.0000 - val_tn: 45385.0000 - val_fn: 10.0000 - val_accuracy: 0.9974 - val_precision: 0.3793 - val_recall: 0.8684 - val_auc: 0.9468 Epoch 3/100 90/90 [==============================] - 1s 6ms/step - loss: 0.7136 - tp: 215.0000 - fp: 1377.0000 - tn: 180582.0000 - fn: 102.0000 - accuracy: 0.9919 - precision: 0.1351 - recall: 0.6782 - auc: 0.8811 - val_loss: 0.0238 - val_tp: 68.0000 - val_fp: 174.0000 - val_tn: 45319.0000 - val_fn: 8.0000 - val_accuracy: 0.9960 - val_precision: 0.2810 - val_recall: 0.8947 - val_auc: 0.9866 Epoch 4/100 90/90 [==============================] - 1s 6ms/step - loss: 0.5158 - tp: 240.0000 - fp: 2141.0000 - tn: 179818.0000 - fn: 77.0000 - accuracy: 0.9878 - precision: 0.1008 - recall: 0.7571 - auc: 0.9124 - val_loss: 0.0334 - val_tp: 69.0000 - val_fp: 257.0000 - val_tn: 45236.0000 - val_fn: 7.0000 - val_accuracy: 0.9942 - val_precision: 0.2117 - val_recall: 0.9079 - val_auc: 0.9951 Epoch 5/100 90/90 [==============================] - 1s 6ms/step - loss: 0.4060 - tp: 256.0000 - fp: 3039.0000 - tn: 178920.0000 - fn: 61.0000 - accuracy: 0.9830 - precision: 0.0777 - recall: 0.8076 - auc: 0.9329 - val_loss: 0.0439 - val_tp: 72.0000 - val_fp: 364.0000 - val_tn: 45129.0000 - val_fn: 4.0000 - val_accuracy: 0.9919 - val_precision: 0.1651 - val_recall: 0.9474 - val_auc: 0.9965 Epoch 6/100 90/90 [==============================] - 1s 6ms/step - loss: 0.4045 - tp: 255.0000 - fp: 4034.0000 - tn: 177925.0000 - fn: 62.0000 - accuracy: 0.9775 - precision: 0.0595 - recall: 0.8044 - auc: 0.9289 - val_loss: 0.0557 - val_tp: 73.0000 - val_fp: 572.0000 - val_tn: 44921.0000 - val_fn: 3.0000 - val_accuracy: 0.9874 - val_precision: 0.1132 - val_recall: 0.9605 - val_auc: 0.9969 Epoch 7/100 90/90 [==============================] - 1s 6ms/step - loss: 0.2990 - tp: 267.0000 - fp: 5024.0000 - tn: 176935.0000 - fn: 50.0000 - accuracy: 0.9722 - precision: 0.0505 - recall: 0.8423 - auc: 0.9544 - val_loss: 0.0723 - val_tp: 74.0000 - val_fp: 825.0000 - val_tn: 44668.0000 - val_fn: 2.0000 - val_accuracy: 0.9819 - val_precision: 0.0823 - val_recall: 0.9737 - val_auc: 0.9971 Epoch 8/100 90/90 [==============================] - 1s 6ms/step - loss: 0.3124 - tp: 267.0000 - fp: 6321.0000 - tn: 175638.0000 - fn: 50.0000 - accuracy: 0.9650 - precision: 0.0405 - recall: 0.8423 - auc: 0.9493 - val_loss: 0.0886 - val_tp: 74.0000 - val_fp: 1043.0000 - val_tn: 44450.0000 - val_fn: 2.0000 - val_accuracy: 0.9771 - val_precision: 0.0662 - val_recall: 0.9737 - val_auc: 0.9971 Epoch 9/100 90/90 [==============================] - 1s 6ms/step - loss: 0.2856 - tp: 276.0000 - fp: 6879.0000 - tn: 175080.0000 - fn: 41.0000 - accuracy: 0.9620 - precision: 0.0386 - recall: 0.8707 - auc: 0.9536 - val_loss: 0.0963 - val_tp: 74.0000 - val_fp: 1120.0000 - val_tn: 44373.0000 - val_fn: 2.0000 - val_accuracy: 0.9754 - val_precision: 0.0620 - val_recall: 0.9737 - val_auc: 0.9972 Epoch 10/100 90/90 [==============================] - 1s 6ms/step - loss: 0.2871 - tp: 272.0000 - fp: 7610.0000 - tn: 174349.0000 - fn: 45.0000 - accuracy: 0.9580 - precision: 0.0345 - recall: 0.8580 - auc: 0.9545 - val_loss: 0.1053 - val_tp: 74.0000 - val_fp: 1219.0000 - val_tn: 44274.0000 - val_fn: 2.0000 - val_accuracy: 0.9732 - val_precision: 0.0572 - val_recall: 0.9737 - val_auc: 0.9972 Epoch 11/100 90/90 [==============================] - 1s 6ms/step - loss: 0.2646 - tp: 273.0000 - fp: 8061.0000 - tn: 173898.0000 - fn: 44.0000 - accuracy: 0.9555 - precision: 0.0328 - recall: 0.8612 - auc: 0.9602 - val_loss: 0.1079 - val_tp: 74.0000 - val_fp: 1242.0000 - val_tn: 44251.0000 - val_fn: 2.0000 - val_accuracy: 0.9727 - val_precision: 0.0562 - val_recall: 0.9737 - val_auc: 0.9973 Epoch 12/100 90/90 [==============================] - 1s 6ms/step - loss: 0.2574 - tp: 277.0000 - fp: 8239.0000 - tn: 173720.0000 - fn: 40.0000 - accuracy: 0.9546 - precision: 0.0325 - recall: 0.8738 - auc: 0.9621 - val_loss: 0.1086 - val_tp: 74.0000 - val_fp: 1223.0000 - val_tn: 44270.0000 - val_fn: 2.0000 - val_accuracy: 0.9731 - val_precision: 0.0571 - val_recall: 0.9737 - val_auc: 0.9973 Epoch 13/100 90/90 [==============================] - 1s 6ms/step - loss: 0.2525 - tp: 279.0000 - fp: 7925.0000 - tn: 174034.0000 - fn: 38.0000 - accuracy: 0.9563 - precision: 0.0340 - recall: 0.8801 - auc: 0.9604 - val_loss: 0.1032 - val_tp: 74.0000 - val_fp: 1153.0000 - val_tn: 44340.0000 - val_fn: 2.0000 - val_accuracy: 0.9747 - val_precision: 0.0603 - val_recall: 0.9737 - val_auc: 0.9974 Epoch 14/100 90/90 [==============================] - 1s 6ms/step - loss: 0.2712 - tp: 273.0000 - fp: 8290.0000 - tn: 173669.0000 - fn: 44.0000 - accuracy: 0.9543 - precision: 0.0319 - recall: 0.8612 - auc: 0.9572 - val_loss: 0.1083 - val_tp: 74.0000 - val_fp: 1193.0000 - val_tn: 44300.0000 - val_fn: 2.0000 - val_accuracy: 0.9738 - val_precision: 0.0584 - val_recall: 0.9737 - val_auc: 0.9973 Epoch 15/100 90/90 [==============================] - 1s 6ms/step - loss: 0.2324 - tp: 283.0000 - fp: 8182.0000 - tn: 173777.0000 - fn: 34.0000 - accuracy: 0.9549 - precision: 0.0334 - recall: 0.8927 - auc: 0.9668 - val_loss: 0.1017 - val_tp: 74.0000 - val_fp: 1131.0000 - val_tn: 44362.0000 - val_fn: 2.0000 - val_accuracy: 0.9751 - val_precision: 0.0614 - val_recall: 0.9737 - val_auc: 0.9974 Epoch 16/100 90/90 [==============================] - 1s 6ms/step - loss: 0.2605 - tp: 277.0000 - fp: 7798.0000 - tn: 174161.0000 - fn: 40.0000 - accuracy: 0.9570 - precision: 0.0343 - recall: 0.8738 - auc: 0.9585 - val_loss: 0.1030 - val_tp: 74.0000 - val_fp: 1134.0000 - val_tn: 44359.0000 - val_fn: 2.0000 - val_accuracy: 0.9751 - val_precision: 0.0613 - val_recall: 0.9737 - val_auc: 0.9973 Epoch 17/100 90/90 [==============================] - 1s 6ms/step - loss: 0.2235 - tp: 281.0000 - fp: 7994.0000 - tn: 173965.0000 - fn: 36.0000 - accuracy: 0.9559 - precision: 0.0340 - recall: 0.8864 - auc: 0.9696 - val_loss: 0.1029 - val_tp: 74.0000 - val_fp: 1127.0000 - val_tn: 44366.0000 - val_fn: 2.0000 - val_accuracy: 0.9752 - val_precision: 0.0616 - val_recall: 0.9737 - val_auc: 0.9973 Epoch 18/100 90/90 [==============================] - 1s 6ms/step - loss: 0.2455 - tp: 278.0000 - fp: 7706.0000 - tn: 174253.0000 - fn: 39.0000 - accuracy: 0.9575 - precision: 0.0348 - recall: 0.8770 - auc: 0.9633 - val_loss: 0.1010 - val_tp: 74.0000 - val_fp: 1099.0000 - val_tn: 44394.0000 - val_fn: 2.0000 - val_accuracy: 0.9758 - val_precision: 0.0631 - val_recall: 0.9737 - val_auc: 0.9973 Epoch 19/100 90/90 [==============================] - 1s 6ms/step - loss: 0.1759 - tp: 285.0000 - fp: 7766.0000 - tn: 174193.0000 - fn: 32.0000 - accuracy: 0.9572 - precision: 0.0354 - recall: 0.8991 - auc: 0.9813 - val_loss: 0.1001 - val_tp: 74.0000 - val_fp: 1079.0000 - val_tn: 44414.0000 - val_fn: 2.0000 - val_accuracy: 0.9763 - val_precision: 0.0642 - val_recall: 0.9737 - val_auc: 0.9973 Epoch 20/100 90/90 [==============================] - 1s 6ms/step - loss: 0.2220 - tp: 283.0000 - fp: 7554.0000 - tn: 174405.0000 - fn: 34.0000 - accuracy: 0.9584 - precision: 0.0361 - recall: 0.8927 - auc: 0.9669 - val_loss: 0.0944 - val_tp: 74.0000 - val_fp: 1009.0000 - val_tn: 44484.0000 - val_fn: 2.0000 - val_accuracy: 0.9778 - val_precision: 0.0683 - val_recall: 0.9737 - val_auc: 0.9973 Epoch 21/100 90/90 [==============================] - 1s 6ms/step - loss: 0.2329 - tp: 282.0000 - fp: 7369.0000 - tn: 174590.0000 - fn: 35.0000 - accuracy: 0.9594 - precision: 0.0369 - recall: 0.8896 - auc: 0.9657 - val_loss: 0.0942 - val_tp: 74.0000 - val_fp: 1011.0000 - val_tn: 44482.0000 - val_fn: 2.0000 - val_accuracy: 0.9778 - val_precision: 0.0682 - val_recall: 0.9737 - val_auc: 0.9973 Epoch 22/100 90/90 [==============================] - 1s 6ms/step - loss: 0.2317 - tp: 281.0000 - fp: 7563.0000 - tn: 174396.0000 - fn: 36.0000 - accuracy: 0.9583 - precision: 0.0358 - recall: 0.8864 - auc: 0.9658 - val_loss: 0.0936 - val_tp: 74.0000 - val_fp: 993.0000 - val_tn: 44500.0000 - val_fn: 2.0000 - val_accuracy: 0.9782 - val_precision: 0.0694 - val_recall: 0.9737 - val_auc: 0.9974 Epoch 23/100 90/90 [==============================] - 1s 6ms/step - loss: 0.2109 - tp: 284.0000 - fp: 7520.0000 - tn: 174439.0000 - fn: 33.0000 - accuracy: 0.9586 - precision: 0.0364 - recall: 0.8959 - auc: 0.9702 - val_loss: 0.0940 - val_tp: 74.0000 - val_fp: 1003.0000 - val_tn: 44490.0000 - val_fn: 2.0000 - val_accuracy: 0.9779 - val_precision: 0.0687 - val_recall: 0.9737 - val_auc: 0.9974 Epoch 24/100 90/90 [==============================] - 1s 6ms/step - loss: 0.2089 - tp: 283.0000 - fp: 7119.0000 - tn: 174840.0000 - fn: 34.0000 - accuracy: 0.9608 - precision: 0.0382 - recall: 0.8927 - auc: 0.9731 - val_loss: 0.0898 - val_tp: 74.0000 - val_fp: 939.0000 - val_tn: 44554.0000 - val_fn: 2.0000 - val_accuracy: 0.9793 - val_precision: 0.0731 - val_recall: 0.9737 - val_auc: 0.9974 Epoch 25/100 90/90 [==============================] - 1s 6ms/step - loss: 0.2062 - tp: 280.0000 - fp: 7392.0000 - tn: 174567.0000 - fn: 37.0000 - accuracy: 0.9592 - precision: 0.0365 - recall: 0.8833 - auc: 0.9749 - val_loss: 0.0969 - val_tp: 74.0000 - val_fp: 1035.0000 - val_tn: 44458.0000 - val_fn: 2.0000 - val_accuracy: 0.9772 - val_precision: 0.0667 - val_recall: 0.9737 - val_auc: 0.9973 Epoch 26/100 90/90 [==============================] - 1s 6ms/step - loss: 0.2006 - tp: 284.0000 - fp: 7149.0000 - tn: 174810.0000 - fn: 33.0000 - accuracy: 0.9606 - precision: 0.0382 - recall: 0.8959 - auc: 0.9754 - val_loss: 0.0949 - val_tp: 74.0000 - val_fp: 1004.0000 - val_tn: 44489.0000 - val_fn: 2.0000 - val_accuracy: 0.9779 - val_precision: 0.0686 - val_recall: 0.9737 - val_auc: 0.9973 Epoch 27/100 90/90 [==============================] - 1s 6ms/step - loss: 0.2169 - tp: 281.0000 - fp: 7592.0000 - tn: 174367.0000 - fn: 36.0000 - accuracy: 0.9582 - precision: 0.0357 - recall: 0.8864 - auc: 0.9727 - val_loss: 0.0969 - val_tp: 74.0000 - val_fp: 1038.0000 - val_tn: 44455.0000 - val_fn: 2.0000 - val_accuracy: 0.9772 - val_precision: 0.0665 - val_recall: 0.9737 - val_auc: 0.9974 Epoch 28/100 90/90 [==============================] - 1s 6ms/step - loss: 0.2110 - tp: 285.0000 - fp: 6957.0000 - tn: 175002.0000 - fn: 32.0000 - accuracy: 0.9617 - precision: 0.0394 - recall: 0.8991 - auc: 0.9697 - val_loss: 0.0922 - val_tp: 74.0000 - val_fp: 988.0000 - val_tn: 44505.0000 - val_fn: 2.0000 - val_accuracy: 0.9783 - val_precision: 0.0697 - val_recall: 0.9737 - val_auc: 0.9974 Epoch 29/100 90/90 [==============================] - 1s 6ms/step - loss: 0.2234 - tp: 280.0000 - fp: 7030.0000 - tn: 174929.0000 - fn: 37.0000 - accuracy: 0.9612 - precision: 0.0383 - recall: 0.8833 - auc: 0.9679 - val_loss: 0.0942 - val_tp: 74.0000 - val_fp: 1024.0000 - val_tn: 44469.0000 - val_fn: 2.0000 - val_accuracy: 0.9775 - val_precision: 0.0674 - val_recall: 0.9737 - val_auc: 0.9974 Epoch 30/100 90/90 [==============================] - 1s 6ms/step - loss: 0.1889 - tp: 288.0000 - fp: 7228.0000 - tn: 174731.0000 - fn: 29.0000 - accuracy: 0.9602 - precision: 0.0383 - recall: 0.9085 - auc: 0.9771 - val_loss: 0.0895 - val_tp: 74.0000 - val_fp: 954.0000 - val_tn: 44539.0000 - val_fn: 2.0000 - val_accuracy: 0.9790 - val_precision: 0.0720 - val_recall: 0.9737 - val_auc: 0.9974 Epoch 31/100 90/90 [==============================] - 1s 6ms/step - loss: 0.1824 - tp: 286.0000 - fp: 6703.0000 - tn: 175256.0000 - fn: 31.0000 - accuracy: 0.9631 - precision: 0.0409 - recall: 0.9022 - auc: 0.9789 - val_loss: 0.0898 - val_tp: 74.0000 - val_fp: 957.0000 - val_tn: 44536.0000 - val_fn: 2.0000 - val_accuracy: 0.9790 - val_precision: 0.0718 - val_recall: 0.9737 - val_auc: 0.9974 Epoch 32/100 90/90 [==============================] - 1s 6ms/step - loss: 0.2014 - tp: 279.0000 - fp: 6918.0000 - tn: 175041.0000 - fn: 38.0000 - accuracy: 0.9618 - precision: 0.0388 - recall: 0.8801 - auc: 0.9756 - val_loss: 0.0933 - val_tp: 74.0000 - val_fp: 993.0000 - val_tn: 44500.0000 - val_fn: 2.0000 - val_accuracy: 0.9782 - val_precision: 0.0694 - val_recall: 0.9737 - val_auc: 0.9974 Epoch 33/100 90/90 [==============================] - 1s 6ms/step - loss: 0.2153 - tp: 283.0000 - fp: 6885.0000 - tn: 175074.0000 - fn: 34.0000 - accuracy: 0.9620 - precision: 0.0395 - recall: 0.8927 - auc: 0.9708 - val_loss: 0.0905 - val_tp: 74.0000 - val_fp: 949.0000 - val_tn: 44544.0000 - val_fn: 2.0000 - val_accuracy: 0.9791 - val_precision: 0.0723 - val_recall: 0.9737 - val_auc: 0.9974 Epoch 34/100 90/90 [==============================] - 1s 6ms/step - loss: 0.1970 - tp: 283.0000 - fp: 6969.0000 - tn: 174990.0000 - fn: 34.0000 - accuracy: 0.9616 - precision: 0.0390 - recall: 0.8927 - auc: 0.9769 - val_loss: 0.0958 - val_tp: 74.0000 - val_fp: 1018.0000 - val_tn: 44475.0000 - val_fn: 2.0000 - val_accuracy: 0.9776 - val_precision: 0.0678 - val_recall: 0.9737 - val_auc: 0.9975 Epoch 35/100 90/90 [==============================] - 1s 6ms/step - loss: 0.1737 - tp: 290.0000 - fp: 7040.0000 - tn: 174919.0000 - fn: 27.0000 - accuracy: 0.9612 - precision: 0.0396 - recall: 0.9148 - auc: 0.9806 - val_loss: 0.0907 - val_tp: 74.0000 - val_fp: 948.0000 - val_tn: 44545.0000 - val_fn: 2.0000 - val_accuracy: 0.9792 - val_precision: 0.0724 - val_recall: 0.9737 - val_auc: 0.9975 Epoch 36/100 90/90 [==============================] - 1s 6ms/step - loss: 0.1906 - tp: 288.0000 - fp: 6663.0000 - tn: 175296.0000 - fn: 29.0000 - accuracy: 0.9633 - precision: 0.0414 - recall: 0.9085 - auc: 0.9764 - val_loss: 0.0872 - val_tp: 74.0000 - val_fp: 904.0000 - val_tn: 44589.0000 - val_fn: 2.0000 - val_accuracy: 0.9801 - val_precision: 0.0757 - val_recall: 0.9737 - val_auc: 0.9975 Epoch 37/100 90/90 [==============================] - 1s 6ms/step - loss: 0.2071 - tp: 284.0000 - fp: 6688.0000 - tn: 175271.0000 - fn: 33.0000 - accuracy: 0.9631 - precision: 0.0407 - recall: 0.8959 - auc: 0.9727 - val_loss: 0.0905 - val_tp: 74.0000 - val_fp: 939.0000 - val_tn: 44554.0000 - val_fn: 2.0000 - val_accuracy: 0.9793 - val_precision: 0.0731 - val_recall: 0.9737 - val_auc: 0.9975 Epoch 38/100 90/90 [==============================] - 1s 6ms/step - loss: 0.1813 - tp: 283.0000 - fp: 6784.0000 - tn: 175175.0000 - fn: 34.0000 - accuracy: 0.9626 - precision: 0.0400 - recall: 0.8927 - auc: 0.9817 - val_loss: 0.0913 - val_tp: 74.0000 - val_fp: 951.0000 - val_tn: 44542.0000 - val_fn: 2.0000 - val_accuracy: 0.9791 - val_precision: 0.0722 - val_recall: 0.9737 - val_auc: 0.9975 Epoch 39/100 90/90 [==============================] - 1s 6ms/step - loss: 0.1897 - tp: 289.0000 - fp: 6872.0000 - tn: 175087.0000 - fn: 28.0000 - accuracy: 0.9621 - precision: 0.0404 - recall: 0.9117 - auc: 0.9767 - val_loss: 0.0916 - val_tp: 74.0000 - val_fp: 959.0000 - val_tn: 44534.0000 - val_fn: 2.0000 - val_accuracy: 0.9789 - val_precision: 0.0716 - val_recall: 0.9737 - val_auc: 0.9975 Epoch 40/100 90/90 [==============================] - 1s 6ms/step - loss: 0.1546 - tp: 295.0000 - fp: 6584.0000 - tn: 175375.0000 - fn: 22.0000 - accuracy: 0.9638 - precision: 0.0429 - recall: 0.9306 - auc: 0.9858 - val_loss: 0.0904 - val_tp: 74.0000 - val_fp: 948.0000 - val_tn: 44545.0000 - val_fn: 2.0000 - val_accuracy: 0.9792 - val_precision: 0.0724 - val_recall: 0.9737 - val_auc: 0.9976 Epoch 41/100 90/90 [==============================] - 1s 6ms/step - loss: 0.2263 - tp: 278.0000 - fp: 6677.0000 - tn: 175282.0000 - fn: 39.0000 - accuracy: 0.9632 - precision: 0.0400 - recall: 0.8770 - auc: 0.9676 - val_loss: 0.0908 - val_tp: 74.0000 - val_fp: 955.0000 - val_tn: 44538.0000 - val_fn: 2.0000 - val_accuracy: 0.9790 - val_precision: 0.0719 - val_recall: 0.9737 - val_auc: 0.9975 Epoch 42/100 90/90 [==============================] - 1s 6ms/step - loss: 0.2033 - tp: 281.0000 - fp: 6777.0000 - tn: 175182.0000 - fn: 36.0000 - accuracy: 0.9626 - precision: 0.0398 - recall: 0.8864 - auc: 0.9756 - val_loss: 0.0953 - val_tp: 74.0000 - val_fp: 1029.0000 - val_tn: 44464.0000 - val_fn: 2.0000 - val_accuracy: 0.9774 - val_precision: 0.0671 - val_recall: 0.9737 - val_auc: 0.9975 Epoch 43/100 90/90 [==============================] - 1s 6ms/step - loss: 0.1865 - tp: 283.0000 - fp: 6824.0000 - tn: 175135.0000 - fn: 34.0000 - accuracy: 0.9624 - precision: 0.0398 - recall: 0.8927 - auc: 0.9792 - val_loss: 0.0945 - val_tp: 74.0000 - val_fp: 1019.0000 - val_tn: 44474.0000 - val_fn: 2.0000 - val_accuracy: 0.9776 - val_precision: 0.0677 - val_recall: 0.9737 - val_auc: 0.9976 Epoch 44/100 90/90 [==============================] - 1s 6ms/step - loss: 0.1857 - tp: 286.0000 - fp: 6655.0000 - tn: 175304.0000 - fn: 31.0000 - accuracy: 0.9633 - precision: 0.0412 - recall: 0.9022 - auc: 0.9781 - val_loss: 0.0927 - val_tp: 74.0000 - val_fp: 975.0000 - val_tn: 44518.0000 - val_fn: 2.0000 - val_accuracy: 0.9786 - val_precision: 0.0705 - val_recall: 0.9737 - val_auc: 0.9977 Epoch 45/100 90/90 [==============================] - 1s 6ms/step - loss: 0.1874 - tp: 282.0000 - fp: 6595.0000 - tn: 175364.0000 - fn: 35.0000 - accuracy: 0.9636 - precision: 0.0410 - recall: 0.8896 - auc: 0.9810 - val_loss: 0.0934 - val_tp: 74.0000 - val_fp: 974.0000 - val_tn: 44519.0000 - val_fn: 2.0000 - val_accuracy: 0.9786 - val_precision: 0.0706 - val_recall: 0.9737 - val_auc: 0.9977 Epoch 46/100 90/90 [==============================] - 1s 6ms/step - loss: 0.1641 - tp: 291.0000 - fp: 6532.0000 - tn: 175427.0000 - fn: 26.0000 - accuracy: 0.9640 - precision: 0.0426 - recall: 0.9180 - auc: 0.9837 - val_loss: 0.0868 - val_tp: 74.0000 - val_fp: 885.0000 - val_tn: 44608.0000 - val_fn: 2.0000 - val_accuracy: 0.9805 - val_precision: 0.0772 - val_recall: 0.9737 - val_auc: 0.9976 Epoch 47/100 90/90 [==============================] - 1s 6ms/step - loss: 0.1909 - tp: 285.0000 - fp: 6662.0000 - tn: 175297.0000 - fn: 32.0000 - accuracy: 0.9633 - precision: 0.0410 - recall: 0.8991 - auc: 0.9780 - val_loss: 0.0874 - val_tp: 74.0000 - val_fp: 899.0000 - val_tn: 44594.0000 - val_fn: 2.0000 - val_accuracy: 0.9802 - val_precision: 0.0761 - val_recall: 0.9737 - val_auc: 0.9978 Epoch 48/100 90/90 [==============================] - 1s 6ms/step - loss: 0.1687 - tp: 287.0000 - fp: 6627.0000 - tn: 175332.0000 - fn: 30.0000 - accuracy: 0.9635 - precision: 0.0415 - recall: 0.9054 - auc: 0.9836 - val_loss: 0.0875 - val_tp: 74.0000 - val_fp: 899.0000 - val_tn: 44594.0000 - val_fn: 2.0000 - val_accuracy: 0.9802 - val_precision: 0.0761 - val_recall: 0.9737 - val_auc: 0.9977 Epoch 49/100 90/90 [==============================] - 1s 6ms/step - loss: 0.1740 - tp: 291.0000 - fp: 6522.0000 - tn: 175437.0000 - fn: 26.0000 - accuracy: 0.9641 - precision: 0.0427 - recall: 0.9180 - auc: 0.9794 - val_loss: 0.0848 - val_tp: 74.0000 - val_fp: 866.0000 - val_tn: 44627.0000 - val_fn: 2.0000 - val_accuracy: 0.9810 - val_precision: 0.0787 - val_recall: 0.9737 - val_auc: 0.9978 Epoch 50/100 90/90 [==============================] - 1s 6ms/step - loss: 0.1773 - tp: 292.0000 - fp: 6370.0000 - tn: 175589.0000 - fn: 25.0000 - accuracy: 0.9649 - precision: 0.0438 - recall: 0.9211 - auc: 0.9785 - val_loss: 0.0794 - val_tp: 74.0000 - val_fp: 791.0000 - val_tn: 44702.0000 - val_fn: 2.0000 - val_accuracy: 0.9826 - val_precision: 0.0855 - val_recall: 0.9737 - val_auc: 0.9978 Epoch 51/100 90/90 [==============================] - 1s 6ms/step - loss: 0.1828 - tp: 289.0000 - fp: 5870.0000 - tn: 176089.0000 - fn: 28.0000 - accuracy: 0.9676 - precision: 0.0469 - recall: 0.9117 - auc: 0.9789 - val_loss: 0.0791 - val_tp: 74.0000 - val_fp: 807.0000 - val_tn: 44686.0000 - val_fn: 2.0000 - val_accuracy: 0.9822 - val_precision: 0.0840 - val_recall: 0.9737 - val_auc: 0.9980 Epoch 52/100 90/90 [==============================] - 1s 6ms/step - loss: 0.1567 - tp: 290.0000 - fp: 5704.0000 - tn: 176255.0000 - fn: 27.0000 - accuracy: 0.9686 - precision: 0.0484 - recall: 0.9148 - auc: 0.9860 - val_loss: 0.0772 - val_tp: 74.0000 - val_fp: 765.0000 - val_tn: 44728.0000 - val_fn: 2.0000 - val_accuracy: 0.9832 - val_precision: 0.0882 - val_recall: 0.9737 - val_auc: 0.9979 Epoch 53/100 90/90 [==============================] - 1s 6ms/step - loss: 0.1752 - tp: 290.0000 - fp: 6278.0000 - tn: 175681.0000 - fn: 27.0000 - accuracy: 0.9654 - precision: 0.0442 - recall: 0.9148 - auc: 0.9808 - val_loss: 0.0854 - val_tp: 74.0000 - val_fp: 873.0000 - val_tn: 44620.0000 - val_fn: 2.0000 - val_accuracy: 0.9808 - val_precision: 0.0781 - val_recall: 0.9737 - val_auc: 0.9979 Epoch 54/100 90/90 [==============================] - 1s 6ms/step - loss: 0.1644 - tp: 289.0000 - fp: 6298.0000 - tn: 175661.0000 - fn: 28.0000 - accuracy: 0.9653 - precision: 0.0439 - recall: 0.9117 - auc: 0.9833 - val_loss: 0.0875 - val_tp: 74.0000 - val_fp: 904.0000 - val_tn: 44589.0000 - val_fn: 2.0000 - val_accuracy: 0.9801 - val_precision: 0.0757 - val_recall: 0.9737 - val_auc: 0.9979 Epoch 55/100 90/90 [==============================] - 1s 6ms/step - loss: 0.1721 - tp: 290.0000 - fp: 6318.0000 - tn: 175641.0000 - fn: 27.0000 - accuracy: 0.9652 - precision: 0.0439 - recall: 0.9148 - auc: 0.9816 - val_loss: 0.0827 - val_tp: 74.0000 - val_fp: 844.0000 - val_tn: 44649.0000 - val_fn: 2.0000 - val_accuracy: 0.9814 - val_precision: 0.0806 - val_recall: 0.9737 - val_auc: 0.9979 Epoch 56/100 90/90 [==============================] - 1s 6ms/step - loss: 0.1841 - tp: 284.0000 - fp: 6052.0000 - tn: 175907.0000 - fn: 33.0000 - accuracy: 0.9666 - precision: 0.0448 - recall: 0.8959 - auc: 0.9798 - val_loss: 0.0872 - val_tp: 74.0000 - val_fp: 911.0000 - val_tn: 44582.0000 - val_fn: 2.0000 - val_accuracy: 0.9800 - val_precision: 0.0751 - val_recall: 0.9737 - val_auc: 0.9977 Epoch 57/100 90/90 [==============================] - 1s 6ms/step - loss: 0.1710 - tp: 289.0000 - fp: 6593.0000 - tn: 175366.0000 - fn: 28.0000 - accuracy: 0.9637 - precision: 0.0420 - recall: 0.9117 - auc: 0.9824 - val_loss: 0.0856 - val_tp: 74.0000 - val_fp: 890.0000 - val_tn: 44603.0000 - val_fn: 2.0000 - val_accuracy: 0.9804 - val_precision: 0.0768 - val_recall: 0.9737 - val_auc: 0.9978 Epoch 58/100 90/90 [==============================] - 1s 6ms/step - loss: 0.1649 - tp: 287.0000 - fp: 6478.0000 - tn: 175481.0000 - fn: 30.0000 - accuracy: 0.9643 - precision: 0.0424 - recall: 0.9054 - auc: 0.9836 - val_loss: 0.0797 - val_tp: 74.0000 - val_fp: 817.0000 - val_tn: 44676.0000 - val_fn: 2.0000 - val_accuracy: 0.9820 - val_precision: 0.0831 - val_recall: 0.9737 - val_auc: 0.9980 Epoch 59/100 90/90 [==============================] - 1s 6ms/step - loss: 0.1752 - tp: 285.0000 - fp: 5981.0000 - tn: 175978.0000 - fn: 32.0000 - accuracy: 0.9670 - precision: 0.0455 - recall: 0.8991 - auc: 0.9827 - val_loss: 0.0813 - val_tp: 74.0000 - val_fp: 842.0000 - val_tn: 44651.0000 - val_fn: 2.0000 - val_accuracy: 0.9815 - val_precision: 0.0808 - val_recall: 0.9737 - val_auc: 0.9980 Epoch 60/100 90/90 [==============================] - 1s 6ms/step - loss: 0.1374 - tp: 298.0000 - fp: 5910.0000 - tn: 176049.0000 - fn: 19.0000 - accuracy: 0.9675 - precision: 0.0480 - recall: 0.9401 - auc: 0.9884 - val_loss: 0.0760 - val_tp: 74.0000 - val_fp: 764.0000 - val_tn: 44729.0000 - val_fn: 2.0000 - val_accuracy: 0.9832 - val_precision: 0.0883 - val_recall: 0.9737 - val_auc: 0.9980 Epoch 61/100 90/90 [==============================] - 1s 6ms/step - loss: 0.1655 - tp: 288.0000 - fp: 5872.0000 - tn: 176087.0000 - fn: 29.0000 - accuracy: 0.9676 - precision: 0.0468 - recall: 0.9085 - auc: 0.9838 - val_loss: 0.0795 - val_tp: 74.0000 - val_fp: 819.0000 - val_tn: 44674.0000 - val_fn: 2.0000 - val_accuracy: 0.9820 - val_precision: 0.0829 - val_recall: 0.9737 - val_auc: 0.9979 Epoch 62/100 90/90 [==============================] - 1s 6ms/step - loss: 0.2024 - tp: 281.0000 - fp: 6087.0000 - tn: 175872.0000 - fn: 36.0000 - accuracy: 0.9664 - precision: 0.0441 - recall: 0.8864 - auc: 0.9758 - val_loss: 0.0841 - val_tp: 74.0000 - val_fp: 872.0000 - val_tn: 44621.0000 - val_fn: 2.0000 - val_accuracy: 0.9808 - val_precision: 0.0782 - val_recall: 0.9737 - val_auc: 0.9979 Epoch 63/100 90/90 [==============================] - 1s 7ms/step - loss: 0.1624 - tp: 288.0000 - fp: 6407.0000 - tn: 175552.0000 - fn: 29.0000 - accuracy: 0.9647 - precision: 0.0430 - recall: 0.9085 - auc: 0.9855 - val_loss: 0.0836 - val_tp: 74.0000 - val_fp: 876.0000 - val_tn: 44617.0000 - val_fn: 2.0000 - val_accuracy: 0.9807 - val_precision: 0.0779 - val_recall: 0.9737 - val_auc: 0.9979 Epoch 64/100 90/90 [==============================] - 1s 7ms/step - loss: 0.1757 - tp: 287.0000 - fp: 6419.0000 - tn: 175540.0000 - fn: 30.0000 - accuracy: 0.9646 - precision: 0.0428 - recall: 0.9054 - auc: 0.9812 - val_loss: 0.0803 - val_tp: 74.0000 - val_fp: 832.0000 - val_tn: 44661.0000 - val_fn: 2.0000 - val_accuracy: 0.9817 - val_precision: 0.0817 - val_recall: 0.9737 - val_auc: 0.9979 Epoch 65/100 90/90 [==============================] - 1s 7ms/step - loss: 0.1609 - tp: 287.0000 - fp: 6008.0000 - tn: 175951.0000 - fn: 30.0000 - accuracy: 0.9669 - precision: 0.0456 - recall: 0.9054 - auc: 0.9854 - val_loss: 0.0805 - val_tp: 74.0000 - val_fp: 838.0000 - val_tn: 44655.0000 - val_fn: 2.0000 - val_accuracy: 0.9816 - val_precision: 0.0811 - val_recall: 0.9737 - val_auc: 0.9980 Epoch 66/100 90/90 [==============================] - 1s 7ms/step - loss: 0.1593 - tp: 290.0000 - fp: 6104.0000 - tn: 175855.0000 - fn: 27.0000 - accuracy: 0.9664 - precision: 0.0454 - recall: 0.9148 - auc: 0.9838 - val_loss: 0.0801 - val_tp: 74.0000 - val_fp: 822.0000 - val_tn: 44671.0000 - val_fn: 2.0000 - val_accuracy: 0.9819 - val_precision: 0.0826 - val_recall: 0.9737 - val_auc: 0.9980 Epoch 67/100 90/90 [==============================] - 1s 8ms/step - loss: 0.1600 - tp: 292.0000 - fp: 5888.0000 - tn: 176071.0000 - fn: 25.0000 - accuracy: 0.9676 - precision: 0.0472 - recall: 0.9211 - auc: 0.9823 - val_loss: 0.0766 - val_tp: 74.0000 - val_fp: 786.0000 - val_tn: 44707.0000 - val_fn: 2.0000 - val_accuracy: 0.9827 - val_precision: 0.0860 - val_recall: 0.9737 - val_auc: 0.9981 Epoch 68/100 90/90 [==============================] - 1s 8ms/step - loss: 0.1803 - tp: 286.0000 - fp: 5871.0000 - tn: 176088.0000 - fn: 31.0000 - accuracy: 0.9676 - precision: 0.0465 - recall: 0.9022 - auc: 0.9792 - val_loss: 0.0785 - val_tp: 74.0000 - val_fp: 788.0000 - val_tn: 44705.0000 - val_fn: 2.0000 - val_accuracy: 0.9827 - val_precision: 0.0858 - val_recall: 0.9737 - val_auc: 0.9980 Epoch 69/100 90/90 [==============================] - 1s 7ms/step - loss: 0.1873 - tp: 284.0000 - fp: 5685.0000 - tn: 176274.0000 - fn: 33.0000 - accuracy: 0.9686 - precision: 0.0476 - recall: 0.8959 - auc: 0.9773 - val_loss: 0.0779 - val_tp: 74.0000 - val_fp: 786.0000 - val_tn: 44707.0000 - val_fn: 2.0000 - val_accuracy: 0.9827 - val_precision: 0.0860 - val_recall: 0.9737 - val_auc: 0.9979 Epoch 70/100 90/90 [==============================] - 1s 7ms/step - loss: 0.1525 - tp: 289.0000 - fp: 6048.0000 - tn: 175911.0000 - fn: 28.0000 - accuracy: 0.9667 - precision: 0.0456 - recall: 0.9117 - auc: 0.9857 - val_loss: 0.0791 - val_tp: 74.0000 - val_fp: 803.0000 - val_tn: 44690.0000 - val_fn: 2.0000 - val_accuracy: 0.9823 - val_precision: 0.0844 - val_recall: 0.9737 - val_auc: 0.9979 Epoch 71/100 90/90 [==============================] - 1s 7ms/step - loss: 0.1971 - tp: 284.0000 - fp: 5980.0000 - tn: 175979.0000 - fn: 33.0000 - accuracy: 0.9670 - precision: 0.0453 - recall: 0.8959 - auc: 0.9753 - val_loss: 0.0811 - val_tp: 74.0000 - val_fp: 837.0000 - val_tn: 44656.0000 - val_fn: 2.0000 - val_accuracy: 0.9816 - val_precision: 0.0812 - val_recall: 0.9737 - val_auc: 0.9980 Epoch 72/100 90/90 [==============================] - 1s 7ms/step - loss: 0.1695 - tp: 286.0000 - fp: 6341.0000 - tn: 175618.0000 - fn: 31.0000 - accuracy: 0.9650 - precision: 0.0432 - recall: 0.9022 - auc: 0.9845 - val_loss: 0.0846 - val_tp: 74.0000 - val_fp: 896.0000 - val_tn: 44597.0000 - val_fn: 2.0000 - val_accuracy: 0.9803 - val_precision: 0.0763 - val_recall: 0.9737 - val_auc: 0.9979 Epoch 73/100 90/90 [==============================] - 1s 7ms/step - loss: 0.1766 - tp: 287.0000 - fp: 6297.0000 - tn: 175662.0000 - fn: 30.0000 - accuracy: 0.9653 - precision: 0.0436 - recall: 0.9054 - auc: 0.9818 - val_loss: 0.0824 - val_tp: 74.0000 - val_fp: 865.0000 - val_tn: 44628.0000 - val_fn: 2.0000 - val_accuracy: 0.9810 - val_precision: 0.0788 - val_recall: 0.9737 - val_auc: 0.9980 Epoch 74/100 90/90 [==============================] - 1s 7ms/step - loss: 0.1623 - tp: 289.0000 - fp: 6086.0000 - tn: 175873.0000 - fn: 28.0000 - accuracy: 0.9665 - precision: 0.0453 - recall: 0.9117 - auc: 0.9851 - val_loss: 0.0805 - val_tp: 74.0000 - val_fp: 829.0000 - val_tn: 44664.0000 - val_fn: 2.0000 - val_accuracy: 0.9818 - val_precision: 0.0819 - val_recall: 0.9737 - val_auc: 0.9979 Epoch 75/100 90/90 [==============================] - 1s 7ms/step - loss: 0.1746 - tp: 286.0000 - fp: 6430.0000 - tn: 175529.0000 - fn: 31.0000 - accuracy: 0.9646 - precision: 0.0426 - recall: 0.9022 - auc: 0.9826 - val_loss: 0.0851 - val_tp: 74.0000 - val_fp: 920.0000 - val_tn: 44573.0000 - val_fn: 2.0000 - val_accuracy: 0.9798 - val_precision: 0.0744 - val_recall: 0.9737 - val_auc: 0.9979 Epoch 76/100 90/90 [==============================] - 1s 7ms/step - loss: 0.1439 - tp: 294.0000 - fp: 6075.0000 - tn: 175884.0000 - fn: 23.0000 - accuracy: 0.9665 - precision: 0.0462 - recall: 0.9274 - auc: 0.9877 - val_loss: 0.0818 - val_tp: 74.0000 - val_fp: 872.0000 - val_tn: 44621.0000 - val_fn: 2.0000 - val_accuracy: 0.9808 - val_precision: 0.0782 - val_recall: 0.9737 - val_auc: 0.9979 Epoch 77/100 82/90 [==========================>...] - ETA: 0s - loss: 0.1624 - tp: 266.0000 - fp: 5807.0000 - tn: 161834.0000 - fn: 29.0000 - accuracy: 0.9652 - precision: 0.0438 - recall: 0.9017 - auc: 0.9853Restoring model weights from the end of the best epoch. 90/90 [==============================] - 1s 7ms/step - loss: 0.1554 - tp: 288.0000 - fp: 6332.0000 - tn: 175627.0000 - fn: 29.0000 - accuracy: 0.9651 - precision: 0.0435 - recall: 0.9085 - auc: 0.9862 - val_loss: 0.0818 - val_tp: 74.0000 - val_fp: 875.0000 - val_tn: 44618.0000 - val_fn: 2.0000 - val_accuracy: 0.9808 - val_precision: 0.0780 - val_recall: 0.9737 - val_auc: 0.9979 Epoch 00077: early stopping ``` ### Check training history ```py plot_metrics(weighted_history) ``` ![png](img/677f173984390980c5f4af3a22313c24.png) ### Evaluate metrics ```py train_predictions_weighted = weighted_model.predict(train_features, batch_size=BATCH_SIZE) test_predictions_weighted = weighted_model.predict(test_features, batch_size=BATCH_SIZE) ``` ```py weighted_results = weighted_model.evaluate(test_features, test_labels, batch_size=BATCH_SIZE, verbose=0) for name, value in zip(weighted_model.metrics_names, weighted_results): print(name, ': ', value) print() plot_cm(test_labels, test_predictions_weighted) ``` ```py loss : 0.07622280716896057 tp : 90.0 fp : 998.0 tn : 55865.0 fn : 9.0 accuracy : 0.982321560382843 precision : 0.08272058516740799 recall : 0.9090909361839294 auc : 0.9769566059112549 Legitimate Transactions Detected (True Negatives): 55865 Legitimate Transactions Incorrectly Detected (False Positives): 998 Fraudulent Transactions Missed (False Negatives): 9 Fraudulent Transactions Detected (True Positives): 90 Total Fraudulent Transactions: 99 ``` ![png](img/00fbc45eaf75c4132a6ea862403f4be4.png) Here you can see that with class weights the accuracy and precision are lower because there are more false positives, but conversely the recall and AUC are higher because the model also found more true positives. Despite having lower accuracy, this model has higher recall (and identifies more fraudulent transactions). Of course, there is a cost to both types of error (you wouldn't want to bug users by flagging too many legitimate transactions as fraudulent, either). Carefully consider the trade-offs between these different types of errors for your application. ### Plot the ROC ```py plot_roc("Train Baseline", train_labels, train_predictions_baseline, color=colors[0]) plot_roc("Test Baseline", test_labels, test_predictions_baseline, color=colors[0], linestyle='--') plot_roc("Train Weighted", train_labels, train_predictions_weighted, color=colors[1]) plot_roc("Test Weighted", test_labels, test_predictions_weighted, color=colors[1], linestyle='--') plt.legend(loc='lower right') ``` ```py ``` ![png](img/13a6ef1c7f66c4208c56677c5ddd6506.png) ## Oversampling ### Oversample the minority class A related approach would be to resample the dataset by oversampling the minority class. ```py pos_features = train_features[bool_train_labels] neg_features = train_features[~bool_train_labels] pos_labels = train_labels[bool_train_labels] neg_labels = train_labels[~bool_train_labels] ``` #### Using NumPy You can balance the dataset manually by choosing the right number of random indices from the positive examples: ```py ids = np.arange(len(pos_features)) choices = np.random.choice(ids, len(neg_features)) res_pos_features = pos_features[choices] res_pos_labels = pos_labels[choices] res_pos_features.shape ``` ```py (181959, 29) ``` ```py resampled_features = np.concatenate([res_pos_features, neg_features], axis=0) resampled_labels = np.concatenate([res_pos_labels, neg_labels], axis=0) order = np.arange(len(resampled_labels)) np.random.shuffle(order) resampled_features = resampled_features[order] resampled_labels = resampled_labels[order] resampled_features.shape ``` ```py (363918, 29) ``` #### Using [`tf.data`](https://tensorflow.google.cn/api_docs/python/tf/data) If you're using [`tf.data`](https://tensorflow.google.cn/api_docs/python/tf/data) the easiest way to produce balanced examples is to start with a `positive` and a `negative` dataset, and merge them. See [the tf.data guide](https://tensorflow.google.cn/guide/data) for more examples. ```py BUFFER_SIZE = 100000 def make_ds(features, labels): ds = tf.data.Dataset.from_tensor_slices((features, labels))#.cache() ds = ds.shuffle(BUFFER_SIZE).repeat() return ds pos_ds = make_ds(pos_features, pos_labels) neg_ds = make_ds(neg_features, neg_labels) ``` Each dataset provides `(feature, label)` pairs: ```py for features, label in pos_ds.take(1): print("Features:\n", features.numpy()) print() print("Label: ", label.numpy()) ``` ```py Features: [-1.72731925 1.77656615 -3.74269876 2.74253414 -1.32668397 -1.34677584 -4.46175762 2.0139002 -2.59309618 -5\. 4.12736453 -5. 0.02274489 -5\. -0.32786349 -5\. -5\. -2.96360886 2.89835815 0.75463714 1.2022707 -0.14114195 -0.95544067 -1.22935903 0.18671861 -0.27928716 3.04376109 0.29779937 -1.45688482] Label: 1 ``` Merge the two together using [`experimental.sample_from_datasets`](https://tensorflow.google.cn/api_docs/python/tf/data/experimental/sample_from_datasets): ```py resampled_ds = tf.data.experimental.sample_from_datasets([pos_ds, neg_ds], weights=[0.5, 0.5]) resampled_ds = resampled_ds.batch(BATCH_SIZE).prefetch(2) ``` ```py for features, label in resampled_ds.take(1): print(label.numpy().mean()) ``` ```py 0.50537109375 ``` To use this dataset, you'll need the number of steps per epoch. The definition of "epoch" in this case is less clear. Say it's the number of batches required to see each negative example once: ```py resampled_steps_per_epoch = np.ceil(2.0*neg/BATCH_SIZE) resampled_steps_per_epoch ``` ```py 278.0 ``` ### Train on the oversampled data Now try training the model with the resampled data set instead of using class weights to see how these methods compare. **Note:** Because the data was balanced by replicating the positive examples, the total dataset size is larger, and each epoch runs for more training steps. ```py resampled_model = make_model() resampled_model.load_weights(initial_weights) # Reset the bias to zero, since this dataset is balanced. output_layer = resampled_model.layers[-1] output_layer.bias.assign([0]) val_ds = tf.data.Dataset.from_tensor_slices((val_features, val_labels)).cache() val_ds = val_ds.batch(BATCH_SIZE).prefetch(2) resampled_history = resampled_model.fit( resampled_ds, epochs=EPOCHS, steps_per_epoch=resampled_steps_per_epoch, callbacks=[early_stopping], validation_data=val_ds) ``` ```py Epoch 1/100 278/278 [==============================] - 7s 25ms/step - loss: 0.4377 - tp: 243925.0000 - fp: 88309.0000 - tn: 253494.0000 - fn: 40578.0000 - accuracy: 0.7942 - precision: 0.7342 - recall: 0.8574 - auc: 0.9062 - val_loss: 0.2355 - val_tp: 75.0000 - val_fp: 1532.0000 - val_tn: 43961.0000 - val_fn: 1.0000 - val_accuracy: 0.9664 - val_precision: 0.0467 - val_recall: 0.9868 - val_auc: 0.9984 Epoch 2/100 278/278 [==============================] - 7s 24ms/step - loss: 0.2055 - tp: 258676.0000 - fp: 18235.0000 - tn: 266800.0000 - fn: 25633.0000 - accuracy: 0.9229 - precision: 0.9341 - recall: 0.9098 - auc: 0.9741 - val_loss: 0.1247 - val_tp: 74.0000 - val_fp: 1074.0000 - val_tn: 44419.0000 - val_fn: 2.0000 - val_accuracy: 0.9764 - val_precision: 0.0645 - val_recall: 0.9737 - val_auc: 0.9981 Epoch 3/100 278/278 [==============================] - 7s 24ms/step - loss: 0.1640 - tp: 263104.0000 - fp: 12413.0000 - tn: 272291.0000 - fn: 21536.0000 - accuracy: 0.9404 - precision: 0.9549 - recall: 0.9243 - auc: 0.9834 - val_loss: 0.0981 - val_tp: 74.0000 - val_fp: 937.0000 - val_tn: 44556.0000 - val_fn: 2.0000 - val_accuracy: 0.9794 - val_precision: 0.0732 - val_recall: 0.9737 - val_auc: 0.9980 Epoch 4/100 278/278 [==============================] - 7s 24ms/step - loss: 0.1417 - tp: 265623.0000 - fp: 10539.0000 - tn: 274390.0000 - fn: 18792.0000 - accuracy: 0.9485 - precision: 0.9618 - recall: 0.9339 - auc: 0.9882 - val_loss: 0.0842 - val_tp: 74.0000 - val_fp: 866.0000 - val_tn: 44627.0000 - val_fn: 2.0000 - val_accuracy: 0.9810 - val_precision: 0.0787 - val_recall: 0.9737 - val_auc: 0.9979 Epoch 5/100 278/278 [==============================] - 7s 24ms/step - loss: 0.1275 - tp: 267112.0000 - fp: 9551.0000 - tn: 275482.0000 - fn: 17199.0000 - accuracy: 0.9530 - precision: 0.9655 - recall: 0.9395 - auc: 0.9909 - val_loss: 0.0745 - val_tp: 74.0000 - val_fp: 822.0000 - val_tn: 44671.0000 - val_fn: 2.0000 - val_accuracy: 0.9819 - val_precision: 0.0826 - val_recall: 0.9737 - val_auc: 0.9978 Epoch 6/100 278/278 [==============================] - 7s 24ms/step - loss: 0.1175 - tp: 268432.0000 - fp: 8918.0000 - tn: 276042.0000 - fn: 15952.0000 - accuracy: 0.9563 - precision: 0.9678 - recall: 0.9439 - auc: 0.9926 - val_loss: 0.0663 - val_tp: 74.0000 - val_fp: 750.0000 - val_tn: 44743.0000 - val_fn: 2.0000 - val_accuracy: 0.9835 - val_precision: 0.0898 - val_recall: 0.9737 - val_auc: 0.9972 Epoch 7/100 278/278 [==============================] - 7s 24ms/step - loss: 0.1088 - tp: 269482.0000 - fp: 8208.0000 - tn: 276504.0000 - fn: 15150.0000 - accuracy: 0.9590 - precision: 0.9704 - recall: 0.9468 - auc: 0.9938 - val_loss: 0.0595 - val_tp: 74.0000 - val_fp: 703.0000 - val_tn: 44790.0000 - val_fn: 2.0000 - val_accuracy: 0.9845 - val_precision: 0.0952 - val_recall: 0.9737 - val_auc: 0.9969 Epoch 8/100 278/278 [==============================] - 7s 24ms/step - loss: 0.1040 - tp: 269902.0000 - fp: 7829.0000 - tn: 276891.0000 - fn: 14722.0000 - accuracy: 0.9604 - precision: 0.9718 - recall: 0.9483 - auc: 0.9943 - val_loss: 0.0527 - val_tp: 74.0000 - val_fp: 634.0000 - val_tn: 44859.0000 - val_fn: 2.0000 - val_accuracy: 0.9860 - val_precision: 0.1045 - val_recall: 0.9737 - val_auc: 0.9962 Epoch 9/100 278/278 [==============================] - 7s 24ms/step - loss: 0.0992 - tp: 271117.0000 - fp: 7567.0000 - tn: 276382.0000 - fn: 14278.0000 - accuracy: 0.9616 - precision: 0.9728 - recall: 0.9500 - auc: 0.9948 - val_loss: 0.0509 - val_tp: 74.0000 - val_fp: 651.0000 - val_tn: 44842.0000 - val_fn: 2.0000 - val_accuracy: 0.9857 - val_precision: 0.1021 - val_recall: 0.9737 - val_auc: 0.9959 Epoch 10/100 278/278 [==============================] - 7s 24ms/step - loss: 0.0960 - tp: 270783.0000 - fp: 7337.0000 - tn: 277379.0000 - fn: 13845.0000 - accuracy: 0.9628 - precision: 0.9736 - recall: 0.9514 - auc: 0.9950 - val_loss: 0.0467 - val_tp: 74.0000 - val_fp: 612.0000 - val_tn: 44881.0000 - val_fn: 2.0000 - val_accuracy: 0.9865 - val_precision: 0.1079 - val_recall: 0.9737 - val_auc: 0.9959 Epoch 11/100 278/278 [==============================] - ETA: 0s - loss: 0.0927 - tp: 271368.0000 - fp: 7017.0000 - tn: 277337.0000 - fn: 13622.0000 - accuracy: 0.9637 - precision: 0.9748 - recall: 0.9522 - auc: 0.9954Restoring model weights from the end of the best epoch. 278/278 [==============================] - 7s 25ms/step - loss: 0.0927 - tp: 271368.0000 - fp: 7017.0000 - tn: 277337.0000 - fn: 13622.0000 - accuracy: 0.9637 - precision: 0.9748 - recall: 0.9522 - auc: 0.9954 - val_loss: 0.0434 - val_tp: 74.0000 - val_fp: 555.0000 - val_tn: 44938.0000 - val_fn: 2.0000 - val_accuracy: 0.9878 - val_precision: 0.1176 - val_recall: 0.9737 - val_auc: 0.9957 Epoch 00011: early stopping ``` If the training process were considering the whole dataset on each gradient update, this oversampling would be basically identical to the class weighting. But when training the model batch-wise, as you did here, the oversampled data provides a smoother gradient signal: Instead of each positive example being shown in one batch with a large weight, they're shown in many different batches each time with a small weight. This smoother gradient signal makes it easier to train the model. ### Check training history Note that the distributions of metrics will be different here, because the training data has a totally different distribution from the validation and test data. ```py plot_metrics(resampled_history) ``` ![png](img/6e877844629f2888009486ac2f5ece0b.png) ### Re-train Because training is easier on the balanced data, the above training procedure may overfit quickly. So break up the epochs to give the [`callbacks.EarlyStopping`](https://tensorflow.google.cn/api_docs/python/tf/keras/callbacks/EarlyStopping) finer control over when to stop training. ```py resampled_model = make_model() resampled_model.load_weights(initial_weights) # Reset the bias to zero, since this dataset is balanced. output_layer = resampled_model.layers[-1] output_layer.bias.assign([0]) resampled_history = resampled_model.fit( resampled_ds, # These are not real epochs steps_per_epoch=20, epochs=10*EPOCHS, callbacks=[early_stopping], validation_data=(val_ds)) ``` ```py Epoch 1/1000 20/20 [==============================] - 1s 56ms/step - loss: 0.9282 - tp: 14181.0000 - fp: 14018.0000 - tn: 51871.0000 - fn: 6459.0000 - accuracy: 0.7634 - precision: 0.5029 - recall: 0.6871 - auc: 0.8612 - val_loss: 0.9177 - val_tp: 68.0000 - val_fp: 32432.0000 - val_tn: 13061.0000 - val_fn: 8.0000 - val_accuracy: 0.2881 - val_precision: 0.0021 - val_recall: 0.8947 - val_auc: 0.8450 Epoch 2/1000 20/20 [==============================] - 1s 26ms/step - loss: 0.7143 - tp: 16471.0000 - fp: 12416.0000 - tn: 7841.0000 - fn: 4232.0000 - accuracy: 0.5936 - precision: 0.5702 - recall: 0.7956 - auc: 0.7514 - val_loss: 0.8173 - val_tp: 70.0000 - val_fp: 28250.0000 - val_tn: 17243.0000 - val_fn: 6.0000 - val_accuracy: 0.3799 - val_precision: 0.0025 - val_recall: 0.9211 - val_auc: 0.9056 Epoch 3/1000 20/20 [==============================] - 1s 26ms/step - loss: 0.5979 - tp: 17119.0000 - fp: 11055.0000 - tn: 9528.0000 - fn: 3258.0000 - accuracy: 0.6506 - precision: 0.6076 - recall: 0.8401 - auc: 0.8194 - val_loss: 0.7158 - val_tp: 70.0000 - val_fp: 22377.0000 - val_tn: 23116.0000 - val_fn: 6.0000 - val_accuracy: 0.5088 - val_precision: 0.0031 - val_recall: 0.9211 - val_auc: 0.9283 Epoch 4/1000 20/20 [==============================] - 1s 28ms/step - loss: 0.5265 - tp: 17496.0000 - fp: 9591.0000 - tn: 10955.0000 - fn: 2918.0000 - accuracy: 0.6946 - precision: 0.6459 - recall: 0.8571 - auc: 0.8542 - val_loss: 0.6276 - val_tp: 72.0000 - val_fp: 16226.0000 - val_tn: 29267.0000 - val_fn: 4.0000 - val_accuracy: 0.6438 - val_precision: 0.0044 - val_recall: 0.9474 - val_auc: 0.9576 Epoch 5/1000 20/20 [==============================] - 1s 28ms/step - loss: 0.4682 - tp: 17653.0000 - fp: 8002.0000 - tn: 12532.0000 - fn: 2773.0000 - accuracy: 0.7369 - precision: 0.6881 - recall: 0.8642 - auc: 0.8793 - val_loss: 0.5534 - val_tp: 75.0000 - val_fp: 10889.0000 - val_tn: 34604.0000 - val_fn: 1.0000 - val_accuracy: 0.7610 - val_precision: 0.0068 - val_recall: 0.9868 - val_auc: 0.9848 Epoch 6/1000 20/20 [==============================] - 1s 28ms/step - loss: 0.4174 - tp: 17837.0000 - fp: 6635.0000 - tn: 13890.0000 - fn: 2598.0000 - accuracy: 0.7746 - precision: 0.7289 - recall: 0.8729 - auc: 0.9003 - val_loss: 0.4905 - val_tp: 75.0000 - val_fp: 6493.0000 - val_tn: 39000.0000 - val_fn: 1.0000 - val_accuracy: 0.8575 - val_precision: 0.0114 - val_recall: 0.9868 - val_auc: 0.9922 Epoch 7/1000 20/20 [==============================] - 1s 28ms/step - loss: 0.3787 - tp: 17939.0000 - fp: 5276.0000 - tn: 15251.0000 - fn: 2494.0000 - accuracy: 0.8103 - precision: 0.7727 - recall: 0.8779 - auc: 0.9162 - val_loss: 0.4380 - val_tp: 75.0000 - val_fp: 4314.0000 - val_tn: 41179.0000 - val_fn: 1.0000 - val_accuracy: 0.9053 - val_precision: 0.0171 - val_recall: 0.9868 - val_auc: 0.9954 Epoch 8/1000 20/20 [==============================] - 1s 28ms/step - loss: 0.3537 - tp: 18025.0000 - fp: 4431.0000 - tn: 15994.0000 - fn: 2510.0000 - accuracy: 0.8305 - precision: 0.8027 - recall: 0.8778 - auc: 0.9242 - val_loss: 0.3942 - val_tp: 75.0000 - val_fp: 3206.0000 - val_tn: 42287.0000 - val_fn: 1.0000 - val_accuracy: 0.9296 - val_precision: 0.0229 - val_recall: 0.9868 - val_auc: 0.9972 Epoch 9/1000 20/20 [==============================] - 1s 27ms/step - loss: 0.3290 - tp: 17975.0000 - fp: 3746.0000 - tn: 16823.0000 - fn: 2416.0000 - accuracy: 0.8496 - precision: 0.8275 - recall: 0.8815 - auc: 0.9341 - val_loss: 0.3560 - val_tp: 75.0000 - val_fp: 2593.0000 - val_tn: 42900.0000 - val_fn: 1.0000 - val_accuracy: 0.9431 - val_precision: 0.0281 - val_recall: 0.9868 - val_auc: 0.9979 Epoch 10/1000 20/20 [==============================] - 1s 27ms/step - loss: 0.3121 - tp: 18157.0000 - fp: 3263.0000 - tn: 17154.0000 - fn: 2386.0000 - accuracy: 0.8621 - precision: 0.8477 - recall: 0.8839 - auc: 0.9397 - val_loss: 0.3239 - val_tp: 75.0000 - val_fp: 2212.0000 - val_tn: 43281.0000 - val_fn: 1.0000 - val_accuracy: 0.9514 - val_precision: 0.0328 - val_recall: 0.9868 - val_auc: 0.9982 Epoch 11/1000 20/20 [==============================] - 1s 28ms/step - loss: 0.2934 - tp: 18151.0000 - fp: 2867.0000 - tn: 17641.0000 - fn: 2301.0000 - accuracy: 0.8738 - precision: 0.8636 - recall: 0.8875 - auc: 0.9471 - val_loss: 0.2972 - val_tp: 75.0000 - val_fp: 1970.0000 - val_tn: 43523.0000 - val_fn: 1.0000 - val_accuracy: 0.9567 - val_precision: 0.0367 - val_recall: 0.9868 - val_auc: 0.9983 Epoch 12/1000 20/20 [==============================] - 1s 28ms/step - loss: 0.2765 - tp: 18454.0000 - fp: 2498.0000 - tn: 17838.0000 - fn: 2170.0000 - accuracy: 0.8860 - precision: 0.8808 - recall: 0.8948 - auc: 0.9537 - val_loss: 0.2727 - val_tp: 75.0000 - val_fp: 1763.0000 - val_tn: 43730.0000 - val_fn: 1.0000 - val_accuracy: 0.9613 - val_precision: 0.0408 - val_recall: 0.9868 - val_auc: 0.9984 Epoch 13/1000 20/20 [==============================] - 1s 27ms/step - loss: 0.2649 - tp: 18355.0000 - fp: 2240.0000 - tn: 18272.0000 - fn: 2093.0000 - accuracy: 0.8942 - precision: 0.8912 - recall: 0.8976 - auc: 0.9577 - val_loss: 0.2528 - val_tp: 75.0000 - val_fp: 1638.0000 - val_tn: 43855.0000 - val_fn: 1.0000 - val_accuracy: 0.9640 - val_precision: 0.0438 - val_recall: 0.9868 - val_auc: 0.9984 Epoch 14/1000 20/20 [==============================] - 1s 28ms/step - loss: 0.2547 - tp: 18293.0000 - fp: 2013.0000 - tn: 18577.0000 - fn: 2077.0000 - accuracy: 0.9001 - precision: 0.9009 - recall: 0.8980 - auc: 0.9609 - val_loss: 0.2338 - val_tp: 75.0000 - val_fp: 1516.0000 - val_tn: 43977.0000 - val_fn: 1.0000 - val_accuracy: 0.9667 - val_precision: 0.0471 - val_recall: 0.9868 - val_auc: 0.9985 Epoch 15/1000 20/20 [==============================] - 1s 27ms/step - loss: 0.2414 - tp: 18414.0000 - fp: 1709.0000 - tn: 18760.0000 - fn: 2077.0000 - accuracy: 0.9076 - precision: 0.9151 - recall: 0.8986 - auc: 0.9641 - val_loss: 0.2187 - val_tp: 75.0000 - val_fp: 1462.0000 - val_tn: 44031.0000 - val_fn: 1.0000 - val_accuracy: 0.9679 - val_precision: 0.0488 - val_recall: 0.9868 - val_auc: 0.9984 Epoch 16/1000 20/20 [==============================] - 1s 27ms/step - loss: 0.2386 - tp: 18465.0000 - fp: 1792.0000 - tn: 18740.0000 - fn: 1963.0000 - accuracy: 0.9083 - precision: 0.9115 - recall: 0.9039 - auc: 0.9660 - val_loss: 0.2044 - val_tp: 75.0000 - val_fp: 1383.0000 - val_tn: 44110.0000 - val_fn: 1.0000 - val_accuracy: 0.9696 - val_precision: 0.0514 - val_recall: 0.9868 - val_auc: 0.9984 Epoch 17/1000 20/20 [==============================] - 1s 27ms/step - loss: 0.2290 - tp: 18685.0000 - fp: 1578.0000 - tn: 18675.0000 - fn: 2022.0000 - accuracy: 0.9121 - precision: 0.9221 - recall: 0.9024 - auc: 0.9673 - val_loss: 0.1913 - val_tp: 75.0000 - val_fp: 1297.0000 - val_tn: 44196.0000 - val_fn: 1.0000 - val_accuracy: 0.9715 - val_precision: 0.0547 - val_recall: 0.9868 - val_auc: 0.9983 Epoch 18/1000 20/20 [==============================] - 1s 27ms/step - loss: 0.2170 - tp: 18526.0000 - fp: 1453.0000 - tn: 19039.0000 - fn: 1942.0000 - accuracy: 0.9171 - precision: 0.9273 - recall: 0.9051 - auc: 0.9714 - val_loss: 0.1812 - val_tp: 75.0000 - val_fp: 1266.0000 - val_tn: 44227.0000 - val_fn: 1.0000 - val_accuracy: 0.9722 - val_precision: 0.0559 - val_recall: 0.9868 - val_auc: 0.9983 Epoch 19/1000 20/20 [==============================] - 1s 28ms/step - loss: 0.2160 - tp: 18362.0000 - fp: 1402.0000 - tn: 19310.0000 - fn: 1886.0000 - accuracy: 0.9197 - precision: 0.9291 - recall: 0.9069 - auc: 0.9715 - val_loss: 0.1716 - val_tp: 75.0000 - val_fp: 1219.0000 - val_tn: 44274.0000 - val_fn: 1.0000 - val_accuracy: 0.9732 - val_precision: 0.0580 - val_recall: 0.9868 - val_auc: 0.9982 Epoch 20/1000 20/20 [==============================] - 1s 28ms/step - loss: 0.2081 - tp: 18722.0000 - fp: 1297.0000 - tn: 19031.0000 - fn: 1910.0000 - accuracy: 0.9217 - precision: 0.9352 - recall: 0.9074 - auc: 0.9735 - val_loss: 0.1633 - val_tp: 75.0000 - val_fp: 1192.0000 - val_tn: 44301.0000 - val_fn: 1.0000 - val_accuracy: 0.9738 - val_precision: 0.0592 - val_recall: 0.9868 - val_auc: 0.9981 Epoch 21/1000 20/20 [==============================] - 1s 27ms/step - loss: 0.2041 - tp: 18929.0000 - fp: 1263.0000 - tn: 18884.0000 - fn: 1884.0000 - accuracy: 0.9232 - precision: 0.9375 - recall: 0.9095 - auc: 0.9742 - val_loss: 0.1571 - val_tp: 75.0000 - val_fp: 1187.0000 - val_tn: 44306.0000 - val_fn: 1.0000 - val_accuracy: 0.9739 - val_precision: 0.0594 - val_recall: 0.9868 - val_auc: 0.9980 Epoch 22/1000 20/20 [==============================] - 1s 27ms/step - loss: 0.2017 - tp: 18834.0000 - fp: 1224.0000 - tn: 19064.0000 - fn: 1838.0000 - accuracy: 0.9252 - precision: 0.9390 - recall: 0.9111 - auc: 0.9752 - val_loss: 0.1508 - val_tp: 75.0000 - val_fp: 1162.0000 - val_tn: 44331.0000 - val_fn: 1.0000 - val_accuracy: 0.9745 - val_precision: 0.0606 - val_recall: 0.9868 - val_auc: 0.9980 Epoch 23/1000 20/20 [==============================] - 1s 28ms/step - loss: 0.1951 - tp: 18612.0000 - fp: 1127.0000 - tn: 19424.0000 - fn: 1797.0000 - accuracy: 0.9286 - precision: 0.9429 - recall: 0.9120 - auc: 0.9764 - val_loss: 0.1458 - val_tp: 75.0000 - val_fp: 1158.0000 - val_tn: 44335.0000 - val_fn: 1.0000 - val_accuracy: 0.9746 - val_precision: 0.0608 - val_recall: 0.9868 - val_auc: 0.9979 Epoch 24/1000 18/20 [==========================>...] - ETA: 0s - loss: 0.1945 - tp: 16932.0000 - fp: 1065.0000 - tn: 17301.0000 - fn: 1566.0000 - accuracy: 0.9286 - precision: 0.9408 - recall: 0.9153 - auc: 0.9765Restoring model weights from the end of the best epoch. 20/20 [==============================] - 1s 27ms/step - loss: 0.1943 - tp: 18796.0000 - fp: 1179.0000 - tn: 19226.0000 - fn: 1759.0000 - accuracy: 0.9283 - precision: 0.9410 - recall: 0.9144 - auc: 0.9765 - val_loss: 0.1401 - val_tp: 75.0000 - val_fp: 1136.0000 - val_tn: 44357.0000 - val_fn: 1.0000 - val_accuracy: 0.9750 - val_precision: 0.0619 - val_recall: 0.9868 - val_auc: 0.9979 Epoch 00024: early stopping ``` ### Re-check training history ```py plot_metrics(resampled_history) ``` ![png](img/217c4e55f89d4a55a78dc082bbdc1e2f.png) ### Evaluate metrics ```py train_predictions_resampled = resampled_model.predict(train_features, batch_size=BATCH_SIZE) test_predictions_resampled = resampled_model.predict(test_features, batch_size=BATCH_SIZE) ``` ```py resampled_results = resampled_model.evaluate(test_features, test_labels, batch_size=BATCH_SIZE, verbose=0) for name, value in zip(resampled_model.metrics_names, resampled_results): print(name, ': ', value) print() plot_cm(test_labels, test_predictions_resampled) ``` ```py loss : 0.23386089503765106 tp : 91.0 fp : 1892.0 tn : 54971.0 fn : 8.0 accuracy : 0.9666444063186646 precision : 0.045890066772699356 recall : 0.9191918969154358 auc : 0.977620542049408 Legitimate Transactions Detected (True Negatives): 54971 Legitimate Transactions Incorrectly Detected (False Positives): 1892 Fraudulent Transactions Missed (False Negatives): 8 Fraudulent Transactions Detected (True Positives): 91 Total Fraudulent Transactions: 99 ``` ![png](img/5224f36bbf08c7f64fa501011640042b.png) ### Plot the ROC ```py plot_roc("Train Baseline", train_labels, train_predictions_baseline, color=colors[0]) plot_roc("Test Baseline", test_labels, test_predictions_baseline, color=colors[0], linestyle='--') plot_roc("Train Weighted", train_labels, train_predictions_weighted, color=colors[1]) plot_roc("Test Weighted", test_labels, test_predictions_weighted, color=colors[1], linestyle='--') plot_roc("Train Resampled", train_labels, train_predictions_resampled, color=colors[2]) plot_roc("Test Resampled", test_labels, test_predictions_resampled, color=colors[2], linestyle='--') plt.legend(loc='lower right') ``` ```py ``` ![png](img/f1fa73dd5b685549afd6264592919903.png) ## Applying this tutorial to your problem Imbalanced data classification is an inherently difficult task since there are so few samples to learn from. You should always start with the data first and do your best to collect as many samples as possible and give substantial thought to what features may be relevant so the model can get the most out of your minority class. At some point your model may struggle to improve and yield the results you want, so it is important to keep in mind the context of your problem and the trade offs between different types of errors.