Customized models and datasets for structured inputs

In this tutorial we will: - Show how to add a model for structural classification - Show how to add a dataloader with structured data preprocessing

We will be using the Northpointe’s Correctional Offender Management Profiling for Alternative Sanctions (COMPAS) score, used in states like California and Florida.

Installation

Again, the first step will be installing our libarary

[1]:

!pip install fairlib

Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/
Collecting fairlib
  Downloading fairlib-0.0.9-py3-none-any.whl (85 kB)
     |████████████████████████████████| 85 kB 2.6 MB/s
Requirement already satisfied: PyYAML in /usr/local/lib/python3.7/dist-packages (from fairlib) (3.13)
Requirement already satisfied: torch in /usr/local/lib/python3.7/dist-packages (from fairlib) (1.12.0+cu113)
Collecting transformers
  Downloading transformers-4.20.1-py3-none-any.whl (4.4 MB)
     |████████████████████████████████| 4.4 MB 22.6 MB/s
Requirement already satisfied: seaborn in /usr/local/lib/python3.7/dist-packages (from fairlib) (0.11.2)
Requirement already satisfied: scikit-learn in /usr/local/lib/python3.7/dist-packages (from fairlib) (1.0.2)
Requirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from fairlib) (4.64.0)
Requirement already satisfied: docopt in /usr/local/lib/python3.7/dist-packages (from fairlib) (0.6.2)
Collecting sacremoses
  Downloading sacremoses-0.0.53.tar.gz (880 kB)
     |████████████████████████████████| 880 kB 62.2 MB/s
Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from fairlib) (1.21.6)
Requirement already satisfied: pandas in /usr/local/lib/python3.7/dist-packages (from fairlib) (1.3.5)
Requirement already satisfied: matplotlib in /usr/local/lib/python3.7/dist-packages (from fairlib) (3.2.2)
Collecting pickle5
  Downloading pickle5-0.0.12-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (256 kB)
     |████████████████████████████████| 256 kB 47.9 MB/s
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->fairlib) (3.0.9)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->fairlib) (1.4.4)
Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->fairlib) (2.8.2)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib->fairlib) (0.11.0)
Requirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from kiwisolver>=1.0.1->matplotlib->fairlib) (4.1.1)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil>=2.1->matplotlib->fairlib) (1.15.0)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.7/dist-packages (from pandas->fairlib) (2022.1)
Requirement already satisfied: regex in /usr/local/lib/python3.7/dist-packages (from sacremoses->fairlib) (2022.6.2)
Requirement already satisfied: click in /usr/local/lib/python3.7/dist-packages (from sacremoses->fairlib) (7.1.2)
Requirement already satisfied: joblib in /usr/local/lib/python3.7/dist-packages (from sacremoses->fairlib) (1.1.0)
Requirement already satisfied: scipy>=1.1.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn->fairlib) (1.7.3)
Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn->fairlib) (3.1.0)
Collecting huggingface-hub<1.0,>=0.1.0
  Downloading huggingface_hub-0.8.1-py3-none-any.whl (101 kB)
     |████████████████████████████████| 101 kB 11.5 MB/s
Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.7/dist-packages (from transformers->fairlib) (21.3)
Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from transformers->fairlib) (2.23.0)
Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.7/dist-packages (from transformers->fairlib) (4.12.0)
Requirement already satisfied: filelock in /usr/local/lib/python3.7/dist-packages (from transformers->fairlib) (3.7.1)
Collecting PyYAML
  Downloading PyYAML-6.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (596 kB)
     |████████████████████████████████| 596 kB 11.8 MB/s
Collecting tokenizers!=0.11.3,<0.13,>=0.11.1
  Downloading tokenizers-0.12.1-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (6.6 MB)
     |████████████████████████████████| 6.6 MB 33.9 MB/s
Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata->transformers->fairlib) (3.8.1)
Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->transformers->fairlib) (2.10)
Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->transformers->fairlib) (3.0.4)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->transformers->fairlib) (1.24.3)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->transformers->fairlib) (2022.6.15)
Building wheels for collected packages: sacremoses
  Building wheel for sacremoses (setup.py) ... done
  Created wheel for sacremoses: filename=sacremoses-0.0.53-py3-none-any.whl size=895260 sha256=9e7ef0aea29edc17a4bd4ba5d5f00a2eb51f47d5ed5f7ec133443d60dce94969
  Stored in directory: /root/.cache/pip/wheels/87/39/dd/a83eeef36d0bf98e7a4d1933a4ad2d660295a40613079bafc9
Successfully built sacremoses
Installing collected packages: PyYAML, tokenizers, huggingface-hub, transformers, sacremoses, pickle5, fairlib
  Attempting uninstall: PyYAML
    Found existing installation: PyYAML 3.13
    Uninstalling PyYAML-3.13:
      Successfully uninstalled PyYAML-3.13
Successfully installed PyYAML-6.0 fairlib-0.0.9 huggingface-hub-0.8.1 pickle5-0.0.12 sacremoses-0.0.53 tokenizers-0.12.1 transformers-4.20.1

[2]:

import fairlib

Download and preprocess the COMPAS dataset

https://github.com/google-research/google-research/blob/master/group_agnostic_fairness/data_utils/CreateCompasDatasetFiles.ipynb

[3]:

from fairlib import datasets
datasets.prepare_dataset("compas", "data")

saving to /content/data/compas-scores-two-years.csv

[4]:

import pandas as pd

pd.read_pickle("data/COMPAS_dev.pkl").keys()

[4]:

Index(['juv_fel_count', 'juv_misd_count', 'juv_other_count', 'priors_count',
       'age', 'c_charge_degree_0', 'c_charge_degree_1', 'c_charge_desc_262',
       'c_charge_desc_43', 'c_charge_desc_55',
       ...
       'c_charge_desc_323', 'c_charge_desc_66', 'c_charge_desc_40',
       'c_charge_desc_33', 'age_cat_1', 'age_cat_0', 'age_cat_2', 'sex',
       'race', 'is_recid'],
      dtype='object', length=450)

Train Models

[5]:

from fairlib import networks, BaseOptions, dataloaders
import torch

[6]:

Shared_options = {
    # The name of the dataset, correponding dataloader will be used,
    "dataset":  "COMPAS",

    # Specifiy the path to the input data
    "data_dir": "./data",

    # Device for computing, -1 is the cpu
    "device_id": -1,

    # The default path for saving experimental results
    "results_dir":  r"results",

    # The same as the dataset
    "project_dir":  r"dev",

    # We will focusing on TPR GAP, implying the Equalized Odds for binay classification.
    "GAP_metric_name":  "TPR_GAP",

    # The overall performance will be measured as accuracy
    "Performance_metric_name":  "accuracy",

    # Model selections are based on DTO
    "selection_criterion":  "DTO",

    # Default dirs for saving checkpoints
    "checkpoint_dir":   "models",
    "checkpoint_name":  "checkpoint_epoch",


    "n_jobs":   1,
}

[7]:

args = {
    "dataset":Shared_options["dataset"],
    "data_dir":Shared_options["data_dir"],
    "device_id":Shared_options["device_id"],

    # Give a name to the exp, which will be used in the path
    "exp_id":"vanilla",

    "emb_size": 450-3,
    "lr": 0.001,
    "batch_size": 128,
    "hidden_size": 32,
    "n_hidden": 1,
    "activation_function": "ReLu",

    "num_classes": 2,
    "num_groups": 3, # Balck; White; and Other
}

# Init the argument
options = BaseOptions()
state = options.get_state(args=args, silence=True)

INFO:root:Unexpected args: ['-f', '/root/.local/share/jupyter/runtime/kernel-1cc80252-6b1d-427f-a63d-8b70911dfc3b.json']
INFO:root:Logging to ./results/dev/COMPAS/vanilla/output.log

2022-07-21 07:10:35 [INFO ]  ======================================== 2022-07-21 07:10:35 ========================================
2022-07-21 07:10:35 [INFO ]  Base directory is ./results/dev/COMPAS/vanilla
2022-07-21 07:10:35 [INFO ]  Exception type : AssertionError
2022-07-21 07:10:35 [INFO ]  Exception message : Not implemented
2022-07-21 07:10:35 [INFO ]  Stack trace : ['File : /usr/local/lib/python3.7/dist-packages/fairlib/src/base_options.py , Line : 486, Func.Name : set_state, Message : train_iterator, dev_iterator, test_iterator = dataloaders.get_dataloaders(state)', 'File : /usr/local/lib/python3.7/dist-packages/fairlib/src/dataloaders/__init__.py , Line : 40, Func.Name : get_dataloaders, Message : ], "Not implemented"']
2022-07-21 07:10:35 [INFO ]  dataloaders need to be initialized!

Customize dataset loader

[8]:

import os
import pandas as pd
import numpy as np

[9]:

class CustomizedDataset(dataloaders.utils.BaseDataset):

    def load_data(self):

        self.data_dir = os.path.join(self.args.data_dir, "COMPAS_{}.pkl".format(self.split))

        data = pd.read_pickle(self.data_dir)

        self.X = data.drop(['sex', 'race', 'is_recid'], axis=1).to_numpy().astype(np.float32)
        self.y = list(data["is_recid"])
        self.protected_label = list(data["race"])

[10]:

customized_train_data = CustomizedDataset(args=state, split="train")
customized_dev_data = CustomizedDataset(args=state, split="dev")
customized_test_data = CustomizedDataset(args=state, split="test")

# DataLoader Parameters
tran_dataloader_params = {
        'batch_size': state.batch_size,
        'shuffle': True,
        'num_workers': state.num_workers}

eval_dataloader_params = {
        'batch_size': state.test_batch_size,
        'shuffle': False,
        'num_workers': state.num_workers}

# init dataloader
customized_training_generator = torch.utils.data.DataLoader(customized_train_data, **tran_dataloader_params)
customized_validation_generator = torch.utils.data.DataLoader(customized_dev_data, **eval_dataloader_params)
customized_test_generator = torch.utils.data.DataLoader(customized_test_data, **eval_dataloader_params)

Loaded data shapes: (4544, 447), (4544,), (4544,)
Loaded data shapes: (505, 447), (505,), (505,)
Loaded data shapes: (2165, 447), (2165,), (2165,)

[11]:

model = networks.classifier.MLP(state)

2022-07-21 07:10:35 [INFO ]  MLP(
2022-07-21 07:10:35 [INFO ]    (output_layer): Linear(in_features=32, out_features=2, bias=True)
2022-07-21 07:10:35 [INFO ]    (AF): ReLU()
2022-07-21 07:10:35 [INFO ]    (hidden_layers): ModuleList(
2022-07-21 07:10:35 [INFO ]      (0): Linear(in_features=447, out_features=32, bias=True)
2022-07-21 07:10:35 [INFO ]      (1): ReLU()
2022-07-21 07:10:35 [INFO ]    )
2022-07-21 07:10:35 [INFO ]    (criterion): CrossEntropyLoss()
2022-07-21 07:10:35 [INFO ]  )
2022-07-21 07:10:35 [INFO ]  Total number of parameters: 14402

[12]:

model.train_self(
    train_generator = customized_training_generator,
    dev_generator = customized_validation_generator,
    test_generator = customized_test_generator,
)

2022-07-21 07:10:35 [INFO ]  Epoch:    0 [      0/   4544 ( 0%)]        Loss: 0.6946     Data Time: 0.05s       Train Time: 0.12s
2022-07-21 07:10:35 [INFO ]  Evaluation at Epoch 0
2022-07-21 07:10:35 [INFO ]  Validation accuracy: 65.35 macro_fscore: 64.41     micro_fscore: 65.35     TPR_GAP: 37.09  FPR_GAP: 37.09  PPR_GAP: 38.61
2022-07-21 07:10:35 [INFO ]  Test accuracy: 67.30       macro_fscore: 66.29     micro_fscore: 67.30     TPR_GAP: 28.96  FPR_GAP: 28.96  PPR_GAP: 33.07
2022-07-21 07:10:35 [INFO ]  Epoch:    1 [      0/   4544 ( 0%)]        Loss: 0.6375     Data Time: 0.00s       Train Time: 0.00s
2022-07-21 07:10:35 [INFO ]  Evaluation at Epoch 1
2022-07-21 07:10:35 [INFO ]  Validation accuracy: 66.14 macro_fscore: 65.92     micro_fscore: 66.14     TPR_GAP: 30.20  FPR_GAP: 30.20  PPR_GAP: 32.54
2022-07-21 07:10:35 [INFO ]  Test accuracy: 67.53       macro_fscore: 67.12     micro_fscore: 67.53     TPR_GAP: 31.45  FPR_GAP: 31.45  PPR_GAP: 36.71
2022-07-21 07:10:35 [INFO ]  Epoch:    2 [      0/   4544 ( 0%)]        Loss: 0.5948     Data Time: 0.01s       Train Time: 0.00s
2022-07-21 07:10:36 [INFO ]  Evaluation at Epoch 2
2022-07-21 07:10:36 [INFO ]  Validation accuracy: 66.14 macro_fscore: 66.00     micro_fscore: 66.14     TPR_GAP: 31.74  FPR_GAP: 31.74  PPR_GAP: 34.83
2022-07-21 07:10:36 [INFO ]  Test accuracy: 68.55       macro_fscore: 68.24     micro_fscore: 68.55     TPR_GAP: 32.02  FPR_GAP: 32.02  PPR_GAP: 38.27
2022-07-21 07:10:36 [INFO ]  Epoch:    3 [      0/   4544 ( 0%)]        Loss: 0.6298     Data Time: 0.01s       Train Time: 0.00s
2022-07-21 07:10:36 [INFO ]  Epochs since last improvement: 1
2022-07-21 07:10:36 [INFO ]  Evaluation at Epoch 3
2022-07-21 07:10:36 [INFO ]  Validation accuracy: 65.94 macro_fscore: 65.71     micro_fscore: 65.94     TPR_GAP: 35.41  FPR_GAP: 35.41  PPR_GAP: 38.06
2022-07-21 07:10:36 [INFO ]  Test accuracy: 68.96       macro_fscore: 68.54     micro_fscore: 68.96     TPR_GAP: 32.02  FPR_GAP: 32.02  PPR_GAP: 37.86
2022-07-21 07:10:36 [INFO ]  Epoch:    4 [      0/   4544 ( 0%)]        Loss: 0.5868     Data Time: 0.00s       Train Time: 0.00s
2022-07-21 07:10:36 [INFO ]  Epochs since last improvement: 2
2022-07-21 07:10:36 [INFO ]  Evaluation at Epoch 4
2022-07-21 07:10:36 [INFO ]  Validation accuracy: 65.35 macro_fscore: 65.32     micro_fscore: 65.35     TPR_GAP: 35.10  FPR_GAP: 35.10  PPR_GAP: 38.32
2022-07-21 07:10:36 [INFO ]  Test accuracy: 68.91       macro_fscore: 68.80     micro_fscore: 68.91     TPR_GAP: 33.54  FPR_GAP: 33.54  PPR_GAP: 40.45
2022-07-21 07:10:36 [INFO ]  Epoch:    5 [      0/   4544 ( 0%)]        Loss: 0.5572     Data Time: 0.00s       Train Time: 0.00s
2022-07-21 07:10:36 [INFO ]  Epochs since last improvement: 3
2022-07-21 07:10:36 [INFO ]  Evaluation at Epoch 5
2022-07-21 07:10:36 [INFO ]  Validation accuracy: 65.54 macro_fscore: 65.24     micro_fscore: 65.54     TPR_GAP: 35.02  FPR_GAP: 35.02  PPR_GAP: 37.96
2022-07-21 07:10:36 [INFO ]  Test accuracy: 68.96       macro_fscore: 68.47     micro_fscore: 68.96     TPR_GAP: 30.91  FPR_GAP: 30.91  PPR_GAP: 36.59
2022-07-21 07:10:36 [INFO ]  Epoch:    6 [      0/   4544 ( 0%)]        Loss: 0.5651     Data Time: 0.00s       Train Time: 0.00s
2022-07-21 07:10:36 [INFO ]  Epochs since last improvement: 4
2022-07-21 07:10:36 [INFO ]  Evaluation at Epoch 6
2022-07-21 07:10:36 [INFO ]  Validation accuracy: 64.36 macro_fscore: 64.22     micro_fscore: 64.36     TPR_GAP: 34.11  FPR_GAP: 34.11  PPR_GAP: 36.69
2022-07-21 07:10:36 [INFO ]  Test accuracy: 69.01       macro_fscore: 68.75     micro_fscore: 69.01     TPR_GAP: 30.58  FPR_GAP: 30.58  PPR_GAP: 37.35
2022-07-21 07:10:36 [INFO ]  Epoch:    7 [      0/   4544 ( 0%)]        Loss: 0.6300     Data Time: 0.00s       Train Time: 0.00s
2022-07-21 07:10:36 [INFO ]  Epochs since last improvement: 5
2022-07-21 07:10:36 [INFO ]  Evaluation at Epoch 7
2022-07-21 07:10:37 [INFO ]  Validation accuracy: 64.75 macro_fscore: 64.75     micro_fscore: 64.75     TPR_GAP: 31.62  FPR_GAP: 31.62  PPR_GAP: 34.74
2022-07-21 07:10:37 [INFO ]  Test accuracy: 68.68       macro_fscore: 68.59     micro_fscore: 68.68     TPR_GAP: 31.42  FPR_GAP: 31.42  PPR_GAP: 38.69

[13]:

debiasing_args = {
    "dataset":Shared_options["dataset"],
    "data_dir":Shared_options["data_dir"],
    "device_id":Shared_options["device_id"],

    # Give a name to the exp, which will be used in the path
    "exp_id":"BT_Adv",

    "emb_size": 450-3,
    "lr": 0.001,
    "batch_size": 128,
    "hidden_size": 32,
    "n_hidden": 1,
    "activation_function": "ReLu",

    "num_classes": 2,
    "num_groups": 3, # Balck; White; and Other

    # Perform adversarial training if True
    "adv_debiasing":True,

    # Specify the hyperparameters for Balanced Training
    "BT":"Resampling",
    "BTObj":"EO",
}

# Init the argument
debias_options = BaseOptions()
debias_state = debias_options.get_state(args=debiasing_args, silence=True)

customized_train_data = CustomizedDataset(args=debias_state, split="train")
customized_dev_data = CustomizedDataset(args=debias_state, split="dev")
customized_test_data = CustomizedDataset(args=debias_state, split="test")

# DataLoader Parameters
tran_dataloader_params = {
        'batch_size': state.batch_size,
        'shuffle': True,
        'num_workers': state.num_workers}

eval_dataloader_params = {
        'batch_size': state.test_batch_size,
        'shuffle': False,
        'num_workers': state.num_workers}

# init dataloader
customized_training_generator = torch.utils.data.DataLoader(customized_train_data, **tran_dataloader_params)
customized_validation_generator = torch.utils.data.DataLoader(customized_dev_data, **eval_dataloader_params)
customized_test_generator = torch.utils.data.DataLoader(customized_test_data, **eval_dataloader_params)

debias_model = networks.classifier.MLP(debias_state)

2022-07-21 07:10:37 [INFO ]  Unexpected args: ['-f', '/root/.local/share/jupyter/runtime/kernel-1cc80252-6b1d-427f-a63d-8b70911dfc3b.json']
2022-07-21 07:10:37 [INFO ]  Logging to ./results/dev/COMPAS/BT_Adv/output.log
2022-07-21 07:10:37 [INFO ]  ======================================== 2022-07-21 07:10:37 ========================================
2022-07-21 07:10:37 [INFO ]  Base directory is ./results/dev/COMPAS/BT_Adv
2022-07-21 07:10:37 [INFO ]  Exception type : AssertionError
2022-07-21 07:10:37 [INFO ]  Exception message : Not implemented
2022-07-21 07:10:37 [INFO ]  Stack trace : ['File : /usr/local/lib/python3.7/dist-packages/fairlib/src/base_options.py , Line : 486, Func.Name : set_state, Message : train_iterator, dev_iterator, test_iterator = dataloaders.get_dataloaders(state)', 'File : /usr/local/lib/python3.7/dist-packages/fairlib/src/dataloaders/__init__.py , Line : 40, Func.Name : get_dataloaders, Message : ], "Not implemented"']
2022-07-21 07:10:37 [INFO ]  dataloaders need to be initialized!
2022-07-21 07:10:37 [INFO ]  SubDiscriminator(
2022-07-21 07:10:37 [INFO ]    (grad_rev): GradientReversal()
2022-07-21 07:10:37 [INFO ]    (output_layer): Linear(in_features=300, out_features=3, bias=True)
2022-07-21 07:10:37 [INFO ]    (AF): ReLU()
2022-07-21 07:10:37 [INFO ]    (hidden_layers): ModuleList(
2022-07-21 07:10:37 [INFO ]      (0): Linear(in_features=32, out_features=300, bias=True)
2022-07-21 07:10:37 [INFO ]      (1): ReLU()
2022-07-21 07:10:37 [INFO ]      (2): Linear(in_features=300, out_features=300, bias=True)
2022-07-21 07:10:37 [INFO ]      (3): ReLU()
2022-07-21 07:10:37 [INFO ]    )
2022-07-21 07:10:37 [INFO ]    (criterion): CrossEntropyLoss()
2022-07-21 07:10:37 [INFO ]  )
2022-07-21 07:10:37 [INFO ]  Total number of parameters: 101103

2022-07-21 07:10:37 [INFO ]  Discriminator built!
Loaded data shapes: (4542, 447), (4542,), (4542,)
Loaded data shapes: (505, 447), (505,), (505,)
Loaded data shapes: (2165, 447), (2165,), (2165,)
2022-07-21 07:10:37 [INFO ]  MLP(
2022-07-21 07:10:37 [INFO ]    (output_layer): Linear(in_features=32, out_features=2, bias=True)
2022-07-21 07:10:37 [INFO ]    (AF): ReLU()
2022-07-21 07:10:37 [INFO ]    (hidden_layers): ModuleList(
2022-07-21 07:10:37 [INFO ]      (0): Linear(in_features=447, out_features=32, bias=True)
2022-07-21 07:10:37 [INFO ]      (1): ReLU()
2022-07-21 07:10:37 [INFO ]    )
2022-07-21 07:10:37 [INFO ]    (criterion): CrossEntropyLoss()
2022-07-21 07:10:37 [INFO ]  )
2022-07-21 07:10:37 [INFO ]  Total number of parameters: 14402

[14]:

debias_model.train_self(
    train_generator = customized_training_generator,
    dev_generator = customized_validation_generator,
    test_generator = customized_test_generator,
)

2022-07-21 07:10:37 [INFO ]  Epoch:    0 [      0/   4542 ( 0%)]        Loss: -0.4072    Data Time: 0.00s       Train Time: 0.05s
2022-07-21 07:10:37 [INFO ]  Evaluation at Epoch 0
2022-07-21 07:10:38 [INFO ]  Validation accuracy: 65.54 macro_fscore: 65.27     micro_fscore: 65.54     TPR_GAP: 32.32  FPR_GAP: 32.32  PPR_GAP: 35.63
2022-07-21 07:10:38 [INFO ]  Test accuracy: 67.44       macro_fscore: 67.00     micro_fscore: 67.44     TPR_GAP: 30.82  FPR_GAP: 30.82  PPR_GAP: 36.06
2022-07-21 07:10:38 [INFO ]  Epoch:    1 [      0/   4542 ( 0%)]        Loss: -0.4403    Data Time: 0.00s       Train Time: 0.02s
2022-07-21 07:10:38 [INFO ]  Evaluation at Epoch 1
2022-07-21 07:10:38 [INFO ]  Validation accuracy: 64.36 macro_fscore: 64.04     micro_fscore: 64.36     TPR_GAP: 31.50  FPR_GAP: 31.50  PPR_GAP: 34.76
2022-07-21 07:10:38 [INFO ]  Test accuracy: 67.67       macro_fscore: 67.06     micro_fscore: 67.67     TPR_GAP: 28.13  FPR_GAP: 28.13  PPR_GAP: 34.07
2022-07-21 07:10:38 [INFO ]  Epoch:    2 [      0/   4542 ( 0%)]        Loss: -0.4799    Data Time: 0.00s       Train Time: 0.03s
2022-07-21 07:10:39 [INFO ]  Evaluation at Epoch 2
2022-07-21 07:10:39 [INFO ]  Validation accuracy: 61.58 macro_fscore: 61.53     micro_fscore: 61.58     TPR_GAP: 32.47  FPR_GAP: 32.47  PPR_GAP: 34.93
2022-07-21 07:10:39 [INFO ]  Test accuracy: 66.65       macro_fscore: 66.41     micro_fscore: 66.65     TPR_GAP: 27.35  FPR_GAP: 27.36  PPR_GAP: 33.06
2022-07-21 07:10:39 [INFO ]  Epoch:    3 [      0/   4542 ( 0%)]        Loss: -0.4201    Data Time: 0.00s       Train Time: 0.02s
2022-07-21 07:10:40 [INFO ]  Evaluation at Epoch 3
2022-07-21 07:10:40 [INFO ]  Validation accuracy: 62.97 macro_fscore: 62.76     micro_fscore: 62.97     TPR_GAP: 35.73  FPR_GAP: 35.73  PPR_GAP: 38.56
2022-07-21 07:10:40 [INFO ]  Test accuracy: 67.62       macro_fscore: 67.05     micro_fscore: 67.62     TPR_GAP: 26.93  FPR_GAP: 26.93  PPR_GAP: 33.23
2022-07-21 07:10:40 [INFO ]  Epoch:    4 [      0/   4542 ( 0%)]        Loss: -0.5529    Data Time: 0.00s       Train Time: 0.01s
2022-07-21 07:10:41 [INFO ]  Epochs since last improvement: 1
2022-07-21 07:10:41 [INFO ]  Evaluation at Epoch 4
2022-07-21 07:10:41 [INFO ]  Validation accuracy: 61.98 macro_fscore: 61.78     micro_fscore: 61.98     TPR_GAP: 35.31  FPR_GAP: 35.31  PPR_GAP: 37.22
2022-07-21 07:10:41 [INFO ]  Test accuracy: 67.67       macro_fscore: 67.16     micro_fscore: 67.67     TPR_GAP: 25.47  FPR_GAP: 25.47  PPR_GAP: 31.31
2022-07-21 07:10:41 [INFO ]  Epoch:    5 [      0/   4542 ( 0%)]        Loss: -0.4623    Data Time: 0.00s       Train Time: 0.02s
2022-07-21 07:10:42 [INFO ]  Evaluation at Epoch 5
2022-07-21 07:10:42 [INFO ]  Validation accuracy: 63.37 macro_fscore: 63.24     micro_fscore: 63.37     TPR_GAP: 32.46  FPR_GAP: 32.46  PPR_GAP: 34.72
2022-07-21 07:10:42 [INFO ]  Test accuracy: 67.02       macro_fscore: 66.56     micro_fscore: 67.02     TPR_GAP: 26.19  FPR_GAP: 26.19  PPR_GAP: 32.26
2022-07-21 07:10:42 [INFO ]  Epoch:    6 [      0/   4542 ( 0%)]        Loss: -0.4727    Data Time: 0.00s       Train Time: 0.01s
2022-07-21 07:10:43 [INFO ]  Epochs since last improvement: 1
2022-07-21 07:10:43 [INFO ]  Evaluation at Epoch 6
2022-07-21 07:10:43 [INFO ]  Validation accuracy: 65.15 macro_fscore: 65.05     micro_fscore: 65.15     TPR_GAP: 32.60  FPR_GAP: 32.60  PPR_GAP: 35.29
2022-07-21 07:10:43 [INFO ]  Test accuracy: 66.70       macro_fscore: 66.36     micro_fscore: 66.70     TPR_GAP: 27.40  FPR_GAP: 27.40  PPR_GAP: 32.54
2022-07-21 07:10:43 [INFO ]  Epoch:    7 [      0/   4542 ( 0%)]        Loss: -0.4569    Data Time: 0.00s       Train Time: 0.02s
2022-07-21 07:10:44 [INFO ]  Epochs since last improvement: 2
2022-07-21 07:10:44 [INFO ]  Evaluation at Epoch 7
2022-07-21 07:10:44 [INFO ]  Validation accuracy: 65.74 macro_fscore: 65.69     micro_fscore: 65.74     TPR_GAP: 32.00  FPR_GAP: 32.00  PPR_GAP: 35.02
2022-07-21 07:10:44 [INFO ]  Test accuracy: 66.84       macro_fscore: 66.62     micro_fscore: 66.84     TPR_GAP: 27.97  FPR_GAP: 27.97  PPR_GAP: 33.59
2022-07-21 07:10:44 [INFO ]  Epoch:    8 [      0/   4542 ( 0%)]        Loss: -0.4861    Data Time: 0.00s       Train Time: 0.01s
2022-07-21 07:10:44 [INFO ]  Epochs since last improvement: 3
2022-07-21 07:10:44 [INFO ]  Evaluation at Epoch 8
2022-07-21 07:10:44 [INFO ]  Validation accuracy: 64.36 macro_fscore: 64.16     micro_fscore: 64.36     TPR_GAP: 39.90  FPR_GAP: 39.90  PPR_GAP: 42.72
2022-07-21 07:10:44 [INFO ]  Test accuracy: 66.28       macro_fscore: 65.64     micro_fscore: 66.28     TPR_GAP: 24.97  FPR_GAP: 24.97  PPR_GAP: 29.19
2022-07-21 07:10:44 [INFO ]  Epoch:    9 [      0/   4542 ( 0%)]        Loss: -0.5028    Data Time: 0.00s       Train Time: 0.01s
2022-07-21 07:10:45 [INFO ]  Epochs since last improvement: 4
2022-07-21 07:10:45 [INFO ]  Evaluation at Epoch 9
2022-07-21 07:10:45 [INFO ]  Validation accuracy: 63.76 macro_fscore: 63.67     micro_fscore: 63.76     TPR_GAP: 31.83  FPR_GAP: 31.83  PPR_GAP: 34.30
2022-07-21 07:10:45 [INFO ]  Test accuracy: 66.61       macro_fscore: 66.15     micro_fscore: 66.61     TPR_GAP: 25.55  FPR_GAP: 25.55  PPR_GAP: 29.76
2022-07-21 07:10:45 [INFO ]  Epoch:   10 [      0/   4542 ( 0%)]        Loss: -0.5265    Data Time: 0.00s       Train Time: 0.01s
2022-07-21 07:10:46 [INFO ]  Epochs since last improvement: 5
2022-07-21 07:10:46 [INFO ]  Evaluation at Epoch 10
2022-07-21 07:10:46 [INFO ]  Validation accuracy: 64.55 macro_fscore: 64.55     micro_fscore: 64.55     TPR_GAP: 30.83  FPR_GAP: 30.83  PPR_GAP: 32.99
2022-07-21 07:10:46 [INFO ]  Test accuracy: 66.28       macro_fscore: 66.11     micro_fscore: 66.28     TPR_GAP: 26.50  FPR_GAP: 26.50  PPR_GAP: 32.48