alibi_detect.cd package

class alibi_detect.cd.ChiSquareDrift(x_ref, p_val=0.05, categories_per_feature=None, preprocess_x_ref=True, update_x_ref=None, preprocess_fn=None, correction='bonferroni', n_features=None, input_shape=None, data_type=None)[source]

Bases: alibi_detect.cd.base.BaseUnivariateDrift

__init__(x_ref, p_val=0.05, categories_per_feature=None, preprocess_x_ref=True, update_x_ref=None, preprocess_fn=None, correction='bonferroni', n_features=None, input_shape=None, data_type=None)[source]

Chi-Squared data drift detector with Bonferroni or False Discovery Rate (FDR) correction for multivariate data.

Parameters

  • x_ref (Union[ndarray, list]) – Data used as reference distribution.

  • p_val (float) – p-value used for significance of the Chi-Squared test for each feature. If the FDR correction method is used, this corresponds to the acceptable q-value.

  • categories_per_feature (Optional[Dict[int, int]]) – Optional dictionary with as keys the feature column index and as values the number of possible categorical values for that feature or a list with the possible values. If you know how many categories are present for a given feature you could pass this in the categories_per_feature dict in the Dict[int, int] format, e.g. {0: 3, 3: 2}. If you pass N categories this will assume the possible values for the feature are [0, …, N-1]. You can also explicitly pass the possible categories in the Dict[int, List[int]] format, e.g. {0: [0, 1, 2], 3: [0, 55]}. Note that the categories can be arbitrary int values. If it is not specified, categories_per_feature is inferred from x_ref.

  • preprocess_x_ref (bool) – Whether to already preprocess and infer categories and frequencies for reference data.

  • update_x_ref (Optional[Dict[str, int]]) – Reference data can optionally be updated to the last n instances seen by the detector or via reservoir sampling with size n. For the former, the parameter equals {‘last’: n} while for reservoir sampling {‘reservoir_sampling’: n} is passed.

  • preprocess_fn (Optional[Callable]) – Function to preprocess the data before computing the data drift metrics. Typically a dimensionality reduction technique.

  • correction (str) – Correction type for multivariate data. Either ‘bonferroni’ or ‘fdr’ (False Discovery Rate).

  • n_features (Optional[int]) – Number of features used in the Chi-Squared test. No need to pass it if no preprocessing takes place. In case of a preprocessing step, this can also be inferred automatically but could be more expensive to compute.

  • input_shape (Optional[tuple]) – Shape of input data.

  • data_type (Optional[str]) – Optionally specify the data type (tabular, image or time-series). Added to metadata.

Return type

None

feature_score(x_ref, x)[source]

Compute Chi-Squared test statistic and p-values per feature.

Parameters

  • x_ref (ndarray) – Reference instances to compare distribution with.

  • x (ndarray) – Batch of instances.

Return type

Tuple[ndarray, ndarray]

Returns

Feature level p-values and Chi-Squared statistics.

class alibi_detect.cd.ClassifierDrift(x_ref, model, backend='tensorflow', p_val=0.05, preprocess_x_ref=True, update_x_ref=None, preprocess_fn=None, preds_type='probs', binarize_preds=False, reg_loss_fn=<function ClassifierDrift.<lambda>>, train_size=0.75, n_folds=None, retrain_from_scratch=True, seed=0, optimizer=None, learning_rate=0.001, batch_size=32, preprocess_batch_fn=None, epochs=3, verbose=0, train_kwargs=None, device=None, dataset=None, dataloader=None, data_type=None)[source]

Bases: object

__init__(x_ref, model, backend='tensorflow', p_val=0.05, preprocess_x_ref=True, update_x_ref=None, preprocess_fn=None, preds_type='probs', binarize_preds=False, reg_loss_fn=<function ClassifierDrift.<lambda>>, train_size=0.75, n_folds=None, retrain_from_scratch=True, seed=0, optimizer=None, learning_rate=0.001, batch_size=32, preprocess_batch_fn=None, epochs=3, verbose=0, train_kwargs=None, device=None, dataset=None, dataloader=None, data_type=None)[source]

Classifier-based drift detector. The classifier is trained on a fraction of the combined reference and test data and drift is detected on the remaining data. To use all the data to detect drift, a stratified cross-validation scheme can be chosen.

Parameters

  • x_ref (Union[ndarray, list]) – Data used as reference distribution.

  • model (Callable) – PyTorch or TensorFlow classification model used for drift detection.

  • backend (str) – Backend used for the training loop implementation.

  • p_val (float) – p-value used for the significance of the test.

  • preprocess_x_ref (bool) – Whether to already preprocess and store the reference data.

  • update_x_ref (Optional[Dict[str, int]]) – Reference data can optionally be updated to the last n instances seen by the detector or via reservoir sampling with size n. For the former, the parameter equals {‘last’: n} while for reservoir sampling {‘reservoir_sampling’: n} is passed.

  • preprocess_fn (Optional[Callable]) – Function to preprocess the data before computing the data drift metrics.

  • preds_type (str) – Whether the model outputs ‘probs’ or ‘logits’

  • binarize_preds (bool) – Whether to test for discrepency on soft (e.g. probs/logits) model predictions directly with a K-S test or binarise to 0-1 prediction errors and apply a binomial test.

  • reg_loss_fn (Callable) – The regularisation term reg_loss_fn(model) is added to the loss function being optimized.

  • train_size (Optional[float]) – Optional fraction (float between 0 and 1) of the dataset used to train the classifier. The drift is detected on 1 - train_size. Cannot be used in combination with n_folds.

  • n_folds (Optional[int]) – Optional number of stratified folds used for training. The model preds are then calculated on all the out-of-fold instances. This allows to leverage all the reference and test data for drift detection at the expense of longer computation. If both train_size and n_folds are specified, n_folds is prioritized.

  • retrain_from_scratch (bool) – Whether the classifier should be retrained from scratch for each set of test data or whether it should instead continue training from where it left off on the previous set.

  • seed (int) – Optional random seed for fold selection.

  • optimizer (Optional[Callable]) – Optimizer used during training of the classifier.

  • learning_rate (float) – Learning rate used by optimizer.

  • batch_size (int) – Batch size used during training of the classifier.

  • preprocess_batch_fn (Optional[Callable]) – Optional batch preprocessing function. For example to convert a list of objects to a batch which can be processed by the model.

  • epochs (int) – Number of training epochs for the classifier for each (optional) fold.

  • verbose (int) – Verbosity level during the training of the classifier. 0 is silent, 1 a progress bar.

  • train_kwargs (Optional[dict]) – Optional additional kwargs when fitting the classifier.

  • device (Optional[str]) – Device type used. The default None tries to use the GPU and falls back on CPU if needed. Can be specified by passing either ‘cuda’, ‘gpu’ or ‘cpu’. Only relevant for ‘pytorch’ backend.

  • dataset (Optional[Callable]) – Dataset object used during training.

  • dataloader (Optional[Callable]) – Dataloader object used during training. Only relevant for ‘pytorch’ backend.

  • data_type (Optional[str]) – Optionally specify the data type (tabular, image or time-series). Added to metadata.

Return type

None

predict(x, return_p_val=True, return_distance=True, return_probs=True, return_model=True)[source]

Predict whether a batch of data has drifted from the reference data.

Parameters

  • x (Union[ndarray, list]) – Batch of instances.

  • return_p_val (bool) – Whether to return the p-value of the test.

  • return_distance (bool) – Whether to return a notion of strength of the drift. K-S test stat if binarize_preds=False, otherwise relative error reduction.

  • return_probs (bool) – Whether to return the instance level classifier probabilities for the reference and test data (0=reference data, 1=test data).

  • return_model (bool) – Whether to return the updated model trained to discriminate reference and test instances.

Return type

Dict[str, Dict[str, Union[str, int, float, Callable]]]

Returns

  • Dictionary containing ‘meta’ and ‘data’ dictionaries.

  • ’meta’ has the model’s metadata.

  • ’data’ contains the drift prediction and optionally the p-value, performance of the classifier

  • relative to its expectation under the no-change null, the out-of-fold classifier model

  • prediction probabilities on the reference and test data, and the trained model.

class alibi_detect.cd.KSDrift(x_ref, p_val=0.05, preprocess_x_ref=True, update_x_ref=None, preprocess_fn=None, correction='bonferroni', alternative='two-sided', n_features=None, input_shape=None, data_type=None)[source]

Bases: alibi_detect.cd.base.BaseUnivariateDrift

__init__(x_ref, p_val=0.05, preprocess_x_ref=True, update_x_ref=None, preprocess_fn=None, correction='bonferroni', alternative='two-sided', n_features=None, input_shape=None, data_type=None)[source]

Kolmogorov-Smirnov (K-S) data drift detector with Bonferroni or False Discovery Rate (FDR) correction for multivariate data.

Parameters

  • x_ref (Union[ndarray, list]) – Data used as reference distribution.

  • p_val (float) – p-value used for significance of the K-S test for each feature. If the FDR correction method is used, this corresponds to the acceptable q-value.

  • preprocess_x_ref (bool) – Whether to already preprocess and store the reference data.

  • update_x_ref (Optional[Dict[str, int]]) – Reference data can optionally be updated to the last n instances seen by the detector or via reservoir sampling with size n. For the former, the parameter equals {‘last’: n} while for reservoir sampling {‘reservoir_sampling’: n} is passed.

  • preprocess_fn (Optional[Callable]) – Function to preprocess the data before computing the data drift metrics. Typically a dimensionality reduction technique.

  • correction (str) – Correction type for multivariate data. Either ‘bonferroni’ or ‘fdr’ (False Discovery Rate).

  • alternative (str) – Defines the alternative hypothesis. Options are ‘two-sided’, ‘less’ or ‘greater’.

  • n_features (Optional[int]) – Number of features used in the K-S test. No need to pass it if no preprocessing takes place. In case of a preprocessing step, this can also be inferred automatically but could be more expensive to compute.

  • input_shape (Optional[tuple]) – Shape of input data.

  • data_type (Optional[str]) – Optionally specify the data type (tabular, image or time-series). Added to metadata.

Return type

None

feature_score(x_ref, x)[source]

Compute K-S scores and statistics per feature.

Parameters

  • x_ref (ndarray) – Reference instances to compare distribution with.

  • x (ndarray) – Batch of instances.

Return type

Tuple[ndarray, ndarray]

Returns

Feature level p-values and K-S statistics.

class alibi_detect.cd.LearnedKernelDrift(x_ref, kernel, backend='tensorflow', p_val=0.05, preprocess_x_ref=True, update_x_ref=None, preprocess_fn=None, n_permutations=100, var_reg=1e-05, reg_loss_fn=<function LearnedKernelDrift.<lambda>>, train_size=0.75, retrain_from_scratch=True, optimizer=None, learning_rate=0.001, batch_size=32, preprocess_batch_fn=None, epochs=3, verbose=0, train_kwargs=None, device=None, dataset=None, dataloader=None, data_type=None)[source]

Bases: object

__init__(x_ref, kernel, backend='tensorflow', p_val=0.05, preprocess_x_ref=True, update_x_ref=None, preprocess_fn=None, n_permutations=100, var_reg=1e-05, reg_loss_fn=<function LearnedKernelDrift.<lambda>>, train_size=0.75, retrain_from_scratch=True, optimizer=None, learning_rate=0.001, batch_size=32, preprocess_batch_fn=None, epochs=3, verbose=0, train_kwargs=None, device=None, dataset=None, dataloader=None, data_type=None)[source]

Maximum Mean Discrepancy (MMD) data drift detector where the kernel is trained to maximise an estimate of the test power. The kernel is trained on a split of the reference and test instances and then the MMD is evaluated on held out instances and a permutation test is performed.

For details see Liu et al (2020): Learning Deep Kernels for Non-Parametric Two-Sample Tests (https://arxiv.org/abs/2002.09116)

Parameters

  • x_ref (Union[ndarray, list]) – Data used as reference distribution.

  • kernel (Callable) – Trainable PyTorch or TensorFlow module that returns a similarity between two instances.

  • backend (str) – Backend used by the kernel and training loop.

  • p_val (float) – p-value used for the significance of the test.

  • preprocess_x_ref (bool) – Whether to already preprocess and store the reference data.

  • update_x_ref (Optional[Dict[str, int]]) – Reference data can optionally be updated to the last n instances seen by the detector or via reservoir sampling with size n. For the former, the parameter equals {‘last’: n} while for reservoir sampling {‘reservoir_sampling’: n} is passed.

  • preprocess_fn (Optional[Callable]) – Function to preprocess the data before applying the kernel.

  • n_permutations (int) – The number of permutations to use in the permutation test once the MMD has been computed.

  • var_reg (float) – Constant added to the estimated variance of the MMD for stability.

  • reg_loss_fn (Callable) – The regularisation term reg_loss_fn(kernel) is added to the loss function being optimized.

  • train_size (Optional[float]) – Optional fraction (float between 0 and 1) of the dataset used to train the kernel. The drift is detected on 1 - train_size.

  • retrain_from_scratch (bool) – Whether the kernel should be retrained from scratch for each set of test data or whether it should instead continue training from where it left off on the previous set.

  • optimizer (Optional[Callable]) – Optimizer used during training of the kernel.

  • learning_rate (float) – Learning rate used by optimizer.

  • batch_size (int) – Batch size used during training of the kernel.

  • preprocess_batch_fn (Optional[Callable]) – Optional batch preprocessing function. For example to convert a list of objects to a batch which can be processed by the kernel.

  • epochs (int) – Number of training epochs for the kernel. Corresponds to the smaller of the reference and test sets.

  • verbose (int) – Verbosity level during the training of the kernel. 0 is silent, 1 a progress bar.

  • train_kwargs (Optional[dict]) – Optional additional kwargs when training the kernel.

  • device (Optional[str]) – Device type used. The default None tries to use the GPU and falls back on CPU if needed. Can be specified by passing either ‘cuda’, ‘gpu’ or ‘cpu’. Only relevant for ‘pytorch’ backend.

  • dataset (Optional[Callable]) – Dataset object used during training.

  • dataloader (Optional[Callable]) – Dataloader object used during training. Only relevant for ‘pytorch’ backend.

  • data_type (Optional[str]) – Optionally specify the data type (tabular, image or time-series). Added to metadata.

Return type

None

predict(x, return_p_val=True, return_distance=True, return_kernel=True)[source]

Predict whether a batch of data has drifted from the reference data.

Parameters

  • x (Union[ndarray, list]) – Batch of instances.

  • return_p_val (bool) – Whether to return the p-value of the permutation test.

  • return_distance (bool) – Whether to return the MMD metric between the new batch and reference data.

  • return_kernel (bool) – Whether to return the updated kernel trained to discriminate reference and test instances.

Return type

Dict[Dict[str, str], Dict[str, Union[int, float, Callable]]]

Returns

  • Dictionary containing ‘meta’ and ‘data’ dictionaries.

  • ’meta’ has the detector’s metadata.

  • ’data’ contains the drift prediction and optionally the p-value, threshold, MMD metric and – trained kernel.

class alibi_detect.cd.LSDDDrift(x_ref, backend='tensorflow', p_val=0.05, preprocess_x_ref=True, update_x_ref=None, preprocess_fn=None, sigma=None, n_permutations=100, n_kernel_centers=None, lambda_rd_max=0.2, device=None, input_shape=None, data_type=None)[source]

Bases: object

__init__(x_ref, backend='tensorflow', p_val=0.05, preprocess_x_ref=True, update_x_ref=None, preprocess_fn=None, sigma=None, n_permutations=100, n_kernel_centers=None, lambda_rd_max=0.2, device=None, input_shape=None, data_type=None)[source]

Least-squares density difference (LSDD) data drift detector using a permutation test.

Parameters

  • x_ref (Union[ndarray, list]) – Data used as reference distribution.

  • backend (str) – Backend used for the LSDD implementation.

  • p_val (float) – p-value used for the significance of the permutation test.

  • preprocess_x_ref (bool) – Whether to already preprocess and store the reference data.

  • update_x_ref (Optional[Dict[str, int]]) – Reference data can optionally be updated to the last n instances seen by the detector or via reservoir sampling with size n. For the former, the parameter equals {‘last’: n} while for reservoir sampling {‘reservoir_sampling’: n} is passed.

  • preprocess_fn (Optional[Callable]) – Function to preprocess the data before computing the data drift metrics.

  • sigma (Optional[ndarray]) – Optionally set the bandwidth of the Gaussian kernel used in estimating the LSDD. Can also pass multiple bandwidth values as an array. The kernel evaluation is then averaged over those bandwidths. If sigma is not specified, the ‘median heuristic’ is adopted whereby sigma is set as the median pairwise distance between reference samples.

  • n_permutations (int) – Number of permutations used in the permutation test.

  • n_kernel_centers (Optional[int]) – The number of reference samples to use as centers in the Gaussian kernel model used to estimate LSDD. Defaults to 1/20th of the reference data.

  • lambda_rd_max (float) – The maximum relative difference between two estimates of LSDD that the regularization parameter lambda is allowed to cause. Defaults to 0.2 as in the paper.

  • device (Optional[str]) – Device type used. The default None tries to use the GPU and falls back on CPU if needed. Can be specified by passing either ‘cuda’, ‘gpu’ or ‘cpu’. Only relevant for ‘pytorch’ backend.

  • input_shape (Optional[tuple]) – Shape of input data.

  • data_type (Optional[str]) – Optionally specify the data type (tabular, image or time-series). Added to metadata.

Return type

None

predict(x, return_p_val=True, return_distance=True)[source]

Predict whether a batch of data has drifted from the reference data.

Parameters

  • x (Union[ndarray, list]) – Batch of instances.

  • return_p_val (bool) – Whether to return the p-value of the permutation test.

  • return_distance (bool) – Whether to return the LSDD metric between the new batch and reference data.

Return type

Dict[Dict[str, str], Dict[str, Union[int, float]]]

Returns

  • Dictionary containing ‘meta’ and ‘data’ dictionaries.

  • ’meta’ has the model’s metadata.

  • ’data’ contains the drift prediction and optionally the p-value, threshold and LSDD metric.

class alibi_detect.cd.LSDDDriftOnline(x_ref, ert, window_size, backend='tensorflow', preprocess_fn=None, sigma=None, n_bootstraps=1000, n_kernel_centers=None, lambda_rd_max=0.2, device=None, verbose=True, input_shape=None, data_type=None)[source]

Bases: object

__init__(x_ref, ert, window_size, backend='tensorflow', preprocess_fn=None, sigma=None, n_bootstraps=1000, n_kernel_centers=None, lambda_rd_max=0.2, device=None, verbose=True, input_shape=None, data_type=None)[source]

Online least squares density difference (LSDD) data drift detector using preconfigured thresholds. Motivated by Bu et al. (2017): https://ieeexplore.ieee.org/abstract/document/7890493 We have made modifications such that a desired ERT can be accurately targeted however.

Parameters

  • x_ref (Union[ndarray, list]) – Data used as reference distribution.

  • ert (float) – The expected run-time (ERT) in the absence of drift.

  • window_size (int) – The size of the sliding test-window used to compute the test-statistic. Smaller windows focus on responding quickly to severe drift, larger windows focus on ability to detect slight drift.

  • backend (str) – Backend used for the LSDD implementation and configuration.

  • preprocess_fn (Optional[Callable]) – Function to preprocess the data before computing the data drift metrics.s

  • sigma (Optional[ndarray]) – Optionally set the bandwidth of the Gaussian kernel used in estimating the LSDD. Can also pass multiple bandwidth values as an array. The kernel evaluation is then averaged over those bandwidths. If sigma is not specified, the ‘median heuristic’ is adopted whereby sigma is set as the median pairwise distance between reference samples.

  • n_bootstraps (int) – The number of bootstrap simulations used to configure the thresholds. The larger this is the more accurately the desired ERT will be targeted. Should ideally be at least an order of magnitude larger than the ert.

  • n_kernel_centers (Optional[int]) – The number of reference samples to use as centers in the Gaussian kernel model used to estimate LSDD. Defaults to 2*window_size.

  • lambda_rd_max (float) – The maximum relative difference between two estimates of LSDD that the regularization parameter lambda is allowed to cause. Defaults to 0.2 as in the paper.

  • device (Optional[str]) – Device type used. The default None tries to use the GPU and falls back on CPU if needed. Can be specified by passing either ‘cuda’, ‘gpu’ or ‘cpu’. Only relevant for ‘pytorch’ backend.

  • verbose (bool) – Whether or not to print progress during configuration.

  • input_shape (Optional[tuple]) – Shape of input data.

  • data_type (Optional[str]) – Optionally specify the data type (tabular, image or time-series). Added to metadata.

Return type

None

predict(x_t, return_test_stat=True)[source]

Predict whether the most recent window of data has drifted from the reference data.

Parameters

  • x_t (Union[ndarray, list]) – A single instance to be added to the test-window.

  • return_test_stat (bool) – Whether to return the test statistic (LSDD) and threshold.

Return type

Dict[Dict[str, str], Dict[str, Union[int, float]]]

Returns

  • Dictionary containing ‘meta’ and ‘data’ dictionaries.

  • ’meta’ has the model’s metadata.

  • ’data’ contains the drift prediction and optionally the test-statistic and threshold.

reset()[source]

Resets the detector but does not reconfigure thresholds.

property t
property test_stats
property thresholds
class alibi_detect.cd.MMDDrift(x_ref, backend='tensorflow', p_val=0.05, preprocess_x_ref=True, update_x_ref=None, preprocess_fn=None, kernel=None, sigma=None, configure_kernel_from_x_ref=True, n_permutations=100, device=None, input_shape=None, data_type=None)[source]

Bases: object

__init__(x_ref, backend='tensorflow', p_val=0.05, preprocess_x_ref=True, update_x_ref=None, preprocess_fn=None, kernel=None, sigma=None, configure_kernel_from_x_ref=True, n_permutations=100, device=None, input_shape=None, data_type=None)[source]

Maximum Mean Discrepancy (MMD) data drift detector using a permutation test.

Parameters

  • x_ref (Union[ndarray, list]) – Data used as reference distribution.

  • backend (str) – Backend used for the MMD implementation.

  • p_val (float) – p-value used for the significance of the permutation test.

  • preprocess_x_ref (bool) – Whether to already preprocess and store the reference data.

  • update_x_ref (Optional[Dict[str, int]]) – Reference data can optionally be updated to the last n instances seen by the detector or via reservoir sampling with size n. For the former, the parameter equals {‘last’: n} while for reservoir sampling {‘reservoir_sampling’: n} is passed.

  • preprocess_fn (Optional[Callable]) – Function to preprocess the data before computing the data drift metrics.

  • kernel (Optional[Callable]) – Kernel used for the MMD computation, defaults to Gaussian RBF kernel.

  • sigma (Optional[ndarray]) – Optionally set the GaussianRBF kernel bandwidth. Can also pass multiple bandwidth values as an array. The kernel evaluation is then averaged over those bandwidths.

  • configure_kernel_from_x_ref (bool) – Whether to already configure the kernel bandwidth from the reference data.

  • n_permutations (int) – Number of permutations used in the permutation test.

  • device (Optional[str]) – Device type used. The default None tries to use the GPU and falls back on CPU if needed. Can be specified by passing either ‘cuda’, ‘gpu’ or ‘cpu’. Only relevant for ‘pytorch’ backend.

  • input_shape (Optional[tuple]) – Shape of input data.

  • data_type (Optional[str]) – Optionally specify the data type (tabular, image or time-series). Added to metadata.

Return type

None

predict(x, return_p_val=True, return_distance=True)[source]

Predict whether a batch of data has drifted from the reference data.

Parameters

  • x (Union[ndarray, list]) – Batch of instances.

  • return_p_val (bool) – Whether to return the p-value of the permutation test.

  • return_distance (bool) – Whether to return the MMD metric between the new batch and reference data.

Return type

Dict[Dict[str, str], Dict[str, Union[int, float]]]

Returns

  • Dictionary containing ‘meta’ and ‘data’ dictionaries.

  • ’meta’ has the model’s metadata.

  • ’data’ contains the drift prediction and optionally the p-value, threshold and MMD metric.

class alibi_detect.cd.MMDDriftOnline(x_ref, ert, window_size, backend='tensorflow', preprocess_fn=None, kernel=None, sigma=None, n_bootstraps=1000, device=None, verbose=True, input_shape=None, data_type=None)[source]

Bases: object

__init__(x_ref, ert, window_size, backend='tensorflow', preprocess_fn=None, kernel=None, sigma=None, n_bootstraps=1000, device=None, verbose=True, input_shape=None, data_type=None)[source]

Online maximum Mean Discrepancy (MMD) data drift detector using preconfigured thresholds.

Parameters

  • x_ref (Union[ndarray, list]) – Data used as reference distribution.

  • ert (float) – The expected run-time (ERT) in the absence of drift.

  • window_size (int) – The size of the sliding test-window used to compute the test-statistic. Smaller windows focus on responding quickly to severe drift, larger windows focus on ability to detect slight drift.

  • backend (str) – Backend used for the MMD implementation and configuration.

  • preprocess_fn (Optional[Callable]) – Function to preprocess the data before computing the data drift metrics.

  • kernel (Optional[Callable]) – Kernel used for the MMD computation, defaults to Gaussian RBF kernel.

  • sigma (Optional[ndarray]) – Optionally set the GaussianRBF kernel bandwidth. Can also pass multiple bandwidth values as an array. The kernel evaluation is then averaged over those bandwidths. If sigma is not specified, the ‘median heuristic’ is adopted whereby sigma is set as the median pairwise distance between reference samples.

  • n_bootstraps (int) – The number of bootstrap simulations used to configure the thresholds. The larger this is the more accurately the desired ERT will be targeted. Should ideally be at least an order of magnitude larger than the ERT.

  • device (Optional[str]) – Device type used. The default None tries to use the GPU and falls back on CPU if needed. Can be specified by passing either ‘cuda’, ‘gpu’ or ‘cpu’. Only relevant for ‘pytorch’ backend.

  • verbose (bool) – Whether or not to print progress during configuration.

  • input_shape (Optional[tuple]) – Shape of input data.

  • data_type (Optional[str]) – Optionally specify the data type (tabular, image or time-series). Added to metadata.

Return type

None

predict(x_t, return_test_stat=True)[source]

Predict whether the most recent window of data has drifted from the reference data.

Parameters

  • x_t (Union[ndarray, list]) – A single instance to be added to the test-window.

  • return_test_stat (bool) – Whether to return the test statistic (squared MMD) and threshold.

Return type

Dict[Dict[str, str], Dict[str, Union[int, float]]]

Returns

  • Dictionary containing ‘meta’ and ‘data’ dictionaries.

  • ’meta’ has the model’s metadata.

  • ’data’ contains the drift prediction and optionally the test-statistic and threshold.

reset()[source]

Resets the detector but does not reconfigure thresholds.

property t
property test_stats
property thresholds
class alibi_detect.cd.TabularDrift(x_ref, p_val=0.05, categories_per_feature=None, preprocess_x_ref=True, update_x_ref=None, preprocess_fn=None, correction='bonferroni', alternative='two-sided', n_features=None, input_shape=None, data_type=None)[source]

Bases: alibi_detect.cd.base.BaseUnivariateDrift

__init__(x_ref, p_val=0.05, categories_per_feature=None, preprocess_x_ref=True, update_x_ref=None, preprocess_fn=None, correction='bonferroni', alternative='two-sided', n_features=None, input_shape=None, data_type=None)[source]

Mixed-type tabular data drift detector with Bonferroni or False Discovery Rate (FDR) correction for multivariate data. Kolmogorov-Smirnov (K-S) univariate tests are applied to continuous numerical data and Chi-Squared (Chi2) univariate tests to categorical data.

Parameters

  • x_ref (Union[ndarray, list]) – Data used as reference distribution.

  • p_val (float) – p-value used for significance of the K-S and Chi2 test for each feature. If the FDR correction method is used, this corresponds to the acceptable q-value.

  • categories_per_feature (Optional[Dict[int, Optional[int]]]) – Dictionary with as keys the column indices of the categorical features and optionally as values the number of possible categorical values for that feature or a list with the possible values. If you know which features are categorical and simply want to infer the possible values of the categorical feature from the reference data you can pass a Dict[int, NoneType] such as {0: None, 3: None} if features 0 and 3 are categorical. If you also know how many categories are present for a given feature you could pass this in the categories_per_feature dict in the Dict[int, int] format, e.g. {0: 3, 3: 2}. If you pass N categories this will assume the possible values for the feature are [0, …, N-1]. You can also explicitly pass the possible categories in the Dict[int, List[int]] format, e.g. {0: [0, 1, 2], 3: [0, 55]}. Note that the categories can be arbitrary int values.

  • preprocess_x_ref (bool) – Whether to already preprocess and infer categories and frequencies for categorical reference data.

  • update_x_ref (Optional[Dict[str, int]]) – Reference data can optionally be updated to the last n instances seen by the detector or via reservoir sampling with size n. For the former, the parameter equals {‘last’: n} while for reservoir sampling {‘reservoir_sampling’: n} is passed.

  • preprocess_fn (Optional[Callable]) – Function to preprocess the data before computing the data drift metrics. Typically a dimensionality reduction technique.

  • correction (str) – Correction type for multivariate data. Either ‘bonferroni’ or ‘fdr’ (False Discovery Rate).

  • alternative (str) – Defines the alternative hypothesis for the K-S tests. Options are ‘two-sided’, ‘less’ or ‘greater’.

  • n_features (Optional[int]) – Number of features used in the combined K-S/Chi-Squared tests. No need to pass it if no preprocessing takes place. In case of a preprocessing step, this can also be inferred automatically but could be more expensive to compute.

  • input_shape (Optional[tuple]) – Shape of input data.

  • data_type (Optional[str]) – Optionally specify the data type (tabular, image or time-series). Added to metadata.

Return type

None

feature_score(x_ref, x)[source]

Compute K-S or Chi-Squared test statistics and p-values per feature.

Parameters

  • x_ref (ndarray) – Reference instances to compare distribution with.

  • x (ndarray) – Batch of instances.

Return type

Tuple[ndarray, ndarray]

Returns

Feature level p-values and K-S or Chi-Squared statistics.

class alibi_detect.cd.ClassifierUncertaintyDrift(x_ref, model, p_val=0.05, backend=None, update_x_ref=None, preds_type='probs', uncertainty_type='entropy', margin_width=0.1, batch_size=32, preprocess_batch_fn=None, device=None, tokenizer=None, max_len=None, data_type=None)[source]

Bases: object

__init__(x_ref, model, p_val=0.05, backend=None, update_x_ref=None, preds_type='probs', uncertainty_type='entropy', margin_width=0.1, batch_size=32, preprocess_batch_fn=None, device=None, tokenizer=None, max_len=None, data_type=None)[source]

Test for a change in the number of instances falling into regions on which the model is uncertain. Performs either a K-S test on prediction entropies or Chi-squared test on 0-1 indicators of predictions falling into a margin of uncertainty (e.g. probs falling into [0.45, 0.55] in binary case).

Parameters

  • x_ref (Union[ndarray, list]) – Data used as reference distribution. Should be disjoint from the data the model was trained on for accurate p-values.

  • model (Callable) – Classification model outputting class probabilities (or logits)

  • backend (Optional[str]) – Backend to use if model requires batch prediction. Options are ‘tensorflow’ or ‘pytorch’.

  • p_val (float) – p-value used for the significance of the test.

  • update_x_ref (Optional[Dict[str, int]]) – Reference data can optionally be updated to the last n instances seen by the detector or via reservoir sampling with size n. For the former, the parameter equals {‘last’: n} while for reservoir sampling {‘reservoir_sampling’: n} is passed.

  • preds_type (str) – Type of prediction output by the model. Options are ‘probs’ (in [0,1]) or ‘logits’ (in [-inf,inf]).

  • uncertainty_type (str) – Method for determining the model’s uncertainty for a given instance. Options are ‘entropy’ or ‘margin’.

  • margin_width (float) – Width of the margin if uncertainty_type = ‘margin’. The model is considered uncertain on an instance if the highest two class probabilities it assigns to the instance differ by less than margin_width.

  • batch_size (int) – Batch size used to evaluate model. Only relevant when backend has been specified for batch prediction.

  • preprocess_batch_fn (Optional[Callable]) – Optional batch preprocessing function. For example to convert a list of objects to a batch which can be processed by the model.

  • device (Optional[str]) – Device type used. The default None tries to use the GPU and falls back on CPU if needed. Can be specified by passing either ‘cuda’, ‘gpu’ or ‘cpu’. Only relevant for ‘pytorch’ backend.

  • tokenizer (Optional[Callable]) – Optional tokenizer for NLP models.

  • max_len (Optional[int]) – Optional max token length for NLP models.

  • data_type (Optional[str]) – Optionally specify the data type (tabular, image or time-series). Added to metadata.

Return type

None

predict(x, return_p_val=True, return_distance=True)[source]

Predict whether a batch of data has drifted from the reference data.

Parameters

  • x (Union[ndarray, list]) – Batch of instances.

  • return_p_val (bool) – Whether to return the p-value of the test.

  • return_distance (bool) – Whether to return the corresponding test statistic (K-S for ‘entropy’, Chi2 for ‘margin’).

Return type

Dict[Dict[str, str], Dict[str, Union[int, float]]]

Returns

  • Dictionary containing ‘meta’ and ‘data’ dictionaries.

  • ’meta’ has the model’s metadata.

  • ’data’ contains the drift prediction and optionally the p-value, threshold and test statistic.

class alibi_detect.cd.RegressorUncertaintyDrift(x_ref, model, p_val=0.05, backend=None, update_x_ref=None, uncertainty_type='mc_dropout', n_evals=25, batch_size=32, preprocess_batch_fn=None, device=None, tokenizer=None, max_len=None, data_type=None)[source]

Bases: object

__init__(x_ref, model, p_val=0.05, backend=None, update_x_ref=None, uncertainty_type='mc_dropout', n_evals=25, batch_size=32, preprocess_batch_fn=None, device=None, tokenizer=None, max_len=None, data_type=None)[source]

Test for a change in the number of instances falling into regions on which the model is uncertain. Performs either a K-S test on uncertainties estimated from an preditive ensemble given either explicitly or implicitly as a model with dropout layers.

Parameters

  • x_ref (Union[ndarray, list]) – Data used as reference distribution. Should be disjoint from the data the model was trained on for accurate p-values.

  • model (Callable) – Regression model outputting class probabilities (or logits)

  • backend (Optional[str]) – Backend to use if model requires batch prediction. Options are ‘tensorflow’ or ‘pytorch’.

  • p_val (float) – p-value used for the significance of the test.

  • update_x_ref (Optional[Dict[str, int]]) – Reference data can optionally be updated to the last n instances seen by the detector or via reservoir sampling with size n. For the former, the parameter equals {‘last’: n} while for reservoir sampling {‘reservoir_sampling’: n} is passed.

  • uncertainty_type (str) – Method for determining the model’s uncertainty for a given instance. Options are ‘mc_dropout’ or ‘ensemble’. The former should output a scalar per instance. The latter should output a vector of predictions per instance.

  • n_evals (int) – The number of times to evaluate the model under different dropout configurations. Only relevant when using the ‘mc_dropout’ uncertainty type.

  • batch_size (int) – Batch size used to evaluate model. Only relevant when backend has been specified for batch prediction.

  • preprocess_batch_fn (Optional[Callable]) – Optional batch preprocessing function. For example to convert a list of objects to a batch which can be processed by the model.

  • device (Optional[str]) – Device type used. The default None tries to use the GPU and falls back on CPU if needed. Can be specified by passing either ‘cuda’, ‘gpu’ or ‘cpu’. Only relevant for ‘pytorch’ backend.

  • tokenizer (Optional[Callable]) – Optional tokenizer for NLP models.

  • max_len (Optional[int]) – Optional max token length for NLP models.

  • data_type (Optional[str]) – Optionally specify the data type (tabular, image or time-series). Added to metadata.

Return type

None

predict(x, return_p_val=True, return_distance=True)[source]

Predict whether a batch of data has drifted from the reference data.

Parameters

  • x (Union[ndarray, list]) – Batch of instances.

  • return_p_val (bool) – Whether to return the p-value of the test.

  • return_distance (bool) – Whether to return the K-S test statistic

Return type

Dict[Dict[str, str], Dict[str, Union[int, float]]]

Returns

  • Dictionary containing ‘meta’ and ‘data’ dictionaries.

  • ’meta’ has the model’s metadata.

  • ’data’ contains the drift prediction and optionally the p-value, threshold and test statistic.

class alibi_detect.cd.SpotTheDiffDrift(x_ref, backend='tensorflow', p_val=0.05, preprocess_fn=None, kernel=None, n_diffs=1, initial_diffs=None, l1_reg=0.01, binarize_preds=False, train_size=0.75, n_folds=None, retrain_from_scratch=True, seed=0, optimizer=None, learning_rate=0.001, batch_size=32, preprocess_batch_fn=None, epochs=3, verbose=0, train_kwargs=None, device=None, dataset=None, dataloader=None, data_type=None)[source]

Bases: object

__init__(x_ref, backend='tensorflow', p_val=0.05, preprocess_fn=None, kernel=None, n_diffs=1, initial_diffs=None, l1_reg=0.01, binarize_preds=False, train_size=0.75, n_folds=None, retrain_from_scratch=True, seed=0, optimizer=None, learning_rate=0.001, batch_size=32, preprocess_batch_fn=None, epochs=3, verbose=0, train_kwargs=None, device=None, dataset=None, dataloader=None, data_type=None)[source]

Classifier-based drift detector with a classifier of form y = a + b_1*k(x,w_1) + … + b_J*k(x,w_J), where k is a kernel and w_1,…,w_J are learnable test locations. If drift has occured the test locations learn to be more/less (given by sign of b_i) similar to test instances than reference instances. The test locations are regularised to be close to the average reference instance such that the difference is then interpretable as the transformation required for each feature to make the average instance more/less like a test instance than a reference instance.

The classifier is trained on a fraction of the combined reference and test data and drift is detected on the remaining data. To use all the data to detect drift, a stratified cross-validation scheme can be chosen.

Parameters

  • x_ref (Union[ndarray, list]) – Data used as reference distribution.

  • backend (str) – Backend used for the training loop implementation.

  • p_val (float) – p-value used for the significance of the test.

  • preprocess_fn (Optional[Callable]) – Function to preprocess the data before computing the data drift metrics.

  • kernel (Optional[Callable]) – Kernel used to define similarity between instances, defaults to Gaussian RBF

  • n_diffs (int) – The number of test locations to use, each corresponding to an interpretable difference.

  • initial_diffs (Optional[ndarray]) – Array used to initialise the diffs that will be learned. Defaults to Gaussian for each feature with equal variance to that of reference data.

  • l1_reg (float) – Strength of l1 regularisation to apply to the differences.

  • binarize_preds (bool) – Whether to test for discrepency on soft (e.g. probs/logits) model predictions directly with a K-S test or binarise to 0-1 prediction errors and apply a binomial test.

  • train_size (Optional[float]) – Optional fraction (float between 0 and 1) of the dataset used to train the classifier. The drift is detected on 1 - train_size. Cannot be used in combination with n_folds.

  • n_folds (Optional[int]) – Optional number of stratified folds used for training. The model preds are then calculated on all the out-of-fold instances. This allows to leverage all the reference and test data for drift detection at the expense of longer computation. If both train_size and n_folds are specified, n_folds is prioritized.

  • retrain_from_scratch (bool) – Whether the classifier should be retrained from scratch for each set of test data or whether it should instead continue training from where it left off on the previous set.

  • seed (int) – Optional random seed for fold selection.

  • optimizer (Optional[Callable]) – Optimizer used during training of the classifier.

  • learning_rate (float) – Learning rate used by optimizer.

  • batch_size (int) – Batch size used during training of the classifier.

  • preprocess_batch_fn (Optional[Callable]) – Optional batch preprocessing function. For example to convert a list of objects to a batch which can be processed by the model.

  • epochs (int) – Number of training epochs for the classifier for each (optional) fold.

  • verbose (int) – Verbosity level during the training of the classifier. 0 is silent, 1 a progress bar.

  • train_kwargs (Optional[dict]) – Optional additional kwargs when fitting the classifier.

  • device (Optional[str]) – Device type used. The default None tries to use the GPU and falls back on CPU if needed. Can be specified by passing either ‘cuda’, ‘gpu’ or ‘cpu’. Only relevant for ‘pytorch’ backend.

  • dataset (Optional[Callable]) – Dataset object used during training.

  • dataloader (Optional[Callable]) – Dataloader object used during training. Only relevant for ‘pytorch’ backend.

  • data_type (Optional[str]) – Optionally specify the data type (tabular, image or time-series). Added to metadata.

Return type

None

predict(x, return_p_val=True, return_distance=True, return_probs=True, return_model=True)[source]

Predict whether a batch of data has drifted from the reference data.

Parameters

  • x (ndarray) – Batch of instances.

  • return_p_val (bool) – Whether to return the p-value of the test.

  • return_distance (bool) – Whether to return a notion of strength of the drift. K-S test stat if binarize_preds=False, otherwise relative error reduction.

  • return_probs (bool) – Whether to return the instance level classifier probabilities for the reference and test data (0=reference data, 1=test data).

  • return_model (bool) – Whether to return the updated model trained to discriminate reference and test instances.

Return type

Dict[str, Dict[str, Union[str, int, float, Callable]]]

Returns

  • Dictionary containing ‘meta’ and ‘data’ dictionaries.

  • ’meta’ has the detector’s metadata.

  • ’data’ contains the drift prediction, the diffs used to distinguish reference from test instances,

  • and optionally the p-value, performance of the classifier relative to its expectation under the

  • no-change null, the out-of-fold classifier model prediction probabilities on the reference and test

  • data, and the trained model.