In this article, we explore the application of the practices of MLops to optimize performance models of Machine Learning and ensure their reliability in production.
In the world of Artificial Intelligence and Machine Learning, many organizations are facing the challenge of moving from a simple prototype to the production of a model. The industrialization of a Machine Learning model indeed poses several problems or technical challenges, such as the evolution of available data, the retraining and automated deployment of models, etc. In this article, we will discuss how MLOps practices can be used to optimize the performance of Machine Learning models and ensure their reliability over time in a production environment.
Implementing an MLOps approach makes it possible to monitor and anticipate the deterioration of the performance of Machine Learning models over time. Once a model has been trained and deployed, it is essential to monitor its performance to ensure that it continues to produce accurate and reliable results. Without proper monitoring, model performance may degrade due to changes in input data, target data, or other external factors.
The answer to our problem lies in the application of MLOps practices, in particular in the monitoring of models. There are various methods for monitoring model performance and detecting possible degradations:
Data quality: Monitoring the quality of the data used for model training and inference is critical. This includes checking the volume of the data, the integrity of the columns, the consistency of the types of data, etc. Any deviation from expectations can be an indicator of a potential problem.
Detection of outliers: In order to improve the reliability of our model and its predictions, we are adding an anomaly detection model whose objective is to evaluate the input data of our prediction model. Sometimes strange data can appear that is very different from the training set, and the prediction is more likely to be wrong.
Data drift: Data “drift” occurs when the new input data changes or differs from what the model was trained on. It is important to detect these drifts to avoid the deterioration of model prediction performances. Statistical methods, such as the Jensen-Shannon divergence and the Wasserstein metric, can be used to measure this data drift.
Concept Drift : Concept drift occurs when the relationship between input data and target data (the data you're trying to predict) changes. This can happen due to changes in the external environment or underlying processes. Although there is no direct statistical method for evaluating conceptual drift, it can be monitored by comparing the drifts of input and target data.
Model performance evaluation: It is essential to collect data on model predictions and compare them to reality when available. This makes it possible to measure the accuracy and consistency of the model under real conditions.
Model performance estimate: The estimation of the performance of the model without knowledge of the real target data can be carried out using methods such as CBPE, from the English Confidence-based Performance Estimation. This provides an idea of the expected performance of the model before actual data is available.
Interpretability: This part does not directly monitor model performance, but interpretability is a key MLOps topic and can be useful for understanding the origin and impact of data/conceptual drift.
NannyML is an open-source Python library that easily allows you to monitor Machine Learning models. NannyML is designed to detect silent model failures: estimate model performance after deployment, detect drifts in data, and monitor performance once target data is available.
Thanks to this library, it is possible in particular to monitor data drift in a univariate manner. That is to say, monitor a drift in the distribution of each variable independently. Here is schematically how this drift is monitored:
It is also possible to monitor data drift in a multivariate manner. That is to say, monitor a drift in the distribution of all variables at the same time. Here is schematically how this drift is monitored:
In conclusion, MLOps practices play an essential role in optimizing the performance of Machine Learning models. Thanks to tools such as NannyML and the establishment of monitoring dashboards, it is possible to continuously and in real time monitor the performance of models, to anticipate degradations and to detect data drifts. This ensures the reliability and relevance of model predictions in a production environment. While implementing these techniques requires technical expertise and close coordination with business experts, the benefits in terms of the quality of future model deployments are worth investing in MLOps practices.
