In this article, we explore the application of practices of MLOps For optimize performance models of Machine Learning And ensure their reliability in production.
Louis REBERGA
Data Scientist
Introduction
In the world of Artificial Intelligence and Machine Learning, many organizations face the challenge of moving from a simple prototype to putting a model into production. The industrialization of a Machine Learning model indeed poses several technical issues or challenges, such as the evolution of the available data, the retraining and automated deployment of models, etc. In this article, we will discuss how MLOps practices can be used to optimize the performance of Machine Learning models and ensure their reliability over time in a production environment.
What problems are solved by MLOps?
Implementing an MLOps approach makes it possible to monitor and anticipate performance degradation of Machine Learning models over time. Once a model has been trained and deployed, it is essential to monitor its performance to ensure that it continues to produce accurate and reliable results. Without proper monitoring, a model's performance can degrade due to changes in input data, target data, or other external factors.
How does MLOps solve these problems?
The answer to our problem lies in the application of MLOps practices, particularly in model monitoring. There are different methods to monitor model performance and detect possible degradations:
Data quality: It is essential to monitor the quality of the data used for model training and inference. This includes checking data volume, column integrity, data type consistency, etc. Any deviation from expectations can be an indicator of a potential problem.
Detection of aberrant data (outliers): In order to improve the reliability of our model and its predictions, we add an anomaly detection model whose objective is to evaluate the input data of our prediction model. Sometimes strange data can appear, very different from the training set, and the prediction is more likely to be wrong.
Data drift: Data drift occurs when new input data changes or differs from what the model was trained on. It is important to detect these drifts to avoid degrading the prediction performance of the models. Statistical methods, such as the Jensen-Shannon divergence and the Wasserstein metric, can be used to measure this data drift.
Conceptual drift : Concept drift occurs when the relationship between the input data and the target data (the data we are trying to predict) changes. This may occur due to changes in the external environment or underlying processes. Although there is no direct statistical method for evaluating conceptual drift, it can be monitored by comparing the drifts of input data and target data.
Model performance evaluation: It is essential to collect data on model predictions and compare them to reality when available. This allows the accuracy and consistency of the model to be measured under real-world conditions.
Estimated model performance: Estimating model performance without knowledge of the actual target data can be done using methods such as Confidence-based Performance Estimation (CBPE). This allows you to get an idea of the expected performance of the model before the actual data is available.
Interpretability: This part does not directly monitor model performance, but interpretability is a key topic in MLOps and can be useful for understanding the origin and impact of data/conceptual drift.
Focus on data drift monitoring
NannyML is an open-source Python library that makes it easy to monitor Machine Learning models. NannyML is designed to detect silent model failures: estimate model performance after deployment, detect data drift, and monitor performance once target data is available.
Thanks to this library, we can monitor data drift in a univariate manner. That is to say, monitor a drift in the distribution of each variable independently. Here is schematically how this drift is monitored:
It is also possible to monitor data drift in a multivariate manner. That is to say, monitor a drift in the distribution of all variables at the same time. Here is schematically how this drift is monitored:
Conclusion
In conclusion, MLOps practices play an essential role in optimizing the performance of Machine Learning models. Thanks to tools such as NannyML and the implementation of monitoring dashboards, it is possible to continuously monitor model performance in real time, anticipate degradations and detect data drift. This ensures that model predictions are reliable and relevant in a production environment. Although implementing these techniques requires technical expertise and close coordination with business experts, the quality benefits of future model deployments make it worth investing in MLOps practices.
A must see
