In recent while I have come across several articles and conversations where Data Science & Analytics leaders across industries have been expressing concerns over performance and reliability of AI/ML. The problem of concept drift or data drift is not new, it is just that it has made its presence felt very significantly with COVID-19. In this article, I will be talking about some best practices to ensure creation of resilient AI/ML solutions to tackle such data shifts.
AI/ML has an extremely broad use-case across all industries and verticals. However, not all industries or use-cases are getting impacted by COVID triggered data shift. As I assess and see around, only areas where AI/ML is being used to capture consumer/end-user behaviour have been impacted, because that is the only thing COVID has changed. So, for e.g., your Alexa has not suddenly stopped understanding English or Self-Driving cars will not forget how to drive, because there is no user behaviour involved here. What has changed is our lifestyle, daily needs, online presence etc. In a B2B market, what has changed is how companies are carrying out their business, what products are they consuming vs which ones are they shedding off. On top of these changes, the other problem is that the change is volatile. We are seeing shifts, which themselves are driven by a lot of extraneous factors on weekly basis.
The problem primarily exists because AI/ML uses historical data to profile behaviours and predict future events. If the recent behaviours have changed drastically and continue to be volatile, it makes sense that static AI/ML solutions would not stand a chance and would be inaccurate at best and misleading at worst.
While there is no definitive and exhaustive guidebook to get around this, and depending upon the problem statement at hand, one may have to employ one or more strategies in different permutations and combinations. Below are some of the ways that I think can help in that direction.
One of the key things I have learnt as a data science leader is that a successful data science team should always have capability to monitor underlying data and analytical features. This is a key step to take control of your advanced analytics strategy, where you have a direct line of sight on how your data is faring and how your models are performing.
I am not talking about ad hoc monitoring, which I sense most of us already do on demand as we see our end goals are not being met by an analytical solution. I am talking about creating proactive measures with ability to alert solution owner on possible deterioration in data shifts, outcome degradation etc.
Agility is key to everything and so is AI/ML. Ask yourself on whether your model requires retraining for every change in feature distribution or has some degree of dynamism built into the way you have engineered features and your model. What is the kind of effort it would take to make your solution more sensitive to recent data to track recent user behaviour? How easy is it to make human corrections to the weights that you are giving to different decision parameters? Can rule based business filters be applied over your AI/ML workflow?
Dynamic features, self-learning models, ability to increase sensitivity of your solutions and ease of introducing human corrections in your AI/ML decision flow are key in deciding how agile your data science & analytical solutions are in making quick tweaks to accommodate for changing data patterns. A termed that is being coined now a days is MLOps similar to DevOps in Technology to bring a similar agility in Data Science domain.
Because what we call as normal has shifted, it does warrant that data scientists revisit the definitions of their outcome variables to see if what they have been calling abnormal is the new normal and vice versa. Can our data fall in a new category that was inexistent prior or have some categories vanished? Questions should also be asked on how your model treats new data points that have not been seen earlier during training? Does it outcast them as a positive or negative outlier or consider them as a part of the normal population, more of a use-case based decision that may need to be taken.
While this section may not be applicable everywhere; however, wherever possible one should explore for methodologies that do not require a normal profile to train rather have intrinsic ability to find outliers in the data basis the given sample. The statement is mainly applicable for outlier based detection systems in unsupervised and hybrid setups.
Given that our current data may have shifted drastically from what we have seen in past, trusting past model performances that we may have established during training, test and validation would no longer provide the real picture. Data science units should move as much as possible to intime validation setups using A/B testing to be able to observe how model is faring against a scenario with no AI/ML solution in the current setup.
As I read it somewhere, AI/ML is living and breathing. So, as with any other living and breathing object, it requires constant care and intervention. One cannot build an AI/ML solution and go away. Data science setups in my perspective should invest around 20-30% of their effort in monitoring, management, and potential enhancements to be successful.
Are you already leveraging some of the above to evaluate your AI/ML solution performance? What other techniques is your team employing to curb the potential impact of frequent data shifts?
-By Mayank Kumar, Data Science Consultant