AI commissioning: How to move from a successful pilot to a full scale deployment
Alexander Khaytin
Alexander Khaytin
Mechanica had the pleasure of participating in Data Insights Week event by Big Data Expo. In his talk, our CEO and Co-founder Alexander Khaytin shares his thoughts on successful AI commissioning.
After finishing their first AI pilots and proving the value AI can bring, many industrial companies have realised that the hardest part is still ahead. Integrating AI into regular operations on the production floor brings completely new challenges, and the industry doesn't have much experience in tackling them.

Having deployed a number of AI-based services and supported them in the first months and years, we have gathered experience on how plants and factories deal with AI models in production.
AI solution adoption stages
In the last years, there are many successful pilots in ai adoption. But trouble comes in the next step — how to move from a pilot to a full scale everyday use.

AI solution adoptions start with an idea: use case scenarios, data availability and measurement procedure predefined at the start. Then comes the pilot project. At this stage, the model is developed and tested. It can be a predictive model or a prescriptive model, but it's the core part of the solution. And than it is tested in some limited scale, to make sure it runs smoothly.

However, there's an after-pilot gap. In daily use, unlike in the pilot, there are no highly qualified specialists and data scientists. You cannot measure success by comparing the results with no AI baseline from the past. The environment is not stable and controlled anymore.
That leads to a pilot purgatory: pilot projects that can't manage to transition die off
You might be ready to scale your solution. But gaining the trust of the shop floor workers in the real challenge.
Measurement approach
When introducing the solution, it is common to use the historical data as an initial benchmark. And though it works for a pilot, as you continue to use this method in a full-scale deployment, the baseline becomes more and more obsolete.

That's when it's time to move to A/B testing. That comes with its own challenges, and you have to pay attention to certain things.
Check the metrics and measurements procedures for the stability
Pay attention to the environment
Educate customer in advance and on the way
Robustness and reliability
Even the best models can go wrong at times. That is why it is crucial to work with the customer on a safe switch rule. It must be something that shows that the model isn't good enough anymore and it's necessary to switch to a backup system.
If the model fails, that can cause serious damage to your client. And developing a clear fallback plan with your customer will not only help to ensure everything runs smoothly, but makes your client trust you more as a specialist too.

That leads us to these takeaways:
Embed monitoring and alert triggers from the 1st day
Send regular reports daily / weekly
Have the fallback option available and agreed in advance
Lead your team to success
Questions and Answers
What's the most unexpected challenge that you encountered when implementing AI?

How fierce was initial resistance! Workers on the shop floor were trying to prove that solution doesn't work. They were not interested in the measurement of the performance. They tried to prove there's no performance. It took some time for us to get to a point where people trusted the solution enough to start implementing it and participating in the pilot.
Can you give an example in which regular performance measurement (A/B testing) was really difficult to do?

Really, in everyday use, any A/B test isn't very comfortable. Because in case you have a recommendation system, you need to split your recommendations in groups and in some cases you don't provide recommendations at all. And it's uncomfortable for the customer because they can't rely on the system psychologically. One time they get a recommendation, another they don't.

In many cases you can't do it in a simple way. You need to develop a design of A/B test which can feed to everyday life of a factory, and that's not so easy. We did it for different processes, and every time we needed to discuss it in details in advance, because people don't have an experience of A/B testing, and for them this concept is kind of alien. We need to introduce this concept in advance, and then find a way how to conduct this test comfortably.

In some cases we can compare groups, for example, one month to another month, in some we need to split events to groups A and B, in some we need to split it other ways. Every time it was difficult, but different problems arised.
Do you think that a certain level of understanding of the solution by the customer is key for successful adoption?

Unfortunately, no.

At the pilot stage yes, definitely: because we have a dedicated team from the customers side, and there are people who know how it works, they have an idea.

But people on the ground don't understand all this measuring stuff. They don't understand AI. They have their own business at hand, and so we can only provide them with a very high level ideas. And what we need to do is to make our solutions simple: we need to prove it works, we need to embed this measurement procedure in the solution so we don't need people on the ground to understand the technology itself. And, unfortunately, I don't think there's another option. Because your end user is an operator on a steel plant, he is an expert in steel making. And he's not going to be an expert in data science and AI at all.
More posts you may like