By John P Desmond, AI Trend Editor
Experiences with AI and machine learning at CVS Health and St. Luke’s Health System in Boise, Idaho, are having practical benefits to the two organizations.
Nels Lindahl, director of Clinical Decision Systems, CVS Health
CVS Health is learning how to scale AI applications using machine learning, especially through the house of machine learning operations (MLOps) tools, according to Nels Lindahl, director of Clinical Decision Systems, speaking in a virtual session at the recent Ai4 Conference held virtually recently.
And St. Luke’s Health Center put a COVID-19 prediction program, a supply chain purchase engine and a demand-based staffing application into initial production using AI and machine learning, said Dr. Justin Smith, senior director of advanced analytics at St. Luke’s, also at a recent Ai4 virtual conference session.
“We are at an MLOps tipping point, where ML has a growing production footprint, with adoption picking up pace and awareness and understanding at an all-time high,” stated Lindahl. “ML tech can now deliver; people are seeing real use cases in the wild and having them grow; it’s real.”
The three primary “ecosystems” for building out an AI footprint are Amazon Web Services (AWS), Microsoft Azure and Google Cloud Platform (GCP), he said. Developers are using open source tools and some they develop on their own to deliver value to their organizations.
He highly recommended having an ML strategy, with executive sponsor, defined budget resources, and a repeatable process to ensure the same approach can be replicated throughout an enterprise. “You need a narrative that helps you push things forward. When you can deliver on a use case consistently, you want to be ready to go,” he said.
In his professional role today, Lindahl is directing IT application development. He has delivered several complex process automation projects within the clinical decision space of pharmacy benefit management.
Tracking Popular API Services on GitHub
In April, he was tracking 19 external API services on axes of scale versus maturity, with for example AWS Enterprise Search and GCP Vision API rated high on the scale and maturity axes, so in the upper right quadrant. He recently updated his chart and is now tracking 49 external services. “The number of APIs out there is growing every day,” he stated.
The external services are becoming more specific, so that instead of one large vision API, it is breaking down into more specializations. “The exact thing you want to do is probably closer to something you can put into production from your API, ” he said. “The AI is out there and ready to go.”
He emphasized again the need for a development organization to have an overall ML strategy. “Just because you can go out and get an API, does not mean that is the right thing to do for your organization. It might be cool technology that is amazing, but there might be no return on that investment for your customer,” he said, adding, “Part of your ML strategy must be about purpose, replication, and reuse. Those are going to be at the heart of getting value back for the organization.”
He recommended looking at the GitHub Trending page, to see the popularity of different services. He tracks them over time, using indicators such as “stars,” which means people who have indicated a continuing interest, like a bookmark. In April, for example, TensorFlow registered over 154,000 stars and Pytorch was at 47,000; those attracted the most interest.
Lindahl is also tracking MLOps: machine learning operations tools that support practices to maintain ML models in production reliably. His tracking of MLOps on GitHub shows 10,000 stars for Kubeflow, a machine learning toolkit for Kubernetes, which is an open source container orchestration platform. What surprised Lindahl more was the rate of growth for MLReef, an open source MLOps platform with a focus on collaboration. “The number of people who downloaded it and are actively using it is going up really fast,” he said. He is researching it to see if he can understand what is driving the increased use.
Three ML Projects at St. Luke’s Delivering Business Value in New Ways
Justin Smith, senior director of advanced analytics, St. Luke’s Health System, Boise, Idaho
The emphasis of Dr. Smith at St. Luke’s was more on current delivered projects, and less on their technical underpinnings. The St. Luke’s complex has eight medical centers serving about one million patients. Dr. Smith described three projects.
The first was an application to predict the rate of spread of COVID-19, so that the hospital could do some better planning. “We were receiving all kinds of wild forecasts for what to expect,” he said. “We wanted to know if we could predict how many patients we would have in our ICU and on general hospital floors.”
The difficulty was that the forecasts were so different. Idaho had not experienced a wave of infections that happened in New York, for example. The team decided to document what they did know, then produce from that data two sets of forecasts: a short-term forecast for one or two weeks, and a medium-term forecast of up to 30 days. They used a new technique, XGBoost, an open source software library popular in applied machine learning for structuring tabular data. It implements gradient-boosted decision trees, which builds a regression tree in a stepwise fashion, measuring the error of each step and correcting it in the next.
“It got us to within five or six patients a month out,” Dr Smith said, using variables including the inpatient census and the positivity rate from their own health system. “We controlled 90% of the testing, so we had good data on positivity rates,” Dr. Smith said. “We showed with strong accuracy whether we would be increasing or decreasing the patient census.”
For the supply chain, the team was asked if they could develop a “purchasing engine” that could achieve savings with more optimal purchasing from the 100 vendors and 400,000 products in the supply chain. Certain volumes of purchase qualify for better pricing, posing an optimization problem. “With such levels of complexity, it’s too large for a human. You can’t solve it on Excel,” Dr. Smith said.
Breaking Analyst Habit of Sticking with Familiar Vendors
An examination of the practices of hospital analysts involved in purchasing, showed that many operated in familiar territory, often choosing vendors they knew and had dealt with in the past. Using advanced analytics, the team wrote algorithms that generated “billions” of scenarios. Providing a view of that data was challenging. “We don’t want to show everything or too little either,” Dr. Smith said, The team took the approach of eliminating the non-viable scenarios, sticking to spending within existing agreements, and to spend the required volume with each vendor to get the best price.
The team also tried to minimize the number of transitions, in which a product would be purchased from a different vendor. The analysis allowed the team to select opportunities representing the lowest number of transitions and the highest potential for savings across the entire health system. “It’s very powerful,” Dr. Smith said. “It’s being rolled out across the enterprise.”
The new system will require renegotiated contracts with a number of suppliers, so Dr. Smith expects it will take several years to become fully implemented.
The third project was around demand-based staffing. “We don’t want to be overstaffed and we don’t want to be understaffed,” he said. “We want to optimize our staffing to match historical patient demands.” While COVID did “wild things” to the patient census in 2020, that data was included. The system tried to avoid high labor costs associated with “on-call” roles, and also having to send medical professionals home when demand is low.
“The solution we created, which is still rolling out, is based on algorithms that offer an optimized schedule that matches staff and census according to predicted demand based on the last three years of data, including 2020,” he said. The system was focused on nurses and their support staff to start. Using a visual picture of the forecasts via Power BI, the business analytics service from Microsoft, human professionals were able to adjust the recommended staffing based on “front-line knowledge.” That would be, for example, knowing there is a football game, a music festival and a rodeo all in the same weekend, something the machine learning algorithm might not pick up.
“The output might be that we need six RNs to cover the 7 am to 7 pm shift, but just four for the 9 am to 5 pm shift,” Dr. Smith said. The system is beginning to be deployed and can project weekly staffing levels needed for four to six weeks out, he said. “It’s very dynamic, and senior executives can look and see how we are doing in hospital staff,” Dr. Smith said.
Learn more at the recent Ai4 Conference, at the GitHub Trending page, at Nels Lindahl and at Dr. Justin Smith.
