The Real Questions to Ask an AI Platform Vendor

Is Artificial Intelligence just a fad – or is it here to stay? That’s the question Gene Saragnese, CEO of MaxQ AI, posed to attendees during a discussion in the Machine Learning Showcase on Monday afternoon.

His answer? AI is with us for the long haul, with the opportunity to help us address some of the toughest medical challenges. “AI is here to stay. It's already having a profound effect on everything we do,” he said.

Saragnese proposed that, as a global society, we can’t afford to ask the wrong questions when evaluating AI solutions for healthcare. For example, globally in 2013 there were 6.5 million stroke deaths, making stroke the second-leading cause of death. On average, every 40 seconds, someone in the US has a stroke. It’s forecasted that by 2030, strokes will cause 3.4 million deaths for a total cost of $240 billion.

According to a 2016 Johns Hopkins study, medical errors are the third-leading cause of death in the United States, driving upwards of 250,000 deaths each year. Despite advancements in medicine, patient care and medical imaging, this rate has remained unchanged for 30 years.

Clearly, said Saragnese, radiologists need solutions and tools that go beyond data and analysis. They need technology that provides answers while seamlessly integrating into their current workflow, PACS systems, medical imaging hardware and healthcare clouds.

But how do radiologists evaluate an AI platform? Saragnese proposes a new set of probing questions that dig deeper to reveal the true value of assessing AI platforms. “There's a lot of confusion out there, so these are they types of real questions you should be asking me and my competitors,” he said.

Instead of asking: Does the application solve an unmet clinical problem? Does it work? How do I really know?

Ask this instead:

• Is this a meaningful clinical problem?
• What was the source of truth for the test set?
• What is the enrichment of the test set?
• IS the test set generalized?
• Only then ask, “What is the measured performance?”
• When can you demonstrate your performance at my site?

Instead of asking: Is it truly integrated in my workflow? How many steps?

Ask this instead:

• Is it integrated directly into my PACS results?
• Can I see results anywhere – scanner side, PACS, etc.?
• How many steps are required to get to an answer?
• Do I initiate the analysis? How many steps? How long do I wait?
• What image format is required? What is the format of the output?
• How much overall time will I spend to see results?

Instead of asking: Does it create an economic return? Is there a way to pay for it?

Ask this instead:

• Is this more analysis or answers?
• Does this deliver radiology productivity (e.g., greater throughput, less stress, faster read times)?
• Does this impact outcomes and quality (e.g., improved accuracy, speed to diagnosis and treatment, extend the reach of expersts)?
• Does this impact population cost for the payer?

Instead of asking: Can I trust the company to be there in six months?

Ask this instead:

• Is this a proven and trusted partner?
• Do they have service infrastructure to support me?
• Is there an upgrade path?
• Will I get improvements from continuous learning?
• Is this a standalone solution or part of a living ecosystem?
• Do I have a trusted point of contact?

If we ask the right questions, we can partner with the right AI vendor who can deliver actionable knowledge that will help radiologists make confident decisions. “Today we read images. Tomorrow, we should be interpreting and diagnosing patients,” Saragnese said. “To me, that is the power of the AI in healthcare, and I think it's a tremendous opportunity.”