The Future of AI in Otolaryngology with Dr. Philip Edgcumbe
Our 25th episode interviews Dr. Philip Edgcumbe, CSOHNS 2024 Keynote Speaker, about the future of AI in healthcare and otolaryngology. Join us to learn more about this exciting area of innovation and how you can be at the forefront of new surgical technologies.
Show Notes
Alison:
Hi everyone and welcome to The Oto Approach - a podcast created by medical students, for medical students, to teach you about all things Otolaryngology. I’m your host, Alison, and I asked Dr. Edgcumbe to speak to us here at the Oto Approach after he was a keynote speaker at the 2024 Canadian Society for Otolaryngology - Head and Neck Surgery meeting in Montreal, where he educated Canadian otolaryngologists on the future of artificial intelligence, hereby referred to as AI, in the specialty.
Dr. Philip Edgcumbe is a radiology resident at the University of British Columbia with a PhD in biomedical engineering and a Bachelor of Applied Science in engineering physics. Alongside his residency, Dr. Edgcumbe has an interest in exponential and disruptive technology. He helps health care organizations adopt the latest innovations to improve the efficiency of care and plan for the future of healthcare. He is also the co-founder of a start-up company called Map Medicine.
So, thank you for joining us today, Dr. Edgcumbe, could you please start by telling us a bit about yourself?
Dr. Edgcumbe:
Well, Alison, it's great to chat again. Just before jumping into to my introduction, I just wanted to say it's an honour to be on the podcast. It's great to connect with you again. It was nice to meet you at the CSOHNS conference, as well as lots of other medical students and residents and physicians in the world of ENT. It's going to be really fun to have a conversation with you about AI in healthcare.
Alison:
Awesome. I'm happy to be speaking with you again too, thanks so much.
Dr. Edgecumbe:
One of the themes that I'll touch on during our discussion is my strong belief that especially when it comes to AI, it's imperative that we have young physicians and medical students that are engaging fully because this is uncharted territory for everyone and it's almost to the advantage of the naïve, you know, medical student who doesn't know what can't be done to be working on this topic. So, I guess that in a sense may represent a little bit of my own career journey in that I started as an engineering physics student and decided to do the UBC MD PhD program because I really wanted to understand first and foremost the problems before trying to apply kind of technological solutions to improve the world of health care. And that is something that I worked on during my PhD. My focus there was surgical robotics for kidney cancer surgery, and in my residency I've continued to have a research interest in biomedical engineering as well as really trying to stay engaged and on top of the advances in the world of AI and health care.
One of the things that I did during my PhD back in 2016, is I participated in this 10-week course called The Global Solutions Program at Singularity University. It was based on a NASA base in Palo Alto, and there were speakers there who spoke about exponential technologies, and that's in domains such as you know, healthcare but also transportation, computing, agriculture, and that experience really opened my eyes to how quickly technology is changing in a whole bunch of fields. And, I think that has continued to inspire me to try to stay at the forefront of the opportunities for AI in healthcare.
Alison:
Yeah, it can definitely be a bit intimidating for medical students to try to dive into this world of new technology, so it's really nice to hear from someone who's been there and who has taken that courageous leap into the unknown and the future. And, I do understand that you've done that as part of your start-up company called Map Medicine?
Dr. Edgcumbe:
Map Medicine is a company that I've co-founded with Dr. Catherine Isaac, a plastic surgeon, and Tony Yang an engineering student. We've called it Map Medicine because we are trying to provide a map of the patient's anatomy on the patient during the surgery so that the surgeon can see into the patient before they cut. And, we're using the Apple Vision Pro headset that was released in the US in February of this year, and in Canada in, I can't recall if it was June or July of this year, and the Apple Vision Pro headset gives us the ability to overlay preoperative imaging and segmented anatomy onto the patient. And so we're really excited about the opportunity to use this to augment the surgeon, to give them extra information to help them find you know, small little blood vessels associated with a flap for example or to help them put that incision in just the right place to be able to access the tumor. And, we think that this extra information for the surgeon when they put on the Apple Vision Pro at particular points in the surgery will actually reduce operative time and complication rates.
And so that has been a real joy for me to work on, and actually at the CSOHNS conference, I was there with, at that time it was the Meta Quest 3 headset, and I must have had maybe 25 or 30 surgeons try on the headset and it was really exciting for me to see the, I guess, the openness that the ENT surgeons had, the interest that they had, and I would say about half of them said, well, I'd like to have something like this in my operating room. So that was a great signal that we're building something that surgeons are going to use, and that will help patient care.
Alison:
So, Dr. Edgcumbe when you say overlaid images do you mean that you're looking through the vision goggles at the patient and their own anatomy is being overlaid on themselves?
Dr. Edgcumbe:
Yes, and these, the Apple Vision Pro goggles, or the Meta Quest 3 goggles, the way they work is, they have these little video cameras in the goggles that are taking a video of the real world, and then those are displayed to the surgeon on these wrap-around glasses, and in the case of the Apple Vision Pro, it's at 4K resolution. And so, you're watching potentially a video of the operative field, and you can see the patient. And, then on top of that since it's now a screen, we can put the CT scan, we can put a particular segmented anatomy, and we can place it with a fair amount of accuracy onto the patient, so that yes, you get to see the CT scan ‘quote on quote’ overlaid onto the patient. And, it's like you're finally getting to review the preoperative imaging in three-dimensional space. It's no longer only something that you could review on the computer screen in the corner of the operating room.
So, I'm really excited to continue to move this forward and I'm hopeful that say, in 5 or 10 years time, in surgical residency programs that it'll just be part of the accreditation that all residents have to have exposure to this technology because it's going to become more and more prevalent in a surgery in the future.
Alison:
Well, just hearing about this technology makes me excited to hopefully interact with it sometime in the future, and I'm sure a lot of our listeners are also feeling the same way.
Dr. Edgcumbe:
Well, I think you might have actually already interacted with it if I recall.
Alison:
Yes I did! I did try on the glasses at the conference. It was really fun. I remember there was a whole lineup of students and physicians alike waiting to get their hands on those glasses to see what it was like. I mean, to me, it was my first ever exposure to like a goggle, or wrap-around headset like that, and it felt very surreal to be able to try them on in such a relaxed and fun setting as the social that night of the conference.
Dr. Edgcumbe:
If I remember correctly, it was a social and a converted Catholic church, was that right?
Alison:
It was a social in a converted Catholic church, with circus performers hanging from the ceiling.
Dr. Edgcumbe
Yeah, so, it was like augmented reality of another kind.
Alison:
It already felt like I was in an alternate universe when I was inside the church, and then I put on the goggles on top of that and it was a lot!
Dr. Edgcumbe:
The alternate universe in the alternate universe.
Well, I mean, I love getting to get feedback from yourself and others, and I was really glad that everyone was willing to just try on the glasses, and imagine what that would look like for the future of surgery.
I also have a bit of a funny personal story which is, on the first night of the conference, I had met Dr. Leitao, who's a pediatric Otolaryngologist, really friendly guy, who was very kind, because I was a bit of an outsider, I guess. And, he was happy to kind of introduce me, and I told him that the only other pediatric Otolaryngologist that I knew was Dr. Kosak. And, I said Dr. Kosak was a bit of a household name in my family because he had actually been the doctor that had looked after my brother for 30 years, because my brother had some hearing concerns. We always looked forward to or I would always hear about the appointments with Dr. Kosak, and he said, well let me you know take you over here and he's like, I present to you the one and only Dr. Kosak. And, It's pretty fun for me to meet Dr. Kosak in person.
Alison:
You finally got to meet the legend you heard about at home.
Dr. Edgcumbe:
Yeah, I had met him as a medical student, but this was fun to meet him again. And, he remembers my brother because my brother was one of his first patients as a new staff physician, and now, Dr. Kosak is an experienced staff physician, but still full of energy and has lots of good jokes to share. So, that was the first night of the conference, and I certainly felt you know, right at home after that, after a warm welcome from Dr. Darren Leitao, and then meeting the superstar doctor of my childhood, one degree removed, being so good for my brother.
Alison
Yeah, the conference… I expected to be educational, and it definitely was, and I learned so much, and was really inspired by all the speakers I listened to. But, I also got to meet so many amazing people in the field, and it was more social than I thought it was going to be, and I really got to meet these people in both the professional and educational setting as well as a more social and conversational setting, as well.
Dr. Edgcumbe:
Anyway, so it was really fun for me to get to know the ENT community more in person, and to get the kind of feedback and interest, that was quite nice.
Alison:
So, that Map Medicine, that is an example of some of the exponential technology you're involved in, is this the same as disruptive technology?
Dr. Edgcumbe:
Yeah, I would say so. I think the concept of exponential technology is highly interlinked with the concept of disruptive technology.
The quintessential example of exponential technology is computational performance. And, and we've all experienced that in our lifetimes, in that the classic Moore's Law predicts that the density of transistors on an integrated circuit doubles every 18 months to 2 years, and so then if you're taking exponential steps that's 2, 4, 8, 16, 32, 64. And, the technology improves you know, dramatically in an exponential way. And when that happens it has the opportunity to disrupt. You know a few examples that I'm sure we're all familiar with is how Netflix, you know, disrupted the Blockbuster model, or Uber disrupted the taxi model, and that was underpinned by you know exponential technology and computational power, internet streaming rates or ability to do GPS mapping. Those exponential technologies did become disruptive as well.
Alison:
Okay, so would I be right in saying that disruptive technology is almost the big major shifts we have in the way we do things?
Dr. Edgcumbe:
Yes, I think it's, you know, a technology that potentially can disrupt significantly how we do things.
Alison:
So, what originally drew you to being involved in this disruptive technology in health care?
Dr. Edgcumbe:
I mean I trained as an engineer so, I was always fascinated by technology and physics, and it seemed to me like the most meaningful place to work (and I'm sure this is a realization shared by many of the listeners) is in medicine. And so, I started my career with a goal to bring technological solutions to medicine, the quintessential bridging of the bench to bedside, but I guess mine might be the computer science lab to the bedside. And, so, by virtue of that, it's important to understand which technologies are going to be exponential, which are going to be disruptive. One of the, I would say, the biggest proponents of the concept of exponential technology is this scientist Dr. Kurzweil and, he was someone who was a pioneer in speech-to-text recognition in the 80s and 90s, and he was originally an entrepreneur, and he became a futurist because as an entrepreneur, he was always trying to anticipate where technology would be at 2 or 3, or 4 years in the future so that he could start building businesses, and designing tools, that would perfectly intersect with the technology performance, improvement curves, such that when he was finally ready to launch, the technology was there to support his product that would be able to you know, actually work. And so, I think that, you know, the combination of wanting to build solutions that work in health care naturally motivates a desire to understand technological trends, and the evolution of technology.
Alison:
Okay. Very cool, thanks for explaining all that. I guess, one of the technology trends there's a lot of talk about, not only in the classroom, but in the media, is artificial intelligence, and I know that you are an expert in artificial intelligence, and I was just wondering if you could please review it for us and tell us a bit about what AI is.
Dr. Edgcumbe
Yes. I would love to. There are many definitions for AI but, I would say that artificial intelligence refers to the field of computer science dedicated to creating systems or machines capable of performing tasks that typically require human intelligence. And it's been the exponential growth of computational power, the exponential growth of our ability to store data, that has enabled tremendous progress in AI and, some examples of AI at work, and impressive AI breakthroughs, in the last few years include things like the AlphaGo competition, where an AI was able to beat the best human at the game, Go, which is like a very complicated game in terms of the number of moves. So, it's kind of like chess… plus plus.
A related AI breakthrough was the development of AlphaFold, which was an AI program that predicted the protein structure based on just the amino acids. And, what AlphaFold allowed us to do is to go from 200,000 known protein structures to effectively 200 million or all of the proteins in the known universe now knowing what their protein structures are. And, that's going to have all sorts of really transformative impacts in terms of drug development in the future.
And, then object image recognition has been also a great example of AI, where you know, computers are able to identify objects in images at or beyond that of human's abilities. And, then I think the most obvious example that's probably front of mind for everybody has been the large language models that came out you know originally with ChatGPT and have shown just phenomenal abilities in terms of being able to really understand language and allow us to interact with computers with just language.
So, those are all examples of, you know, significant artificial intelligence breakthroughs.
Alison:
Yeah, one thing I find really interesting is how you defined artificial intelligence as systems or machines that can perform tasks that typically require human intelligence, but it just seems like the computers, or machines, or systems are going above and beyond what we do, and doing it at such a faster rate. I can only imagine it's going to be having a huge impact on the medical world.
Dr. Edgcumbe:
Yes, I wholeheartedly agree. I would also say there's a much simpler definition to AI which can also be useful to keep in mind and this one comes from Ajay Agrawal, who's a professor at the University of Toronto. And, he actually wrote a book called Prediction Machines and, in that book he really made the case that AI is just something that's good at making predictions. So, an AI can predict a protein structure. An AI can predict what is the next move that's good to make in Go. An AI can predict what is the right answer to a question. And, so his perspective was just think of AI as this incredibly powerful prediction machine.
To your point, Alison, a lot of medicine is about prediction, and if you can predict what disease someone has, or predict how they're going to do with a particular treatment, that's very significant, and very important. If you can predict, you know, which patient most urgently needs a surgery or predict which patient should have been discharged but hasn't, I mean there's so many potential applications.
Alison:
Yeah, well it just seems like predicting is what physicians and other health care workers are trying to do every single day. So, to think that we could have a technology that can help us with that, is just very exciting.
Dr. Edgcumbe:
I would just say, we do a lot of predicting on a regular basis but, we also do a lot of judgment. And, that is something which is still probably best done by humans in that, it's all very well to predict and diagnose that someone has some kind of neck cancer, but then, it's up to them and their physician to decide what is the best treatment for them. You know, some people would rather surgery over radiation or couldn't fathom the idea of having a dry mouth for 30 years. So, there's inherently like a judgment aspect that comes in which is uniquely human, and I think that's the real value that physicians will continue to bring to the table for many years.
Alison:
It's really nice to hear, because I think a lot of physicians and medical students are worried about what their role will be in health care, if AI is involved but, bringing the judgment into it kind of reminds us that although the systems and computers and machines can be very smart and efficient there's something uniquely human about health care and the compassion and the understanding that comes with it that these systems aren't able to do just yet.
Dr. Edgcumbe:
Yeah.
Alison:
Do you have any examples of the impact AI has had in the medical world, like, in the past few months?
Dr. Edgcumbe:
Yeah, I think there's quite an exciting study that came out in April in Nature Medicine, and the study is called AI Enabled Electrocardiography Alert Intervention and All-Cause Mortality: A Pragmatic Randomized Clinical Trial. So, it's a bit of a mouthful but, up until April 2024, there'd been lots of studies of AI in hospitals, and most of those studies looked at endpoints like detection accuracy, maybe even discharge time, things like that, but the study that I just referenced was one of, if not the first, example of a randomized control trial where the they showed that the deployment of AI in a hospital actually reduced death, like reduced all-cause mortality.
Specifically this study was a randomized control trial with 16,000 patients at two hospitals in Taiwan. The intervention group had AI ECG assessment and alert systems. So, the physician was alerted if the AI thought that there was a concerning finding on the ECG, and then, could proceed accordingly, oftentimes reassessing the patient, and sometimes elevating their level of care. And what they found was that among the high-risk group patients, there was actually a 31% relative reduction of deaths, at 90 days for all-cause mortality in the AI group and that was pretty impressive.
Alison
Well, of course and part of health care is doing anything we can to make someone's life better or to prolong someone's life in a meaningful way. So, if AI can help us do that, why wouldn't we want to adopt it?
Dr. Edgcumbe:
Yeah. Why wouldn't we want to adopt it? That's a very good question. How long do we have? Um, yeah we’ll stop there but, there's lots of things to consider about rolling out a new kind of technology and intelligence into a health care system.
Alison:
Yeah, that’s a whole debate that could probably take a couple more hours of podcasting. If we were to get more specific about AI in Otolaryngology, or surgery, what is the role of AI there?
Dr. Edgcumbe:
There are two FDA-approved AI machine learning algorithms in the field of ENT, one is called VisionEar and FlowEar, from this company called Pacific MD Biotech and then, there's another AI ML enabled medical device that BrainLab has the approval for and, that's related to helping with surgical navigation and image-guided surgery. So, there are two products that are available for purchase that are ultimately underpinned by AI in the field of ENT.
But I guess, the thing about surgery is yes, it's a tactile thing, in that you're cutting and sewing, but it's also a very much a visual thing. And so, there's lots of interesting opportunities for AI, as like an image recognition tool in surgery. And someone who I've really learned a lot from and turned to as a real kind of thought leader in this space is Dr. Pieter De Backer, he's a friend and colleague of mine, he's actually a Urology resident in Belgium, but he also is the Surgical Artificial Intelligence lead for this research institute called The Orsi Academy. And so, in our conversations, some of the areas that he's highlighted as AI being useful in surgery includes automatic smoke removal so that it just kind of cleans up the view digitally if you're using a laparoscopic camera. AI can always be watching during a surgery. So, for example it can track when a needle goes out of frame, which is something to be avoided, it can potentially assess the quality of a needle or a suture bite, the quality and the consistency of one's grasp of a needle when they're picking it up with a surgical instrument, and so if you have this AI analysis running in the background at the end of one's surgery, or practice in the cadaver lab, or whatever the AI can actually generate a report card and it can say this is how many bites were good, this is how many times you grabbed the suture properly or incorrectly, this is how many times the suture went out of frame. And, it could even take you back to those moments in the surgery where it was working well, or where it didn't. And, I think that kind of always-present coaching can really help with just the fundamental skills of surgery.
Alison:
Totally, that's really cool. As physicians and surgeons, I think everyone is always trying to improve their technique and make everything more efficient and effective. So the idea that AI could play a role in this is exciting and really interesting, something I definitely haven't thought of when I think of the role of AI in medicine.
Dr. Edgcumbe
Yeah, and you can go a step further and, this is something that Pieter De Backer and his team have been working on, which is to really identify in a very systematic way, and it starts, this starts with humans. The human experts have to define in a surgery “what are the common steps in a surgery” and for a radical prostatectomy, there could be 81 steps. And, then within each of those steps “what are the operative errors that could occur?”. Well, you could accidentally cut the ureter at this point, or the aorta is nearby at this point. I don't know the specifics but, you can imagine what all those operative errors are. And, then of those operative errors, “what are critical errors”. And then, you can train an AI algorithm to potentially identify what step you are in the surgery and to be looking to see if those operative errors are occurring or flagging to the user that there's a risk of these particular operative errors, at that point, and the dream is that in the fullness of time, that, you have such a rich data set of surgical outcomes, with the surgical videos that you can really start to apply like data science to say okay, well there were a few mistakes that were made in the surgery, did it affect the operative outcomes? And really get a very granular understanding of the moves and the steps in the surgery that really matter. And so that's also something that kind of intelligence can just always be running in the background and further informing surgical approaches.
Alison:
Amazing. Thanks for going through all that with us. And I remember Dr. Edgcumbe, when we were talking, before we started recording you mentioned something about eustachian tube balloon dilation, a way AI can shape Otolaryngology. Could you talk a little bit about that?
Dr. Edgcumbe:
Yeah. I mean I should say when I was at the CSOHNS conference, I really enjoyed connecting with Dr. Jennifer Siu. She had a presentation there on AI and surgical skills evaluation and application to Otolaryngology training. So I think, there are already Otolaryngologists that are working on AI for surgical skill assessment and training and so that's really exciting that there is that expertise and interest within Canada.
As it relates to that question about eustachian tube dilation, that's another potential application for AI and surgery. There was a paper that recently came out, that was published by a friend of mine and recent UBC ENT resident grad, his name is Dr. Ameen Amanian, and he published a paper in Otolaryngology Head & Neck Surgery and the title was A Deep Learning Framework for Analysis of the Eustachian Tube and the Internal Carotid Artery. So the question is, well, why does that matter? And what Ameen explored was automatic segmentation of these two structures, because when a surgeon is doing a eustachian tube balloon dilation, there's always a risk that they could damage the internal carotid artery. And, just if you kind of over dilate, the internal carotid artery is near the eustachian tube, there's a risk, small risk but, a significant complication, there. And so what he proposed, was using deep learning to automatically segment these structures and calculate the distance between them, and so, one can imagine, in the future you let's say, Alison, as a you know, future ENT surgeon going into a case for eustachian tube dilation balloon dilation and not only can you review the preoperative imaging and the radiologist's report but you also have this automatically generated metric which is this is the distance from the eustachian tube to the internal carotid artery, and that's just always running in the background based on segmentation of those structures.
Alison:
Very cool. So all of this research and innovation done by Canadian physicians, and Canadian medical student graduates, is really exciting, but I think, it can be a little daunting for current medical students who are worried about transitioning into practice and transitioning into this exciting time of AI innovation. I was wondering if you had any advice for people like myself who may be a bit daunted by this new exciting world of AI that will be coming into the healthcare system, in the future, and now I guess, already, as you spoke about.
Dr. Edgcumbe:
I mean it's a great question, there's a lot to learn in the conventional world of medicine. Just getting through clerkship is an accomplishment in and of itself and it's important, at all stages of training to make sure that first and foremost you're a competent medical student, or resident, or physician, and able to achieve the standard of care that you're expected to be able to do. But, above and beyond that, you know, I believe that there are lots of medical students and residents that can find extra meaning and purpose and impact in their career by exploring the world of research and especially thinking about opportunities for the application of AI in health care.
And, I'd say we're all learning together. You know, ChatGPT only came out within the last few years and, so there is no one who's necessarily an expert on it. And, so, medical students are just as well placed as an attending with 30 years of experience to be diving in, maybe even better placed, and thinking about where these new AI tools could be applied in health care.
Alison:
That's really cool, and it's inspiring to know that, even a lot of medical students, like myself, are only so far into their current education, it's nice to know that we can still get involved in this exciting area of innovation. And, I think you mentioned before that you knew some current students, or residents that are already involved in AI research?
Dr. Edgcumbe:
Yeah, I think it's you know it's important to highlight that there's already lots of medical students that are diving full-on into this world of AI research, I'd highlight two here, one is Eddie Gou, he developed OSCE GPT which you can go to, just by OSCEgpt.com and effectively his thought process was, can I just create a bunch of OSCE scenarios, put it through ChatGPT and ask it for feedback both on the medical diagnostic accuracy, but also on you know the way I was conversing with the simulated patient?
And he has built out a pretty impressive OSCE curriculum, so you can go and practice OSCE's in all sorts of subspecialties on that website And so, he's done that as a medical student and has been the co-lead on that project.
Alison:
And, what an exciting thing for medical students like myself and others that are listening, that now we have this resource that we can go to and help us study for our exams.
Are there any other students besides Eddie?
Dr. Edgcumbe:
Daniel Shivani is a UBC medical student. I know him well because we're at UBC and we have this shared interest in entrepreneurship and innovation, he actually came out to my beach birthday celebration just yesterday. So, we were catching up about his start-up, which is called QUAS AI and it's an AI scribe built around ChatGPT. And so it's a product which will listen in to a patient encounter, and provide an appropriate summary of that encounter. It generates kind of a SOAP note on the fly. My girlfriend is a vet and she actually tried using it in her practice, and was very impressed with it and I think there's lots of physicians in BC that are now using it. And Daniel is rolling it out more broadly across Canada, you know, he had a some background in computer science, not an actual degree but an interest in coding, and when ChatGPT came out he just started to work with ChatGPT to do the prompt engineering on the back side, which is effectively what is the question? How do you set it up, such that ChatGPT does a good job at writing a note? What is the prompt that you need to give it? You know you need to ask that it say, you know, pretend that you are an experienced physician interviewing a patient in the summary, don't mention yourself (as in the AI chatbot) and write a note that an expert physician would write after doing this interview, or something like that. And by the way you might benefit by reviewing this chapter on these common ENT conditions before you do this summary because that might be helpful for you.
So that's what's running in the back end, and Daniel's done the work to address the legal considerations around storing and sending some of the information down to the ChatGPT servers, at openAI they delete all of the recordings within a month. And, so he's built a real product that physicians are using everyday and I believe you could take that, and do that over and over again in all sorts of different specialties or in the realm of patient education about certain medical conditions, the sky’s really the limit.
Alison:
That's really exciting, especially to hear about people around the same place in their education as myself and most of our listeners. So, thank you so much for sharing those examples with us. Is there anything else, we didn't talk about yet that you would like to mention to our listeners?
Dr. Edgcumbe:
Well, I think you know, we've talked about large language models vis a vis, kind of understanding language. We've talked about image recognition for detecting were you holding the surgical needle correctly, what stage of the surgery are you in? But, really what has really just come onto the scene in the last year is this concept of multi-modal AI. So, can you build an AI that not only listens, but also,watches and then puts that information together?
We do that all the time in that say we're walking through the city, we're listening and, looking at the road, hopefully, before we cross it, as long as we don't have our noise canceling, air pods in but, assuming you don't have those like, we are inherently multimodal. When we're walking down the street, we're listening and watching for cars. When we're talking to a friend, we're listening for what they're saying, the tone of their voice, their facial expression. When we're operating, or doing a biopsy, you know, we're using our sensory input, as well with our hands, and so the question is can you create multimodal, generative AI applications, which is able to make predictions based on these multimodal inputs, both visual and audio? The answer is seemingly, yes. As of just May 14th, 2024 Google came out with Gemini 1.5 flash, which is one of the leading multi modal generative AI neural nets, and it's just phenomenal, the size of the data that it can take in.
So, you can drop a 1-hour video into Gemini Flash and ask it to analyze it and as it related to my talk at the CSOHNS, I wanted to make sure that I was flagging to the ENT community what are some of the most recent in advances in AI, and I was able to team up with a medical student at UBC who was interested in ENT, his name's Norbert, and to do this video analysis where he found a video just on YouTube from Mayo Clinic, which was a level two-to-four select neck dissection but, it also had narration from the surgeon. And the surgeon would talk about the importance of identifying the accessory nerve. He would describe the accessory nerve. And we just opened a free Gemini 1.5 account and we took this 24-minute video and just dropped it onto this multi model AI, and then just started asking it questions, having a conversation.
And so we wanted to see how did it do if we just gave it the audio? You know, a description of the surgery or if we gave it the just, the video. Because we wanted to try to figure out how much information it was taking from the narration of the surgery versus the video of the surgery itself. When it came to identifying the accessory nerve, when we gave it the video and the audio, it was able to say the accessory nerve has shown up at minute 12 or whatever it was, but that's because it was listening. The narrator said, okay the accessory nerve is now in view, and when we gave it just the video it couldn't come up with the time that the accessory nerve had come into view, so that demonstrates, it didn't quite have the anatomical or you know visual understanding on it's own to make sense of when the accessory nerve occurred. When we asked it to identify bleeding it could do it, with or without the audio. And then, we even gave it kind of some second-order questions, like can you identify a time where there'd be a higher risk of damage to the thoracic duct? And the narrator never mentioned thoracic duct, but it did mention other anatomical structures that are near the thoracic duct so the AI was able to say, “okay, look at minute, you know, 10 there's these other anatomical structures that are near the thoracic duct, so that's probably at high risk of injury”. So, it was able to kind of use some second-order reasoning. But, again when you took out the audio and just gave it the video, it couldn't make sense of the anatomical structures let alone when the thoracic duct was in view. So, it just gives, you an idea of how easy it is to do research because this multimodal tool really only became available to the public in the last few months. And so the questions we were asking were very novel. We were probably some of the first people in the world to say okay, how does this multimodal generative AI tool do when applied to head and neck surgeries? And so I'm hopeful that there will be people listening to the interview today who are already having those gears grind in their mind and thinking about okay, well how else could we use this in medicine? Now that we have AIs that can take both image and audio together, where's that going to be useful?
Alison:
Yes. And I remember at the conference listening to yourself and Norbert talk about this technology. How exciting it was for me, and that's why I wanted to have you on the show just to share a bit for every student interested in Otolaryngology, or any medical student in general, to learn about all these exciting innovations we have. So thank you so much for taking the time out of your busy day to share all this information with us.
Dr. Edgcumbe:
Well Alison, it's a great pleasure to do so. I certainly was glad to have met you and many of the other students that are in the ENT community, just I guess it was a couple months ago now at CSOHNS. And I am fundamentally an optimist and hopeful about the advances that we are going to be able to bring and what our generation will be able to do in terms of improving healthcare. And I am sure that the strategic deployment of AI is going to be critical to the improvements that we will make.
