Transcript
First, I'd like to thank Brainlab for inviting me to come here and allow me to talk about a slightly different topic from some of the other speakers. I promised to remove all the math equations, which I've succeeded at. I have a few disclosures but none of that are relevant to the current talk. So if I have one slide that I want ever everyone to take away from my talk and to remember it's this one, and it's that deep learning is automated feature engineering.
And so what I mean by that is, let's say you are seeing a patient and you notice a new neurologic deficit, a new pronator drift, or you're sitting at your workstation and you're contouring a tumor choosing a therapeutic dose. These are all examples of you manually futurizing data. There's some raw input. You interacting with the patient or looking at the imaging. And then you're extracting some kind of categorical observations from it, a new neurologic deficit or a therapeutic dose. And you can record that in a spreadsheet and run statistics on it. And this is how we do clinical research, and it's what we're all familiar with.
And deep learning then is going to automate that process for you. And you can imagine there's a lot of advantages to that. One, it's scalable. Those two examples, I just mentioned require highly trained physicians, which there's only so many. And physician time is expensive. And the other one is we don't always know how to featurize data. If I gave anyone here a dataset of neutron spectroscopy to look for large vessel occlusions, how would we even start to analyze that? Deep learning can find features automatically for us.
My second technical slide is this one and it's for supervised learning with computer vision. There's five fundamental tasks. And I'd like to encourage you to try to remember this slide. I'm gonna refer back to it through the rest of my talk. And the tasks range from the simple, whole image classification, given an image, tell what it is of, all the way up to semantic segmentation, given an image, assign a class or a label to every individual pixel within the image or voxel if you're working with three-dimensional imaging.
And so when I was thinking about this talk, I thought that I don't really wanna talk about how can machine learning impact medical outcomes, really ask a more prosaic but I think more interesting question. What problems do we face as physicians and how can we translate them as machine learning problems to then impact outcomes? And I'm gonna work through some examples now referring back to that slide about fundamental tasks and computer vision.
First, the neural ICU, how many times do we go to an ICU and say, we wish we could have more frequent neurologic checks? I'm worried about this patient. So a team at the University of Florida, which is currently being replicated by my lab, decided to do this. Can we have continuous, real-time neurologic assessments specifically in their case to look at ICU delirium? So using actigraphy, hooking up, essentially accelerometers to the patient's wrists, elbows, and heads, as well as a series of cameras trained on the room, on the patients, on the patient's bed, they set out to do this in an automated fashion.
I'm gonna skip the actigraphy data in the interest of time, but here's the data from the cameras. And I think this is so cool that you can see here using just the cameras, they could detect clear features that as physicians we recognized or associated with deliria, patients whose heads were more frequently tilted to one side or the other, not in a normal vertical position, tended to be delirious. Patients with fewer visitors during the day tended to be delirious. Patients with more activity at night tended to be delirious, and patients who were more frequently supine and not upright or walking around again, tended to be delirious. And none of these features are surprising to physicians. We know that patients who have those features tend to be delirious, but what's exciting is that they're doing this continuously and in real-time, continuous real-time neurologic assessment.
And so how did they do this? They started with the cameras. They trained the cameras on these patients and said, "Fundamentally, what we're interested in is how the patients are positioned. How are they interacting with the world around them?"
And this really is a classification and localization task, the second type of computer vision task. And in fact, if you're interested in patient position, we even have a specific name for this in computer vision. It's such a well-characterized problem. It's called pose detection. Deposed was the first algorithm to do this in 2013. And so this is the basic paradigm. I'd like to encourage people to think of a clinical problem, continuous real-time neurologic assessment, a machine learning problem, pose detection, and a computer vision solution deposed.
Another example, brain biopsies. Brain biopsies have a 1% risk of serious neurologic complication. What if we could get everything we wanted to from a brain biopsy, tissue diagnosis, molecular information off of just the imaging? And there's a whole subfield cropping up around this is called radiomics. And my lab does a lot of this and so does a group at NYU, which I have their results here. So using a kind... radiomics type analysis, they found that they could predict IDH1 expression, 94% of gliomas. They could predict MGMT hypermethylation, 83% of the time. And they could predict 1P19Q co-deletion, again, 92% of the time.
But, you know, how did they do this? So break it down into the tasks. First, you need to find your tumor in the MR imaging, which that's a semantic segmentation problem. We wanna assign class labels and say these parts of the MRI image belong to the tumor, another well-delineated computer vision problem, 3D units. Once you've identified the tumor and the image, then we wanna classify it based on the gene expression, a simple feed-forward convolutional neural network. So the paradigm, clinical problem, we wanna get information from brain biopsy off of just imaging, a computer vision problem, semantic segmentation followed by classification and computer vision solutions, a 3D unit, and a feed-forward convolutional neural network.
My third example, this is a recent paper for my lab, which we published in "Nature Medicine" a few weeks ago. And our question was, can we have an AI-run surveillance on all of the head CTs being done in the Mount Sinai health system and try to detect acute neurologic events of interest to neurosurgeons hemorrhage, stroke, hydrocephalus? How do we do this? We put together the largest dataset ever of head CTs, 37,000 head CTs. We had one neural network, a recurrent neural network label, all of the head CTs based on the radiology reports. And we trained a second weekly supervised convolutional neural network to classify them.
We then ran a randomized control double-blinded clinical trial and showed that radiologists working with the AI were able to recognize acute neurologic events sooner. It takes the AI about 600 milliseconds to process an image. That's far faster than any human across every head CT scanner in the system. But again, kind of the paradigm, right?
Head CT surveillance, we can tackle this is a weekly supervised problem, whole image. Weekly supervised convolutional neural network aided by the recurrent neural network for the training process of teaching it what to do. So for the purposes of this talk, since a lot of us are interested in radiosurgery, I kind of went through this process and figured if I wanted to tackle radiosurgery as a whole, what would that look like? Well, here's the workflow, right? Simulation, we do registration, contouring, dose planning, and treatment delivery.
And here's maybe some clinical goals that I'd be interested in here. Speeding the MRI acquisition. Could we acquire an MRI scan in a minute? What about contouring? Superhuman accuracy and contouring normal and critical structures. And wrote the doses. Rather than based on my clinical experience, why not choose doses based on outcomes like we just saw in the past talk? Here's the doses that I'm using to get these outcomes in my patients. These are all solved problems in computer science. Simulation is really image reconstruction, which you can do under-sample imagery reconstruction with convolutional neural networks.
Contouring again is essentially semantic segmentation, an extremely well-tackled problem computer vision. Planning your dose is a regression study on your own data, but we could use deep neural networks again, the DNA regressor. And registration, you know, that's fundamentally is a mathematical problem, and modern AI techniques to registration are incredibly fast for deformable non-rigid registration with Quicksilver and [inaudible [00:09:15].
Though, this is all very cool, there are some disclaimers that come with any kind of AI talk, which I feel obligated to discuss. One, generalization is not guaranteed. We have strong theoretical guarantees, in fact, that we can fit our training data, the data that the network learns on, but no guarantees that it'll work on out of sample data. And here's a good example of this. These are a bunch of images of natural things, planes, cars, a deer. And in the texture, you can see a deep neural networks predictions of what these are, airplane, automobile, cat. It has superhuman performance on this, over 98% accuracy.
Now though, you'll see that an individual pixel in each image has been changed, which your eyes probably write off as just some noise or air, you know, they're not good photos. In the parentheses, that same superhuman deep neural network, those are its predictions now with that one pixel changed, it calls the cat a dog. This dog is a sheep. These aren't artificial intelligences, really. This is just a lot of linear algebra running on modern high-performance computing hardware. You have to be very careful working with deep neural networks.
To make a medical example of this, my group reproduced the Stanford results on detecting pneumonia in chest x-rays with the checks net, which we did at Mount Sinai and ours was 97% accurate at detecting pneumonias, which sounded great. But we noticed that it wasn't...it said that no one had pneumonia in our outpatient clinics, which we also felt sounded odd. And it turns out that the deep neural network was cheating. It wasn't detecting pneumonias. It was recognizing which chest x-rays were getting taken in our ICUs, which is correlated to pneumonia.
And so, you know, you have to be very careful, deep neural networks aren't going to do what you want them to do. They're going to do what you train them to do based on the data you show them, which brings me into my next point.
Data is everything. Deep learning's data-intensive. And as I just showed it's data-sensitive, highly sensitive to data. And then this is where a lot of physicians can play a role in this. Not all of us know how to program these things or know the math behind them, but we're the most proximal individuals to patients and to patient data.
And I'd like to suggest that in the 21st century, being custodians of medical data and responsible stewards of it is actually one of our obligations as physicians. Medical data is ultimately paid for in human lives. I remember I had a conversation with one of my graduate students that said, "Oh, you know, I'd like more data on stroke patients." And we reflected on that for a second and realized that, in order to get that, we needed more patients to have strokes. That's the cost of medical data and why us being custodians of it is critical as physicians.
So in conclusion, you know, make your problems, clinical problems, and machine learning problems. We have solutions in machine learning. Trust no algorithms that haven't been rigorously vetted and gather your data. Thank you.
And so what I mean by that is, let's say you are seeing a patient and you notice a new neurologic deficit, a new pronator drift, or you're sitting at your workstation and you're contouring a tumor choosing a therapeutic dose. These are all examples of you manually futurizing data. There's some raw input. You interacting with the patient or looking at the imaging. And then you're extracting some kind of categorical observations from it, a new neurologic deficit or a therapeutic dose. And you can record that in a spreadsheet and run statistics on it. And this is how we do clinical research, and it's what we're all familiar with.
And deep learning then is going to automate that process for you. And you can imagine there's a lot of advantages to that. One, it's scalable. Those two examples, I just mentioned require highly trained physicians, which there's only so many. And physician time is expensive. And the other one is we don't always know how to featurize data. If I gave anyone here a dataset of neutron spectroscopy to look for large vessel occlusions, how would we even start to analyze that? Deep learning can find features automatically for us.
My second technical slide is this one and it's for supervised learning with computer vision. There's five fundamental tasks. And I'd like to encourage you to try to remember this slide. I'm gonna refer back to it through the rest of my talk. And the tasks range from the simple, whole image classification, given an image, tell what it is of, all the way up to semantic segmentation, given an image, assign a class or a label to every individual pixel within the image or voxel if you're working with three-dimensional imaging.
And so when I was thinking about this talk, I thought that I don't really wanna talk about how can machine learning impact medical outcomes, really ask a more prosaic but I think more interesting question. What problems do we face as physicians and how can we translate them as machine learning problems to then impact outcomes? And I'm gonna work through some examples now referring back to that slide about fundamental tasks and computer vision.
First, the neural ICU, how many times do we go to an ICU and say, we wish we could have more frequent neurologic checks? I'm worried about this patient. So a team at the University of Florida, which is currently being replicated by my lab, decided to do this. Can we have continuous, real-time neurologic assessments specifically in their case to look at ICU delirium? So using actigraphy, hooking up, essentially accelerometers to the patient's wrists, elbows, and heads, as well as a series of cameras trained on the room, on the patients, on the patient's bed, they set out to do this in an automated fashion.
I'm gonna skip the actigraphy data in the interest of time, but here's the data from the cameras. And I think this is so cool that you can see here using just the cameras, they could detect clear features that as physicians we recognized or associated with deliria, patients whose heads were more frequently tilted to one side or the other, not in a normal vertical position, tended to be delirious. Patients with fewer visitors during the day tended to be delirious. Patients with more activity at night tended to be delirious, and patients who were more frequently supine and not upright or walking around again, tended to be delirious. And none of these features are surprising to physicians. We know that patients who have those features tend to be delirious, but what's exciting is that they're doing this continuously and in real-time, continuous real-time neurologic assessment.
And so how did they do this? They started with the cameras. They trained the cameras on these patients and said, "Fundamentally, what we're interested in is how the patients are positioned. How are they interacting with the world around them?"
And this really is a classification and localization task, the second type of computer vision task. And in fact, if you're interested in patient position, we even have a specific name for this in computer vision. It's such a well-characterized problem. It's called pose detection. Deposed was the first algorithm to do this in 2013. And so this is the basic paradigm. I'd like to encourage people to think of a clinical problem, continuous real-time neurologic assessment, a machine learning problem, pose detection, and a computer vision solution deposed.
Another example, brain biopsies. Brain biopsies have a 1% risk of serious neurologic complication. What if we could get everything we wanted to from a brain biopsy, tissue diagnosis, molecular information off of just the imaging? And there's a whole subfield cropping up around this is called radiomics. And my lab does a lot of this and so does a group at NYU, which I have their results here. So using a kind... radiomics type analysis, they found that they could predict IDH1 expression, 94% of gliomas. They could predict MGMT hypermethylation, 83% of the time. And they could predict 1P19Q co-deletion, again, 92% of the time.
But, you know, how did they do this? So break it down into the tasks. First, you need to find your tumor in the MR imaging, which that's a semantic segmentation problem. We wanna assign class labels and say these parts of the MRI image belong to the tumor, another well-delineated computer vision problem, 3D units. Once you've identified the tumor and the image, then we wanna classify it based on the gene expression, a simple feed-forward convolutional neural network. So the paradigm, clinical problem, we wanna get information from brain biopsy off of just imaging, a computer vision problem, semantic segmentation followed by classification and computer vision solutions, a 3D unit, and a feed-forward convolutional neural network.
My third example, this is a recent paper for my lab, which we published in "Nature Medicine" a few weeks ago. And our question was, can we have an AI-run surveillance on all of the head CTs being done in the Mount Sinai health system and try to detect acute neurologic events of interest to neurosurgeons hemorrhage, stroke, hydrocephalus? How do we do this? We put together the largest dataset ever of head CTs, 37,000 head CTs. We had one neural network, a recurrent neural network label, all of the head CTs based on the radiology reports. And we trained a second weekly supervised convolutional neural network to classify them.
We then ran a randomized control double-blinded clinical trial and showed that radiologists working with the AI were able to recognize acute neurologic events sooner. It takes the AI about 600 milliseconds to process an image. That's far faster than any human across every head CT scanner in the system. But again, kind of the paradigm, right?
Head CT surveillance, we can tackle this is a weekly supervised problem, whole image. Weekly supervised convolutional neural network aided by the recurrent neural network for the training process of teaching it what to do. So for the purposes of this talk, since a lot of us are interested in radiosurgery, I kind of went through this process and figured if I wanted to tackle radiosurgery as a whole, what would that look like? Well, here's the workflow, right? Simulation, we do registration, contouring, dose planning, and treatment delivery.
And here's maybe some clinical goals that I'd be interested in here. Speeding the MRI acquisition. Could we acquire an MRI scan in a minute? What about contouring? Superhuman accuracy and contouring normal and critical structures. And wrote the doses. Rather than based on my clinical experience, why not choose doses based on outcomes like we just saw in the past talk? Here's the doses that I'm using to get these outcomes in my patients. These are all solved problems in computer science. Simulation is really image reconstruction, which you can do under-sample imagery reconstruction with convolutional neural networks.
Contouring again is essentially semantic segmentation, an extremely well-tackled problem computer vision. Planning your dose is a regression study on your own data, but we could use deep neural networks again, the DNA regressor. And registration, you know, that's fundamentally is a mathematical problem, and modern AI techniques to registration are incredibly fast for deformable non-rigid registration with Quicksilver and [inaudible [00:09:15].
Though, this is all very cool, there are some disclaimers that come with any kind of AI talk, which I feel obligated to discuss. One, generalization is not guaranteed. We have strong theoretical guarantees, in fact, that we can fit our training data, the data that the network learns on, but no guarantees that it'll work on out of sample data. And here's a good example of this. These are a bunch of images of natural things, planes, cars, a deer. And in the texture, you can see a deep neural networks predictions of what these are, airplane, automobile, cat. It has superhuman performance on this, over 98% accuracy.
Now though, you'll see that an individual pixel in each image has been changed, which your eyes probably write off as just some noise or air, you know, they're not good photos. In the parentheses, that same superhuman deep neural network, those are its predictions now with that one pixel changed, it calls the cat a dog. This dog is a sheep. These aren't artificial intelligences, really. This is just a lot of linear algebra running on modern high-performance computing hardware. You have to be very careful working with deep neural networks.
To make a medical example of this, my group reproduced the Stanford results on detecting pneumonia in chest x-rays with the checks net, which we did at Mount Sinai and ours was 97% accurate at detecting pneumonias, which sounded great. But we noticed that it wasn't...it said that no one had pneumonia in our outpatient clinics, which we also felt sounded odd. And it turns out that the deep neural network was cheating. It wasn't detecting pneumonias. It was recognizing which chest x-rays were getting taken in our ICUs, which is correlated to pneumonia.
And so, you know, you have to be very careful, deep neural networks aren't going to do what you want them to do. They're going to do what you train them to do based on the data you show them, which brings me into my next point.
Data is everything. Deep learning's data-intensive. And as I just showed it's data-sensitive, highly sensitive to data. And then this is where a lot of physicians can play a role in this. Not all of us know how to program these things or know the math behind them, but we're the most proximal individuals to patients and to patient data.
And I'd like to suggest that in the 21st century, being custodians of medical data and responsible stewards of it is actually one of our obligations as physicians. Medical data is ultimately paid for in human lives. I remember I had a conversation with one of my graduate students that said, "Oh, you know, I'd like more data on stroke patients." And we reflected on that for a second and realized that, in order to get that, we needed more patients to have strokes. That's the cost of medical data and why us being custodians of it is critical as physicians.
So in conclusion, you know, make your problems, clinical problems, and machine learning problems. We have solutions in machine learning. Trust no algorithms that haven't been rigorously vetted and gather your data. Thank you.