Kategorie: KI Use Cases

The “interdisciplinary approach” is something that has been lauded for decades for its ability to break down silos and create new integrated approaches to research.

For Munther Dahleh, founding director of the MIT Institute for Data, Systems, and Society (IDSS), showing the community that data science and statistics can transcend individual disciplines and form a new holistic approach to addressing complex societal challenges has been crucial to the institute’s success.

“From the very beginning, it was critical that we recognized the areas of data science, statistics, AI, and, in a way, computing, as transdisciplinary,” says Dahleh, who is the William A. Coolidge Professor in Electrical Engineering and Computer Science. “We made that point over and over — these are areas that embed in your field. It is not ours; this organization is here for everyone.”

On April 14-15, researchers from across and beyond MIT joined together to celebrate the accomplishments and impact IDSS has had on research and education since its inception in 2015. Taking the place of IDSS’s annual statistics and data science conference SDSCon, the celebration also doubled as a way to recognize Dahleh for his work creating and executing the vision of IDSS as he prepares to step down from his director position this summer.

In addition to talks and panels on statistics and computation, smart systems, automation and artificial intelligence, conference participants discussed issues ranging from climate change, health care, and misinformation. Nobel Prize winner and IDSS affiliate Professor Esther Duflo spoke on large scale immunization efforts, former MLK Visiting Professor Craig Watkins joined a panel on equity and justice in AI, and IDSS Associate Director Alberto Abadie discussed synthetic controls for policy evaluation. Other policy questions were explored through lightning talks, including those by students from the Technology and Policy Program (TPP) within IDSS.

A place to call home

The list of IDSS accomplishments over the last eight years is long and growing. From creating a home for 21st century statistics at MIT after other unsuccessful attempts, to creating a new PhD preparing the trilingual student who is an expert in data science and social science in the context of a domain, to playing a key role in determining an effective process for Covid testing in the early days of the pandemic, IDSS has left its mark on MIT. More recently, IDSS launched an initiative using big data to help effect structural and normative change toward racial equity, and will continue to explore societal challenges through the lenses of statistics, social science, and science and engineering.

“I’m very proud of what we’ve done and of all the people who have contributed to this. The leadership team has been phenomenal in their commitment and their creativity,” Dahleh says. “I always say it doesn’t take one person, it takes the village to do what we have done, and I am very proud of that.”

Prior to the institute’s formation, Dahleh and others at MIT were brought together to answer one key question: How would MIT prepare for the future of systems and data?

“Data science is a complex area because in some ways it’s everywhere and it belongs to everyone, similar to statistics and AI,” Dahleh says “The most important part of creating an organization to support it was making it clear that it was an organization for everyone.” The response the team came back with was to build an Institute: a department that could cross all other departments and schools.

While Dahleh and others on the committee were creating this blueprint for the future, the events that would lead early IDSS hires like Caroline Uhler to join the team were also beginning to take shape. Uhler, now an MIT professor of computer science and co-director of the Eric and Wendy Schmidt Center at the Broad Institute, was a panelist at the celebration discussing statistics and human health.

In 2015, Uhler was a faculty member at the Institute of Science and Technology in Austria looking to move back to the U.S. “I was looking for positions in all different types of departments related to statistics, including electrical engineering and computer science, which were areas not related to my degree,” Uhler says. “What really got me to MIT was Munther’s vision for building a modern type of statistics, and the unique opportunity to be part of building what statistics should be moving forward.”

The breadth of the Statistics and Data Science Center has given it a unique and a robust character that makes for an attractive collaborative environment at MIT. “A lot of IDSS’s impact has been in giving people like me a home,” Uhler adds. “By building an institute for statistics that is across all schools instead of housed within a single department, it has created a home for everyone who is interested in the field.”

Filling the gap

For Ali Jadbabaie, former IDSS associate director and another early IDSS hire, being in the right place at the right time landed him in the center of it all. A control theory expert and network scientist by training, Jadbabaie first came to MIT during a sabbatical from his position as a professor at the University of Pennsylvania.

“My time at MIT coincided with the early discussions around forming IDSS and given my experience they asked me to stay and help with its creation,” Jadbabaie says. He is now head of the Department of Civil and Environmental Engineering at MIT, and he spoke at the celebration about a new MIT major in climate system science and engineering.

A critical early accomplishment of IDSS was the creation of a doctoral program in social and engineering systems (SES), which has the goal of educating and fostering the success of a new type of PhD student, says Jadbabaie.

“We realized we had this opportunity to educate a new type of PhD student who was conversant in the math of information sciences and statistics in addition to an understanding of a domain — infrastructures, climate, political polarization — in which problems arise,” he says. “This program would provide training in statistics and data science, the math of information sciences and a branch of social science that is relevant to their domain.”

“SES has been filling a gap,” adds Jadbabaie. “We wanted to bring quantitative reasoning to areas in social sciences, particularly as they interact with complex engineering systems.”

“My first year at MIT really broadened my horizon in terms of what was available and exciting,” says Manxi Wu, a member of the first cohort of students in the SES program after starting out in the Master of Science in Transportation (MST) program. “My advisor introduced me to a number of interesting topics at the intersection of game theory, economics, and engineering systems, and in my second year I realized my interest was really about the societal scale systems, with transportation as my go-to application area when I think about how to make an impact in the real world.”

Wu, now an assistant professor in the School of Operations Research and Information Engineering at Cornell, was a panelist at the Celebration’s session on smart infrastructure systems. She says that the beauty of the SES program lies in its ability to create a common ground between groups of students and researchers who all have different applications interests but share an eagerness to sharpen their technical skills.

“While we may be working on very different application areas, the core methodologies, such as mathematical tools for data science and probability optimization, create a common language,” Wu says. “We are all capable of speaking the technical language, and our diversified interests give us even more to talk about.”

In addition to the PhD program, IDSS has helped bring quality MIT programming to people around the globe with its MicroMasters Program in Statistics and Data Science (SDS), which recently celebrated the certification of over 1,000 learners. The MicroMasters is just one offering in the newly-minted IDSSx, a collection of online learning opportunities for learners at different skill levels and interests.

“The impact of branding what MIT-IDSS does across the globe has been great,” Dahleh says. “In addition, we’ve created smaller online programs for continued education in data science and machine learning, which I think is also critical in educating the community at large.”

Hopes for the future

Through all of its accomplishments, the core mission of IDSS has never changed.

“The belief was always to create an institute focused on how data science can be used to solve pressing societal problems,” Dahleh says. “The organizational structure of IDSS as an MIT Institute has enabled it to promote data and systems as a transdiciplinary area that embeds in every domain to support its mission. This reverse ownership structure will continue to strengthen the presence of IDSS in MIT and will make it an essential unit within the Schwarzman College of Computing.”

As Dahleh prepares to step down from his role, and Professor Martin Wainwright gets ready to fill his (very big) shoes as director, Dahleh’s colleagues say the real key to the success of IDSS all started with his passion and vision.

“Creating a new academic unit within MIT is actually next to impossible,” Jadbabaie says. “It requires structural changes, as well as someone who has a strong understanding of multiple areas, who knows how to get people to work together collectively, and who has a mission.“

“The most important thing is that he was inclusive,” he adds. “He didn’t try to create a gate around it and say these people are in and these people are not. I don’t think this would have ever happened without Munther at the helm.”

Using an artificial intelligence algorithm, researchers at MIT and McMaster University have identified a new antibiotic that can kill a type of bacteria that is responsible for many drug-resistant infections.

If developed for use in patients, the drug could help to combat Acinetobacter baumannii, a species of bacteria that is often found in hospitals and can lead to pneumonia, meningitis, and other serious infections. The microbe is also a leading cause of infections in wounded soldiers in Iraq and Afghanistan.

“Acinetobacter can survive on hospital doorknobs and equipment for long periods of time, and it can take up antibiotic resistance genes from its environment. It’s really common now to find A. baumannii isolates that are resistant to nearly every antibiotic,” says Jonathan Stokes, a former MIT postdoc who is now an assistant professor of biochemistry and biomedical sciences at McMaster University.

The researchers identified the new drug from a library of nearly 7,000 potential drug compounds using a machine-learning model that they trained to evaluate whether a chemical compound will inhibit the growth of A. baumannii.

“This finding further supports the premise that AI can significantly accelerate and expand our search for novel antibiotics,” says James Collins, the Termeer Professor of Medical Engineering and Science in MIT’s Institute for Medical Engineering and Science (IMES) and Department of Biological Engineering. “I’m excited that this work shows that we can use AI to help combat problematic pathogens such as A. baumannii.”

Collins and Stokes are the senior authors of the new study, which appears today in Nature Chemical Biology. The paper’s lead authors are McMaster University graduate students Gary Liu and Denise Catacutan and recent McMaster graduate Khushi Rathod.

Drug discovery

Over the past several decades, many pathogenic bacteria have become increasingly resistant to existing antibiotics, while very few new antibiotics have been developed.

Several years ago, Collins, Stokes, and MIT Professor Regina Barzilay (who is also an author on the new study), set out to combat this growing problem by using machine learning, a type of artificial intelligence that can learn to recognize patterns in vast amounts of data. Collins and Barzilay, who co-direct MIT’s Abdul Latif Jameel Clinic for Machine Learning in Health, hoped this approach could be used to identify new antibiotics whose chemical structures are different from any existing drugs.

In their initial demonstration, the researchers trained a machine-learning algorithm to identify chemical structures that could inhibit growth of E. coli. In a screen of more than 100 million compounds, that algorithm yielded a molecule that the researchers called halicin, after the fictional artificial intelligence system from “2001: A Space Odyssey.” This molecule, they showed, could kill not only E. coli but several other bacterial species that are resistant to treatment.

“After that paper, when we showed that these machine-learning approaches can work well for complex antibiotic discovery tasks, we turned our attention to what I perceive to be public enemy No. 1 for multidrug-resistant bacterial infections, which is Acinetobacter,” Stokes says.

To obtain training data for their computational model, the researchers first exposed A. baumannii grown in a lab dish to about 7,500 different chemical compounds to see which ones could inhibit growth of the microbe. Then they fed the structure of each molecule into the model. They also told the model whether each structure could inhibit bacterial growth or not. This allowed the algorithm to learn chemical features associated with growth inhibition.

Once the model was trained, the researchers used it to analyze a set of 6,680 compounds it had not seen before, which came from the Drug Repurposing Hub at the Broad Institute. This analysis, which took less than two hours, yielded a few hundred top hits. Of these, the researchers chose 240 to test experimentally in the lab, focusing on compounds with structures that were different from those of existing antibiotics or molecules from the training data.

Those tests yielded nine antibiotics, including one that was very potent. This compound, which was originally explored as a potential diabetes drug, turned out to be extremely effective at killing A. baumannii but had no effect on other species of bacteria including Pseudomonas aeruginosa, Staphylococcus aureus, and carbapenem-resistant Enterobacteriaceae.

This “narrow spectrum” killing ability is a desirable feature for antibiotics because it minimizes the risk of bacteria rapidly spreading resistance against the drug. Another advantage is that the drug would likely spare the beneficial bacteria that live in the human gut and help to suppress opportunistic infections such as Clostridium difficile.

“Antibiotics often have to be administered systemically, and the last thing you want to do is cause significant dysbiosis and open up these already sick patients to secondary infections,” Stokes says.

A novel mechanism

In studies in mice, the researchers showed that the drug, which they named abaucin, could treat wound infections caused by A. baumannii. They also showed, in lab tests, that it works against a variety of drug-resistant A. baumannii strains isolated from human patients.

Further experiments revealed that the drug kills cells by interfering with a process known as lipoprotein trafficking, which cells use to transport proteins from the interior of the cell to the cell envelope. Specifically, the drug appears to inhibit LolE, a protein involved in this process.

All Gram-negative bacteria express this enzyme, so the researchers were surprised to find that abaucin is so selective in targeting A. baumannii. They hypothesize that slight differences in how A. baumannii performs this task might account for the drug’s selectivity.

“We haven’t finalized the experimental data acquisition yet, but we think it’s because A. baumannii does lipoprotein trafficking a little bit differently than other Gram-negative species. We believe that’s why we’re getting this narrow spectrum activity,” Stokes says.

Stokes’ lab is now working with other researchers at McMaster to optimize the medicinal properties of the compound, in hopes of developing it for eventual use in patients.

The researchers also plan to use their modeling approach to identify potential antibiotics for other types of drug-resistant infections, including those caused by Staphylococcus aureus and Pseudomonas aeruginosa.

The research was funded by the David Braley Center for Antibiotic Discovery, the Weston Family Foundation, the Audacious Project, the C3.ai Digital Transformation Institute, the Abdul Latif Jameel Clinic for Machine Learning in Health, the DTRA Discovery of Medical Countermeasures Against New and Emerging Threats program, the DARPA Accelerated Molecular Discovery program, the Canadian Institutes of Health Research, Genome Canada, the Faculty of Health Sciences of McMaster University, the Boris Family, a Marshall Scholarship, and the Department of Energy Biological and Environmental Research program.

Despite their enormous size and power, today’s artificial intelligence systems routinely fail to distinguish between hallucination and reality. Autonomous driving systems can fail to perceive pedestrians and emergency vehicles right in front of them, with fatal consequences. Conversational AI systems confidently make up facts and, after training via reinforcement learning, often fail to give accurate estimates of their own uncertainty.

Working together, researchers from MIT and the University of California at Berkeley have developed a new method for building sophisticated AI inference algorithms that simultaneously generate collections of probable explanations for data, and accurately estimate the quality of these explanations.

The new method is based on a mathematical approach called sequential Monte Carlo (SMC). SMC algorithms are an established set of algorithms that have been widely used for uncertainty-calibrated AI, by proposing probable explanations of data and tracking how likely or unlikely the proposed explanations seem whenever given more information. But SMC is too simplistic for complex tasks. The main issue is that one of the central steps in the algorithm — the step of actually coming up with guesses for probable explanations (before the other step of tracking how likely different hypotheses seem relative to one another) — had to be very simple. In complicated application areas, looking at data and coming up with plausible guesses of what’s going on can be a challenging problem in its own right. In self driving, for example, this requires looking at the video data from a self-driving car’s cameras, identifying cars and pedestrians on the road, and guessing probable motion paths of pedestrians currently hidden from view.  Making plausible guesses from raw data can require sophisticated algorithms that regular SMC can’t support.

That’s where the new method, SMC with probabilistic program proposals (SMCP3), comes in. SMCP3 makes it possible to use smarter ways of guessing probable explanations of data, to update those proposed explanations in light of new information, and to estimate the quality of these explanations that were proposed in sophisticated ways. SMCP3 does this by making it possible to use any probabilistic program — any computer program that is also allowed to make random choices — as a strategy for proposing (that is, intelligently guessing) explanations of data. Previous versions of SMC only allowed the use of very simple strategies, so simple that one could calculate the exact probability of any guess. This restriction made it difficult to use guessing procedures with multiple stages.

The researchers‘ SMCP3 paper shows that by using more sophisticated proposal procedures, SMCP3 can improve the accuracy of AI systems for tracking 3D objects and analyzing data, and also improve the accuracy of the algorithms‘ own estimates of how likely the data is. Previous research by MIT and others has shown that these estimates can be used to infer how accurately an inference algorithm is explaining data, relative to an idealized Bayesian reasoner.

George Matheos, co-first author of the paper (and an incoming MIT electrical engineering and computer science [EECS] PhD student), says he’s most excited by SMCP3’s potential to make it practical to use well-understood, uncertainty-calibrated algorithms in complicated problem settings where older versions of SMC did not work.

“Today, we have lots of new algorithms, many based on deep neural networks, which can propose what might be going on in the world, in light of data, in all sorts of problem areas. But often, these algorithms are not really uncertainty-calibrated. They just output one idea of what might be going on in the world, and it’s not clear whether that’s the only plausible explanation or if there are others — or even if that’s a good explanation in the first place! But with SMCP3, I think it will be possible to use many more of these smart but hard-to-trust algorithms to build algorithms that are uncertainty-calibrated. As we use ‘artificial intelligence’ systems to make decisions in more and more areas of life, having systems we can trust, which are aware of their uncertainty, will be crucial for reliability and safety.”

Vikash Mansinghka, senior author of the paper, adds, „The first electronic computers were built to run Monte Carlo methods, and they are some of the most widely used techniques in computing and in artificial intelligence. But since the beginning, Monte Carlo methods have been difficult to design and implement: the math had to be derived by hand, and there were lots of subtle mathematical restrictions that users had to be aware of. SMCP3 simultaneously automates the hard math, and expands the space of designs. We’ve already used it to think of new AI algorithms that we couldn’t have designed before.”

Other authors of the paper include co-first author Alex Lew (an MIT EECS PhD student); MIT EECS PhD students Nishad Gothoskar, Matin Ghavamizadeh, and Tan Zhi-Xuan; and Stuart Russell, professor at UC Berkeley. The work was presented at the AISTATS conference in Valencia, Spain, in April.

Imagine you’re enjoying a picnic by a riverbank on a windy day. A gust of wind accidentally catches your paper napkin and lands on the water’s surface, quickly drifting away from you. You grab a nearby stick and carefully agitate the water to retrieve it, creating a series of small waves. These waves eventually push the napkin back toward the shore, so you grab it. In this scenario, the water acts as a medium for transmitting forces, enabling you to manipulate the position of the napkin without direct contact.

Humans regularly engage with various types of fluids in their daily lives, but doing so has been a formidable and elusive goal for current robotic systems. Hand you a latte? A robot can do that. Make it? That’s going to require a bit more nuance. 

FluidLab, a new simulation tool from researchers at the MIT Computer Science and Artificial Intelligence Laboratory (CSAIL), enhances robot learning for complex fluid manipulation tasks like making latte art, ice cream, and even manipulating air. The virtual environment offers a versatile collection of intricate fluid handling challenges, involving both solids and liquids, and multiple fluids simultaneously. FluidLab supports modeling solid, liquid, and gas, including elastic, plastic, rigid objects, Newtonian and non-Newtonian liquids, and smoke and air. 

At the heart of FluidLab lies FluidEngine, an easy-to-use physics simulator capable of seamlessly calculating and simulating various materials and their interactions, all while harnessing the power of graphics processing units (GPUs) for faster processing. The engine is “differential,” meaning the simulator can incorporate physics knowledge for a more realistic physical world model, leading to more efficient learning and planning for robotic tasks. In contrast, most existing reinforcement learning methods lack that world model that just depends on trial and error. This enhanced capability, say the researchers, lets users experiment with robot learning algorithms and toy with the boundaries of current robotic manipulation abilities.

To set the stage, the researchers tested said robot learning algorithms using FluidLab, discovering and overcoming unique challenges in fluid systems. By developing clever optimization methods, they’ve been able to transfer these learnings from simulations to real-world scenarios effectively. 

“Imagine a future where a household robot effortlessly assists you with daily tasks, like making coffee, preparing breakfast, or cooking dinner. These tasks involve numerous fluid manipulation challenges. Our benchmark is a first step towards enabling robots to master these skills, benefiting households and workplaces alike,” says visiting researcher at MIT CSAIL and research scientist at the MIT-IBM Watson AI Lab Chuang Gan, the senior author on a new paper about the research. “For instance, these robots could reduce wait times and enhance customer experiences in busy coffee shops. FluidEngine is, to our knowledge, the first-of-its-kind physics engine that supports a wide range of materials and couplings while being fully differentiable. With our standardized fluid manipulation tasks, researchers can evaluate robot learning algorithms and push the boundaries of today’s robotic manipulation capabilities.”

Fluid fantasia 

Over the past few decades, scientists in the robotic manipulation domain have mainly focused on manipulating rigid objects, or on very simplistic fluid manipulation tasks like pouring water. Studying these manipulation tasks involving fluids in the real world can also be an unsafe and costly endeavor. 

With fluid manipulation, it’s not always just about fluids, though. In many tasks, such as creating the perfect ice cream swirl, mixing solids into liquids, or paddling through the water to move objects, it’s a dance of interactions between fluids and various other materials. Simulation environments must support “coupling,” or how two different material properties interact. Fluid manipulation tasks usually require pretty fine-grained precision, with delicate interactions and handling of materials, setting them apart from straightforward tasks like pushing a block or opening a bottle. 

FluidLab’s simulator can quickly calculate how different materials interact with each other. 

Helping out the GPUs is “Taichi,” a domain-specific language embedded in Python. The system can compute gradients (rates of change in environment configurations with respect to the robot’s actions) for different material types and their interactions (couplings) with one another. This precise information can be used to fine-tune the robot’s movements for better performance. As a result, the simulator allows for faster and more efficient solutions, setting it apart from its counterparts.

The 10 tasks the team put forth fell into two categories: using fluids to manipulate hard-to-reach objects, and directly manipulating fluids for specific goals. Examples included separating liquids, guiding floating objects, transporting items with water jets, mixing liquids, creating latte art, shaping ice cream, and controlling air circulation. 

“The simulator works similarly to how humans use their mental models to predict the consequences of their actions and make informed decisions when manipulating fluids. This is a significant advantage of our simulator compared to others,” says Carnegie Mellon University PhD student Zhou Xian, another author on the paper. “While other simulators primarily support reinforcement learning, ours supports reinforcement learning and allows for more efficient optimization techniques. Utilizing the gradients provided by the simulator supports highly efficient policy search, making it a more versatile and effective tool.”

Next steps

FluidLab’s future looks bright. The current work attempted to transfer trajectories optimized in simulation to real-world tasks directly in an open-loop manner. For next steps, the team is working to develop a closed-loop policy in simulation that takes as input the state or the visual observations of the environments and performs fluid manipulation tasks in real time, and then transfers the learned policies in real-world scenes.

The platform is publicly publicly available, and researchers hope it will benefit future studies in developing better methods for solving complex fluid manipulation tasks.

“Humans interact with fluids in everyday tasks, including pouring and mixing liquids (coffee, yogurts, soups, batter), washing and cleaning with water, and more,” says University of Maryland computer science professor Ming Lin, who was not involved in the work. “For robots to assist humans and serve in similar capacities for day-to-day tasks, novel techniques for interacting and handling various liquids of different properties (e.g. viscosity and density of materials) would be needed and remains a major computational challenge for real-time autonomous systems. This work introduces the first comprehensive physics engine, FluidLab, to enable modeling of diverse, complex fluids and their coupling with other objects and dynamical systems in the environment. The mathematical formulation of ‘differentiable fluids’ as presented in the paper makes it possible for integrating versatile fluid simulation as a network layer in learning-based algorithms and neural network architectures for intelligent systems to operate in real-world applications.”

Gan and Xian wrote the paper alongside Hsiao-Yu Tung a postdoc in the MIT Department of Brain and Cognitive Sciences; Antonio Torralba, an MIT professor of electrical engineering and computer science and CSAIL principal investigator; Dartmouth College Assistant Professor Bo Zhu, Columbia University PhD student Zhenjia Xu, and CMU Assistant Professor Katerina Fragkiadaki. The team’s research is supported by the MIT-IBM Watson AI Lab, Sony AI, a DARPA Young Investigator Award, an NSF CAREER award, an AFOSR Young Investigator Award, DARPA Machine Common Sense, and the National Science Foundation.

The research was presented at the International Conference on Learning Representations earlier this month.

A robot manipulating objects while, say, working in a kitchen, will benefit from understanding which items are composed of the same materials. With this knowledge, the robot would know to exert a similar amount of force whether it picks up a small pat of butter from a shadowy corner of the counter or an entire stick from inside the brightly lit fridge.

Identifying objects in a scene that are composed of the same material, known as material selection, is an especially challenging problem for machines because a material’s appearance can vary drastically based on the shape of the object or lighting conditions.

Scientists at MIT and Adobe Research have taken a step toward solving this challenge. They developed a technique that can identify all pixels in an image representing a given material, which is shown in a pixel selected by the user.

The method is accurate even when objects have varying shapes and sizes, and the machine-learning model they developed isn’t tricked by shadows or lighting conditions that can make the same material appear different.

Although they trained their model using only “synthetic” data, which are created by a computer that modifies 3D scenes to produce many varying images, the system works effectively on real indoor and outdoor scenes it has never seen before. The approach can also be used for videos; once the user identifies a pixel in the first frame, the model can identify objects made from the same material throughout the rest of the video.

In addition to applications in scene understanding for robotics, this method could be used for image editing or incorporated into computational systems that deduce the parameters of materials in images. It could also be utilized for material-based web recommendation systems. (Perhaps a shopper is searching for clothing made from a particular type of fabric, for example.)

“Knowing what material you are interacting with is often quite important. Although two objects may look similar, they can have different material properties. Our method can facilitate the selection of all the other pixels in an image that are made from the same material,” says Prafull Sharma, an electrical engineering and computer science graduate student and lead author of a paper on this technique.

Sharma’s co-authors include Julien Philip and Michael Gharbi, research scientists at Adobe Research; and senior authors William T. Freeman, the Thomas and Gerd Perkins Professor of Electrical Engineering and Computer Science and a member of the Computer Science and Artificial Intelligence Laboratory (CSAIL); Frédo Durand, a professor of electrical engineering and computer science and a member of CSAIL; and Valentin Deschaintre, a research scientist at Adobe Research. The research will be presented at the SIGGRAPH 2023 conference.

A new approach

Existing methods for material selection struggle to accurately identify all pixels representing the same material. For instance, some methods focus on entire objects, but one object can be composed of multiple materials, like a chair with wooden arms and a leather seat. Other methods may utilize a predetermined set of materials, but these often have broad labels like “wood,” despite the fact that there are thousands of varieties of wood.

Instead, Sharma and his collaborators developed a machine-learning approach that dynamically evaluates all pixels in an image to determine the material similarities between a pixel the user selects and all other regions of the image. If an image contains a table and two chairs, and the chair legs and tabletop are made of the same type of wood, their model could accurately identify those similar regions.

Before the researchers could develop an AI method to learn how to select similar materials, they had to overcome a few hurdles. First, no existing dataset contained materials that were labeled finely enough to train their machine-learning model. The researchers rendered their own synthetic dataset of indoor scenes, which included 50,000 images and more than 16,000 materials randomly applied to each object.

“We wanted a dataset where each individual type of material is marked independently,” Sharma says.

Synthetic dataset in hand, they trained a machine-learning model for the task of identifying similar materials in real images — but it failed. The researchers realized distribution shift was to blame. This occurs when a model is trained on synthetic data, but it fails when tested on real-world data that can be very different from the training set.

To solve this problem, they built their model on top of a pretrained computer vision model, which has seen millions of real images. They utilized the prior knowledge of that model by leveraging the visual features it had already learned.

“In machine learning, when you are using a neural network, usually it is learning the representation and the process of solving the task together. We have disentangled this. The pretrained model gives us the representation, then our neural network just focuses on solving the task,” he says.

Solving for similarity

The researchers’ model transforms the generic, pretrained visual features into material-specific features, and it does this in a way that is robust to object shapes or varied lighting conditions.

The model can then compute a material similarity score for every pixel in the image. When a user clicks a pixel, the model figures out how close in appearance every other pixel is to the query. It produces a map where each pixel is ranked on a scale from 0 to 1 for similarity.

“The user just clicks one pixel and then the model will automatically select all regions that have the same material,” he says.

Since the model is outputting a similarity score for each pixel, the user can fine-tune the results by setting a threshold, such as 90 percent similarity, and receive a map of the image with those regions highlighted. The method also works for cross-image selection — the user can select a pixel in one image and find the same material in a separate image.

During experiments, the researchers found that their model could predict regions of an image that contained the same material more accurately than other methods. When they measured how well the prediction compared to ground truth, meaning the actual areas of the image that are comprised of the same material, their model matched up with about 92 percent accuracy.

In the future, they want to enhance the model so it can better capture fine details of the objects in an image, which would boost the accuracy of their approach.

“Rich materials contribute to the functionality and beauty of the world we live in. But computer vision algorithms typically overlook materials, focusing heavily on objects instead. This paper makes an important contribution in recognizing materials in images and video across a broad range of challenging conditions,” says Kavita Bala, Dean of the Cornell Bowers College of Computing and Information Science and Professor of Computer Science, who was not involved with this work. “This technology can be very useful to end consumers and designers alike. For example, a home owner can envision how expensive choices like reupholstering a couch, or changing the carpeting in a room, might appear, and can be more confident in their design choices based on these visualizations.”

The advent of generative artificial intelligence models like ChatGPT has prompted renewed calls for AI in health care, and its support base only appears to be broadening.

The second annual MIT-MGB AI Cures Conference, hosted on April 24 by the Abdul Latif Jameel Clinic for Machine Learning in Health (Jameel Clinic), saw its attendance nearly double this year, with over 500 attendees from an array of backgrounds in computer science, medicine, pharmaceuticals, and policy. 

In contrast to the overcast Boston weather that morning, many of the speakers took an optimistic view of AI in health and reiterated two key ideas throughout the day: that AI has the potential to create a more equitable health-care system, and AI won’t be replacing clinicians anytime soon — but clinicians who know how to use AI will eventually replace clinicians who don’t incorporate AI into their daily practice. 

„Collaborations with our partners in government, especially collaborations at the intersection of policy and innovation, are critical to our work,” MIT Provost Cynthia Barnhart stated in her opening remarks to the audience. “All of the pioneering activity you’ll hear about today leaves me very hopeful for the future of human health.” 

Massachusetts General Brigham’s (MGB) president and CEO Anne Klibanski’s remarks reflected a similar optimism: “We have visionaries here in AI, we have visionaries here in health care. If this group can’t come together in a meaningful way to impact health care, we have to ask ourselves why we’re here … this is a time when we have to rethink health care.” Klibanski called attention to the work of Jameel Clinic AI faculty lead, AI Cures co-chair, and MIT Professor Regina Barzilay and MGB Center for Innovation in Early Cancer Detection Director Lecia Sequist, whose research in lung cancer risk assessment is an example of how the continued collaboration between MIT and MGB could yield fruitful results for the future of AI in medicine. 

“Is AI going to be the thing that cures everything with our ailing health care system?” asked newly inaugurated Massachusetts Secretary of Health and Human Services Kate Walsh. “I don’t think so, but I think it’s a great place to start.” Walsh highlighted the pandemic as a wake-up call for the health care system and focused on AI’s potential to establish more equitable care, particularly for those with disabilities, as well as augment an already burdened workforce. “We absolutely have to do better … AI can look across populations and develop insights into where the health care system is failing us and redistribute the health care system so it can do more.” 

Barzilay called out the marked absence of AI in health care today with a reference to the No Surprises Act implemented last year, which requires insurance companies to be transparent about billing codes. “The FDA has approved over 500 AI tools in the last few years and from the 500 models, only 10 have associated billing codes that are actually used,” she said. “What this shows is that AI’s outcome on patients is really limited, and my hope is this conference brings together people who develop AI, clinicians who are the ones bringing innovation to patients, regulators, and people from biotech who are translation these innovations into products. With this forum we have a chance to change that.” 

Despite the enthusiasm, speakers did not sugarcoat the potential risks, nor did they downplay importance of safety in the development and implementation of clinical AI tools.

“You’ve got those who think that AI is going to solve all the world’s problems in the health-care space, replace the world’s physicians, and revolutionize health care. And then you have the other side of the spectrum that says how bad AI is for our economy and how it’s going to take over the world, developing an intelligence of its own,” Jameel Clinic principal investigator, AI Cures speaker, and MIT Professor Collin Stultz said. “None of these concepts are new, but like most things in life, the truth is somewhere in the middle.”  

„There are always potential unintended consequences,” CEO of Cambridge Health Alliance and the Cambridge Commissioner of Public Health Assaad Sayah pointed out during the conference’s regulatory panel. “At the end of the day, it’s hard to predict what are the potential consequences and have the appropriate safeguards … many things are really inappropriately inequitable for certain sub-populations … there’s so much data that’s been hard to contain. I would implore all of you to keep this in mind.” 

Today, the Abdul Latif Jameel Water and Food Systems Lab (J-WAFS) announced its ninth round of seed grants to support innovative research projects at MIT. The grants are designed to fund research efforts that tackle challenges related to water and food for human use, with the ultimate goal of creating meaningful impact as the world population continues to grow and the planet undergoes significant climate and environmental changes.

Ten new projects led by 15 researchers from seven different departments will be supported this year. The projects address a range of challenges by employing advanced materials, technology innovations, and new approaches to resource management. The new projects aim to remove harmful chemicals from water sources, develop monitoring and other systems to help manage various aquaculture industries, optimize water purification materials, and more.

“The seed grant program is J-WAFS’ flagship grant initiative,” says J-WAFS executive director Renee J. Robins. “The funding is intended to spur groundbreaking MIT research addressing complex issues that are challenging our water and food systems. The 10 projects selected this year show great promise, and we look forward to the progress and accomplishments these talented researchers will make,” she adds.

The 2023 J-WAFS seed grant researchers and their projects are:

Sara Beery, an assistant professor in the Department of Electrical Engineering and Computer Science (EECS), is building the first completely automated system to estimate the size of salmon populations in the Pacific Northwest (PNW).

Salmon are a keystone species in the PNW, feeding human populations for the last 7,500 years at least. However, overfishing, habitat loss, and climate change threaten extinction of salmon populations across the region. Accurate salmon counts during their seasonal migration to their natal river to spawn are essential for fisheries’ regulation and management but are limited by human capacity. Fish population monitoring is a widespread challenge in the United States and worldwide. Beery and her team are working to build a system that will provide a detailed picture of the state of salmon populations in unprecedented, spatial, and temporal resolution by combining sonar sensors and computer vision and machine learning (CVML) techniques. The sonar will capture individual fish as they swim upstream and CVML will train accurate algorithms to interpret the sonar video for detecting, tracking, and counting fish automatically while adapting to changing river conditions and fish densities.

Another aquaculture project is being led by Michael Triantafyllou, the Henry L. and Grace Doherty Professor in Ocean Science and Engineering in the Department of Mechanical Engineering, and Robert Vincent, the assistant director at MIT’s Sea Grant Program. They are working with Otto Cordero, an associate professor in the Department of Civil and Environmental Engineering, to control harmful bacteria blooms in aquaculture algae feed production.

Aquaculture in the United States represents a $1.5 billion industry annually and helps support 1.7 million jobs, yet many American hatcheries are not able to keep up with demand. One barrier to aquaculture production is the high degree of variability in survival rates, most likely caused by a poorly controlled microbiome that leads to bacterial infections and sub-optimal feed efficiency. Triantafyllou, Vincent, and Cordero plan to monitor the microbiome composition of a shellfish hatchery in order to identify possible causing agents of mortality, as well as beneficial microbes. They hope to pair microbe data with detail phenotypic information about the animal population to generate rapid diagnostic tests and explore the potential for microbiome therapies to protect larvae and prevent future outbreaks. The researchers plan to transfer their findings and technology to the local and regional aquaculture community to ensure healthy aquaculture production that will support the expansion of the U.S. aquaculture industry.

David Des Marais is the Cecil and Ida Green Career Development Professor in the Department of Civil and Environmental Engineering. His 2023 J-WAFS project seeks to understand plant growth responses to elevated carbon dioxide (CO2) in the atmosphere, in the hopes of identifying breeding strategies that maximize crop yield under future CO2 scenarios.

Today’s crop plants experience higher atmospheric CO2 than 20 or 30 years ago. Crops such as wheat, oat, barley, and rice typically increase their growth rate and biomass when grown at experimentally elevated atmospheric CO2. This is known as the so-called “CO2 fertilization effect.” However, not all plant species respond to rising atmospheric CO2 with increased growth, and for the ones that do, increased growth doesn’t necessarily correspond to increased crop yield. Using specially built plant growth chambers that can control the concentration of CO2, Des Marais will explore how CO2 availability impacts the development of tillers (branches) in the grass species Brachypodium. He will study how gene expression controls tiller development, and whether this is affected by the growing environment. The tillering response refers to how many branches a plant produces, which sets a limit on how much grain it can yield. Therefore, optimizing the tillering response to elevated CO2 could greatly increase yield. Des Marais will also look at the complete genome sequence of Brachypodium, wheat, oat, and barley to help identify genes relevant for branch growth.

Darcy McRose, an assistant professor in the Department of Civil and Environmental Engineering, is researching whether a combination of plant metabolites and soil bacteria can be used to make mineral-associated phosphorus more bioavailable.

The nutrient phosphorus is essential for agricultural plant growth, but when added as a fertilizer, phosphorus sticks to the surface of soil minerals, decreasing bioavailability, limiting plant growth, and accumulating residual phosphorus. Heavily fertilized agricultural soils often harbor large reservoirs of this type of mineral-associated “legacy” phosphorus. Redox transformations are one chemical process that can liberate mineral-associated phosphorus. However, this needs to be carefully controlled, as overly mobile phosphorus can lead to runoff and pollution of natural waters. Ideally, phosphorus would be made bioavailable when plants need it and immobile when they don’t. Many plants make small metabolites called coumarins that might be able to solubilize mineral-adsorbed phosphorus and be activated and inactivated under different conditions. McRose will use laboratory experiments to determine whether a combination of plant metabolites and soil bacteria can be used as a highly efficient and tunable system for phosphorus solubilization. She also aims to develop an imaging platform to investigate exchanges of phosphorus between plants and soil microbes.

Many of the 2023 seed grants will support innovative technologies to monitor, quantify, and remediate various kinds of pollutants found in water. Two of the new projects address the problem of per- and polyfluoroalkyl substances (PFAS), human-made chemicals that have recently emerged as a global health threat. Known as “forever chemicals,” PFAS are used in many manufacturing processes. These chemicals are known to cause significant health issues including cancer, and they have become pervasive in soil, dust, air, groundwater, and drinking water. Unfortunately, the physical and chemical properties of PFAS render them difficult to detect and remove.

Aristide Gumyusenge, the Merton C. Assistant Professor of Materials Science and Engineering, is using metal-organic frameworks for low-cost sensing and capture of PFAS. Most metal-organic frameworks (MOFs) are synthesized as particles, which complicates their high accuracy sensing performance due to defects such as intergranular boundaries. Thin, film-based electronic devices could enable the use of MOFs for many applications, especially chemical sensing. Gumyusenge’s project aims to design test kits based on two-dimensional conductive MOF films for detecting PFAS in drinking water. In early demonstrations, Gumyusenge and his team showed that these MOF films can sense PFAS at low concentrations. They will continue to iterate using a computation-guided approach to tune sensitivity and selectivity of the kits with the goal of deploying them in real-world scenarios.

Carlos Portela, the Brit (1961) and Alex (1949) d’Arbeloff Career Development Professor in the Department of Mechanical Engineering, and Ariel Furst, the Cook Career Development Professor in the Department of Chemical Engineering, are building novel architected materials to act as filters for the removal of PFAS from water. Portela and Furst will design and fabricate nanoscale materials that use activated carbon and porous polymers to create a physical adsorption system. They will engineer the materials to have tunable porosities and morphologies that can maximize interactions between contaminated water and functionalized surfaces, while providing a mechanically robust system.

Rohit Karnik is a Tata Professor and interim co-department head of the Department of Mechanical Engineering. He is working on another technology, his based on microbead sensors, to rapidly measure and monitor trace contaminants in water.

Water pollution from both biological and chemical contaminants contributes to an estimated 1.36 million deaths annually. Chemical contaminants include pesticides and herbicides, heavy metals like lead, and compounds used in manufacturing. These emerging contaminants can be found throughout the environment, including in water supplies. The Environmental Protection Agency (EPA) in the United States sets recommended water quality standards, but states are responsible for developing their own monitoring criteria and systems, which must be approved by the EPA every three years. However, the availability of data on regulated chemicals and on candidate pollutants is limited by current testing methods that are either insensitive or expensive and laboratory-based, requiring trained scientists and technicians. Karnik’s project proposes a simple, self-contained, portable system for monitoring trace and emerging pollutants in water, making it suitable for field studies. The concept is based on multiplexed microbead-based sensors that use thermal or gravitational actuation to generate a signal. His proposed sandwich assay, a testing format that is appealing for environmental sensing, will enable both single-use and continuous monitoring. The hope is that the bead-based assays will increase the ease and reach of detecting and quantifying trace contaminants in water for both personal and industrial scale applications.

Alexander Radosevich, a professor in the Department of Chemistry, and Timothy Swager, the John D. MacArthur Professor of Chemistry, are teaming up to create rapid, cost-effective, and reliable techniques for on-site arsenic detection in water.

Arsenic contamination of groundwater is a problem that affects as many as 500 million people worldwide. Arsenic poisoning can lead to a range of severe health problems from cancer to cardiovascular and neurological impacts. Both the EPA and the World Health Organization have established that 10 parts per billion is a practical threshold for arsenic in drinking water, but measuring arsenic in water at such low levels is challenging, especially in resource-limited environments where access to sensitive laboratory equipment may not be readily accessible. Radosevich and Swager plan to develop reaction-based chemical sensors that bind and extract electrons from aqueous arsenic. In this way, they will exploit the inherent reactivity of aqueous arsenic to selectively detect and quantify it. This work will establish the chemical basis for a new method of detecting trace arsenic in drinking water.

Rajeev Ram is a professor in the Department of Electrical Engineering and Computer Science. His J-WAFS research will advance a robust technology for monitoring nitrogen-containing pollutants, which threaten over 15,000 bodies of water in the United States alone.

Nitrogen in the form of nitrate, nitrite, ammonia, and urea can run off from agricultural fertilizer and lead to harmful algal blooms that jeopardize human health. Unfortunately, monitoring these contaminants in the environment is challenging, as sensors are difficult to maintain and expensive to deploy. Ram and his students will work to establish limits of detection for nitrate, nitrite, ammonia, and urea in environmental, industrial, and agricultural samples using swept-source Raman spectroscopy. Swept-source Raman spectroscopy is a method of detecting the presence of a chemical by using a tunable, single mode laser that illuminates a sample. This method does not require costly, high-power lasers or a spectrometer. Ram will then develop and demonstrate a portable system that is capable of achieving chemical specificity in complex, natural environments. Data generated by such a system should help regulate polluters and guide remediation.

Kripa Varanasi, a professor in the Department of Mechanical Engineering, and Angela Belcher, the James Mason Crafts Professor and head of the Department of Biological Engineering, will join forces to develop an affordable water disinfection technology that selectively identifies, adsorbs, and kills “superbugs” in domestic and industrial wastewater.

Recent research predicts that antibiotic-resistance bacteria (superbugs) will result in $100 trillion in health care expenses and 10 million deaths annually by 2050. The prevalence of superbugs in our water systems has increased due to corroded pipes, contamination, and climate change. Current drinking water disinfection technologies are designed to kill all types of bacteria before human consumption. However, for certain domestic and industrial applications there is a need to protect the good bacteria required for ecological processes that contribute to soil and plant health. Varanasi and Belcher will combine material, biological, process, and system engineering principles to design a sponge-based water disinfection technology that can identify and destroy harmful bacteria while leaving the good bacteria unharmed. By modifying the sponge surface with specialized nanomaterials, their approach will be able to kill superbugs faster and more efficiently. The sponge filters can be deployed under very low pressure, making them an affordable technology, especially in resource-constrained communities.

In addition to the 10 seed grant projects, J-WAFS will also fund a research initiative led by Greg Sixt. Sixt is the research manager for climate and food systems at J-WAFS, and the director of the J-WAFS-led Food and Climate Systems Transformation (FACT) Alliance. His project focuses on the Lake Victoria Basin (LVB) of East Africa. The second-largest freshwater lake in the world, Lake Victoria straddles three countries (Uganda, Tanzania, and Kenya) and has a catchment area that encompasses two more (Rwanda and Burundi). Sixt will collaborate with Michael Hauser of the University of Natural Resources and Life Sciences, Vienna, and Paul Kariuki, of the Lake Victoria Basin Commission.

The group will study how to adapt food systems to climate change in the Lake Victoria Basin. The basin is facing a range of climate threats that could significantly impact livelihoods and food systems in the expansive region. For example, extreme weather events like droughts and floods are negatively affecting agricultural production and freshwater resources. Across the LVB, current approaches to land and water management are unsustainable and threaten future food and water security. The Lake Victoria Basin Commission (LVBC), a specialized institution of the East African Community, wants to play a more vital role in coordinating transboundary land and water management to support transitions toward more resilient, sustainable, and equitable food systems. The primary goal of this research will be to support the LVBC’s transboundary land and water management efforts, specifically as they relate to sustainability and climate change adaptation in food systems. The research team will work with key stakeholders in Kenya, Uganda, and Tanzania to identify specific capacity needs to facilitate land and water management transitions. The two-year project will produce actionable recommendations to the LVBC.

To study ocean currents, scientists release GPS-tagged buoys in the ocean and record their velocities to reconstruct the currents that transport them. These buoy data are also used to identify “divergences,” which are areas where water rises up from below the surface or sinks beneath it.

By accurately predicting currents and pinpointing divergences, scientists can more precisely forecast the weather, approximate how oil will spread after a spill, or measure energy transfer in the ocean. A new model that incorporates machine learning makes more accurate predictions than conventional models do, a new study reports.

A multidisciplinary research team including computer scientists at MIT and oceanographers has found that a standard statistical model typically used on buoy data can struggle to accurately reconstruct currents or identify divergences because it makes unrealistic assumptions about the behavior of water.

The researchers developed a new model that incorporates knowledge from fluid dynamics to better reflect the physics at work in ocean currents. They show that their method, which only requires a small amount of additional computational expense, is more accurate at predicting currents and identifying divergences than the traditional model.

This new model could help oceanographers make more accurate estimates from buoy data, which would enable them to more effectively monitor the transportation of biomass (such as Sargassum seaweed), carbon, plastics, oil, and nutrients in the ocean. This information is also important for understanding and tracking climate change.

“Our method captures the physical assumptions more appropriately and more accurately. In this case, we know a lot of the physics already. We are giving the model a little bit of that information so it can focus on learning the things that are important to us, like what are the currents away from the buoys, or what is this divergence and where is it happening?” says senior author Tamara Broderick, an associate professor in MIT’s Department of Electrical Engineering and Computer Science (EECS) and a member of the Laboratory for Information and Decision Systems and the Institute for Data, Systems, and Society.

Broderick’s co-authors include lead author Renato Berlinghieri, an electrical engineering and computer science graduate student; Brian L. Trippe, a postdoc at Columbia University; David R. Burt and Ryan Giordano, MIT postdocs; Kaushik Srinivasan, an assistant researcher in atmospheric and ocean sciences at the University of California at Los Angeles; Tamay Özgökmen, professor in the Department of Ocean Sciences at the University of Miami; and Junfei Xia, a graduate student at the University of Miami. The research will be presented at the International Conference on Machine Learning.

Diving into the data

Oceanographers use data on buoy velocity to predict ocean currents and identify “divergences” where water rises to the surface or sinks deeper.

To estimate currents and find divergences, oceanographers have used a machine-learning technique known as a Gaussian process, which can make predictions even when data are sparse. To work well in this case, the Gaussian process must make assumptions about the data to generate a prediction.

A standard way of applying a Gaussian process to oceans data assumes the latitude and longitude components of the current are unrelated. But this assumption isn’t physically accurate. For instance, this existing model implies that a current’s divergence and its vorticity (a whirling motion of fluid) operate on the same magnitude and length scales. Ocean scientists know this is not true, Broderick says. The previous model also assumes the frame of reference matters, which means fluid would behave differently in the latitude versus the longitude direction.

“We were thinking we could address these problems with a model that incorporates the physics,” she says.

They built a new model that uses what is known as a Helmholtz decomposition to accurately represent the principles of fluid dynamics. This method models an ocean current by breaking it down into a vorticity component (which captures the whirling motion) and a divergence component (which captures water rising or sinking).

In this way, they give the model some basic physics knowledge that it uses to make more accurate predictions.

This new model utilizes the same data as the old model. And while their method can be more computationally intensive, the researchers show that the additional cost is relatively small.

Buoyant performance

They evaluated the new model using synthetic and real ocean buoy data. Because the synthetic data were fabricated by the researchers, they could compare the model’s predictions to ground-truth currents and divergences. But simulation involves assumptions that may not reflect real life, so the researchers also tested their model using data captured by real buoys released in the Gulf of Mexico.

In each case, their method demonstrated superior performance for both tasks, predicting currents and identifying divergences, when compared to the standard Gaussian process and another machine-learning approach that used a neural network. For example, in one simulation that included a vortex adjacent to an ocean current, the new method correctly predicted no divergence while the previous Gaussian process method and the neural network method both predicted a divergence with very high confidence.

The technique is also good at identifying vortices from a small set of buoys, Broderick adds.

Now that they have demonstrated the effectiveness of using a Helmholtz decomposition, the researchers want to incorporate a time element into their model, since currents can vary over time as well as space. In addition, they want to better capture how noise impacts the data, such as winds that sometimes affect buoy velocity. Separating that noise from the data could make their approach more accurate.

“Our hope is to take this noisily observed field of velocities from the buoys, and then say what is the actual divergence and actual vorticity, and predict away from those buoys, and we think that our new technique will be helpful for this,” she says.

“The authors cleverly integrate known behaviors from fluid dynamics to model ocean currents in a flexible model,” says Massimiliano Russo, an associate biostatistician at Brigham and Women’s Hospital and instructor at Harvard Medical School, who was not involved with this work. “The resulting approach retains the flexibility to model the nonlinearity in the currents but can also characterize phenomena such as vortices and connected currents that would only be noticed if the fluid dynamic structure is integrated into the model. This is an excellent example of where a flexible model can be substantially improved with a well thought and scientifically sound specification.”

This research is supported, in part, by the Office of Naval Research, a National Science Foundation (NSF) CAREER Award, and the Rosenstiel School of Marine, Atmospheric, and Earth Science at the University of Miami.

Medical researchers are awash in a tsunami of clinical data. But we need major changes in how we gather, share, and apply this data to bring its benefits to all, says Leo Anthony Celi, principal research scientist at the MIT Laboratory for Computational Physiology (LCP). 

One key change is to make clinical data of all kinds openly available, with the proper privacy safeguards, says Celi, a practicing intensive care unit (ICU) physician at the Beth Israel Deaconess Medical Center (BIDMC) in Boston. Another key is to fully exploit these open data with multidisciplinary collaborations among clinicians, academic investigators, and industry. A third key is to focus on the varying needs of populations across every country, and to empower the experts there to drive advances in treatment, says Celi, who is also an associate professor at Harvard Medical School. 

In all of this work, researchers must actively seek to overcome the perennial problem of bias in understanding and applying medical knowledge. This deeply damaging problem is only heightened with the massive onslaught of machine learning and other artificial intelligence technologies. “Computers will pick up all our unconscious, implicit biases when we make decisions,” Celi warns.

Sharing medical data 

Founded by the LCP, the MIT Critical Data consortium builds communities across disciplines to leverage the data that are routinely collected in the process of ICU care to understand health and disease better. “We connect people and align incentives,” Celi says. “In order to advance, hospitals need to work with universities, who need to work with industry partners, who need access to clinicians and data.” 

The consortium’s flagship project is the MIMIC (medical information marked for intensive care) ICU database built at BIDMC. With about 35,000 users around the world, the MIMIC cohort is the most widely analyzed in critical care medicine. 

International collaborations such as MIMIC highlight one of the biggest obstacles in health care: most clinical research is performed in rich countries, typically with most clinical trial participants being white males. “The findings of these trials are translated into treatment recommendations for every patient around the world,” says Celi. “We think that this is a major contributor to the sub-optimal outcomes that we see in the treatment of all sorts of diseases in Africa, in Asia, in Latin America.” 

To fix this problem, “groups who are disproportionately burdened by disease should be setting the research agenda,” Celi says. 

That’s the rule in the “datathons” (health hackathons) that MIT Critical Data has organized in more than two dozen countries, which apply the latest data science techniques to real-world health data. At the datathons, MIT students and faculty both learn from local experts and share their own skill sets. Many of these several-day events are sponsored by the MIT Industrial Liaison Program, the MIT International Science and Technology Initiatives program, or the MIT Sloan Latin America Office. 

Datathons are typically held in that country’s national language or dialect, rather than English, with representation from academia, industry, government, and other stakeholders. Doctors, nurses, pharmacists, and social workers join up with computer science, engineering, and humanities students to brainstorm and analyze potential solutions. “They need each other’s expertise to fully leverage and discover and validate the knowledge that is encrypted in the data, and that will be translated into the way they deliver care,” says Celi. 

“Everywhere we go, there is incredible talent that is completely capable of designing solutions to their health-care problems,” he emphasizes. The datathons aim to further empower the professionals and students in the host countries to drive medical research, innovation, and entrepreneurship.

Fighting built-in bias 

Applying machine learning and other advanced data science techniques to medical data reveals that “bias exists in the data in unimaginable ways” in every type of health product, Celi says. Often this bias is rooted in the clinical trials required to approve medical devices and therapies. 

One dramatic example comes from pulse oximeters, which provide readouts on oxygen levels in a patient’s blood. It turns out that these devices overestimate oxygen levels for people of color. “We have been under-treating individuals of color because the nurses and the doctors have been falsely assured that their patients have adequate oxygenation,” he says. “We think that we have harmed, if not killed, a lot of individuals in the past, especially during Covid, as a result of a technology that was not designed with inclusive test subjects.” 

Such dangers only increase as the universe of medical data expands. “The data that we have available now for research is maybe two or three levels of magnitude more than what we had even 10 years ago,” Celi says. MIMIC, for example, now includes terabytes of X-ray, echocardiogram, and electrocardiogram data, all linked with related health records. Such enormous sets of data allow investigators to detect health patterns that were previously invisible. 

“But there is a caveat,” Celi says. “It is trivial for computers to learn sensitive attributes that are not very obvious to human experts.” In a study released last year, for instance, he and his colleagues showed that algorithms can tell if a chest X-ray image belongs to a white patient or person of color, even without looking at any other clinical data. 

“More concerningly, groups including ours have demonstrated that computers can learn easily if you’re rich or poor, just from your imaging alone,” Celi says. “We were able to train a computer to predict if you are on Medicaid, or if you have private insurance, if you feed them with chest X-rays without any abnormality. So again, computers are catching features that are not visible to the human eye.” And these features may lead algorithms to advise against therapies for people who are Black or poor, he says. 

Opening up industry opportunities 

Every stakeholder stands to benefit when pharmaceutical firms and other health-care corporations better understand societal needs and can target their treatments appropriately, Celi says. 

“We need to bring to the table the vendors of electronic health records and the medical device manufacturers, as well as the pharmaceutical companies,” he explains. “They need to be more aware of the disparities in the way that they perform their research. They need to have more investigators representing underrepresented groups of people, to provide that lens to come up with better designs of health products.” 

Corporations could benefit by sharing results from their clinical trials, and could immediately see these potential benefits by participating in datathons, Celi says. “They could really witness the magic that happens when that data is curated and analyzed by students and clinicians with different backgrounds from different countries. So we’re calling out our partners in the pharmaceutical industry to organize these events with us!” 

Words, data, and algorithms combine,
An article about LLMs, so divine. 
A glimpse into a linguistic world, 
Where language machines are unfurled.

It was a natural inclination to task a large language model (LLM) like CHATGPT with creating a poem that delves into the topic of large language models, and subsequently utilize said poem as an introductory piece for this article.

So how exactly did said poem get all stitched together in a neat package, with rhyming words and little morsels of clever phrases? 

We went straight to the source: MIT assistant professor and CSAIL principal investigator Jacob Andreas, whose research focuses on advancing the field of natural language processing, in both developing cutting-edge machine learning models and exploring the potential of language as a means of enhancing other forms of artificial intelligence. This includes pioneering work in areas such as using natural language to teach robots, and leveraging language to enable computer vision systems to articulate the rationale behind their decision-making processes. We probed Andreas regarding the mechanics, implications, and future prospects of the technology at hand.

Q: Language is a rich ecosystem ripe with subtle nuances that humans use to communicate with one another — sarcasm, irony, and other forms of figurative language. There’s numerous ways to convey meaning beyond the literal. Is it possible for large language models to comprehend the intricacies of context? What does it mean for a model to achieve „in-context learning“? Moreover, how do multilingual transformers process variations and dialects of different languages beyond English? 

A: When we think about linguistic contexts, these models are capable of reasoning about much, much longer documents and chunks of text more broadly than really anything that we’ve known how to build before. But that’s only one kind of context. With humans, language production and comprehension takes place in a grounded context. For example, I know that I’m sitting at this table. There are objects that I can refer to, and the language models we have right now typically can’t see any of that when interacting with a human user. 

There’s a broader social context that informs a lot of our language use which these models are, at least not immediately, sensitive to or aware of. It’s not clear how to give them information about the social context in which their language generation and language modeling takes place. Another important thing is temporal context. We’re shooting this video at a particular moment in time when particular facts are true. The models that we have right now were trained on, again, a snapshot of the internet that stopped at a particular time — for most models that we have now, probably a couple of years ago — and they don’t know about anything that’s happened since then. They don’t even know at what moment in time they’re doing text generation. Figuring out how to provide all of those different kinds of contexts is also an interesting question.

Maybe one of the most surprising components here is this phenomenon called in-context learning. If I take a small ML [machine learning] dataset and feed it to the model, like a movie review and the star rating assigned to the movie by the critic, you give just a couple of examples of these things, language models generate the ability both to generate plausible sounding movie reviews but also to predict the star ratings. More generally, if I have a machine learning problem, I have my inputs and my outputs. As you give an input to the model, you give it one more input and ask it to predict the output, the models can often do this really well.

This is a super interesting, fundamentally different way of doing machine learning, where I have this one big general-purpose model into which I can insert lots of little machine learning datasets, and yet without having to train a new model at all, classifier or a generator or whatever specialized to my particular task. This is actually something we’ve been thinking a lot about in my group, and in some collaborations with colleagues at Google — trying to understand exactly how this in-context learning phenomenon actually comes about.

Q: We like to believe humans are (at least somewhat) in pursuit of what is objectively and morally known to be true. Large language models, perhaps with under-defined or yet-to-be-understood „moral compasses,“ aren’t beholden to the truth. Why do large language models tend to hallucinate facts, or confidently assert inaccuracies? Does that limit the usefulness for applications where factual accuracy is critical? Is there a leading theory on how we will solve this? 

A: It’s well-documented that these models hallucinate facts, that they’re not always reliable. Recently, I asked ChatGPT to describe some of our group’s research. It named five papers, four of which are not papers that actually exist, and one of which is a real paper that was written by a colleague of mine who lives in the United Kingdom, whom I’ve never co-authored with. Factuality is still a big problem. Even beyond that, things involving reasoning in a really general sense, things involving complicated computations, complicated inferences, still seem to be really difficult for these models. There might be even fundamental limitations of this transformer architecture, and I believe a lot more modeling work is needed to make things better.

Why it happens is still partly an open question, but possibly, just architecturally, there are reasons that it’s hard for these models to build coherent models of the world. They can do that a little bit. You can query them with factual questions, trivia questions, and they get them right most of the time, maybe even more often than your average human user off the street. But unlike your average human user, it’s really unclear whether there’s anything that lives inside this language model that corresponds to a belief about the state of the world. I think this is both for architectural reasons, that transformers don’t, obviously, have anywhere to put that belief, and training data, that these models are trained on the internet, which was authored by a bunch of different people at different moments who believe different things about the state of the world. Therefore, it’s difficult to expect models to represent those things coherently.

All that being said, I don’t think this is a fundamental limitation of neural language models or even more general language models in general, but something that’s true about today’s language models. We’re already seeing that models are approaching being able to build representations of facts, representations of the state of the world, and I think there’s room to improve further.

Q: The pace of progress from GPT-2 to GPT-3 to GPT-4 has been dizzying. What does the pace of the trajectory look like from here? Will it be exponential, or an S-curve that will diminish in progress in the near term? If so, are there limiting factors in terms of scale, compute, data, or architecture?

A: Certainly in the short term, the thing that I’m most scared about has to do with these truthfulness and coherence issues that I was mentioning before, that even the best models that we have today do generate incorrect facts. They generate code with bugs, and because of the way these models work, they do so in a way that’s particularly difficult for humans to spot because the model output has all the right surface statistics. When we think about code, it’s still an open question whether it’s actually less work for somebody to write a function by hand or to ask a language model to generate that function and then have the person go through and verify that the implementation of that function was actually correct.

There’s a little danger in rushing to deploy these tools right away, and that we’ll wind up in a world where everything’s a little bit worse, but where it’s actually very difficult for people to actually reliably check the outputs of these models. That being said, these are problems that can be overcome. The pace that things are moving at especially, there’s a lot of room to address these issues of factuality and coherence and correctness of generated code in the long term. These really are tools, tools that we can use to free ourselves up as a society from a lot of unpleasant tasks, chores, or drudge work that has been difficult to automate — and that’s something to be excited about.