Episode Transcript
This is a computer-generated transcript. While our team has reviewed it, there may be errors.
Morgan Sung: My colleague Francesca Fenzi, digital community producer at KQED, spends a lot of time online.
Francesca Fenzi: I’m on KQED’s Discord server, on Reddit, all the various chat threads and usual places.
Morgan Sung: After President Trump was sworn into office earlier this year, she noticed a new wave of viral content about him and his administration. There were clips claiming all kinds of things. One of these viral clips shows Musk and his son, X, during a visit to the White House. X was four at the time. While Musk speaks to the press, X runs around the Oval Office and makes his way the president’s desk.
Francesca Fenzi: At one point, he peels off and he makes a side comment to President Trump. This is all happening while Elon Musk is speaking, so it’s very hard to hear what he’s saying. But a lot of people started interpreting what the boy was saying as saying, “You’re not the president and you need to go away,” to Donald Trump.
Elon Musk: Part of the presidency is to restore, is to restore democracy.
Morgan Sung: Again, X is very young. This was a four-year-old babbling at the president. Another moment from the same meeting seems to show X telling the president to, “shush his mouth.”
Elon Musk: Taxes from one year to the next are the same.
Francesca Fenzi: I did watch this clip. It does seem like he might have said something like this, but my audio producer brain also lit up right away because I can tell you that that clip was not clear. It was not the kind of thing that you can clean up in post. So when people started quoting this as fact, as something that was happening, my interest was piqued.
Morgan Sung: The whole thing set Francesca down a research rabbit hole, which — relatable. This clip felt different from other pieces of viral misinformation she’d seen online.
Francesca Fenzi: We’ve been really focused on AI’s potential to spread false information and how it can trick people into believing things that never actually happened. But even without AI, our senses can be fooled the old fashioned way. And what I was interested in in this situation was just trying to understand how our brains process what we see and hear and how those senses can be manipulated, not by AI, but just by old fashioned audio and video tricks. And so my research spiral, led me to this whole world of auditory illusions.
Morgan Sung: Remember those Magic Eye posters? They were those optical illusions that looked like random patterns until you unfocused your eyes just right and a hidden image popped out. Well, it turns out it’s not just our eyes that can be fooled. Our ears can play tricks on us too. There’s a whole world of auditory illusions that seem to say one thing when they’re really saying something else. And that matters, especially in an age of misinformation.
In today’s episode, we’re looking at some phenomena that can completely change what we think we hear. And we explore how understanding the limits of our perception might actually make us better at spotting disinformation online.
This is Close All Tabs. I’m Morgan Sung, tech journalist, and your chronically online friend, here to open as many browser tabs as it takes to help you understand how the digital world affects our real lives. Let’s get into it.
Okay, so we’re gonna start with something that feels super commonplace, but might not be as reliable as we think.
New tab. Can we trust lip reading?
Okay, so in this video with Elon Musk’s son and President Trump, a lot of people were really relying on a combination of very poor audio and lip reading to decipher what he was saying. So let’s start with lip reading. Francesca, how reliable is it?
Francesca Fenzi: Well, it’s estimated that only 30 to 40% of speech can actually be lip read, and that’s even under the best conditions. Lip reading is a really useful tool for people who are hard of hearing because it helps to piece together context around other pieces of information, right? Like partial audio or even like hand movements in sign language. So some lip readings seem really good when we think that we have the context associated with them. One example of that is there’s a TikTok creator who I really like, her handle is “It’s Jackie G”, and she interprets celebrity red carpet moments. So she’ll take moments of celebrities being recorded and will lip read the conversation that’s too far away from the camera for us to be able to hear accurately.
Morgan Sung: Here’s Jackie G, lip reading Zendaya at this year’s Met Gala.
Jackie G: She’s so fab. So fabulous. I love it. It’s so funny because she’s, I would say when we, when it cuts off.
Francesca Fenzi: The reason that those work is because as a fan of a certain kind of celebrity, you are probably a little bit aware of how they feel about the movie that they’re promoting or their relationship to other stars who they might be interacting with, and they seem really plausible because of that. But the more removed lip reading is from its context, the less you understand about the true nature of the relationship of the speakers, the more likely you are to be misled. That’s part of what makes the YouTube series Bad Lip Reading so possible and successful.
Morgan Sung: Bad Lip Reading is a YouTube channel that intentionally misinterprets what people are saying in movies or TV shows and then voices them over. Like this scene from Star Wars, a conversation between Obi-Wan Kenobi and some stormtroopers.
Stormtrooper: Hey guys, we’re collecting donations for the Jawa Orphanage. Do you have any spare change?
Obi-Wan Kenobi: Hey, you should know that you stink kinda like fish.
Stormtrooper: Wait, what?
Obi-Wan Kenobi: Everyone knows it except for you.
Francesca Fenzi: You’re removing context from the audio that you’re hearing and you’re replacing the story with sound that maybe mirrors some lip movements but is totally nonsensical to the scenario. And that’s the source of the humor in those videos. I spoke to Nicholas Davidenko, who’s a researcher at the High Level Perceptions Lab at UC Santa Cruz. He studies auditory illusions and I asked him why bad lip reading videos look so convincing.
Nicolas Davidenko: The reason they work so well is because lip reading is a much more ambiguous cue. So there’s actually a lot of words that could fit the shapes of my lips as I talk.
Francesca Fenzi: Davidenko studies this type of phenomena in his lab. He’s been researching how to help people with something called misophonia. That is when you have an extreme negative reaction to certain sounds. There’s some sounds that all of us find a little bit unpleasant, but these are folks who have a really extreme triggered reaction to sounds like chewing or teeth mashing. Those are some common ones.
And Davidenko found that you can actually pair a different kind of video with the audio that would normally trigger misophonia for somebody. So for example, if the sound that triggers you is that sound of chewing, you can replace an image of somebody chewing with another plausible sound source. In his lab, they use an example of somebody stepping on leaves to kind of mimic that crunching sound that might originate with chewing on food. And by swapping that image out, people start to interpret that sound differently.
Nicolas Davidenko: If the visual signal is telling you something, you trust it more than the auditory signal and when there’s a conflict, you tend to go with whatever visual system is telling you.
Morgan Sung: Got it. So the thing we’re seeing with our eyes is overruling what we’re hearing.
Francesca Fenzi: And that’s because of something called the McGurk Effect. There was this famous study in 1976, McGurk and McDonald, and what they found was that what we see can actually change what we can hear. In fact, I’m gonna demonstrate. Can I play you a video, Morgan?
Morgan Sung: Please do.
Francesca Fenzi: I’m going to play you clip, and in this first one, I just want to hear what sound you hear.
McGurk Video: Bah bah bah
Morgan Sung: It’s a close-up of a person’s mouth.
McGurk Video: Bah bah bah
Morgan Sung: Okay, so I’m hearing bah with a B, like baby.
Francesca Fenzi: Yeah, yeah, that’s right. Okay, so now I’m gonna play you a different clip and tell me this time what you hear.
McGurk Video: Bah, bah, bah.
Morgan Sung: Okay, now I’m hearing fa, like with an F, like fabulous.
Francesca Fenzi: Right. So it’s actually the exact same audio. So if you were listening and you heard exactly the same thing both times, you’re not crazy. Morgan’s being tricked. The audio…
Morgan Sung: That is wild.
Francesca Fenzi: Yeah, it’s crazy, right? It happened to me too the first time I watched this. The audio is actually exactly the same, but what listeners aren’t getting in this case that you are is the lip motion is different from one to the next, and that’s actually changing the that you’re hearing the audio. So when you see that B shape being made with the mouth, you hear “ba,” and when you see that F shape being with the mouth, you’re actually hearing it like “fa.”
Morgan Sung: Right, like I saw the person’s bottom lip hit their teeth. It’s like, yeah, that’s an F shape.
Francesca Fenzi: That’s the effect that bad lip readings take advantage of, is they’re taking those lip shapes with plausible sounds and they’re kind of swapping them for things that are similar phonetically.
Morgan Sung: So experiencing the McGurk effect in a bad lip reading video is pretty funny, but I can imagine that if this falls into the wrong hands it can go very poorly?
Francesca Fenzi: For sure. Something else that Davidenko explained is that we can be misled by someone telling us what to hear or see ahead of time. That’s kind of playing into this idea that contextualizing those clips changed the way we hear it too. So it’s not just the McGurk effect, it’s also the expectation we have coming into a video. In this case, Davidenko also worked on an experiment in his lab called Mind Controlled Motion. That is how he named it, Mind Controlled Motion.
In that experiment, what researchers did is they showed people a set of randomly refreshing images that were just pixels on a screen. So they’re just truly randomly refreshing pixels popping up and disappearing. There’s no logical motion behind them. But when researchers said something like, “left, right, left, right,” or “up, down, up, down,” over and over, when people viewed these images, then people were actually seeing the motion that they were told to see. And he said that there was a 90% compliance rate, meaning like 90% of the people who watched these and got those prompts saw the motion that they are being told to.
Morgan Sung: Yeah, this really makes me think of those like kind of rage bait body language reading videos that we always see online. Like an infamous example is Couch Guy, where a girl walked into her long distance boyfriend’s apartment and he just didn’t seem as excited to see her as people thought he should have been.
Couch Guy Commenter: All right, there his arm goes to the side of his pants, grabs his phone from old girl, acts like he’s laughing to pull it up through the middle and then boop, there went her hand.
Morgan Sung: And some people were like, no, that’s a totally normal reaction. Like, look at his body language. He’s just surprised. Well, as other people were, like, this is, like he hates her. Look at his, body language. Does this explain, like why people see completely different things in the same viral videos?
Francesca Fenzi: Yeah. Yeah, yeah. It doesn’t have to be a deep fake to be misleading. It can be a real situation that really happened. But how something is presented has a lot to do with influencing our perception of the relationship of the before and after that surround that moment.
Morgan Sung: Okay, so when I’m watching something and also listening to it at the same time, it’s like, “yeah, my eyes can deceive me,” but at least audio on its own is safe, right?
Francesca Fenzi: Uh, unfortunately, not exactly. Morgan, do you remember the whole Laurel, Yanny thing that broke the internet a few years ago?
Morgan Sung: It feels like a lifetime ago.
Francesca Fenzi: Yeah, it was like the dress, but for your ears.
Morgan Sung: And we’re going to hear about that after this break. Okay, welcome back. Time to open a new tab. Can people hear different things in the same audio? So I remember this whole Laurel versus Yanny thing back in 2018, feels like forever ago in internet time. So can you remind us what it was all about?
Francesca Fenzi: Yeah, so this actually started with a group of high school students who were studying vocabulary words for their English class. They were on vocabulary.com and they were sending recordings of different words to each other on Instagram and they discovered that when they recorded one word, they were hearing totally different things one person to the next.
So some people were hearing the word Laurel and others were hearing the word Yanny, which are so different that it kind of kicked off this debate in their friend group. And then eventually someone posted to Reddit and Stranger started weighing in as well. And it became this divisive litmus test of sorts where people were hearing either Laurel or Yanny and then being fiercely adamant that it was not the other. Actually, I’m gonna play the clip for you now. I wanna hear what you hear in this.
Laurel vs Yanny: Laurel, laurel laurel, laurel, laurel.
Morgan Sung: You know what’s crazy? When I first heard this, um, what was this, like seven years ago in 2018, I swore it was saying Yanny. And now… I’m hearing Laurel.
Francesca Fenzi: Okay, I think I may have an explanation for that. I hear Laurel too, and the unflattering reason behind that, or one of the theories, is that it may be related to our age. So I hate to tell you, but I think you’ve arrived in Millennialville.
Morgan Sung: It’s time. It’s coming for us all.
Francesca Fenzi: So when this was first circulating, the New York Times and Wired and a bunch of other news outlets took it super seriously getting to the bottom of what was happening. And the New York Times actually created this tool to help people hear both sides. So if you only heard one or the other in this clip, I can play a clip of audio that shows you what it should sound like moving from the Laurel to Yanny spectrum.
Laurel vs Yanny: Laurel, laurel, laurel, laurel, yanny, yanny, yanny, yanny, yanny.
Morgan Sung: Okay, I was hearing like hints of Yanny. It was so weird. It was like my brain was fighting the Yanny.
Francesca Fenzi: Yeah, it feels like it’s like fighting to come to the surface.
Morgan Sung: Yeah.
Francesca Fenzi: But eventually like it kicks over.
Morgan Sung: Exactly! So there’s some actual science behind why some people hear Laurel and others hear Yanny, right?
Francesca Fenzi: Yes, so it’s still hard to know for sure what caused it, but there’s a prevailing theory and essentially two things are what people think is happening in this clip. One is that the recording of the recording has added and introduced new frequencies to the audio. So remember, these were high school kids who were recording a clip played from online of a vocab word.
The original word is Laurel. So if that’s what you heard, you were correct. But when Laurel was recorded through computer speakers into a phone and then sent across the internet, it introduced some additional frequencies to the audio. And our brains are choosing which of those frequencies to prioritize.
And this is where the age part comes in. Older people tend to hear lower frequencies and less of the high frequencies. And then younger listeners, they have a broader range of those high frequencies available to them. So the theory is that if you hear Laurel, you’re probably prioritizing those lower frequencies, and if you hear Yanny, your brain is prioritizing those higher frequencies. And younger people may be more inclined to prioritize those because they can actually hear more of them.
Morgan Sung: My God. Sorry, I’m still coming to terms with my ancient, decrepit ears.
Francesca Fenzi: I know. It’s the worst.
Morgan Sung: So are there any bigger implications for this phenomenon or is it just like an oddity of the digital age?
Francesca Fenzi: So it means that there’s just more opportunity for ambiguity and misinterpretation. We’re listening to audio, if you think about it, in all these different forms all the time now, we’re playing them through computer speakers, from our phone speakers on crowded busses, in our car stereos, which means that there’s a lot of opportunity for us to hear things differently.
Francesca Fenzi: So is this why I sometimes miss your song lyrics? Like I swore, god, back in the day, Taylor Swift’s Blank Space. I swore she was saying “Starbucks lovers.”
Taylor Swift: Got a long list of ex-lovers. They’ll tell…
Francesca Fenzi: Yes, that is a common one. The actual lyric is “long list of ex-lovers,” but I get why people hear that.
Morgan Sung: And here I am, like, I knew it was the wrong lyric, and for the last, I don’t know, 10 years, I’ve just been like, “I don’t need to learn it, it’s Starbucks lovers.”
Francesca Fenzi: Well, that’s actually a different phenomena, but just as fascinating. It turns out our brains do make up words that aren’t there. And that’s what’s happening when you’re listening to song lyrics sometimes.
Morgan Sung: Let’s hear about that, in a new tab. Can my brain make up words that aren’t there?
Francesca Fenzi: You’re definitely not the only person this happens to. I actually went around and I asked our colleagues about some of the songs that they’ve misheard.
Susie Britton: My brother and I, when we were little, used to play what we called the wrong song game. One that I remember was Bonnie Tyler’s It’s a Heartache.
Bonnie Tyler: It’s a heartache
Susie Britton: And we heard that as, it’s a hard egg.
Mark Nieto: I learned this very, very late in life, which I’m embarrassed to admit, but yeah, the song lyric is “revved up like a deuce” from Blinded by the Light by Manfred Mann.
Manfred Mann: Blinded by the light. Revved up like a deuce another runner in the night.
Mark Nieto: For the vast majority of my life, I thought it was “wrapped up like a douche.” And I know I’m not the only one because if you Google that lyric, you get a ton of hits for it.
Marlena Jackson-Retondo: Beat It by Michael Jackson, I always thought was, “beat it, just beat it. You don’t wanna beat it beat it”
Michael Jackson: Beat it, beat it, no one wants to be defeated.
Marlena Jackson-Retondo: But it’s actually, “no one wants to be defeated.”
Blanca Torres: When Chappell Roan’s song, Hot to Go, was really popular.
Chappell Roan: H-O-T-T-O-G-O You can take me hot to go.
Blanca Torres: My then five-year-old loved that song, would always sing along to it, and then one time I was singing along to, and I said, “hot to go,” and she was like, Mom, that’s wrong, it’s “out to go.” She wasn’t catching on to the spelling, so she was just like, “you’re wrong, it’s out to, do it right.”
Bryan Vo: Growing up, one of the childhood bangers was T-Pain Buy You a Drink. I saw a T-pain tweet about it in like 2017. And then I found out that instead of just harmonizing random lyrics, “ooh wee.” It was actually “and then.”
Francesca Fenzi: Wait, what?
Bryan Vo: Exactly what?
Morgan Sung: I’m glad I’m not the only one. So is there a name for this phenomenon, too?
Francesca Fenzi: Very perceptive. I feel like you’re getting the hang of this. It’s called the Mondegreen effect. And that’s when you mishear phrases or words and so assign them a new meaning. Sometimes it’s hearing words that you do know, but in an order that’s not what was actually being said, like Starbucks lovers, like those are real words. And then sometimes it’s just inventing a totally new word, which is what happens to me the most. I just make up a new thing.
So, this is most common with song lyrics, and it comes from the mid-century American writer, Sylvia Wright. She coined this term based on her childhood. She remembered mis-hearing the line in a Scottish ballad called The Bonnie Earl O’ Moray. And there’s a line in the song that goes, “laid him on the green,” which she interpreted as Lady Mondegreen.
The Bonnie Earl O’ Moray: Oh where have ye been, they slain The Earl O’ Moray, and laid him on the green.
Francesca Fenzi: Which is another really common way that we do this is we just make up formal nouns or things that feel like names when we don’t really know what’s being said. So Morgan, researching the Mondegreen effect actually led me to researcher Diana Deutsch. She is at Stanford and UC San Diego, and she is like the audio illusion researcher. Been doing this for decades. She discovered that when you take two audio sources, and you play the same word or syllable slightly out of sync, after about 10 seconds of listening, people start to invent phantom words in that overlap. They start to hear different things.
Morgan Sung: Okay, here’s a clip from one of these audio experiments.
Auditory Illusion: WELWELWELLWEL WEL WELL WELLWELL WEL
Francesca Fenzi: And even with the same audio, the phantom words that they’re hearing are often unique to the listener. So here’s Diana listing some of the words that people have reported hearing in the same piece of audio.
Diana Deustch: Window, welcome, love me, run away, no brain, rainbow, raincoat, bueno, nombre, when oh when, mango, window pane, broadway, even rogain.
Morgan Sung: That is fascinating. So why would people hear totally different phantom words in the same audio?
Francesca Fenzi: This is similar to something called periodolia, where people perceive familiar patterns in random or ambiguous stimuli. This happens a lot with visual things. So like seeing a face on the moon or Jesus in a flower tortilla. It turns out that that can happen with audio too.
Diana Deustch: Generally people hear words or phrases that refer to things that are on their mind. So for example, if someone’s on a diet, they might hear the phrase, “feel fat.” And it often happens when I present these to a group of students close to exam time, they’ll hear things like “no brain.” So these illusions show that when people believe that they’re hearing meaningful messages from the outside world, their brains are actively reconstructing sounds that make sense to them.
Francesca Fenzi: Diana says that the patterns we hear are influenced by things like our mood, what we’ve thought about or discussed that day, whether we’re tired or sad or scared. We assign meaning to sounds based on our internal narratives. For example, that’s part of what might be going on when people report hearing electronic voice phenomena in ghost hunting. Being scared or heightened or thinking about ghosts may lead you to hear certain phrases in ambiguous audio.
Ghost Hunters: That’s amazing. Strange, right? How it says- Goose bumps up my spine. Clearly a voice. I hear something negative, like no or don’t.
Francesca Fenzi: And Diana actually told me that one thing that’s quite common is for people who have recently experienced a loss, they’re more likely in their day-to-day lives to hear what sounds like words or phrases or even voices associated with their lost loved one.
Morgan Sung: Earlier, you mentioned that having less information, like audio without visual cues, makes us more likely to assign new meaning to what we hear. So let’s bring this full circle. So after all of this research, what would you tell someone who’s absolutely convinced that they know what Elon Musk’s son said in that clip?
Francesca Fenzi: I guess I would say that that probably says more about you and how you feel about Elon Musk or President Trump than necessarily what the three people in that audio clip were saying. It doesn’t mean that you’re wrong. It’s just that we don’t know. There’s no way to really hear truth in an audio clip that convoluted.
Morgan Sung: And what are some of the ways bad actors can purposely take advantage of how suggestible our senses and our brains are?
Francesca Fenzi: I think that the easiest way to be manipulated is when somebody is taking a clip or a small snippet of something and then abbreviating the context and telling you what goes before and after. And social media is designed in this way to give us bite-sized samples of the world. But when you’re taking just a bite, it means that you might be missing the whole meal around it, and you might get the flavors wrong. And you might kinda misunderstand what’s being served to you.
Morgan Sung: How can understanding all of this brain trickery help us spot actual misinformation?
Francesca Fenzi: Yeah, that’s the ultimate question, right? We can’t trust our eyes, we can’t trust our ears, we can’t trust AI. I think that the real takeaway, I guess, that I have after this research spiral is trust but verify. Double check your own thinking when you’re encountering one of these clips online. Is it too good to be true? Maybe it takes a little extra Googling to see if you can get to the bottom of it. And maybe, I guess another takeaway I would have is being okay with a little bit of ambiguity. Sometimes there are mysteries that Google can’t answer for us. Just like the answer of what was really said in the White House in that moment.
Morgan Sung: No amount of Reddit threads can do it.
Francesca Fenzi: But knowing that there are unknowns, I feel like being aware that people claiming to have a definitive answer might not be telling the truth.
Morgan Sung: Well Francesca, thank you so much for joining us. Thanks so much for telling us all about, I don’t know, these crazy auditory illusions.
Francesca Fenzi: Thank you for having me, Morgan. This was super fun.
Morgan Sung: Voices from KQED staff in this episode included Susie Britton, Mark Nieto, Marlena Jackson-Retondo, Brian Vo, and Blanca Torres. Francesca runs KQED’s Discord server and Close All Tabs has its own channel. Come say hi, share your thoughts, and chat with other listeners about the show. Join us at discord.gg/KQED.
Now, let’s close all these tabs.
Close All Tabs is a production of KQED Studios and is reported and hosted by me, Morgan Sung. Our Producer is Maya Cueva. Chris Egusa is our Senior Editor. Jen Chien is KQED’s Director of Podcasts and helps edit the show. Sound design by Maya Cueva. Original music by Chris Egusa. Additional music by APM. Mixing and mastering by Brendan Willard and Katherine Monahan.
Audience engagement support from Maha Sanad and Alana Walker. Katie Sprenger is our Podcast Operations Manager, and Holly Kernan is our Chief Content Officer. Support for this program comes from Birong Hu and supporters of the KQED Studios Fund. Some members of the kqed podcast team are represented by the Screen Actors Guild, American Federation of Television and Radio Artists, San Francisco, Northern California Local.
Keyboard sounds were recorded on my purple and pink dust silver K-84 wired mechanical keyboard with Gateron Red switches.
If you have feedback, or a topic you think we should cover, hit us up at [email protected], follow us on Instagram at CloseAllTabsPod, and if you’re enjoying the show, give us a rating on Apple Podcasts or whatever platform you use.
Thanks for listening!
