Naughty Dog: Next Uncharted Will “Benefit Heavily” From The Last of Us Dialogue System
There’s a lot more to a dialogue system than meets the ears. In a recent interview with The Frontliner, the Naughty Dog audio team explained how The Last of Us progressed the company’s dialogue technology and in turn the AI, and how it will all benefit the next Uncharted game.
Speaking to The Frontliner, Naughty Dog Audio Programmer Jonathan Lanier said, “The dialogue system in The Last Of Us is amazing, and (Dialogue Supervisor) James should get a lot of credit for working with (Lead Programmer) Jason Gregory on it. We never had anything remotely as good for the Uncharted series. The next Uncharted will benefit heavily from this.”
The comments come after the Naughty Dog audio team went into detail on The Last Us’ audio technology and how programmers wanted “intensive dialogue systems” for the Artificial Intelligence.
“We built breathing systems for all the characters and tried to make it as realistic as possible. It was nice to see it all come together,” said James.
Jonathan goes on to explain, “One of the problems we had with not only The Last Of Us, but the Uncharted games also is that these games are heavily story-centric. A lot of the exposition of that story happens in the dialogue. This creates nightmares for us, because game designers can create situations where they’re not thinking about dialogue, and yet dialogue is very tricky to create.”
He says the Naughty Dog audio team used various audio tricks to simulate the effect of listening to real-world dialogue while also balancing the levels with music and environmental effects. According to Jonathan, problems caused by level layouts and characters having to hold conversations while moving around potential obstructions is a common problem for audio designers.
“A lot of games don’t bother with addressing things like that. They just play all the audio in the center channel or don’t pan it, which is sort of a cop-out.”
“I came up with a plan: We used a psycho-acoustic technique to trick the listener. We created a separate reverb unit that had a very short delay on it, and dedicated it to dialogue in cases where characters were obstructed. We called it the “Dialogue Sweetener”. This allowed us to create a tiny bit of reflection that was enough to trick your brain into thinking that the sound was bouncing around in the room. So if Ellie walks around a corner, you can still hear her as if she was close by.”
Naughty Dog’s custom ‘dialogue manager’ allows characters to interact verbally in a natural manner, but this represents something of a “double-edged sword” according to James:
“The more you allow characters to interact verbally, the more telling it is when they break form or do something unrealistic. We call it the “uncanny valley”, which means that we’ve made things look so realistic that anything which doesn’t look real will stand out. When your dialogue manager is so sophisticated that you can feel character’s moods and feeling, the in-game Artificial Intelligence can’t afford to make mistakes.”
“If the AI makes a dumb move, it’ll be telegraphed through the dialogue. So one of our biggest problems with raising the bar for dialogue is to create a smarter AI, because it can easily do things that are wrong.”
James compounds his point with a staggering fact, “We shipped this game with close to 70,000 lines of dialogue, which is huge amount for a non-RPG game this size.”
“We also built a context awareness system so that enemies can call out your location when you’re fighting. In order for that to work, you have to record every single object and location that are in the different environments. “He’s by the van!” or “She’s by the bar!” or “She’s across the street!“. So once the levels were fleshed out, we’d have to go through each area and take note of the fact that we might have a store, an office, a street and an alleyway present, with vans and trucks parked outside.”
This type of context aware audio system should allow the next Uncharted to be even more immersive than its predecessors, which were considered some of the finest examples of cinematic storytelling on the PlayStation 3.
Unfortunately, with great dialogue and AI systems comes great strain on the vocal chords of voice actors:
“Not only did all of that dialogue need to be recorded, but it had to be recorded multiple times with variations, so that it didn’t become repetitive and unnatural, ” says James, “Our poor voice actors were screaming “He’s over there” and “She’s by the car“, for 4 hours over the course of 5 recording sessions. I’m sure they all hate me, haha. But that’s what it takes to model this kind of thing, and as games continue to model reality even more, we’ll get into this kind of territory even more. Sorry voice-actors….”
If you’re interested in how audio plays a role in game design and development, I highly recommend reading the full interview with the Naughty Dog audio team on The Frontliner here.
