I strongly disagree. Silent protagonists gives much more freedom to me to truly develop the character I'm playing as. In Legend of Zelda Link being silent makes sense because he's designed to be more of an avatar than a completely premade character that you're merely taking control of. I personally think Breath of the Wild in particular fits well with the silent protagonist approach with its non-linear design and a main character that in many ways is a blank slate.
As for Xenoblade X I really liked that they let you create your own character and it would've been difficult to give you that much freedom over your main character had that character not been silent. My problem with that game was that there seemed to be a conflict between character-driven and story-driven design. The main story should've been less linear and more flexible to fit with the open-world exploration approach. For example it killed the immersion for me when the story acted like it was my first encounter with certain locations or creatures when I had already discovered those. Instead it should've been something like: explore -> discover new location/species -> open up new story branch.
Despite its flaws I still loved X and I hope they make another one with even more focus on roleplaying, exploration, interaction with the world and dynamic storytelling.
