My reaction is that it's a silly thing to say, because there is no way to ever get a concensus on any of those things. No one has listened to music from every artist of the last 15 years in order to compare, and import and influence won't be truly known until a long time from now anyway (probably still won't be a concensus then either).
...You don't need to have listened to every artist to see his influence. He literally changed the trajectory of both hip hop and pop. Some of the biggest musicians of today literally wouldn't exist today without the path Kanye paved for them with the music he's put out.
Drake, B.o.B., Kid Cudi, Childish Gambino, Frank Ocean, even The Weeknd would not be relevent as they are now if Kanye hadn't introduced the kinds of sounds and subject matter they've since built their entire careers off of in 808s and Heartbreak to the mainstream. Some of them wouldn't even have careers. Same with Graduation. College Dropout literally redefinied hip hop when it came out. It redefined the sound, the dress code, the subject matter, the culture, and the image. Notice the severe drop in baggy pants, oversized tees, and flaunted boxers, replaced instead by skinny jeans and fitted polos in young black urban culture in the past 15 years? Who do you think did that? Eminem? 50 Cent? Bruno Mars? No, it was Kanye West and his pink polo. And that's not to even remotely get into his revolution of the perception of rap and hip hop in white culture.
My Beautiful Dark Twisted Fantasy is inarguably one of the greatest mainstream albums of all time. It's definitely one of the most dense, well produced, well organized albums ever created. No one makes albums like that. Some of the most important musicians of the past century - MJ, Queen, The Beetles, some of the greatest musicians of all time, haven't made albums as tight as MBDTF. If 808s introduced the sound that would go one to define much, if not most, of modern pop (which has been the most dominating and successful genre of, what, the last 35 years or more?) MBDTF pressured that sound to a lazer focus and perfected it.
Even his most recent album, Yeezus, which is his most controversial album since 808s, is already showing blooming influence in Rihanna's most recent album Anti, which came out less than a month ago. That new wave of dark, industrial, distorted sounds will only drip more into the mainstream over the next five years as the top artists become influenced by other big early adopters like Rihanna and the success of these new sounds.
So no, it's not a silly thing to say. The consensus isn't just clear - it's screaming. Love him or hate him, no musician is more important to the 21st century than Kanye West. Not one.