Simon Cowell will be judging AGT next season, a source confirms to Us. Credit: Joel Ryan/Invision/AP

America’s Got Talent contestants better brace themselves for some brutal feedback. That’s because Simon Cowell will be taking over Howard Stern’s now-empty spot at the judges’ podium come the new season, Us Weekly can exclusively confirm.

"We've been talking about it for a few years, funny enough, me and NBC," Cowell tells Us, noting that he and the network first discussed the job five years ago, but that it didn't pan out at the time. "This year ... everything worked. So it was perfect."

The reality TV personality, 56, is known for his unflinching honesty when it comes to critiquing performers. He crushed countless dreams with gestures as simple as an eye roll to biting quips during his tenure as a judge on Fox’s American Idol, from its premiere in 2002 until he left in 2011 after nine seasons.

But for those wondering whether being a dad now means Cowell will show a softer side, forget it. "I'm not going to lie to people when I think they're hopless, I just couldn't do that!" he says, laughing. "But no, of course I'm not going to be rude for the sake of being rude. That would be ridiculous. I'll take it act by act."

Judging a competition featuring a wide range of talents beyond just singing won’t be new to the music and television producer. In addition to American Idol, Cowell has judged both the American and British versions of The X Factor, as well as Britain’s Got Talent.

Shock jock Stern, 61, joined the popular NBC show in 2012 for Season 7, and stepped down after Season 10 concluded in September. He announced his decision to vacate his seat at the judges’ table during his SiriusXM show in June, saying that he couldn’t do his radio program, AGT, and a possible upcoming TV opportunity all at the same time.

AGT is hosted by Nick Cannon, and judged by Howie Mandel, Heidi Klum, and Mel B., though none have been officially confirmed for Season 11 yet.

The show will be holding auditions for the upcoming season in cities including Detroit, Kansas City, Los Angeles, New York, and more. For a full list of cities and dates, check the AGT auditions website.

Can't get enough of Us? Sign up now for the Us Weekly newsletter packed with the latest celeb news, hot pics, and more!