The couple's son, whom they've named Kelsey Gabriel Elias Grammer, was born on Tuesday, July 22, at 8:28 a.m. in L.A. He weighed in at 8 pounds, 2 ounces.
"Our son will be called Gabriel as there is a tradition in our family of going by our middle names," the Emmy-winning new dad, 59, told ET. "We are blessed and excited to have this lovely young man join our family—he is magnificent!"
Gabriel joins big sister Faith Evangeline Elisa, 2. Grammer also has four other children—Spencer, 30, Greer, 22, Mason, 12, and Jude, 9—from previous relationships, and is a grandfather to Spencer's son Emmett, 2.
Kayte is the actor's fourth wife. They married in February 2011, weeks after he finalized his divorce from his third wife, former Real Housewives of Beverly Hills star Camille Grammer.
Sign up now for the Us Weekly newsletter to get breaking celebrity news, hot pics and more delivered straight to your inbox!