Nicole Richie, mother of two, says she isn't sure she wants more kids. Credit: Michael Kovac/Getty Images

Nicole Richie is glad to be done with diaper duty. During an Apr. 23 appearance on Chelsea Lately, the Fashion Star mentor and mom to Harlow, 5, and Sparrow, 3, revealed she's not sure she's ready for a third child.

"You haven't had a baby in forever. What's your problem?" host Chelsea Handler joked. "Are you being selfish right now?"

Richie fired back, "I'm just chillin'. Let me just chill! I really like that they're doing their own thing. They're at school until 3. I don't have to, like, look after them -- or even think about them! It's like every other day."

When asked point blank if she wants to expand her family, Richie replied, "I don't know." Unlike some mothers, Joel Madden's wife said she's not one to brag about her kids' every attribute. "They want to talk about their spit-up and how cute their boogers are. I try to not be that mom," she explained. "I know what is really cute and what I only think is cute."

The former Simple Life star is enjoying some free time now that her children are becoming more independent. In fact, Richie recently went on a childless vacation with a group of friends (including fellow mother of two Jessica Alba). "Everyone left their kids at home," the 31-year-old House of Harlow designer said. "It worked out great."