Credit: Ilya Dreyvitser/FilmMagic; Victor Chavez/; Denise Truscello/

Chelsea Handler may be tight with her fellow E! stars, but that doesn't mean she wants to see them mingling with her new A-list friends.

"Jennifer [Aniston] said she saw the Kardashians at dinner last night," the Chelsea Lately host tells The Hollywood Reporter. "She was like, 'They're nice.' And I said: 'Don't get photographed with the Kardashians!' People will blame me for it!"

Kidding aside, Handler defends her brash sense of humor (which often includes poking fun at celebrities), arguing that she only makes fun of people who "are acting irresponsibly or are making fools of themselves."

"There's no reason to talk about Reese Witherspoon. I've never talked about those people and anyone I've become friends with," Handler explains. "I'm not going to be friends with them if they're idiots in the first place."

Fortunately for the Kardashians and other celebrity targets, Handler's Chelsea Lately contract expires in 2012 -- and she may not renew the network's offer to extend her hit late night show.

"I want to do something that's going to utilize my brain a little more than this show," she says. "If Lately is the show that I'm going to do, it's going to change. But it may turn out that I'm done with it altogether. I can't keep doing the same thing; my brain is bleeding. I want to do something that's more mindful and isn't celebrity-centered. I'm not looking to totally bail on E! They've done a lot for me, and I like it here...if you take away the Kardashians."