Matthew Fox: I’m Quitting TV After Lost

 Marcia Campbell Photography/FilmMagic

When Lost wraps up this May, don't expect Matthew Fox to find himself on another TV show.

"I won't be doing any more television," the star, 43, tells

Fox, who starred as the eldest brother in the '90s drama Party of Five for six seasons, joined the cast of the ABC drama in 2004 and is eagerly anticipating the end of the series' sixth and final season.

"I've done two six-year shows, about 300 hours of television," he told Us at the show's season premiere event on Waikiki Beach in Hawaii on Sunday (it airs nationwide Tuesday). "I'm done with that."

Fox — who stars as Dr. Jack Shephard — hasn't sworn off acting entirely.

"I'm either going to make movies with filmmakers I want to work with or I'll be doing something else entirely," he says, adding that his lifestyle doesn't require him to earn "millions" of dollars.

In fact, the father of two — he has a daughter, Kyle Allison, 11, and son Byron, 8, with wife Margherita — is ready to put his family first.

"My family will move to Oregon, back to the country I love," he tells Us of his plans to relocate from Hawaii and L.A. "I miss the four seasons climate. We're building a house up there and I'll be closer to my brothers and my mother. Our kids will be closer to their first cousins."

Still, he'll always have a soft spot for the Lost experience.

"Being in Hawaii and working here has been an incredible, fantastic experience," he says. "Hawaii and Lost has been a great thing for me."

Sign up now for the Us Weekly newsletter to get breaking celebrity news, hot pics and more delivered straight to your inbox!

Want stories like these delivered straight to your phone? Download the Us Weekly iPhone app now!