Credit: Jon Furniss/

Fans worldwide swoon over Robert Pattinson these days, but the British actor says he used to be the victim of multiple beat-downs.

"I got beaten up by a lot of people when I was younger," Pattinson, 23, tells

The star -- promoting the romance Remember Me, out this Friday -- says the pummeling occurred when he was an unknown actor trying to get parts. "I was a bit of an idiot, but I always thought the assaults were unprovoked," he says.

"I liked to behave like an actor, or how I thought an actor was supposed to be, and that apparently provoked a lot of people into hitting me."

And although he's been romantically linked to many high-profile women -- from Twilight costar Kristen Stewart to Remember Me's Emilie de Ravin -- the modest Pattinson claims he's all thumbs with the ladies.

"I don't even remember the last time I asked someone out on a date," he claims. "Like, just went up to them and that's the first thing I did... I tend to hold back."

Indeed. In a recent interview with Details magazine, Pattinson lamented that he's "allergic to vaginas."

"The only emotional connection of relevance is with my dog," he continued. "My relationship with my dog, it's ridiculous."