When it comes to Jesse James' alleged adultery, maybe the apple doesn't fall far from the tree.

"We all saw it coming," a family source tells UsMagazine.com of James, 40, accused of cheating on wife Sandra Bullock with tattoo model and stripper Michelle "Bombshell" McGee. As others claim in the new issue of Us Weekly that the McGee affair "is not an isolated incident," the source says that James' struggle to stay faithful is "his nature."

PHOTOS: Stars caught cheating

The insider points to the example of James' father, Larry -- who, the source claims, frequently cheated on his wives and girlfriends during the Monster Garage star's childhood.

"Every time Jesse would get close to a stepmom or a girlfriend, the dad would cheat and somebody new was in the picture," the source says. "It just wasn't a good way for Jesse to grow up...The stability really wasn't there."

Of his marriage to wholesome Bullock, 45, the insider muses, "He wants the stability, yet he's still attracted to the wild ones...women just [throw] themselves at him."

James and his dad are "estranged" and haven't spoken "for years," the source says. "I don't think they'll ever get back together."