The R&B singer, 35, made the accusation while talking to singer Stacy Francis on the British reality show.
Francis, who was friends with Ray J's sister, Brandy, said she was staying at the Moesha star's house and she saw Kardashian styling Ray J's clothes. "She was really into you, really in love with you," Francis, 47, said of the Selfish author.
"Nah, but she was a player though," the former For the Love of Ray J star insisted.
"She was not a player, she was crying her eyes out for you," said Francis, who appeared on the first season of The X Factor in the U.S.
"You don't really know what happened though," Ray J said. "Put it like this, we were both players. We were both cheaters."
"I don't believe that about her," Francis retorted. "I remember her being very heartbroken over you. I think she really loved you very much."
Kardashian, now 36, met Ray J when she was working as a stylist for Brandy. They dated on and off for three years from 2003 to 2006. The exes made headlines the following year after a sex tape they'd made when they were dating was released by Vivid Entertainment. Kardashian sued over the video, which was titled Kim K Superstar, and reportedly received a multimillion-dollar settlement.
Ray J told Francis she didn't know the whole story about his relationship with Kardashian, while admitting that they'd loved each other. "Every relationship I was in, I loved the girl. It's just the moral values and standards was off. I was loving and lying."
The singer — who had a minor hit with the 2013 song "I Hit It First," which was seen as a dig at Kardashian and her then-boyfriend, Kanye West — said that he hasn't "crossed paths" with the Keeping Up With the Kardashians star since their split.
He said that he's run into her mom, Kris Jenner, and sisters at basketball games and said hi, and added that he had no bad feelings toward West.
Sign up now for the Us Weekly newsletter to get breaking celebrity news, hot pics and more delivered straight to your inbox!