Driver, 33, starred opposite Fisher in Star Wars: The Force Awakens in 2015. In the J.J. Abrams film, it's revealed that Driver's character, Kylo Ren, is Princess Leia and Han Solo's (Harrison Ford) son.
"We finished the second one [Star Wars: Episode VIII] last summer," the Girls star revealed. "I think that was the last time I saw her, actually, was on set being Princess Leia."
"As you can kind of tell from meeting her, she burns very bright and has such a great, very generous energy," he added. "For that suddenly not to be on set, or we're talking about it in publicity or just in life, when you kind of see her around at a Star Wars event, or anywhere, to have ... her character — not just her character in the movie, but her character missing from that very small unit — is a tragedy."
As previously reported, Fisher died at the age of 60 on December 27. Her mother, Debbie Reynolds, passed away just one day later, at 84. They were both remembered at a star-studded memorial on Thursday and will be buried at Forest Lawn Memorial Park on Friday.
Watch Driver and Colbert's interview in the video above.
Can't get enough of Us? Sign up now for the Us Weekly newsletter packed with the latest celeb news, hot pics and more!