Whoopi Goldberg and Joy Behar walked off the set of ABC's The View Thursday after a fiery debate with guest Bill O'Reilly over the mosque near Ground Zero.

O'Reilly, 61, set the two off when he declared Muslims "killed us on 9/11" (see video above).

Their exit prompted host Barbara Walters to reprimand her castmates, telling the audience, "This should not have happened," that they all should be able to discuss issues without anyone throwing in the towel.

The two later rejoined the show.