I wouldn't, so long as it sticks to facts. No need to be unreasonably negative.

"Honestly, he only worked here for two weeks, and then quit without notice. I didn't even know he'd use me as a reference."

"Honestly, I don't know how much help I'll be. He worked here for two weeks, and then turned in his resignation stating he wasn't comfortable discussing his work with others. I didn't realize I'd be a reference before you called".

These are facts, the first two easily verifiable in court - who would likely have no trouble believing the third.