Unit tests are exactly one of those situations I had in mind. Unit tests are always kind of painful that way: you need to construct things by hand, using interfaces that were designed for automated use.
The same fact is what makes unit tests so brittle. Ideally, you do as little of that as possible: you less fake stuff you make, the more robust the test will be.
There are days you can't, and that sucks. But the language is giving you a nudge in the direction of how to do that, by making it easier to construct actual POJOs than unstructured intermediate data.