Sure, I mean, go for it. You are free to fully embrace "death of the author" and believe what you like, irrespective of Tolkien's direct statements.

Tolkien's direct statements carry a fair bit of weight with me. His private writings were extensive, consistent, and deeply introspective.

It's as you say: primary sources aren't everything. But they're not nothing. In Tolkien's case, they're really something.