If I recall the introduction correctly, the lectures were kindof a failed pedagogical experiment because of this - it worked well for the best students, but the mediocre/worse students did a fair amount worse than in previous years. I don't think Caltech continued to teach the first two years in the Feynman style after this.
But regardless, the lecture series is a wonderful overview on the actual physics, even if it doesn't work that well as a straight textbook.
Failed experiment or no, it's probably the single most influential set of undergraduate physics classes ever taught. Grad students attened these classes too, and nearly everyone in the field has a copy of these lectures sitting on the shelf.
They're not a singular reference, but they're really good. His rachet and pawl lecture is timeless.
These lectures are awesome if you have already encountered the material and want to start over from scratch really internalizing it this time. If you use them as a first-time learner you are likely to be pretty confused.
But regardless, the lecture series is a wonderful overview on the actual physics, even if it doesn't work that well as a straight textbook.