It seems to work nicely on simple queries, however there are some rough corners which I don't think have a simple solution. For example the query "how to set the timezone in react-datepicker" first offers a Stack Overflow solution from 2019, however that answer is outdated and no longer works. The other solution offered copies code from a different Stack Overflow answer verbatim, which is problematic since it doesn't correctly license the code — code on SO is CC BY-SA which means you have to both attribute credit and link to the license.
This is one of the biggest challenges that I see for LLMs in relation to codig: Giving answers that work on the particular language and library version(s) you're developing for.
Most of the data these LLMs are trained on aren't labeled as to version number, so they really have no way of determining which version of a particular language or library the code they provide will work on.
It might work if you're doing something generic enough. Otherwise you're going to have to rely on luck on it working with your particular version.
