The answer is it’s very bad at large amounts of code. It has no skill at reasoning across a large amount of files where say you’d need to make many small changes in a handful of places and know that those changes all work together.
These models are amazing at regurgitation. They have no skill at reasoning and any solution outside their training set. They have a distinct disability for precision in anything remotely novel or large. And they will bullshit subtle bugs all over the place at scale.
Even a junior programmer would produce more value than the next generations of this, simply because they can take vague English instructions and actually make iterative progress across a huge codebase, test those changes for all sorts of vague constraints, and be precise. There’s no ability for that here.
I have a feeling though that tools like this will lower the skill threshold for programming by a lot.
I gave it this code I just made and asked it what it does and it figured it out:
public class mean_sink {
double sum;
long n;
public mean_sink(){
}
public void add(double d){
sum += d;
n++;
}
public double getMean(){
if(n == 0)
return Double.NaN;
return sum/n;
}
}
I asked it to improve the numerical accuracy and it did it with BigDecimal instead. I asked it to only use doubles and it did some strange reciprocal summation when I was "hoping" for Kahan summation algorithm. Its "reciprocal summation" did not work. But it made a correct Kahan when I asked for it.
I dunno. This is insane anyway. It is really hard to grasp the consequences.
Except you must have the underlying knowledge to effectively review code. So how that would obviate a career in programming is a strange choice by the grandparent commenter. Anyone can rubber stamp bad code, but to stop bad code from entering projects you need experts.
The bad code it produces is not just stylistically bad, it often doesn't solve the problem.
It seems to be making the same kind of errors in code as we've seen in speech, you get a lot of plausible but subtly wrong hallucinated content that only someone who knows the subject can identify. It prefers probable output over matching the specifications so it'll add unnecessary branches just because it's seen them millions of times.
Among errors I've seen it make there is bad indexing, adding conditions that serve no purpose or assuming a variable has a specific structure when it doesn't.
>Even a junior programmer would produce more value than the next generations of this
At the rate this is evolving that wont be a limitation for long.
My 14 yr old son thats taught himself to code in c# over the last few years is going to be put of a potential job before he cab be taught to code professionally.
These models are amazing at regurgitation. They have no skill at reasoning and any solution outside their training set. They have a distinct disability for precision in anything remotely novel or large. And they will bullshit subtle bugs all over the place at scale.
Even a junior programmer would produce more value than the next generations of this, simply because they can take vague English instructions and actually make iterative progress across a huge codebase, test those changes for all sorts of vague constraints, and be precise. There’s no ability for that here.