Last night I entered the page-long instructions for Advent of Code day 4, and it spewed out perfectly readable code and solved it on the first try [1]. And we're not talking about a common algorithm that has been solved many times before, but a convoluted story that is full of "elves cleaning up overlapping sections of the camp" (!), and ChatGPT was still able to understand it, write the code to solve it, and even explain how it works.
Are you sure this whole problem with its solution wasn't indexed in ChatGPT training set, so all it did here is retrieve the whole answer to the problem similar to how google will find the solution pretty quickly if you search for "advent of code day 4 solution"?
Can you see if it solves an advent of code problem whose solution hasn't been posted yet? If it did I will consider this an almost publishable breakthrough.
It wasn't in the training set - the puzzle was released today, as part of this year's advent of code. ChatGPT is eminently publishable and has demonstrably produced plenty of novel, if boring, code.
Yes, but you'll have to chop the project up into different parts and describe each sufficiently well.
For example, start with "write a level generator in JavaScript which outputs pacman like levels encoded in ASCII which could be used as the basis for a pacman clone."
Then you'd say "Great, now write a pacman clone in JavaScript which takes the input of the previously generated level generator and takes interactive input on the console."
Then you'd have to ask it to generate the HTML and CSS. It won't do it all at once. But it will do it piece by piece.
Yes, it's absolutely feasible. I did exactly that with a simpler version of this (Tic Tac Toe, and console only, no actual visual interface) but it worked pretty well.
The main thing is having a good idea of what the overall program structure will look like. If you have that, I think you can go pretty far with it.
Btw, in case anyone wants to see what working like this may look like, I recorded myself while doing that little experiment: https://youtu.be/TBMQIDMwI5E
I'm trying that right now. It can probably do it, the big issue right now is that it won't give really long answers and just stop in the middle, which can be avoided by asking him to finish it, but it doesn't always works.
If you can break down what you want into smaller functions, instead of just asking for the whole thing at once, you can actually do a lot with it, including telling it to out it all together at the end.
Won't link it again so as to not get too spammy, but I posted a recording of an experiment I did playing around with ChatGPT in that way on a sibling comment if you want to check that out.
I've tried giving it languages like Turbo Pascal or telling it to obscure frameworks like fp-ts and while sometimes it's not syntactically perfect, but I can guarntee it's not just scraping code snippets from somewhere.
It's nothing short of a phenomenal milestone.
[1] https://twitter.com/GuiAmbros/status/1599282083838296064