This is a very simple case that doesn’t reflect the complexity of a real project. Like so many attempts before to produce code, using a little effort, it degrades when the complexity level increases even slightly. Once there are more tables, ones that have names which cannot be easily translated from English, it breaks down quickly. These types of tools work ok for brand new projects, but work on existing projects will prove to be exponentially harder or more difficult than it is worth.

Nonetheless, it could prove useful for looking up algorithms, patterns, and generating boilerplate code. However, an important issue is will it generate similar code if queried at a later time? Not likely, which will make it less useful or result in an inconsistent codebase. Maybe you can request a version of the code generation? In-house code generators will generate consistent code, so it will be interesting to see how it is used in real projects.

Here's a more extreme example, using SQL as an API to give model access to game world state to reason about it.

https://gist.github.com/int19h/4f5b98bcb9fab124d308efc19e530...

Note that in this case it isn't even asked to write specific queries for specific tasks - it's just given one high-level task and the schema to work with.

You're right, though, that the effectiveness of this approach depends very much on schema design and things like descriptive table/column names etc (and even then sometimes you have to make it more explicit than a human would need). You really need to design the schema around the capabilities of the model for best results, which makes it that much harder to integrate with legacy stuff. Similarly, not all representations of data work equivalently well - originally, I gave the model direct access to the typed object graph, and it handles that much worse than SQL. So if your legacy software has a data model that is not easy to map to relational, too bad.

On the other hand, GPT-4 is already vastly better at this kind of task than GPT-3.5, so I think we can't assume that this will remain a limitation with larger models.

> You really need to design the schema around the capabilities of the model for best results, which makes it that much harder to integrate with legacy stuff.

This may end up being a feature of some high level frameworks … “compatible with ChatGPT” or “designed to work with xxx LLM”.

It will be very amusing if, eventually, our jobs as software engineers will be crafting bespoke AIs to maximize the efficiency of their use by an LLM.