The question is: Is it hard essentially, or is it hard accidentally? That is, can you remove unnecessary complexity from programming and suddenly it'll become easy?

That was the proposition behind first LightTable, and then Eve as originally conceived. But neither of them really found a satisfying answer, a way to say "hey, for making your web app or whatever, if you throw away your existing stuff and use this tool/process, now it's super-easy."

That implies heavily that a lot of the complexity and difficulty is essential. Not all of it -- things will get easier and better over time, as they have over the last ten years -- but enough so that blowing it all up and starting from scratch isn't likely to lead to wins.

Actually, we think we did. We're just not choosing that as the primary focus of the workflow in Eve. Now with a better version of the editor and with a bit more work on the UI builder, I suspect we could rebuild the entire foursquare clone in under a week. If we had VCS so that multiple people could work together on it, it might only be a couple of days. This foundation for programming has lots of implications for building "real software" - it's actually based on research for making distributed systems much easier to build. [1]

We'll see more of that as we go since we're bootstrapping bits and pieces. One of the first things that will transition over is the compiler, if that gives you any indication of the level of sophistication you can achieve with this programming model.

[1]: http://db.cs.berkeley.edu/papers/eurosys10-boom.pdf

I agree that blowing it all up is not a win, which is why accidental complexity goes away only gradually, and it's partially a process of hiding it rather than removing it, with lots of politics along the way.

Still I think there must be some hard lower limits on amount of incidental complexity. Nature just can't allow you to get rid of all of it (impossibility results from distributed systems theory come to mind) just as in manufacturing transport costs can't be zero (goods and materials can't be transported between factories faster than the speed of light after all). It will be interesting to see how eve team works around these issues.

Reminds me of the old:

"It is better to have 100 functions operate on one data structure than 10 functions on 10 data structures." —Alan Perlis

Not exactly either, but certainly a bit of both. Programming is hard primarily because it is so poorly understood. The entire field is in its infancy. Comparing it to art, I'm pretty sure we haven't even reached the "stick figures scrawled on a cave wall" stage yet. As Alan Kay pointed out we sure didn't invent an arch yet: http://squab.no-ip.com/collab/uploads/61/IsSoftwareEngineeri...

It's nice to see the Eve team trying to do something at least slightly different from the same-old, same-old. Even if it looks a lot like some horrors of yore (FoxPro) when I squint.

But I also think there's a lot of value in continuing to try new things. This complexity is so expensive that the payoff from a successful project could be enormous. And even when projects fail, we can often learn a lot from understanding how they fail. See that recent HN discussion on literate programming for an interesting example.

If you can narrow the problem space to some finite number of goals and workflows, you have a viable application. Application environments with sufficiently many workflows always reincorporate programming as a way to let the user build their own solution - and most things are implicitly programmable, whether through a defined API or through tricky reverse-engineered methods.

What I think muddies the picture is the line between "design" workflows and "engineering" workflows - in the first, you're piecing together the existing technology in a different way, while in the second, you are transferring math and science knowledge into original technology. As individuals we experience personal bias as to which side of programming is more "necessary," which is reflected in the resulting choices of tooling, code style, and preferred problem domains.

Sometimes you want design-heavy programming - e.g., you add some business logic and a UI on top of a database. Other times you want to add engineering to an existing design - you're writing hardware drivers using a common protocol, a data formatting plugin for an application, etc.

Library code acts as a way to expose units of engineering, while a framework defines a broad, but still configurable design space. Sometimes you have overlapping design spaces - you can have client code that works with a GUI framework, but also talks to an internal model and remote data sources.

One of the things that is exciting about programming's evolution is how much it is based on an ecosystem of technologies. Outside of some embedded fields, the era where you are given a hardware manual and are told to come up with your own development environment is over. Successful technologies tend to act parasitically on prior ones. This leads to a lot of compromises, but the general direction remains toward "better fit."

http://incidentalcomplexity.com/

:)

I find it really useful to use Ruby's built-in CSV library to process data for spreadsheets then visualize it with a graph in Numbers. I pull data from copy-paste tables, extract from Sqlite databases embedded in applications, etc. But for something like scanning Facebook friends as they suggested I'd have to first figure out the API or how to scrape the data. Once I figured that out it probably wouldn't be too hard to write a script for it. I could probably even put it in a crontab (LaunchAgent plist on a Mac).

Anyway, I think there's a huge opportunity to empower more people who are inquisitive and maybe somewhat technical but are not programmers. Make it easy for these people to solve problems rather than trying to teach them to "program". The end goal in my mind is the StarTrek programming model where you discuss with the computer what you want to do to solve a problem. Those crew members are scientists and engineers but only a few of them are "programmers". (StarTrek is a useful yardstick because we seem to like copying technology out of it. E.g. StarTrek communicator which is the late 90s cell phone and StarTrek PADD which is the SmartPhone/tablet.)

We can come up with better and better formal languages and ways for editing these languages (and probably should), but I would guess even a visual formal language, will still pose many of the same difficulties for users that existing formal languages do.

2. Why denigrate group ware?

While this may not succeed, I have little reason to believe future improvements will all be evolutionary (I know you hedged there, saying "probably"). A lot can be learned from shedding assumptions. If the result of the project is inspiration for building tools on top of traditional programming models, the originators may be disappointed, but I'd call that success in basic research.

Probably, yes -- but working on a problem starting from the beginning is something more people should do.

It is riskier, for sure. And I am glad their team is doing it.

There is nothing technically difficult about these problems; and it's work that's really more suited to a business analyst anyway. While developers should obviously check their logic to make sure it's sound; gathering the requirements is 90% of the work in these situations.

Anyway, that market seems a lot easier to compete in than the straight up dev tools market. Dev tools are so personalized, with every person/team/project/company having different needs and requirements that it seems like the only way to succeed would be with a niche product (which naturally limits the scale of your success).

I do not believe it is reasonable to draw that broad conclusion based on one example