I'm not allowed to forget a lot of this pain. I teach programming, so I see it anew every semester.

And of course I still experience much of it in my own work. So, yes, let's deal with these issues better.

One little disagreement. In the "What?" and "Why?" sections the writer present some ideas for debuggers. While these are good ideas, I prefer to think of the inadequacies of existing debuggers as motivation for good practices: modularity, loose coupling, unit testing, etc. Certainly, it would be nice (say) to be able to examine the entire state of a running program easily. But I would rather code in such a way that I do not need to.

So to those who would write the World's Greatest Debugger, I say, "Good for you." But even better would be to turn your efforts to producing languages, libraries, frameworks, and programming environments that make such a debugger superfluous.

But if you could examine the entire state, why wouldn't you? People build correct intuitions by comparing their mental model to reality. If you don't ever show see reality, or only peek at it through sprinkled print statements, then it becomes harder to build an accurate mental model. This applies whether we are talking about a beginner trying to understand how pointers work or an expert grappling with a complex system.

Programming is always limited by our ability to comprehend complexity. So enabling simpler systems is a win, sure, but so is getting better at understanding things.

Well, I would, of course. As I said, I think his debugger ideas are good ones.

OTOH, there is a limited amount of effort available for improving programming, and I would prefer that such effort be directed toward other ends.

Part of the problem is that, in my experience, debuggers tend to encourage sloppy thinking. And the better the debugger is, the sloppier is the result. Using a debugger, together with thinking hard about a problem, can be a powerful combination. But all too often, I see debuggers used as substitutes for thinking hard; the way debuggers are designed seems to encourage this.

To put it in the terms you used, constructing a mental model and comparing it with what the debugger shows, is a great idea. But in practice, I'm faced with a choice: work on my mental model now or fire up the debugger now. The latter is less taxing, and is often chosen; this is likely harmful in the long run.

I wish there was a device that would help me think less hard...oh wait, there is! Its called a computer!

The lack of a decent debugging experience in languages like Haskell mandates deep thinking upfront along with powerful bondage-style type systems. I can imagine that a language with a more advanced debugger would push things the other way.

[1] I couldn't entirely parse your sentence as posted, so please forgive me if I misunderstood it.

Oh, and don't worry, I would never pull the I'm offended by X card! I don't get offended easily, and I don't think you do either ;). This is all friendly argumentativeness (is that a word?) as far as I'm concerned, I hope you feel the same.

Equational reasoning is useful but it is still reasoning that I have to do. The whole point of having a computer is for it to do some of the thinking for me.

It is not the learning curve that gets me; I know many languages already, some of them with very steep curves. It is the thought processes that the language promotes, the order that it forces me to write my programs, the flow doesn't suit me well. I want the freedom to work at my own rhythm.

[1] http://www.cs.ox.ac.uk/ralf.hinze/WG2.8/

I love paranoid type systems: they debug my thinking up front.

I've never needed to read a paper to understand any other language. If I need one to learn Haskell, that's an immediate downside.

Haskell isn't dependently typed, at least not yet.

For example, I am trying to make my first Rails app outside a tutorial.

Okay, I want to use bootstrap. Should I use gem? Guess that's the ruby way. Okay seems like there a few of them. Tried one, another one. Doesn't work. Don't know why, since I am just copying half-cryptic stuff because I am new to Rails.

A friend suggests to use Bower. It's easier. Right! I had totally forgotten about bower, it rocks! Google: bower rails. Okay, there is that thing sprockets which I apparently need to configure. Googled, a few blog posts opened, they offer a bit different advice each, let's try. Doesn't work. Nope. Not really... Google: bower rails bootstrap sprockets. And yay! Works.

Two hours later, I have included Bootstrap. Properly.

Real disadvantages such as updating becomes so boring that you will probably never do that. And 'bower search <something>' followed by 'bower install <found it!>' is less distracting than googling and unzipping etc.

Being lazy pays off.

Of course, in a narrow sense, this means such a person isn't really designing it at all: the real engineers are, which may cause resentment. There is a very good chance that a non-programmer can design a programming IDE that is two or three orders of magnitude better (by whatever standard) than the status quo. This means such a person can't actually implement any part of it, or even know exactly what it's doing.

Quite a surprising conclusion.

By the way I have experienced this myself, when designing for a target I didn't know yet: after/while I was learning it, the resulting design iteration process was much worse than when I didn't know the implementation details. It's harder to think from the user's perspective, after you have been forced to think from the implementation's perspective.

So if building the application engenders learning x, then you might totally miss this design problem, unless you step away long enough to forget x again yourself - and that may never happen.

It is a bit too much to ask for one person to learn all of the details and totally own the implementation space, and yet be able to instantly forget all those details and go back to when they didn't know anything, to look at things from the user's perspective.

Why does it take teams of developers to create and manage applications which do really simple tasks in the grand scheme of things (and I realize the amount of complexity in building applications is staggering, but large portions of it could be better automated). Where are the auto generated GUIs, where is the ability to ship execution control to arbitrary devices, where is a hypermedia layer with independent view and presentation code?

I'm approaching this from a different angle than the light table guys appear to be (I agree with everything they are saying). My angle is an attempt to build a cross platform module system (where platform includes runtime and programming language as well operating system and architecture): https://github.com/OffByOneStudios/massive-dangerzone

My argument is: before we can build the next generation of useful tools, we need a framework for managing both generations. massive-dangerzone is an attempt at bringing next generation usage (like that described in the article) to existing tools. It's still a big work in progress though, and is barely useful at the moment.

Do people even remember what life was like before Google?

I type 'email python' and get back a link to an email module. Am I being closed-minded in thinking that can't get much easier?

Yeah, I need to understand a bit as to how email works, smtp and imap/pop and what not, and how to send vs receive email, but some level of understanding is just necessary.

With package management systems standardized within a language (such as Go) the problem domain is reduced, but extending the solution across multiple languages sounds like another third-party tool than a language feature.

For example, if I see that a repo hasn't been touched in 3 years, I'm probably not going to pull it into my project. If the setup required to use a library looks clunky, I'm likely to avoid it. If the documentation is dense and difficult to parse, I'd much prefer an alternative.

These are the problems that I don't see an automatic package-fetching IDE solving anytime soon. Not that it's beyond our capacity as developers to solve such problems, but because I doubt that the number (hundreds, easily) of hours it would take to develop such a solution would be justified the 10 minutes it might save a developer looking for a library. Also, it will be quite a while before the general faith in an IDE's ability to pick a good library for us is high enough that we'd trust it with such a responsibility. A mistake would easily cost more than the 10 minutes it saved, and given that computers can't even pick the right music for me with 100% accuracy, I don't see something like this being a reality anytime soon; and I'm fine with that. I'd rather we focus on solving real problems.

http://techcrunch.com/2014/02/17/microsoft-launches-smart-vi...

http://visualstudiogallery.msdn.microsoft.com/a1166718-a2d9-...

I postulate the problem domain changes. I don't have experience with Go's package management system or NuGet, but I do with PHP's Packagist (Composer packages).

Finding packages is easy; evaluating their quality, if they do what they claim, and how buggy they are.. not so much. The problem's shifted to become one of too many libraries for any task, most of which are a liability in any project.

It certainly could be streamlined. To quote the preceding paragraph:

> For common tasks google will probably find you entire code samples or at the very least some javadocs. The samples will be missing lots of implicit information such as how to install the necessary libraries and how to deal with missing dependencies and version conflicts. Transcribing and modifying the examples may lead to bugs that suck up time. It's not terrible, mostly thanks to sites like stackoverflow, but it's still a lot of unnecessary distractions from the task at hand.

cough smalltalk, emacs, etc. cough :)

"The samples will be missing lots of implicit information such as how to install the necessary libraries and how to deal with missing dependencies and version conflicts. Transcribing and modifying the examples may lead to bugs that suck up time. It's not terrible, mostly thanks to sites like stackoverflow, but it's still a lot of unnecessary distractions from the task at hand."

So many times, the actual programming isn't tough, it's just getting all of the stuff around it setup that is hard.

I co-developed the initial version of Verse with Eskil; I think it was a bit before its time perhaps. It was hard to get real traction for it, but at least the things Eskil has gone on to build on top have gotten some real attention. Great, and very well deserved!

Verse was a huge inspiration for me and it's a shame to see it largely ignored. We're exploring similar ideas for getting our language tooling away from plain text and towards structured, networked representations. I would love to hear your thoughts on how to do that, or any advice you have from building Verse - jamie@scattered-thoughts.net

EDIT I also thinks Eskil's ideas on 'what we can do' vs 'what we can get done' applies just as well to the rest of the programming industry. We're so focused on scaling up to millions of lines of code that we end up with environments where lunch_app takes a week to build and deploy.

What's hard -- and often impossible -- is fixing those big problems, because a lot of times they're genuinely intractable; and when they're not, they're often so difficult that they might as well be.

So just sitting around and complaining about them sounds insightful, but it doesn't really get anything done. And yeah, I know that they're allegedly working on "fixes" for these issues, but based on the track record so far (LightTable promised all sorts of revolutionary views of what an IDE could be; it's delivered... not a whole lot of revolution), I don't have any faith that Aurora is going to amount to much either.

And I don't want to be too negative, because sometimes a previously-intractable problem turns out to now be tractable, and it takes someone who was willing to question long-accepted pain to find that out. So I'd be pleasantly surprised if this project delivered something that had even as much effect as the development of, say, git or xunit. But I'm not holding my breath.

IMHO, all these things went down to the bottom of history because things like these:

> Anon the intern needs to be able to open up Programmingâ„¢ and click 'New Web Form'

are adequate for people who usually don't program, and extremely inadequate for people who usually do. Generally, and for good reasons, programmers will dislike a tool that hides implementation details for ease of operation. Past a certain level of complexity, the time spent manually doing the right cruft becomes significantly smaller than the time spent manually cleaning up after a smart tool.

I sympathize with Anon the intern, but perhaps he should rethink his expectations about complexity; if discoverability is a problem, perhaps he could switch to something that's better documented?

And at the risk of sounding like an elitist schmuck who rants about how things were back in his day, maybe he ought to start with something other than web programming. The size and complexity of that tech stack is humongous, to the extent that a large proportion of those who use it don't understand it more than two layers of abstraction down. Programs are also hard to pack and the environment that runs them is hard to setup. Because it involves at least two servers, possibly with several add-ons in order to allow the server-side languages to run, learning at least three languages (assuming server-side JS is an option), two of which (HTML and CSS) aren't quite being used for their original purpose. This is a beginner's nightmare and it has exactly nothing to do with the development tools.

And then there are things that are far harder to solve than they originally seem:

> I want to just type 'email' and see a list of functions and libraries relating to email.

Related how :-)? Should MIME-related functions, needed to reason about attachments, also come up here? HTML parsing/converting, in case you need to deal with HTML email? Information cluttering does nothing to alleviate the opposite problem of information breadth: if Anon the intern's problem is he doesn't know how to Google for libraries or how to make efficient use of documentation, an IDE that presents him with a gazillion of possibly related things won't help him. Especially when, like all beginning programmers, one of his main difficulties is correctly defining the problem he's working on which, in turn, makes it likely for the solutions presented by the IDE to be nowhere even close to the one he needs, because the IDE (like Anon himself) thinks Anon is trying to solve another problem.

There is, on the other hand, a lot more truth in this:

> Tightening the feedback loop between writing code and seeing the results reduces the damage caused by wrong assumptions, lightens the cognitive load of tracking what should be happening and helps build accurate mental models of the system.

I do think that the real resolution to this problem is writing simpler programs whose state is easier to track. On the other hand, programming tools today suck considerably at presenting program meaning. Things like evaluating what an expression comprising entirely of constants, or at least evaluating it based on the default values of the variables involved, are well within reach for today's tools, and yet programmers' calculators are still employed because 99% of the available IDEs couldn't evaluate ADDR_MASK & IO_SEGMENT if the life of every kid in Africa depended on it.

This is wicked cool: http://repository.cmu.edu/cgi/viewcontent.cgi?article=1165&c... . However, I also find myself thinking that the very fact that we need debuggers that are this smart is proof enough that we don't reason about our programs well enough. Except for the fringe case of having to quickly debug a (possibly horrible) codebase I haven't written, I'd much rather prefer being good enough a programmer to avoid the need for a debugger that can tell me why foo is not 8 despite the fact that I fucking said foo = 8 ten lines above, than being a programmer with good enough tools to help me when I'm stupid.

[0] And this is just my opinion because I have no actual rigorous research to back this up. At least I can admit that, contrary to most of the people who are agitating for approach A or B to software development.

EDIT: s/studies/rigorous research/. "Study" can mean anything from "I though about it for 5 minutes" to "rigourous randomised trials", so I shouldn't use that term.

You might find https://www.cs.utexas.edu/users/EWD/transcriptions/EWD10xx/E... an interesting read, if you haven't read it before. There's been some interesting discussion of it on HN in the past, too.

I am strongly in favour of mathematical methods but that doesn't preclude exploratory work or visualisation. The idea that mathematicians spend all their time constructing rigorous theories and proofs is a complete illusion. It is common practice (outside of the Bourbaki school, at least) to alternate between building intuition through examples and solidifying intuition through rigorous proofs. See eg http://terrytao.wordpress.com/career-advice/there%E2%80%99s-... (by Terence Tao, one of the most talented mathematicians alive).

By comparison, we are extraordinarily lucky in programming in that everything we deal with is computable and representable. It is far easier for us to display examples and visualise systems. So while I agree that more understanding of mathematics and proof would aid in programming complex systems and I have certainly found that my degree in maths was a good preparation for programming, I don't see why we shouldn't also take advantage of the power of computers to help build the intuition that is necessary for rigour.

What's meant by reliability?

Additionally, frankly, "reliable software" isn't something that's needed at least 99 times out of 100. Software that more-or-less gets some specific task done is. Those tasks can change from day to day, and generally require a few tries before the person who gave the task is even sure what they want, so RAD is more of a necessity than spec-based development.

Basically, we now have this entire class of information workers, and very few tools to give them that are any better than an Excel spreadsheet for the most part. I don't see the article as trying to give traditional programmers new tools, it's trying to give information workers new tools, and probably traditional programmers will find some of those useful.

That puts it much more clearly than I managed. Thanks :)

At the very least the modding system required some abstract analysis to account for trolls and such.

Nice concise mathematical specs are rare in practice, not because programmers are incapable of expressing such specs (and most probably aren't), but most programming problems are not expressible rigorously via math; they might not even be amenable to much analysis (so called "wicked" problems are common in programming).

Someone might say "but Haskell itself doesn't solve practical programming problems", or whatever. The point isn't Haskell or pure functional programming per se; it's just an example of things that easily lend themselves to formal specs. Whether more complicated, perhaps effectful programs lend themselves to formal specs (I think ATS might be able to express such specifications) is more hard to answer.

Spec xor exploratory programming. The codebase is the spec.

  > Anon the intern needs to be able to open up Programmingâ„¢ and click 'New Web Form'
  
  are adequate for people who usually don't program, and extremely inadequate
  for people who usually do.

I think the best solution for this is the way Go does it. It takes the best of both worlds.

  http.ListenAndServe(":8080", nil)

The problem is when you press 'Start Web Server' or 'New Web Form' and it generates 300+ lines of boilerplate code that you now have to look at, maintain, and waste your attention on. Advanced users don't want that, and novice users aren't happy about it either.

Instead, take high level things and make them available via high level APIs, that are also extensible and flexible to allow tweaking of what you want [1], if/when you need to.

[1] https://github.com/golang/gddo/pull/143#issuecomment-3194318...

It includes things like a source browser that allows looking up ways to deal with eg: email, if you have the/some source code, you can easily run it, a built in debugger and a tight write/test/fix-loop etc.

Chris Hancock's dissertation [1] is really good at making this point:

> And yet, if making a live programming environment were simply a matter of adding "continuous feedback," there would surely be many more live programming languages than we now have. As a thought experiment, imagine taking Logo (or Java, or BASIC, or LISP) as is, and attempting to build a live programming environment around it. Some parts of the problem are hard but solvable, e.g. keeping track of which pieces of code have been successfully parsed and are ready to run, and which ones haven’t. Others just don’t make sense: what does it mean for a line of Java or C, say a = a + 1, to start working as soon as I put it in? Perhaps I could test the line right away—many programming environments allow that. But the line doesn't make much sense in isolation, and in the context of the whole program, it probably isn't time to run it. In a sense, even if the programming environment is live, the code itself is not

i.e. Smalltalk code simply isn't live.

[1] http://llk.media.mit.edu/papers/ch-phd.pdf

Or are you talking about some strict old Smalltalk-80?

While you could change a parameter like color in some text-representation of the live code, wouldn't it be more natural (in Smalltalk) to simply change the object [ed: object instance] in question (and see the immediate change)?

I'm considering things like:

http://pharo.gforge.inria.fr/PBE1/PBE1ch7.html#x32-111006r22...

If Smalltalk allows editing values of variables of running code, and hot code loading -- what is missing? Is it the fact that other systems define the source code as "the truth", and consider the byte/compiled code to be a "secondary artefact" that causes (my) confusion?

Because as far as I can tell, Smalltalk does allow you to manipulate the objects that make up your system live?

The problem with editing the program's objects directly is that it is not very powerful in its capability: fine if you want w to always be red, but not fine if their are a hundred w's to edit individually, or if w is red because of some kind of active condition. In any case, hot code replacement doesn't really help either; its useful for sure, but the next step is to change the program's state retroactively [1] to match the new code. Smalltalk never went there, and probably couldn't given the hardware constraints of the time. But now, I think we can go there.

The Pharo folks are claiming that their system supports live programming capabilities, but it is just the same old live object/fix-and-continue features that Smalltalk has had for ages. I suspect that they just really don't know what live programming means, and are latching onto the word because its "trendy."

[1] http://dynamicaspects.org/blog/2012/08/15/changing-the-past-...

If we look at what is possible, namely changing data and instructions in ram -- it seems to me that Smalltalk already provides that.

I suppose there is some value to viewing a program as a kind of differential equation over time (where we assume the future has not yet happened, and all input has been given at certain points in time) -- and so be able to change the state at a given present, by changing the program, and accounting for how that change would translate to a different state in the present, given the same inputs... But we cannot stop time, nor the real world -- so most real systems can only be approximated in this way.

The best we can do (with any real system that isn't isolated from the outside) is to take a snapshot at a given time t, make changes to see how that state could have been at t with our new code -- but we then would have to bring our system back into the present -- either by playing back recorded input or by some other means...

I don't think your points about w manage to put weight behind (or communicate) your point: in Smalltalk you could either programaticaly alter all ws, or if you changed the condition on which w depended, all ws would update (in a GUI, or other system with an event loop...). I get that that's not your point though.

Anyways, this an active research topic...see:

http://research.microsoft.com/pubs/211297/managedtime.pdf

If you haven't yet.

[1] http://www.opencobalt.org/about/history

[2] I think the paper I read is this one -- but I can't verify it from here:

"Designing croquet's TeaTime: a real-time, temporal environment for active object cooperation", David P. Reed http://dl.acm.org/citation.cfm?id=1094861

See also: http://www.opencobalt.org/about/synchronization-architecture

"Implementing atomic actions on decentralized data" http://dl.acm.org/citation.cfm?id=357355

Again, the goal here is not live programming, but the "boxing" of blocks of code into "atomic actions" feels very similar to the MS paper.

[edit: Some interesting parallels with the earlier story on Soundcloud's Roshi system too https://news.ycombinator.com/item?id=7732696 ]

"In the Smalltalk model, for instance, state is persistent and code changes don't affect data. In the Clojure model, code is "mostly functional", with a small amount of carefully-managed state. Either model could be a starting point for a system where continuous code changes can be seen as continuous effects." -- Bret Victor http://worrydream.com/LearnableProgramming/

i.e. live programming isn't programming. It's just a toy now.

> * storing code in a networked database with version control and realtime sync

This sounds a lot like Monticello and/or what Lively Kernel does over Webdav?

> * a structured editor to enable rich ASTs with unique UUIDs

I'd say something similar is achieved by Smalltalk with it's view that there is no "source code" -- there is only the compiled code that is also viewable/editable as text (this is not so much about code, but a feature of a truly object oriented system).

> * managing environments declaratively so that evaluating code is always safe

I don't see how "evaluating code" can "always" be safe? Or is this similar to having the ability to deploy a separate dev-image in Smalltalk terms?

> * a uniform (logical) data model where every piece of state is globally addressable

One example of this would be an object graph?

> * a model for change that tracks history and causality

I take it this is the "live programming" bit discussed alongside here. If all input and output is via messages, it would seem feasible to simply log messages (possibly collapse some messages (ie: +1, -1, +1, -1, +1 becomes just "+1").

> * a powerful query language that can be used for querying code, runtime state, causal graphs, profiling data etc

I'm not entirely convinced we need a "powerful" query language. But something to specify context (show me all counters in the date widget) would be good -- but it would be fine to express it as "in:someDateModule var:count". Maybe I just read "powerful query language" differently than the author (intended).

> * composable gui tools with transparent guts

We seem to be reinventing these forever.

> * a smooth interface to the old world so we don't end up sharing a grave with smalltalk

I really didn't want to go all "Smalltalk did that" (I'm not even sure it did I only know new-ish Smalltalks) -- but I think the eco-system is in the process of proving the death-by-different wrong: We use Apps on phones and through the web -- and the fact that eg: Google spreadsheets has pretty a pretty crappy interface with "the old world" doesn't seem to bother anyone. I mean, yeah, we know that we probably need to interface with some form of external file system -- but now we can pretty much just throw http/dav at the problem.

I guess I'm just grumpy -- arguably there's no difference between running a javascript vm on bare metal vs running a Smalltalk system on bare metal (except for the language part, but we can implement the language(s) we need on top of js anyway…). I just can't help but feel we might do better with a system better designed to be a system, rather than the web browser that accidentally became a vm and virtualized display driver. It doesn't strike me as likely that we'll see a proper security model that actually works for javascript in the near future, for example.

It's a shame that it's not being said more clearly and explicitly in the post; but that 'model for change' implies immutable/persistent data structures and all that comes along with that.

The foundational assumption that data is never deleted or updated but just garbage collected does really change a lot.

Essentially (I'm hoping!) they're just saying: let's build another Smalltalk but with persistent data structures and reactive programming instead of Smalltalk's MVC model.

But yeah; it does sound like they need to study the Smalltalk ecosystem a bit better to figure out that apart from those two things Smalltalk did already do pretty much everything they want to be doing here; or find a marketing angle that's less off putting to those that know Smalltalk well...

https://news.ycombinator.com/item?id=7762535

Visual Basic was phenomenally successful as an end-user programming tool. There are entire businesses that still run off VB6. It worked because they nailed the early learning curve. It doesn't matter if your language is faster or more maintainable or less bug-ridden and insane - the majority of people will always gravitate towards the tool that is easy to get started with. This is why VB6 and PHP are so widespread.

The moral of the story is that if you want people to program in a sane and scalable way you must make it the path of least resistance. No amount of advertising or education has ever been able to overcome the fact that most people don't give a shit about programming and are just trying to get stuff done.

> if Anon the intern's problem is he doesn't know how to Google for libraries

Anon the intern's problem is that the information returned by google is incomplete and requires manual effort. I don't want to replace stackoverflow, I want to make it executable.

> I do think that the real resolution to this problem is writing simpler programs whose state is easier to track.

Well, that's where the managed state part comes in. We are trying to separate essential state which is actually part of the program logic from accidental state which is just an implementation detail (an idea expounded in http://shaffner.us/cs/papers/tarpit.pdf). That's a whole other post though.

> I'd much rather prefer being good enough a programmer to avoid the need for a debugger that can tell me why foo is not 8 despite the fact that I fucking said foo = 8 ten lines above, than being a programmer with good enough tools to help me when I'm stupid.

There is an implicit assertion here that not having a debugger makes you a smarter programmer. Would removing other feedback print statements, unit tests and the repl help too? Why even execute code at all, just print it out and think hard.

Thinking hard doesn't work if part of your mental model is wrong. Being human (unfortunately) my mental model is often wrong and a quick glance at a stack trace does more to enable correct thinking than hours of working the problem out by hand with pen and paper.

The whole point of having a computer is that it can do some of the thinking for you.

> maybe he ought to start with something other than web programming

The goal is not for Anon to learn to program, it's to build lunch_app. The web sucks but it is the only good distribution platform we have. Either we use that or we build something easier on top of it.

If anon wants to build software then learning how to should not be optional. The obvious alternative is to buy software, just as we buy most other things (cars, books, bread, etc).

Simplifying the parts you mention does nothing to simplify the important things in software development, which, as VB has demonstrated, leads to two things: crappy software and people who know how to create a barely working program, but not how to structure it to be maintainable or to prevent data loss in somewhat unexpected situations.

Was? So what happenened, where did it go, why isn't it still filing the role you are aiming at?

Another example, held in somewhat better regard by programmers, of a phenomenally successful tool that let end-users build working programs, is Hypercard. Which definitely isn't around anymore.

Looking at the prior art of tools that have succeeded at letting end-users build programs is probably a good bet. What they did right, what they did wrong, why they don't exist anymore even though they were succesful, if they don't exist anymore.

The last proper version of Visual Basic was released in 1998 -- Microsoft discontinued it in favor of VB.NET. If they continued to develop VB, I think it would still be a heavily used platform.

Hypercard is not the same thing. People didn't build real applications in Hypercard -- it was a toy. I spent a lot of time playing with Hypercard and it was very VB-like when programming for it but the UI was more like Flash. It's best use was making simple games.

The real problem with simple tools is that they solve simple problems. Once you have to start doing something outside of the tools range, you're stuck. This happens with every tool but more complex tools have a bigger range.

Remarkably, they did! Wikipedia:

>A number of commercial software products were created in HyperCard, most notably the original version of the interactive game narrative Myst, the Voyager Company's Expanded Books, and multimedia CD-ROMs of Beethoven's Ninth Symphony CD-ROM, the Beatles' A Hard Day's Night, and the Voyager MacBeth.

It's true that Hypercard was not an ideal platform for development. Apparently, Myst developers had trouble making the high resolution images load quickly on machines of the time.

I wrote several commercial software products in HyperCard and SuperCard back then. It provided awesome tooling leverage. I worked on an EDI application that was written in SuperCard (with the EDI processing running in a C plugin) which was working, but later the company's president decided we needed to re-write the whole thing in C++ with the MacApp framework (http://en.wikipedia.org/wiki/MacApp) because C++ was a "real" programming language. This was the cfront era -- build times were long, debugging meant manually de-mangling names, etc. Crazy slow. We spent a long time on it but the schedule slipped long past the end of my time there.

The common knowledge about why Apple never supported HyperCard whole-heartedly is that they didn't know what it was - it wasn't a programming environment, it wasn't a database, it wasn't a paint program, it wasn't a multimedia environment. Yet it was still all of the above.

MS killed it in favour of a 'real' programming language. See eg http://msdn.microsoft.com/en-us/magazine/jj133828.aspx

> Looking at the prior art of tools that have succeeded at letting end-users build programs is probably a good bet

Indeed. We have a 50 odd page google doc full of exactly that :)

VB6 still runs an astounding amount of apps that are used by millions of people every day.

I know you're not implying anything, but, there's several hundred billions of dollars done in the US annually with just VB6, excel macros, and shitty Access DBs. It's incredible, really.

Another humungous company (think 10.000 employees) has its core system in VB6 (and it works REALLY WELL).

The company I work for has > 80% of its programs in VB6.

It didn't go anywhere. I'm not sure what the spiritual successor is, though.

I made this lengthy introduction because I really like LightTable. What you folks are doing is impressive! I'm not particularly fond of the tech stack behind it but it's the first significant, organized kick in the tools' butt I've seen for many years now that isn't confined to academia.

> There is an implicit assertion here that not having a debugger makes you a smarter programmer.

The implicit assumption I'm making if that I were a smart (and careful!) enough programmer, I wouldn't need a debugger :-).

So yeah, if I had the choice between being so good that I don't need a debugger and having a really, really good debugger, I'd pick the former any time. Sadly, that's not... quite a choice, so I'll grumpily let gdb tag along.

> The whole point of having a computer is that it can do some of the thinking for you.

This is something to be enthusiastic about, certainly! I was glancing over Moseley and Marks' paper (I read it a while ago, but not too carefully) and, now that I think about it, the mantra about writing simple programs is held in highest esteem by communities that traditionally had to struggle with their tools (Unix, because 1970s, and more recently the web folks, because they were building complex web applications long before JavaScript had anything that would even resemble a reasonable debugger). Much of this was done so as to eschew the limitations of the platforms themselves. PDP-11/20 had 64 KB of addressable RAM; the fact that you could write and compile programs on it alone was remarkable. Letting the computer take some of the thinking load was quite certainly impossible.

I'm still debating with myself whether allowing the computer to do some of the thinking for me would be good for the quality of my programs. I'm a very clingy father :).

> The goal is not for Anon to learn to program, it's to build lunch_app.

I don't wish to sound rude, but this sounds like a lump made by marketing types :-). Is lunch_app a program? If so, Anon has to learn at least the basics of programming, much like someone who builds a desk has to learn at least the basics of carpentry or someone who wants to learn to play a musical instrument has to learn at least the basics of music.

More helpful tools will definitely help Anon, but there are things no tools will, or should, do for him: he will still have to understand how web applications are served, how they are hosted on the same server along with other applications and so on. Not knowing these things will be a lot more detrimental to everyone using his application than spending a couple of weeks learning all this boring stuff will be to Anon.

Isn't this the destiny of most software☨, and another step along the craft to utility curve [0]? Software of this type is needed to do a chore, and honestly if the tools are in place, it could be re-created in an afternoon rather than extended when the time comes.

I would use the analogy of printing; we used to need a typesetter and go to a printer; now we select a font, type and hit 'print' and - if the damned printer on our desk cooperates - we get our work back. Websites are reaching this level of utility; there's no reason simple web apps or desktop software won't too.

[0] This graph http://1.bp.blogspot.com/-lrxB3ntRzKQ/U05ClSIXwGI/AAAAAAAAEy... taken from http://blog.gardeviance.org/2012/06/pioneers-settlers-and-to.... The graph matches what I see in the web industry; his blog is a fun read

☨ Because thinking of all software ever written, including scripting for automation, the really complicated and large and needing well-engineered software is far less common than we sometimes think, and not what someone like Anon is considering building with these tools.

I agree that it's supposed to do a chore, so it should be as easy to churn it out as possible. However, if we were to follow your printer analogy, the proper way to do that with a printer would be to just build a machine that makes it very easy to create a custom printer that types out exactly what you want it to type out! What we have instead is a very general typesetter that can be convinced to print a particular picture (by feeding it an automatically-generated program in e.g. PS!).

The application in question is pretty much a simple CRUD that generates a report in the end so that whoever is ordering knows what to order and from where. IMHO, just like printing doesn't require the user to know PS, this should not involve programming. It's not that programming should be easy, it's just that the CRUD model is so easily understood and so orthogonal that building one should not require anything more than a wizard asking you about the data it should collect.

(Which is a great thing for carpenters and furniture store owners. We're those people in the analogy, so why get uppity about people needing our services eventually?)

No, seriously. IKEA didn't have a store open around here when I was a kid; some of the less pretentious furniture in my parents' house was made by my father, and since I was a kid, he had no choice but to need my assistance. Putting wooden things together with screws based on drawings was literally the first thing I learned about carpentry, just before sawing and all that.

The fact that it's so easy should be no surprise. Patching together some stuff in BASIC is exactly as easy.

Yeah, it's fucking disgusting that today's "personal computers" are such impersonally cluttered, patchy, uselessly complex designs that even experienced programmers can barely wield them.

And then if you really care (and this is where the non-text/denser ASTs come in; as well as a deeper understanding of computer science and programming) you can add annotations to your description of the data structure like "I want reverse traversal of ordering Alpha to be O(n)", annotations could even cause a compile error like, "I want to insert via index Beta in O(1) time, with an O(n) in order traversal time of Beta". And even fine grained annotations like "For index Beta use a red-black tree" or more specifically "To implement index Beta use module 'foo.bar.baz' by 'FancyBeansCorp', using class 'AwesomeEnterpriseIndex'"

Let the tools do the heavy lifting! Let data structure and systems programming experts write data structure implementation strategies. Hell, I bet you could build a marketplace of code just for optimized data structures, if it was easy to swap them in and out. Heck those sorts of dependencies should be at the company level, without even touching the project in question (a company level code policy to use a specific library of data structure implementation strategies, regardless of project being build (unless the project or user overrides it)). Let alone a market place of implementations for algorithms, general code patterns (conditionals, whiles), AI, networking, UI, etc.

Compilers should take a long time. A very long time. Sure we can short circuit them for debugging, and just use the best workable code we got in a few seconds, so we can test (and besides, with constant compilation, it should have already solved most of the problems reasonably well by the time you click run). Our compilers should be smart enough they can change their output for different machines not just at an assembly optimization level (better intrinsic, etc), but for different cache sizes, for different memory performance characteristics (changing between intensive memory data algorithm to intensive processor algorithm) as necessary. And yes, I expect it to be deterministic for every platform.

- It would be cool if these data structures could be transparently saved to disk, because data usually lives longer than code;

- It would be cool if they could be used by multiple applications, with some sort of guarantee that each application sees a consistent view of the data at any particular time;

- It would also be cool to have a declarative language for accessing several of these data structures at once, "joining" them on the values of certain fields and dynamically picking the best indices to use. Since you want a marketplace of implementations, it would make sense to standardize that language, so different data structure libraries can support it. A good name for it would be Structured Query Language, or SQL for short.

Tl;dr: You are completely right that network databases (object graphs and special-purpose data structures) are inferior to relational databases (declarative general-purpose data structures that separate querying and indexing, and allow a host of other general-purpose functionality). That idea has the potential to transform the whole software industry and lead to billion-dollar profits, as it has amply demonstrated since the 1970s. But since today's programmers are a forgetful lot, perhaps you could rebrand the idea and sell it to them as "Big Data 2.0" or something.

* Micro-optimizations. The implementations should be granular, not monolithic like SQL servers (let alone the fact that optimized SQL rarely translates between the implementations, as most optimization commands are not cross platform).

* Embeddable DSL, this one is honestly a problem with programming languages in general, but I've yet to see a static programming language which can tell me when my query is malformed at compile time (or hell, even provide decent syntax highlighting). Let alone importable, custom, extensions to SQL, or embedding lazy functions within the data structure.

* Intent vs Exact. The ability to describe partial intent, rather than exactly how to organize, the data (e.g. describing the operations expected to be executed across it, rather than the indexes to build). As well as having information available from the compiler about how the data structure is used (so that without the explicit declarations, the intent can still be optimized for).

SQL is a fine direction to approach the big picture problem from, but it isn't anywhere near the goal I care about. I'm not interested in dealing with big data. I'm just tired of having to know the difference between arcane implementation names for ordered sequences [1], I want to be able to describe what I want my ordered sequence to do, and let the compiler figure out the best implementation available.

[1] http://www.programcreek.com/2013/03/arraylist-vs-linkedlist-...

I have a lot to say about separating specification from implementation and trying to better capture programmer intent (eg http://www.freelists.org/post/luajit/Ramblings-on-languages-...) but the blog post was already pretty long.

There seems to be this dissonance between dynamic languages, which tend to be popular with new programmers for a variety of reasons, and better tools enabled by static analysis of (usually) static langues.

The static languages seem to present this "hump" that a lot of people are averse to, and if we really want these smarter tools, we're going to have to find a way to get programmers to use more static techniques; we need to get them over that "hump", possibly then these smarter tools would ease things in the long run, but doesn't have quite the same immediate-gratification aspect of dynamic languages due to the up-front effort.

Dynamic languages are preferred by those who don't want the conservative verbose static type system to get in the way of writing code. You can design a static type system that is less verbose (e.g. via more inference), but then it can become more conservative (H&M's inability to deal very well with semi-unification in the form of subtyping and assignment). That is the "hump."

Smarter "more magical" compilers are a nice idea in theory, but hard to realize in practice. There are even limits to the kind of analysis we can do dynamically, but they are a bit less constrained then what we can do statically.

It's probably not the dynamic language that is the problem, but its constructs.

The core issue is programming does not really focus on any one problem which is why there are no great general solutions. People forget that just because you call something a database does not mean SQL and noSQL solve interchangeable problems there just local maxima. For all lisps power and simplicity people don't actually use it to write a lot of embodied code.

PS: It might be helpful to consider to note in general case compression does not actually work. But, 7zip exists because we don't live in a generic world we live in this one. So, don't hand people a steam shovel and ask them to bake you a cake.

Could it not be possible to take this even further, to the point where I say "I want to do ..." and all of the "code," be it high or low level is generated for me.

I realize making something capable of this level of abstraction would be incredibly difficult, but it would certainly be fun to try.

In Uruguay, a tool called Genexus enjoys a high level of success promising something like that :) .

For simple CRUD apps, it's pretty much magical - and it looks awesome in demos, heck, it can generate code for every mobile phone out there just throwing around some controls, specifying your data fields and writing a few rules, but if you want to make it do something that isn't explicitly considered, it's very annoying (and it has some UI and UX problems). And don't ever try to touch the autogenerated code :P .

That said, it's probably the easiest way to get from 0 to a complete CRUD app, once you learn how it works.

http://www.genexus.com/products/genexus?en

Excited because I found myself agreeing with every point: isn't it obvious to everyone that the programming experience could be so much better? We created wonderful tools for graphical expression and number crunching, that keep getting better [1] [2], while our own day-to-day tools remain basically in a rut.

Disappointed because it seems LightTable is foregoing an incremental path to reach the goal, choosing the boil-the-ocean approach. We do need more long term start-from-scratch rethink everything kind of projects; like what VPRI's STEPS project aims to achieve [3]. But there is a lot that can be done to improve the programming experience today, and I don't see enough work in this area. IMO, the latest meaningful improvement in software development tooling was Intellij Idea around 2001 (arguably the functional programming renaissance represents another meaningful improvement, but the real breakthroughs there happened in the 70s). LightTable moving to the Blue plane leaves the PinkPlane unattended.

[1] http://www.adobe.com/technology/projects/content-aware-fill.... [2] https://www.youtube.com/watch?v=UccfqwwOCoY [3] http://www.viewpointsresearch.org/html/writings.php

There is an inherent tradeoff in creating any system: you can make the system very simple and it becomes too simplistic for the advanced users who want more control and detail. You allow too much control and detail and you alienate users who want simplicity.

A programming language/IDE is a system that affords advanced users the ability to solve an immense set of problems. That unrestrained ability comes at the expense of simplicity. The more constrained and specific that ability becomes, the simpler it can be (i.e. Excel is more constrained but far simpler than a Hadoop cluster)

Love this (although I don't like extra long lines). He's talking to you Pep8.

Arguably this wouldn't much of an improvement over the VB6 + Access status quo (or Visual FoxPro, or FileMaker Pro, etc), except that the individual components can be much more robust and scalable, monitored, auditable, and so on, without Anon Intern having to worry about any of that.

> There are probably dedicated apps that cover this particular example but we are more concerned with how an end-user would solve this kind of problem in general...

VB6, Access and Excel are exactly the kind of tools we seek to emulate on the usability front. Half of the world runs on Excel precisely because it is easy to get started, easy to understand and yet is capable of handling fairly complex simulations. If we can make something equally approachable but backed by a sane, maintainable and powerful language then we can give programming powers to a lot of people who need them.

Docker pretty much solves this problem. Writing samples for your new project? Start with step 1 `docker pull yourproj/your_tutorial_image`.

If I select a function from autocomplete, its dependencies should be automatically added to the project without any fuss.

For the point, that that post makes, this one is at least available if you use IntelliJ with Java or Scala.

Every file in a large project starts with dozens of redundant import lines. Why don't we have imports go the way of the inferred type?

What if tomorrow I add another library which implement his own foo()? (this is trickier)

When I read the code, what is foo()? (here the IDE may be smart and show me the right documentation).

All that said, I feel that the pythonic "Explicit is better than implicit" is better.

It's also true that when I see a file whose first 30 lines are used for including dependencies it doesn't sound right.

The problem lies with the fact that we use text to store structured data. If we were to save source code as binaries we could have the source code in a "file|object|item" and the dependencies in another item, linked to the source code (I'm not advocating this, source code in text files has many advantages, but also his drawbacks).

This has the neat side-effect that ids are globally unique so you can figure out what libraries you need to pull down just by looking up ids in your package repository.

That and the odd namespace clash that a compiler can't really resolve, but that is more due to implementation and system issues, not programming proper.

HOWEVER, I still think most everyone is missing the REAL issue here. As evidenced by the powerful tools cited in this article that address various aspects of the "programming" problem, there have been numerous efforts to move the state-of-the-art in software development forward. And to a great degree those efforts have _proven_ quite a few superior paradigms.

And yet, we haven't seen those new paradigms become truly mainstream for most programmers. Why? I do NOT believe it is because the approaches haven't integrated the right new concepts, because there are so many existing useful tools combining many different new ideas effectively which have failed to become mainstream among programmers.

I think many of those new approaches could and should have become normal operating mode for programmers.

I think the reason they did not is this: the core definition of "programming" (and by extension "software engineering" etc.) is an antagonistic and largely manual process of creating complex textual source code that can be used to describe system behavior. Period.

For example, if I want to call myself a web "developer" I DARE NOT use an interactive GUI tool to generate and maintain the source code for my web site. Web developers will always say, for example, they avoid this because the source code generated that way is less maintainable by hand. In many cases that may be true. In some cases with advanced code generation it is not. Regardless, I don't believe that is the actual reason. Its just a rationalization.

What do we call someone who's entire job entails creating a web page using a graphical user interface? In other words, this is a hypothetical person who has found a hypothetical GUI tool that can accommodate all of his web site design and implementation needs without any manual edits to source code. If he builds a web site or web application this way, and writes zero lines of code, do we refer to him as a very smart and advanced web "developer"? Or do we call him a web designer or simply a WordPress user (for example)?

We do NOT refer to him as a web "developer". He has no right to refer to himself as a "developer" or "programmer" because he has not wrestled through an antagonistic manual process to create complex textual source code. And that's what people are not understanding. The reason we don't call him a programmer is NOT because he didn't create an effective program or website. Its because the way he did it wasn't hard enough and doesn't match our outdated definition of what "programming" is.

To create a POPULAR system (among "real" "programmers") that makes programming more practical or easier using new paradigms, you must either redefine programming to include the possibility of new paradigms and a non-antagonistic process, or perhaps somehow trick programmers into thinking what they are doing is actually harder than it is. Maybe if there are a few places to type a command to generate source code, that will be sufficiently complex to still be considered "programming".

If you are too successful without doing those things then you will just have another tool that only "users" or "beginners" would ever admit to using.

But while the process of programming became easier, the role of the programmer did not. Instead, the programmer was expected to build bigger, more complex systems, and do it faster than before.

I would hesitate to call someone a programmer for just making a web form in HTML and sending it to a database, and a GUI tools to do this already exist, most notably Microsoft Access. But suppose a GUI tool was invented that allowed its user to create any website, from Google search to eBay and anything in between, including both the front- and back-end. Anyone well-versed in using that tool would be a programmer for sure.

Let's keep creating better ways to program. Future generations will be thankful for it, even if today's developers can't appreciate it.

http://www.lighttable.com/2014/03/27/toward-a-better-program...

The short version is yes, we are making a new language. The kind of workflow I discussed in the OP was the original goal for Light Table and it turns out that it can't just be hacked onto existing languages.

There will be another Light Table release in the next few weeks, featuring CodeMirror 4 (better performance, multiple cursors).

We've only been working on Aurora since February. Designing a new language takes time. We may show some early demos in the next few months, depending on how the current prototype works out.

Also, the fact that Aurora is intended to be simple and easy does not mean that we are building a toy. Quite the opposite - the intention is to remove some of the incidental complexity in programming to leave you free to focus on the actual problems. We intend to bootstrap the compiler and editor by way of dogfooding. I look forward to doing most of my programming in Aurora in the future.

The advantage of being incremental is that you can reliably produce something in a reasonable amount of time. This is not incremental, and it's quite risky; be patient.

(Sure, there's always room for more polish...)

Categorizing people into programmers and non-programmers neglects the people who need programming but not at an expert level. Some interns and students won't excel as programmers, but they still can bring programming into what they do instead, especially if efforts like Light Table succeed.

I suppose I shouldn't be surprised by this kind of pushback against reimagining programming — I've seen it as a response to some of Bret Victor's projects. Skepticism is great: Maybe the ideas behind Light Table are wrong or badly implemented. But we gain nothing from glib dismissals that seem to regard any attempt at improving or simplifying programming as beneath them.

There is progress to be made, and I'll be excited again when I see some.

I'm not always impressed by Light Table, but I appreciate the work the team has put into the problem and think they have good ideas.

[0]: http://www.chris-granger.com/2012/04/12/light-table---a-new-...

[0] http://www.evanmiller.org/dont-kill-math.html

I believe in the concept and potential; it's the execution that I'm finding fault with.