Check out Kakoune[1] for a different take on an editor’s scripting language. It offloads as much as possible to existing Unix tools such as bash, sort, fmt, etc. It uses a clever system of string substitutions that’s a bit unintuitive at first, but not too difficult to grasp quickly. I spent two weeks trying to fix a bug in a Vim plugin that ended up being a 3 line change. After that I felt like I barely understood just a small section of vimscript.
I ended up switching to Kakoune after trying it for a bit because I realized that in the two weeks of me exploring it I had entirely understood it’s “scripting language”, or at the very least understood how to use the onboard docs to write any plugin I wanted to.
Check this out [1]: kak script evaluating sh script which in turn call perl which constructs kak script commands and output them to stdout to be evaluated by kak.
Or this one [2]: same, as before, but now it's calling awk, which calls `date` binary (so it's kak → shell → awk → date → kak).
It seems that it's done this way because sh is not good programming language to use for manipulating text. Moreover, sh is not good programming language to write complicated programs, which later can be successfully maintained & tested.
For me, kak approach is past the border of unmaintainability.
Also, it raises question of why not to use embedded language like python or lua instead of creating such a limited editor-specific language.
To give a bit of context, the idea behind this design is that those scripts should ultimately only be glue scripts that convert from some editor agnostic tools into Kakoune commands.
I do agree that sh is not a nice programing language, and forking programs all the time is not very efficient. I think that there is a tradeoff between the different approaches, and Kakoune's has proven to make quick'n'dirty very easy while being suboptimal for complex tasks.
I am okay with that tradeoff because I still believe most complex tasks do not need to be editor specific, and should be provided by separate tools written in a more maintainable language.
Another reason is that I've been trying to design Kakoune in a minimalistic way, and one thing I tried to unify was the user and scripting interface, which means I needed a command language that was easy to write interactively (`edit filename` vs `edit("filename")`). This is again a tradeoff that helps with simple things (as users already know the scripting language if they know how to use the editor), while making complex things harder.
I'm interested in seeing a response to this. I sympathize with wanting to use unix tool composition, but I've personally found shell languages inadequate for more interesting applications.
I wouldn't say that they're badly designed. They make it easy to deal with text, the bread and butter of shell, while neglecting other data structures, which is fine 99% of the time, unless you're using these shell languages as general purpose.
They also neglect parallel, concurrent, async programming. Which agan is fine for a shell language. The shell does have use cases for simple parallel processing and you can delegate to dedicated unix tools for that and it works great. If you need more, you're expected to jump into a more powerful environment.
So they're poor in data structure variety, and more interesting (nonsequential) ways of executing programs, which, again, is fine for a shell language, but those were exactly the things I wanted. I wanted to pipe and tee a bunch of programs programs into one "executable graph" a la dataflow programming using bash or something similar, but quickly understood that due to the above reasons I need a different environment. I'm still on the lookout for good dataflow programming environments.
My issue is not that shell scripting languages have constraints and are not general purpose programming languages.
My issue is their tons of subtle points (e.g. behaviour of [ vs [[ blocks), blatant disregard for the principle of least surprise, lack of some very basic items (not parallel, async etc programming -- a mere list and a mere map with an obvious syntax and no BS caveats would do), inelegant ad-hoc accretion of features, bad error handling, and so on. Those things can (and frequently do) trouble programmers even an one-liner.
I don't with e.g. something like bash supports writing a full 10K program with it.
But I wish it didn't make a 100-line shell script so clumsy and frail.
You can definitely do parallel / concurrent / async programming directly in the shell, without any helper programs. It's not exactly ergonomic, but with a few patterns it's not too difficult.
Async:
long_running_cmd & > results.txt
cmd_pid=$!
# other stuff
wait $cmd_pid
# do something with results, if you want
Parallel:
for script in "${scripts[@]}" ; do
"$script" &
done
wait
But if you want a useful dataflow environment, I really recommend checking out [Nextflow](https://www.nextflow.io/). I use it at work for constructing bioinformatics pipelines, and it's really natural. The "preview" DSL2 is worth looking at, as well.
I was dismissing powershell for a great deal of time until I eventually looked into it more and I have to say, I'm impressed by it. Though it can be used on linux, it doesn't feel like a first-class citizen, but non-the-less I would like to have an object-based shell environment for linux with old tools rewritten for it (like top would output a list of objects with properties like CPU usage, name etc and a later grep could be used to sort them by whatever we want. No more awk/sed the nth column hack)
I burst out laughing. I've not seen spaghetti code like that in a serious program before. Why is syntax highlighting being implemented in a shell language anyway?
Coming from a video game industry background, I have been running Kakoune in Windows since its early days, but always in a posix environment (cygwin, then WSL).
It was a concious design decision to rely on posix extensively, one reason being that if you do not care about not having posix available, you are probably not that interested in what Kakoune provides.
So, if the new language is fundamentally incompatible with the old VimScript, why keep using VimScript? Lua already seems pretty fast, why not just make a fork where Lua is the default scripting language?
(Note, not considering JavaScript in the benchmarks seems like a pretty big oversight before making a conclusion of new default language.)
Why can't the default be to write extensions in C/C++/Rust/zig/jai or any compiled language with which one can produce a .dll or.so?
Why can't application developers abandon this strange habit of adding a slow "scripting" language?
Why not provide a versioned C API and be done with it?
A compiled language would allow us to use the best debugger and compiler on our systems and performance would be in the extension writer's hands and would eliminate the frustrating need to learn yet another scripting language.
>Why can't application developers abandon this strange habit of adding a slow "scripting" language?
How is it strange? Your suggestion sounds even more bizarro. It's not like those developers don't know about shared libs and .so files.
It's that:
a) Programming in C/C++/Rust is more cumbersome (than in a dynamic/scripting/gc language), and throws off tons of potential extension developers.
b) building extensions as .so even more so, adds different builds per OS/architecture, compilation steps, and tons of potential issues, both for setting up your extension building and for delivering it.
c) encouraging extensions written as .so in different languages, means a smaller pool of common reusuable extension code (each extension will reinvent several wheels). Whereas people could easily develop libs in the editors extension language and share them for re-use in others' extensions.
d) an issue in a C/C++/unsafe Rust etc extension will bring down the whole editor. In an embedded language that just calls API you can mitigate it easily.
f) same for memory leaks and such.
AAA game studios, which are hardcode C++ users with high performance / low latency needs, and have gone to using embedded scripting languages (with great success) and for some reason mere editor developers should abandon them and go pure C ABI?
a) I personally find using scripting languages more cumbersome in the medium to long term when I need to maintain, debug code, and optimise code. Every time I see a scripting/dynamic language, I'm turned off from extending the software that embeds it because debugability and performance will inevitably become a problem.
b) I'm not convinced of the alleged "tons of potential issues" that arise from using a compiled language. The extension can be distributed as C code (or whatever compiled language the text editor's community has decided upon). Every platform has a C compiler. I'd argue that C is the most portable language available. Extensions that don't compile will simply fade away, as do extensions in any scripting language, when they don't work. (There is no need for a complex build process or autoconf or whatever other frustrating arcane/convoluted build systems we've somehow been convinced are necessary.)
c) A dll is a shared library. By definition, that is reusable code. There's no necessity that "different languages" be encouraged, merely _a_ compiled language. Even if one extension were to be written in rust/zig and I choose to write an extension in C, is it really so hard for me to read the code I need from that extension and transcribe it for my purposes? I'm not convinced by the "reusable extension code" argument because I have not seen any statistics to justify it and I'm wary of anecdotes, including my own. How much of vim plugins or emacs extensions or vscode extensions are reused code from other extensions? Is it significant enough to raise this (that scripting language code is somehow more reusable) as an argument in defence of scripting languages as extension languages?
d) Patently false I think. (Recall also that emacs didn't successfully mitigate this case for a long time and extensions in elisp could crash the editor.) It's just as easy, if not easier, to mitigate this kind of thing in C. A loaded dll or function can be run easily in a separate process or thread or whatever else one desires. (Didn't plan9 ACME implement communicating processes, an architecture that might be suitable as mitigation of the sort you describe?)
(Also, reinventing the wheel is a bugbear that we should let go of. Attempting to reinvent a wheel is the only way by which one can truly understand how a wheel works. We shouldn't lose this ability or desire. The more we reinvent wheels, the better we get at examining fundamentals efficiently. If we don't reinvent wheels, we'll eventually lose the ability to make wheels.)
f) Claiming "memory leaks and such" is a bit too abstract for me to understand what you mean. I've seen an emacs extension gobble up gigabytes of memory and crash the editor. I suspect I can write such an extension even today. The fault and responsibility would be mine, the extension writer's. I've seen python scripts consume all of a system's memory. There is no magic bullet that solves how much memory a program needs and when the program needs the memory. The programmer has to solve these things.
I do not have any experience in AAA game studios. That said, I haven't heard of scripting languages being used extensively in recent years: A cursory glance at GDC presentations over the past 5 years doesn't reveal much emphasis on scripting languages or on the great successes you allude to. I gather from this that they've mostly been abandoned, excised from engines, or play a relatively insignificant role in the engines used for such games.
I'm not convinced that compiled languages constitute a significant barrier to entry merely because they are compiled and have low level facilities. That would only be possible if programmers somehow didn't have a working understanding of how CPUs and memory work, could only function with a garbage collector, and were unable/unwilling to run compilers like gcc or debuggers like visual studio, but such a person wouldn't meet the definition of "programmer" anyway, I don't think. My assumption is that a text editor, in particular, is being written for programmers.
(a) I don't think we can agree in the end starting from this though. In general, scripting languages are extremely popular and just fine for most people, and editors with tons of quality extensions (Emacs, Vim, Sublime Text, VS Code) all have them written in those. Even if your argument was valid about building a large system, here we're talking editor extensions.
(b) If you're accustomed to compiled languages (e.g. seasoned C/C++ programmer), then sure you are not. But setting up the correct environment, debugging, etc is more difficult for C/C++ than for any scripting language, and a big barrier to entry.
(c) The necessity arises from wanting to get more extensions, and thus attract more coders. If you only allow C extensions, you'll get much less extensions (and not necessarily higher quality either). So you either give a scripting extension language (low barrier, easy access, lots can start using it), or many scripting langauges (even better), or you allow for many compiled languages though an API/ABI (higher barrier, but at least not limited to a single one).
I hardly believe this is a controversial point. Literally almost any app that ever wanted to get more extensions from users went that route. They didn't tell them: "just set up GCC, get the headers, and be on your way", because they knew the majority of potential users (extension authors) would be put off.
In fact, even hardcore C++ AAA game developers add a scripting engine, not for end users, but to make it easier and faster for themselves to develop the logic, behaviors, etc, of the game, exactly for the same reason: outside of the critical paths C++ is overkill, and slows down momentum.
(d) So, not exactly patently false, but only "patently false if you also make a special architecture choice for it" (off-process, messaging, etc).
> f) Claiming "memory leaks and such" is a bit too abstract for me to understand what you mean.
(f) A scripting GC makes it from less easy to impossible to have memory leaks in your extension that e.g. C.
I do not have any experience in AAA game studios. That said, I haven't heard of scripting languages being used extensively in recent years: A cursory glance at GDC presentations over the past 5 years doesn't reveal much emphasis on scripting languages or on the great successes you allude to. I gather from this that they've mostly been abandoned, excised from engines, or play a relatively insignificant role in the engines used for such games.
I'm not convinced that compiled languages constitute a significant barrier to entry merely because they are compiled and have low level facilities. That would only be possible if programmers somehow didn't have a working understanding of how CPUs and memory work, could only function with a garbage collector, and were unable/unwilling to run compilers like gcc or debuggers like visual studio, but such a person wouldn't meet the definition of "programmer" anyway, I don't think. My assumption is that a text editor, in particular, is being written for programmers.
>I gather from this that they've mostly been abandoned, excised from engines, or play a relatively insignificant role in the engines used for such games.
The scripting language is also the configuration. Does it make sense to have non-portable pre-compiled configuration, or configuration that requires a compiler invocation on startup?
I don't know what kdbus is but you're never going to have less latency than copying a value into your instruction register.
And the fact you think a software bus is good enough is the reason so many things are so slow these days. It just depends on what your standard for latency is.
So you can use all the zillions of existing extensions/scripts, it really only makes sense to rewrite using VimScript as the scripting language to make it faster. Then you have a built in user base.
But according to the README, the new language is syntactically incompatible with the old VimScript. So you can’t reuse those extensions, they have to be rewritten.
Yeah, I didn't notice that. If that is the case, then perhaps there is no purpose in creating a new language. Perhaps the only benefit to creating the new language could be that it might be a little easier to mechanically convert from the existing VimScript to the new style VimScript they are thinking of.
Unless you want to be in some way compatible with Emacs, elisp is not a language I would use. Anyone who has written any non-trivial amount of elisp knows this.
There are other lispy languages, that are easier to embed without many of the warts of elisp. If you really want to have elisp, you have 3 ootions:
* Implementing it yourself
* Trying to break elisp and Emacs apart (not likely)
* Using guile's elisp implementation. It is complete and all that, but then you should just use guile scheme instead.
Sure it does. It is just unsupported :) I cross compiled 2.0 without much fuzz at all many moons ago without other changes than a small work-around for a bug in mingw. 2.2 should be about the same. 3.0 (or rather, the 2.9 beta) could possibly be quite a lot harder
Why would people want to adopt and use Vim9 instead of NeoVim, which eliminates a lot of cruft and uses Lua? Interested in knowledgeable responses from Vimmers and NeoVimmers.
VimScript isn't all that bad. I think a lot of the hate it gets here is undeserved. It's a domain-specific language, sure, but scripting an editor is a domain-specific thing. I'm not sure if Lua would really give me a massive amount of benefit for the things I write in VimScript. Performance has never been too bad of an issue, and I'm not sure if I like programming plugins in Lua more (but I haven't tried it, so I'm hesitant to "knock it 'till I try it").
Note that in my experience the biggest performance hog tends to be syntax highlighting.
Neovim changed some defaults and such, and I need to frob with my vimrc and such to spend time on it. I'm at a point in my career where I just want to build "useful stuff", and not muck about with my vimrc. For example, the default colours don't work well with my light background.
Neovim also removed encryption, which I use. Yes, I'm familiar with all the arguments for removing it and I've engaged in discussions on this in the past (so won't repeat it here): it's "good enough security" for what I use it for, and a much better UX than the alternatives. Again, more effort.
And what do I get back for it? I can't really name a single thing that would give me an immediate benefit to be honest. To be fair, I'm a "simple" user, and don't use stuff like :term and whatnot.
Also, I'm put off by the Neovim developers' attitude sometimes, things like a Neovim dev condescendingly describing VimScript2 as "BramScript" earlier today on Lobsters is ... not great, and neither is describing various disagreements as "cop outs" (e.g. t_* settings).
I’m torn by the idea of replacing vimscript with lua. I honestly enjoy both languages (vimscript is not as bad as its reputation!) but they each shine in different use-cases. Lua is more ergonomic for writing programs/plugins, but vimscript is more ergonomic for interactive use (and maybe config files, more below).
For non-vimmers, a good analogy is the shell: would you want to replace sh with a more robust language like lua? Few people seem to enjoy writing larger programs in sh, and are quick to jump to perl/python/ruby once a script crosses some complexity threshold. However, I doubt many people would want to give up sh for interactive use. Mainly[0] due to omitting “cruft” like parens/quotes/commas, and also pipe syntax. Minimal example:
ls -laht | grep foo
is nicer than
grep(ls(‘laht’), ‘foo’)
especially when you’re doing thousands of commands of varying complexity every day.
Vimscript is similar, again mainly due to omitting “cruft” like parens/quotes/commas, and also some specific sugar. Minimal examples:
iabbr cosnt const
is nicer than
iabbr(‘cosnt’, ‘const’)
and
s/foo/bar/g
is nicer than
sub(‘foo’, ‘bar’, ‘g’)
especially when you’re doing hundreds (thousands?) of commands of varying complexity every day.
Besides setting options, the vast majority of the “meat” of my config file boils down to these types of commands, and is thus much more readable with the cruft omitted, even though I only have to type it once.
This could probably be addressed with some light syntax sugar on top of lua, and I think I remember seeing some comments from the Neovim folks about this very thing, but I’m not able to find anything concrete at the moment. Does anyone know where this stands?
—
[0] Of course, another issue (for both shell-scripting and vim) is the huge amount of tools built on top of the existing language, which we may be remiss to abandon. For shells, it’s probably practically impossible. For vim, I suspect it is possible, though it would be painful enough to merit a long hard think.
I you are having your editor change "cosnt" to "const", or "isntall" to "install", you are programming your fingers to be unable to type them correctly. You will regret it, but it will be too late.
I also never want to learn a language for a single environment. With Lua and the like, I can use them in many other places, making learning them much more useful.
This is true only if what you're building is big enough.
For something like vimscript, where most people are likely to script minor stuff, the marginal cost of learning the scripting language itself is non-trivial and increases the "barrier of entry" significantly.
For minor stuff you spend most time on learing ex commands (i.e. VIM API). Also, vimscript is a rather simple language. It's not haskell, we're talking about.
Many of the comments here appear to have been made without reading the full post. Maybe because I'm not a Vim user it made sense to read the post in its entirety.
From the post:
"Attempts have been made to implement functionality with built-in script languages such as Python, Perl, Lua, Tcl and Ruby. This never gained much foothold, for various reasons.
Instead of using script language support in Vim:
Encourage implementing external tools in any language and communicate with them. The job and channel support already makes this possible. Really any language can be used, also Java and Go, which are not available built-in.
Phase out the built-in language interfaces, make maintenance a bit easier and executables easier to build. They will be kept for backwards compatibility, no new features.
Improve the Vim script language, so that it can be used when an external tool is undesired.
All together this creates a clear situation: Vim with the +eval feature will be sufficient for most plugins, while some plugins require installing a tool that can be written in any language. No confusion about having Vim but the plugin not working because some specific language is missing. This is a good long term goal."
So, as I read this, his argument is that it makes sense to offer both external language support and Vimscript.
I find it quite impressive that Bram still finds time to not just maintain Vim but also adding drastic new features like this, while doing a full time job at Google at the same time.
I have the displeasure of knowing vimscript. I have a couple plugins and contribute to others. vimscript is terrible, it's unintuitive, neither fp nor oop, and fails to integrate any domain specific features. It could and should be replaced by any other fast scripting language.
Kill vimscript adopt Lua for compatibly or Gluon for fp.
Vimscript is a collection of ex commands (i.e., a DSL for editors) with a few functions bolted on that provide for a simple form of OOP. I don't think your argument is 100% fair.
Being a collection of commands that can be entered in a repl does not make it a text centric DSL Text manipulation in vimscript is more verbose than Rust including all ownership syntax. What elements of vimscripts design make it good at parsing and manipulating text?
I can imagine a beautiful text centric EDSL in Haskell or a lisp that integrates tree sitter to provide syntactic hooks. I don't blame vimscript for not having this modern integration, but I do blame it for being a poorly designed language and I condemn any attempts to prolong its life. Over in NeoVim land treesitter is being integrated. Once it lands someone should write a EDSL in the Lua lisp.
I've seen this posted 4-5 times (via the "past link") and 0 comments in all.
Are the usual posts with comments (e.g. on some minor version release of a framework) on HN so much more interesting than a potential feature direction of a staple UNIX editor with a redesigned scripting language?
I think the wording of the title is likely making people think this is "just another fork". However, this is made by Bram and he's looking to drastically optimize the way VimScript functions are called (as opposed to redesigning the scription language).
Either way, I'm happy that I saw it 1 of the handful of times it's been posted. Being a NeoVim regular, though, I'm more interested about how/if this carries over and what the gains would be there.
> Seems like some impressive improvement in benchmarks, but for me vim already just works.
> I never feel like it's slowing me down, and I can get everything I need from it.
I feel the same way personally, but do you do a lot of scripting of vim? I don't, and I get the impression that that is a lot of pain points for vimmers much more experienced than I, so that that may explain why people like us don't see the need for such optimisation.
You’re right about Neovim’s effect on Vim improvements and lighting a fire in Bram.
Even if neovim goes belly-up (unlikely), or totally diverges (slightly less unlikely), they’ve done a huge service to the vim community simply by existing and offering compelling, thoughtful enhancements while respecting the core spirit of vim - and being exceptionally mindful of practicality in how they approach changes.
Probably one of the best-ever examples of a fork that is Doing It Right.
>are the Vim9 benchmarks not underwhelming compared to Lua or did I miss something?
Apparently that's the case, Bram did not compare against LuaJIT making the benchmarks pointless. They would quickly point out that Vimscript 2 also is, if this Reddit comment thread is to be believed: https://www.reddit.com/r/vim/comments/ejfn57/vim9/fcxi99k/ where justinmk is one of the Neovim core devs.
As an end user, Vim feels (IMO) like a dead-end. It's cool that Bram has started trying new things but he comes off as too protective of his creation, which is fine, but then it's normal for the community to move as well.
Why not have it as an incremental improvement? Similar to javascript engines, jit the easily executable code and fall back to the slow path if any of the listed slow features are encountered.
Incrementalism is powerful, even if it also takes a lot of work; breaking changes in Vim script are probably not worth it for anyone.
I ended up switching to Kakoune after trying it for a bit because I realized that in the two weeks of me exploring it I had entirely understood it’s “scripting language”, or at the very least understood how to use the onboard docs to write any plugin I wanted to.
[1]: https://github.com/mawww/kakoune