NGS is programming-language-first while other modern shells are typically shell-first. Multiple dispatch would probably be the most prominent manifestation of this approach in NGS.
> NGS is programming-language-first while other modern shells are typically shell-first.
Not always no.
Even if your point were true, being programmer-first is not always a desirable feature. The vast majority of shell work is basic and repetitive. Most of the time people just want something that functions a little like Bash but less shitty for scripting. The fact that Powershell, LISP nor Python haven’t taken over the world for shell usage proves that a higher level language REPL generally makes for a shit daily shell. So usually you end up with just a small few purists using it. And that’s not really good enough.
Whereas Oil, Elvish, Murex and even Fish (to a less dramatic extent) are looking at what makes a good shell and then fixing shell scripting within that shell. Having gone through the REPL phase myself with a great many different languages and found them all painful for daily use, I’m inclined to agree with shell-first approach.
> Multiple dispatch would probably be the most prominent manifestation of this approach in NGS.
Again, NGS isn’t unique in that regard. Murex has methods, Powershell has methods. I’ve not seen anything new in NGSs “Multiple Dispatch” nor “MultiMethod” docs that other alt shells aren’t also doing.
Don’t take this the wrong way, it looks very impressive what you’re doing. But it’s not unique these days. And I say this having tried a great many options out there.
> being programmer-first is not always a desirable feature.
programming-language-first allows convenient scripting. I could not think of a way to make anything shell-first be convenient for scripting (anything beyond tiny scale).
> higher level language REPL generally makes for a shit daily shell.
I think it proves that general purpose languages, where using them as a shell was afterthought "makes for a shit daily shell".
NGS, on the other hand, has somewhat-bash-like (read: easily run external programs) syntax at the top level, which should be good for CLI. Want to use more advanced features that are typically associated with "real" programming languages? OK, pay the price, switch syntax with { ... } and have full blown programming language at your disposal, in the shell.
> NGS isn’t unique in that regard.
I do scan alternative shells from time to time. I could have missed but I didn't see multiple dispatch in any of them. Which ones have it? Just to clarify: In which shell you can define several methods with the same name and when called, the method to invoke is selected based on the types of the arguments?
There is a huge difference in having methods and multiple dispatch.
> not unique these days
NGS is a mixture of "borrowed" features and unique ones. Multiple dispatch is an old concept and has been implemented in other programming languages. Examples of things in NGS that I have not seen anywhere else: syntax for run-command-and-parse-output, proper handling of exit codes.
Regarding exit codes. Typical approach to exit codes varies. Python - "I don't care, the programmer should handle it". Some Python libraries and other places - "Non-zero is an error". That's simplistic and does not reflect the reality in which some utilities return 1 for "false". bash (and probably other shells too) is unable to handle in a straightforward manner situation where external command can return exit codes for "true", "false" and "error". It just doesn't fit in the "if" with two branches. NGS does handle it with "if" with two branches + possible exception thrown for "error" exit code.
Edit: and some other features around handling exit codes such as short syntax to provide expected exit code (any other exit code throws exception)
Edit: clarification - NGS knows that some external programs have exit code 1 which does not signify an error.
> I do scan alternative shells from time to time. I could have missed but I didn't see multiple dispatch in any of them. Which ones have it? Just to clarify: In which shell you can define several methods with the same name and when called, the method to invoke is selected based on the types of the arguments?
Murex does this but in a slightly different way. Rather than method overloading (which is a bad feature in any language in my opinion but I get it has its fans) murex has APIs that allow for writing data type agnostic methods. in practice that means the same tools to query a length of an array, or grab an item in a map (etc) work irrespective of whether the data type is a JSON array or map, YAML, CSV, S-Expressions or even just ‘ps’ (etc) output…and so on and so forth. This expands out to all methods, meaning you basically have a ‘jq’ like toolset that works exactly the same - same commands etc - for most data formats (certainly the ones I use daily). Which in effect is the same end result as what you’re describing except without the method overloading and instead allowing me as a shell script writer not to worry less about the data type or format of the piped data.
There are a few specific methods that do allow for overloading though, but that’s where functionality is a little more complex (these are usually event handlers though. I don’t use them much so I can’t say how well they work)
> NGS does handle it with "if" with two branches + possible exception thrown for "error" exit code.
Murex does this too. I’m pretty sure I’ve seen other shells do that as well (possibly Elvish?)
Iconically it was actually the error handling that drew me to murex and the smart handling of data types within methods that kept me there. Basically all the same features you’re promoting in NGS. Which is why I’m saying there is a lot out there that does the same.
Anyhow, it’s been interesting reading about your work on NGS. Good luck
murex author here. Hopefully I can answer some questions:
> I do scan alternative shells from time to time. I could have missed but I didn't see multiple dispatch in any of them. Which ones have it? Just to clarify: In which shell you can define several methods with the same name and when called, the method to invoke is selected based on the types of the arguments?
I've not heard of the term "multiple dispatch" but reading the thread it sounds like you're describing function overloading. Powershell does support this with classes[0].
murex does it's overloading at the API level[1]. The reason behind that decision is to keep the methods simple (eg a pipeline might contain JSON or CSV data) because you as a shell user don't want to run into situations where you've written a function that supports one data type but not another, you just want it to work first time. So murex automatically abstracts that part away for you. In addition to the point the other poster mentioned about consistent `jq`-like methods that are data type agnostic, murex allows for easy iteration through objects agnostic of their data type (eg `open somedata -> foreach { do stuff }` -- where you don't need to think about data types, file formats, etc, murex does the heavy lifting for you in a consistent and predictable way).
There are some specific builtins that support "overloading" of sorts but instead of overloading the function they have handlers that call user defined functions[2][3]. This removes the mystery behind function overloading because you can easily trace which handlers exist for which data types.
It's also worth adding that function overloading is supported in the same way that its also supported in Bash too where you might have a function but if there is a alias of the same name that will take priority. There are also private[4] functions (which are namespaced) so you have additional controls to avoid accidental overloading when writing modules too. And the interactive shell expands out any command (eg displays a hint text stating if a command is a function, alias, external command, etc) so there's a reduced risk of being unaware if `ls` is an alias, function, `/bin/ls`, or even a symlink to something else.
Named parameters are optional because neither Windows nor POSIX have any understanding of named arguments in their processes. I did consider abstracting that into murex's functions regardless like Python et al would but I couldn't design a way that wasn't jarring nor cumbersome to write in a hurry nor worked transparently with Windows and POSIX ARGS[]. So I've come up with an optional builtin called `args`[5] which allows you to define flags, which are effectively the same thing as named parameters except they're supported natively by Windows and POSIX ARGS[], so you can write a murex script as a native shell script without leaving the user to write another abstraction layer themselves interpreting an indexed argument into an named arguments. Thus giving you the flexibility to write really quick Bash-like functions or more verbose scripting style functions depending on your need.
> Examples of things in NGS that I have not seen anywhere else: syntax for run-command-and-parse-output, proper handling of exit codes.
Both of these are baked into murex as well. The run-command-and-parse-output is the API stuff mentioned above. It's where the overloading happens.
As for error handling: any command is considered a failure if there is either a non-zero exit code, STDERR contains a failed message (eg "false", "failed", etc) or STDERR is > STDOUT[6]. This has covered every use case I've come across.
Additionally STDERR is highlighted red by default and when a process fails you're given a mini-stack trace showing where in the script the error happened. You have try/catch blocks, saner `if` syntax etc that all use the same API for detecting if a process has failed too. Keeping everything consistent.
The other thing murex has to help catch bugs and error is a testing framework baked into the shell language. The docs for test[7] need expanding but in essence:
- you can write proper unit tests
- you can intercept STDOUT (even if it's mid pipeline) and test that output. This works around instances where (1) you can't have full on unit tests due to side effects in a function that can't be easily mocked (2) you want to add your own debugging routines (rather than just printing values to STDOUT or running commands manually to see their output -- like one would normally have to do with shell scripting)
- you can add state watches. Except in murex the watches are Turing complete so you're not just adding noise to your terminal output but rather putting meaningful debug messages and scripts.
And all of the debugging stuff can be written straight into your normal shell routines and cause no additional execution overhead unless you've purposely enabled test mode.
So its fair to say I've spent a significant amount of time designing smarter ways of handling failures than your average shell scripting language. As I'm sure you have too.
> Regarding exit codes. Typical approach to exit codes varies. Python - "I don't care, the programmer should handle it". Some Python libraries and other places - "Non-zero is an error". That's simplistic and does not reflect the reality in which some utilities return 1 for "false". bash (and probably other shells too) is unable to handle in a straightforward manner situation where external command can return exit codes for "true", "false" and "error". It just doesn't fit in the "if" with two branches. NGS does handle it with "if" with two branches + possible exception thrown for "error" exit code.
If you're describing that as a "two branch" approach then technically murex could be argued as having "three branches" because it checks exit code, STDERR contents, and payload size too. Personally I just describe it as "error handling" because in my view this is how peoples expectations are rather than the reality of handling forked executables.
I guess where NGS and murex really differ is NGS likes to expose its smart features whereas in murex they're abstracted away a little (they can still be altered, customised, etc) to keep the daily mundane shell usage as KISS (keep it simple stupid) as possible. eg you can overload function calls if you really wanted but that can often cause unforeseen complications or other unexpected annoyances right when you least want it to. So murex keeps that stuff around for when you need it but finds ways to avoid people needing to rely on it and furthermore unwraps the covers of what a routine does in the interactive terminal. It's all about offering abstractions but removing surprises.
> NGS knows that some external programs have exit code 1 which does not signify an error.
Having different behaviours for different executables hard coded into the shell is one behaviour I purposely avoided. I do completely understand the incentive behind wanting to do this and wouldn't criticise others for doing that but given external programs can change without the shell being aware, they can be overloaded with aliases, functions, etc, and they might even just differ between Linux, BSD, Windows, etc -- well it just seemed like hard coding executable behaviour causes more potential issues than it solves. You also then run into problems where users expect the shell to understand all external executables but there are some you haven't anticipated thus breaking expectation / assumptions. Ultimately it creates a kind of special magic that is unpredictable and outside the control of the shell itself. So instead I've relied on having foundational logic that is consistent. It's the one side of shell programming where I've placed the responsibility onto the developer to get right rather than "automagically" doing what I think they are expecting. This of course does mean I have to focus even harder on ensuring all other aspects of the shell are predictable and low maintenance so that the developer can cover any specific edge cases with ease. Which goes a long way to explaining why I've chosen the path I've chosen (ie reducing the amount of syntax sugar, overloading, etc needed for daily use that one might rely on in a more traditional programming language).
Somewhat similar. Since methods do not live in classes in NGS, I would argue the mechanism in NGS is simpler (more elegant?). You can just define your_method(c1:Class1, c2:Class2).
> murex does it's overloading at the API level[1].
mmm. I see the support but looking at documentation at https://murex.rocks/docs/apis/Unmarshal.html , I see it's not exposed into the Murex language, it's in Go. Is this correct?
> where you don't need to think about data types, file formats, etc, murex does the heavy lifting for you in a consistent and predictable way
That is good and what I would expect.
( sorry, running out of time, to be continued :) )
Learn something new every day. Seen (and used) this methodology before but wasn't aware it was called that. I'd always heard of it as "overloading" which is conceptually similar but not identically the same.
> mmm. I see the support but looking at documentation at https://murex.rocks/docs/apis/Unmarshal.html , I see it's not exposed into the Murex language, it's in Go. Is this correct?
It's APIs written in Go (for performance and convenience -- you wouldn't want to write a YAML marshaller in a shell scripting language). So the methods are exposed as builtins. However you can add method written in murex if you wanted.
The idea being get 99.9% right by default but leave some flexibility for the user to customise if they want.
Murex builtins are also written in a way that they're all optional includes into the core project. Thus you can easily write your own builtins, marshallers, etc in Go if you want performance and then call them from your shell script (or you could write them in Python, Java, Perl, etc and run them as an external executable if you wanted too. But if you do that you lose access to murex's typed pipelines. Which is where the really interesting stuff happens (I haven't yet figured out a non-shitty way to send typed data over POSIX pipes to external executables).
> sorry, running out of time, to be continued :)
Any time :) I'm finding this conversation really interesting and educational
I should have added (outside of edit time now) that I do like what you've done with NGS. Nothing I've posted above is intended to suggest I disagree with your approach. We're covering mostly the same problems but we've just looked at it from different angles. Which is good -- the more options out there the better I say.
In NGS the common case for multiple dispatch would be to define your own type and then add methods to existing multimethod to handle that type; additionally I've simplified the dispatch algorithm exactly for this reason - unclear answer to "what's going to be called actually?".
> function overloading is supported in the same way that its also supported in Bash too where you might have a function but if there is a alias of the same name that will take priority.
I don't think it is called overloading. That's very different from having two methods with the same name and the "right one" is called based on types of passed arguments.
> interpreting an indexed argument into an named arguments.
In NGS it happens in exactly one place, when main() is called. At that point command line arguments are automatically parsed based on main() parameters and passed to main().
I don't think I understand your reasoning behind not including named parameters.
> NGS likes to expose its smart features whereas in murex they're abstracted away a little
Sounds about right. Power to the people!
> Having different behaviours for different executables hard coded into the shell is one behaviour I purposely avoided.
I can see why. NGS prefers to do the right thing in most cases and have shorter and cleaner code. Yes adding some risk, which I see as not that big - exception when an exit code was "ok" for unknown program (unknown external programs default to exception on non-zero exit code).
I also dream about moving the hard coded information about programs into separate schema, which could be re-used between different shells. That would be similar to externally provided typescript definitions for existing code, described at https://www.typescriptlang.org/docs/handbook/declaration-fil...
> differ between Linux, BSD, Windows,
"detect program variant" feature in the schema I mentioned above
> hard coding executable behaviour
Yep, smells a bit
> causes more potential issues than it solves.
I think in particular situation with exit codes the risk is low.
> users expect the shell to understand all external executables but there are some you haven't anticipated thus breaking expectation / assumptions
Yes. Downside. That's the kind of surprises that suck. Need to make sure at least the docs are clearly warning about this.
> So instead I've relied on having foundational logic that is consistent.
Totally understandable.
> STDERR contains a failed message (eg "false", "failed", etc) or STDERR is > STDOUT[6]. This has covered every use case I've come across.
Sounds also like potentially surprising, somewhat similar to handling exit codes. Yes, it's not per program and that's why it's better but still.
> Additionally STDERR is highlighted red by default
As it should! Yes!
> mini-stack trace
Yes, please!
> testing framework baked into the shell language.
> So its fair to say I've spent a significant amount of time designing smarter ways of handling failures than your average shell scripting language. As I'm sure you have too.
I guess. Differently of course :)
> Personally I just describe it as "error handling" because in my view this is how peoples expectations are rather than the reality of handling forked executables.
Sounds like I was not clear enough as if something is not uniform in this regard in NGS. Exceptions are not only for external programs. It just happens that external program can be called inside "if" condition; an exception might occur there.
> > function overloading is supported in the same way that its also supported in Bash too where you might have a function but if there is a alias of the same name that will take priority.
> I don't think it is called overloading. That's very different from having two methods with the same name and the "right one" is called based on types of passed arguments.
Yeah you're right that's not overloading. I'm not really sure why included that in my description.
> In NGS it happens in exactly one place, when main() is called. At that point command line arguments are automatically parsed based on main() parameters and passed to main().
> I don't think I understand your reasoning behind not including named parameters.
there isn't really an entry point in murex scripts so
function foobar {
out foobar
}
foobar
is no different from
#!/usr/bin/env murex
out foobar
and no different from typing the following into the interactive terminal
$ out foobar
Which means every call to a builtin, function or external executable is parsed an executed as if it has been forked, at least with regards to how parameters are just an array (actually on Windows they're not even an array. Parameters are passed just as one long string, whitespace and quotation marks and all. Which is just another example of why Windows is a shit platform). This is a limitation of POSIX (and Windows) that I decided I didn't want to work around because otherwise some parts of the language would behave like POSIX (eg calling external programs and how named parameters are passed as `--key value` style flags vs scripting functions where named parameters positional arguments).
I wasn't really interested in creating a two tier language where parameters are conceptually different depending on what's being called so I decided to make flags easy instead:
function hippo {
# Hungry hippo example function demonstrating the `args` builtin
args: args {
"Flags": {
"--name": "str",
"--hungry": "bool"
}
}
$args[Flags] -> set flags
if { $flags[--hungry] } then {
out: "$flags[--name] is hungry"
} else {
out: "$flags[--name] is not hungry"
}
}
That all said. I'm still not 100% happy with this design:
- `args` still contains more boilerplate code than I'm happy with
- murex cannot use `args` to automatically generate an autocomplete suggestion
So I expect this design will change again.
> Sounds also like potentially surprising, somewhat similar to handling exit codes. Yes, it's not per program and that's why it's better but still.
I don't see how that's surprising because it's literally the same thing you were describing with NGS (if I understood you correctly).
> Sounds like I was not clear enough as if something is not uniform in this regard in NGS. Exceptions are not only for external programs. It just happens that external program can be called inside "if" condition; an exception might occur there.
It was me who was unclear as I assumed you meant it was the same error handling for builtins and functions too (as is also the case with murex). I just meant the "reality of handling forked executables" as an example rather than as a commentary about the scope of your error handling :)
> there isn't really an entry point in murex scripts
I have a nice trick in NGS for that. Under the idea that "small scripts should not suffer", script is running top to bottom without "entry point". However, if the script has defined main() function, it is invoked (with command line arguments passed).
> `args` still contains more boilerplate code than I'm happy with
Is there anything preventing you to have exactly the same functionality but with syntactic sugar that it looks like parameters declaration? (Just to be clear, keeping all the ARGV machinery).
Something like (assuming local variables are supported; if not, it could still be $args[Flags] etc):
function hippo(name:str, hungry:bool) {
if { $hungry } then {
out: "$name is hungry"
} else {
out: "$name is not hungry"
}
}
> I don't see how that's surprising because it's literally the same thing you were describing with NGS (if I understood you correctly).
Yes it is almost the same in NGS with regards to exit codes, which you preferred not to do in Murex. On the other hand, checking stderr looks very similar to knowing about exit codes and here you decided to go for it. I'm puzzled why. It is somewhat fragile, like knowing about exit codes.
> It was me who was unclear as I assumed you meant it was the same error handling for builtins and functions too (as is also the case with murex). I just meant the "reality of handling forked executables" as an example rather than as a commentary about the scope of your error handling :)
Here I lost you completely but I hope it's fine :)
In NGS, exception handling is used throughout the language, consistently, well at least I hope it is.
> Is there anything preventing you to have exactly the same functionality but with syntactic sugar that it looks like parameters declaration? (Just to be clear, keeping all the ARGV machinery).
> Something like (assuming local variables are supported; if not, it could still be $args[Flags] etc):
> example code
oooh I like that idea. Thank you for the suggestion.
> Yes it is almost the same in NGS with regards to exit codes, which you preferred not to do in Murex. On the other hand, checking stderr looks very similar to knowing about exit codes and here you decided to go for it. I'm puzzled why. It is somewhat fragile, like knowing about exit codes.
No, murex still checks for exit codes. Essentially a program is considered successful unless the following conditions are met:
+ exit code is > 0
+ STDERR in ('false', 'no', 'off', 'fail', 'failed')
+ or []byte(STDERR) > []byte(STDOUT)
The exit code is self explanatory
STDERR messages are useful for functions that return a message rather than exit code. Particularly inside `if` blocks, eg
= 1 == 2 -> set math
if { $math } then { out: $math } else { err: $math }
will return `false` to STDERR because $math == "false" (ie `= 1 == 2` will return "false")
As for the STDERR > STDOUT test, that covers an edge case where utilities don't return non-zero exit codes but do spam STDERR when there are problems. It's an uncommon edge case but the only time this condition is checked is when a command is evaluated in a way where you wouldn't normally want the STDOUT nor STDERR to be written to the terminal.
This also allows you to get clever and do stuff like
if { which: foobar } else { err: "foobar is not installed }
where `if` is evaluating the result of `which` and you don't need to do more complex tests like you would in Bash to test the output of `which`.
As in not throwing exception or as in evaluates to true?
> exit code is > 0
You refused to get into specific exit codes for specific programs citing potentially surprising behavior (which I agree will sometimes be an issue).
> STDERR in ('false', 'no', 'off', 'fail', 'failed')
I correlated this with the following source code:
> if len(s) == 0 || s == "null" || s == "0" || s == "false" || s == "no" || s == "off" || s == "fail" || s == "failed" || s == "disabled" { return false
... which also has a potential for surprising behavior. (side note: appears to work on stdout, not stderr)
> or []byte(STDERR) > []byte(STDOUT)
Does that mean len(stderr) > len(stdout) ?
... also has a potential for surprising behavior. I can easily imagine a program with lots of debug/warnings/log on stderr and truthy/non-error output.
> STDERR messages are useful for functions that return a message rather than exit code.
Might be problematic because typically exit codes are used for that.
> where `if` is evaluating the result of `which` and you don't need to do more complex tests like you would in Bash to test the output of `which`.
Mmm.. Never did this. In which situation looking just at exit code of `which` is not good enough?
> if { which: foobar } else { err: "foobar is not installed }
Just to show off :) in NGS that would be:
if not(Program('foobar')) {
error("foobar is not installed")
}
