+1 on nushell, it is incredible. Having all data typed and structured is an insane superpower.
Common worries I hear from people that were non-issues in practice:
- Not POSIX compatible: Nushell doesn't aim to replace POSIX, there's no problem with dropping back to bash to execute some shell snippets
- You need to re-learn everything: I'm not a huge fan of how the commands are organized, but I still didn't find it that difficult. nushell's prompt/readline comes with an inline help menu with fuzzy search. Hit CTRL+O to edit a command/pipeline in your IDE/editor of choice with LSP backed intellisense, type-checking and in-editor command docs/help. The syntax is very simple and intuitive.
- Just use python: Sure, but python comes with a lot of disadvantages. It's slow and uses dynamic typing. Static typing in nushell catch typos in pipelines & scripts before they execute. It also makes in-shell and IDE LSP tab-completions very accurate. Large files process quickly though it will still consume more memory if you aren't able to process all the data in a streaming fasion. It's like having jq but with autocomplete and it works on all command output & shell variables. Though if you really like python, check out Oil/OSH/YSH: https://oils.pub/
- All Unix commands output text, structured data is useless in a shell: `detect columns` (https://www.nushell.sh/commands/docs/detect_columns.html) - now it's structured. Or use `from <format>` if the command outputs CSV, JSON, INI, YAML, etc... Or don't, cause GNU tools work fine in nushell if you keep everything in text format
And there are other crazy features too.
- Write nushell plugins in your language of choice, plugins can work with structured data
- Plugins can run in the background and maintain state, nushell can automatically start a plugin when it is first used and stop the plugin when it is idle
- e.g. a plugin can open a SQL connection and use it across multiple commands. There's a built-in plugin for opening in-mem/on-disk SQLite databases
- Data can carry metadata, e.g. binary data can carry its mime type, strings often carry metadata about which line and file the string was read from.
- Ongoing work on DAP suppprt to allow debugging scripts from your IDE
- Create your own hooks to customize how different types of data are displayed. Display structured data in table/tree form, display binary data in hex, etc...
- Collect related commands/variables into modules. Load a module knowing that you can easily unload the whole module later, module contents don't pollute global state. Variable declarations, env vars and loaded modules are scoped to the current code block and disappear after the closing bracket, lowering the odds of a name collision.
- Native support of Polars dataframes to work with even moar data
- Complex parllelism: Message-passing/actor architecture background jobs. Turn-key parallelism: transform every element of a list in parallel - `par-each` (https://www.nushell.sh/commands/docs/par-each.html)
The biggest downside of nushell is that it hasn't hit 1.0 yet so commands occasionally get renamed. Expect that you may occasionally need to tweak a script to get it working again. Definitely a pain point.
Nushell is fun. When I tried it years ago, it was only half-baked.
But a lot of the structured data transformation use cases I encounter, I find myself tackling in DuckDB. It's a little harder for the simplest things, but it pulls ahead quickly. Or at least it does if you need to remember SQL anyway...
Ah, it underwent a complete rewrite somewhat recently, I heard it was a significant improvement.
But I must agree that dedicated data analysis tools like DuckDB and jq are more powerful, intuitive, and performant. I guess what makes nushell appealing is how the data is already in nushell. It's where I stash any inputs I plan to use, any output commands produce and also any datasets I'm currently working on.
The true value of nushell is it's role as a data exchange that preserves typing+structure and in providing tools so ingesting structured data is easy and parsing unstructured data is not daunting. I'm less pushing for nushell specifically and more hoping that it encourages more people ro think about some larger questions:
- It's time to question the role of "UTF-8 text" as "the basic fundamental unit of data in the POSIX ecosystem"
- Typed/structured data brings significant value and is not harder to work with
- How can we improve data interchange between tools/apps without causing breakage? There's been quite a bit of thinking on if CLI tools can negotiate with each other to switch to communicating using higher-level data formats. Optimally it should work over common transports like SSH too. Unfortunately, I haven't seen any proposals that don't also introduce new problems. The nushell authors are looking into this as well.
- How can we evolve terminals from the simple, reliable text renderers (that they never were) into simple, reliable renderers of general structured data?
I tried really hard to get into Nushell but gave up after a month or two. Muscle memory with backgrounding was my big issue -- I tend to edit in helix and then background for a while and then foreground, and I seem to recall this totally freezing or crashing nushell somehow. Tried to learn some kind of recommended alternate workflow with pqueue (? I think) but just couldn't get there.
Yeah prior to background job support, pausing the foreground process with CTRL-Z was a major footgun—nushell didn't detect when the foreground process was paused, nor did it offer any way to unpause it. The terminal was rendered useless until you manually resumed/killed the paused process via another terminal or GUI.
I totally understand the confusion it brought to users, but I'm equally disappointed in how quickly the idea was abandoned, especially by the broader Android app ecosystem. It would be so useful to be able to open multiple Amazon app tabs to compare products for example.
The feature is still in the OS, so apps that declare support can allow users to open multiple simultaneous windows of itself. Most native Android apps can probably add support for this feature with minimal code changes, as Android "best practices" have pushed apps towards good reactivity support and rigorous handling of app state in these types of edge cases.
"Gamergate was actually 8chan communists fighting sensasionalist journalism but their message was then twisted and used against them to push people into far-right MAGA."
Amazing... I can't tell if you are trolling or seriously think this.
Communists no, young progressives yes. It's kind of insane to believe that a majority audience of gamers in the year 2014 would be anything but progressive, at least until the entire media turned on them and orchestrated a misinformation campaign out of a combination of a core of malice then a majority of laziness.
File managers need full access as you can use that ability to extract and inspect the code of any apps installed on the system. It is a very useful feature and I would hate for it to be removed.
youtube-dl just went largely dormant. There was a fiasco involving a unit test specifically downloading copyrighted content, but it was corrected. yt-dlp just became the more active fork.
The unlisting from GitHub was precisely due to the reason I mentioned, and Nat Friedman himself, CEO of GitHub at the time, dropped into the youtube-dl development IRC, assured the team that he had their back, and that the moment the infringing test was fixed, he would personally restore access, which he did posthaste.
Regarding the website being taken down, it was hosted in Germany and it was a German court order. Germany is notorious for this stuff, and it should never have been hosted there. If they wanted, they could have found a more reasonable host.
I understand the burnout, but it comes with the territory, and powerful enough people made it clear that the team did have their support. With some effort, the project could have continued on at full pace at least as uninhibited as its forks.
Now the URL just redirects to the yt-dlp GitHub repository, anyway.
Specifically testing the extra code needed to download certain videos - they didn't pick them just for the hell of it. It seemed unwise to have that in the public repo but I wouldn't describe it as a fiasco.
Common worries I hear from people that were non-issues in practice:
- Not POSIX compatible: Nushell doesn't aim to replace POSIX, there's no problem with dropping back to bash to execute some shell snippets
- You need to re-learn everything: I'm not a huge fan of how the commands are organized, but I still didn't find it that difficult. nushell's prompt/readline comes with an inline help menu with fuzzy search. Hit CTRL+O to edit a command/pipeline in your IDE/editor of choice with LSP backed intellisense, type-checking and in-editor command docs/help. The syntax is very simple and intuitive.
- Just use python: Sure, but python comes with a lot of disadvantages. It's slow and uses dynamic typing. Static typing in nushell catch typos in pipelines & scripts before they execute. It also makes in-shell and IDE LSP tab-completions very accurate. Large files process quickly though it will still consume more memory if you aren't able to process all the data in a streaming fasion. It's like having jq but with autocomplete and it works on all command output & shell variables. Though if you really like python, check out Oil/OSH/YSH: https://oils.pub/
- All Unix commands output text, structured data is useless in a shell: `detect columns` (https://www.nushell.sh/commands/docs/detect_columns.html) - now it's structured. Or use `from <format>` if the command outputs CSV, JSON, INI, YAML, etc... Or don't, cause GNU tools work fine in nushell if you keep everything in text format
And there are other crazy features too.
- Write nushell plugins in your language of choice, plugins can work with structured data
- Plugins can run in the background and maintain state, nushell can automatically start a plugin when it is first used and stop the plugin when it is idle
- Data can carry metadata, e.g. binary data can carry its mime type, strings often carry metadata about which line and file the string was read from.- Closures, generators, ranges, errors/exceptions + try-catch
- Ongoing work on DAP suppprt to allow debugging scripts from your IDE
- Create your own hooks to customize how different types of data are displayed. Display structured data in table/tree form, display binary data in hex, etc...
- Collect related commands/variables into modules. Load a module knowing that you can easily unload the whole module later, module contents don't pollute global state. Variable declarations, env vars and loaded modules are scoped to the current code block and disappear after the closing bracket, lowering the odds of a name collision.
- Native support of Polars dataframes to work with even moar data
- Complex parllelism: Message-passing/actor architecture background jobs. Turn-key parallelism: transform every element of a list in parallel - `par-each` (https://www.nushell.sh/commands/docs/par-each.html)
The biggest downside of nushell is that it hasn't hit 1.0 yet so commands occasionally get renamed. Expect that you may occasionally need to tweak a script to get it working again. Definitely a pain point.
reply