Hacker Newsnew | past | comments | ask | show | jobs | submit | ayuhito's commentslogin

DuckDB has one of my favourite articles on this topic if you want something a little more high level: https://duckdb.org/2022/10/28/lightweight-compression

> My experience with GHA default caches is that it’s absolutely dog slow.

For reference, oven-sh/setup-bun opted to install dependencies from scratch over using GHA caching since the latter was somehow slower.

https://github.com/oven-sh/setup-bun/issues/14#issuecomment-...


I strongly relate to this in many ways.

Because of OSS, I’ve never actually applied for a job or done a Leetcode interview. I’ve gotten multiple direct offers through Twitter DMs (I don’t post) and multiple referrals through random encounters that I never used.

E.g. Debugging an interesting issue with GitHub customer support eventually led to a referral for Microsoft by an MD. Similar stories with Cloudflare and more.

It’s not limited to OSS, but just having any sort of backing credibility to your name without going through the whole CV/CL process unlocks a whole slew of opportunities since people can “pre-screen” you from the start.


You're sort of describing networks more broadly whether you initiate a connection or someone at a company does. Latterly, I didn't apply in the usual sense for 25 years or so.

Don't code a lot but have written books which led to book signings at conferences that probably led to other opportunities if I had the need to exploit them.


I feel it’s an evolution of the term “Devrel” which still feels tacky.

Nor would you want someone who built most of their career as an actual engineer to suddenly drop that term and become a generic someone in “marketing”. They’re more than that for sure.

I quite like the terminology the more I think about it.

https://github.com/aarondfrancis


Totally. An engineer, who (at the the time) works in marketing! Makes sense to me :D


Also worth checking out libxev[1] by Mitchell Hashimoto. It’s a Zig based event loop (similar to libuv) inspired by Tigerbeetle’s implementation.

[1] https://github.com/mitchellh/libxev


Also, the Zig 0.16.0 preview nightly builds includes a new Io library[0]. I have not used libxev or Tigerbeetles event loop, but I believe the standard Zig implementation is likely largely influenced by those two.

[0] https://ziglang.org/documentation/master/std/#std.Io, or https://ziglang.org/documentation/0.16.0/std/#std.Io after the release


I’m curious, how do you know it was inspired by tiger beetles impl?

They look very similar so that makes sense, just curious on the order of events.

Also I tried using libxev for a project of mine and found it really broke the zig way of doing things. All these callbacks needed to return disarm/rearm instead of error unions so I had to catch every single error instead of being able to use try.

I could have reworked it further to make try work but found the entire thing very verbose and difficult to use with 6 params for all the callback functions.

Thankfully my use case was such that poll() was more than sufficient and that is part of zigs posix namespace so that was what I went with.


> I’m curious, how do you know it was inspired by tiger beetles impl?

Describing its design, the readme for libxev says "credit to this awesome blog post" and links to the same Tigerbeetle post in this submission.


I recently created this, which is largely based on ideas from libxev, but my optimized for my zio runtime. https://github.com/lalinsky/aio.zig


> I'd love to know how much LLM was used to write this if any, and how much effort went into it as well (if it was LLM-assisted.)

Are people supposed to be obligated to post such a report nowadays?

I enjoyed the article and found it really interesting, but seeing these types of comments always kind of puts a damper on it afterwards.


> Are people supposed to be obligated to post such a report nowadays?

No, typically when I ask questions it's optional.

> I enjoyed the article and found it really interesting, but seeing these types of comments always kind of puts a damper on it afterwards.

That is why I waited half a day, and until after there were lots of comments praising the article. Still, I'm sorry if it put a damper on it for you.

Also the whole reason I asked about the source is because I think the article has a lot of merit and so I am curious if it's because the author put a lot of work in (LLM-assisted or not.) Usually when I get that feeling it's followed by a realization I'm wasting my time on something the author didn't even read closely.

But I didn't get that this time, and I'd love more examples of LLMs being used (with effort, presumably) to produce something the author could take pride in.


> But I didn't get that this time,

Actually, I take it back. I did think I was wasting my time when I noticed it was written by an LLM. But then I came back to HN an saw only praise and decided to wait a bit to see if people kept finding it useful before commenting.

I was somewhat excited by the prospect of this article being useful, but I've started to come around to my initial impression after another day. I don't really trust it.


In fact, v1 code usually uses v2 code under the hood, but with different options to maintain backwards compatibility.

You still get performance improvements even if you don’t switch over to the new import!


A good example is io/ioutil. It's useful to migrate to eliminate the deprecation messages, but you don't need to do it right away.


Also, most of this can be automated with `go install golang.org/x/tools/gopls/internal/analysis/modernize/cmd/modernize@latest && modernize -fix ./...`


“Chemicals” are overused as a term for sure, but there is a huge difference between what’s legal in America and Europe that brings a shred of truth to the previous statements.

For example, common ingredients like potassium bromate or ADA are straight up banned in the EU for health concerns.

Reading the ingredient list of American bread is plain shocking at times.


And there are a handful of chemicals banned in the US for health concerns that the EU is fine with.


Individual cases are interesting. For example, Wikipedia says this of E122:

> In the US, this color was listed in 1939 as Ext. D&C Red No. 10 for use in externally applied drugs and cosmetics. It was delisted in 1963 because no party was interested in supporting the studies needed to establish safety. It was not used in food in the US.

> Azorubine has shown no evidence of mutagenic or carcinogenic properties and an acceptable daily intake (ADI) of 0–4 mg/kg was established in 1983 by the WHO.

Wikipedia's article on E180 is a stub. Wikipedia's article on E105 says it's now banned in both the US and EU, but it doesn't say when it was banned: did the US ever approve it?


Go also suffers from this form of “subtle coloring”.

If you’re working with goroutines, you would always pass in a context parameter to handle cancellation. Many library functions also require context, which poisons the rest of your functions.

Technically, you don’t have to use context for a goroutine and could stub every dependency with context.Background, but that’s very discouraged.


Having all async happen completely transparently is not really logically possible. asynchronous logic is frequently fundamentally different from synchronous logic, and you need to do something different one way or the other. I don't think that's really the same as "function colouring".

And context is used for more than just goroutines. Even a completely synchronous function can (and often does) take a context, and the cancellation is often useful there too.


Context is not required in Go and I personally encourage you to avoid it. There is no shame in blazing a different path.


Why do you encourage avoiding it? Afaik it's the only way to early-abort an operation since Goroutines operate in a cooperative, not preemptive, paradigm. To be very clear, I'm asking this completely in good faith looking to learn something new!


> Afaik it's the only way to early-abort an operation since Goroutines operate in a cooperative, not preemptive, paradigm.

I'm not sure what you mean here. Preemptive/coorporative terminology refers to interrupting (not aborting) a CPU-bound task, in which case goroutines are fully preemptive on most platforms since Go 1.14, check the release notes for more info. However, this has nothing to do with context.

If you're referring to early-aborting IO operations, then yes, that's what context is for. However, this doesn't really have anything to do with goroutines, you could do the same if the runtime was built on OS threads.


Goroutines are preemptive only to the runtime scheduler. You, the application developer merely using the language, cannot directly preempt a goroutine.

This makes goroutines effectively cooperative still from the perspective of the developer. The preemptive runtime "just" prevents things like user code starving out the garbage collector. To interrupt a goroutine, your options are generally limited to context cancelation and closing the channel or socket being read, if any. And the goroutine may still refuse to exit (or whatever else you want it to do), though that's largely up to how you code it.

This difference is especially stark when compared with Erlang/BEAM where you can directly address, signal, and terminate its lightweight processes.


Exactly, thank you. I knew that the runtime gained the ability to preempt but the clear fact that you cannot get a handle to a goroutine (eg `gr := go fn()` is proof you have no way to take advantage of this ability as a user.


You are expecting them to actually check the value, there is nothing preemptive.

Another approach is special messages over a side channel.


So instead of a context you need to pass a a channel. Same problem.


Not necessarily, that is one of the reasons OOP exists.

Have a struct representing the set of associated activities, owning the channel.


soo, a context?


You can take that view, yes.

But if you store your context in a struct (which is not the recommend “best practice” – but which you can do) it's no longer a function coloring issue.

I do that in on of my libraries and I feel that it's the right call (for that library).


If the struct has a well-scoped and short-lived lifecycle, then it is actually better to put the context in the struct. Many Go libraries including the stdlib do this despite not being "best practice".

An exception to the short-lived rule is to put context in your service struct and pass it as the base context when constructing the HTTP server, so that when you get a service shutdown signal, one can cancel requests gracefully.


It's well scoped, but not short lived; it's an SQLite connection.

But the API surface is huge, with 100s of methods on the connection and derived objects, with it being unclear which might block and be worthy of asynchronous cancellation. You never know when pulling an additional column if that one might be an overflow text/blob that does additional IO.

The solution, while not amazing is a method that you use like this:

  old := conn.SetInterrupt(ctx)
  defer conn.SetInterrupt(old)
This changes the “interrupt” context for the duration of your function scope, and covers all potentially blocking calls that you might make. Also, from the name, it's quite clear that this context is used only for interruption/cancellation (interrupt is the SQLite name for this, which I try to adhere to).


Nope, because I didn't mention it was to be passed around as a compulsory parameter, rather have your logic organised across structs with methods, hide most details behind interfaces.


It's not required, but eschewing it ends up going against the grain, since so much of the ecosystem is written to use contexts, including the standard library.

For example, say you instead of contexts, you use channels for cancellation. You can have a goroutine like this:

    go func() {
      for {
        select {
        case <-stop:
          return
        case <-time.After(1*time.Second):
          resp := fetchURL(url)
          processResult(resp.Body)  // Simplified, of course
        }
      }
    }()
If you want to be able to shut this goroutine down gracefully, you're going to have an issue where http.Get() may stall for a long time, preventing the goroutine from quitting.

Likewise, processResult() may be doing stuff that cannot be aborted just by closing the stop channel. You could pass the stop channel to it, but now you're just reinventing contexts.

Of course, you can choose to only use contexts where you're forced to, and invent wrappers around standard library stuff (e.g. the HTTP client), but at that point you're going pretty far to avoid them.

I do think the context is problematic. For the purposes of cancellation, it's invasive and litters the call graph with parameters and variables. Goroutines really ought to have an implicit context inherited from its parent, since everything is using it anyway.

Contexts are wildly abused for passing data around, leading to bloated contexts and situations where you can't follow the chain of data-passing without carefully reviewing the entire call graph. I always recommend not being extremely discriminating about where to pass values in context. A core principle is that it has to be something that is so pervasive that it would be egregious to pass around explicitly, such as loggers and application-wide feature flags.


What would you use in its place? I've never had an issue with it. I use it for 1) early termination 2) carrying custom request metadata.

I don't really think it is fully the coloring problem because you can easily call non-context functions from context functions (but not other way around, so one way coloring issue), but you need to be aware the cancellation chain of course stops then.


I think the main point is in something like Go, the approach is non-viral. If you are 99 levels deep in synchronous code and need to call something with context, well, you can just create one. With C#, if you need to refactor all 99 levels above (or use bad practices which is of course what everyone does).

Also, in general cancellation is something that you want to optionally have with any asynchronous function so I don't think there really exists an ideal approach that doesn't include it. In my opinion the approach taken by Zig looks pretty good.


Like you said you dont NEED context. Its just something thats available if you need it. I still think Go/Erlang has one of the best concurrency stories out there.


> If you’re working with goroutines, you would always pass in a context parameter to handle cancellation.

The utility of context could be called a subtle coloring. But you do NOT need context at all. If your dealing with data+state (around queue and bus processing) its easy to throw things into a goroutine and let the chips fall where they will.

> which poisons the rest of your functions. You are free to use context dependent functions without a real context: https://pkg.go.dev/context#TODO


The thing about context is it can be a lot more than a cancellation mechanism. You can attach anything to it—metadata, database client, logger, whatever. Even Io and Allocator if you want to. Signatures are future-proof as long as you take a context for everything.

At the end of the day you have to pass something for cooperative multitasking.

Of course it’s also trivial to work around if you don’t like the pattern, “very discouraged” or not.


Good thing most of my tasks don’t require novelty, just working code.


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: