> Either<Either<A,B>,A>> doesn't express our intent for a function return or parameter type if we don't care about the position of A, just whether it is an A
> so we'd want all nested variations normalized to Either<A,B>.
Sorry, perhaps my thinking is shaped by nominal type systems rather than structural, but if the only thing we care about is whether the type is A, then how do we end up having Either<Either<A, B>, A>> in the first place? Thinking about this in terms of a nominal type system, the specific type you present here has to have some specific meaning associated with, specifically, this type, otherwise we would have chosen some other type. So the key thing here is that if we have Either<A, A> then it HAS to be distinct from simply A, otherwise we wouldn't have this type in the first place. Us constructing it means we associate it with a specific meaning so it has to be distinct from A.
But if we DON'T care, then, I guess, we shouldn't use this type? Use the type we do care about? The same goes for Either<A, B> and Either<B, A>.
> or we need to hide the complexity by using more abstract tools like e.g. monad transformers
This is interesting, how do monad transformers relate to this problem?
Well, you might use a different custom named type than Either, but sum types only give you basically that meaning - you can't enforce the invariants we want. You could use other type mechanisms of your language (e.g. type classes or dependent types) to embed some form of set theoretic types and hopefully leak less of your abstraction (needing "bookkeeping" to keep the invariants) or deal with restricted forms (the type violating some of the invariants we want in some situations).
The examples above or Either<A,A> could result from polymorpic functions that would return a set of types that the function is abstracting about, something like: pickRandom<S,T> : S, T -> S|T. With Either<S,T> you would get pickRandom<A,A> a1 a2 : Either<A,A> (requiring cleanup if you want the invariants I wrote about), with set theoretic types you'd get A. If you have pickRandom<A|B, B|C> x y you would get nested Either's or just A|B|C respectively.
Either is a Monad and so Haskell and others allow us to hide a bunch of complexity of reducing nestings by using abstractions and custom magic syntax (do notation) built for them - but the underlying complexity of the type and necessary mental model remains. Monad transformers become a necessity because you already needed the Monad magic for the cleanup, but you also have another Monad you care much more about then Either (like IO), see e.g the answer here https://stackoverflow.com/questions/67617871/reduce-nestedne...
Note that this isn't talking about nested Either's, just the nested syntax for handling them without using it as a Monad and do notation, with actual nested Either's you'd need to do more cleanup.
> If you have pickRandom<A|B, B|C> x y you would get nested Either's
If this wasn't the case, how would the information about what you got be retained? It's either positional, or by a tag/key (row-polymorphic variants), or none retained.
I don't see why would you want to use monadic API for approaching an "anonymous sum type" problem in the first place. As I said before, there are fundamentally just 2 operations you would want to use: inject and project. Maybe you could also mention assoc for re-association but I'd say if you're using it you're likely handling the problem the wrong way. So I still don't see how monad transformers play into this. They are a nice (decent, at least) trick for dealing with some situations but the problem we're talking about here isn't one of them.
> so we'd want all nested variations normalized to Either<A,B>.
Sorry, perhaps my thinking is shaped by nominal type systems rather than structural, but if the only thing we care about is whether the type is A, then how do we end up having Either<Either<A, B>, A>> in the first place? Thinking about this in terms of a nominal type system, the specific type you present here has to have some specific meaning associated with, specifically, this type, otherwise we would have chosen some other type. So the key thing here is that if we have Either<A, A> then it HAS to be distinct from simply A, otherwise we wouldn't have this type in the first place. Us constructing it means we associate it with a specific meaning so it has to be distinct from A. But if we DON'T care, then, I guess, we shouldn't use this type? Use the type we do care about? The same goes for Either<A, B> and Either<B, A>.
> or we need to hide the complexity by using more abstract tools like e.g. monad transformers
This is interesting, how do monad transformers relate to this problem?