Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Are you aware of any framework or could use for such tasks (ML and alike)?


The problem you face is not so much a lack of such things existing but such a staggering multiplicity that it is hard to poke through them all, and despite the sheer quantity, you may still find that your particular problem doesn't fit any of them terribly well. You can find anything from OpenMP, which amounts to "let's try to treat a whole bunch of resources a lot like one big computer" https://en.wikipedia.org/wiki/OpenMP, through things like Kubernetes which can be used to deploy all kinds of "worker nodes" for all sorts of tasks even if it doesn't do the orchestration of the task itself, and so many combinations of anything in between, plus you have all sorts of clustering technologies, message bus technologies, on-demand VMs spinning up, so many primitives that even if they aren't designed for this can be relatively easily put together into whatever it is you actually need that it is all rather about bewildering abundance than shortage.

There's also the entire category of "data lakes" and other nouns that have "data" applied as an adjective that includes various orchestration techniques because just being storage isn't enough, that is its own entire market segment.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: