> One model drafts a result, an independent read-only critic from a different model family reviews it
Multiple model vendors is key here, the cascade pattern doesn't need it, but the critique pattern does.
Last Nov, my team wrote a paper ("Team of Rivals") on the difference between using an OpenAI model to Critique an Anthropic model's output vs running a self-review agent loop on the same vendor.
The ablations [1] proved that neither company alone was better than using both.
The paper was a general response to "What does your company do that Anthropic can't?" but more so a demonstration of how to make something 90%+ good with models which eval at 60% or so (& Gas Town post unblocked our "this is a trade secret" argument about the paper).
There are other quite elegent methods for triangle and simplices.
For a triangle, drawing α and β uniform over [0,1) the barycentric coordinates given by (1-sqrt(α), sqrt(alpha)(1-β), βsqrt(alpha)) is uniform over the triangle. No rejection and no test for flipping.
For simplices (triangle, tetrahedron, 5-cell etc) barycentric coordinates obtained by drawing uniformly from (0,1] taking a log and normalizing will be uniform within the simplex.
I wrote about this and other related sampling below.
There are places I've found the topological perspective useful, but after a decade of grappling with trying to understand what goes on inside neural networks, I just haven't gotten that much traction out of it.
I've had a lot more success with:
* The linear representation hypothesis - The idea that "concepts" (features) correspond to directions in neural networks.
* The idea of circuits - networks of such connected concepts.
This is an interesting blogpost. While the general conclusion ("We need batching") is true, inference of mixture of experts (MoE) models is actually a bit more nuanced.
The main reason we want big batches is because LLM inference is not limited by the compute, but my loading every single weight out of VRAM. Just compare the number of TFLOPS of an H100 with the memory bandwidth, there's basically room for 300 FLOP per byte loaded. So that's why we want big batches: we can perform a lot of operations per parameter/weight that we load from memory. This limit is often referred to as the "roofline model".
As models become bigger, this does not scale anymore because the model weights will not fit into GPU memory anymore and you need to distribute them across GPUs or across nodes. Even with NVLink and Infiniband, these communications are slower than loading from VRAM. NVlink is still fine for tensor parallelism, but across nodes this is quite slow.
So what MoE allows is expert parallelism, where different nodes keep different experts in memory and don't need to communicate as much between nodes. This only works if there are enough nodes to keep all experts in VRAM and have enough overhead for other stuff (KV cache, other weights, etc). So naturally the possible batch size becomes quite large. And of course you want to maximize this to make sure all GPUs are actually working.
jsonnet[1] and kapitan[2] are the tools I currently use. Their learning curve is not optimal (and I tried to contribute to smoothen it with a jsonnet course[3] and a 'get started wit kapitan' blog post[4]), but once used to it it's hard to do without, and their combination makes them even more useful (esp. if you deploy K8s).
In Ruud's case, Jsonnet might have been worth looking at as Hashicorp tools can be configured with json in addition to HCL. But that would have been less fun I guess ;-)
I hope for Ruud it finds its niche, there's quite some competition in this field!
I worked with Ward Cunningham for about a year, and he said once that he regretted coining the phrase “technical debt.” He said it allowed people to think of the debt in a bottomless way: once you’ve accumulated some, why not a little more? After all, the first little bit didn’t hurt us, did it?
The end result of this thinking is the feature factory, where a company only ever builds new features, usually to attract new customers. Necessary refactors are called “tech debt” and left to pile up. Yes, this is just another view of bad management, but still, Ward thought that the metaphor afforded it too easily.
He said he wished instead that he’d coined “opportunity,” as in, producing or consuming it. Good practices produce opportunity. Opportunity can then be consumed in order to meet certain short-term goals.
So it flips the baseline. Rather than having a baseline of quality then dipping below it into tech debt, you’d produce opportunity to put you above the baseline. Once you have this opportunity, you consume it to get back to baseline but not below.
I’m not convinced that the concept phrased thus would have the same traction. Still, I love this way of looking at it, like I love much of Ward’s POV on the world.
At Netflix, we're doing a mix of what Dan calls "CPU Pinning and Isolation" (ie, host-level scheduling controlled by user-space logic) [1] and "Oversubscription at the cluster scheduler level" (through a bunch of custom k8s controllers) to avoid placing unhappy neighbors on the same box in the first place, while oversubscribing the machines based on containers usage patterns.
- Reducing DNS calls and server round trips. Loading fewer resources from fewer domains makes a huge difference. Using server push also helps, although it might get deprecated.
- Responsive images. Load small images on small displays. Don't load 2x images on lower density displays.
- Using the right load event for JavaScript. load fires long after DOMContentLoaded on some pages, especially on slow connections.
- Setting the size of elements before they load, to prevent the content from jumping all over the place.
- Loading fewer fonts, and serving them yourself. Google Fonts is significantly slower.
- Gzipping everything you send. It makes a significant difference on slow connections.
- Static caching. This will shave whole seconds off your page load time. Cache both on the server and in the browser, but mostly on the server.
- Consider perceived performance too. It doesn't matter how fast you force a coercive cookie notice on your readers.
- Performance extends to the content of the page. Don't make users chase answers. Give clear, concise information and use typography to guide your users to it.
I've been looking into making something like that myself. My beef with current platforms is that they are good for extracting metrics and routing logs to their correct team (by service/deployment/...) but not so much for analysis.
When there's a problem in my system, I want to bring up multiple views, manually annotate, highlight the new debug log statements I just added, mark events as good/bad from complicated rules, etc.
None of the current systems (except maybe Grafana+Loki) really do this, they focus on ahead-of-time ingestion rules and tags. I would love a desktop or web tool that can do that, working from a log file that I can easily extract from my log platform.
I really enjoyed taking Ullman's Automata course on Coursera. I found it was great for better appreciating topics like
* searching
* implementation of automata in electronic circuits
* challenges of formal specifications for things like protocols and grammars, as well as for verifying their correctness; implementation strategies for applying these specifications
* computability and complexity
* programming language theory
* history of computer science
* LANGSEC arguments
in addition to having an austere mathematical beauty.
I did Strang's linear algebra course[0] (link goes to Youtube playlist of lectures) several years after graduating and recommend it highly. I was looking for refresher but I gained a deeper understanding of several important concepts; in particular it's fair to say I barely understood, or perhaps even misunderstood, SVD until Strang. If you're not sure if you need something like that, I suggest doing something like testing yourself on this video[1] or the MIT problem sets[2] it's easy to tell yourself that you "know" linear algebra when it would be closer to the truth to say that you used to know linear algebra, but can't answer even basic questions today. After Strang, Golub's book on Matrix Computations is also really incredible.[3]
So I've been at places that literally wouldn't give me a title increase with no monetary benefit. I'd saved them boatloads of money and doing far more than my job description. Upper management had a bonus structure based on cutting costs. Not actually performing the job, just how much they could cut.
I left that place and immediately made nearly 20k more per year, then next job another 25k, then another 20k and I'm now in a job where I get to set my own hours, work from home and get paid stupid amounts of money getting raises and bonuses I haven't even asked for. To this day I have no idea how I've gotten so damned lucky.
Some general things that have helped me, and believe me I started from self taught tech skills, poor people skills, and massive introversion, so chances are you're smarter than I am and able to do just the same if you're willing to put in the work.
1. Luck - not much you can do to get lucky, but you can stack the deck in your favor by making sure you're ready when your opportunity does come.
2. Know what you're looking for in advance. That means the salary, type of work environment, and position you're working for. If you're comfortable where you are position wise figure out how you're going to improve your skill set through either diversification or specialization. Diversification makes you more useful at startups, specialization at mid to mature companies. Being comfortable means you're not growing which means you're losing value from an employee standpoint because you're probably forgetting something every day.
3. Know how much you're looking for. Don't value yourself low. one way I heard which seemed to help was to start saying amounts in a mirror. When you got to a point where you thought it was silly ask for slightly higher. Worst they can say is no and you've lost nothing.
4. Be willing to walk when opportunity comes. Nothing says you're serious like walking away. Let's be honest if you're unhappy enough that you're looking externally, it's probably a sign you're ready to go anyways.
5. Be able to sell yourself. Know your past achievements and be able to show how your past achievements will be able to help the company make more money. Know what the company desires to do, who their current customers are, and what they're excited about. Nothing impresses a new company more than a prospective employee who has done their homework. I've literally had people come into interviews being told they would be asked about the company and been unprepared, they didn't get the job. For your current company, show them how you want to grow into a role that can give them more for the money. Being excited about your company and what they do is genuinely hard to find. Some jobs make it nearly impossible to do so, especially mature or more often dying companies.
6. People skills, people skills, people skills. If you suck at talking to people, get better. Toastmasters is good for speeches, church meetings for general conversations, friends groups, networking events, etc. If you're looking for step by step instructions look into the classic "How to Win Friends and Influence People" by Dale Carnegie. It's old but it's a classic. Vanessa Van Edwards is younger but also provides really great information on learning to read people's body language, something which doesn't come naturally to me. Even if you're not in sales or management read books aimed at them. Chet Holmes is solid for sales information, James Collins for management. Knowing how your salespeople bring in clients and how your management thinks means you can often be the person who is solving their problems before they come up. Good salespeople and managers take care of people who make their lives easier, but they also lean on them more. Just never make the mistake of thinking that because they like you, compliment you, or take care of you that you're their friend. Unless a relationship exists outside of work it's a work relationship. Good salespeople and managers can make you feel great even when they are screwing you over.
7. Start doing good things for other people before they do them for you. If you know someone in your network is out of work and you're not looking but a recruiter sends you a job that you know they might fit, send it along. If someone left a job for different pastures, leave them a genuine recommendation or send them a personal message letting them know how much you appreciated them. People too often don't let others know how much they appreciate good people, and no one hates being genuinely appreciated.
I think I've blabbered on for long enough, but these are just a few of the guidelines I've worked through to get to have an amazing job. Hope you find yours. :)
Dependencies (coupling) is an important concern to address, but it's only 1 of 4 criteria that I consider and it's not the most important one. I try to optimize my code around reducing state, coupling, complexity and code, in that order. I'm willing to add increased coupling if it makes my code more stateless. I'm willing to make it more complex if it reduces coupling. And I'm willing to duplicate code if it makes the code less complex. Only if it doesn't increase state, coupling or complexity do I dedup code.
The reason I put stateless code as the highest priority is it's the easiest to reason about. Stateless logic functions the same whether run normally, in parallel or distributed. It's the easiest to test, since it requires very little setup code. And it's the easiest to scale up, since you just run another copy of it. Once you introduce state, your life gets significantly harder.
I think the reason that novice programmers optimize around code reduction is that it's the easiest of the 4 to spot. The other 3 are much more subtle and subjective and so will require greater experience to spot. But learning those priorities, in that order, has made me a significantly better developer.
Multiple model vendors is key here, the cascade pattern doesn't need it, but the critique pattern does.
Last Nov, my team wrote a paper ("Team of Rivals") on the difference between using an OpenAI model to Critique an Anthropic model's output vs running a self-review agent loop on the same vendor.
The ablations [1] proved that neither company alone was better than using both.
The paper was a general response to "What does your company do that Anthropic can't?" but more so a demonstration of how to make something 90%+ good with models which eval at 60% or so (& Gas Town post unblocked our "this is a trade secret" argument about the paper).
[1] - https://github.com/t3rmin4t0r/critique-evals