I love OpenTelemetry and we want to trace almost every span happening. We’d be bankrupt if we went any vendor. We wired opentelemetry with Java magic, 0 effort and pointed to a self hosted Clickhouseand store 700m+ span per day with a 100$ EC2.
Professional managers, particularly American MBA types is what destroys businesses. Not KPIs, Agile, Scrum or whatever the fad of the moment to blame may be.
Management is a service task, it does not produce value itself. Good management is important to prioritize, allocate resources and resolve conflicts. But that is not how MBAs seem themselves, they see themselves as the most important thing around, they're the ones in charge after all. And damn physics and reality, the spreadsheets say you can get get a baby out in a month if you use 9 women to do it. Soon management captures all the attention and sucks away all the money, and shortly thereafter end up with a moribund shell of the former company. You can tell how close to moribund a company is by looking at the manager to worker ratio. Get out of there once it climbs much above 1 to 10.
There are good managers around, but they're almost as rare as unicorns.
My brother passed away very suddenly a few years ago, and I was put in charge of wrapping up and archiving his "digital" life. We were very lucky that we had access to a recovery email for his main gmail account (as well as a couple of passwords that his partner knew) and was able to access and archive virtually all data we could think of (services like Google Takeout were invaluable). I realized that if this had happened to me, it would have been virtually impossible to do, as all my passwords and credentials are in my password manager, and the password to that was only in my head.
It's a good thing to plan for this eventuality, to make it easy for your family and friends to wind up your "digital life" after you've passed. 1Password has a very good solution for this, with a "recovery document" you can print out and write down your password on, which contains instructions anyone else would need to access your 1Password account. I gave a copy of this document printed out to a small number of people I trust implicitly.
You never know when something sudden can happen to you. For the sake of those you leave behind, it's a nice gift to plan for this eventuality, even if it seems far off at the moment.
This is not a great interview question. It’s not even a good interview question. It barely evaluates any technical abilities and it ignores most technical abilities that matter.
This question could be improved as follows:
* present the naive solution. Explain that this is a PR that was submitted in 20 minutes by our new intern from Waterloo. Ask them to help improve it.
* suggest to them the intern’s argument why the lookup is O(1) and explain that it’s actually O(n). Ask them to write a PR comment to break the news to the intern.
* explain the optimal solution with tries. Ask what happens if the vocabulary is big enough—and the prod environment is resource constrained—so that the trie is rarely in memory when the function is called. How would the trie solution compare to a binary search through a length normalized list of words?
This kind of question is barely useful to interview new hires—when you still may have basic questions about how much undergrad material “stuck”. Within a few years when they hit full-performance level you should care much less about this kind of question and much more about “higher order” thinking about engineering.
How do you communicate bad news? How do you identify, communicate, and mitigate risk? How do you plan large tasks at at appropriate level without over engineering the solution? What questions should you ask at different project stages to capture the right requirements, anticipate issues, and successfully execute over time?
But whatever, I’m sure you get lots of signal out of repeating “a hash table lookup is O(1)” “nuh uh” on loop for 10 years. :eyeroll:
I did enjoy reading the preview book for EventStorming, seems like a decent format for getting stakeholders involved in the design process and capturing business details. I'm pretty sure you can also do this with an org-mode doc and emailing the right questions to the right people, but discovery is hard and having a big post-it note brain storm with 30 people does sound like more fun!
I don't know, I feel like tech people are regularly amazed that other people can describe the events, processes, and data needs of their jobs if you just ask them. Likewise I grant that many people are poorly onboarded and end up "faking it to make it" in their careers. I've seen similar "people lighting up as they finally understand what the company does" moments through the use of literally any form of high-level modeling of the business. People are starved for context at most jobs.
Making telemetry opt-in would not have fixed this issue, the author is simply using this example to make an unrelated point about telemetry.
The real solution here is to remove all optional code from the critical path - this not only includes telemetry, but also things like in-app tutorials, automatic updates, crash reporting... If some code is not implementing a core function of the application, it should be allowed to crash and the application continue working. This is a core principle of good software engineering and it needs to be talked about more.
> “One of the most common misconceptions about microservices is that they solve a technology problem,” says Johnston. “But really they solve a business or organization scalability problem.”
I disagree. Service oriented architecture (note I don't say microservices, because I think that often goes too far) does solve technical problems. But they are problems that articles like this often ignore (maybe the authors have some misconceptions about microservices themselves).
For me the main benefit of seperate services isn't about organization of code or teams, it is about isolation.
The technical problems I think service seperation solves are:
How do I ensure that a security vulnerability in X doesn't compromise Y (least privilege)?
How do I minimize the blast radius if an unusual request or traffic patternconsumes an unexpected amount of resources (related to above, but doesn't necessarily need to be malicious)?
How do I deploy a change to X without having to rebuild and retest everything else?
How do I give a group of people access to manage X without giving them permissions for Y.
How do I optimize usage of some resources? (Note: multiple services have overhead, so for this to apply, you need to have enough scale that the benefits of fine-grained scaling and rightsizing of individual services outweighs the costs, but contrary to some comments you don't have to be quite as big as google for it to matter).
For some of these,"microservices" are not the only solution, and depending on your exact situation it may or may not be the best solution.
The important thing is to understand what specific problems you are trying to solve, why, and what the tradeoffs are.
I grew up a geek in a rural area, learning how to filter conversations for your audience was a critical skill.
Topics and conversational style amongst my D&D group was very different than amongst my other high school classmates and very different than when at my summer jobs.
This is just an article about people who never learned that skill.
I'm reminded of a few moments in my life that drove this home ...
A coworker (60 year old railway worker) who said "You shouldn't use ten dollar words in a twenty-five cent conversation."
A girlfriend who described the difference as "Weird is when you know you're weird and compensate when needed, strange is when you don't know you're strange."
2 minutes in, Paul types out [xfs] and then later [xtc] — what do you think these symbols are for?
My best guess from watching the video is that they are temporary place holders, breadcrumbs he leaves himself to return to, allowing him not to break his flow/train of thought. If that's the case, then the technique reminds me of Neil Strauss's "Power of the TK":
When you get suck on a word, or cant’ think of the right fact, what shouldn’t you do?
ImageOptim is fantastic. You can use it via CLI, so I've added it to a few different archiving / publishing workflows in the past. One thing that some people don't realize is you can run it multiple times on the same (PNG) images and get better results. Each filter is dependent on the input from the previous filter, so running them again can give you better results even after one run. This works out because filter 3 might give you some savings or rearrange the data in a way that filter 1 can now take advantage of.
The other JPG filters are lossless, but be aware that Guetzli is lossy!
I was impressed with just how much it could losslessly compress some massive JPGs until I did a visual diff. I can't see the difference, like one-bit deltas sort of thing, but it's not 1:1 lossless as I would expect.
First, a key limitation that every architect should pay attention. Redis reaches the limits of what you can do in well-written single-threaded C. One of those limits is that you really, really, *really* don't want to go outside of RAM. Think about what is stored, and be sure not to waste space. (It is surprisingly easy to leak memory.)
Second, another use case. Replication in Redis is cheap. If your data is small and latency is a concern (eg happened to me with an adserver), then you can locate read-only Redis replicas everywhere. The speed of querying off of your local machine is not to be underestimated.
And third, it is worth spending time mastering Redis data structures. For example suppose you have a dynamic leaderboard for an active game. A Redis sorted set will happily let you instantly display any page of that leaderboard, live, with 10 million players and tens of thousands of updates per second. There are a lot of features like that which will be just perfect for the right scenario.
I'm fond of Vale, where you can write your own rules or grab others off github. I wish we could crowd source a grammarly clone using it. Some efforts have been made to do so, but didn't get far.
After 20 years of programming, this is what comes to my mind when I hear "How do Design Programs":
1: Start with the data structure
The tools to operate on the data can be rewritten or improved without much pain. Changing the data structure is what hurts. If the data structure is elegant, the code will fall into place. If the data structure is messy, the code will grow into a giant hairy ball of madness.
2: Don't use a DB. Use human-readable text files
It makes everything so much easier. You can see if your data structure is elegant. Because then the files will be easy to read and navigate.
You will be able to start up your software 20 years from now without any installation and without having to go through the hell that is setting up the environment and updating dependencies. The website you are reading this on, Hacker News, stores all data in text files.
3: Build small tools to operate on the data
Like Linus started git by writing small tools "git-add", "git-commit", "git-log" etc. On the web you can have one url to do one thing (/urser, /item, /edit here on HN) and map them to one code file on your disk. If you want to use a framework like Django, Laravel or Express, those will unfortunately make it a bit harder to do that. But you can still stick somewhat closely to this approach.
4: Let people use it right away
If you are building it for yourself, you already have a user. Great. If not, let your target group use it right away. Otherwise, you will waste many years of your life building software features nobody understands or even wants.
I'll second the recommendation above. For folks coming to algorithms for the first time I would recommend _Grokking Algorithms
An illustrated guide for programmers and other curious people_ as a nice quick _tour_ of the field.
And the _Common Sense Guide_ (mentioned above) as the next book. For complete beginners this book has the best simple explanations of algorithms I've read in any book (and I have a whole bunch of them).
A major component of a strategy is constraint. In other words, what are you going to give up? If you're going to do All The Things, that isn't a strategy, it's a wishlist. You have to bring constraints into the picture before anything like a strategy emerges.
CPU utilization. Modern CPU can execute like 4-6 uops per cycle IIRC. Multiply that by clock freq and number of cores and you get theoretical max. Then take your executed instructions per sec and divide by this theoretical max. The better the ratio, the more efficient your program is.
I worked at Google and been on teams where OKRs are used very ineffectively. Since then, I've read one thing that really changed the way I think about "goals" and could alleviate the problem you mention here. It's called the product strategy stack:
If you have time, listen to the podcast. This is really the most comprehensive treatment I have seen related to OKRs and goal setting. Basically, goals are the last thing you consider. What is often missing is the high levels of the stack clearly articulated such that the goals make sense and measure progress towards a strategic outcome:
"Our strategy is to increase revenue by 5%’ or ‘Increase retention by 10%.’ That’s not a strategy, that’s a goal. It’s great if you can achieve that goal, but only if it’s actually part of a larger strategy that the company is trying to advance,” Mehta says.
“I often see teams get into a mode where they’re just doing anything and everything to move the goal, without actually realizing they’re headed in the wrong direction from a strategic standpoint to create long-term value.” "
I would respectfully say that you are wrong. I speak from experience. At Netflix we tried to hire for around the clock coverage. But what ended up working much better was taking that same team and having each person on call for a week at a time, all based in Pacific Time.
Yes, you would get calls at 2am, sometimes multiple days in a row. But you were only on call once every six to eight weeks, and we scheduled out well in advance so you could plan your life accordingly.
As a bonus, for the five weeks you weren't on call, you were highly incentivized (and had the time) to build tools or submit patches to fix the problems that woke you at 2am.
> It requires (oh, the horror!) actually maintaining production standards, runbooks, and other documentation.
I disagree with this too. Documentation and runbooks are useless in an outage. Instead of runbooks, write code to do the thing. Instead of documentation, comment the code and build automation to make the documentation unnecessary, or at least surface the right information automatically if you can't automate it.
My most eye opening leadership experience was last year. I was made tech lead of a small team, and asked to to accomplish something with a hard and somewhat unrealistic deadline.
It was a success, but it had nothing to do with me being an incredibly inspirational leader. Or being a brilliant developer - I don't think I actually contributed any code. It honestly wasn't much to do with me at all. I basically just set the stage for them to work effectively:
- I booted out all the non-coding people from our standup. The CIO in particular would waffle on, and it was draining everyones will to work (and possibly live).
- I ruthlessly pared down requirements. We had some written specs, and whenever there was a question about which way we should go I chose the one that either didn't involve us doing anything, or was easiest.
- I handled dealing with the stakeholders myself, and reported back to them the relevant info.
The end result was that they were the first dev team to deliver something on time in the history of that company. No one had to work any over time (demo was Friday mid-day, so I got clearance for them to leave early).
So yeah, the best project I ever lead, I barely 'lead'. I was a supporting player. I played interference. I made sure everyone left them the hell alone, that they had the information they needed, and told them that if it fucked up it was all my fault so relax (always blame the contractor).
And and they pretty much did it all by themselves. Turns out developers can develop if you trust them and get out of their way.
I think this article could have done a better job explaining the underlining data structures, which would have then given the readers much better intuition why this can actually work, and when.
In a nutshell, a table can be:
- Heap-based: the rows are in the unordered "heap", with any number of indexes (B-trees) pointing to them physically (by holding the row's physical address).
- Or clustered (aka. index-organized): the rows are in a primary/clustered index (B-tree), with any number of secondary indexes (also B-trees) pointing to them logically (by holding the copy of the primary key).
Fetching a row through any index in a heap-based table, or any secondary index in a clustered table, requires a separate step: physical heap access in the former case, or a clustered index seek in the later.
So...
- if you are using a heap-based table,
- or using a clustered table where pagination order differs from the clustering order
...then it makes sense to avoid the price of fetching the entire row for those rows that you are going to discard anyway. You just scan the index that matches the pagination order until you reach the correct page, and only then start fetching the rows.
It's still "offset" pagination, but can be much faster in practice than the naive implementation (for later pages).
OTOH, if you are paginating in the clustering order, then there is no separate step for fetching the row, so this method would be unlikely to bring any benefit.
---
A DBMS with a decent query planner should be able to do this kind of optimization automatically, through "predicate pushdown". But it may still be a good idea not to rely on it and make things blindingly obvious to the planner by structuring the SQL as shown in the article.
---
BTW, "cursor pagination" is also known as "keyset pagination" or "seek method". Curiously, actual database cursors can be used for pagination, assuming very tight transactional coordination between client and server (and very nasty consequences if that's lacking), but that's different from the "cursor pagination" as mentioned in the article.
This definitely resonates. I've often talked about how the key transition to 'lead engineer' is when you learn how to wield a team of programmers to solve a technical problem, instead of solving it yourself, and to higher levels it's when you learn how to wield an entire organization to some technical purpose.
These ARE much clunkier tools than a compiler.
Another way I've expressed it is how the tool you use to get started on implementing a change shifts as you grow more senior (if you'll forgive an IDE-centric kind of view - this is meant to reflect life in a Big Org - feel free to substitute with appropriate emacs commands)
- Junior Dev: File > Open
- Dev: File > New
- Senior Dev: File > New Project
- Lead Dev: File > New Diagram
- Principal Dev: File > New Powerpoint Presentation
It saddens me that a lot of people don't use debuggers and default to adding print statements. As far as I can tell, it's for several reasons:
1. The debugger is primitive (e.g. Godot GDScript - no conditional breakpoints or watches).
2. The debugger is unstable (e.g. Android Studio - frequently hangs, or takes a long time to populate data needlessly)
3. The debugger's UI is not friendly (e.g. Android Studio - hitting a breakpoint in multiple threads causes unexpected jumps or loss of current state; VSCode C++ debugger - doesn't display information properly or easily (arrays of objects) or displays too much information (CPU registers, flags, memory addresses); C++ debugger for D - doesn't display D data types).
4. The debugger is not properly integrated into the environment - can't find symbols, libraries or source files, or finds the wrong source files, etc. Need to jump through hoops to configure those.
5. Platforms don't support debuggers properly (e.g. again Android - ANRs when debugging the main thread, can't leave a debugging session overnight without some timer killing the process)
6. Developers got used to the workflow of "add a print statement, rerun and check the console" since high school and nobody taught them a more powerful tool
7. Developers code all day, so adding print statements by coding feels more natural than switching to the debugger's UI and way of doing things. (e.g. "if (i == 100) console.log(value)" allows you to stay in the same code, as opposed to setting a breakpoint, finding out how to add the 'i == 100' condition and pray that there's no issue with variables being optimized out at runtime).
I like Replay's features and that it's improving the state of the current tools. At the end of the day, adding print statements in Replay doesn't seem to affect the state of the application, so in that sense it's similar to gdb commands in that it's just a UI choice, but I wouldn't go as far as encouraging print-based debugging.
Outside of Replay, print-based debugging is still a primitive way of analyzing the state of the app and promoting this state of affairs reduces the pool of people who use and would hopefully improve the existing debuggers.
We all appreciated Firebug and the Chrome DevTools because of the powerful features they give us to inspect the state of the application. Imagine a person who adds print statements to their code every time they want to inspect the DOM or check the current CSS attributes. It works, but we have better tools, and we should make them even better.
Taking this opportunity to promote the author of this article (Geoffrey de Smet) and the project he's been the lead of for some very long amount of time (Optaplanner):
If you've never heard of constraints programming -- it's really interesting (personal opinion).
It lets you programmatically state "rules" and "goals" of some system, and solvers will automatically attempt to find the best solution.
Optaplanner is one of the most advanced of such engines that I'm aware of:
Under the hood, OptaPlanner combines sophisticated Artificial Intelligence optimization algorithms (such as Tabu Search, Simulated Annealing, Late Acceptance and other metaheuristics) with very efficient score calculation and other state-of-the-art constraint solving techniques.
But it's also really approachable since they re-wrote it in 2020 to accept regular code for rules, instead of the custom DSL language they used before (Drools).
I got interested in this area some years ago, and reached out to Geoffrey personally. He took the time to reply to me and answer a few questions. Great person, he's brilliant and super passionate about the domain.
If you have any sort of practical usecase for this (scheduling, vehicle routing, cost optimization, etc) and this sounds at all interesting, highly recommend giving it a shot!
As an example of what you can do in a few lines with this, here's one from the docs:
The following code snippet first groups all processes by the computer they run on, sums up all the power required by the processes on that computer using the ConstraintCollectors.sum(… ) collector, and finally penalizes every computer whose processes consume more power than is available.
private Constraint requiredCpuPowerTotal(ConstraintFactory constraintFactory) {
return constraintFactory.from(CloudProcess.class)
.groupBy(CloudProcess::getComputer, sum(CloudProcess::getRequiredCpuPower))
.filter((computer, requiredCpuPower) -> requiredCpuPower > computer.getCpuPower())
.penalize("requiredCpuPowerTotal",
HardSoftScore.ONE_HARD,
(computer, requiredCpuPower) -> requiredCpuPower - computer.getCpuPower());
}
---
As a bonus, it's natively integrated into Quarkus and the Quarkus ecosystem.
Quarkus is the best thing since sliced bread if you're on the JVM and are writing API's.
> I have a hard time wrapping my head around why companies are mandating 100% return to office.
Hot take: There's a whole way of communicating and handling team organization that grew up around 20th century business management culture, and it just doesn't work well with remote teams. Methods that do work well with remote teams, though, don't work well with traditional deeply hierarchical management structures. At best, they tend to make all those management layers somewhat superfluous.
This means getting people back into the office may be a matter of self-preservation for career managers, whether they realize it consciously or not.
I used to be one. I’ve also been a 1/10x programmer.
Knowing and mastering your tool chain, language, editor infrastructure etc intimately; can have tremendous productivity gains.
Having zero personal life also helps tremendously.
Features and products just flow out of you.
Personally I found having a family has many benefits, but I also lost that edge. Health issues also take away a lot.
Kids getting old enough to not require constant attention is allowing me to regain some of that.
https://clickhouse.com/blog/how-we-used-clickhouse-to-store-...