It's easy to underestimate just how slow digital systems are in many real time scenarios compared to analog systems.
Consider it in the context of camera based self-driving cars, it's tangential to this discussion but it's an easy to visualize metaphor:
- A car traveling 60mph is traveling at 88 feet per second
- Assuming a 60hz camera, there would be a 16.67 ms gap between each frame
- The car is traveling 1.5 feet between each frame interval
- A certain amount of exposure time is necessary for the camera to generate even 1 frame or it will be blurry
- High framerate cameras often work around this by staggering/interlacing multiple sensors, but doing this implicitly increases the latency of each frame
- A 120hz camera might deliver double the frames per second, but each frame could be arriving 4 frames late in exchange
- 4 frames would be imperceptible to humans, it would be 3 feet for the car
- You haven't even processed any of these frames yet.
- Your off the shelf library introduces a random 1 second delay for some reason and costs you 88 miles in processing time
- The car can drive as fast as 120mph
All digital sensors implicitly have a sampling frequency, and the fundamental disconnect is always high sampling frequency =/= low latency. People constantly make this mistake over and over, and by the time you notice you are already too deep into development to make a change.
Decreasing latency is expensive, and requires specialized knowledge. Often you get expert software engineers who end up bottlenecked by the hardware limitations they can't even comprehend or the reverse, hardware guys bottlenecked by the software they can't introspect. The latency is only truly understood when you get to integration testing.
Nearly every step of the way you discover you need specialized hardware, software, operating systems, sensors. Every part of the chain each costing you more latency. It's like it's own ecosystem where almost everyone writes everything from scratch and doesn't share anything. It's gotten better in recent years though.
Full disclosure: I work in medtech and don't actually deal with cars, but it's a very similar problem space. We often use the same hardware/software cars use for this reason.
88ft not 88 miles. I don't understand your 120hz camera line, I also have a degree in imaging tech and that part I do not understand, would you be so kind as to expand upon it??
Wew I'm getting a lot of pedantic replies here. Yeah I got the units wrong there my bad, you get the point I was making though right?
In regards to the 120hz camera line, I would be happy to expand on that for you. To be clear, I'm specifically talking about how when you try to increase sampling rate by interlacing multiple concurrent digital sensors you need to deal with the following potential problems (this is a property of all digital sensors):
- Real time synchronization between concurrent sensors requires additional processing time to ensure proper ordering. The samples need to be processed together in small batches. This adds latency and the more things you are trying to synchronize the more latency is introduced.
- Inter-sensor calibration to account for variations between individual sensors can be used to reduce the latency introduced by synchronization but lacking this, you are bottlenecked by the slowest sensor. There are a lot of different ways to handle this though so I'm speaking very generally.
- Broadly speaking most signals need to go through some kind of filter to remove noise, most digital filters have a certain amount of algorithmic latency built-in that is physically unavoidable. When you are interlacing multiple different sensors, you are getting more noise, so the likelihood of a filter being required starts to increase.
I want to stress here that these are not impossible challenges. In fact they are largely solved problems. But they are not universally solved in the same way, you need to balance between precision manufacturing, signal quality and signal latency. In practice most people are not prioritizing latency, so a 120hz camera might be optimizing for video recordings and not live processing scenerios. So long as you know what you are looking for you can avoid this when choosing which camera to use.
Computers can be fast, but fast can mean different things when dealing with real-time situations in high speeds. Bottlenecks need to be considered from all levels. The clock speed of the computers CPU often gives product managers weird ideas about what is possible. This was the main point I was trying to make here.
Honestly I'm just a dumb web dev and even though many of our performance problems can be solved by horizontal scaling, there are a minority of cases where latency is important and that's when things get really fun :-) No more downloading random halfbaked libraries off the web, architecture starts to matter, vm settings start to matter, gateways/load balancers in the middle start to matter, even code optimization starts to matter again just like in good old days.
> Assuming a 60hz camera, there would be a 16.67 ms gap between each frame. The car is traveling 1.5 feet between each frame interval.
Ok? So? You are just stating this as if we should understand the implications. I do in fact work with self-driving cars. What you say is true, but it is not a big deal. Why do you feel this maters? Or what is your point?
> A certain amount of exposure time is necessary for the camera to generate even 1 frame or it will be blurry
This is a confused statement. A certain amount of exposure is necessary for the camera to collect light. If you don’t have long enough exposure the picture will be dark, not blurry. Your statement makes it sound as if avoiding blur is why we need exposure time, which is a complete nonsense.
In reality none of this is a problem. There are automotive grade cameras which can collect enough light fast enough that the images are not blurry in practice. Yes, these cameras have a non-zero exposure time. Yes, this adds latency. No, this is not a problem.
> Your off the shelf library introduces a random 1 second delay for some reason and costs you 88 miles in processing time
You mean 88 feet. If my off the shelf library introduces a random 1 second delay i will chuck it in the bin post haste. Use stuff whose performance characteristics are well understood by you and is not terrible.
> Nearly every step of the way you discover you need specialized hardware, software, operating systems, sensors.
> I do in fact work with self-driving cars. What you say is true, but it is not a big deal. Why do you feel this maters? Or what is your point?
Sorry not trying to dunk on you here, but this reads like something a junior engineer would complain to me about. These are not trivial problems, and I'm sure your co-workers who resolved them already so you didn't need to worry about them would agree with me.
> In reality none of this is a problem. There are automotive grade cameras which can collect enough light fast enough that the images are not blurry in practice. Yes, these cameras have a non-zero exposure time. Yes, this adds latency. No, this is not a problem.
You are contradicting yourself here, yes there are automotive grade cameras, but if this wasn't a problem, why would automotive grade cameras need to exist? My post wasn't saying these were impossible problems but hard ones.
> You mean 88 feet. If my off the shelf library introduces a random 1 second delay i will chuck it in the bin post haste. Use stuff whose performance characteristics are well understood by you and is not terrible.
Look I might have typed the wrong unit but it's a bit ironic you gave me this word salad right after complaining about this... Yeah you obviously don't use the library that adds a 1 second delay, but often you don't have the luxury of knowing that until after you learn about it through integration testing.
Libraries don't usually come with latency stats calibrated to your desired hardware right on the tin, would be pretty sweet if they did though.
My complaint is that it is not clear why you think what you describe is a problem. You describe that by the time the next image arrives the car traveled a certain distance. And that is correct. But you imply that it is a problem without spelling out why it is a problem in your opinion.
You seem to assume it is so trivial to understand that you don’t even need to spell out the problem. But it is not. Because i don’t know what is in your head I can’t argue with the details of it. I know that whatever you feel is a problem is not a problem in practice.
Definietly not a problem you would solve by having higher frame rate cameras. So what I’m seeing is that you are unclear on the problem and jumping at a non-solution. One which adds other complexities without actually solving anything for you. And that is a certified junior engineer behaviour.
> if this wasn't a problem, why would automotive grade cameras need to exist?
Automotive grade cameras are special in their supported temperature range (they won’t die if you leave them baking in the sun) and their physical and electrical intefaces being resilient to vibration and electrical interference. You can point your smartphone out the window of your car and see that it can record clear images.
"My complaint is that it is not clear why you think what you describe is a problem."
Well, while you read, do the math in your head and think about what OP was trying to say. The point - latency is a cold and often uncalculated bitch that will screw with reliability and integrity of the system - got across to me quite easily. I was already on that page by the 4th sentence.
> while you read, do the math in your head and think about what OP was trying to say.
I did the math in my head. Before i read their comment in fact. Did calculations both on paper and spreadsheets too. Run simulations, and field tests too. Based on what i know i do not believe that the latency imposed by the frame rate of the cameras is a limiting factor on self-driving car performance or safety. I was asking to see if they have some particular scenairo in their mind to check if maybe i’m missing something. If anything I’m much more concerned about the latency of brake actuators.
> The point - latency is a cold and often uncalculated bitch that will screw with reliability and integrity of the system - got across to me quite easily.
Ah, ok. If we are just catching the vibes, then sure. The vibes are great.
> Based on what i know i do not believe that the latency imposed by the frame rate of the cameras is a limiting factor on self-driving car performance or safety.
This is would be pretty tangential to the discussion so not sure why you are bringing it up? Did you think my post was illustrating that camera based self driving cars were infeasible? Also it appears you are mistaking framerate for latency, something I specifically warned about in my post... I don't really know what to say here.
> Did you think my post was illustrating that camera based self driving cars were infeasible?
You said this line: “The car is traveling 1.5 feet between each frame interval” As I said before, this is a true statement. What I’m asking is what do you think is the significance of this value. You wrote it. It does indeed sound like you were saying this is an impediment, or at least a complication for implementing a camera based system. But I don’t know because you didn’t spelled it out, and I can’t read your mind. Hence I asked why do you feel that number is a problem. (If you don’t think it is a problem, just an interesting fact you wanted to share of course you can say that too.)
> it appears you are mistaking framerate for latency
I’m not. If you are measuring latency from “something is happening in the world” to “robot is reacting to it” then a component of that end-to-end latency is the time delay before your sensor captures a new measurement. That delay is a function of your framerate. This delay is only part of the full end-to-end latency. It appeared to me you were concerned about this delay and that is why you calculated how much the vehicle travels during this time.
> I don't really know what to say here.
I asked specific questions. If you would care to answer them we would be ahead. When i wrote “I can’t quite understand your comment.” that is not a retorical snipe. It literally means what i wrote: I do not understand your comment and I’m asking for clarification. I even quoted the bits which I don’t understand and asked my specific questions about them.
> This is would be pretty tangential to the discussion so not sure why you are bringing it up?
Because it seemed to me you were bringing it up in your comment. Or at least that is how i could interpret those few lines in your comment I was talking about.
Tangent:
why do so many people (even smart articulate ones like you) in our field (which involves precision with syntax and grammar) get apostrophes wrong?
"It's [IT IS] like its [ITS/HIS/HER/MY/YOUR/THEIR] own ecosystem..."
I anticipate downvotes for picking nits (let alone mentioning karma points), but FTR my intent is to help non-native English speakers (and mostly-literate English language-natives). Getting this "it's : its" distinction wrong is increasingly common and sometimes leads to signal loss.
Yesco, rereading your comment makes me slightly ashamed of this apostrophe rant, I hope others comment on the substance here.
/ end tangent
As someone who spent years doing web performance optimization for a living, your observations resonate. Beyond obvious low-hanging fruit, latency gains are rarely simple to achieve in practice; tradeoffs abound.
To your tangent. When I was taught its vs it's, I was never given the association to his/hers/etc. It was just arbitrary and something you just had to memorize. Reading your post I am shocked I never made the connection earlier but it was just never knowledge I was provided with to work from.
Ha yeah it reads a bit silly, to be honest I lazily typed my original post on my phone while sitting on a hammock outside, I didn't really review it much before posting. Autocorrect can make it challenging for me to do apostrophes correctly since it often overassumes my intent.
For most of the article I felt like the author's learnings were pretty obvious, particularly in regards to the water jet direction and the minimum necessary dish arrangement. They didn't even get into detail about the heat and steam.
Then they mentioned the part about dishwashers having a "filter", and in that moment, my heart was instantly filled with shock, horror, anxiety, and finally, resignation.
I apparently live in a world where dishwashers have filters.
I'm afraid to look, but I'm pretty sure mine probably does as well, in fact I think I've seen it with my own eyes, yet somehow it never registered. Oh no.
I regret to inform you that your bank, credit card company and every other financial institution you use is already selling it off in real time... it's how they make credit reports :(
Also my bank only knows what goes on in my checking account. They have a lot less information than the IRS does (although what they do have is probably a lot more detailed).
I've been surprised at how many people will happily hand over their detailed financial data to Plaid. I can understand how many people might not know how sophisticated malicious actors can be with it, but everyone has heard of identity theft at this point and having someone taking out credit cards in your name is a huge pain.
- Steam Runtime: A set of common libraries Linux native games target for multi-distro compatibility, I believe this still uses Ubuntu as the upstream
- Steam OS: An arch based distro pre-configured to run Steam out of the box, used by the Steam Deck, comes with extra stuff like gamescope to smooth over various issues other distros have with VRR, HDR, etc.
They technically worked but they were expensive, painful, and basically just kept you alive on a bed. In essence they were like a more invasive version of the iron lung but required a team to maintain.
For this reason while you may have heard it used it various trials with "success" they were considered a failure in the end and rejected by the FDA. I've spoken to some people who worked on it and the feelings are pretty bittersweet.
Learning from this, Abiomed later brought a heart pump called the "Impella" to market which works to assist your heart instead of replace it. This device is cool for different reasons and can actually save peoples lives instead of merely prolong them a few years.
The new artificial heart in the OP is more sophisticated than the AbioCor, the science keeps improving.
Our modern peace is not from enlightenment but because war became too destructive. Peace should never be assumed.
reply