2. As I mentioned above, I went to work at Knight as a DevOps on a team that deal directly with the team mentioned in the blog post.
There are lots of stories around this but I will share this one:
Late 2012 is when Apple rolled out the "emergency weather notification" function. I was in the office and the notification went off on multiple people's phones. Knight was also experimenting with call notifications.
So when the alert goes off, someone yells "God damn it! Not again!!" (thinking there was another big outage)
3. People outside of finance have no idea of the different types of outage that can happen due to all sorts of factors.
4. In finance in general, the amount of legacy code that behaves in weird ways or was written by someone 10 years ago who is no longer with the firm is ASTOUNDING.
Coupled with the billions of combinations of regulations, internal controls, multiple countries and jurisdictions etc makes accounting for every single edge case impossible. To use an infosec term the "attack surface" of possible user actions that could lead to bugs is enormous.
Typical case:
- User says they want to see reports for a couple days worth of trading for all securities
- User also says they want to see FULL history for one security
- User never says they might want to see FULL history for ALL securities at the same time
- This being HN, someone will say "you should have thought of that"
- Sure, but then they pull only some of the history for a Ukranian bond that has a 182 (not 180 like most) day bond. This is the only example of this type of bond. Ever. Did you think of that? What should the system have done?
- An oh, btw, this system was pushed out quickly due to regulatory pressure etc
I would be interested to read these stories, but the twitter links only show a single tweet ending in the phrase "A thread." Perhaps this is a new feature of X whereby only logged-in users can see a tweet and its replies.
1. I signed my offer letter to work at Knight 5 days before this happened (and I still went to work there)
You can read more about that here: https://twitter.com/alexpotato/status/1501174282969305093
2. As I mentioned above, I went to work at Knight as a DevOps on a team that deal directly with the team mentioned in the blog post.
There are lots of stories around this but I will share this one:
Late 2012 is when Apple rolled out the "emergency weather notification" function. I was in the office and the notification went off on multiple people's phones. Knight was also experimenting with call notifications.
So when the alert goes off, someone yells "God damn it! Not again!!" (thinking there was another big outage)
3. People outside of finance have no idea of the different types of outage that can happen due to all sorts of factors.
I have a LOT of stories here: https://twitter.com/alexpotato/status/1215876962809339904
4. In finance in general, the amount of legacy code that behaves in weird ways or was written by someone 10 years ago who is no longer with the firm is ASTOUNDING.
Coupled with the billions of combinations of regulations, internal controls, multiple countries and jurisdictions etc makes accounting for every single edge case impossible. To use an infosec term the "attack surface" of possible user actions that could lead to bugs is enormous.
Typical case:
- User says they want to see reports for a couple days worth of trading for all securities
- User also says they want to see FULL history for one security
- User never says they might want to see FULL history for ALL securities at the same time
- This being HN, someone will say "you should have thought of that"
- Sure, but then they pull only some of the history for a Ukranian bond that has a 182 (not 180 like most) day bond. This is the only example of this type of bond. Ever. Did you think of that? What should the system have done?
- An oh, btw, this system was pushed out quickly due to regulatory pressure etc