I never did bother with actually making it an SQL injection; it was meant to be an in-joke between me and whoever at the client with tech chops set up the billing record, nothing more :)

Did it have an impact on your business? i.e. was it easier or harder to find clients? I would guess harder, but for me personally I'd be more likely to check you out with such an awesome name, so I'm quite curious

When I was running it, I was marketing myself - the company was (HMRC, if I tell you to stop reading this comment now, you're legally required to stop, right?) mostly a vehicle for billing clients and "correctly and appropriately accounting for the appropriate legal tax requirements" rather than something that was actively marketed for inbound business.

I must admit that I do always try and maintain a separate data backup for true disaster recovery scenarios - but those are mainly focused around AWS locking me out of our AWS account (and hence we can't access our data or backups) or recovering from a crypto scam hack that also corrupts on-platform backups, for example.

I would never use this as part of the backup and restore plan; but I was lucky when a bunch of customer files were deleted due to a bug in a release. Something like 100k files were deleted from Google Storage without us having backup. In a panic we contact GCP. We were able to provide a list of all the file names from our logs. In the end, all but 6 files were recovered.

I think it took around 2-3 days to get all the files restored, which was still a big headache and impactful to people.

If you have to take care of availablity and redundancy and delete protection and backups then why pay the premium S3 is charging ?

Either you don't trust the cloud and you can run NAS or equivalent (with s3 APIs easily today) much cheaper or trust them to keep your data safe and available.

No point in investing in S3 and then doing it again yourself.

I mean that's just obviously wrong, though.

There is a point.

> Either you don't trust the cloud and you can run NAS or equivalent (with s3 APIs easily today) much cheaper or trust them to keep your data safe and available.

What if you trust the cloud 90%, and you trust yourself 90%, and you think it's likely that the failure cases between the two are likely to be independent? Then it seems like the smart decision would be to do both.

Your position is basically arguing that redundant systems are never necessary, because "either you trust A or you trust B, why do both?" If it's absolutely critical that you don't suffer a particular failure, then having redundant systems is very wise.

You can argue that you protect against different threats than AWS does . So far I have not seen a meaningful argument of threats a on Prem protects differently than the cloud that you need both.

Say for example your solution is to put all your data backups on the moon then it makes sense to do both, AWS does not protect against threat to planet wide issues.

However if you are both protecting against exact same risks having just provider redundancy only protects against events like AWS goes down for days /months or goes bankrupt.

All business decisions have some risk , provider redundancy does not seem a risk to mitigate for the cost it would mean for most businesses I have seen.

Even Amazon.com or Google apps host on their own cloud and not use multi cloud after all, their regular businesses are much bigger than their cloud biz , they would still risk those to stick to their cloud/services only.

This is a really confusing question. Redundancy requires more than 1 option. It's not about it being better than AWS, it's that in order to have it you need something besides just AWS. AWS may provide redundant drives, but they don't provide a redundant AWS. AWS can protect against many things, but it cannot protect against AWS being unavailable.

This is probably true with Google, but AWS contributes > 50% of Amazon's operating income. [1]

[1] https://www.techradar.com/news/aws-is-now-a-bigger-part-of-a...

Their retail/e-commerce side is less profitable than AWS but the absolute revenue is still massive and the risk of losing that a chunk of that revenue(and income) due to tech issues is still enormous risk for Amazon .

You and AWS are using similar chips similar hard disks even with similar failure rates.

If you both use same hardware from say batch both can defects and fail at similar times.or you use the same file systems, that say corrupts both your backups.

90% is not a magic number , you need to know AWS supply chains and practices thoroughly and keep yours different enough not to have same risks as AWS does for your system to have independent probability of failures.

(Consider a buggy software release which incorrectly deletes a backup. Depending on the bug it’s very possible it will delete in both places.)

But you still have some risks here, yes, with a super low probability, but a company-killing impact.

In some industries - banking, finance, anything regulated, or really (I'd argue) anywhere where losing all of your data is company killing - you will need a disaster recovery strategy in place.

The risks requiring non-AWS backups are things like:

- A failed payment goes unnoticed and AWS locks us out of your AWS account, which also goes unnoticed and the account and data are deleted

- A bad actor gains access to the root account through faxing Amazon a fake notarized letter, finding a leaked AWS key, social engineering one of your DevOps team, and encrypts all of your data while removing your AWS-based backups

- An internal bad actor deletes all of your AWS data because they know they're about to be fired

...and so on.

There's so many scenarios that aren't technical which can result in a single vendor dependency for your entire business being unwise.

A storage array in a separate DC somewhere where your platform can send (and only send! not access or modify) backups of your business critical data ticks off those super low probability but company-killing impact risks.

This is why risk matrices have separate probability and impact sections. Miniscule probability but "the company directors go to jail" impact? Better believe I'm spending some time on that.

Between these two protections, it's pretty hard to lose data from S3 if you really want to keep it. I would guess they are better protections than you could achieve in your own self managed DC.

I'm guessing AWS has some clause in their contract that means they can refuse to deal with you or even return any of your data if they feel like it. Not sure if that's ever happened, but still worth considering it.

For most companies what AWS.or Azure offers is more than adequate.

An internal bad actor with that level of privileged access can delete your local backups or external one can all things you he can do to AWS he can likely do easier to your company storage DC too.

Bottom-line it doesn't matter if customers can pay for all this low probability stuff that can only happen on the cloud and not on Prem sure go ahead. Half the things customers pay for they don't need or use anyway.

[1] assuming your business model allows you to spend the expense outlay you need for the threat model

Sure, your threat model may vary. But relying on cloud only for your backup is simply not enough. If you split access for your AWS backup and your DC backup to two different people, you mitigated your thread model. If you only have 1 backup location, that's going to be very hard.

Every cloud provider has compliance locks which even root user cannot disable, version history and you can setup your own copy workflow storage container to second container without delete/update access to second one to two different people or whatever.

You don't need to do any of it offsite.

Having had to restore databases from tapes and removable drives for a compliance/legal incident, we had a failure rate of >50% on the tapes and about 33% for the removable drives.

I came away not trusting any backup that wasn’t on line.

The extra expense outlay for the 2 additional backups is approximately $50/month, so it's not going to break the bank.

At $50/month scale a lot of things are possible. Most companies cannot store their data in a hard disk in a safe. If you can, then cloud is a convenience not a necessity for you. I.e. you are perfectly fine running your storage stack for the most part.

My company is not very big(100ish employees) and we pay $200k+ for AWS in just storage and AWS is not even out primary cloud. If we have to do what you have, it is probably in bandwidth costs alone another $500k. Add running costs in another cloud and recurring bandwidth for transfers , retrieval from Glacier for older data on top of that.[1]

Over 3 years that would be easily $1-$1.5 million in net new expenses for us scale.

No sane business is going to sign off on +3x storage costs on a risk that cannot be easily modeled[2] and costs that cannot be priced into the product, just so one sysadmin can sleep better at night.

[1]your hard disk in a safe third component is not sensible discussion point at reasonable scale.

[2] this would be probability of data loss with AWS * business cost of losing that data > cost of secondary system.

Or probability of data availablity event(like now) * business cost of that > cost of an active secondary system .

For almost no business in the world the either equation would be valid.

For example even the cost is 100B dollars in revenue with 6 nines of durability the expected loss would be only $10,000 (100B * 0.000001) a secondary system is much costlier than that.

I don't get how this is relevant at all, it's more about how much data your company stores than how many employees it has.

I've worked for a company with 5000 employees that stored less data (fewer data?) than my current employer that has less than 100.

> No sane business is going to sign off on +3x storage costs on a risk that cannot be easily modeled

Probably not, but for us the cost is about 0.1x our aws storage costs, so it's a no-brainer.

Also, what we saw on Dec 7th was that the complexity of Amazon's infrastructure introduces risks of downtime that simply cannot be fully mitigated by Amazon, or by any other single provider. More redundancy introduces more complexity at both the micro level and macro level.

It doesn't really cost that much to at least store replicated data in an independent cloud, particularly a low-cost one like Digital Ocean.

Customers don't care if it's you're fault or not, they only care that your stuff is broken. That safety blanket of having a vendor to blame for the problem might feel like it'll protect your job but the fact is that there are many points in your career where there is one customer we can't afford to lose for financial or political reasons, and if your lack of pessimistic thinking loses us that customer, then you're boned. You might not be fired, but you'll be at the top of the list for a layoff round (and if the loss was financial, that'll happen).

In IT, we pay someone else to clean our offices and restock supplies because it's not part of our core business. It's fine to let that go. If I work at a hotel or a restaurant, though, 'we' have our own people that clean the buildings and equipment. Because a hotel is a clean, dry building that people rent in increments of 24 hours. Similarly, a restaurant has to build up a core competency in cleanliness or the health department will shut them down. If we violate that social contract, we take it in the teeth, and then people legislate away our opportunities to cut those corners.

For the life of me I can't figure out why IT companies are running to AWS. This is the exact same sort of facilities management problem that physical businesses deal with internally.

I have saved myself and my teams from a few architectural blunders by asking the head of IT or Operations what they think of my solution. Sometimes the answer starts with, "nobody would ever deploy a solution that looked like that". Better to get that feedback in private rather than in a post-mortem or via veto in a launch meeting. But I have had less and less access to that sort of domain knowledge over the last decade, between Cloud Services and centralized, faceless IT at some bigger companies. It's a huge loss of wisdom, and I don't know that the consequences are entirely outweighed by the advantages.

Relevant blog post, https://aws.amazon.com/blogs/aws/preview-aws-backup-adds-sup...

Ah, the Vinnie Boombatz treatment.

In my experience, the kind of person that argues about "arrogant 25 year olds that know everything" is the kind of person that only sees their side of a discussion and refuses to understand the whole context. Maybe OP was in the right, maybe they weren't. But the fact that they are focusing on age and making ad hominem attacks is a red flag in my book.

It's also possible the response was "That's an excellent point! I think we should put that on the backlog. Since this data is already a backup of our DB data, I think we should focus on getting the feature out rather than replicating to GCP."

Those are two plausible conversations. Instead, what we have is "these arrogant 25 year olds that have 1-2 years of experience and know it all." That's a red flag to me.

And this is of course valid reason to ignore basic data preservation approaches.

Myself I am an old fart and I realize that I am too independent / cautious. But I see way too many young programmers who just read sales pitch and honestly believe that once data is on Amazon/Azure/Google it is automatically safe, their apps are automatically scalable, etc. etc.

And again, my point isn't that you never need backups. My point is that it is entirely plausible that at that point in time backups from S3 weren't a priority.

Add object versioning for your bucket (1 click) and mirror/sync your bucket to another bucket (a few more clicks).

Yes, your S3 costs will double, but usually they're peanuts compared to all the other costs, anyway.

Debating it takes longer than configuring it.

I have my family photos on a RAIDed NAS. It took me years to get that setup simply because there were higher priority things in my life. I never once thought "ahh I don't need backups of our data" I just had more important things to do.

I'd say leaving the overall roadmap (which is all this produces, at the end of the day, if you ignore the estimation piece) fuzzy and allowing the team to work that out with users/subject matter experts is the right approach, imo.

I was lucky in that I was dealing (in both cases where I've run similar flows to this) with above-board exec teams who wanted the best quality information I could give them - even assuming that estimates are just assumptions - even if it meant having some tough conversations about scope or headcount.

I'm just not sure why software projects are "special" -- if you can avoid it being a project and instead make it ongoing OpEx like, for example, GDS managed for the UK in 2016, then great, you've sidestepped that, but until the entire PM industry discovers how to improve overall project management techniques, I don't see why we'd consider our industry "above" them.

Every non-tech startup founder who's approached me with "how long will it take to build an MVP for my startup idea?", I've answered with this. Development is an ongoing cost, a process, not a once-off capex cost.

I recommend to them going the other way. Start with "how much can you afford to pay a dev team sustainably?" then work out how many devs that works out to, then work out how long your MVP will take to build based on their estimates (and estimates are not deadlines).

Not quite the same as #no_estimates (which I also try to argue for whenever possible), but close.

How long until we cure retinoblastoma? Everyone understands there is no way to produce a meaningful timeline. There are too many inter-related unknowns - various causes, various treatment modalities, varying funding on various fundamental and applied research, no real idea if the final answer is gene therapy, nano-something, chemo, etc.

I used to develop software for a defense contractor, and it was pretty waterfall-y. But we built risks in, to an extent. Not by multiplying Sally by 1.3x, Joe by 2.7x or whatever, but you'd chart it all out, showing interconnections (x depends on Y, which depends on Z and Q, which...). And then roughly figure out risks of each of those sub-tasks going long.

The idea NOT being you then just multiply by a weight, and ta-da, you have an accurate schedule. The idea is that you have now identified particularly risky chain of events, and now you at least have a chance of managing risk. Every day/week you'd have a risk assessment meeting. Where are we on X, Y, Z. What can we do to get X back on track? Can't, okay, can we we-jigger the dependencies, or is this a hard slip? And so on. I've never seen this done on the commercial side, and it just seems like people are flying blind as a result.

"Waterfall is terrible" you reply. Sure. But when you are building an airplane, ya kinda need the 1553B/ARINC bus installed before you install and test the avionics. You can't attach the engines if the wings haven't shown up. You can't redesign the fuselage after the wing builder started building the wings (in general). These are hard, unavoidable dependencies, and changes are often extremely to destructively expensive (hence the endless mind numbing meetings arguing about change control).

It is just (IMO) not an unsolved problem, but unsolvable. Too many unknowns results in unpredictability. Your only bet is to manage the risks, adjust as necessary, and accept some things are just unknowable. Agile does that in one way, sophisticated waterfall in another.

Obviously as you progress through your project, you predicted end date should become more and more accurate.

A project plan, though, is a prediction of the future. I've not seen anyone that can predict the future perfectly. That's not to say that you shouldn't do it, but defective project management creates a project plan on Day 0 and then tries to bend reality to meet the plan.

*Obviously all the above is caveated with reasonableness - you do try and bend reality a certain amount to meet your plan, and you try and keep to your plan as much as possible.

I tried to think up an accessible example that didn't require too much context on the part of the reader, so obviously, as you correctly point out, all the numbers are made up, and I'm trying to use it solely to demonstrate the workflow :)

The problem with the real world examples is the business domain, which was hyper complex and the specific "pieces" of work I described wouldn't have been easy to grasp for most not familiar with the esoteric side of fintech that the project took place in.

So I went with a simpler, albeit contrived and more accessible example.

Asking b/c I've taken two courses in dev process or project management in my academic career, and neither provided substantial value or benefit to how I've lead projects professionally.

However, there were techniques (like critical path analysis) that as I've got more senior, started working at larger companies, and started stepping down the senior leadership path, I've started to see some applicability to. Not direct applications - they still need taking with a massive pinch of salt, and modifying for modern learnings in industry, but they do start to provide some value, even if it's just learning what the grey-hairs in the exec are used to seeing :)

Lots of other old school stuff that we "just turning grey" people need to translate for the new kids.

Realistic estimates aren't padded, but they still have significant probability of being inaccurate. After all, they're estimates, not information from the future transmitted to the past.

I can't talk in too much detail, but in general, the deadline date was fixed through commercial contracts signed at a high enough level that engineering didn't have sight of them. The concept and commercial case was sound, but the implementation hadn't been worked out yet, when a date was set.

My strong preference would be for estimation to come first, of course, before a deadline is picked (and even then, only picked if it is really a necessary deadline), which is then based on reality, and also include some slack for unintended discoveries.

I delivered a spreadsheet with detailed information of how long it would take to develop each feature of the new module they wanted. It totalled around 3 months of development.

Fancy suit folks told me 3 months wouldn't do because sales promised it would be ready in 2 months. Then I was asked where could shortcuts be taken.

In the end we had to cut features that will certainly upset our client given their initial expectation.

How do you incentivise devs to hit those timeframes?

Hope you're doing well too!

Once the project is ongoing and you'll need to account for changes, I've either done it manually (which takes an age) or handed it off to PM's to oversee using either MS Project, airtable with a custom-authored set of actions/etc, or PrimaVera.

There's a reason medical certifications are so hard to get, and medical software is so expensive.

You're storing patient information in postgres. What certifications do you have to assert that the patient data is stored securely, in line with your government guidelines on patient/medical data? There's a damn good reason this is the "holy grail" of information security certifications.

You've got critical alerting built into the browser window using JavaScript.

This "alerting" is the kind of critical thing that sometimes needs *immediate" intervention, or someone could die. What happens if your browser experiences a JavaScript error blocking processing? And your alerts don't fire?

What happens if they fire too often and you get "alert fatigue" because they're not tuned correctly or in line with the other alerts available at the bedside/nursing station?

How much testing have you done to correctly assert that you're interpreting the HL7 or other specs correctly? And aren't misinterpreting data for some conditions or types of individual?

The "throw things together quickly" startup mentality might (I stress might!) Be okay where it's the difference between nothing at all and something that can save lives, in a country like Sri Lanka, during a global pandemic, fine.

But afterwards, this is so much junk without serious thought and time put into certifying it.

Medical, Aerospace -- really, any safety critical industry where your code working or not could mean someone is seriously injured or dies as a result -- is an industry that needs disruption, but that disruption should happen slowly, carefully, and safely.

If this is some small town hospital in Srilanka, the choice is between an unaffordable certified solution and not having any monitoring. If Medical software didn’t bleed them dry, they wouldn’t go this route.

> disruption should happen slowly, carefully, and safely

Disruption always happens this way - same way Uber broke existing laws. Yes, few people will die. But this isn’t surprising when the alternative is even worse.

Always consider the alternative. This could be a hospital in a remote part of a third-world country. Maybe they're understaffed. How are they currently handling the task of gathering information from monitoring devices and reacting to alarms?

Maybe, their nursing staff has to run from bed to bed to check patient's vital signs and device alarms. Emergencies would frequently be missed because they are understaffed and checking is irregular. Now, you could introduce software which provides centralized monitoring. If it's introduced on top of the existing activities (i.e., running from bed to bed), it leads to a net benefit - you catch emergencies earlier and consequences of malfunctions are less severe. But if it's introduced to replace the existing activities, it may lead to patient harm.

Sure, it's self-coded, browser-based and buggy - but you always need to weigh risks with benefits, and those depend on usage context.

Of course, in most western countries, this would be completely illegal. But these are also the countries in which medical software looks like it's from the 90s, with catastrophic usability and missing features.

We need to ask ourselves: Right now, we heavily prioritize patient safety over innovation - but have we got that balance right? What are patients missing out on if we could just bring a few more of the latest advances in technology to their bedside?

You know, not machine learning, the blockchain or the internet of things. Rather things like browser-based applications which "just work" and have great usability.

Note: I'm a physician, software developer and consultant for medical software certification :)

It feels to me like the management has misunderstood the cost of the software vs not having the software. It feels like they're saying "this software is expensive, and doing nothing is free" when they should be saying "having all these healthcare professionals spending time putting on and taking off PPE the check patients is costing us this much per year".

As you probably know, an ICU will go through 30 sets of PPE per patient per day. That's a lot of time putting stuff on and taking it off.

So, why not use those to build the "something is better than nothing" solution?

Just availability.

This was a quick and dirty hack to improve access to patient data done with what was on-hand, for a constrained deployment using specific known devices. They didn't have anyone with knowledge on using any of the tech you mentioned, some of which requires spending months setting up unless you have practical experience in delivering on the platforms. Just getting a more safety-minded setup for a MCU using free software can be a harrowing experience.

And they don't have the money to just contract it out or pay for the commercial grade stuff.

They did what they could with what they had, with explicit mention that it's not good on safety and security - but it brings some benefit now.

Here in Poland, a few weeks into lockdown, nobody asked for certifications on volunteer made PPE parts anymore. Because a shoddy PPE with no certification was still better than none.

I really hope they 1. Open source it 2. continue to work on this throughout the crisis and get it to a state where its actually suitable for critical care, and then 3. Work on achieving the relevant certification.

It sounds (just guessing) like the device vendor sells their own software separately, and is unwilling to budge on price during this time, forcing an already stretched hospital to look for new solutions.

This could likely be "good enough" for those that have no other options if open sourced.