On Fragile Systems

2026.11.15

CLXXVI

[Fragile Systems; Privacy Screens; On Prem Compute; Open Source; Smart Home; Not AI Pilled; Gardening & Hunting; Robust Systems]

Thesis: Every time we choose distributed systems with independent failure points, we make the world a more robust place, and often save money.

[Fragile Systems]

Our world is increasingly interconnected and interdependent. One part of the system going down can take out others.

I wonder how much business was stopped on Thursday when a huge part of Microsoft went down, and how much business and LIFE was stopped on Friday when the FAA issue happened?

I think in a lot of ways, we're laying more and more of these bottleneck traps for ourselves. We're far higher trust society than we should be, and build systems on top of this trust.

Sometimes, the trade offs make sense: by being a part of the larger 'network', you get benefits with amortized costs across the whole system. Meaning, you get features and capabilities that would be impossible to maintain on your own.

And if I didn’t fundamentally believe that this trade off was worth it in some cases, I’d be a fool for building software, which is predicated on this trade off making sense at least sometimes.

But just as often, this is a bad trade in the long term! Yet, it seems to be the default one that most people make!

This post is me jumping around and exploring fragility and robustness and a general caution around depending on external systems when you don’t have to. It is something of a dellusion (or, an incredibly ambitious undertaking) to think you can escape from it entirely; yet, that doesn't mean we should just accept it exactly as it is.

We'll go over some decisions I've made and am making in this regard that I'd argue are positive ROI. One of them has objectively saved WhiteWhale over $100K as of this month.

Some of them might seem weird, but you're new here if you'd think I'd actually care about that.

Not much on the FAA because I don’t really understand it other than to mention for all of it’s amazing benefits, it clearly represents a single point of failure; more on Microsoft below.

[Privacy Screens]

I'm increasingly a proponent of privacy screens, especially while traveling.

Do you ever look at what's happening on your neighbors screens while in a plane? I'd be lying if I said I didn't... so obviously, I should expect the same of them for my screen!

While I was waiting for the bathroom on the flight I'm writing this, I saw a woman working on deck for the office of the CFO at a $100B publicly traded company outlining how they will be migrating away from Oracle Cloud in 2027 to expand margins. This is private information that you’d definitely not want shared if you work at that company until you’re certain of it and ready to brag about margin expansion.

The funniest part is that the company the woman works for... does cyber security?! You'd think they'd mandate things as simple as privacy screens, but I guess not??

Now, maybe part of the risk mitigation is that a random arbitrary person on the street wouldn't be able to see a wall of text and parse this information out of it rapidly, but also, you're not worried about that person knowing this, I wouldn't think. The kind of person who you don't want to parse it is the kind of person who will.

Same with cellphone privacy screens.

You might think, "Oh, I have nothing to hide," but everything information leak is potentially leverage.

I might sound insanely paranoid, and maybe I am, but it's also so low cost to use things like privacy screens.

And don't get me wrong, I don't do this without fail; the immediate return feels non existent. Still, I think it's an easy way to limit the amount of information that leaks about you without your knowledge.

Inline with this, I also ask them not to take my photo at TSA. Just one less place with a repository of true positive identifications of my face.

With an increasing surveillance state, I do wonder if it's all an exercise in futility. Still, I'm opting for at least some level of hygiene.

[On Prem Compute]

I've now saved over 6 figures for WhiteWhale since we started by using on prem servers. This is a lot of money for a bootstrapped SaaS.

Also, I'd argue that it's substantially more robust.

Cloud providers are f*cking expensive. And, they're centralized points of failure.

WhiteWhale still has an lynchpin issue with one bottle neck on a specific cloud provider. I'd like to add in more redundancy to this at some point. With creeping costs, it's quite likely that I do.

And you might say, but Noah, the cloud has all these backups and there's so much risk, etc etc.

Yeah, but you could realistically recreate this in a well designed office with a backup generator, redundant internet connection (starlink?), and maybe amn MSP for a separate location to write to, or even use a dirt cheap cloud like backblaze as a fallback for storage.

Maybe it's a pipe dream, but if our cloud bill is on track to be 6 figures within a year (even with some stuff already on prem!), this decision becomes seriously attractive very quickly.

And yes, you have to learn a lot more about how computers actually work, but I’d argue that this makes you a substantially superior developer, at least if you’re doing anything that requires even a little performance.

[Open Source]

Adjacent to this, the machine I’m writing this on is a framework that’s operating omarchy with the text editor Neovim. I'll still do the final edits and send it out via Beehiiv, but I do like depending less on the same suites of software everyone does.

On that note, when I logged into Beehiiv this morning, there was a totally random error logging in that I couldn’t do anything about! It ended up self resolving quickly, but it very well could have not.

The massive Microsoft outage on Thursday, regardless of why it happened, should illustrate the risk of global systems depending on one provider.

Isn't it outlandish that one outage can put a stopper in so much work? A piece of software goes down and it’s one of the most material things that happened during the entire week, beat only be Mr Huang’s treasonous tryst* & the FAA’s outage?

The worst part is, Microsoft's software really isn't even that good at all! It's annoyingly mid and overpriced.

I think it's a much more robust and secure world when different people use different pieces of software and hardware. Think back to the Crowdstrike caused outages in 2024… and then think about how much harder it would be for a malicious hacker to exploit systems if 62% of all computers didn’t use the same centralized OS!

God, Microsoft has made itself such a target!

_{*If you even indirectly funnel billions of dollars of chips to an adversary that is not supposed to have them, yes, this is treason. And yeah, it does make it worse if part of the reason you did it was because you were horny.}

[Smart Home]

I hate those battery powered digital locks. Yeah, maybe it's good for an airbnb so you can set different passwords for each guest, but it just creates another failure point. What happens if your LOCK runs out of battery? Now what, you just can't use it?

Also, since it's cloud based, I'm sure the device is quite hackable, or at the least, a good target for hacking. I like those mechanical ones where you have to punch in the buttons way more.

And then the cameras that are connected to cloud apps, too. Damn, that feels risky. Yet another centralized target. And hopefully these companies design the storage in a super robust way, but who knows if they do!

It'd be so much more secure if everyone's home security system ran on a local server. But Noah, how would I view the system from afar?

You can still do that!! You can stream from the server just fine. You'd just need a very thin application on top of it to do so, with a very solid piece of security for entering, simple SSH keys.

And this is part of the weird nuance that goes on: arguments for interdependence and interconnectedness are often times based on false trade offs. Just because things aren’t on the cloud, it doesn’t mean you can’t access them remotely!

Don't get me wrong, there are arguments for these things, but it's just not as cut and dry as people make it out to be.

[Not AI Pilled]

And, of course, I'm still not a die hard vibe coder.

I've been using AI more; there are some good pros to it! Especially when it does things I would never take the time to do, like build out dummy databases to dry run queries to make sure the shape is right.

Then you can get it to also do things you'd never do on your own, like have it fill out a bunch of dummy data to test the query plan at scale or with a bunch of concurrent connections to see how well it works under load. You can also get it to write test suites for you, which are super helpful to maintain high throughput rate of production changes without breaking stuff.

But I still notice so many fundamental issues, even using one of the most up to date systems (Claude Code with Fable 5.1*). Like:

• Hallucinate variable names

• Randomly change side effects without telling me

• Import random dependencies that aren't needed

• Make incorrect assumptions about data structs without checking

• Leave the current directory and go to other ones without asking

• Not share the complete diff

With all of this, and the fact that I make so many of my design decisions in the middle of implementing the code, I have a really, really hard time believing people when they say things like "you don't need to be technical at all, it's so over" or whatever it is they say.

Yes, it's very useful and cool, but the game is not over. A big refactor still took me a week, even heavily using this thing and giving it a spec that it took me 2 days to write.

Also, of course, I worry that depending on it... well, that depending on it makes you depend on it! Why would I put on a blind fold to my own code base for maybe a marginal boost when instead I can use it symbiotically, steer and get the speed boost, but actually understand what's going on?

I’m considering forking OpenCode or Hermes to try to engineer some of the security risks & annoyances out, eg adding things like deterministically displaying all changed files since last chat, intercepting all terminal commands to lock it in a specific directory, and blacklisting specific files. (right now, I don’t trust Claude Code enough to let it run around the repo fully and only send it the code it needs).

_{*I swear to god if you tell me, "No, bro, 5.2 is a game changer, your opinion is invalid until you use that", I'm going to lose my shit. We can't perpetually do this every time a 'new model' comes out because someone releases one every week. I'm judging your claims from the last model against the outputs I am getting from the last model. You don't get to redact your hype vomit just because the next one is here…}

[Gardening & Hunting]

My parents garden and hunt. During the lettuce issues over the summer, they still had lettuce. We have pounds of venison in the freezer at all times.

Their house has a backup generator that runs on natural gas, as well as a few gas ones lining around that take in the same stuff that fuels cars.

This is awesome. My house will have the same, as well as redundant internet connections (starlink) as well as a local server with quite the library on it.

If you enjoyed this post give it a subscribe - I’m here each sunday.

[Robust Systems]

All that is to say, it's not like there is a simple, perfect way to build a more robust system, but there are some obvious, easy things you can do to take yourself out of the flow at least a little.

Maybe I'm staring at computers too much. I had a dream I was at a hot chocolate and they asked if I wanted to sign up for their loyalty program. I said sure, and they flipped the monitor around to reveal a postgres CLI where I entered a raw sql query to save my info down.

I asked them what they'd do if I typed drop table customers and they said, "Oh, yea, I guess that'd be bad."

That's how I feel about the robustness of most systems out there. I think we can all work together to do better.

Live Deeply,