On Iterative Hypothesis Testing

2023.09.24

Lindy Expectancy: 26 Weeks

Iterative Hypothesis Testing is just my nerdy way of saying “repeated trial and error.” I genuinely believe that when used correctly, trial and error can be the single most powerful force in the world.

Complexity

A lot of cool things come from the study of complexity; the goal is to understand tools that help us live in a ‘complex’ world, one in which there are billions of different players competing at different games, constantly evolving their strategies and goals.

The amount of things you can do in this world are unending, unbounded. It can be overwhelming.

A tool to help function amidst all of that is the iterative hypothesis test: the scientific method, over and over again. You have an idea, you try it, and if it doesn’t work as well as you’d like, you fine tune it.

The Hypothesis

What exactly do we mean by a hypothesis, here?

Hypothesis: An assumption that you can test.

Simple as that. A hypothesis is this thing you think might be true but aren’t really sure about. So, we should go and test it!

That is the critical part: you absolutely have to be able to test these assumptions in such a way that you can measure the results.

For now, we’ll keep what we mean by hypothesis bound to things that don’t have large, negative externalities for other people. We’re working with assumptions on the scale of tweaking your running form, not with assumptions about geopolitical or macro economic cause and effect.

Trial One–The Hypothesis

I’m rambling on about you needing a hypothesis, some idea or thought to test, that should ideally be an updated version of your last hypothesis.

What if we’re on trial one, though? How do we even know where to start with our hypothesis?

Well, chances are someone has done something similar to what you’re trying to do; why don’t you go and ask them? Or read the book they wrote about it?

One of my first posts focused on learning from experts, and my last post focused on taking others' advice as a framework for filling out your own experience. In this context, both of these things hold true. They can help you seed a hypothesis.

The Test

Okay, so let’s say you read this article suggesting that cutting out carbs may lead to weight loss without changing any other variables–you can eat the same amount of calories and lose weight? Sounds good to me.

So, the hypothesis has been seeded, and now it’s time to test it. Which should be easy enough in theory, we’re just cutting carbs.

Not so fast–we’re doing the scientific method, remember? Give me at least a little more rigor. We need parameters, here. Are you eating zero carbs a day? Good luck.

Let’s instead say we do below 30 grams of carbs a day, and let’s say we run the test for a month. Now we have something to work with.

Selection Pressure

Still, we actually need to measure the results or we just went on a diet for nothing. In this case, the measurement we take over the course of the test would probably be our weight. Does it get lower as we go along?

If we want to get more specific, it would be body fat percentage, but if you’re not concurrently making an effort to put on muscle mass, then this nuance can be safely ignored.

We also have to be careful to not change a bunch of other variables dramatically, like how much we’re working out and sleeping, or we might mess up the results.

I’ve actually done this specific test (cutting out carbs) a number of times before, and it works. The hypothesis, the assumption, was tested, and it passed.

Fine Tuning

Okay, so we had our hypothesis test. That was cool. What now? What about that word iterative?

This is where the fun part comes in. Now that we had a hypothesis that worked, we’re going to change it.

What?! Why would we do that? We just went on a diet for a month and lost weight and now we’re going to jeopardize our results?

Yes. Yes we are.

Iterative: Involving repetition

Iterative Hypothesis Testing means we just keep testing stuff out, trying to see if something works better than our current solution. There was certainly something to that no carb diet… but, now there’s this other paper on cutting processed carbs, not just all carbs. That’s a nice nuance.

Afterall, it’s never just about weight: remember Goodhart’s Law:

Every measure that becomes a target falls under pressure to no longer be a good measure.

We don’t just care about weight, we care about health, and energy levels, and how we feel. So, let’s try it again. This time, we’ll allow for fruit and raw honey. We’ve updated our hypothesis, and we can give it a go. We’re now in our second iteration.

Maybe this time we don’t lose quite as much weight, but we’re more satisfied. This is just as important–when dealing with our health, we can zoom in and zoom out as we’re trying to find what really matters.

The Meta Goal

Each hypothesis test implies something about us–what we think we care about. If my test is about losing body fat, that implies I care about losing body fat. Really, though, that’s only true up to a point. My actions have never indicated that I really care that much below below 15%, certainly not as much as I care about eating tasty food.

This is an important distinction–there is never ONE thing we are optimizing for in life, there’s always a collection of a million different variables, and the iterative hypothesis tests are as much about finding out what those are as they are about hitting some range of that goal.

Entrepreneurship

I just talked about diet, but this framework can be applied to literally anything. I think about it in terms of fitness, presenting, writing, coding, and, of course, entrepreneurship.

A very high octane example is from Elon Musk. If you haven’t started reading Walter Isaacson’s new book on him, I highly recommend it.

Musk failed a lot with SpaceX.

Musk blew up three rockets before he got one with its payload into orbit. At each of the three failed launches, something obviously went wrong. What’s really important, though, is that each time, something different went wrong… Musk & SpaceX were not just making the same mistake over and over again. They got closer each time.

Ultima is not launching rockets into space, but we have had a lot of failed hypotheses, thinking that if we do X, so and so will pay Y. If we get closer each time, though, if we keep failing up, and we fail fast enough, we will have something worth something to someone.

Progress, not perfection.

There are two main reasons I started writing these notes: I enjoy doing so and they help me to share what’s going on in my life with those I care about.

Now, I’m really starting to appreciate the challenge of communicating complex ideas in as simple of words and thoughts as possible. I think it’s a good challenge and measure of my own comprehension–putting ideas on paper challenges them, forces you to refine them and make them real. If I can’t share these ideas with you, they’re either bullshit or I don’t understand them enough.

Writing is part of a processes of iteratively hypothesis testing ideas. I hope you enjoy.

Live Deeply,