How to understand the most important equation in information theory using nothing but a humble triangle

---

The Goal

We’re going to completely understand this equation:

$$H = -\sum_{i} p_i \log_2 p_i$$

It looks scary. It has Greek letters. It has weird symbols. It has subscripts.

By the end of this document, you’ll see it’s just triangles all the way down.

No math will be left behind. Every symbol will be explained. A 12-year-old can understand this.

Let’s go.

---

Part I: The Triangle — Our Only Tool

What Is a Triangle?

Three points. Three lines connecting them. That’s it.

        A
       /\
      /  \
     /    \
    /______\
   B        C

A triangle is the simplest closed shape. You can’t make a closed shape with fewer than three sides. Two lines can’t enclose an area. Three can.

The 45-45-90 Triangle: Our Hero

Among all triangles, one is special for understanding information:

        /|
       / |
      /  |
   1 /   | 1/√2
    /    |
   / 45° |
  /______|
    1/√2

This triangle has:

• One right angle (90°)
• Two equal angles (45° each)
• Two equal legs (both 1/√2 if the hypotenuse is 1)

Why is this triangle important?

Because when you don’t know which of two things will happen, and they’re equally likely — that’s 50-50. And 50% = 1/2. And 1/2 comes from this triangle.

$$(1/\sqrt{2})^2 = 1/2 = 50%$$

The 45-45-90 triangle is the geometry of maximum uncertainty.

---

Part II: What Is Distinction?

The Most Basic Idea

Before we can talk about information, we need to understand distinction.

Distinction means: telling A from B.

• Is the light on or off? That’s a distinction.
• Is the answer yes or no? That’s a distinction.
• Is the coin heads or tails? That’s a distinction.

Without distinction, there is no information.

If everything is the same, there’s nothing to say. “It’s A” only means something if there’s a B it’s not.

The Bit: One Distinction

Shannon invented a word for “one distinction between two equally likely things”:

The bit.

• 1 bit = enough information to pick between 2 possibilities
• 2 bits = enough to pick between 4 possibilities
• 3 bits = enough to pick between 8 possibilities

See the pattern?

Bits	Possibilities
1	2
2	4
3	8
4	16
5	32
n	2^n

n bits can distinguish 2^n things.

But wait — what does 2^n mean?

---

Part III: Exponents — Repeated Multiplication as Triangles

What Is an Exponent?

When we write $$2^3$$, we mean:

$$2^3 = 2 \times 2 \times 2 = 8$$

The small number (3) tells us how many times to multiply the big number (2) by itself.

• $$2^1 = 2$$
• $$2^2 = 2 \times 2 = 4$$
• $$2^3 = 2 \times 2 \times 2 = 8$$
• $$2^4 = 2 \times 2 \times 2 \times 2 = 16$$

Exponents as Triangle Stacking

Imagine each “multiplication by 2” as doubling the area of a triangle.

Start with a triangle of area 1:

Step 0: Area = 1
    /\
   /  \
  /____\

Multiply by 2 (double the area):

Step 1: Area = 2
      /\
     /  \
    /    \
   /      \
  /________\

Multiply by 2 again:

Step 2: Area = 4
        /\
       /  \
      /    \
     /      \
    /        \
   /          \
  /____________\

Each step doubles. After n steps, you have $$2^n$$ times the original area.

The Branching Tree View

Another way to see exponents: a tree that branches.

                    *           Level 0: 1 item
                   / \
                  *   *         Level 1: 2 items
                 /|   |\
                * * * *         Level 2: 4 items
               /|/|/|\|\
              ********          Level 3: 8 items

At each level, every branch splits into 2. After n levels: $$2^n$$ branches.

Each split is one bit of information. n bits = $$2^n$$ possibilities.

---

Part IV: Logarithms — Counting the Splits

The Question Logarithms Answer

Exponents ask: “I have n bits. How many possibilities?”

Answer: $$2^n$$

Logarithms ask the reverse question: “I have N possibilities. How many bits?”

Answer: $$\log_2(N)$$

Example

If there are 8 equally likely possibilities, how many yes/no questions do you need to narrow it down to one?

Start: 8 possibilities {A, B, C, D, E, F, G, H}

Question 1: "Is it in the first half {A,B,C,D}?"
  If YES: 4 possibilities remain
  If NO: 4 possibilities remain

Question 2: "Is it in the first half of what's left?"
  If YES: 2 possibilities remain
  If NO: 2 possibilities remain

Question 3: "Is it the first one?"
  If YES: 1 possibility remains
  If NO: 1 possibility remains

Done! 3 questions for 8 possibilities.

So: $$\log_2(8) = 3$$

Because: $$2^3 = 8$$

Logarithms as Counting Triangle Halvings

Remember how exponents were “doubling”? Logarithms are “halving”.

Start with a big triangle (say, area 8):

Step 0: Area = 8
            /\
           /  \
          /    \
         /      \
        /        \
       /          \
      /____________\

Halve it:

Step 1: Area = 4
        /\
       /  \
      /    \
     /______\

Halve again:

Step 2: Area = 2
      /\
     /  \
    /____\

Halve once more:

Step 3: Area = 1
    /\
   /__\

It took 3 halvings to get from 8 to 1.

$$\log_2(8) = 3$$ = the number of times you halve 8 to reach 1.

The General Rule

$$\log_2(N) = \text{how many times you double 1 to reach N}$$

Or equivalently:

$$\log_2(N) = \text{how many times you halve N to reach 1}$$

N	$$\log_2(N)$$	Why
1	0	$$2^0 = 1$$
2	1	$$2^1 = 2$$
4	2	$$2^2 = 4$$
8	3	$$2^3 = 8$$
16	4	$$2^4 = 16$$
32	5	$$2^5 = 32$$

---

Part V: What About Non-Whole Numbers?

Fractional Exponents

What does $$2^{1/2}$$ mean?

It’s the number that, when multiplied by itself, gives 2.

$$2^{1/2} \times 2^{1/2} = 2^{(1/2 + 1/2)} = 2^1 = 2$$

So: $$2^{1/2} = \sqrt{2} \approx 1.414$$

And guess what? $$\sqrt{2}$$ is the diagonal of a unit square — which is made of two 45-45-90 triangles!

    1
  ┌───┐
  │ / │
  │/  │  diagonal = √2
  └───┘
    1

Fractional Logarithms

What is $$\log_2(3)$$?

It’s asking: “2 to what power gives 3?”

We know:

• $$2^1 = 2$$
• $$2^2 = 4$$

So 3 is between $$2^1$$ and $$2^2$$. The answer must be between 1 and 2.

$$\log_2(3) \approx 1.585$$

This means: $$2^{1.585} \approx 3$$

Fractional bits are real! If you have 3 equally likely possibilities, it takes about 1.585 bits to specify one. Not a whole number — because 3 isn’t a power of 2.

---

Part VI: The Summation Symbol Σ

What Does Σ Mean?

The Greek letter sigma (Σ) just means “add them all up.”

$$\sum_{i=1}^{3} x_i = x_1 + x_2 + x_3$$

Let’s break this down:

• The Σ says “add”
• The $$i=1$$ at the bottom says “start counting at 1”
• The $$3$$ at the top says “stop counting at 3”
• The $$x_i$$ says “add the thing called x with index i”

Example with Numbers

$$\sum_{i=1}^{4} i = 1 + 2 + 3 + 4 = 10$$

Here, $$x_i = i$$ (the index itself). We add 1+2+3+4.

Example with Probabilities

Imagine a coin flip:

• $$p_1 = 1/2$$ (probability of heads)
• $$p_2 = 1/2$$ (probability of tails)

Then:

$$\sum_{i=1}^{2} p_i = p_1 + p_2 = \frac{1}{2} + \frac{1}{2} = 1$$

The probabilities add to 1. This always happens — something has to occur!

Σ as Triangle Addition

Think of each term as a triangle. Σ means “stack all the triangles”:

  p₁        p₂        p₃        Σpᵢ
  /\        /\        /\       /\  /\  /\
 /  \  +   /  \  +   /  \  =  /  \/  \/  \
/____\    /____\    /____\   /______________\

 Area     Area      Area      Total Area
  p₁       p₂        p₃        = p₁+p₂+p₃

Summation is just collecting areas.

---

Part VII: Probability — The Language of Uncertainty

What Is Probability?

Probability measures how uncertain you are about something.

• Probability 1 = certain to happen (no uncertainty)
• Probability 0 = certain NOT to happen (no uncertainty)
• Probability 1/2 = maximally uncertain between two options

Probability as Triangle Area

Imagine the “space of all possibilities” as a big triangle with area 1:

Total possibility space (area = 1)
              /\
             /  \
            /    \
           /      \
          /        \
         /          \
        /__Outcome___\
       /   A = p_A    \
      /________________\

Each outcome occupies a fraction of that space. The fraction is its probability.

For a fair coin:

        /\
       /  \
      / H  \     Heads: area = 1/2
     /______\
    /   T    \   Tails: area = 1/2
   /__________\

Total area = 1/2 + 1/2 = 1

For a biased coin (70% heads):

        /\
       /  \
      /    \
     / HEAD \    Heads: area = 0.7
    /  (70%) \
   /----------\
  /   TAIL     \  Tails: area = 0.3
 /    (30%)     \
/________________\

Total area = 0.7 + 0.3 = 1

---

Part VIII: Surprise — The Information in an Event

Less Likely = More Surprising = More Information

When something unlikely happens, you learn more.

If I tell you “the sun rose this morning” — no surprise. You already knew that would happen. Low information.

If I tell you “a meteor hit your house” — huge surprise! You didn’t expect that. High information.

Shannon’s Key Insight

Shannon defined the information content of an event with probability p as:

$$I = \log_2\left(\frac{1}{p}\right) = -\log_2(p)$$

Let’s see why this makes sense.

Example 1: Fair Coin (p = 1/2)

$$I = \log_2\left(\frac{1}{1/2}\right) = \log_2(2) = 1 \text{ bit}$$

Learning the outcome of a fair coin flip gives you 1 bit of information.

Example 2: Certain Event (p = 1)

$$I = \log_2\left(\frac{1}{1}\right) = \log_2(1) = 0 \text{ bits}$$

Learning that a certain event happened gives you no information. You already knew!

Example 3: Rare Event (p = 1/8)

$$I = \log_2\left(\frac{1}{1/8}\right) = \log_2(8) = 3 \text{ bits}$$

Learning that a 1-in-8 event happened gives you 3 bits of information.

The Triangle Interpretation

Information = “How many halvings is 1/p?”

If p = 1/8:

• 1/p = 8
• How many halvings? 3 (because 8 → 4 → 2 → 1)
• Information = 3 bits

Probability 1/8 means the event occupies 1/8 of possibility space:

        /\
       /  \
      /    \
     /      \
    /        \
   /          \
  /____________\
 |##|           |  <-- This tiny slice (1/8 of total) is your event

The smaller the slice, the more surprising when it happens.

---

Part IX: Entropy — Average Surprise

What If Events Have Different Probabilities?

A fair coin: both outcomes have p = 1/2.

But what about a biased coin? Or a die? Or weather forecasts?

We need to average the surprise across all possible outcomes.

The Formula for Entropy

$$H = \sum_{i} p_i \times I_i = \sum_{i} p_i \times (-\log_2 p_i) = -\sum_{i} p_i \log_2 p_i$$

Let’s unpack this step by step.

Step 1: For each outcome i, calculate its surprise

$$I_i = -\log_2(p_i)$$

Step 2: Weight each surprise by how often it happens

$$p_i \times I_i$$

(If an outcome is rare, its surprise matters less to the average.)

Step 3: Add up all the weighted surprises

$$H = \sum_{i} p_i \times I_i$$

This is the expected (average) surprise. Shannon called it entropy.

---

Part X: Entropy Calculated — Examples

Example 1: Fair Coin

Two outcomes: heads (p = 1/2) and tails (p = 1/2).

Step 1: Surprise for each outcome

$$I_{\text{heads}} = -\log_2(1/2) = -(-1) = 1 \text{ bit}$$

$$I_{\text{tails}} = -\log_2(1/2) = 1 \text{ bit}$$

Step 2: Weight by probability

$$p_{\text{heads}} \times I_{\text{heads}} = \frac{1}{2} \times 1 = 0.5$$

$$p_{\text{tails}} \times I_{\text{tails}} = \frac{1}{2} \times 1 = 0.5$$

Step 3: Add them up

$$H = 0.5 + 0.5 = 1 \text{ bit}$$

A fair coin has entropy 1 bit. Maximum uncertainty for two outcomes!

Example 2: Biased Coin (90% heads)

$$p_{\text{heads}} = 0.9, \quad p_{\text{tails}} = 0.1$$

Surprises:

$$I_{\text{heads}} = -\log_2(0.9) \approx 0.152 \text{ bits}$$

$$I_{\text{tails}} = -\log_2(0.1) \approx 3.322 \text{ bits}$$

(Tails is rare, so very surprising when it happens!)

Weighted:

$$0.9 \times 0.152 \approx 0.137$$

$$0.1 \times 3.322 \approx 0.332$$

Total entropy:

$$H \approx 0.137 + 0.332 = 0.469 \text{ bits}$$

Less than 1 bit! The biased coin is more predictable, so less entropy.

Example 3: Certain Outcome (100% heads)

$$p_{\text{heads}} = 1, \quad p_{\text{tails}} = 0$$

Surprise for heads:

$$I_{\text{heads}} = -\log_2(1) = 0 \text{ bits}$$

Entropy:

$$H = 1 \times 0 = 0 \text{ bits}$$

Zero entropy! No uncertainty means no information when we learn the outcome.

---

Part XI: The Entropy Formula as Triangle Geometry

Each Term is a Rectangle Under a Curve

Remember: $$p_i \times (-\log_2 p_i)$$

This is the area of a rectangle:

• Width = $$p_i$$
• Height = $$-\log_2 p_i$$

Height = -log₂(pᵢ)
    │
    │  ┌─────┐
    │  │     │
    │  │     │
    │  │     │
    └──┴─────┴──
       Width = pᵢ

Entropy H = sum of all these rectangle areas.

The Entropy Curve

Plot $$f(p) = -p \log_2(p)$$ for p from 0 to 1:

f(p)
 │
 │    *****
 │   *     *
 │  *       *
 │ *         *
 │*           *
 └─────────────── p
 0    0.5    1

Maximum at p = 0.5!

• At p = 0: f(0) = 0 (no probability, no contribution)
• At p = 1: f(1) = 0 (certain event, no surprise)
• At p = 0.5: f(0.5) = 0.5 (maximum contribution)

The 45-45-90 triangle (p = 0.5) gives maximum entropy per outcome!

Why Does the 45-45-90 Triangle Maximize Entropy?

When you have two outcomes with probabilities p and (1-p):

$$H(p) = -p\log_2(p) – (1-p)\log_2(1-p)$$

This function is maximized when p = 0.5.

H(p)
 │
 1│      *
 │     * *
 │    *   *
 │   *     *
 │  *       *
 └──────────── p
 0   0.5   1

At p = 0.5: H = 1 bit (maximum for two outcomes)
At p = 0 or p = 1: H = 0 bits (minimum — no uncertainty)

The 45-45-90 triangle represents the equal split (p = 0.5) that maximizes uncertainty!

---

Part XII: Putting It All Together

The Full Formula, One More Time

$$H = -\sum_{i} p_i \log_2 p_i$$

Now you know every piece:

Symbol	Meaning	Triangle Connection
H	Entropy (average surprise)	Total area of weighted triangles
Σ	“Add them all up”	Stack triangles together
i	Index (which outcome)	Label for each triangle
$$p_i$$	Probability of outcome i	Area of outcome’s slice in possibility space
$$\log_2$$	“How many halvings?”	Counting triangle subdivisions
$$-\log_2 p_i$$	Surprise of outcome i	How many halvings to reach that small slice

The Recipe

1. List all possible outcomes (i = 1, 2, 3, …)
2. For each outcome, find its probability $$p_i$$
3. For each outcome, calculate its surprise $$-\log_2 p_i$$
4. For each outcome, weight surprise by probability $$p_i \times (-\log_2 p_i)$$
5. Add all the weighted surprises → This is H!

The Meaning

Entropy H tells you how uncertain you are on average.

• High H → Very uncertain → Takes many bits to describe what happened
• Low H → Pretty certain → Takes few bits to describe what happened
• H = 0 → Completely certain → No bits needed (you already knew!)

---

Part XIII: Information as Distinction — The Deepest View

Every Bit Is a Distinction

Shannon’s entropy counts distinctions.

When you flip a fair coin and learn the result:

• Before: 2 possibilities (can’t distinguish heads from tails)
• After: 1 possibility (you can distinguish)
• Distinction gained: 1 bit

When you roll a die and learn the result:

• Before: 6 possibilities
• After: 1 possibility
• Distinction gained: $$\log_2(6) \approx 2.58$$ bits

The Triangle as the Atom of Distinction

The 45-45-90 triangle represents the minimal distinction:

        /|\
       / | \
      /  |  \
     / A | B \
    /___ |____\

Two regions, A and B, with equal area.
Learning "it's A" or "it's B" = 1 bit.

All information is built from these binary distinctions. More complex situations are just multiple triangles stacked:

4 possibilities = 2 bits = 2 triangle splits

        /\
       /  \
      /    \
     /______\         First split: left or right? (1 bit)
    /\      /\
   /  \    /  \       Second split: which half? (1 bit)
  /____\  /____\

Total: 2 bits

Entropy Counts the Average Number of Triangle Splits

When outcomes aren’t equally likely, you need fractional splits. That’s what the logarithm gives you.

• Certain event (p=1): 0 splits needed
• Fair coin (p=1/2): 1 split needed
• 1-in-8 chance (p=1/8): 3 splits needed

Entropy averages these, weighted by how often each happens.

---

Part XIV: The Relational Meaning

Information Is About Relationships

Shannon’s formula doesn’t care about what the outcomes mean. It only cares about the relationship between observer and observed.

Before measurement: uncertainty (potential distinctions)
After measurement: certainty (actualized distinctions)

Information = the transition from potential to actual distinction.

The Triangle as Relationship

A triangle isn’t three points in space. A triangle is the relationship between three nodes.

      A
     /|\
    / | \
   /  |  \
  B───┼───C

• The relationship A-B (one edge)
• The relationship B-C (one edge)
• The relationship A-C (one edge)

The triangle doesn’t exist “in” space. The triangle defines a structure of relations.

Similarly, information doesn’t exist “in” messages. Information is the structure of distinctions between states.

---

Part XV: Summary — The Equation Demystified

$$H = -\sum_{i} p_i \log_2 p_i$$

In plain English:

“Add up, for each possible outcome, the probability of that outcome times the number of yes/no questions needed to identify it.”

In triangle language:

“Stack up, for each slice of possibility space, the area of that slice times the number of halvings needed to reach that slice.”

In distinction language:

“Average the number of binary distinctions needed to resolve your uncertainty.”

---

Final Thought: Why Triangles?

We started with Shannon’s formula and explained it using only triangles. But this isn’t just a teaching trick.

Triangles are fundamental because distinction is fundamental.

• The triangle is the minimal closed structure
• Two regions require at least one boundary
• One boundary dividing two regions = one bit
• All information is stacked distinctions
• All distinctions are geometrically triangular

Shannon discovered that information has a geometry. That geometry is made of triangles.

When you understand the 45-45-90 triangle, you understand the bit. When you understand the bit, you understand information. When you understand information, you understand… everything.

---

“Information is the resolution of uncertainty.” — Claude Shannon

“The triangle is the resolution of distinction.” — This course

---

Now go forth and count your halvings.

---

Sources:

• Shannon, C. E., “A Mathematical Theory of Communication,” 1948
• Cover & Thomas, “Elements of Information Theory,” 2006
• The 45-45-90 triangle: Pythagoras, ~500 BCE (still going strong)