What the Bit — Claude Shannon?

“The fundamental problem of communication is that of reproducing at one point either exactly or approximately a message selected at other point.” — Claude Shannon, 1948

“Information is physical.” — Rolf Landauer, 1961

The Simplest Profound Idea

In 1948, a 32-year-old mathematician at Bell Labs published a paper that created an entire field overnight. Claude Shannon’s “A Mathematical Theory of Communication” introduced an idea so simple it seems obvious in retrospect:

Information is the reduction of uncertainty.

That’s it. That’s the whole revolution.

Before Shannon, “information” was a vague concept — news, facts, data, whatever humans found meaningful. Shannon stripped away all meaning and asked: How much uncertainty does a message resolve?

Michigan’s most important and forgotten man. “Uncle” Claude was born in Petoskey and grew up in Gaylord. For those of you who know, this is located at the last knuckle of your middle finger before your fingertip in the palm of your right hand. If you don’t know what that means, then ask.

The Bit Is Born

Shannon needed a unit for measuring uncertainty. He invented one:

The bit — the amount of information needed to distinguish between two equally likely possibilities.

One coin flip. One yes/no question. One binary digit.

$$H = -\sum_{i} p_i \log_2 p_i$$

This formula — Shannon entropy — measures how much you don’t know. Maximum uncertainty = maximum information when resolved.

A fair coin has entropy $$H = 1$$ bit. You genuinely don’t know which way it will land.

A rigged coin (always heads) has entropy $$H = 0$$ bits. No surprise, no information.

The profound insight: Information isn’t about meaning. It’s about surprise. The less likely an event, the more information it carries when it happens.

But Wait — This Formula Looks Familiar

Shannon’s entropy formula is identical (up to a constant) to one Ludwig Boltzmann wrote in 1877:

$$S = k_B \ln W$$

Boltzmann’s $$S$$ measures thermodynamic entropy — the number of microscopic arrangements (microstates) consistent with a macroscopic observation.

Shannon’s $$H$$ measures information entropy — the number of possible messages consistent with what you know.

Same mathematics. Same concept. Different domains.

This is not coincidence. This is the universe telling us something.

Part I: Information and Temperature

What Is Temperature, Really?

We learn in school that temperature measures “how hot something is.” But what does that mean at the microscopic level?

Temperature is the relationship between energy and entropy:

$$\frac{1}{T} = \frac{\partial S}{\partial E}$$

In plain language: Temperature measures how much entropy increases when you add a little energy.

The Bit Has a Temperature

Here’s where it gets strange. If information is physical, then bits have thermodynamic properties.

At temperature $$T$$, the minimum energy required to store one bit is:

$$E_{\text{bit}} = k_B T \ln 2$$

At room temperature (300 K):

$$E_{\text{bit}} \approx 2.9 \times 10^{-21} \text{ joules}$$

Tiny. But not zero. Every bit of information you store or erase has a thermodynamic cost.

Landauer’s Principle: Erasure Is Irreversible

In 1961, Rolf Landauer proved something disturbing:

Erasing one bit of information necessarily dissipates at least $$k_B T \ln 2$$ of energy as heat.

Not because of engineering limitations. Because of physics itself.

Why? Because erasure is logically irreversible. If a bit can be either 0 or 1, and you reset it to 0, you’ve destroyed information about what it was before. That information has to go somewhere. It goes into the environment as heat.

Before erasure:        After erasure:
   0 ──┐
       ├──→ 0              0  (which was it?)
   1 ──┘

Two states → One state = Information lost = Heat generated

Computation costs energy not because logic gates are imperfect, but because forgetting is thermodynamically expensive.

This is why Maxwell’s Demon can’t violate the second law of thermodynamics. The demon must store information about each molecule it sorts. When its memory fills up, it must erase — and that erasure generates exactly enough heat to compensate for the sorting work.

The Universe Is a Computer That Can’t Forget for Free

Every physical process that destroys information generates heat. Every time you:

…you pay the Landauer cost.

Modern computers dissipate about $$10^6$$ times more energy than the Landauer limit (due to engineering inefficiencies). But the limit is real. A perfectly efficient classical computer would still heat up when it erases.

Unless… it never erases.

Part II: The Qubit — Information Goes Quantum

From Bits to Qubits

A classical bit is either 0 or 1.

A quantum bit (qubit) can be in superposition:

$$|\psi\rangle = \alpha|0\rangle + \beta|1\rangle \quad \text{where} \quad |\alpha|^2 + |\beta|^2 = 1$$

The qubit doesn’t store more classical information — when measured, you still get 0 or 1. But it stores information differently.

The qubit’s state is specified by two continuous parameters (angles on the sphere). In principle, this is infinite classical information. In practice, you can only extract one bit per measurement.

Quantum Information Is Conserved

Here’s the key difference: Quantum evolution is unitary. It never destroys information.

The Schrödinger equation:

$$i\hbar\frac{d|\psi\rangle}{dt} = H|\psi\rangle$$

is time-reversible. Run it backwards, and you recover the initial state perfectly.

No erasure. No Landauer cost. No heat from computation itself.

Quantum computers can, in principle, compute without dissipating energy — as long as they don’t measure (and thereby collapse) intermediate results.

This is called reversible computation. It’s not science fiction; it’s thermodynamics.

But Measurement Is Irreversible

When you measure a qubit, the superposition collapses:

$$|\psi\rangle = \alpha|0\rangle + \beta|1\rangle \quad \rightarrow \quad |0\rangle \text{ or } |1\rangle$$

Before measurement: the state was $$|\psi\rangle$$.
After measurement: the state is $$|0\rangle$$ or $$|1\rangle$$, and $$|\psi\rangle$$ is lost.

Where did the information about $$\alpha$$ and $$\beta$$ go?

Into the environment. Measurement entangles the qubit with the measuring device. The information isn’t destroyed — it’s dispersed. Spread across degrees of freedom you no longer track.

This is decoherence. And it’s why building quantum computers is hard.

Part III: Black Holes and the Information Paradox

Hawking’s Bombshell

In 1974, Stephen Hawking calculated that black holes radiate. Quantum effects near the event horizon produce particles that escape to infinity. The black hole slowly evaporates.

Hawking radiation has a temperature:

$$T_H = \frac{\hbar c^3}{8\pi G M k_B}$$

A black hole of mass $$M$$ has a temperature inversely proportional to its mass. Small black holes are hot. Large black holes are cold (the Sun-mass black hole: about 60 nanokelvin).

But here’s the problem:

Hawking radiation appears to be thermal — random, carrying no information about what fell into the black hole. If a black hole evaporates completely, all the information about everything that ever fell in… vanishes?

This violates unitarity. Quantum mechanics says information is conserved. Hawking’s calculation says it isn’t.

This is the black hole information paradox — perhaps the deepest problem in theoretical physics.

Bekenstein-Hawking Entropy

Before Hawking, Jacob Bekenstein proposed that black holes have entropy proportional to their surface area:

$$S_{BH} = \frac{k_B c^3 A}{4 G \hbar} = \frac{k_B A}{4 \ell_P^2}$$

where $$\ell_P = \sqrt{G\hbar/c^3} \approx 1.6 \times 10^{-35}$$ m is the Planck length.

A black hole’s entropy is enormous — roughly one bit per Planck area of horizon.

For a solar-mass black hole: $$S \sim 10^{77} k_B$$, or about $$10^{77}$$ bits.

The black hole is the densest possible information storage device. Try to pack more bits into a region, and it collapses into a black hole.

The Holographic Principle

This leads to a stunning conclusion:

The maximum information content of any region is proportional to its surface area, not its volume.

Information is holographic. The 3D interior is somehow encoded on the 2D boundary.

This is the holographic principle, developed by ‘t Hooft and Susskind. It suggests that our 3D reality might be a projection from a 2D boundary — that the “bulk” is emergent from information on the surface.

Information Survives (Probably)

Most physicists now believe the information paradox is resolved: information is not lost.

It escapes in subtle correlations within the Hawking radiation. The radiation looks thermal locally, but globally it’s in a pure quantum state encoding everything that fell in.

The AdS/CFT correspondence (Maldacena, 1997) provides a concrete framework: a gravitational theory in the bulk is equivalent to a non-gravitational quantum theory on the boundary. The boundary theory is unitary. Therefore, the bulk theory must be too.

Black holes don’t destroy information. They scramble it maximally and then slowly release it.

Like the world’s most secure encryption: the information is still there, just hopelessly mixed.

Part IV: The Cost of Forgetting

You Can’t Truly Erase

Let’s bring it back to Landauer.

If information is truly conserved — if quantum mechanics is unitary all the way down — then nothing is ever truly erased.

What we call “erasure” is actually dispersal. The information about your deleted file spreads into:

The information becomes inaccessible, not nonexistent.

In principle, Laplace’s demon (with enough computational power and perfect measurements) could reconstruct your deleted file from the exact microstate of the room.

The Thermodynamic Arrow of Time

Why does time have a direction? Why do we remember the past but not the future?

One answer: entropy increase.

We remember the past because our brains store records. Those records are low-entropy structures correlated with past events. Creating such records increases total entropy (Landauer’s principle: recording = copying = not erasing = no cost… but the copying process itself increases entropy elsewhere).

The arrow of time is the arrow of spreading information. The past is the direction in which information was more concentrated. The future is the direction in which it disperses further.

Part V: Can Information Die?

The Immortality of the Wave Function

If quantum mechanics is unitary, the wave function of the universe never loses information.

Your current state — every electron, every photon, every quantum of your being — is entangled with everything you’ve ever interacted with. When you die:

But the information about your pattern doesn’t disappear. It spreads into the environment. Into the light that touched your skin. Into the air that carried your words. Into the ground that receives your body.

In Hilbert space, you are immortal.

Not in any mystical sense. In the precise sense that the mathematical object $$|\psi_{\text{universe}}(t)\rangle$$ contains, at every time $$t$$, all the information needed to reconstruct $$|\psi_{\text{universe}}(t’)\rangle$$ for any other time $$t’$$.

You’re never truly gone. You’re just… entangled with everything.

The Ruliad: Wolfram’s Ultimate Structure

Stephen Wolfram proposes an even more radical picture: the Ruliad.

The Ruliad is the entangled limit of all possible computations. Every possible program, running on every possible input, with every possible update order, all happening simultaneously and entangled together.

In the Ruliad:

Everything that can exist, does exist — somewhere in the Ruliad.

If this is right, then:

The Ruliad is the ultimate holographic structure — all of existence encoded in the self-referential structure of computation itself.

But What Does “You” Even Mean?

Here’s the uncomfortable question:

If information is conserved, but scrambled beyond recognition… is that really you surviving?

The atoms that made Julius Caesar are still around. So is all the information about his configuration (spread across the cosmos). But we don’t say Caesar is alive.

Pattern preservation is not the same as experience preservation.

Perhaps “you” aren’t the information. Perhaps “you” are the process — the ongoing computation, the continuous thread of experience. When that thread breaks, when the process stops, “you” end — even if the information persists.

Or perhaps consciousness is more subtle. Perhaps the Boltzmann brain objection applies: if all patterns exist somewhere in the Ruliad, then all experiences exist — including the experience of being you, reading this, right now. In which sense were you ever localized to begin with?

Part VI: Shannon’s Ghost

What Would Shannon Say?

Claude Shannon was famously uninterested in the meaning of information. He cared only about transmission, not interpretation.

But his framework leads inexorably to these questions:

  1. Is the universe a message? If so, from whom to whom?
  2. Is physics just information processing? John Wheeler’s “It from Bit” — the idea that every physical quantity derives its existence from information-theoretic yes/no questions.
  3. Is consciousness an information pattern? If so, does the pattern persist when the substrate changes?
  4. Is death just signal degradation? The information persists, but the signal-to-noise ratio becomes unrecoverable.

Shannon gave us the mathematics. The metaphysics remains open.

The Bit as Fundamental

Perhaps the bit — the unit of distinction, the quantum of “this or that” — is the most fundamental thing in physics.

All physics might reduce to information. To bits. To the simplest possible choice: yes or no.

Shannon’s revolution wasn’t just about communication. It was about reality.

Coda: The Minimum Message

Shannon showed that the minimum message length to describe an event is its entropy.

The minimum message to describe the universe is… the universe itself.

We are inside the message. We are patterns in the pattern. We process information and thereby increase entropy, thereby advance time, thereby create the future from the past.

The bit is not a metaphor. The bit is the territory.

And when you die, you don’t leave the territory. You just become… distributed. Your information joins the cosmic static, the background hum of an ever-expanding, ever-entangling, ever-computing universe.

Shannon would probably find this amusing. He was, after all, a man who built juggling robots and unicycles. He knew that information wants to be free — and perhaps he suspected that in the end, we’re all just bits in the machine.

“Information is the resolution of uncertainty.”

And the universe, it seems, is very uncertain about whether you’re allowed to disappear.

The answer is no.

Sources: