Naar de inhoud
OneKitly

English Carries About One Bit Per Letter and the Rest Is Insurance

Gepubliceerd op 13-11-2026 · 3 min leestijd · Ontwikkelaarstools

Daniel Okonkwo

Daniel Okonkwo — Front-enddeveloper en techredacteur bij OneKitly

Webprestaties · Bestandsformaten

Getoetst aan 2 bronnen

Profiel bekijken →
Kort gezegd

Shannon entropy measures how much a symbol tells you, in bits, and it is largest when every outcome is equally likely. Four equally likely symbols carry exactly 2 bits each; skew them to 50, 25, 12.5 and 12.5 per cent and the average drops to 1.75 bits, a redundancy of 12.5%. Written English is far more skewed than that: with real letter frequencies each letter carries 4.18 bits against a maximum of 4.70, and once the surrounding words are taken into account Shannon's own experiments put the figure near 1.1. Roughly three quarters of every English text is therefore predictable — which is what makes both compression and reading in a noisy room possible.

Advertentie

26 equally likely letters would carry 4.70 bits each. Real letter frequencies give 4.18, and context cuts it to about 1.1 — a redundancy of roughly 77%.

Redundancy is a feature, not waste

A language with no redundancy would be unreadable the moment a letter was smudged, because every possible string would be a valid word and no error could be detected. Human languages sit around three quarters redundant, which is why a sentence survives a bad phone line, a foreign accent and a page of handwriting. The same principle is designed deliberately into machines: error-correcting codes add redundancy on purpose, and a QR code still scans with a third of its area destroyed for exactly the reason a smudged word is still readable.

What a compressor is actually removing

Entropy sets a hard floor: no lossless compressor can average fewer bits per symbol than the source's entropy, and every one of them is a machine for approaching that floor. Text compresses well because there is a great deal of redundancy to take out; already-compressed files, encrypted files and random noise compress not at all, since their entropy is already at the maximum. That is a useful field test — anything that refuses to shrink is either already compressed or is not the kind of data you thought it was.

Redundancy
Entropy of four symbols under two distributions
DistributionEntropyRedundancy
25, 25, 25, 252.00 bits0%
50, 25, 12.5, 12.51.75 bits12.5%
English letters4.18 of 4.70 bits11.2%

Uitgewerkt met onze eigen calculator

Calculator shannon-entropie

Gegeven

Kansen (tellen op tot 1)
0.25, 0.25

Resultaat

Entropie (bits)
1
Maximale entropie (bits)
1
Redundantie
0%

Deze cijfers komen uit de calculator hieronder, ze zijn niet met de hand ingetypt — ze worden opnieuw berekend telkens als de tool verandert.

Reken het door met je eigen cijfers →
Advertentie

Veelgestelde vragen

Why measure information in bits?
Because a bit is the answer to one yes-or-no question, and any uncertainty can be resolved by asking enough of them. Two bits identify one of four equally likely options, ten bits identify one of 1024. Entropy is simply the average number of such questions needed, which is why it comes out fractional: 1.75 bits means that a long run of symbols can be pinned down with 1.75 questions each on average, even though no individual question can be three quarters asked.
Do other languages come out the same?
Close, once the alphabet size is accounted for. Languages with more letters start from a higher maximum and end up in a similar place, because the redundancy is imposed by how humans speak and listen rather than by the writing system. Where the measurements diverge is with scripts that encode syllables or whole words, since a symbol then carries far more information and the per-symbol entropy rises accordingly — comparisons across writing systems have to be made per unit of meaning, not per character.

Artikelen die je misschien interesseren

Alle gidsen →
UitlegPassword Entropy: What a Strength Meter Cannot KnowEntropy measures the process that produced a password, not the characters in it. H = L x log2(R) is only true when every character was chosen at random — which is exactly why a meter scoring a human-invented password on its character classes is measuring the wrong thing.UitlegIf Three People Each Name Three Contacts Among Four, Someone Is WrongThe list 3, 3, 3, 1 cannot describe any real network. 3, 3, 2, 2 can, with five links. Two centuries-old rules catch survey errors before any analysis starts.UitlegThree to One Means One Chance in Four, Not ThreeFractional odds count the ways to lose against the ways to win. 3/1 is 25%, 1/3 is 75%, and 5/2 is 28.57% — the fraction is never the probability.UitlegA Secret Santa Draw Fails 63 % of the Time, Whatever the Group SizeOf the 120 ways five people can draw names, 44 give nobody their own — 36.7 %. With ten people it is 36.788 %, and with twenty it is 36.788 %. The proportion stops moving almost immediately.UitlegThe Pigeonhole Principle Proves a Thing Exists Without Finding ItTen items in three containers force four into one of them. 367 people force a shared birthday. Neither statement tells you which container or which pair.UitlegIf the Variance Is Not the Mean, It Is Not PoissonWith an average of five events per interval, exactly three occur 14.0 % of the time, none at all 0.7 %, and ten or more 3.2 % — about once every 31 intervals. The distribution has one parameter, and it sets both the mean and the variance.

Vergelijkbare tools

Bronnen

Een fout in dit artikel gezien?