Friedman Test for Vigenère Cipher Key Length

Suppose a Vigenère ciphertext contains 600 alphabetic characters and its measured Index of Coincidence is:

Using the Friedman test with common English-language reference values, the estimated repeating-key period is:

That does not prove that the key has exactly six letters. It means that a period around 6 is statistically plausible and deserves to be tested.

This is what the Friedman test contributes to Vigenère cryptanalysis: it converts the overall coincidence rate of a ciphertext into a direct estimate of the repeating period.

Unlike the Kasiski examination, it does not require visible repeated trigrams or other repeated ciphertext sequences. It works from the letter-frequency distribution of the ciphertext as a whole.

If you already have the key and simply need to process a message, use the Vigenère Cipher Decoder and Encoder. The Friedman test is useful when the repeating key is unknown.

What the Friedman test is estimating

The phrase key length is commonly used when describing the Friedman test, but the more precise target is the period of the repeating keystream.

If the Vigenère key is:

the sequence of shifts repeats every five positions:

Its period is therefore 5.

If a written key contains its own internal repetition, the effective period can be shorter than the number of letters written in the key. For example:

repeats after three positions, even if someone originally describes the keyword as CATCAT.

Statistical period-finding methods respond to the pattern actually repeated by the cipher, not necessarily to the way a keyword was written down.

The Friedman formula

Let:

Using the commonly rounded English values 0.065 for language-like coincidence and 0.038 for near-uniform A–Z coincidence, the finite-sample Friedman estimate is:

The number 0.027 is simply the difference:

A more general way to write the same idea is:

where:

This general form is useful because the constants are assumptions, not universal cryptographic constants.

Why the formula can estimate a Vigenère period

The logic comes from asking what happens when two ciphertext letters are selected from different positions in a repeating-key message.

There are two important cases.

The first possibility is that both positions were encrypted by the same key position.

If the key period is m, this happens roughly one time in every m position classes.

Both characters then received the same Caesar-style shift. A fixed shift changes which letters appear, but it preserves the uneven frequency distribution of the underlying language.

Their coincidence probability therefore behaves approximately like ordinary plaintext:

The second possibility is that the letters came from different key positions.

Those positions normally use different shifts. When several shifted distributions are mixed, their relationship is much closer to the coincidence rate expected from a uniform alphabet:

The ciphertext-wide IC is therefore a mixture of these two kinds of pairs.

The large-sample version makes the intuition clearer

Ignoring finite-text corrections for a moment, the expected ciphertext IC can be approximated as:

Rearranging:

This equation shows the relationship immediately.

If m=1, every character is encrypted under the same shift:

That is what we expect from a monoalphabetic shift: the frequency distribution remains as uneven as ordinary language.

As the period becomes longer, the fraction 0.027/m becomes smaller:

The whole-text IC therefore tends to move closer to the uniform baseline as more independently shifted alphabets are mixed together.

The finite-sample Friedman formula refines this idea by accounting for the fact that a real ciphertext contains a limited number of letters.

Work through the estimate step by step

Return to the example from the beginning:

Start with the numerator:

Now calculate the denominator:

Divide:

So the Friedman estimate is:

The sensible conclusion is not that a key can somehow contain 6.08 letters.

The conclusion is that 6 is a strong period to investigate, while nearby possibilities should remain open until additional evidence is checked.

Do not blindly round the answer

A Friedman result is an estimate built from observed statistics. Sampling variation means it rarely lands exactly on an integer.

Suppose three ciphertexts produce these results:

The decimal value contains uncertainty. Rounding it hides that uncertainty but does not remove it.

A useful workflow is to treat the estimate as the center of a small candidate range and then ask whether those periods produce the statistical behavior expected from recurring single-shift streams.

Use raw IC in the formula, not normalized IC

This is an easy way to obtain a completely meaningless Friedman result.

The formula shown above expects a raw Index of Coincidence on a scale such as:

Some tools and books use a normalized IC instead. A common normalization multiplies the raw value by 26:

Feeding a value around 1.7 into a formula whose constants are around 0.038 and 0.065 mixes two different scales.

Before calculating the Friedman estimate, confirm which IC convention your source or software reports.

Why some Friedman formulas use 0.0265 instead of 0.027

You may encounter a formula that looks slightly different:

This is not a fundamentally different test.

It uses the more precise uniform 26-letter baseline:

Rounded to 0.0385, the difference from 0.065 is:

Other sources use slightly different English-language coincidence values as well.

The important rule is to keep the constants internally consistent. Do not take the numerator from one convention and the random baseline from another.

The English reference value is an assumption

A value such as 0.065 represents an expected coincidence rate for English-like text. It is not a physical constant.

Real English samples vary according to vocabulary, subject matter, author style, and length.

More importantly, other languages have different letter-frequency distributions.

If the plaintext language has a substantially different expected IC, applying English constants changes the estimated period.

The general Friedman formula makes that dependency explicit:

Choose Kp to match the expected plaintext source and Kr to match the alphabet.

For an alphabet of q equally likely symbols, the uniform baseline is:

This is one reason results from different Friedman calculators can disagree slightly even when they analyze the same ciphertext.

Short ciphertext can move the estimate dramatically

The Friedman test uses observed letter frequencies to estimate an underlying probability distribution.

With little text, those observed frequencies can be noisy.

Imagine that two ciphertexts were generated with the same repeating period. One contains 1,000 letters and the other contains only 50.

A few accidental repeated letters make only a small difference in the 1,000-letter sample. The same few coincidences can move the IC of the 50-letter sample substantially.

The Friedman estimate can therefore move substantially as well.

A precise-looking result such as 7.31 should not be mistaken for high precision when it came from a small sample.

Why estimates can become extremely large or even impossible

Look again at the denominator:

When the observed IC is very close to the random baseline, that denominator can become very small.

Dividing by a very small number produces a very large period estimate.

A tiny change in the measured IC can then produce a large change in the answer.

With short or noisy samples, the observed IC may even fall below the theoretical uniform reference. The formula can then return a negative or otherwise nonsensical result.

That does not mean the cipher has a “negative key length.”

It means the statistical assumptions and the available sample are not supporting a useful Friedman estimate.

A broken-looking result should be treated as a warning about the evidence, not repaired by forcing the answer into a positive integer.

Repeated key letters can affect the approximation

The standard derivation treats letters from different key positions as if their substitutions are sufficiently different that their coincidence behavior approaches the random baseline.

A real keyword does not always cooperate perfectly with that model.

Consider:

Different positions in this six-letter key include repeated key letters. Two separate key positions using T, for example, apply the same shift.

Some pairs drawn from nominally different positions can therefore behave more like same-shift pairs than the simple model assumes.

This is another reason the Friedman result should be treated as an estimate rather than a deterministic measurement.

Friedman and Kasiski answer the same problem with different evidence

Both techniques can narrow the unknown repeating period, but they approach the problem from different directions.

Their errors are not identical.

Kasiski can struggle when useful repeated sequences do not appear. Friedman can struggle when the text is too short or its measured coincidence rate is unstable.

Agreement between different methods is therefore stronger evidence than either technique used mechanically on its own.

What to do with the Friedman result

Estimating the period is not the same as recovering the Vigenère key.

If the test suggests a period near 6, the next question is whether dividing the ciphertext into six recurring position streams produces the behavior expected when each stream has been encrypted with one Caesar-style shift.

If the evidence supports that period, each key position can then be investigated separately.

The broader sequence, including recovering the individual shifts and reconstructing the plaintext, is covered in How to Break the Vigenère Cipher .

The calculation assumes a repeating-key Vigenère structure

The Friedman test is not a universal key-length detector that can be applied to any ciphertext.

Its interpretation depends on a model: a polyalphabetic substitution system with a repeating period, language-like plaintext statistics, and an alphabet for which the coincidence reference values make sense.

If the keystream does not repeat in the assumed way, there may be no short period for the formula to recover.

Likewise, unusual plaintext, heavily transformed input, very small samples, or a different cipher family can produce an IC that happens to generate a number without making that number meaningful.

Understanding the construction being tested is therefore as important as performing the arithmetic. If the underlying mechanics are unclear, the relationship between key positions and repeated shifts is explained in How Does the Vigenère Cipher Work? .

Why the test is associated with William Friedman

William F. Friedman developed coincidence-based methods into a systematic statistical tool for cryptanalysis in the early twentieth century.

His work on the Index of Coincidence showed how the probability of matching letters could reveal structure that was difficult to see by inspecting ciphertext directly.

The significance of the Friedman approach is not simply the formula bearing his name.

It turns a qualitative observation — that polyalphabetic encryption flattens language frequencies — into a quantity that can be measured and used to estimate an unknown period.

How to interpret a Friedman estimate responsibly

The Friedman test is most useful when it is treated as a statistical clue rather than an automatic answer.

Before trusting the result, verify that the IC is on the raw scale, that the alphabet and language constants match the text being analyzed, and that the ciphertext is long enough to provide useful frequency evidence.

Then treat the decimal result as a region of interest.

An estimate of 6.08 makes six a strong candidate. An estimate of 6.6 is weaker evidence for any one integer and should encourage testing several nearby periods.

The calculation is valuable because it can turn an unknown period into a much smaller search problem. Its limitation is equally important: it estimates the repeating structure expected from the ciphertext statistics; it does not prove the key length or reveal the key itself.