Brute-Force and Dictionary Attacks on Vigenère

Two programs can attack the same Vigenère ciphertext by trying possible keys, yet perform fundamentally different searches.

One program might test:

until every possible five-letter A–Z key has been considered.

That is a brute-force attack.

Another program might try:

using a prepared list of words or other likely keys.

That is a dictionary attack.

Both methods repeatedly decrypt the ciphertext with candidate keys, but the search strategy is different. Brute force aims to cover an entire defined keyspace. A dictionary attack deliberately ignores most of that keyspace in exchange for testing likely human choices much sooner.

If you already know the key and only need the message, use the Vigenère Cipher Decoder and Encoder. These attacks are relevant when the key is unknown and must be searched for.

What “brute force” means for a Vigenère key

An exhaustive attack needs a clearly defined search space.

Suppose the cipher uses the standard 26-letter alphabet and the attacker knows that the key contains exactly m letters.

Each key position has 26 possibilities, so the number of possible key strings is:

For a three-letter key:

For a five-letter key:

For an eight-letter key:

Every extra A–Z key character multiplies the search space by 26.

These figures describe the mathematical number of candidate strings. They do not by themselves tell you how many seconds an attack will take. Actual speed depends on the implementation, hardware, ciphertext length, scoring method, and how much work is performed for each candidate.

Unknown key length changes what “exhaustive” means

Saying that an attack tries “every key” is incomplete unless the maximum length is specified.

If the attacker knows only that an A–Z key is between one and five characters long, the complete search is:

If the permitted maximum increases to eight characters:

Without some bound on the alphabet and candidate lengths, there is no finite list that deserves to be called an exhaustive search.

In practice, knowledge about the expected key length can therefore matter almost as much as the raw speed of the search program.

A dictionary attack makes a completely different bet

A dictionary attack does not try to represent the entire keyspace.

Instead, it assumes the key was chosen from a predictable set.

That set might contain ordinary words, names, place names, historical terms, previously used keys, topic-specific vocabulary, or other strings that a human might reasonably remember and select.

Suppose the suspected key length is five characters.

Exhaustive A–Z search has:

If an analyst has a hypothetical list containing 10,000 plausible five-letter keywords, a dictionary attack tests only those 10,000.

That is a dramatic reduction in work.

But the reduction is purchased by giving up completeness.

If the true key is QJXMV and the dictionary contains only ordinary words, the correct candidate may never be tested.

The program can execute perfectly and still fail because its assumption about how the key was chosen was wrong.

Work through a dictionary attack

Consider this ciphertext:

Assume there is reason to believe that the key is an ordinary five-letter English word.

A small candidate list might include:

The program does not compare those words directly with the ciphertext. It uses each word as a Vigenère key and decrypts the entire message.

With WATER:

With STONE:

With RAVEN:

The third candidate produces coherent English:

The candidate key is therefore:

This example looks obvious to a human because only a few candidates are displayed. A real wordlist can contain thousands or millions of entries, so an automated attack also needs a way to rank the resulting plaintexts.

Trying keys is only half of the attack

A search program has two separate jobs:

The first job receives most of the attention, but the second can be just as important.

Exhaustive search through millions of keys is not useful if every decrypted result has to be inspected manually.

Candidate plaintexts therefore need a fitness function or other recognition rule.

A simple scorer might count recognizable words. A stronger language scorer can examine common letter sequences such as bigrams, trigrams, or tetragrams and reward candidate plaintexts whose sequence frequencies resemble the expected language.

This is different from the column frequency analysis used to recover individual Vigenère shifts. Here, a complete candidate key has already produced a complete candidate plaintext, so normal adjacency between letters has been restored and sequence statistics become meaningful.

Why Index of Coincidence alone is a poor final key scorer

A tempting shortcut is to decrypt with every candidate key and choose whichever plaintext has an Index of Coincidence close to English.

That is not enough.

The Index of Coincidence measures how uneven the letter-frequency distribution is. It does not care which letters carry those frequencies or whether the letters occur in sensible order.

Two texts can therefore have similar IC values while only one is readable English.

A substitution can also preserve frequency unevenness without producing plaintext.

IC can be useful as one filter, but complete key search benefits from a scoring method that evaluates more information than coincidence alone.

Short ciphertext creates a recognition problem

A large keyspace is not the only difficulty in brute-force search.

Very short ciphertext can make it hard to determine which candidate is actually correct.

Consider a candidate plaintext only six letters long.

Many wrong keys may produce fragments that accidentally resemble words, names, abbreviations, or pronounceable strings.

Statistical language scores are also much less stable when they have only a few characters to evaluate.

This produces an important trade-off. Longer ciphertext makes each decryption attempt slightly more expensive, but it can make the correct plaintext much easier to distinguish from false candidates.

Key length dominates exhaustive-search growth

If a candidate key requires decrypting N ciphertext letters, a simple exhaustive attack on all A–Z keys of fixed length m performs work on the order of:

Increasing the ciphertext from 100 characters to 200 roughly doubles the decryption work per candidate.

Increasing the key length from five letters to six multiplies the number of candidate keys by 26:

This is why key length changes exhaustive search much more dramatically than a modest increase in message length.

Absolute timing claims age quickly and vary greatly between implementations. Keyspace mathematics is more useful because it remains true regardless of the computer performing the search.

A known period can shrink the search before brute force begins

Brute force should not be confused with refusing to use information.

If earlier cryptanalysis suggests that the repeating period is 5, there is no reason to exhaustively test every key length from 1 through 12.

The search can be restricted to the candidate structure supported by the evidence.

In many Vigenère ciphertexts, structural methods can reduce the problem further by identifying individual shift candidates rather than treating the complete keyword as one opaque string.

That is why exhaustive key enumeration is not always the most efficient first attack against repeating-key Vigenère. The cipher leaks periodic structure that can often be investigated before a large raw search is attempted.

The relationship between period discovery and later key recovery is covered in the broader guide to breaking the Vigenère cipher .

Dictionary filtering becomes stronger when the period is known

Suppose period analysis strongly suggests a five-letter key.

A dictionary attack no longer needs to test two-letter, seven-letter, or twelve-letter words unless there is a reason to believe the written key and effective period differ.

The candidate list can be filtered before decryption:

This is an example of two different kinds of evidence working together: structural cryptanalysis narrows the shape of the key, while the dictionary supplies plausible human choices within that shape.

Dictionary coverage matters more than dictionary size

A wordlist containing a million entries is not automatically better than one containing ten thousand.

What matters is whether it represents the kinds of keys the sender was likely to choose.

If the key is a person’s name and the list contains only ordinary vocabulary, adding hundreds of thousands of obscure dictionary words may not solve the real coverage problem.

Similarly, the actual key might be:

Some are compounds, some are names, and some may not appear in a conventional dictionary at all.

A dictionary attack is therefore best understood as an assumption about key selection, not as a magical list of every memorable key.

Hybrid attacks sit between a dictionary and full brute force

Human-chosen keys are not always exact dictionary entries.

An analyst can start with likely words and generate limited variations rather than immediately exploring the full A–Z keyspace.

For a base candidate such as:

a targeted candidate generator might also test combinations or transformations suggested by the particular puzzle or context.

This strategy is broader than a literal dictionary lookup but still narrower than exhaustive enumeration.

Key normalization can silently make an attack fail

Classical Vigenère implementations often normalize keys before use.

A key written as:

might be processed as:

Another implementation may reject spaces or handle them differently.

Case is normally irrelevant in an A–Z Vigenère cipher, but punctuation, accented letters, and nonalphabetic symbols require an explicit convention.

If the candidate generator and the original encryption process normalize keys differently, the right human idea can be present in the list while the exact key sequence used by the cipher is never tested.

The same consistency is required when processing the ciphertext itself: candidate decryption must advance the key under the same character rules used during encryption.

A dictionary failure does not make brute force the automatic next step

Suppose every candidate in a large wordlist has been tested and none produces convincing plaintext.

Several explanations remain possible.

Good cryptanalysis diagnoses those possibilities before committing to a vastly larger exhaustive search.

What brute force guarantees, and what it does not

An exhaustive attack can make a strong claim only about the space it actually searched.

If every A–Z key of exactly five characters is tested, then every candidate in that defined set has been examined.

That does not prove the message used a five-letter key.

It does not prove the underlying plaintext is English.

It does not prove the ciphertext is standard repeating-key Vigenère.

And it does not remove the need to recognize which candidate plaintext is correct.

“Exhaustive” describes the search relative to stated boundaries. The boundaries themselves still have to be justified.

The practical distinction between the two attacks

Brute force and dictionary search are often grouped together because both can be implemented as repeated key guesses.

Their logic is different enough that the distinction matters.

The first trades computational work for completeness.

The second trades completeness for a better search order.

Against Vigenère, neither should be considered in isolation from the structure of the cipher. Evidence about the period, plaintext language, key-selection habits, and ciphertext length can determine whether a small targeted search is sensible or whether exhaustive enumeration is wasteful.

The most important lesson is therefore not simply that computers can “try lots of keys.” It is that effective key search combines a well-defined candidate space with a reliable way to recognize the correct plaintext.