Keyed-Alphabet Vigenère Explained: How Changing the Alphabet Changes the Cipher

Standard Vigenère assumes one familiar ordering:

ABCDEFGHIJKLMNOPQRSTUVWXYZ

That ordering does more than display the letters.

It assigns their numeric values.

Under standard Vigenère:

A = 0
B = 1
C = 2
...
Z = 25

Change the alphabet order and those values change with it.

That means the same visible plaintext and the same visible repeating key can produce a completely different ciphertext even though the arithmetic still looks like:

C = (P + K) mod 26

This article uses one exact keyed-alphabet convention and keeps it consistent throughout.

The alphabet keyword is:

CIPHER

It produces the mixed alphabet:

CIPHERABDFGJKLMNOQSTUVWXYZ

The repeating Vigenère key is:

ORBIT

Under ordinary Vigenère:

ATTACKATDAWN
+
ORBITORBITOR
=
OKUIVYRULTKE

Under the keyed alphabet defined here:

ATTACKATDAWN
+
ORBITORBITOR
=
WYCBTPJCFZKU

Both results are internally correct.

They belong to different cipher specifications.

If you only need to encode or decode standard Vigenère text, use the Vigenère Cipher Decoder and Encoder. The purpose here is to understand what changes when A–Z is replaced by a keyword-mixed ordering.

“Keyed-alphabet Vigenère” needs an explicit definition

The phrase is not used with perfect consistency across classical-cipher literature and puzzle tools.

Some systems use a mixed alphabet as the symbol order for ordinary Vigenère arithmetic.

Some construct a shifted tableau from one or more keyed alphabets.

Some puzzle traditions use the phrase “keyed Vigenère” for constructions closely related to Quagmire ciphers, where indicator alignment introduces additional conventions.

Those systems can produce different ciphertext from the same visible inputs.

So the label alone is not enough.

This article uses the following direct custom-index convention:

1. Build one keyed permutation of A–Z.
2. Use that same permutation to assign numeric positions to plaintext letters.
3. Use it to assign numeric positions to repeating-key letters.
4. Add those positions modulo 26.
5. Convert the result back through the same keyed alphabet.

If the keyed alphabet is A*, encryption is:

C = A*[(index_A*(P) + index_A*(K)) mod 26]

Decryption is:

P = A*[(index_A*(C) - index_A*(K)) mod 26]

Everything that follows uses those two equations.

There are two different keywords doing two different jobs

A keyed-alphabet construction commonly has two pieces of key material that are easy to confuse.

In this example:

Alphabet keyword: CIPHER
Repeating key:    ORBIT

CIPHER does not repeat across the plaintext.

It is used once to construct the alphabet.

ORBIT does repeat across the plaintext.

It controls the periodic shifts.

Their jobs are:

Calling both of them simply “the key” can make an implementation specification ambiguous.

It is better to distinguish:

alphabet keyword
repeating key

throughout the design.

How to build a keyed alphabet

Start with the alphabet keyword:

CIPHER

Write its letters in order, keeping only the first occurrence of each symbol.

Then append every unused letter from the standard alphabet in normal A–Z order.

For CIPHER, all six letters are already unique:

C I P H E R

The remaining letters are:

A B D F G J K L M N O Q S T U V W X Y Z

So the final alphabet is:

CIPHERABDFGJKLMNOQSTUVWXYZ

with positions:

If the alphabet keyword contains duplicates, remove later occurrences.

For example:

BALLOON

becomes the unique keyword prefix:

BALON

and the resulting keyed alphabet is:

BALONCDEFGHIJKMPQRSTUVWXYZ

A valid keyed alphabet must still contain every A–Z letter exactly once.

Changing the order changes the numeric value of a visible letter

Under the standard alphabet, the key ORBIT has values:

O = 14
R = 17
B = 1
I = 8
T = 19

Under:

CIPHERABDFGJKLMNOQSTUVWXYZ

the same visible key has values:

O = 16
R = 5
B = 7
I = 1
T = 19

The string:

ORBIT

has not changed.

Its numeric meaning has.

That is why taking a normal Vigenère implementation and merely changing the final display alphabet is not enough.

The alphabet order must be used consistently for:

plaintext indexing
key indexing
ciphertext lookup
decryption

If even one stage falls back to standard A=0 ordering, the implementation becomes a different cipher.

The first letter already produces a different result

Take the first plaintext/key pair:

Plaintext: A
Key:       O

Standard Vigenère

Standard positions are:

A = 0
O = 14

Therefore:

0 + 14 = 14

and standard position 14 is:

O

Keyed alphabet

Under the mixed alphabet:

A = 6
O = 16

Therefore:

6 + 16 = 22 mod 26

Position 22 in the keyed alphabet is:

W

So the first ciphertext character changes from:

O

to:

W

before anything else about the message has changed.

Complete worked encryption

The plaintext is:

ATTACKATDAWN

The repeating key is:

ORBITORBITOR

Using the keyed alphabet:

CIPHERABDFGJKLMNOQSTUVWXYZ

gives:

The result is:

WYCBTPJCFZKU

The same plaintext and repeating key under ordinary A–Z Vigenère produce:

OKUIVYRULTKE

The difference is caused entirely by the alphabet ordering.

Decryption uses the same keyed positions in reverse

For the first ciphertext letter:

W

the keyed-alphabet position is:

22

The first key letter is:

O

with keyed position:

16

Subtract:

22 - 16 = 6

Position 6 in the keyed alphabet is:

A

So the first plaintext character is recovered correctly.

Apply the same operation across the message:

Ciphertext:
WYCBTPJCFZKU

Repeating key:
ORBITORBITOR

Recovered plaintext:
ATTACKATDAWN

The encryption and decryption are exact inverses because they use the same alphabet and the same index convention.

A standard Vigenère decoder will not recover this ciphertext

Suppose someone receives:

WYCBTPJCFZKU

and knows the repeating key:

ORBIT

but assumes ordinary A–Z Vigenère.

Standard decryption returns:

IHBTABSBXGWD

not:

ATTACKATDAWN

The key is correct.

The alphabet assumption is wrong.

This is one reason cipher metadata matters.

To reproduce or decode a keyed-alphabet message, the recipient needs at least:

the alphabet construction rule
the alphabet keyword or resulting alphabet
the repeating key
the exact arithmetic/alignment convention
the text-normalization policy

The phrase “Vigenère with key ORBIT” is insufficient.

The keyed alphabet changes the tableau as well as the arithmetic view

Standard Vigenère is often taught with a tabula recta:

ABCDEFGHIJKLMNOPQRSTUVWXYZ
BCDEFGHIJKLMNOPQRSTUVWXYZA
CDEFGHIJKLMNOPQRSTUVWXYZAB
...

Each row is a rotation of the standard alphabet.

Under the direct keyed-alphabet convention, the equivalent cycle starts from:

CIPHERABDFGJKLMNOQSTUVWXYZ

and its rotations.

The zero-shift row is:

CIPHERABDFGJKLMNOQSTUVWXYZ

A one-position rotation is:

IPHERABDFGJKLMNOQSTUVWXYZC

A two-position rotation is:

PHERABDFGJKLMNOQSTUVWXYZCI

The same modular structure remains.

What changed is the circular ordering of the symbols.

This is why a keyed alphabet can be implemented either with a tableau or with numeric indexing.

The numeric form is usually easier to test because the construction rule is explicit.

The symbol set has not changed

This distinction is important.

The keyed alphabet:

CIPHERABDFGJKLMNOQSTUVWXYZ

contains exactly the same 26 symbols as:

ABCDEFGHIJKLMNOPQRSTUVWXYZ

Only their order changed.

Therefore the modulus remains:

26

and the message alphabet is still ordinary A–Z.

This is different from designing a cipher alphabet that contains additional symbols such as:

A–Z
digits
punctuation
accented letters

That would change the alphabet size and therefore the modular arithmetic.

A keyed alphabet is a permutation of a fixed symbol set.

It is not automatically a larger alphabet.

If the keyed alphabet is public, this is a relabeling of Vigenère

Suppose everyone knows:

CIPHERABDFGJKLMNOQSTUVWXYZ

Then each visible symbol can be mapped to its mixed-alphabet index.

For example:

C → 0
I → 1
P → 2
...

Once that mapping is known, the arithmetic is still addition modulo 26.

Structurally, the cipher is Vigenère expressed through a different labeling of the 26 positions.

That means a public keyed alphabet should not be treated as a modern security upgrade.

The periodic key still repeats.

The finite alphabet is still small.

The underlying operation is still classical modular substitution.

The change is primarily representational unless the alphabet itself is secret.

A secret alphabet changes the attacker’s post-period problem

If the alphabet order is unknown, the situation becomes more difficult.

In standard Vigenère, once the period is known, each column is an ordinary Caesar shift over:

ABCDEFGHIJKLMNOPQRSTUVWXYZ

The attacker can try 26 standard shifts and compare the results with language frequencies.

With an unknown mixed alphabet, each column is still a fixed substitution generated by a rotation of the hidden cycle, but that cycle itself is unknown.

The attacker now has to reason about:

the repeating period
the shift relationships
the hidden alphabet order

rather than only standard Caesar shifts.

That makes the post-period recovery problem different.

It does not make the cipher secure by modern standards.

It adds another classical secret structure.

Periodicity does not disappear

The most important weakness of repeating-key Vigenère remains:

the repeating key repeats

If the repeating key has length 5, the same key position returns every five encrypted letters.

Changing the alphabet order does not remove that periodic schedule.

That means techniques based on recurring position structure can still matter.

For example:

• repeated-sequence spacings can still reveal periodic relationships;
• column-based coincidence can still rise at compatible periods;
• multiples of the true period can still appear as strong candidates.

The Kasiski-vs-IC-vs-Friedman-vs-autocorrelation benchmark deals with period detection in ordinary Vigenère. A hidden mixed alphabet changes what happens after a period is suspected more than it changes the fact that a repeated period exists.

Index of Coincidence survives symbol relabeling

Index of Coincidence depends on symbol counts:

IC = Σ fᵢ(fᵢ - 1) / N(N - 1)

If the letters are merely permuted, the frequency counts are also merely permuted.

The sum does not change.

For example, if a substitution relabels:

E → P
T → C
A → R

the counts move to different symbol names, but the collection of count values remains the same.

Therefore a fixed substitution permutation preserves IC.

That is useful in keyed-alphabet analysis because it explains why changing letter labels does not erase every statistical signal.

The identities of the frequent letters change.

The coincidence concentration does not.

Standard frequency tables need the alphabet mapping

There is an important distinction between:

frequency concentration

and:

frequency identity

IC mostly cares about concentration.

Chi-squared Caesar recovery cares about which symbol should correspond to:

E
T
A
O
...

If the keyed alphabet is known, the expected language distribution can be remapped into the keyed ordering.

If the alphabet is unknown, ordinary “try all Caesar shifts against English A–Z frequencies” is no longer the right model.

That is why a keyed alphabet can leave period detection relatively recognizable while making direct key-letter recovery less straightforward.

The alphabet keyword does not necessarily equal the final alphabet prefix literally

The construction removes duplicates.

For:

BALLOON

the prefix is:

BALON

not:

BALLOON

This matters when two implementations exchange only the alphabet keyword.

They must agree on details such as:

case normalization
duplicate removal
base alphabet order
handling of unsupported symbols
whether keyword spaces are ignored or rejected

A reproducible implementation should either publish the final alphabet directly or publish a precise construction function.

The site’s Python Vigenère implementation guide makes the same general point about cipher profiles: arithmetic is not enough when preprocessing rules differ.

The exact Python implementation used for this article

The following script constructs the alphabet, encrypts, decrypts, compares standard Vigenère, and demonstrates the failure of a standard decoder on keyed-alphabet ciphertext.

import string

STANDARD_ALPHABET = string.ascii_uppercase


def build_keyed_alphabet(keyword, base_alphabet=STANDARD_ALPHABET):
    """
    Put the first occurrence of each keyword symbol first, then append
    every unused symbol from the base alphabet in its original order.
    """
    keyword = keyword.upper()

    if len(set(base_alphabet)) != len(base_alphabet):
        raise ValueError("Base alphabet must not contain duplicate symbols.")

    if any(ch not in base_alphabet for ch in keyword):
        raise ValueError("Alphabet keyword contains a symbol outside the base alphabet.")

    seen = set()
    result = []

    for ch in keyword + base_alphabet:
        if ch not in seen:
            seen.add(ch)
            result.append(ch)

    mixed = "".join(result)

    if len(mixed) != len(base_alphabet) or set(mixed) != set(base_alphabet):
        raise AssertionError("Keyed alphabet must be a permutation of the base alphabet.")

    return mixed


def _validate_repeating_key(key, alphabet):
    key = key.upper()
    if not key:
        raise ValueError("Repeating key must not be empty.")
    if any(ch not in alphabet for ch in key):
        raise ValueError("Repeating key contains a symbol outside the cipher alphabet.")
    return key


def keyed_vigenere_encrypt(text, repeating_key, keyed_alphabet):
    """
    Direct custom-index convention used in the article:
      C = A[(index_A(P) + index_A(K)) mod 26]
    where A is the same keyed alphabet for plaintext, key, and ciphertext.
    """
    text = text.upper()
    repeating_key = _validate_repeating_key(repeating_key, keyed_alphabet)

    if any(ch not in keyed_alphabet for ch in text):
        raise ValueError("Text contains a symbol outside the cipher alphabet.")

    index = {ch: i for i, ch in enumerate(keyed_alphabet)}
    m = len(keyed_alphabet)
    out = []

    for i, ch in enumerate(text):
        p = index[ch]
        k = index[repeating_key[i % len(repeating_key)]]
        out.append(keyed_alphabet[(p + k) % m])

    return "".join(out)


def keyed_vigenere_decrypt(text, repeating_key, keyed_alphabet):
    """
    Inverse of keyed_vigenere_encrypt:
      P = A[(index_A(C) - index_A(K)) mod 26]
    """
    text = text.upper()
    repeating_key = _validate_repeating_key(repeating_key, keyed_alphabet)

    if any(ch not in keyed_alphabet for ch in text):
        raise ValueError("Text contains a symbol outside the cipher alphabet.")

    index = {ch: i for i, ch in enumerate(keyed_alphabet)}
    m = len(keyed_alphabet)
    out = []

    for i, ch in enumerate(text):
        c = index[ch]
        k = index[repeating_key[i % len(repeating_key)]]
        out.append(keyed_alphabet[(c - k) % m])

    return "".join(out)


def standard_vigenere_encrypt(text, repeating_key):
    return keyed_vigenere_encrypt(text, repeating_key, STANDARD_ALPHABET)


def standard_vigenere_decrypt(text, repeating_key):
    return keyed_vigenere_decrypt(text, repeating_key, STANDARD_ALPHABET)


if __name__ == "__main__":
    alphabet_keyword = "CIPHER"
    repeating_key = "ORBIT"
    plaintext = "ATTACKATDAWN"

    keyed_alphabet = build_keyed_alphabet(alphabet_keyword)
    keyed_ciphertext = keyed_vigenere_encrypt(
        plaintext,
        repeating_key,
        keyed_alphabet,
    )
    standard_ciphertext = standard_vigenere_encrypt(
        plaintext,
        repeating_key,
    )

    print("Alphabet keyword:", alphabet_keyword)
    print("Keyed alphabet:", keyed_alphabet)
    print("Repeating key:", repeating_key)
    print("Plaintext:", plaintext)
    print("Standard Vigenere:", standard_ciphertext)
    print("Keyed-alphabet Vigenere:", keyed_ciphertext)
    print(
        "Keyed decrypt:",
        keyed_vigenere_decrypt(
            keyed_ciphertext,
            repeating_key,
            keyed_alphabet,
        ),
    )
    print(
        "Wrong standard decrypt:",
        standard_vigenere_decrypt(
            keyed_ciphertext,
            repeating_key,
        ),
    )

Its exact output is:

Alphabet keyword: CIPHER
Keyed alphabet: CIPHERABDFGJKLMNOQSTUVWXYZ
Repeating key: ORBIT
Plaintext: ATTACKATDAWN
Standard Vigenere: OKUIVYRULTKE
Keyed-alphabet Vigenere: WYCBTPJCFZKU
Keyed decrypt: ATTACKATDAWN
Wrong standard decrypt: IHBTABSBXGWD

That gives four independent checks:

standard encryption:
OKUIVYRULTKE

keyed-alphabet encryption:
WYCBTPJCFZKU

correct keyed decryption:
ATTACKATDAWN

incorrect standard-alphabet decryption:
IHBTABSBXGWD

The result is therefore not based on a hand-written tableau alone.

It is executable and reversible.

How to test a keyed-alphabet implementation

A useful test suite should include more than one round trip.

Alphabet construction test

Verify:

CIPHER
→
CIPHERABDFGJKLMNOQSTUVWXYZ

and confirm that:

length = 26
all symbols are unique
set of symbols = A–Z

Duplicate-removal test

Verify:

BALLOON
→
BALONCDEFGHIJKMPQRSTUVWXYZ

This catches an implementation that accidentally leaves repeated keyword letters in the alphabet.

Known-answer encryption test

Verify:

plaintext:        ATTACKATDAWN
alphabet keyword: CIPHER
repeating key:    ORBIT
ciphertext:       WYCBTPJCFZKU

Known-answer decryption test

Verify the inverse:

WYCBTPJCFZKU
→
ATTACKATDAWN

under the same alphabet and key.

Standard-alphabet reduction test

If the keyed alphabet is:

ABCDEFGHIJKLMNOPQRSTUVWXYZ

the implementation should reduce exactly to ordinary Vigenère.

For the same plaintext and key:

ATTACKATDAWN + ORBIT
→
OKUIVYRULTKE

This proves that the keyed routine is a generalization of the chosen standard convention rather than an unrelated transformation.

Interoperability requires the convention, not just the alphabet

Two programs can both display:

CIPHERABDFGJKLMNOQSTUVWXYZ

and still disagree if they interpret the repeating key differently.

For example, one may assign key-letter values using the mixed alphabet.

Another may continue to interpret key letters using standard:

A = 0 ... Z = 25

while using the mixed alphabet only for output rows.

Those are different ciphers.

So a complete format specification should say something like:

Alphabet:
CIPHERABDFGJKLMNOQSTUVWXYZ

Plaintext indexing:
positions in the keyed alphabet

Repeating-key indexing:
positions in the keyed alphabet

Encryption:
C = A*[(P + K) mod 26]

Decryption:
P = A*[(C - K) mod 26]

Key advancement:
one position per encrypted letter

That is enough for an independent implementation to reproduce the example.

Keyed alphabet and repeating key solve different design problems

The alphabet keyword changes:

which visible letter occupies each numeric position

The repeating key changes:

which modular offset is applied at each message position

Using a strong-looking alphabet keyword does not stop the repeating key from repeating.

Using a long repeating key does not hide the alphabet construction if the mixed alphabet is published.

They are independent parameters.

This separation is important when analyzing or documenting a ciphertext.

Do not confuse a changed alphabet with a changed plaintext language

A keyed alphabet can make:

E

occupy numeric position 4 in one construction and another position in another construction.

That does not change the fact that the underlying plaintext may be English, Spanish, French, or German.

Language statistics still belong to the plaintext.

The alphabet is a symbol mapping layered on top of it.

When the mapping is known, language frequencies can be translated through the mapping.

When the mapping is secret, the analyst has to recover or work around that permutation.

The non-English Vigenère benchmark addresses a different source of model mismatch: the plaintext language itself.

The two problems should not be conflated.

What a keyed alphabet does—and does not—change

A keyed alphabet changes:

letter-to-number assignments
visible tableau order
ciphertext produced by a given visible key
the substitutions seen in each repeating column
interoperability requirements
post-period recovery when the alphabet is secret

It does not automatically change:

the 26-symbol alphabet size
the modulus
the fact that the repeating key is periodic
the need for consistent normalization
the requirement that encryption and decryption use the same convention

Those distinctions make the variant easier to reason about.

The practical takeaway

A keyed-alphabet Vigenère cipher is not created by decorating a standard Vigenère result with a scrambled alphabet after the fact.

The alphabet ordering is part of the arithmetic.

For the convention used here:

Alphabet keyword:
CIPHER

Keyed alphabet:
CIPHERABDFGJKLMNOQSTUVWXYZ

Repeating key:
ORBIT

Plaintext:
ATTACKATDAWN

Ciphertext:
WYCBTPJCFZKU

Ordinary Vigenère with the same visible plaintext and repeating key gives:

OKUIVYRULTKE

and an ordinary decoder applied to the keyed ciphertext gives:

IHBTABSBXGWD

The lesson is simple:

once the alphabet order changes, every symbol’s numeric meaning changes with it. A decoder must know not only the repeating key, but also the exact alphabet and the convention used to interpret that alphabet.

---

Written by Marcus Hale

Editorially reviewed by VigenereDecoder.com editorial team