Vigenère With Custom Alphabets: Generalizing Modulo 26 to Modulo q

The number 26 is not built into the mathematical idea of Vigenère.

It comes from one particular alphabet:

ABCDEFGHIJKLMNOPQRSTUVWXYZ

Once those 26 symbols are mapped to:

0, 1, 2, ..., 25

standard Vigenère becomes addition modulo 26.

Change the symbol set, and the modulus changes with it.

For an ordered alphabet containing q unique symbols:

A = [a₀, a₁, ..., a(q-1)]

map each symbol to its position:

a₀ → 0
a₁ → 1
...
a(q-1) → q - 1

Then repeating-key Vigenère becomes:

Cᵢ = (Pᵢ + Kᵢ) mod q

with decryption:

Pᵢ = (Cᵢ - Kᵢ) mod q

Standard A–Z Vigenère is simply the special case:

q = 26

This article verifies the same construction with:

q = 36
alphabet = ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789

and with a digit-only alphabet:

q = 10
alphabet = 0123456789

The important change is not cosmetic. A larger alphabet changes symbol values, wraparound, the Vigenère square, the keyspace for a fixed key length, the random-coincidence baseline, and the statistical model needed for cryptanalysis.

If you only need standard A–Z transformation, use the Vigenère Cipher Decoder and Encoder. The purpose here is to define what remains valid when the cipher alphabet itself no longer contains exactly 26 symbols.

Standard Vigenère is addition in a 26-symbol cycle

For ordinary A–Z Vigenère:

A = 0
B = 1
...
Z = 25

With plaintext value P and key value K:

C = (P + K) mod 26

Decryption reverses the addition:

P = (C - K) mod 26

The repeating key determines which shift is applied at each position.

The number 26 enters only because there are 26 symbols in the chosen alphabet.

Mathematically, the same additive construction can be performed in:

Z_q

for another alphabet size q.

Published mathematical descriptions of Vigenère commonly formalize the standard cipher over Z_26; the generalization is obtained by replacing the 26-element symbol set with an alphabet of size q while keeping the same addition-and-subtraction structure.

The generalized definition

Let the ordered cipher alphabet be:

A = [a₀, a₁, ..., a(q-1)]

with exactly:

q

distinct symbols.

Define:

index(aⱼ) = j

for:

j = 0, 1, ..., q - 1

If the repeating key has length m, then its symbol at message position i is:

K[i mod m]

Encryption is:

cᵢ
=
(index(Pᵢ) + index(K[i mod m])) mod q

and the ciphertext symbol is:

Cᵢ = A[cᵢ]

Decryption is:

pᵢ
=
(index(Cᵢ) - index(K[i mod m])) mod q

followed by:

Pᵢ = A[pᵢ]

Nothing in those equations requires:

q = 26

The requirements are instead that the alphabet is ordered, each symbol has one unique index, and both sides use the same specification.

q does not need to be prime

This is an important difference from some other modular ciphers.

The generalized Vigenère construction uses:

addition
and
subtraction

modulo q.

Every residue modulo q has an additive inverse for any integer:

q ≥ 2

So:

q = 10
q = 26
q = 30
q = 36
q = 96

can all support the additive Vigenère rule.

There is no requirement that q be prime.

There is also no requirement that a key shift be coprime to q.

That type of restriction appears when a cipher needs a multiplicative inverse, as an affine cipher does.

For additive Vigenère:

decrypt(encrypt(P, K), K)
=
(P + K - K) mod q
=
P mod q

for every valid symbol value.

The 36-symbol example below therefore works even though:

36

is composite.

A 36-symbol alphabet

Use:

ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789

The first 26 positions remain familiar:

A = 0
...
Z = 25

The digits continue the sequence:

The modulus is now:

q = 36

The cipher cycles through all 36 positions before wrapping.

Wraparound changes as soon as q changes

Consider:

plaintext symbol: Z
key symbol:       B

Their values are:

Z = 25
B = 1

Under standard A–Z Vigenère:

(25 + 1) mod 26
=
0
→ A

Under the 36-symbol alphabet:

(25 + 1) mod 36
=
26
→ 0

So:

Z + B

encrypts to:

A

under modulo 26, but to:

0

under modulo 36.

The plaintext and key symbols are visually identical.

The alphabet specification changes the result.

Complete q=36 example

Use:

Alphabet:
ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789

Plaintext:
MEETATGATE7PM

Repeating key:
R2D2

The key values are:

R = 17
2 = 28
D = 3
2 = 28

The key repeats as:

R2D2R2D2R2D2R

The complete calculation is:

The verified ciphertext is:

36HLRLJ2A6AH3

Decrypting it with the same key and alphabet returns:

MEETATGATE7PM

exactly.

Digits can appear even when the plaintext position contained a letter

The first character illustrates this immediately.

Plaintext:

M = 12

Key:

R = 17

Add:

12 + 17 = 29

Position 29 in the 36-symbol alphabet is:

3

So:

M + R → 3

This is not a formatting trick.

The digit is a normal ciphertext symbol because digits are members of the cipher alphabet.

Later positions can move in the opposite direction. A plaintext digit combined with a key symbol can produce a letter.

For example, position 11 uses:

plaintext 7 = 33
key D      = 3

Therefore:

(33 + 3) mod 36
=
0
→ A

The symbol categories “letter” and “digit” have no special mathematical status once all 36 symbols belong to one ordered cycle.

A digit-only Vigenère cipher is the same construction with q=10

Take:

Alphabet:
0123456789

Plaintext:
20261018

Key:
314

Now:

q = 10

and the repeating key is:

31431431

The verified ciphertext is:

51692449

Decryption modulo 10 returns:

20261018

The arithmetic is identical to the A–Z case.

Only the symbol mapping and modulus changed.

This example makes the abstraction easier to see:

Vigenère is not fundamentally a letter table.
It is repeated modular addition over an agreed ordered symbol set.

The standard cipher falls out as q=26

A generalized implementation should reduce exactly to ordinary Vigenère when passed:

ABCDEFGHIJKLMNOPQRSTUVWXYZ

The verified code in this article produces:

ATTACKATDAWN
+
LEMON
→
LXFOPVEFRNHR

with:

q = 26

That reduction test matters.

If a supposed generalized implementation cannot reproduce established A–Z vectors when the alphabet is standard, then it is not merely “custom Vigenère”; it has changed some other rule as well.

The site’s verified test-vector reference is useful for checking that standard reduction.

This is different from a keyed alphabet

A custom alphabet and a keyed alphabet can look similar in code because both require an explicit symbol sequence.

They answer different questions.

A keyed alphabet such as:

CIPHERABDFGJKLMNOQSTUVWXYZ

contains the same 26 A–Z symbols in a different order.

Therefore:

q remains 26

Only the mapping changes.

The keyed-alphabet Vigenère guide examines that case.

A larger custom alphabet such as:

ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789

contains 36 symbols.

Therefore:

q changes from 26 to 36

Both the mapping and the size of the modular space are different.

The distinction is:

Alphabet order is still part of the specification

Even when the focus is alphabet size, ordering cannot be ignored.

These two alphabets both have:

q = 36

but are not equivalent as visible encodings:

ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789

0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ

In the first:

A = 0
0 = 26

In the second:

0 = 0
A = 10

The same plaintext and visible key can therefore produce different ciphertext.

A complete custom-alphabet specification needs the exact ordered symbol sequence, not merely:

“36 characters”

The plaintext and key must belong to the same algebraic alphabet in this profile

The direct construction used here requires every plaintext symbol and every key symbol to exist in the declared alphabet.

For the 36-symbol profile:

R2D2

is valid because:

R
2
D

all exist in the alphabet.

A key containing:

-

would be invalid unless - were also added to the alphabet.

The same rule applies to plaintext.

This keeps encryption and decryption unambiguous:

every symbol
→ exactly one index
→ exactly one modular value

A user-facing program can choose to preserve unsupported formatting outside the cipher stream, as the Python implementation guide does for one A–Z profile.

But that is an additional text-processing rule, not part of the generalized modular equation itself.

An alphabet must not contain duplicate symbols

Suppose someone defines:

AABC

as a four-position alphabet.

What is the index of:

A

?

It could be:

0

or:

1

The mapping is no longer a function unless another disambiguation rule is invented.

For direct Vigenère indexing, alphabet symbols therefore need to be unique.

The implementation in this article rejects duplicate alphabets rather than guessing.

The Vigenère square becomes q by q

For standard A–Z Vigenère, the familiar tabula recta has:

26 rows
×
26 columns

For the 36-symbol example, the analogous table has:

36 × 36

entries.

Row 0 begins:

ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789

Row 1 is a one-position rotation:

BCDEFGHIJKLMNOPQRSTUVWXYZ0123456789A

and the cycle continues through all 36 shifts.

Nothing conceptually new is required.

The table simply reflects addition modulo 36 instead of modulo 26.

For software, numeric indexing is usually clearer than materializing the full table.

The random coincidence baseline changes with q

For two independent symbols drawn uniformly from an alphabet of size q, the probability of equality is:

1 / q

So the uniform random baseline is:

q = 26:
1/26 ≈ 0.03846

q = 36:
1/36 ≈ 0.02778

That does not mean a real 36-symbol plaintext will have IC near 0.02778.

Real text is not uniformly distributed.

If digits are rare, or some symbols occur much more often than others, the plaintext coincidence rate can be substantially higher.

The important point is that cryptanalytic thresholds calibrated for a 26-symbol model should not be copied blindly into a different alphabet.

The alphabet distribution has changed.

Frequency models must include the new symbols

A chi-squared attack on A–Z usually compares 26 observed categories with 26 expected language frequencies.

For:

A–Z0–9

the model now has:

36

categories.

What should the expected probabilities of:

0
1
2
...
9

be?

There is no universal answer.

It depends on the plaintext domain.

A dataset of product codes may contain many digits.

Ordinary prose may contain relatively few.

A timestamp field may be almost entirely numeric.

So simply extending an English 26-letter frequency table with ten arbitrary small probabilities creates a new modeling assumption.

The non-English Vigenère benchmark showed how a mismatched language model can alter key recovery even when the alphabet size stays fixed. A custom alphabet creates the same general requirement at another level: the statistical model must match the symbol system and data domain actually being encrypted.

Kasiski-style spacing does not depend on q in the same way

Repeated-sequence spacing starts from literal repetitions in the ciphertext.

If the same ciphertext substring appears twice, the distance between occurrences is an integer number of positions regardless of whether:

q = 26

or:

q = 36

So the basic Kasiski idea of looking for common divisors among repeat spacings still applies to a repeating-key construction.

What changes is the probability of accidental repetitions and the symbol statistics underneath the ciphertext.

A larger or differently distributed alphabet can therefore change how much repeat evidence appears even though the distance arithmetic itself is unchanged.

A larger alphabet increases the written-key space—but does not remove repetition

If the key has fixed length m and every alphabet symbol is allowed at every key position, the number of possible written keys is:

q^m

For a four-symbol key:

q = 26:
26^4 = 456,976

q = 36:
36^4 = 1,679,616

The 36-symbol written-key space is about:

3.68 times

larger for the same key length.

That sounds useful, but it should not be confused with modern security.

If the four-symbol key repeats:

K₁ K₂ K₃ K₄ K₁ K₂ K₃ K₄ ...

the periodic structure remains.

The classical weakness of repeating-key Vigenère is not merely that each key symbol has only 26 choices.

It is that the same finite key positions are reused across the message.

Increasing q does not remove that reuse.

Composite q does not create a decryption problem here

Because the cipher uses addition, every key symbol produces a permutation:

x → x + k mod q

of the q possible values.

For q = 36, shift 18 maps every symbol halfway around the cycle.

Shift 12 moves every symbol by a third of the cycle.

Shift 6 moves by a sixth.

Even though those shifts share factors with 36, each mapping is still one-to-one.

Its inverse is simply:

x → x - k mod 36

The requirement:

gcd(k, q) = 1

would matter if the cipher multiplied by k.

It does not apply to plain additive Vigenère.

The exact Python implementation

The following is the complete script used to generate the verified examples.

import string

def validate_alphabet(alphabet):
    if len(alphabet) < 2:
        raise ValueError("Alphabet must contain at least two symbols.")
    if len(set(alphabet)) != len(alphabet):
        raise ValueError("Alphabet symbols must be unique.")
    return alphabet


def _validate_text(text, alphabet, label):
    missing = [ch for ch in text if ch not in alphabet]
    if missing:
        raise ValueError(
            f"{label} contains symbols outside the alphabet: {missing!r}"
        )


def vigenere_encrypt(text, key, alphabet):
    """
    Generalized additive Vigenere over an ordered alphabet of size q.

    C_i = A[(index(P_i) + index(K_i)) mod q]
    """
    alphabet = validate_alphabet(alphabet)

    if not key:
        raise ValueError("Key must not be empty.")

    _validate_text(text, alphabet, "Text")
    _validate_text(key, alphabet, "Key")

    index = {symbol: i for i, symbol in enumerate(alphabet)}
    q = len(alphabet)

    return "".join(
        alphabet[
            (
                index[ch]
                + index[key[i % len(key)]]
            ) % q
        ]
        for i, ch in enumerate(text)
    )


def vigenere_decrypt(text, key, alphabet):
    """
    Inverse of vigenere_encrypt.

    P_i = A[(index(C_i) - index(K_i)) mod q]
    """
    alphabet = validate_alphabet(alphabet)

    if not key:
        raise ValueError("Key must not be empty.")

    _validate_text(text, alphabet, "Text")
    _validate_text(key, alphabet, "Key")

    index = {symbol: i for i, symbol in enumerate(alphabet)}
    q = len(alphabet)

    return "".join(
        alphabet[
            (
                index[ch]
                - index[key[i % len(key)]]
            ) % q
        ]
        for i, ch in enumerate(text)
    )


if __name__ == "__main__":
    alphabet36 = string.ascii_uppercase + string.digits
    plaintext36 = "MEETATGATE7PM"
    key36 = "R2D2"

    ciphertext36 = vigenere_encrypt(
        plaintext36,
        key36,
        alphabet36,
    )

    print("36-symbol alphabet:", alphabet36)
    print("q:", len(alphabet36))
    print("Plaintext:", plaintext36)
    print("Key:", key36)
    print("Ciphertext:", ciphertext36)
    print(
        "Decrypted:",
        vigenere_decrypt(ciphertext36, key36, alphabet36),
    )

    print()
    print("Wraparound check, q=36:")
    print("Z + B ->", vigenere_encrypt("Z", "B", alphabet36))

    alphabet10 = string.digits
    plaintext10 = "20261018"
    key10 = "314"
    ciphertext10 = vigenere_encrypt(
        plaintext10,
        key10,
        alphabet10,
    )

    print()
    print("Digit-only alphabet:", alphabet10)
    print("q:", len(alphabet10))
    print("Plaintext:", plaintext10)
    print("Key:", key10)
    print("Ciphertext:", ciphertext10)
    print(
        "Decrypted:",
        vigenere_decrypt(ciphertext10, key10, alphabet10),
    )

    alphabet26 = string.ascii_uppercase
    print()
    print("Standard reduction:")
    print(
        vigenere_encrypt(
            "ATTACKATDAWN",
            "LEMON",
            alphabet26,
        )
    )

Its exact output is:

36-symbol alphabet: ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789
q: 36
Plaintext: MEETATGATE7PM
Key: R2D2
Ciphertext: 36HLRLJ2A6AH3
Decrypted: MEETATGATE7PM

Wraparound check, q=36:
Z + B -> 0

Digit-only alphabet: 0123456789
q: 10
Plaintext: 20261018
Key: 314
Ciphertext: 51692449
Decrypted: 20261018

Standard reduction:
LXFOPVEFRNHR

The implementation takes the alphabet itself as an argument.

There is no hard-coded:

26

inside the transformation.

Instead:

q = len(alphabet)

determines the modulus.

That is the central implementation requirement for a genuinely generalized version.

What the code assumes about a “symbol”

The Python implementation treats each individual Python character as one alphabet symbol.

That works cleanly for:

A
7
?
é

when each is represented as one code point in the supplied string.

It does not automatically mean that every visible grapheme is one symbol.

Some visible Unicode characters can be represented by multiple code points, such as a base character plus a combining mark.

If a custom alphabet is intended to work with complex Unicode grapheme clusters, the program needs an explicit tokenization layer rather than ordinary character iteration.

That is a software-modeling issue, not a change to the modulo-q mathematics.

The mathematical requirement remains:

q distinct symbols
with one stable index each

What has to be shared for interoperability

A custom-alphabet ciphertext cannot be reproduced from the repeating key alone.

The recipient needs the exact ordered alphabet and the key.

They also need the same text-handling policy if some input characters are preserved rather than encrypted.

For the worked 36-symbol example, the complete core specification is:

Alphabet:
ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789

q:
36

Key:
R2D2

Encryption:
Cᵢ = (Pᵢ + Kᵢ) mod 36

Key advancement:
one key symbol per encrypted symbol

Plaintext:
MEETATGATE7PM

Ciphertext:
36HLRLJ2A6AH3

That is enough for an independent strict-profile implementation to reproduce the vector.

A useful validation sequence

Before trusting a generalized implementation, verify these cases in order:

1. Confirm that the alphabet contains at least two unique symbols and no duplicates.
2. Reproduce a standard A–Z known-answer vector with q = 26.
3. Verify a non-26 vector such as MEETATGATE7PM + R2D2 → 36HLRLJ2A6AH3.
4. Verify wraparound across a new symbol boundary, such as Z + B → 0 under the 36-symbol alphabet.
5. Verify a second alphabet size, such as the digit-only q = 10 example.
6. Check decrypt(encrypt(P, K), K) = P for every supported input profile.
7. Reject duplicate alphabets, empty keys, and symbols that are outside the declared alphabet.

A generalized cipher should pass the standard case and the new cases.

Otherwise the code may have changed more than just the modulus.

The practical takeaway

Vigenère is not inherently a modulo-26 cipher.

Modulo 26 is the consequence of choosing the 26-letter English alphabet.

For any ordered alphabet of q unique symbols, the same additive construction becomes:

Cᵢ = (Pᵢ + Kᵢ) mod q
Pᵢ = (Cᵢ - Kᵢ) mod q

With:

ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789

the modulus is:

36

and:

MEETATGATE7PM
+
R2D2

produces:

36HLRLJ2A6AH3

The larger alphabet changes the cycle, symbol values, wraparound, keyspace size, table dimensions, and cryptanalytic baselines.

What it does not change is the central weakness of repeating-key Vigenère:

the same finite key still returns to the same positions again and again.

Generalizing the alphabet changes the arithmetic space.

It does not turn a classical repeating-key cipher into a modern secure encryption scheme.

---

Written by Marcus Hale

Editorially reviewed by VigenereDecoder.com editorial team