A romanized word may look correct on screen while being stored incorrectly underneath.

The character é, for example, can be encoded as one precomposed Unicode character or as the letter e followed by a combining acute accent. These forms may render identically, but software that compares their raw code-unit sequences can treat them as different strings.

The same problem affects scholarly and technical romanization containing characters such as:

• ā
• č
• ḥ
• ñ
• ṛ
• ṣ
• š
• ṭ
• ū
• ž
• ʿ
• ʾ

A database may store visually identical transliterations in different normalization forms. A search engine may fail to match marked and unmarked variants. A text field may count a letter and its diacritics as several code points. An accent-removal function may accidentally delete Arabic vowel marks, Indic vowel signs, or other characters essential to the original writing system.

Reliable transliteration software therefore requires more than a character-conversion table. It must understand:

• Unicode code points
• Combining characters
• Canonical equivalence
• Normalization forms
• Grapheme clusters
• Transliteration standards
• Diacritic preservation
• Search normalization
• Reversibility
• Security and mixed-script text

This guide explains how these concepts fit together and how to build safer romanization and transliteration tools.

Quick Answer

Unicode transliteration converts text between scripts, while Unicode normalization standardizes how canonically or compatibly equivalent text is encoded.

They solve different problems:

Process	Primary purpose
Transliteration	Convert text from one script or writing convention to another
Normalization	Standardize equivalent Unicode character sequences
Diacritic folding	Create a simplified form with selected marks removed
Case folding	Create a case-insensitive comparison form
Collation	Sort or compare text according to language-sensitive rules
Segmentation	Identify user-perceived characters, words or sentences

A reliable transliteration pipeline commonly follows this pattern:

1. Preserve the original input.
2. Identify the source language and script.
3. Normalize the input.
4. Apply a named transliteration transform.
5. Normalize the output again.
6. Store the full marked result.
7. Generate simplified search and ASCII forms separately.
8. Test round-trip behavior where reversibility is claimed.

Unicode Standard Annex #15 defines four normalization forms—NFC, NFD, NFKC and NFKD—and explains how normalized strings provide consistent binary representations for equivalent text. The current annex is aligned with Unicode 17.0.0.

Transliteration Is Not Normalization

The terms are sometimes confused because both processes transform text.

Transliteration

Transliteration converts text from one writing system into another.

Examples include:

• Москва → Moskva
• 東京 → Tōkyō
• 한국 → Hanguk
• العربية → al-ʿArabīyah
• संस्कृत → saṃskṛta

Unicode CLDR defines transliteration as converting characters from one script to another without translating the underlying words. Transliteration may be designed around a language, a script, or a named institutional standard.

Normalization

Normalization does not normally convert one script into another.

Instead, it places equivalent Unicode text into a standardized encoded form.

For example, these two representations may look identical:

• Precomposed: é
• Decomposed: e + combining acute accent

Normalization can convert both into a common representation so that storage, comparison and processing become more predictable.

Translation

Translation changes the language and meaning representation.

For example:

• Japanese 東京 → Tōkyō is transliteration.
• Japanese 東京 → eastern capital is translation.

Transliteration, normalization and translation should be implemented as separate operations.

Unicode Code Points Are Not Always User-Perceived Characters

Programmers often use the word character ambiguously.

It may refer to:

• A Unicode code point
• A code unit in UTF-8, UTF-16 or UTF-32
• A combining character sequence
• A grapheme cluster
• A user-perceived letter or symbol
• A byte

These are not equivalent.

Base Characters and Combining Marks

Unicode can represent a letter with diacritics as a sequence consisting of:

1. A base character
2. One or more combining marks

For example:

• a + combining macron → ā
• s + combining dot below → ṣ
• r + combining ring below → r̥
• r + ring below + macron → r̥̄

The Unicode Standard stores combining marks after the base character on which they depend, even when the mark is displayed above, below or around it.

Some combinations also have precomposed Unicode characters. Others can only be represented with combining sequences.

For example:

Display	Possible encoding model
ā	Precomposed character or a plus macron
ṛ	Precomposed character or r plus dot below
r̥	Usually r plus combining ring below
r̥̄	r plus ring below plus macron

Software must not assume that one visible letter equals one Unicode code point.

Grapheme Clusters

A grapheme cluster is a sequence that approximates one user-perceived character.

It may contain:

• A base letter
• Combining accents
• Vowel signs
• Join controls
• Hangul jamo
• Emoji components
• Other dependent characters

Unicode recommends extended grapheme clusters for text editing and user-facing operations. These clusters include combining marks and spacing vowel signs used in scripts such as Devanagari and Tamil. Their boundaries remain stable across canonically equivalent NFC and NFD representations.

This has practical consequences.

A text editor should not place a cursor between a Latin base letter and its macron as though they were unrelated visible characters. A truncation function should not cut a name after the base letter while leaving its combining mark behind. A character counter intended for users should normally count grapheme clusters, not UTF-8 bytes or UTF-16 code units.

Incorrect Assumption

visible characters = string length

This fails in many programming environments.

A visually single transliterated letter may contain:

• One code point
• Two code points
• Three or more code points
• Several UTF-8 bytes
• More than one UTF-16 code unit

Use a Unicode grapheme-segmentation library for cursor movement, truncation, selection, deletion and user-visible character counts.

Canonical Equivalence

Two Unicode strings are canonically equivalent when they represent the same abstract text but use different encoded sequences.

The most familiar example is:

U+00E9              é
U+0065 U+0301       e + combining acute accent

These strings may render identically but fail a raw binary comparison.

Unicode normalization exists partly to make canonically equivalent sequences predictable for processing. When text is converted into a selected normalization form, equivalent strings receive the same normalized representation.

Canonical equivalence matters for transliteration because diacritic-rich output can be produced by:

• A keyboard
• Copy and paste
• A database import
• An ICU transform
• A browser form
• A word processor
• A precomposed-character lookup table
• A combining-mark rule

Two converters may therefore generate visually identical text with different underlying sequences.

The Four Unicode Normalization Forms

Unicode defines four standard normalization forms.

NFC: Normalization Form C

NFC performs canonical decomposition followed by canonical composition.

It generally favors precomposed characters where Unicode defines them.

Examples:

e + ◌́  → é
a + ◌̄  → ā
s + ◌̣  → ṣ

NFC is commonly a sensible default for:

• Stored display text
• User input
• Published romanization
• Database values
• API responses
• HTML content

It produces compact, conventional forms where precomposed characters exist.

NFC does not guarantee that every visible diacritic becomes part of one code point. Forms such as r̥̄ may still require combining marks.

NFD: Normalization Form D

NFD performs canonical decomposition.

It separates precomposed characters into their base characters and combining marks where canonical decompositions exist.

Examples:

é → e + ◌́
ā → a + ◌̄
ṣ → s + ◌̣

NFD is especially useful internally when software needs to:

• Inspect diacritics
• Reorder combining marks canonically
• Apply mark-sensitive transliteration rules
• Remove selected accents
• Generate comparison keys

ICU recommends designing transform rules so that they operate consistently regardless of the input’s starting normalization. Its rule tutorial commonly places an NFD stage at the beginning and NFC at the end when accent manipulation is involved.

NFKC: Normalization Form KC

NFKC applies compatibility decomposition followed by canonical composition.

Compatibility normalization can merge characters that Unicode treats as stylistic, positional or compatibility variants.

Potential examples include transformations involving:

• Full-width and half-width forms
• Superscripts
• Circled characters
• Presentation variants
• Some ligatures
• Compatibility symbols

NFKC is useful for certain identifiers, matching systems and restricted technical keys. It should be used cautiously for authoritative linguistic text because compatibility normalization can remove distinctions that a publisher, cataloger or scholar intended to preserve. Unicode provides separate guidance for using normalization in identifiers.

NFKD: Normalization Form KD

NFKD applies compatibility decomposition without recomposition.

It is sometimes useful as an intermediate step in aggressive search folding or identifier preparation. It is usually too destructive to use as the sole stored form of scholarly romanization or original-script text.

Comparison

Form	Canonical decomposition	Compatibility decomposition	Recomposition
NFC	Yes	No	Yes
NFD	Yes	No	No
NFKC	Yes	Yes	Yes
NFKD	Yes	Yes	No

For most romanization display fields, NFC is the safest general default. NFD is valuable inside a transform pipeline. NFKC and NFKD should be reserved for clearly defined technical purposes.

Normalization Does Not Remove Diacritics

A common misconception is that Unicode normalization simplifies accented text into plain ASCII.

It does not.

NFD may separate ā into a plus a combining macron, but the macron remains present. NFC may combine them again.

To remove the mark, software must perform an additional filtering step.

ICU gives the following conceptual transform for removing accents:

NFD;
[:Nonspacing Mark:] Remove;
NFC;

This sequence decomposes characters, removes nonspacing marks and recomposes the remaining text.

However, this technique is dangerous when applied indiscriminately.

Why Removing All Combining Marks Is Unsafe

Combining marks are not always optional accents.

They can encode essential language information.

Examples include:

• Arabic vowel and pronunciation marks
• Hebrew vowel points
• Devanagari and other Indic vowel signs
• Thai and Lao marks
• Vietnamese tone marks
• Vedic accents
• Quranic annotations
• Musical and mathematical marks
• Diacritics distinguishing separate letters in scholarly romanization

Unicode’s grapheme-cluster model includes Indic vowel signs and other spacing combining marks as parts of the same user-perceived unit as their base characters. Removing marks globally can therefore damage words rather than merely simplifying typography.

Consider:

Full form	Aggressively stripped form	Information lost
Kṛṣṇa	Krsna	Vocalic ṛ and retroflex ṣ
Tōkyō	Tokyo	Vowel length
Ḥasan	Hasan	Distinct Arabic consonant
Zhōngguó	Zhongguo	Mandarin tones
Čajkovskij	Cajkovskij	Cyrillic Ч distinction under the selected system
Sanskrit vowel signs	Consonant skeleton	Vowel identity

The stripped form can be useful as a search alias. It should not replace the authoritative value.

Better Strategy

Create separate fields:

Field	Example
Original script	कृष्ण
Standard transliteration	Kṛṣṇa
Diacritic-insensitive search key	krsna
Common spelling	Krishna
URL slug	krishna

Each representation has a defined purpose.

Diacritics Are Linguistic Data

In romanization systems, diacritics may mark:

• Vowel length
• Tone
• Stress
• Retroflexion
• Emphasis
• Pharyngeal articulation
• Palatalization
• Separate source characters
• Historical distinctions
• Nasalization

For example:

Pair	Possible distinction
a / ā	Short and long vowel
s / ṣ	Different source consonants
h / ḥ	Different Arabic letters
t / ṭ	Dental versus retroflex or plain versus emphatic
z / ž	Different Slavic or Cyrillic-source values
r / ṛ	Consonantal versus vocalic or retroflex value
o / ō	Short versus long vowel

Removing a diacritic can make the transliteration irreversible.

A software interface should never describe a diacritic-free output as identical to the complete standard form. It is a simplified derivative.

Precomposed and Decomposed Diacritics

Unicode includes precomposed characters for many common Latin combinations:

• ā
• é
• ñ
• š
• ṭ
• ž

Other combinations require combining marks:

• r̥
• l̥̄
• m̐
• letters with several stacked diacritics

Software should support both.

Storage Recommendation

Normalize display and authoritative transliteration fields to NFC.

This produces precomposed forms where available while retaining combining sequences where necessary.

Processing Recommendation

Convert to NFD temporarily when:

• Inspecting marks
• Reordering marks
• Removing selected marks
• Applying decomposition-based rules

Normalize the final output back to NFC.

Do Not Restrict Input to Precomposed Characters

A validation rule that accepts ā but rejects a plus combining macron is not Unicode-safe. Both can represent canonically equivalent text.

Normalize first, then validate the normalized form.

Canonical Ordering of Combining Marks

One base letter can carry multiple combining marks.

Unicode assigns combining classes that allow marks to be placed into a canonical order during normalization.

For example, a letter might contain:

• A dot below
• A macron above
• A tone mark
• Another scholarly sign

Different input methods can produce the marks in different code-point orders. Normalization places canonically reorderable marks into a consistent order.

This is important for transform rules.

A rule expecting:

base + dot below + macron

may fail when the input arrives as:

base + macron + dot below

NFD normalizes the combining-mark order before the rule is applied.

Unicode CLDR Transforms

The Unicode Common Locale Data Repository contains transform data for script and language conversion.

CLDR transform definitions can specify:

• Source script or language
• Target script or language
• Variant
• Direction
• Aliases
• Visibility
• Rule sequences

Examples of transform variants can identify standards or authorities such as:

• BGN
• UNGEGN
• ISO 9
• U.S. Library of Congress
• UK PCGN
• National government systems

The LDML transform format records source, target, variant and direction, allowing software to distinguish a generic script transform from a named institutional convention.

This matters because:

Cyrillic-Latin

does not necessarily produce the same output as:

Russian-Latin/BGN
Russian-Latin/ISO9
Russian-Latin/USLC

A trustworthy interface should show the exact transform identity.

ICU Transliteration

ICU—International Components for Unicode—provides programmable transforms based on Unicode and CLDR data.

ICU transforms can perform:

• Script transliteration
• Language-specific transliteration
• Normalization
• Case conversion
• Character removal
• Width conversion
• Custom rule-based transformations

Transforms can be chained into pipelines, and UnicodeSet filters can restrict which characters are affected.

Example Transform Chain

A search-index pipeline might conceptually use:

Any-Latin;
NFD;
[:Nonspacing Mark:] Remove;
Lower;
NFC;

This may produce a broad Latin, diacritic-free, lowercase search key.

It should not be used as the authoritative romanization because:

• Any-Latin may choose a general transform rather than a required named standard.
• Diacritic removal destroys distinctions.
• Several original strings may collapse to one key.
• Language-specific readings may require dictionary analysis.
• The transform may not preserve identity or official spellings.

Transform Filters

A UnicodeSet filter can restrict a transform to selected scripts or characters.

This is useful when a string contains:

• Original script
• Latin text
• Punctuation
• Numbers
• Product codes
• Mixed-language content

Without a filter, a transform may alter text that should remain untouched.

For example, a Japanese-to-Latin transform should not necessarily retransliterate Latin brand names already embedded in Japanese text.

Script Transforms vs Language Transforms

A script-level transform answers:

How should characters in this writing system be mapped?

A language-level transform answers:

How should this language be represented in another script?

The difference is crucial.

Script-Level Example

A generic Cyrillic-to-Latin transform may map characters consistently without knowing whether the text is:

• Russian
• Ukrainian
• Bulgarian
• Serbian
• Mongolian

Language-Level Example

A Ukrainian transform can apply Ukrainian mappings such as:

• Г → h
• Ґ → g

A Russian transform would treat shared letters differently.

Likewise, converting Han characters requires knowing whether the intended reading is Mandarin, Japanese, Korean or another language.

CLDR distinguishes script-to-script transforms from language-specific and variant transforms. Language-to-language transforms are often transcription-oriented and are not necessarily lossless.

Reversibility

A transliteration is reversible when the original source can be reconstructed from the result.

Unicode CLDR’s design goals prioritize lossless conversion to Latin where practical, while acknowledging that language-specific transcription and many script transforms cannot be perfectly reversible.

Why Reversibility Fails

Reversibility may be lost when:

• Two source letters share one Latin form.
• Tone marks are removed.
• Long and short vowels are merged.
• Word boundaries are changed.
• Diacritics are stripped.
• Pronunciation replaces spelling.
• The source script omits vowels.
• One Latin form can come from several scripts.
• Compatibility normalization removes distinctions.

For example, a diacritic-free key such as:

siva

might correspond to several marked or source-script forms. It is useful for retrieval, but not for reconstruction.

Round-Trip Testing

Where a transform claims reversibility, test:

source → Latin → source

The final source should be exactly equivalent to the original under the system’s documented rules.

Tests should include:

• Ordinary letters
• Diacritics
• Combining sequences
• Word boundaries
• Punctuation
• Historical characters
• Mixed scripts
• Edge cases
• Canonically equivalent input variants

Recommended Software Architecture

A romanization platform should preserve several layers rather than forcing one string to serve every purpose.

Suggested Data Fields

Field	Purpose
original_text	Exact source text
source_language	Language identifier
source_script	Script identifier
normalized_original	NFC-normalized source
standard_romanization	Fully marked output
transform_id	Exact standard or ICU/CLDR transform
transform_version	Unicode, CLDR or application version
simplified_romanization	Reader-friendly output
ascii_form	Restricted technical fallback
search_key	Folded retrieval form
official_form	Official or preferred spelling
manual_override	Human-reviewed correction
reversible	Declared round-trip status

Preserve User Input

Do not discard the user’s original sequence merely because the system normalizes it.

Store either:

• The original submitted value plus a normalized processing value, or
• The original authoritative text in a normalized form after a clearly documented ingestion policy

For editing systems, preserving the user’s intended orthography and punctuation may be important even when equivalent normalization is used internally.

Input and Storage Pipeline

A practical input pipeline can follow these stages.

1. Validate Encoding

Reject malformed byte sequences before applying language processing.

2. Preserve the Original

Keep the submitted source for audit and review.

3. Normalize to NFC

Use NFC for the main stored processing and display form unless the application has a documented reason to use another normalization.

4. Detect or Request Language and Script

Do not rely on script detection alone.

5. Select a Named Transform

Examples:

• Hanyu Pinyin
• Revised Romanization
• ISO 9
• ALA-LC Arabic
• IAST
• ISO 15919
• Hepburn

6. Apply Transliteration

Use language-sensitive rules, dictionaries or human review where needed.

7. Normalize the Result to NFC

This stabilizes diacritic representation for storage and comparison.

8. Generate Derived Forms

Create separate:

• Diacritic-free key
• ASCII key
• URL slug
• Search aliases
• Alternative-system results

9. Record Version and Provenance

Transform data can evolve. Store enough metadata to reproduce or explain the result later.

Searching Transliterated Text

Search should not depend on one representation.

A user may search with:

• Original script
• Fully marked romanization
• Romanization without diacritics
• Another standard
• Historical spelling
• Common public spelling
• A typing approximation

A search index can maintain several parallel forms.

Example

Type	Form
Original	Чайковский
ISO-style	Čajkovskij
Library form	Chaĭkovskiĭ
Practical form	Chaykovskiy
Conventional English	Tchaikovsky
Folded search key	chaikovskiy

The primary display value should remain separate from the search key.

Accent-Insensitive Search

To support accent-insensitive matching:

1. Normalize the authoritative value.
2. Copy it into a search-only field.
3. Decompose the search copy.
4. Remove an explicitly defined set of marks.
5. Apply case folding where appropriate.
6. Recompose or store in the search engine’s expected form.

Do not mutate the display field.

Normalization Is Not Collation

Normalization answers whether strings have equivalent encoded forms.

Collation determines:

• Sorting order
• Search strength
• Accent sensitivity
• Case sensitivity
• Language-specific ordering

A normalized list is not automatically sorted correctly.

For example, the ordering of:

• a
• ā
• ä
• å
• æ

depends on the selected language and collation rules.

Likewise, a transliteration containing č, š and ž may sort differently under English, Czech, Croatian or a specialized scholarly order.

Use a Unicode-aware collation library rather than byte ordering. Normalization and collation are distinct processes; normalization establishes equivalence but does not define rank order.

Case Folding

Lowercasing and case folding are also different from normalization.

Case folding creates a form suitable for caseless comparison. It may produce results different from ordinary display lowercase rules.

For identifiers or search keys, a pipeline may combine:

• Normalization
• Case folding
• Script filtering
• Transliteration
• Diacritic folding

The order must be documented and tested because changing the sequence can change the result.

Unicode UAX #31 provides guidance for normalization and case handling in identifiers.

Do not apply aggressive case or compatibility folding to authoritative names without keeping the original form.

Regular Expressions and String Slicing

A regular expression that matches only ASCII letters will miss marked Latin characters.

For example:

[A-Za-z]+

does not reliably match:

• Čajkovskij
• Kṛṣṇa
• Muḥammad
• Tōkyō
• Zhōngguó

Use Unicode character properties where supported, such as categories for letters and marks.

A pattern often needs to allow:

• p{L} for letters
• p{M} for combining marks

The exact syntax varies by programming language.

For cursor movement, deletion and truncation, use extended grapheme clusters rather than code-point indexes. Unicode identifies grapheme clusters as a strong basis for matching that remains stable across canonically equivalent normalization forms.

URLs and Slugs

A URL slug is not a scholarly transliteration.

A page may use:

Field	Value
Original	東京
Standard display	Tōkyō
Common form	Tokyo
URL	/tokyo/

The slug should be:

• Stable
• Lowercase where appropriate
• Easy to type
• Redirect-safe
• Separate from the displayed standard form

Avoid regenerating URLs every time a transliteration rule or Unicode version changes. Store the chosen slug as a persistent identifier.

Security Considerations

Unicode normalization and transliteration do not eliminate security risks.

Different scripts contain characters that can look alike:

• Latin a
• Cyrillic а
• Greek α

A malicious identifier can combine scripts to imitate a trusted name.

Unicode Technical Standard #39 provides mechanisms for:

• Identifier restriction profiles
• Mixed-script detection
• Confusable detection
• Confusable skeleton generation
• Script analysis

It warns that Unicode’s large character repertoire can expose systems to spoofing and other security problems when text is handled incorrectly.

Important Distinction

A confusable skeleton is for internal comparison. It should not be displayed as the normalized user name or domain because it may contain a mixture of scripts and is not intended as readable output.

Recommended Identifier Policy

For usernames, domains, project slugs or account handles:

• Normalize consistently.
• Apply a documented identifier profile.
• Restrict unexpected script mixing.
• Check confusable forms.
• Preserve the person’s real display name separately.
• Do not assume that transliteration proves identity.
• Review high-risk names manually.

Mixed-Script Input

Mixed scripts can be legitimate.

Examples include:

• Japanese text containing Latin brand names
• Arabic titles containing numbers
• Cyrillic names containing a Latin acronym
• Hindi text containing an English product name
• A personal name containing punctuation
• Academic text containing both original script and transliteration

A security or transliteration system should not reject all mixed-script text indiscriminately.

Instead, distinguish:

• Expected mixing
• Unexpected mixing
• Identifier contexts
• General prose
• Names
• Source quotations
• Product and technical codes

UTS #39 notes that mixed-script detection can produce legitimate matches and must be interpreted within context.

Fonts and Rendering

Correct Unicode encoding does not guarantee correct display.

Rendering also depends on:

• Font coverage
• Mark positioning
• Shaping support
• Script engine
• Browser or operating system
• Line-height settings
• CSS
• Character fallback

A font may contain the base letter but not position multiple combining marks correctly. Another font may show a missing-glyph box for a specialized modifier letter.

Test transliteration output using:

• Common browser fonts
• Mobile devices
• Multiple operating systems
• Copy and paste
• PDF export
• Email
• Database administration interfaces
• Search result snippets

Do not replace a correct Unicode character merely because one unsuitable font renders it poorly.

Versioning

Unicode, CLDR and ICU evolve.

A production system should record:

• Unicode version
• CLDR version
• ICU or library version
• Transform ID
• Transform variant
• Application rule version
• Conversion timestamp

Transform behavior, security data and script coverage can change between releases.

Unicode’s security specification explicitly warns that persistent indexes may need migration when confusable mapping data is updated because mappings are not guaranteed to remain identical between versions.

For stored romanizations, avoid silently recalculating every value after a library update. Compare old and new results, preserve provenance and migrate deliberately.

Testing a Transliteration System

A robust test suite should cover more than ordinary words.

Canonical Equivalence Tests

Provide both NFC and NFD versions of the same input and verify that they produce equivalent outputs.

Example:

ā
a + combining macron

Combining-Mark Order Tests

Supply canonically equivalent marks in different orders.

Grapheme Tests

Verify cursor movement, truncation and deletion around:

• Multiple diacritics
• Indic vowel signs
• Hangul sequences
• Join controls
• Emoji adjacent to text

Round-Trip Tests

For reversible transforms:

source → Latin → source

Mixed-Script Tests

Include:

• Original script plus Latin
• Numbers
• Punctuation
• Brand names
• Abbreviations

Loss Tests

Verify that simplified output is labeled as lossy.

Search Tests

Ensure that:

• Marked and unmarked forms can be found.
• Original script remains searchable.
• Alternative standards are indexed.
• Search normalization does not change the displayed name.

Security Tests

Test:

• Mixed-script confusables
• Invisible characters
• Excessive combining marks
• Isolated combining marks
• Unexpected join controls
• Spoofed identifiers

Common Implementation Mistakes

Comparing Raw Strings Without Normalization

Visually identical strings may have different encoded sequences.

Normalizing Only One Side of a Comparison

Normalize both stored data and incoming queries according to the same policy.

Using NFKC for Every Stored Field

Compatibility normalization may remove meaningful distinctions. Keep authoritative linguistic text in a canonical form such as NFC unless a specific standard requires otherwise.

Removing Every Mark Character

This can destroy original-script vowels and essential transliteration distinctions.

Treating Code Points as Visible Characters

A visible letter may contain several code points.

Translating by Script Without Identifying the Language

Cyrillic, Arabic script, Han characters and Devanagari are each used by multiple languages.

Using a Generic Transform When a Standard Is Required

Any-Latin is not equivalent to ISO 9, ALA-LC, Pinyin, Hepburn or Revised Romanization.

Overwriting Original Text

Romanization should be an additional representation.

Storing Only an ASCII Form

ASCII is useful for search and URLs but may be irreversible and linguistically incomplete.

Assuming Transliteration Is Safe for Usernames

Unicode confusables and mixed-script spoofing require a separate security policy.

Failing to Version the Transform

Library updates may change results or available characters.

Recommended Design for Romanization.org

Romanization.org should treat text conversion as a transparent, versioned pipeline.

Input Panel

Display:

• Original text
• Detected script
• Selected language
• Normalization status
• Code-point and grapheme information when requested

Standard Selection

Let users choose:

• Language-specific system
• Script-level system
• Institutional standard
• Simplified public form
• Reversible or non-reversible mode

Output Layers

Provide:

Output	Purpose
Standard romanization	Fully marked reference form
NFC representation	Stable display and storage
Decomposed view	Technical inspection
Diacritic-free form	Search convenience
ASCII form	Restricted systems
Alternative standards	Comparison
Reverse conversion	Where supported

Technical Inspector

A useful inspector can show:

• Unicode code points
• Character names
• Combining classes
• NFC/NFD/NFKC/NFKD outputs
• Grapheme-cluster boundaries
• Script properties
• Transform ID
• Unicode and CLDR versions

Warnings

Display warnings such as:

• Input was not normalized.
• Diacritics were removed.
• Output is not reversible.
• Language was inferred from script.
• Several readings are possible.
• Compatibility normalization changed characters.
• Mixed scripts were detected.
• Official personal or geographic spelling may differ.

Frequently Asked Questions

What is Unicode normalization?

Unicode normalization converts equivalent text into one of four standardized encoded forms: NFC, NFD, NFKC or NFKD.

What is the difference between NFC and NFD?

NFC favors canonically composed forms where possible. NFD uses canonically decomposed base characters and combining marks.

Which normalization form should romanization software use?

NFC is generally appropriate for stored and displayed transliteration. NFD is useful internally for manipulating diacritics. NFKC and NFKD should be used only for defined compatibility or identifier purposes.

Does normalization remove accents?

No. Decomposition separates some accents from their base letters, but the marks remain unless software explicitly removes them.

Why do two identical-looking words fail to match?

They may use different Unicode sequences, normalization forms, punctuation characters or modifier letters.

Is ā always one Unicode character?

No. It can be a precomposed character or a sequence containing a and a combining macron.

What is a combining mark?

It is a Unicode character that modifies a preceding base character, such as an accent, dot, macron or vowel sign.

What is a grapheme cluster?

It is a sequence approximating one user-perceived character, potentially containing a base, combining marks and other dependent characters.

Can I remove all combining marks to create ASCII?

You can create a lossy search key that way in selected Latin text, but applying the method globally can corrupt Arabic, Indic and other scripts.

What is CLDR transliteration?

CLDR contains data and rules for script, language and standard-specific transforms used by internationalized software.

What is ICU?

ICU is a set of Unicode internationalization libraries that implements transforms, normalization and many other text-processing services. Its transform framework can chain normalization, transliteration, case conversion and filtering operations.

Is Any-Latin an official romanization standard?

No. It is a broad transform intended to convert many scripts toward Latin. Use a specifically named transform when an institutional standard is required.

Is transliteration reversible?

Sometimes. Reversibility depends on whether the system preserves every necessary distinction and whether marks or boundaries have been removed.

Does normalization make transliteration reversible?

No. Normalization standardizes equivalent encodings; it does not restore information lost by the transliteration rules.

Should search ignore diacritics?

Search can offer diacritic-insensitive matching, but the marked authoritative form should remain stored and displayed.

Is normalization the same as sorting?

No. Sorting requires collation rules. Normalization standardizes equivalent encodings but does not define alphabetical order.

Can Unicode text create security risks?

Yes. Similar-looking characters from different scripts and unexpected mixed-script identifiers can be used for spoofing. UTS #39 defines detection mechanisms and security profiles.

Should databases store NFD or NFC?

NFC is a common practical choice for general stored text. The most important requirement is to choose a consistent policy, normalize at defined boundaries and preserve authoritative content.

Should URLs contain diacritics?

They may, but many sites use stable ASCII slugs for usability and compatibility. The slug should remain separate from the complete display form.

Final Checklist

Before releasing a Unicode transliteration feature, verify:

• The original input is preserved.
• The source language and script are identified.
• The exact transliteration standard is named.
• Input is normalized consistently.
• Output is normalized after transformation.
• NFC and NFD inputs produce equivalent results.
• Diacritics are preserved in the authoritative form.
• Simplified output is labeled as lossy.
• Grapheme clusters are used for editing and truncation.
• Search fields are separate from display fields.
• Original-script search is supported.
• Alternative standards are stored as aliases where useful.
• Generic and language-specific transforms are distinguished.
• Round-trip behavior is tested where claimed.
• Mixed-script and confusable identifiers are checked.
• Unicode, CLDR and transform versions are recorded.
• Font and rendering behavior has been tested.
• Automated conversions can be manually reviewed.

Conclusion

Unicode makes global transliteration possible, but it does not make text processing automatic or risk-free.

A romanized form is more than the letters visible on screen. It may contain precomposed characters, combining sequences, modifier letters and several code points inside one user-perceived character.

Normalization provides consistency:

• NFC favors canonical composition.
• NFD exposes canonical components.
• NFKC and NFKD apply broader compatibility mappings.

Transliteration provides script conversion:

• Generic transforms convert scripts broadly.
• Language-specific transforms incorporate linguistic rules.
• Named variants implement standards such as ISO, BGN/PCGN or library conventions.

Diacritic folding provides search convenience, but it is normally lossy.

Security processing addresses mixed-script and visually confusable text, but it is separate from both normalization and transliteration.

The strongest software architecture therefore preserves several clearly defined layers:

1. Original source text
2. NFC-normalized source
3. Fully marked standard romanization
4. Named transform and version
5. Simplified search form
6. ASCII or URL form
7. Alternative systems and aliases
8. Security and reversibility metadata

Romanization.org should not merely display a converted string. It should show how that string is encoded, which standard produced it, whether its diacritics are preserved, whether the conversion is reversible, and which transformations were applied along the way.

That transparency turns transliteration from a fragile text substitution into a dependable multilingual software process.

FAQ

What is the difference between Unicode normalization and transliteration?

Normalization standardizes how canonically or compatibly equivalent Unicode text is encoded, while transliteration converts text from one script to another without changing its meaning.

Which Unicode normalization form should I use when building a transliteration pipeline?

NFC is usually a safe default for storage and display; however, NFC or NFKC may be applied before transliteration and the opposite form after, depending on whether you need to preserve or discard compatibility characters.

How can I preserve diacritics during transliteration and still support search‑friendly forms?

Store the fully marked transliteration, then generate a secondary “folded” version where selected diacritics are removed (diacritic folding) for indexing and search.

What libraries can I use for Unicode normalization and grapheme‑cluster handling?

Popular options include ICU (C/C++/Java), the Python unicodedata module, the JavaScript String.prototype.normalize() API, and language‑specific packages such as Ruby’s UnicodeUtils.

How do I test round‑trip reversibility of a transliteration scheme?

Apply the transliteration transform to source text, then apply the inverse transform (if defined) to the output and compare the result to the original, ideally after normalizing both strings to the same form.