API reference

Package interface

class indicate.IndicLLMTransliterator(source_lang, target_lang, provider=None, model=None, api_key=None, temperature=0.3, cache_examples=True)[source]

Bases: object

LLM-based transliterator for Indic languages.

Parameters:
  • source_lang (str) – Source language (e.g., ‘hindi’, ‘tamil’)

  • target_lang (str) – Target language (e.g., ‘english’)

  • provider (str | None) – LLM provider (openai, anthropic, etc.). Auto-detected if not provided.

  • model (str | None) – Specific model to use. Uses provider defaults if not provided.

  • api_key (str | None) – API key. Uses environment variables if not provided.

  • temperature (float) – LLM temperature for consistency (lower = more consistent).

  • cache_examples (bool) – Whether to cache generated few-shot examples.

INDIC_LANGUAGES: ClassVar[dict[str, dict[str, str]]] = {'bengali': {'iso': 'bn', 'native': 'বাংলা', 'script': 'bengali'}, 'english': {'iso': 'en', 'native': 'English', 'script': 'latin'}, 'gujarati': {'iso': 'gu', 'native': 'ગુજરાતી', 'script': 'gujarati'}, 'hindi': {'iso': 'hi', 'native': 'हिन्दी', 'script': 'devanagari'}, 'kannada': {'iso': 'kn', 'native': 'ಕನ್ನಡ', 'script': 'kannada'}, 'malayalam': {'iso': 'ml', 'native': 'മലയാളം', 'script': 'malayalam'}, 'marathi': {'iso': 'mr', 'native': 'मराठी', 'script': 'devanagari'}, 'odia': {'iso': 'or', 'native': 'ଓଡ଼ିଆ', 'script': 'odia'}, 'punjabi': {'iso': 'pa', 'native': 'ਪੰਜਾਬੀ', 'script': 'gurmukhi'}, 'sanskrit': {'iso': 'sa', 'native': 'संस्कृतम्', 'script': 'devanagari'}, 'tamil': {'iso': 'ta', 'native': 'தமிழ்', 'script': 'tamil'}, 'telugu': {'iso': 'te', 'native': 'తెలుగు', 'script': 'telugu'}, 'urdu': {'iso': 'ur', 'native': 'اردو', 'script': 'arabic'}}
DEFAULT_MODELS: ClassVar[dict[str, str]] = {'anthropic': 'claude-3-opus-20240229', 'cohere': 'command-r-plus', 'google': 'gemini-pro', 'openai': 'gpt-5.4-mini'}
generate_few_shot_examples(num_examples=5)[source]

Generate few-shot transliteration examples for the language pair.

Parameters:

num_examples (int) – Number of examples to generate.

Returns:

List of dictionaries with ‘source’ and ‘target’ keys.

Return type:

list[dict[str, str]]

transliterate(text, use_few_shot=True, num_examples=5)[source]

Transliterate text from source language to target language.

Parameters:
  • text (str) – Text to transliterate.

  • use_few_shot (bool) – Whether to use few-shot examples.

  • num_examples (int) – Number of few-shot examples to use.

Returns:

Transliterated text.

Raises:

RuntimeError – If the LLM transliteration call fails.

Return type:

str

default_max_tokens_for(texts)[source]

Estimate max output tokens for transliterating texts as a group.

Parameters:

texts (list[str])

Return type:

int

build_group_messages(texts, examples=None)[source]

Build chat messages for transliterating a numbered group of texts.

Shared by the synchronous transliterate_batch and the async Batch-API path (indicate.batch) so both produce identical prompts.

Parameters:
Return type:

list[dict[str, str]]

transliterate_batch(texts, batch_size=10, use_few_shot=True)[source]

Transliterate multiple texts efficiently.

Parameters:
  • texts (list[str]) – List of texts to transliterate.

  • batch_size (int) – Number of texts to process in one API call.

  • use_few_shot (bool) – Whether to use few-shot examples.

Returns:

List of transliterated texts.

Return type:

list[str]

indicate.detect_indic_script(text)[source]

Auto-detect Indic script from Unicode ranges.

Parameters:

text (str) – Text to analyze.

Returns:

Detected script name or None if not Indic.

Return type:

str | None

indicate.detect_language_from_script(text)[source]

Detect the most likely language based on script and context.

Parameters:

text (str) – Text to analyze.

Returns:

Detected language name or None.

Return type:

str | None

indicate.hindi2english(input, n=1)

Transliterate one text (thin wrapper over transliterate_batch).

Parameters:
  • input (str) – source-language text

  • n (int) – number of candidates. n == 1 (default) returns a single best string; n > 1 returns a list of up to n ranked candidates (requires beam search).

Returns:

str when n == 1; list[str] when n > 1.

Raises:
Return type:

str | list[str]

indicate.punjabi2english(input, n=1)

Transliterate one text (thin wrapper over transliterate_batch).

Parameters:
  • input (str) – source-language text

  • n (int) – number of candidates. n == 1 (default) returns a single best string; n > 1 returns a list of up to n ranked candidates (requires beam search).

Returns:

str when n == 1; list[str] when n > 1.

Raises:
Return type:

str | list[str]

Seq2seq transliterator

class indicate.transliterator.Seq2SeqTransliterator[source]

Bases: object

Base char-level seq2seq transliterator: lazy-loaded singleton.

Subclasses set SUBDIR (the per-language folder) and the tokenizer filenames. Files are resolved local-first (indicate/data/<SUBDIR>/... — present after training) and otherwise downloaded from the HF model repo (HF_REPO @ HF_REVISION) and cached, so the wheel ships no weights. All mutable state is assigned on the concrete subclass, so each language’s model loads and caches independently.

Return type:

Self

HF_REPO: str = 'soodoku/indicate'
HF_REVISION: str = 'v0.7.0'
SUBDIR: str = ''
INPUT_VOCAB: str = ''
TARGET_VOCAB: str = ''
ENCODER_FILE: str = 'saved_weights/encoder.safetensors'
DECODER_FILE: str = 'saved_weights/decoder.safetensors'
embedding_dim: int = 256
units: int = 1024
max_length_input: int = 64
max_length_output: int = 64
START_TOKEN: str = '^'
END_TOKEN: str = '$'
BEAM_WIDTH: int = 5
RERANKER: Reranker | None = None
MASK_PADDING: bool = False
input_lang_tokenizer: CharTokenizer | None = None
target_lang_tokenizer: CharTokenizer | None = None
encoder: Encoder | None = None
decoder: Decoder | None = None
classmethod get_model_path()[source]

Local saved_weights dir (for display; weights may be on HF instead).

Return type:

str

classmethod get_input_vocab()[source]
Return type:

str

classmethod get_target_vocab()[source]
Return type:

str

classmethod transliterate(input, n=1)[source]

Transliterate one text (thin wrapper over transliterate_batch).

Parameters:
  • input (str) – source-language text

  • n (int) – number of candidates. n == 1 (default) returns a single best string; n > 1 returns a list of up to n ranked candidates (requires beam search).

Returns:

str when n == 1; list[str] when n > 1.

Raises:
Return type:

str | list[str]

classmethod transliterate_batch(inputs, n=1)[source]

Transliterate many texts at once — the batched decode engine.

All words across all inputs are decoded in a single batch (one encoder / decoder pass per step), which is much faster than calling transliterate per item. Returns one result per input, aligned to inputs: a str each when n == 1, else a list[str] of up to n candidates each.

Parameters:
Return type:

list[str] | list[list[str]]

Hindi model

class indicate.hindi2english.HindiToEnglish[source]

Bases: Seq2SeqTransliterator

Hindi (Devanagari) → English transliteration model.

Return type:

Self

SUBDIR = 'hindi_to_english'
INPUT_VOCAB = 'hindi_tokens.json'
TARGET_VOCAB = 'english_tokens.json'
max_length_input = 47
max_length_output = 173

Punjabi model

class indicate.punjabi2english.PunjabiToEnglish[source]

Bases: Seq2SeqTransliterator

Punjabi (Gurmukhi) → English transliteration model.

Return type:

Self

SUBDIR = 'punjabi_to_english'
INPUT_VOCAB = 'punjabi_tokens.json'
TARGET_VOCAB = 'english_tokens.json'
max_length_input = 32
max_length_output = 32

Encoder

class indicate.encoder.Encoder(vocab_size, embedding_dim, enc_units)[source]

Bases: Module

LSTM encoder: embedding -> LSTM.

Returns the full output sequence (for attention) and the final (hidden, cell) state used to initialise the decoder.

Parameters:
  • vocab_size (int)

  • embedding_dim (int)

  • enc_units (int)

forward(x)[source]

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

Parameters:

x (Tensor)

Return type:

tuple[Tensor, tuple[Tensor, Tensor]]

Decoder

class indicate.decoder.Decoder(vocab_size, embedding_dim, dec_units)[source]

Bases: Module

LSTM decoder with Luong (dot-product) attention.

Mirrors the original Keras model: the attention query is a linear projection of the target embedding (not the recurrent state), attention is unscaled dot-product over the encoder outputs (Keras Attention with use_scale=False), and the attention context is concatenated with the embedding before the LSTM.

forward works for both the full target sequence (training, teacher forcing) and a single step (autoregressive inference) by carrying state across calls.

Parameters:
  • vocab_size (int)

  • embedding_dim (int)

  • dec_units (int)

forward(inputs, encoder_outputs, state=None, src_mask=None)[source]

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

Parameters:
  • inputs (Tensor)

  • encoder_outputs (Tensor)

  • state (tuple[Tensor, Tensor] | None)

  • src_mask (Tensor | None)

Return type:

tuple[Tensor, tuple[Tensor, Tensor]]

LLM backend

LLM-based transliteration for Indic languages using LiteLLM.

class indicate.llm_indic.IndicLLMTransliterator(source_lang, target_lang, provider=None, model=None, api_key=None, temperature=0.3, cache_examples=True)[source]

Bases: object

LLM-based transliterator for Indic languages.

Parameters:
  • source_lang (str) – Source language (e.g., ‘hindi’, ‘tamil’)

  • target_lang (str) – Target language (e.g., ‘english’)

  • provider (str | None) – LLM provider (openai, anthropic, etc.). Auto-detected if not provided.

  • model (str | None) – Specific model to use. Uses provider defaults if not provided.

  • api_key (str | None) – API key. Uses environment variables if not provided.

  • temperature (float) – LLM temperature for consistency (lower = more consistent).

  • cache_examples (bool) – Whether to cache generated few-shot examples.

INDIC_LANGUAGES: ClassVar[dict[str, dict[str, str]]] = {'bengali': {'iso': 'bn', 'native': 'বাংলা', 'script': 'bengali'}, 'english': {'iso': 'en', 'native': 'English', 'script': 'latin'}, 'gujarati': {'iso': 'gu', 'native': 'ગુજરાતી', 'script': 'gujarati'}, 'hindi': {'iso': 'hi', 'native': 'हिन्दी', 'script': 'devanagari'}, 'kannada': {'iso': 'kn', 'native': 'ಕನ್ನಡ', 'script': 'kannada'}, 'malayalam': {'iso': 'ml', 'native': 'മലയാളം', 'script': 'malayalam'}, 'marathi': {'iso': 'mr', 'native': 'मराठी', 'script': 'devanagari'}, 'odia': {'iso': 'or', 'native': 'ଓଡ଼ିଆ', 'script': 'odia'}, 'punjabi': {'iso': 'pa', 'native': 'ਪੰਜਾਬੀ', 'script': 'gurmukhi'}, 'sanskrit': {'iso': 'sa', 'native': 'संस्कृतम्', 'script': 'devanagari'}, 'tamil': {'iso': 'ta', 'native': 'தமிழ்', 'script': 'tamil'}, 'telugu': {'iso': 'te', 'native': 'తెలుగు', 'script': 'telugu'}, 'urdu': {'iso': 'ur', 'native': 'اردو', 'script': 'arabic'}}
DEFAULT_MODELS: ClassVar[dict[str, str]] = {'anthropic': 'claude-3-opus-20240229', 'cohere': 'command-r-plus', 'google': 'gemini-pro', 'openai': 'gpt-5.4-mini'}
generate_few_shot_examples(num_examples=5)[source]

Generate few-shot transliteration examples for the language pair.

Parameters:

num_examples (int) – Number of examples to generate.

Returns:

List of dictionaries with ‘source’ and ‘target’ keys.

Return type:

list[dict[str, str]]

transliterate(text, use_few_shot=True, num_examples=5)[source]

Transliterate text from source language to target language.

Parameters:
  • text (str) – Text to transliterate.

  • use_few_shot (bool) – Whether to use few-shot examples.

  • num_examples (int) – Number of few-shot examples to use.

Returns:

Transliterated text.

Raises:

RuntimeError – If the LLM transliteration call fails.

Return type:

str

default_max_tokens_for(texts)[source]

Estimate max output tokens for transliterating texts as a group.

Parameters:

texts (list[str])

Return type:

int

build_group_messages(texts, examples=None)[source]

Build chat messages for transliterating a numbered group of texts.

Shared by the synchronous transliterate_batch and the async Batch-API path (indicate.batch) so both produce identical prompts.

Parameters:
Return type:

list[dict[str, str]]

transliterate_batch(texts, batch_size=10, use_few_shot=True)[source]

Transliterate multiple texts efficiently.

Parameters:
  • texts (list[str]) – List of texts to transliterate.

  • batch_size (int) – Number of texts to process in one API call.

  • use_few_shot (bool) – Whether to use few-shot examples.

Returns:

List of transliterated texts.

Return type:

list[str]

Batch transliteration

Batch-mode LLM transliteration via provider Batch APIs (LiteLLM).

The synchronous IndicLLMTransliterator issues one litellm.completion call per request. For transliterating the millions of unique tokens in an electoral roll, that is slow and expensive. This module routes the work through a provider’s asynchronous Batch API instead (~50% cheaper), with checkpointing and resume.

Because a batch can take up to 24h to finish, submit and collect are separate steps; transliterate_tokens_batched() is a convenience driver that submits then polls to completion. State is durable, so a killed process resumes from the checkpoint rather than resubmitting finished work:

  • checkpoint_path – a JSONL file of resolved {"token", "translit"} pairs (append-only; the durable result map). Kept dependency-free (no pandas/pyarrow).

  • checkpoint_path + ".batchstate.json" – in-flight batch ids and the custom_id -> [tokens] mapping needed to align results back to tokens.

Provider support is LiteLLM’s batch support: openai, azure, vertex_ai, bedrock, vllmnot native Anthropic. To use Claude in batch mode, go through Bedrock (provider="bedrock", model "anthropic.claude-sonnet-4-6") or add a native Anthropic messages.batches adapter later. The default provider is openai.

class indicate.batch.BatchJob(batch_id, input_file_id, custom_id_to_tokens, status='submitted', output_file_id=None)[source]

Bases: object

One submitted batch (a provider batch id + the tokens it covers).

Parameters:
batch_id: str
input_file_id: str
custom_id_to_tokens: dict[str, list[str]]
status: str = 'submitted'
output_file_id: str | None = None
class indicate.batch.BatchState(provider, model, source_lang, target_lang, group_size, temperature, use_few_shot, jobs=<factory>, submitted_at=None)[source]

Bases: object

Durable record of an in-flight transliteration run.

Parameters:
provider: str
model: str
source_lang: str
target_lang: str
group_size: int
temperature: float
use_few_shot: bool
jobs: list[BatchJob]
submitted_at: float | None = None
indicate.batch.submit_transliteration_batches(tokens, source_lang, target_lang, *, checkpoint_path, provider=None, model=None, api_key=None, group_size=25, completion_window='24h', use_few_shot=True, temperature=0.3, max_requests_per_batch=50000)[source]

Submit unique tokens to the provider Batch API. Does not block.

Already-resolved tokens (present in the checkpoint) and duplicates/blanks are skipped. Returns the persisted BatchState.

Parameters:
  • tokens (list[str])

  • source_lang (str)

  • target_lang (str)

  • checkpoint_path (str | Path)

  • provider (str | None)

  • model (str | None)

  • api_key (str | None)

  • group_size (int)

  • completion_window (str)

  • use_few_shot (bool)

  • temperature (float)

  • max_requests_per_batch (int)

Return type:

BatchState

indicate.batch.collect_transliteration_batches(checkpoint_path, *, transliterator=None)[source]

Poll in-flight batches once and append any completed results.

Returns (all_done, resolved_map). A group whose result count does not match the request, or whose batch failed, is left out of resolved_map (the driver requeues such tokens). Safe to call repeatedly.

Parameters:
Return type:

tuple[bool, dict[str, str]]

indicate.batch.transliterate_tokens_batched(tokens, source_lang, target_lang, *, checkpoint_path, provider=None, model=None, api_key=None, group_size=25, completion_window='24h', use_few_shot=True, temperature=0.3, poll_interval=60.0, max_wait=None, requeue_passes=2, max_requests_per_batch=50000)[source]

Submit tokens in batch mode and poll to completion; return token->translit.

Resumable: if a batch is already in flight for checkpoint_path this skips submission and resumes polling. Tokens whose group output was malformed are requeued one-per-request (up to requeue_passes). If max_wait elapses with batches still running, returns what is resolved so far – rerun later to resume.

Parameters:
  • tokens (list[str])

  • source_lang (str)

  • target_lang (str)

  • checkpoint_path (str | Path)

  • provider (str | None)

  • model (str | None)

  • api_key (str | None)

  • group_size (int)

  • completion_window (str)

  • use_few_shot (bool)

  • temperature (float)

  • poll_interval (float)

  • max_wait (float | None)

  • requeue_passes (int)

  • max_requests_per_batch (int)

Return type:

dict[str, str]

Reranking

class indicate.rerank.Reranker(words, alpha=0.9, order=3, k=1.0)[source]

Bases: object

Re-rank beam candidates by interpolating model score with a char LM.

Following IndicXlit (Aksharantar, EMNLP Findings 2023), the top-k beam hypotheses are re-scored as F = alpha * model_score + (1 - alpha) * lm. Here the LM is a character n-gram model over the training-side English romanizations (so it generalizes to unseen names, unlike a word unigram), with both scores length-normalized to keep them comparable.

Parameters:
lm_logprob(word)[source]

Length-normalized char-n-gram log-probability (add-k smoothed).

Parameters:

word (str)

Return type:

float

best(candidates)[source]

Return the candidate text maximizing the interpolated score.

candidates are (text, model_score) where model_score is the beam’s length-normalized log-probability.

Parameters:

candidates (list[tuple[str, float]])

Return type:

str

Indic text utilities

Utilities for Indic language detection and validation.

indicate.indic_utils.detect_indic_script(text)[source]

Auto-detect Indic script from Unicode ranges.

Parameters:

text (str) – Text to analyze.

Returns:

Detected script name or None if not Indic.

Return type:

str | None

indicate.indic_utils.detect_language_from_script(text)[source]

Detect the most likely language based on script and context.

Parameters:

text (str) – Text to analyze.

Returns:

Detected language name or None.

Return type:

str | None

indicate.indic_utils.is_indic_script(script)[source]

Check if a script is an Indic script.

Parameters:

script (str) – Script name.

Returns:

True if script is Indic, False otherwise.

Return type:

bool

indicate.indic_utils.validate_indic_language_pair(source, target)[source]

Validate that at least one language is Indic.

Parameters:
  • source (str) – Source language or script.

  • target (str) – Target language or script.

Returns:

True if valid (at least one is Indic), False otherwise.

Return type:

bool

indicate.indic_utils.normalize_text_for_transliteration(text)[source]

Normalize Indic text for better transliteration.

Parameters:

text (str) – Input text.

Returns:

Normalized text.

Return type:

str

indicate.indic_utils.split_mixed_script_text(text)[source]

Split text containing multiple scripts into segments.

Parameters:

text (str) – Text potentially containing multiple scripts.

Returns:

List of (text_segment, script) tuples.

Return type:

list[tuple[str, str]]

indicate.indic_utils.get_language_info(language)[source]

Get detailed information about a language.

Parameters:

language (str) – Language name.

Returns:

Dictionary with language information.

Return type:

dict

File utilities

Safe file handling utilities for transliteration operations.

class indicate.file_utils.OutputFormat[source]

Bases: object

Output format definitions and handlers.

TEXT = 'text'
JSON = 'json'
VALID_FORMATS: ClassVar[list[str]] = ['text', 'json']
class indicate.file_utils.TransliterationResult(line_number, input_text, output_text, source_lang, target_lang, confidence='unknown', error=None, processing_time=None)[source]

Bases: object

Represents a single transliteration result with metadata.

Parameters:
  • line_number (int)

  • input_text (str)

  • output_text (str)

  • source_lang (str)

  • target_lang (str)

  • confidence (str)

  • error (str | None)

  • processing_time (float | None)

to_dict()[source]

Convert transliteration result to dictionary format.

Returns:

Dictionary representation suitable for JSON serialization.

Return type:

dict[str, Any]

class indicate.file_utils.BatchProgress(total_lines, output_path)[source]

Bases: object

Tracks progress of batch transliteration operations.

Parameters:
  • total_lines (int)

  • output_path (Path)

results: list[TransliterationResult]
add_result(result)[source]

Add a transliteration result.

Parameters:

result (TransliterationResult)

save_progress()[source]

Save current progress to disk for recovery.

classmethod load_progress(progress_file)[source]

Load progress from disk.

Parameters:

progress_file (Path)

Return type:

BatchProgress | None

cleanup()[source]

Remove progress file after successful completion.

indicate.file_utils.validate_file_paths(input_path, output_path)[source]

Validate input and output file paths for safety.

Parameters:
  • input_path (Path | None) – Input file path.

  • output_path (Path | None) – Output file path.

Raises:

ValueError – If paths are invalid or dangerous.

Return type:

None

indicate.file_utils.create_backup(file_path)[source]

Create a backup of a file before overwriting.

Parameters:

file_path (Path) – File to backup.

Returns:

Path to backup file, or None if backup wasn’t needed.

Return type:

Path | None

indicate.file_utils.write_output_safely(results, output_path, output_format, source_lang, target_lang, atomic=True)[source]

Write transliteration results to file safely.

Parameters:
  • results (list[TransliterationResult]) – List of transliteration results.

  • output_path (Path) – Path to output file.

  • output_format (str) – Output format (text, json, csv).

  • source_lang (str) – Source language.

  • target_lang (str) – Target language.

  • atomic (bool) – Whether to use atomic writing.

Raises:

ValueError – If output_format is not a supported format.

Return type:

None

indicate.file_utils.read_input_file(input_path)[source]

Read input file safely, handling both text and JSON formats.

Parameters:

input_path (Path) – Path to input file.

Returns:

List of text lines to transliterate.

Return type:

list[str]

indicate.file_utils.check_resume_possibility(output_path)[source]

Check if a previous batch operation can be resumed.

Parameters:

output_path (Path) – Output file path.

Returns:

BatchProgress object if resume is possible, None otherwise.

Return type:

BatchProgress | None

Tokenizer and decoding utilities

class indicate.utils.CharTokenizer(word_index)[source]

Bases: object

Character-level tokenizer holding the word_index/index_word maps.

Loaded from the JSON files that were serialised by the original Keras Tokenizer so the vocabulary indices stay identical across the migration.

Parameters:

word_index (dict[str, int])

property vocab_size: int
indicate.utils.load_tokenizer(path)[source]

Load a character tokenizer from a Keras-serialised tokenizer JSON file.

Parameters:

path (str)

Return type:

CharTokenizer

indicate.utils.sequence_to_chars(tokenizer, sequence)[source]

Convert a sequence of indices back to characters, skipping padding (0).

Parameters:
Return type:

str

indicate.utils.batch_candidates(words, input_lang_tokenizer, target_lang_tokenizer, encoder, decoder, max_length_input, max_length_output, beam_width=1, mask_padding=False)[source]

Ranked candidates for many words at once (the batched decode engine).

Returns one ranked (text, score) list per input word, aligned to words (empty/OOV words -> []). Inputs are padded to max_length_input exactly as the single-word path, so outputs are identical — just faster.

Parameters:
Return type:

list[list[tuple[str, float]]]

Command-line interface

Modern CLI for the indicate package using Click.