API reference¶
Package interface¶
- class indicate.IndicLLMTransliterator(source_lang, target_lang, provider=None, model=None, api_key=None, temperature=0.3, cache_examples=True)[source]
Bases:
objectLLM-based transliterator for Indic languages.
- Parameters:
source_lang (str) – Source language (e.g., ‘hindi’, ‘tamil’)
target_lang (str) – Target language (e.g., ‘english’)
provider (str | None) – LLM provider (openai, anthropic, etc.). Auto-detected if not provided.
model (str | None) – Specific model to use. Uses provider defaults if not provided.
api_key (str | None) – API key. Uses environment variables if not provided.
temperature (float) – LLM temperature for consistency (lower = more consistent).
cache_examples (bool) – Whether to cache generated few-shot examples.
- INDIC_LANGUAGES: ClassVar[dict[str, dict[str, str]]] = {'bengali': {'iso': 'bn', 'native': 'বাংলা', 'script': 'bengali'}, 'english': {'iso': 'en', 'native': 'English', 'script': 'latin'}, 'gujarati': {'iso': 'gu', 'native': 'ગુજરાતી', 'script': 'gujarati'}, 'hindi': {'iso': 'hi', 'native': 'हिन्दी', 'script': 'devanagari'}, 'kannada': {'iso': 'kn', 'native': 'ಕನ್ನಡ', 'script': 'kannada'}, 'malayalam': {'iso': 'ml', 'native': 'മലയാളം', 'script': 'malayalam'}, 'marathi': {'iso': 'mr', 'native': 'मराठी', 'script': 'devanagari'}, 'odia': {'iso': 'or', 'native': 'ଓଡ଼ିଆ', 'script': 'odia'}, 'punjabi': {'iso': 'pa', 'native': 'ਪੰਜਾਬੀ', 'script': 'gurmukhi'}, 'sanskrit': {'iso': 'sa', 'native': 'संस्कृतम्', 'script': 'devanagari'}, 'tamil': {'iso': 'ta', 'native': 'தமிழ்', 'script': 'tamil'}, 'telugu': {'iso': 'te', 'native': 'తెలుగు', 'script': 'telugu'}, 'urdu': {'iso': 'ur', 'native': 'اردو', 'script': 'arabic'}}
- DEFAULT_MODELS: ClassVar[dict[str, str]] = {'anthropic': 'claude-3-opus-20240229', 'cohere': 'command-r-plus', 'google': 'gemini-pro', 'openai': 'gpt-5.4-mini'}
- generate_few_shot_examples(num_examples=5)[source]
Generate few-shot transliteration examples for the language pair.
- transliterate(text, use_few_shot=True, num_examples=5)[source]
Transliterate text from source language to target language.
- Parameters:
- Returns:
Transliterated text.
- Raises:
RuntimeError – If the LLM transliteration call fails.
- Return type:
- default_max_tokens_for(texts)[source]
Estimate max output tokens for transliterating
textsas a group.
- build_group_messages(texts, examples=None)[source]
Build chat messages for transliterating a numbered group of texts.
Shared by the synchronous
transliterate_batchand the async Batch-API path (indicate.batch) so both produce identical prompts.
- transliterate_batch(texts, batch_size=10, use_few_shot=True)[source]
Transliterate multiple texts efficiently.
- indicate.detect_indic_script(text)[source]
Auto-detect Indic script from Unicode ranges.
- indicate.detect_language_from_script(text)[source]
Detect the most likely language based on script and context.
- indicate.hindi2english(input, n=1)
Transliterate one text (thin wrapper over
transliterate_batch).- Parameters:
- Returns:
str when
n == 1; list[str] whenn > 1.- Raises:
TypeError – If input is None
ValueError – If input is not a string
- Return type:
- indicate.punjabi2english(input, n=1)
Transliterate one text (thin wrapper over
transliterate_batch).- Parameters:
- Returns:
str when
n == 1; list[str] whenn > 1.- Raises:
TypeError – If input is None
ValueError – If input is not a string
- Return type:
Seq2seq transliterator¶
- class indicate.transliterator.Seq2SeqTransliterator[source]¶
Bases:
objectBase char-level seq2seq transliterator: lazy-loaded singleton.
Subclasses set
SUBDIR(the per-language folder) and the tokenizer filenames. Files are resolved local-first (indicate/data/<SUBDIR>/...— present after training) and otherwise downloaded from the HF model repo (HF_REPO@HF_REVISION) and cached, so the wheel ships no weights. All mutable state is assigned on the concrete subclass, so each language’s model loads and caches independently.- Return type:
Self
- input_lang_tokenizer: CharTokenizer | None = None¶
- target_lang_tokenizer: CharTokenizer | None = None¶
- classmethod get_model_path()[source]¶
Local saved_weights dir (for display; weights may be on HF instead).
- Return type:
- classmethod transliterate(input, n=1)[source]¶
Transliterate one text (thin wrapper over
transliterate_batch).- Parameters:
- Returns:
str when
n == 1; list[str] whenn > 1.- Raises:
TypeError – If input is None
ValueError – If input is not a string
- Return type:
- classmethod transliterate_batch(inputs, n=1)[source]¶
Transliterate many texts at once — the batched decode engine.
All words across all inputs are decoded in a single batch (one encoder / decoder pass per step), which is much faster than calling
transliterateper item. Returns one result per input, aligned toinputs: astreach whenn == 1, else alist[str]of up toncandidates each.
Hindi model¶
- class indicate.hindi2english.HindiToEnglish[source]¶
Bases:
Seq2SeqTransliteratorHindi (Devanagari) → English transliteration model.
- Return type:
Self
- SUBDIR = 'hindi_to_english'¶
- INPUT_VOCAB = 'hindi_tokens.json'¶
- TARGET_VOCAB = 'english_tokens.json'¶
- max_length_input = 47¶
- max_length_output = 173¶
Punjabi model¶
- class indicate.punjabi2english.PunjabiToEnglish[source]¶
Bases:
Seq2SeqTransliteratorPunjabi (Gurmukhi) → English transliteration model.
- Return type:
Self
- SUBDIR = 'punjabi_to_english'¶
- INPUT_VOCAB = 'punjabi_tokens.json'¶
- TARGET_VOCAB = 'english_tokens.json'¶
- max_length_input = 32¶
- max_length_output = 32¶
Encoder¶
- class indicate.encoder.Encoder(vocab_size, embedding_dim, enc_units)[source]¶
Bases:
ModuleLSTM encoder: embedding -> LSTM.
Returns the full output sequence (for attention) and the final
(hidden, cell)state used to initialise the decoder.- forward(x)[source]¶
Define the computation performed at every call.
Should be overridden by all subclasses.
Note
Although the recipe for forward pass needs to be defined within this function, one should call the
Moduleinstance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.
Decoder¶
- class indicate.decoder.Decoder(vocab_size, embedding_dim, dec_units)[source]¶
Bases:
ModuleLSTM decoder with Luong (dot-product) attention.
Mirrors the original Keras model: the attention query is a linear projection of the target embedding (not the recurrent state), attention is unscaled dot-product over the encoder outputs (Keras
Attentionwithuse_scale=False), and the attention context is concatenated with the embedding before the LSTM.forwardworks for both the full target sequence (training, teacher forcing) and a single step (autoregressive inference) by carryingstateacross calls.- forward(inputs, encoder_outputs, state=None, src_mask=None)[source]¶
Define the computation performed at every call.
Should be overridden by all subclasses.
Note
Although the recipe for forward pass needs to be defined within this function, one should call the
Moduleinstance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.
LLM backend¶
LLM-based transliteration for Indic languages using LiteLLM.
- class indicate.llm_indic.IndicLLMTransliterator(source_lang, target_lang, provider=None, model=None, api_key=None, temperature=0.3, cache_examples=True)[source]¶
Bases:
objectLLM-based transliterator for Indic languages.
- Parameters:
source_lang (str) – Source language (e.g., ‘hindi’, ‘tamil’)
target_lang (str) – Target language (e.g., ‘english’)
provider (str | None) – LLM provider (openai, anthropic, etc.). Auto-detected if not provided.
model (str | None) – Specific model to use. Uses provider defaults if not provided.
api_key (str | None) – API key. Uses environment variables if not provided.
temperature (float) – LLM temperature for consistency (lower = more consistent).
cache_examples (bool) – Whether to cache generated few-shot examples.
- INDIC_LANGUAGES: ClassVar[dict[str, dict[str, str]]] = {'bengali': {'iso': 'bn', 'native': 'বাংলা', 'script': 'bengali'}, 'english': {'iso': 'en', 'native': 'English', 'script': 'latin'}, 'gujarati': {'iso': 'gu', 'native': 'ગુજરાતી', 'script': 'gujarati'}, 'hindi': {'iso': 'hi', 'native': 'हिन्दी', 'script': 'devanagari'}, 'kannada': {'iso': 'kn', 'native': 'ಕನ್ನಡ', 'script': 'kannada'}, 'malayalam': {'iso': 'ml', 'native': 'മലയാളം', 'script': 'malayalam'}, 'marathi': {'iso': 'mr', 'native': 'मराठी', 'script': 'devanagari'}, 'odia': {'iso': 'or', 'native': 'ଓଡ଼ିଆ', 'script': 'odia'}, 'punjabi': {'iso': 'pa', 'native': 'ਪੰਜਾਬੀ', 'script': 'gurmukhi'}, 'sanskrit': {'iso': 'sa', 'native': 'संस्कृतम्', 'script': 'devanagari'}, 'tamil': {'iso': 'ta', 'native': 'தமிழ்', 'script': 'tamil'}, 'telugu': {'iso': 'te', 'native': 'తెలుగు', 'script': 'telugu'}, 'urdu': {'iso': 'ur', 'native': 'اردو', 'script': 'arabic'}}¶
- DEFAULT_MODELS: ClassVar[dict[str, str]] = {'anthropic': 'claude-3-opus-20240229', 'cohere': 'command-r-plus', 'google': 'gemini-pro', 'openai': 'gpt-5.4-mini'}¶
- generate_few_shot_examples(num_examples=5)[source]¶
Generate few-shot transliteration examples for the language pair.
- transliterate(text, use_few_shot=True, num_examples=5)[source]¶
Transliterate text from source language to target language.
- Parameters:
- Returns:
Transliterated text.
- Raises:
RuntimeError – If the LLM transliteration call fails.
- Return type:
- default_max_tokens_for(texts)[source]¶
Estimate max output tokens for transliterating
textsas a group.
- build_group_messages(texts, examples=None)[source]¶
Build chat messages for transliterating a numbered group of texts.
Shared by the synchronous
transliterate_batchand the async Batch-API path (indicate.batch) so both produce identical prompts.
Batch transliteration¶
Batch-mode LLM transliteration via provider Batch APIs (LiteLLM).
The synchronous IndicLLMTransliterator issues one
litellm.completion call per request. For transliterating the millions of unique
tokens in an electoral roll, that is slow and expensive. This module routes the work
through a provider’s asynchronous Batch API instead (~50% cheaper), with
checkpointing and resume.
Because a batch can take up to 24h to finish, submit and collect are separate
steps; transliterate_tokens_batched() is a convenience driver that submits then
polls to completion. State is durable, so a killed process resumes from the checkpoint
rather than resubmitting finished work:
checkpoint_path– a JSONL file of resolved{"token", "translit"}pairs (append-only; the durable result map). Kept dependency-free (no pandas/pyarrow).checkpoint_path + ".batchstate.json"– in-flight batch ids and thecustom_id -> [tokens]mapping needed to align results back to tokens.
Provider support is LiteLLM’s batch support: openai, azure, vertex_ai, bedrock,
vllm – not native Anthropic. To use Claude in batch mode, go through Bedrock
(provider="bedrock", model "anthropic.claude-sonnet-4-6") or add a native
Anthropic messages.batches adapter later. The default provider is openai.
- class indicate.batch.BatchJob(batch_id, input_file_id, custom_id_to_tokens, status='submitted', output_file_id=None)[source]¶
Bases:
objectOne submitted batch (a provider batch id + the tokens it covers).
- Parameters:
- class indicate.batch.BatchState(provider, model, source_lang, target_lang, group_size, temperature, use_few_shot, jobs=<factory>, submitted_at=None)[source]¶
Bases:
objectDurable record of an in-flight transliteration run.
- Parameters:
- indicate.batch.submit_transliteration_batches(tokens, source_lang, target_lang, *, checkpoint_path, provider=None, model=None, api_key=None, group_size=25, completion_window='24h', use_few_shot=True, temperature=0.3, max_requests_per_batch=50000)[source]¶
Submit unique
tokensto the provider Batch API. Does not block.Already-resolved tokens (present in the checkpoint) and duplicates/blanks are skipped. Returns the persisted
BatchState.
- indicate.batch.collect_transliteration_batches(checkpoint_path, *, transliterator=None)[source]¶
Poll in-flight batches once and append any completed results.
Returns
(all_done, resolved_map). A group whose result count does not match the request, or whose batch failed, is left out ofresolved_map(the driver requeues such tokens). Safe to call repeatedly.
- indicate.batch.transliterate_tokens_batched(tokens, source_lang, target_lang, *, checkpoint_path, provider=None, model=None, api_key=None, group_size=25, completion_window='24h', use_few_shot=True, temperature=0.3, poll_interval=60.0, max_wait=None, requeue_passes=2, max_requests_per_batch=50000)[source]¶
Submit
tokensin batch mode and poll to completion; return token->translit.Resumable: if a batch is already in flight for
checkpoint_paththis skips submission and resumes polling. Tokens whose group output was malformed are requeued one-per-request (up torequeue_passes). Ifmax_waitelapses with batches still running, returns what is resolved so far – rerun later to resume.
Reranking¶
- class indicate.rerank.Reranker(words, alpha=0.9, order=3, k=1.0)[source]¶
Bases:
objectRe-rank beam candidates by interpolating model score with a char LM.
Following IndicXlit (Aksharantar, EMNLP Findings 2023), the top-k beam hypotheses are re-scored as
F = alpha * model_score + (1 - alpha) * lm. Here the LM is a character n-gram model over the training-side English romanizations (so it generalizes to unseen names, unlike a word unigram), with both scores length-normalized to keep them comparable.
Indic text utilities¶
Utilities for Indic language detection and validation.
- indicate.indic_utils.detect_indic_script(text)[source]¶
Auto-detect Indic script from Unicode ranges.
- indicate.indic_utils.detect_language_from_script(text)[source]¶
Detect the most likely language based on script and context.
- indicate.indic_utils.validate_indic_language_pair(source, target)[source]¶
Validate that at least one language is Indic.
- indicate.indic_utils.normalize_text_for_transliteration(text)[source]¶
Normalize Indic text for better transliteration.
File utilities¶
Safe file handling utilities for transliteration operations.
- class indicate.file_utils.OutputFormat[source]¶
Bases:
objectOutput format definitions and handlers.
- TEXT = 'text'¶
- JSON = 'json'¶
- class indicate.file_utils.TransliterationResult(line_number, input_text, output_text, source_lang, target_lang, confidence='unknown', error=None, processing_time=None)[source]¶
Bases:
objectRepresents a single transliteration result with metadata.
- Parameters:
- class indicate.file_utils.BatchProgress(total_lines, output_path)[source]¶
Bases:
objectTracks progress of batch transliteration operations.
- Parameters:
total_lines (int)
output_path (Path)
- results: list[TransliterationResult]¶
- add_result(result)[source]¶
Add a transliteration result.
- Parameters:
result (TransliterationResult)
- classmethod load_progress(progress_file)[source]¶
Load progress from disk.
- Parameters:
progress_file (Path)
- Return type:
BatchProgress | None
- indicate.file_utils.validate_file_paths(input_path, output_path)[source]¶
Validate input and output file paths for safety.
- Parameters:
- Raises:
ValueError – If paths are invalid or dangerous.
- Return type:
None
- indicate.file_utils.write_output_safely(results, output_path, output_format, source_lang, target_lang, atomic=True)[source]¶
Write transliteration results to file safely.
- Parameters:
- Raises:
ValueError – If
output_formatis not a supported format.- Return type:
None
- indicate.file_utils.read_input_file(input_path)[source]¶
Read input file safely, handling both text and JSON formats.
- indicate.file_utils.check_resume_possibility(output_path)[source]¶
Check if a previous batch operation can be resumed.
- Parameters:
output_path (Path) – Output file path.
- Returns:
BatchProgress object if resume is possible, None otherwise.
- Return type:
BatchProgress | None
Tokenizer and decoding utilities¶
- class indicate.utils.CharTokenizer(word_index)[source]¶
Bases:
objectCharacter-level tokenizer holding the
word_index/index_wordmaps.Loaded from the JSON files that were serialised by the original Keras
Tokenizerso the vocabulary indices stay identical across the migration.
- indicate.utils.load_tokenizer(path)[source]¶
Load a character tokenizer from a Keras-serialised tokenizer JSON file.
- Parameters:
path (str)
- Return type:
- indicate.utils.sequence_to_chars(tokenizer, sequence)[source]¶
Convert a sequence of indices back to characters, skipping padding (0).
- Parameters:
tokenizer (CharTokenizer)
- Return type:
- indicate.utils.batch_candidates(words, input_lang_tokenizer, target_lang_tokenizer, encoder, decoder, max_length_input, max_length_output, beam_width=1, mask_padding=False)[source]¶
Ranked candidates for many words at once (the batched decode engine).
Returns one ranked
(text, score)list per input word, aligned towords(empty/OOV words ->[]). Inputs are padded tomax_length_inputexactly as the single-word path, so outputs are identical — just faster.
Command-line interface¶
Modern CLI for the indicate package using Click.