API reference

Package interface

Transliterate Indic text, choosing how much machinery you pay for.

exception indicate.BackendsUnavailableError[source]

Bases: RuntimeError

No backend in the chain could obtain what it needed to answer anything.

Raised rather than returning "", which is indistinguishable from a legitimate empty answer and exits zero.

exception indicate.UnknownLanguageError[source]

Bases: ValueError

A language name that is not in LANGUAGES or ALIASES.

exception indicate.UnsupportedPairError[source]

Bases: ValueError

No backend in the requested engine can transliterate this direction.

indicate.detect_indic_script(text)[source]

Auto-detect the dominant script of a string.

Parameters:

text (str) – Text to analyze.

Returns:

Script name, "latin" for mostly-ASCII text, or None.

Return type:

str | None

indicate.detect_language_from_script(text)[source]

Guess the language of a string from its script.

Parameters:

text (str) – Text to analyze.

Returns:

Language name, or None if the script maps to none.

Return type:

str | None

indicate.supported(backend=None)[source]

Report every direction this install can transliterate, and by what.

Parameters:

backend (str | None) – Restrict to one backend, or None for all of them.

Returns:

(source, target) to the backends that cover it, in chain order.

Return type:

dict[tuple[str, str], tuple[str, …]]

indicate.supports(source, target, backend)[source]

Report whether one backend can transliterate one direction.

Parameters:

  • source (str) – Canonical source language name.

  • target (str) – Canonical target language name.

  • backend (str) – A member of BACKENDS.

Returns:

True if that backend covers the direction on this machine.

Return type:

bool

indicate.transliterate(text, *, source=None, target='english', engine=None, n=1, beam=None, reranker=None, llm=None, **llm_kwargs)[source]

Transliterate one text.

Parameters:

  • text (str) – Source-language text.

  • source (str | None) – Source language; None detects it from text.

  • target (str) – Target language.

  • engine (Sequence[str] | str | None) – Backend names in order, or None for ("lookup", "model").

  • n (int) – Number of candidates. 1 returns a single string.

  • beam (int | None) – Beam width override for the model backend.

  • reranker (Reranker | None) – Optional LM re-ranker.

  • llm (IndicLLMTransliterator | None) – An existing IndicLLMTransliterator to reuse.

  • **llm_kwargs (Any) – Provider settings for the llm backend.

Return type:

str | list[str]

Propagates UnsupportedPairError from transliterate_batch() when the direction is unsupported or undetectable; it is not raised here.

Returns:

A str when n == 1; a list of up to n candidates otherwise.

Raises:
Parameters:
Return type:

str | list[str]

indicate.transliterate_batch(texts, *, source=None, target='english', engine=None, n=1, beam=None, reranker=None, llm=None, **llm_kwargs)[source]

Transliterate many texts at once.

Batching is what makes the local model usable: every word that reaches the decoder across every input is decoded in a single pass.

Parameters:

  • texts (Sequence[str]) – Source-language texts.

  • source (str | None) – Source language; None detects it from the texts.

  • target (str) – Target language.

  • engine (Sequence[str] | str | None) – Backend names in order, or None for ("lookup", "model").

  • n (int) – Candidates per input. 1 returns one string each.

  • beam (int | None) – Beam width override for the model backend.

  • reranker (Reranker | None) – Optional LM re-ranker, applied to model candidates only.

  • llm (IndicLLMTransliterator | None) – An existing IndicLLMTransliterator to reuse.

  • **llm_kwargs (Any) – Provider settings for the llm backend.

Return type:

list[str] | list[list[str]]

Propagates UnsupportedPairError from resolve_pair() when the direction cannot be detected, or when no backend in the chain supports it. It is documented here rather than under Raises because nothing in this body raises it directly.

Returns:

a str each when n == 1, else a list of up to n candidates each.

Return type:

One result per input

Parameters:

Seq2seq transliterator

The local seq2seq decoder, as one loadable object per language pair.

This used to be a class hierarchy – Seq2SeqTransliterator with a subclass per direction, holding its weights in class attributes on a __new__-cached singleton. Adding a language meant adding a module, and the mutable class state leaked between callers badly enough that the tests had to restore it in tearDown.

Now a direction is a Pair – data – and the loaded weights are an instance, cached per pair in _MODELS. Adding a language is a dict entry.

This module deliberately knows nothing about lookup tables, LLMs or chains. It decodes words with torch and stops there; indicate.engine composes it with everything else. torch is imported inside Seq2SeqModel.load(), so importing this module costs nothing.

indicate.transliterator.EMBEDDING_DIM = 256

both languages were trained with the same encoder/decoder shape, and changing either would need new weights rather than a new setting.

Type:

Architecture of the shipped weights. Not per-pair

indicate.transliterator.DEFAULT_BEAM = 5

Beam width when a caller does not pick one. Beam search is the shipped default: +1.5 points exact-match on Hindi Dakshina against greedy.

class indicate.transliterator.Seq2SeqModel(pair, *, mask_padding=False)[source]

Bases: object

Char-level encoder/decoder weights for one language pair, loaded lazily.

Files resolve local-first (indicate/data/<subdir>/, present after training) and otherwise download from the Hugging Face model repo and cache, so the wheel ships no weights.

Parameters:
input_tokenizer: CharTokenizer | None
target_tokenizer: CharTokenizer | None
encoder: Encoder | None
decoder: Decoder | None
path(rel)[source]

Resolve one of this pair’s files, downloading it if necessary.

Parameters:

rel (str) – Path relative to the pair’s data directory.

Returns:

A path on disk.

Return type:

str

property weights_dir: str

Local weights directory, for display. Weights may live on HF instead.

load()[source]

Load tokenizers and weights. Idempotent.

Raises:

RuntimeError – If any file is missing or fails to load.

Return type:

None

candidates(words, beam=5)[source]

Decode words in one batch.

Parameters:

  • words (Sequence[str]) – Source-language words.

  • beam (int) – Beam width; 1 is greedy.

Returns:

Scored candidates per word, best first.

Return type:

list[list[tuple[str, float]]]

indicate.transliterator.model_for(pair)[source]

Return the cached model for a pair, creating it if needed.

Creating it does not load weights; that happens on first decode.

Parameters:

pair (Pair) – The direction wanted.

Returns:

The shared model object for that pair.

Return type:

Seq2SeqModel

indicate.transliterator.clear_models()[source]

Drop every loaded model. Intended for tests.

Return type:

None

Transliteration API

The public transliteration API: one function, any supported direction.

indicate.transliterate(text) detects the script, picks the language pair, and runs the default engine chain. Everything else is a keyword:

transliterate("राजशेखर चिंतालपति")                    # auto-detected Hindi
transliterate("ਰਵਿ ਸ਼ਰਮਾ", source="punjabi")
transliterate("नमस्ते", n=3)                          # n-best
transliterate("मुंबई", engine="model")                # no table
transliterate("मुंबई", engine=["lookup", "llm"])      # table intercepts the LLM

Text is split on spaces and each word is resolved independently, which is how the local model was trained and what lets a chain answer different words from different backends. Reassembly preserves order, so a mixed result reads back in the order it was given.

indicate.api.transliterate(text, *, source=None, target='english', engine=None, n=1, beam=None, reranker=None, llm=None, **llm_kwargs)[source]

Transliterate one text.

Parameters:

  • text (str) – Source-language text.

  • source (str | None) – Source language; None detects it from text.

  • target (str) – Target language.

  • engine (Sequence[str] | str | None) – Backend names in order, or None for ("lookup", "model").

  • n (int) – Number of candidates. 1 returns a single string.

  • beam (int | None) – Beam width override for the model backend.

  • reranker (Reranker | None) – Optional LM re-ranker.

  • llm (IndicLLMTransliterator | None) – An existing IndicLLMTransliterator to reuse.

  • **llm_kwargs (Any) – Provider settings for the llm backend.

Return type:

str | list[str]

Propagates UnsupportedPairError from transliterate_batch() when the direction is unsupported or undetectable; it is not raised here.

Returns:

A str when n == 1; a list of up to n candidates otherwise.

Raises:
Parameters:
Return type:

str | list[str]

indicate.api.transliterate_batch(texts, *, source=None, target='english', engine=None, n=1, beam=None, reranker=None, llm=None, **llm_kwargs)[source]

Transliterate many texts at once.

Batching is what makes the local model usable: every word that reaches the decoder across every input is decoded in a single pass.

Parameters:

  • texts (Sequence[str]) – Source-language texts.

  • source (str | None) – Source language; None detects it from the texts.

  • target (str) – Target language.

  • engine (Sequence[str] | str | None) – Backend names in order, or None for ("lookup", "model").

  • n (int) – Candidates per input. 1 returns one string each.

  • beam (int | None) – Beam width override for the model backend.

  • reranker (Reranker | None) – Optional LM re-ranker, applied to model candidates only.

  • llm (IndicLLMTransliterator | None) – An existing IndicLLMTransliterator to reuse.

  • **llm_kwargs (Any) – Provider settings for the llm backend.

Return type:

list[str] | list[list[str]]

Propagates UnsupportedPairError from resolve_pair() when the direction cannot be detected, or when no backend in the chain supports it. It is documented here rather than under Raises because nothing in this body raises it directly.

Returns:

a str each when n == 1, else a list of up to n candidates each.

Return type:

One result per input

Parameters:

Backends

Backends, and the order they are tried in.

A word is resolved by the first backend that will answer it. lookup reads a table, model decodes with the local seq2seq weights, llm asks a provider. The chain is ordinary data:

("lookup", "model")           # the default: table, then decode the tail
("model",)                    # what a benchmark must use
("lookup",)                   # table only -- is my corpus already covered?
("lookup", "llm")             # the table intercepts the paid path
("lookup", "model", "llm")    # decode locally, escalate only what is left

This generalizes the hit/miss/reassemble loop the transliterator already ran; it is not a new concept, just one that can now be spelled.

Declining is the whole protocol. Backend.resolve() returns one entry per word: a candidate list, or None/[] meaning I will not answer this. Treating an empty list as a decline is a real change – a decoder exception used to yield [] and become an empty string in the output, silently. Now it falls through, so ("lookup", "llm", "model") degrades instead of losing a word.

Declining is not the same as being unavailable. A backend that loaded its table and had no entry for a word has declined – ordinary, silent, and the whole point of a chain. A backend that could not obtain its asset at all is unavailable, and if every backend in the chain is unavailable the caller is about to receive an empty string that looks like an answer. That case raises BackendsUnavailableError instead. The distinction matters because engine=["lookup"] over an uncovered corpus declines every word by design, and must stay quiet.

Nothing heavy is imported until a backend actually runs. torch lives inside ModelBackend.resolve() and litellm inside LLMBackend.resolve(), so a text whose every word hits the table imports neither. That property is worth about 4x on cold start and is asserted in tests/test_lookup_bench.py.

indicate.engine.Candidates

One romanization and its score. Scores are comparable only within a backend.

alias of list[tuple[str, float]]

indicate.engine.DEFAULT_ENGINE: tuple[str, ...] = ('lookup', 'model')

The chain used when a caller does not choose one.

indicate.engine.KNOWN = frozenset({'llm', 'lookup', 'model'})

Valid backend names, re-exported so callers validate against one set.

indicate.engine.AUTHORITATIVE = 0.0

Score attached to an answer from a backend that does not rank.

A table or an LLM returns exactly one candidate, so this value never orders anything within a word. It exists so phrase assembly works: beam scores are length-normalized log-probs and therefore negative, and 0.0 sits above them, which keeps a single-candidate word from dragging a phrase down. It is not a probability and must never be compared across backends.

exception indicate.engine.BackendsUnavailableError[source]

Bases: RuntimeError

No backend in the chain could obtain what it needed to answer anything.

Raised rather than returning "", which is indistinguishable from a legitimate empty answer and exits zero.

class indicate.engine.Backend(*args, **kwargs)[source]

Bases: Protocol

Something that can answer some words and decline the rest.

name: str
unavailable: bool

Set by resolve when the backend could not obtain its asset at all.

resolve(words)[source]

Answer what it can.

Parameters:

words (Sequence[str]) – Words still unresolved, in order.

Returns:

candidates best-first, or None to decline.

Return type:

One entry per word

class indicate.engine.LookupBackend(pair)[source]

Bases: object

Answers from the packaged word table. No torch, no network.

Parameters:

pair (Pair)

name = 'lookup'
property table

The loaded table, or None when this pair ships none.

resolve(words)[source]

Return the table’s answer for each word it knows.

Parameters:

words (Sequence[str]) – Words to look up.

Returns:

A single authoritative candidate per hit, None per miss.

Return type:

list[Candidates | None]

class indicate.engine.ModelBackend(pair, *, beam=5, reranker=None)[source]

Bases: object

Decodes with the local seq2seq weights.

Parameters:
name = 'model'
resolve(words)[source]

Decode every word in one batch.

Parameters:

words (Sequence[str]) – Words to decode.

Returns:

Candidates per word; None for any word the decoder dropped, so it can fall through instead of becoming an empty string.

Return type:

list[Candidates | None]

class indicate.engine.LLMBackend(pair, *, transliterator=None, group_size=25, **client_kwargs)[source]

Bases: object

Asks a provider. Deduplicates first, because this one costs money.

Parameters:
name = 'llm'
client()[source]

The provider client, constructed on first use.

resolve(words)[source]

Transliterate the distinct words in as few requests as possible.

A failure declines the whole group rather than raising, so a chain like ("lookup", "llm", "model") falls back to local decoding when the network or the API key is not there.

Parameters:

words (Sequence[str]) – Words to transliterate.

Returns:

One authoritative candidate per word the provider answered.

Return type:

list[Candidates | None]

indicate.engine.normalize_engine(engine)[source]

Coerce an engine argument to a validated tuple of backend names.

Parameters:

engine (Sequence[str] | str | None) – A backend name, a sequence of them, or None for the default.

Returns:

The chain, in order.

Raises:

ValueError – If a name is not a known backend, or the chain is empty.

Return type:

tuple[str, …]

indicate.engine.build(engine, pair, *, beam=5, reranker=None, llm=None, **llm_kwargs)[source]

Construct the backends for a chain, without running any of them.

Backends this pair has no support for are dropped rather than failing, so ("lookup", "model") still works for a language that ships no table. If that leaves nothing, the direction is genuinely unsupported and saying so is better than falling through to a backend the caller did not ask to pay for.

Parameters:

  • engine (Sequence[str] | str | None) – Backend names in order, or None for the default.

  • pair (Pair) – The direction being transliterated.

  • beam (int) – Beam width for the model backend.

  • reranker (Reranker | None) – Optional re-ranker for the model backend.

  • llm (IndicLLMTransliterator | None) – An existing LLM client to reuse.

  • **llm_kwargs (Any) – Provider settings for the LLM backend.

Returns:

Constructed backends, in chain order.

Raises:

UnsupportedPairError – If no backend in the chain supports this direction.

Return type:

list[Backend]

indicate.engine.resolve_words(words, backends)[source]

Run the chain: each backend sees only what the previous ones declined.

Parameters:

  • words (Sequence[str]) – Words to resolve, in order.

  • backends (Sequence[Backend]) – The chain.

Returns:

One candidate list per word, aligned to words. A word every backend declined gets [].

Raises:

BackendsUnavailableError – If there were words, nothing was resolved, and every backend reported itself unavailable – i.e. the caller is about to get an empty string not because the answer is empty but because nothing was able to run.

Return type:

list[Candidates]

Encoder

class indicate.encoder.Encoder(vocab_size, embedding_dim, enc_units)[source]

Bases: Module

LSTM encoder: embedding -> LSTM.

Returns the full output sequence (for attention) and the final (hidden, cell) state used to initialise the decoder.

Parameters:

  • vocab_size (int)

  • embedding_dim (int)

  • enc_units (int)

forward(x)[source]

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

Parameters:

x (Tensor)

Return type:

tuple[Tensor, tuple[Tensor, Tensor]]

Decoder

class indicate.decoder.Decoder(vocab_size, embedding_dim, dec_units)[source]

Bases: Module

LSTM decoder with Luong (dot-product) attention.

Mirrors the original Keras model: the attention query is a linear projection of the target embedding (not the recurrent state), attention is unscaled dot-product over the encoder outputs (Keras Attention with use_scale=False), and the attention context is concatenated with the embedding before the LSTM.

forward works for both the full target sequence (training, teacher forcing) and a single step (autoregressive inference) by carrying state across calls.

Parameters:

  • vocab_size (int)

  • embedding_dim (int)

  • dec_units (int)

forward(inputs, encoder_outputs, state=None, src_mask=None)[source]

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

Parameters:

  • inputs (Tensor)

  • encoder_outputs (Tensor)

  • state (tuple[Tensor, Tensor] | None)

  • src_mask (Tensor | None)

Return type:

tuple[Tensor, tuple[Tensor, Tensor]]

LLM backend

LLM-based transliteration for Indic languages using LiteLLM.

class indicate.llm_indic.IndicLLMTransliterator(source_lang, target_lang, provider=None, model=None, api_key=None, temperature=0.3, cache_examples=True)[source]

Bases: object

LLM-based transliterator for Indic languages.

Parameters:

  • source_lang (str) – Source language (e.g., ‘hindi’, ‘tamil’)

  • target_lang (str) – Target language (e.g., ‘english’)

  • provider (str | None) – LLM provider (openai, anthropic, etc.). Auto-detected if not provided.

  • model (str | None) – Specific model to use. Uses provider defaults if not provided.

  • api_key (str | None) – API key. Uses environment variables if not provided.

  • temperature (float) – LLM temperature for consistency (lower = more consistent).

  • cache_examples (bool) – Whether to cache generated few-shot examples.

INDIC_LANGUAGES: ClassVar[dict[str, dict[str, str]]] = {'bengali': {'iso': 'bn', 'native': 'বাংলা', 'script': 'bengali'}, 'english': {'iso': 'en', 'native': 'English', 'script': 'latin'}, 'gujarati': {'iso': 'gu', 'native': 'ગુજરાતી', 'script': 'gujarati'}, 'hindi': {'iso': 'hi', 'native': 'हिन्दी', 'script': 'devanagari'}, 'kannada': {'iso': 'kn', 'native': 'ಕನ್ನಡ', 'script': 'kannada'}, 'malayalam': {'iso': 'ml', 'native': 'മലയാളം', 'script': 'malayalam'}, 'marathi': {'iso': 'mr', 'native': 'मराठी', 'script': 'devanagari'}, 'odia': {'iso': 'or', 'native': 'ଓଡ଼ିଆ', 'script': 'odia'}, 'punjabi': {'iso': 'pa', 'native': 'ਪੰਜਾਬੀ', 'script': 'gurmukhi'}, 'sanskrit': {'iso': 'sa', 'native': 'संस्कृतम्', 'script': 'devanagari'}, 'tamil': {'iso': 'ta', 'native': 'தமிழ்', 'script': 'tamil'}, 'telugu': {'iso': 'te', 'native': 'తెలుగు', 'script': 'telugu'}, 'urdu': {'iso': 'ur', 'native': 'اردو', 'script': 'arabic'}}
DEFAULT_MODELS: ClassVar[dict[str, str]] = {'anthropic': 'claude-3-opus-20240229', 'cohere': 'command-r-plus', 'google': 'gemini-pro', 'openai': 'gpt-5.4-mini'}
generate_few_shot_examples(num_examples=5)[source]

Generate few-shot transliteration examples for the language pair.

Parameters:

num_examples (int) – Number of examples to generate.

Returns:

List of dictionaries with ‘source’ and ‘target’ keys.

Return type:

list[dict[str, str]]

transliterate(text, use_few_shot=True, num_examples=5)[source]

Transliterate text from source language to target language.

Parameters:

  • text (str) – Text to transliterate.

  • use_few_shot (bool) – Whether to use few-shot examples.

  • num_examples (int) – Number of few-shot examples to use.

Returns:

Transliterated text.

Raises:

RuntimeError – If the LLM transliteration call fails.

Return type:

str

default_max_tokens_for(texts)[source]

Estimate max output tokens for transliterating texts as a group.

Parameters:

texts (list[str])

Return type:

int

build_group_messages(texts, examples=None)[source]

Build chat messages for transliterating a numbered group of texts.

Shared by the synchronous transliterate_batch and the async Batch-API path (indicate.batch) so both produce identical prompts.

Parameters:
Return type:

list[dict[str, str]]

transliterate_batch(texts, batch_size=10, use_few_shot=True)[source]

Transliterate multiple texts efficiently.

Parameters:

  • texts (list[str]) – List of texts to transliterate.

  • batch_size (int) – Number of texts to process in one API call.

  • use_few_shot (bool) – Whether to use few-shot examples.

Returns:

List of transliterated texts.

Return type:

list[str]

Batch transliteration

Batch-mode LLM transliteration via provider Batch APIs (LiteLLM).

The synchronous IndicLLMTransliterator issues one litellm.completion call per request. For transliterating the millions of unique tokens in an electoral roll, that is slow and expensive. This module routes the work through a provider’s asynchronous Batch API instead (~50% cheaper), with checkpointing and resume.

Because a batch can take up to 24h to finish, submit and collect are separate steps; transliterate_tokens_batched() is a convenience driver that submits then polls to completion. State is durable, so a killed process resumes from the checkpoint rather than resubmitting finished work:

  • checkpoint_path – a JSONL file of resolved {"token", "translit"} pairs (append-only; the durable result map). Kept dependency-free (no pandas/pyarrow).

  • checkpoint_path + ".batchstate.json" – in-flight batch ids and the custom_id -> [tokens] mapping needed to align results back to tokens.

Provider support is LiteLLM’s batch support: openai, azure, vertex_ai, bedrock, vllm – not native Anthropic. To use Claude in batch mode, go through Bedrock (provider="bedrock", model "anthropic.claude-sonnet-4-6") or add a native Anthropic messages.batches adapter later. The default provider is openai.

The cheapest request is the one you do not send. Before submitting anything, submit_transliteration_batches() runs the local part of engine – everything before "llm" – and writes what it answers straight to the checkpoint. Measured on 1M rows of the Punjab roll with the default ("lookup", "llm"):

unique tokens to resolve

46,902

1,497

batch requests submitted

1,877

60

That is 96.8% of unique tokens, which is the number that matters here – a batch API deduplicates, so token frequency buys nothing. It is this high because the roll shares a vocabulary with the corpus the table was built from; general text would see much less.

engine=("lookup", "model", "llm") goes further and decodes the table’s misses locally, submitting only what both decline. engine=("llm",) submits everything.

indicate.batch.DEFAULT_BATCH_ENGINE: tuple[str, ...] = ('lookup', 'llm')

Answer from the table first, submit only what it declines.

class indicate.batch.BatchJob(batch_id, input_file_id, custom_id_to_tokens, status='submitted', output_file_id=None)[source]

Bases: object

One submitted batch (a provider batch id + the tokens it covers).

Parameters:
batch_id: str
input_file_id: str
custom_id_to_tokens: dict[str, list[str]]
status: str = 'submitted'
output_file_id: str | None = None
class indicate.batch.BatchState(provider, model, source_lang, target_lang, group_size, temperature, use_few_shot, jobs=<factory>, submitted_at=None)[source]

Bases: object

Durable record of an in-flight transliteration run.

Parameters:
provider: str
model: str
source_lang: str
target_lang: str
group_size: int
temperature: float
use_few_shot: bool
jobs: list[BatchJob]
submitted_at: float | None = None
indicate.batch.submit_transliteration_batches(tokens, source_lang, target_lang, *, checkpoint_path, provider=None, model=None, api_key=None, group_size=25, completion_window='24h', use_few_shot=True, temperature=0.3, max_requests_per_batch=50000, engine=('lookup', 'llm'))[source]

Submit unique tokens to the provider Batch API. Does not block.

Already-resolved tokens (present in the checkpoint) and duplicates/blanks are skipped. Returns the persisted BatchState.

Backends in engine before "llm" run locally first; whatever they answer is written straight to the checkpoint and never submitted. Pass engine=("llm",) to send everything to the provider.

Parameters:
Return type:

BatchState

indicate.batch.collect_transliteration_batches(checkpoint_path, *, transliterator=None)[source]

Poll in-flight batches once and append any completed results.

Returns (all_done, resolved_map). A group whose result count does not match the request, or whose batch failed, is left out of resolved_map (the driver requeues such tokens). Safe to call repeatedly.

Parameters:
Return type:

tuple[bool, dict[str, str]]

indicate.batch.transliterate_tokens_batched(tokens, source_lang, target_lang, *, checkpoint_path, provider=None, model=None, api_key=None, group_size=25, completion_window='24h', use_few_shot=True, temperature=0.3, poll_interval=60.0, max_wait=None, requeue_passes=2, max_requests_per_batch=50000, engine=('lookup', 'llm'))[source]

Submit tokens in batch mode and poll to completion; return token->translit.

Resumable: if a batch is already in flight for checkpoint_path this skips submission and resumes polling. Tokens whose group output was malformed are requeued one-per-request (up to requeue_passes). If max_wait elapses with batches still running, returns what is resolved so far – rerun later to resume.

engine decides how much is answered before anything is submitted. The default reads the packaged table first; ("lookup", "model", "llm") also decodes locally, and ("llm",) submits everything.

Parameters:
Return type:

dict[str, str]

Reranking

class indicate.rerank.Reranker(words, alpha=0.9, order=3, k=1.0)[source]

Bases: object

Re-rank beam candidates by interpolating model score with a char LM.

Following IndicXlit (Aksharantar, EMNLP Findings 2023), the top-k beam hypotheses are re-scored as F = alpha * model_score + (1 - alpha) * lm. Here the LM is a character n-gram model over the training-side English romanizations (so it generalizes to unseen names, unlike a word unigram), with both scores length-normalized to keep them comparable.

Parameters:
lm_logprob(word)[source]

Length-normalized char-n-gram log-probability (add-k smoothed).

Parameters:

word (str)

Return type:

float

best(candidates)[source]

Return the candidate text maximizing the interpolated score.

candidates are (text, model_score) where model_score is the beam’s length-normalized log-probability.

Parameters:

candidates (list[tuple[str, float]])

Return type:

str

Indic text utilities

Script detection and text-shape helpers for Indic input.

The language, script and alias tables these functions used to carry are now in indicate.languages; there had been four overlapping copies. These are the thin, text-facing wrappers around them.

indicate.indic_utils.detect_indic_script(text)[source]

Auto-detect the dominant script of a string.

Parameters:

text (str) – Text to analyze.

Returns:

Script name, "latin" for mostly-ASCII text, or None.

Return type:

str | None

indicate.indic_utils.detect_language_from_script(text)[source]

Guess the language of a string from its script.

Parameters:

text (str) – Text to analyze.

Returns:

Language name, or None if the script maps to none.

Return type:

str | None

indicate.indic_utils.is_indic_script(script)[source]

Report whether a script name is an Indic script.

Parameters:

script (str) – Script name.

Returns:

True if Indic.

Return type:

bool

indicate.indic_utils.validate_indic_language_pair(source, target)[source]

Report whether at least one side of a pair is Indic.

Parameters:

  • source (str) – Source language or script name.

  • target (str) – Target language or script name.

Returns:

True if either side is Indic.

Return type:

bool

indicate.indic_utils.normalize_text_for_transliteration(text)[source]

Normalize Indic text for better transliteration.

Parameters:

text (str) – Input text.

Returns:

Normalized text.

Return type:

str

indicate.indic_utils.split_mixed_script_text(text)[source]

Split text containing multiple scripts into segments.

Parameters:

text (str) – Text potentially containing multiple scripts.

Returns:

List of (text_segment, script) tuples.

Return type:

list[tuple[str, str]]

indicate.indic_utils.get_language_info(language)[source]

Get detailed information about a language.

Parameters:

language (str) – Language name.

Returns:

Dictionary with language information.

Return type:

dict

File utilities

Safe file handling utilities for transliteration operations.

class indicate.file_utils.OutputFormat[source]

Bases: object

Output format definitions and handlers.

TEXT = 'text'
JSON = 'json'
VALID_FORMATS: ClassVar[list[str]] = ['text', 'json']
class indicate.file_utils.TransliterationResult(line_number, input_text, output_text, source_lang, target_lang, confidence='unknown', error=None, processing_time=None)[source]

Bases: object

Represents a single transliteration result with metadata.

Parameters:

  • line_number (int)

  • input_text (str)

  • output_text (str)

  • source_lang (str)

  • target_lang (str)

  • confidence (str)

  • error (str | None)

  • processing_time (float | None)

to_dict()[source]

Convert transliteration result to dictionary format.

Returns:

Dictionary representation suitable for JSON serialization.

Return type:

dict[str, Any]

class indicate.file_utils.BatchProgress(total_lines, output_path)[source]

Bases: object

Tracks progress of batch transliteration operations.

Parameters:

  • total_lines (int)

  • output_path (Path)

results: list[TransliterationResult]
add_result(result)[source]

Add a transliteration result.

Parameters:

result (TransliterationResult)

save_progress()[source]

Save current progress to disk for recovery.

classmethod load_progress(progress_file)[source]

Load progress from disk.

Parameters:

progress_file (Path)

Return type:

BatchProgress | None

cleanup()[source]

Remove progress file after successful completion.

indicate.file_utils.validate_file_paths(input_path, output_path, *, create_dirs=True)[source]

Validate input and output file paths for safety.

Parameters:

  • input_path (Path | None) – Input file path.

  • output_path (Path | None) – Output file path.

  • create_dirs (bool) – Create a missing output directory as part of validating it. Pass False for --dry-run, which promises to write nothing and must not leave a directory behind to prove it thought about it.

Raises:

ValueError – If paths are invalid or dangerous.

Return type:

None

indicate.file_utils.create_backup(file_path)[source]

Create a backup of a file before overwriting.

Parameters:

file_path (Path) – File to backup.

Returns:

Path to backup file, or None if backup wasn’t needed.

Return type:

Path | None

indicate.file_utils.write_output_safely(results, output_path, output_format, source_lang, target_lang, atomic=True)[source]

Write transliteration results to file safely.

Parameters:

  • results (list[TransliterationResult]) – List of transliteration results.

  • output_path (Path) – Path to output file.

  • output_format (str) – Output format (text, json, csv).

  • source_lang (str) – Source language.

  • target_lang (str) – Target language.

  • atomic (bool) – Whether to use atomic writing.

Raises:

ValueError – If output_format is not a supported format.

Return type:

None

indicate.file_utils.read_input_file(input_path)[source]

Read input file safely, handling both text and JSON formats.

Parameters:

input_path (Path) – Path to input file.

Returns:

List of text lines to transliterate.

Return type:

list[str]

indicate.file_utils.check_resume_possibility(output_path)[source]

Check if a previous batch operation can be resumed.

Parameters:

output_path (Path) – Output file path.

Returns:

BatchProgress object if resume is possible, None otherwise.

Return type:

BatchProgress | None

Tokenizer and decoding utilities

class indicate.utils.CharTokenizer(word_index)[source]

Bases: object

Character-level tokenizer holding the word_index/index_word maps.

Loaded from the JSON files that were serialised by the original Keras Tokenizer so the vocabulary indices stay identical across the migration.

Parameters:

word_index (dict[str, int])

property vocab_size: int
indicate.utils.load_tokenizer(path)[source]

Load a character tokenizer from a Keras-serialised tokenizer JSON file.

Parameters:

path (str)

Return type:

CharTokenizer

indicate.utils.sequence_to_chars(tokenizer, sequence)[source]

Convert a sequence of indices back to characters, skipping padding (0).

Parameters:
Return type:

str

indicate.utils.batch_candidates(words, input_lang_tokenizer, target_lang_tokenizer, encoder, decoder, max_length_input, max_length_output, beam_width=1, mask_padding=False)[source]

Ranked candidates for many words at once (the batched decode engine).

Returns one ranked (text, score) list per input word, aligned to words (empty/OOV words -> []). Inputs are padded to max_length_input exactly as the single-word path, so outputs are identical — just faster.

Parameters:
Return type:

list[list[tuple[str, float]]]

Command-line interface

Command line interface.

One transliteration command, because the language and the backend are arguments rather than separate programs:

indicate transliterate "राजशेखर चिंतालपति"
indicate transliterate "ਰਵਿ ਸ਼ਰਮਾ" --from punjabi --engine lookup
indicate transliterate --input names.txt --output roman.txt --format json
indicate transliterate "नमस्ते" --engine lookup,llm --provider openai

This replaces hindi2english, punjabi2english and llm. The language used to come out of the command name – which meant feeding Gurmukhi to hindi2english silently produced garbage – and the backend came out of --lookup/--no-lookup, which could not express a chain of more than two.

class indicate.cli.EngineChain[source]

Bases: ParamType

A comma-separated backend chain, validated against the known names.

name: str = 'engine'

the descriptive name of this type

convert(value, param, ctx)[source]

Parse and validate lookup,model into a tuple.

Parameters:

  • value (Any) – The raw option text.

  • param (Parameter | None) – The Click parameter, for error reporting.

  • ctx (Context | None) – The Click context, for error reporting.

Returns:

The chain, in order.

Return type:

tuple[str, …]

indicate.cli.main()[source]

Entry point for the indicate console script.

Return type:

None

Languages and pairs

What this install can transliterate, and how it knows.

One registry replaces four overlapping tables that had grown up separately: IndicLLMTransliterator.INDIC_LANGUAGES plus three dicts inside indic_utils (script ranges, script-to-language, the Indic-script set). They agreed by maintenance rather than by construction, which is the kind of agreement that stops holding.

Two things live here and nothing else:

Languages – name, native spelling, script and ISO code, with the aliases callers actually type (hi, hin, pan).

Pairs – the (source, target) combinations a local seq2seq model exists for, and the files that model is made of. Replacing the old class-per-pair (HindiToEnglish, PunjabiToEnglish) with data means adding a language is a dict entry rather than a module.

Support is per backend, not global: ("tamil", "english") is answerable by an LLM and by nothing else on this machine. supports() says so, and UnsupportedPairError is raised rather than quietly falling through to a backend that costs money.

indicate.languages.BACKENDS = frozenset({'llm', 'lookup', 'model'})

Backends that can appear in an engine chain, in no particular order.

class indicate.languages.Language(name, native, script, iso)[source]

Bases: object

A language this package can name.

Parameters:
name

Canonical lowercase name, the key into LANGUAGES.

Type:

str

native

The language’s name in its own script.

Type:

str

script

Writing system, a key of SCRIPT_RANGES.

Type:

str

iso

ISO 639-1 code.

Type:

str

indicate.languages.ALIASES: dict[str, str] = {'ben': 'bengali', 'eng': 'english', 'guj': 'gujarati', 'hin': 'hindi', 'kan': 'kannada', 'mal': 'malayalam', 'mar': 'marathi', 'ori': 'odia', 'oriya': 'odia', 'pan': 'punjabi', 'pun': 'punjabi', 'san': 'sanskrit', 'tam': 'tamil', 'tel': 'telugu', 'urd': 'urdu'}

Three-letter and colloquial spellings people actually type.

indicate.languages.SCRIPT_RANGES: dict[str, tuple[int, int]] = {'arabic': (1536, 1791), 'bengali': (2432, 2559), 'devanagari': (2304, 2431), 'gujarati': (2688, 2815), 'gurmukhi': (2560, 2687), 'kannada': (3200, 3327), 'lao': (3712, 3839), 'malayalam': (3328, 3455), 'myanmar': (4096, 4255), 'odia': (2816, 2943), 'sinhala': (3456, 3583), 'tamil': (2944, 3071), 'telugu': (3072, 3199), 'thai': (3584, 3711), 'tibetan': (3840, 4095)}

Unicode blocks used to identify a script. Wider than LANGUAGES, because detection should be able to say “this is Thai” and then decline.

indicate.languages.SCRIPT_TO_LANGUAGE: dict[str, str] = {'arabic': 'urdu', 'bengali': 'bengali', 'devanagari': 'hindi', 'gujarati': 'gujarati', 'gurmukhi': 'punjabi', 'kannada': 'kannada', 'latin': 'english', 'malayalam': 'malayalam', 'odia': 'odia', 'tamil': 'tamil', 'telugu': 'telugu'}

Which language a script most likely encodes. Several are ambiguous – Devanagari carries Hindi, Marathi, Sanskrit and Nepali – so this is a default for auto-detection, never an assertion. Pass source= to override.

class indicate.languages.Pair(source, target, subdir, input_vocab, target_vocab, max_input, max_output)[source]

Bases: object

A direction a local seq2seq model exists for.

Parameters:

  • source (str)

  • target (str)

  • subdir (str)

  • input_vocab (str)

  • target_vocab (str)

  • max_input (int)

  • max_output (int)

source

Source language name.

Type:

str

target

Target language name.

Type:

str

subdir

Directory under indicate/data holding this pair’s tokenizers, weights and lookup table.

Type:

str

input_vocab

Source tokenizer filename.

Type:

str

target_vocab

Target tokenizer filename.

Type:

str

max_input

Longest source sequence the weights were trained for.

Type:

int

max_output

Longest output sequence to decode.

Type:

int

property key: tuple[str, str]

The (source, target) tuple this pair is registered under.

exception indicate.languages.UnknownLanguageError[source]

Bases: ValueError

A language name that is not in LANGUAGES or ALIASES.

exception indicate.languages.UnsupportedPairError[source]

Bases: ValueError

No backend in the requested engine can transliterate this direction.

indicate.languages.normalize(name)[source]

Resolve a language name, ISO code or alias to its canonical name.

Parameters:

name (str) – What the caller typed, e.g. "Hindi", "hi", "hin".

Returns:

The canonical lowercase name.

Raises:

UnknownLanguageError – If nothing matches.

Return type:

str

indicate.languages.detect_script(text)[source]

Identify the dominant script of a string.

Parameters:

text (str) – Any text.

Returns:

The script with the most characters, "latin" if the text is mostly ASCII, or None when neither applies.

Return type:

str | None

indicate.languages.detect(text)[source]

Guess the source language of a string from its script.

Parameters:

text (str) – Any text.

Returns:

A canonical language name, or None if the script is unrecognized or maps to no language this package names.

Return type:

str | None

indicate.languages.READY = 'ready'

What a backend can actually do on this machine, right now.

indicate.languages.status(source, target)[source]

Report what each backend can actually do for a direction, offline.

supports() answers “is this direction in scope for this backend”, which is what drives build(). This answers the different and more useful question a user asks: will it work right now. Reporting lookup as available when no table exists and none can be fetched is how the CLI came to advertise a backend that silently returned empty strings.

Parameters:

  • source (str) – Canonical source language name.

  • target (str) – Canonical target language name.

Returns:

Backend name to status, for the backends that support the direction.

Return type:

dict[str, str]

indicate.languages.supports(source, target, backend)[source]

Report whether one backend can transliterate one direction.

Parameters:

  • source (str) – Canonical source language name.

  • target (str) – Canonical target language name.

  • backend (str) – A member of BACKENDS.

Returns:

True if that backend covers the direction on this machine.

Return type:

bool

indicate.languages.supported(backend=None)[source]

Report every direction this install can transliterate, and by what.

Parameters:

backend (str | None) – Restrict to one backend, or None for all of them.

Returns:

(source, target) to the backends that cover it, in chain order.

Return type:

dict[tuple[str, str], tuple[str, …]]

indicate.languages.resolve_pair(source, target, text='')[source]

Normalize a requested direction, detecting the source if it was omitted.

Parameters:

  • source (str | None) – Source language, or None to detect from text.

  • target (str) – Target language.

  • text (str) – Sample text used only when source is None.

Returns:

The canonical (source, target).

Raises:

UnsupportedPairError – If source is None and detection fails. Guessing would silently transliterate Gurmukhi with a Devanagari model.

Return type:

tuple[str, str]

Lookup table

Fast word-level lookup: answer from a table, decode only the tail.

On the Punjab electoral roll a table built from data/punjabi.csv.gz answers 99.1% of token mass, so the decoder handles 0.9% of the work. Measured back to back on one machine (training/bench_lookup.py, warm filesystem cache; absolute values move with machine load, the ratios do not):

measurement

with lookup

model only

ratio

end-to-end on roll names

10,937 tok/s

258 tok/s

42x

cold start, all-hit input

0.10 s

0.44 s

4.4x

raw table read (zipf, cached)

16.9M/s

–

–

Cold start is the smaller ratio but the one users feel per invocation, and it is only 4.4x because a fully-hit input never imports torch. That property is load-bearing, not incidental: it is why the torch-importing modules are imported inside indicate.engine.ModelBackend.resolve() rather than at module scope, and tests/test_lookup_bench.py fails if anything moves them back.

Accuracy does not pay for that speed; it improves. On the Dakshina test words a table happens to contain, answering from the table and falling through to the model on the rest beats either alone – Punjabi 77.6% against the model’s 77.0%, Hindi 78.8% against 76.2%. The gap comes from training/build_lookup.py refusing to answer where the corpus has no majority, so ambiguity reaches the model instead of being resolved by a coin toss.

There is a reason the two are complementary rather than redundant. training/build_v2.py withholds from training every source word appearing in the evaluation sets, which removed ਕੌਰ, ਕੁਮਾਰ, ਦਾਸ, ਬਟਾਲਾ and ਗੁਰਦਾਸਪੁਰ – the most frequent tokens in the corpus. The model is weakest precisely where the table is strongest, which is why it scrambles ਨਗਰ into ganag.

A table carries a romanization convention. The roll table speaks the long-vowel style of its GPT-4o annotation (azaadi, apaar, ambee) where Dakshina prefers azadi, apar, ambi. Neither is wrong, but a string that mixes table hits with model misses could mix styles, so the convention header records which one a table speaks.

That worry turns out to be small, and training/seam_check.py is what measured it rather than assuming. On roll strings containing both a hit and a miss, the table and the model already produce the same string for 93.0% of hit words. On the rest the style gap between the hit half and the miss half is +0.0095 doubled vowels per character wider than in the all-model rendering of the same input – roughly one extra long vowel per 100 characters. The direction is not even consistent (ਕੁਮਾਰੀ: table kumaari, model kumari, but ਦਾਸ: table das, model daas), which is why it does not accumulate.

indicate.lookup.LOOKUP_FILE = 'lookup.tsv.gz'

Filename inside each per-language data directory.

indicate.lookup.FORMAT_VERSION = 1

Bump when the file layout changes incompatibly.

indicate.lookup.SHIPPED = frozenset({'hindi_to_english', 'punjabi_to_english'})

Directories a table may exist for. Asking for any other language returns None without touching the network – otherwise a caller transliterating, say, Tamil would pay a failed Hugging Face round trip to learn nothing.

Membership means “look for it”, not “it is there”. Use DOWNLOADABLE for the stronger claim.

indicate.lookup.DOWNLOADABLE: frozenset[str] = frozenset({})

Directories whose table can be fetched from the model repo.

Empty, and deliberately so. The tables are derived from data/hindi.csv.gz (which blends CC-BY-NC IIT Bombay pairs) and data/punjabi.csv.gz (from the IRB-restricted electoral-roll deposit), so they are not ours to redistribute. Build one locally in a few seconds:

uv run --group train python training/build_lookup.py --lang punjabi

Until this is non-empty, a fresh install has no lookup backend, and saying so here is what stops indicate.supported() advertising one. It also saves every fresh install a guaranteed-404 round trip per language per process. tests/e2e/test_hf_contract.py asserts that whatever is listed here is actually present in the repo.

indicate.lookup.split_edges(word)[source]

Split a surface word into its stripped prefix, table key, and suffix.

Parameters:

word (str) – A token as it appears in text.

Returns:

(prefix, key, suffix). The key is "" when nothing lookup-worthy remains, in which case the edges are empty too.

Return type:

tuple[str, str, str]

indicate.lookup.lookup_key(word)[source]

Return the table key for a surface word.

Build time and query time must agree exactly or every lookup misses, so both go through this one function.

Parameters:

word (str) – A token as it appears in text.

Returns:

The normalized key, or "" when nothing lookup-worthy remains.

Return type:

str

class indicate.lookup.Lookup(table, meta)[source]

Bases: object

An immutable word-level romanization table.

Parameters:
property convention: str

Romanization style this table speaks, e.g. "roll".

property meta: dict[str, str]

The table’s header fields.

get(word)[source]

Return the romanization for word, or None on a miss.

Edge punctuation and digit prefixes are put back. The key is built by stripping them, so returning the bare stored value silently deleted source text: ਸਿੰਘ, became singh, (ਸਿੰਘ) became singh, and the roll serial in 022-ਖੇਮਕਰਨ vanished. That happened on the default chain, because the table answers first.

Parameters:

word (str) – A token as it appears in text.

Returns:

The stored romanization with the original edges, or None.

Return type:

str | None

classmethod from_path(path)[source]

Read a table from a gzipped TSV file.

A table is optional infrastructure, so every failure disables it rather than raising: a missing file, a corrupt file, or a table built with a different normalizer, whose keys would silently never match.

Parameters:

path (Path) – Path to the lookup.tsv.gz file.

Returns:

The table, or None if it could not be used.

Return type:

Lookup | None

classmethod load(subdir)[source]

Load the table for a language directory, caching the result.

Parameters:

subdir (str) – Per-language directory under indicate/data, e.g. "punjabi_to_english".

Returns:

The table, or None when there is none to be had.

Return type:

Lookup | None

indicate.lookup.clear_cache()[source]

Forget every loaded table. Intended for tests.

Return type:

None

Key normalization

Canonical key normalization for gazetteer lookup.

Gazetteer keys must be produced identically when the corpus is built and when it is queried, so this module is the single definition of the ladder. Both the builder under gazetteer/ and the runtime import from here; nothing re-implements it.

Lookup keys come in two levels, tried in order:

LEVEL_EXACT

The surface form, minus invisible formatting noise. An exact hit means the string was literally observed, which keeps the corpus auditable.

LEVEL_CANONICAL

Unicode-canonical: one encoding per grapheme, ASCII digits. Lossless with respect to the word’s identity, so it can never merge two different words.

There is deliberately no lossy lookup level. An earlier design had one that folded vowel length, nasalization, gemination and the AA matra, on the theory that these are scribal noise. Ablating each component against the 3,833 Punjab electoral-roll surfaces with at least 100 occurrences showed otherwise – every component merged more distinct words than duplicate spellings:

component

merges

benign

harmful

vowel length

73

43

30

diphthong

62

25

37

drop nasal

83

6

77

drop addak

73

13

60

drop nukta

130

6

124

drop AA matra

198

23

175

ਬੁਟਾ/buta and ਬੂਟਾ/boota are different names; ਰਤਨ/ratan and ਰੱਤਨ/rattan are different names. A lookup key that merges them returns a confidently wrong romanization, which is worse than a miss.

So folding survives only as alias_candidate_key(), which proposes pairs of surface forms that might be spelling variants. A proposal becomes an alias only when source evidence agrees on the romanization – a data decision, not a Unicode one. This module therefore never invents a join.

indicate.normalize.NORMALIZER_VERSION = 2

Bump whenever a level’s output changes. Corpora record the version they were built with, so a mismatched reader can refuse to use stale keys.

indicate.normalize.EDGE_NOISE = '()[]{}<>.,;:!?"\'`/\\|-–—_*# 0123456789०१२३४५६७८९੦੧੨੩੪੫੬੭੮੯'

Administrative codes (“022-<name>”), brackets and stray punctuation cling to corpus tokens on both sides of a label; the name itself is what gets keyed.

indicate.normalize.strip_edge_noise(token)[source]

Remove leading and trailing punctuation and digit codes from a token.

Parameters:

token (str) – A raw corpus or query token.

Returns:

The token with edge noise removed; may be empty.

Return type:

str

indicate.normalize.gaz_key(token, *, level=1)[source]

Return the gazetteer lookup key for token.

Both levels are identity-preserving: two spellings share a key only when they are the same string up to encoding.

Parameters:

  • token (str) – A surface form, in Indic script or Latin.

  • level (int) – LEVEL_EXACT or LEVEL_CANONICAL.

Returns:

The normalized key, or "" for empty or whitespace-only input.

Raises:

ValueError – If level is not a defined lookup level. Level 2 was a lossy fold in an earlier revision and is rejected explicitly so callers cannot silently reacquire it.

Return type:

str

indicate.normalize.latin_form(text)[source]

Return the canonical Latin form of a romanization candidate.

Candidates from different sources have to be comparable before they can be voted on. This strips diacritics and punctuation but leaves spelling alone, so rāj and raj agree while kumaari and kumari still count as a genuine disagreement for adjudication to resolve.

Parameters:

text (str) – A candidate romanization.

Returns:

Lowercased ASCII letters, digits, spaces, apostrophes and hyphens, with whitespace collapsed. "" when nothing Latin survives, which is the signal that the candidate is not a romanization at all.

Return type:

str

indicate.normalize.alias_candidate_key(token)[source]

Return a lossy key grouping surface forms that might be variants.

This exists to generate alias proposals for the corpus builder – never to look anything up. Two tokens sharing this key are candidates for merging; whether they actually merge is decided by whether independent sources agree on their romanization. See the module docstring for the measured reason.

Parameters:

token (str) – A surface form, in Indic script or Latin.

Returns:

The folded proposal key, or "" for empty or whitespace-only input.

Return type:

str

Data resolution

Locate packaged data files, local-first then Hugging Face.

Model weights are too large for the wheel and the lookup tables derive from corpora with their own license and provenance constraints, so both are resolved the same way: use the file in indicate/data/ if a checkout provides it, otherwise download and cache it from the model repo.

One implementation shared by the model loader and the lookup table, rather than two copies that can drift.

indicate.resources.HF_REPO = 'soodoku/indicate'

Hugging Face repo holding per-language directories.

indicate.resources.DATA_DIR_ENV = 'INDICATE_DATA_DIR'

Environment variable naming a directory to look in before the package.

Without this an installed package could only ever read from site-packages/indicate/data/, while training/build_lookup.py writes into whatever checkout it was run from – so the documented “build your own lookup table” instruction ended at a file the package would never open, and the only way to finish it was to copy into site-packages by hand, which an upgrade discards. Point this at a directory laid out the same way:

$INDICATE_DATA_DIR/<subdir>/<file>

It covers weights as well as tables, because every loader resolves through local_data_path().

indicate.resources.local_data_path(subdir, rel)[source]

Return the local path for a data file, which may not exist.

Checks DATA_DIR_ENV first, then the packaged location.

Parameters:

  • subdir (str) – Per-language directory under indicate/data.

  • rel (str) – Path relative to that directory.

Returns:

An absolute filesystem path.

Return type:

str

indicate.resources.resolve_data(subdir, rel, *, repo='soodoku/indicate', revision='v0.7.0')[source]

Resolve a data file, downloading from Hugging Face if it is not local.

Parameters:

  • subdir (str) – Per-language directory under indicate/data.

  • rel (str) – Path relative to that directory.

  • repo (str) – Hugging Face repo id.

  • revision (str) – Repo revision to pin.

Returns:

A path to the file on disk.

Return type:

str

indicate.resources.try_resolve_data(subdir, rel, *, repo='soodoku/indicate', revision='v0.7.0')[source]

Resolve a data file, or return None when it cannot be found.

Used for optional assets, where absence should disable a feature rather than fail a transliteration.

Parameters:

  • subdir (str) – Per-language directory under indicate/data.

  • rel (str) – Path relative to that directory.

  • repo (str) – Hugging Face repo id.

  • revision (str) – Repo revision to pin.

Returns:

A path to the file, or None if it is neither local nor downloadable.

Return type:

str | None

Logging