Yasmin Moslem

Iterative Layer Pruning for Efficient Inference

Mon, 17 Nov 2025 00:00:00 +0000

Model pruning is a compression technique, that aims to remove redundant components without significantly compromising the model’s performance or accuracy. This process facilitates efficient deployment of complex models by making them smaller and faster.

Pruning is a hardware-agnostic compression approach. Unlike some other compression approaches such as quantisation, models resulting from structured pruning can be deployed on any modern GPU with similar performance gains. Moreover, pruning can be part of a sophisticated compression pipeline that incorporates other techniques such as quantisation and efficient fine-tuning (e.g. LoRA, QLoRA), which can lead to higher compression and efficiency levels.

In this article, we will cover some insights from our two papers about Iterative Layer Pruning at IWSLT 2025 [1] and WMT 2025 [2].

Iterative Layer Pruning

The process of Iterative Layer Pruning involves incrementally identifying and removing layers with minimal contribution to translation or generation quality, one layer at a time. The pruning process is usually followed by fine-tuning the resulting models on relevant training data to restore the translation quality. Moreover, knowledge distillation data from the baseline (teacher) model can be used to help the pruned (student) model to reach the quality of the teacher model.

Layer Importance Evaluation

We conduct layer importance evaluation by measuring translation performance without each layer. The process is as follows:

Remove one layer of the model.
Evaluate the model (chrF++).
Repeat for the rest of the layers.
Prune the least important layer (best chrF++ without it).
Repeat #1 to #4 until reaching the pruning target.

Evaluation Results

For translation from Czech to German (CES-DEU), pruning 8 layers and then fine-tuning the resulting model retains 98% of the translation quality (as measured by COMET), while achieving considerable speedup gains. Interestingly, for translation from English to Egyptian Arabic (ENG-ARZ), the model resulting from pruning up to 16 layers and then fine-tuning outperforms the Aya-Expanse-8B baseline for this language pair. Pruned models achieve up to ~2× speedup.

Knowledge Distillation

Knowledge Distillation aims at transferring knowledge from a larger model (teacher) to a smaller one (student). In “sequence-level” knowledge distillation, the student model is trained to generate sequences that match the teacher’s sequence outputs. Fine-tuning the pruned models on a combination of authentic and synthetic data (from Aya-Expanse-32B) improved the Czech to German (CES-DEU) translation quality, with the 24-layer pruned model nearly matching the performance of the Aya-Expanse-8B baseline.

Further performance gains

It is highly recommended to use an efficient inference engine such as vLLM, which outperforms inference with the Transformers framework. In both cases, pruned models demonstrate up to ~2× speedup.

Moreover, you can quantise the pruned models for further compression. For example, in our IWSLT 2025 paper, we applied QLoRA after pruning the models. However, notice that low-precision quantisation (e.g. 4-bit and 8-bit) requires special hardware (e.g. H100 or H200) to observe performance gains.

Questions and Answers

Q. Why iterative layer pruning instead of random or middle layer pruning?
- A. Iterative layer pruning relies on layer importance analysis. Hence, only the layers with minimal contribution to the output quality can be removed. In our experiments, iterative layer pruning achieves better results than middle layer pruning.
Q. Can we prune other parts of the model other than layers?
- A. There are two types of pruning, structured pruning and unstructured pruning. In structured pruning, you can remove whole layers, attention heads, or other entire computational blocks, while in unstructured pruning, you can remove individual weights from a neural network. While unstructured pruning can achieve higher compression rates, it requires specialised hardware for efficient deployment. On the contrary, models resulting from structure pruning can be deployed on standards GPUs.
Q. Is it better to fine-tune the baseline model before pruning?
- A. If your task, domain, or language is very different from the distribution of the baseline model, it is better to fine-tune the baseline model first. Otherwise, you can prune the baseline directly. On the contrary, fine-tuning after pruning is always required to restore the quality of the baseline model.
Q. Can we fine-tune after each layer pruning step?
- A. We experimented with both fine-tuning after each layer pruning step, and after a number of pruned layers. In both cases, there was no difference than just fine-tuning once after the end of the pruning process. This might be because of overfitting resulted from several fine-tuning passes.
Q. Can the same approach be applied to encoder-decoder models?
- A. Yes, we applied this iterative layer pruning approach to both a decoder-only model, Aya-Expanse, and an encoder-decoder model, Qwen2-Audio. However, for encoder-decoder models, we observed that only pruning the decoder leads to better overall performance.

GitHub Repository

References

Efficient Speech Translation through Model Compression and Knowledge Distillation (Moslem, IWSLT 2025)
Iterative Layer Pruning for Efficient Translation Inference (Moslem et al., WMT 2025)

Adaptive Translation and Terminology with Large Language Models

Wed, 10 Jan 2024 00:00:00 +0000

Large-scale language models (LLMs) have shown interesting capabilities of in-context learning, where they learn to replicate certain input-output text generation patterns, without further fine-tuning. By feeding an LLM at inference time with a prompt that consists of a list of translation pairs, it can then simulate the domain, terminology, and style characteristics.

Adaptive Machine Translation with Large Language Models

First preprint: January 2023

Peer-reviewed: EAMT 2023

Abstract:

Consistency is a key requirement of high-quality translation. It is especially important to adhere to pre-approved terminology and adapt to corrected translations in domain-specific projects. Machine translation (MT) has achieved significant progress in the area of domain adaptation. However, real-time adaptation remains challenging. Large-scale language models (LLMs) have recently shown interesting capabilities of in-context learning, where they learn to replicate certain input-output text generation patterns, without further fine-tuning. By feeding an LLM at inference time with a prompt that consists of a list of translation pairs, it can then simulate the domain and style characteristics. This work aims to investigate how we can utilize in-context learning to improve real-time adaptive MT. Our extensive experiments show promising results at translation time. For example, GPT-3.5 can adapt to a set of in-domain sentence pairs and/or terminology while translating a new sentence. We observe that the translation quality with few-shot in-context learning can surpass that of strong encoder-decoder MT systems, especially for high-resource languages. Moreover, we investigate whether we can combine MT from strong encoder-decoder models with fuzzy matches, which can further improve translation quality, especially for less supported languages. We conduct our experiments across five diverse language pairs, namely English-to-Arabic (EN-AR), English-to-Chinese (EN-ZH), English-to-French (EN-FR), English-to-Kinyarwanda (EN-RW), and English-to-Spanish (EN-ES).

@inproceedings{moslem-etal-2023-adaptive,
    title = "Adaptive Machine Translation with Large Language Models",
    author = "Moslem, Yasmin  and
      Haque, Rejwanul  and
      Kelleher, John D.  and
      Way, Andy",
    booktitle = "Proceedings of the 24th Annual Conference of the European Association for Machine Translation",
    month = jun,
    year = "2023",
    address = "Tampere, Finland",
    publisher = "European Association for Machine Translation",
    url = "https://aclanthology.org/2023.eamt-1.22/",
    pages = "227--237",
}

Domain Terminology Integration into Machine Translation: Leveraging Large Language Models

First preprint: October 2023

Peer-reviewed: WMT 2023

Abstract:

This paper discusses the methods that we used for our submissions to the WMT 2023 Terminology Shared Task for German-to-English (DE-EN), English-to-Czech (EN-CS), and Chinese-to-English (ZH-EN) language pairs. The task aims to advance machine translation (MT) by challenging participants to develop systems that accurately translate technical terms, ultimately enhancing communication and understanding in specialised domains. To this end, we conduct experiments that utilise large language models (LLMs) for two purposes: generating synthetic bilingual terminology-based data, and post-editing translations generated by an MT model through incorporating pre-approved terms. Our system employs a four-step process: (i) using an LLM to generate bilingual synthetic data based on the provided terminology, (ii) fine-tuning a generic encoder-decoder MT model, with a mix of the terminology-based synthetic data generated in the first step and a randomly sampled portion of the original generic training data, (iii) generating translations with the fine-tuned MT model, and (iv) finally, leveraging an LLM for terminology-constrained automatic post-editing of the translations that do not include the required terms. The results demonstrate the effectiveness of our proposed approach in improving the integration of pre-approved terms into translations. The number of terms incorporated into the translations of the blind dataset increases from an average of 36.67% with the generic model to an average of 72.88% by the end of the process. In other words, successful utilisation of terms nearly doubles across the three language pairs.

@inproceedings{moslem-etal-2023-domain,
    title = "Domain Terminology Integration into Machine Translation: Leveraging Large Language Models",
    author = "Moslem, Yasmin  and
      Romani, Gianfranco  and
      Molaei, Mahdi  and
      Kelleher, John D.  and
      Haque, Rejwanul  and
      Way, Andy",
    booktitle = "Proceedings of the Eighth Conference on Machine Translation",
    month = dec,
    year = "2023",
    address = "Singapore",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.wmt-1.82/",
    doi = "10.18653/v1/2023.wmt-1.82",
    pages = "902--911",
}

Fine-tuning Large Language Models for Adaptive Machine Translation

First preprint: December 2023

Published as: thesis chapter

Abstract:

This paper presents the outcomes of fine-tuning Mistral 7B, a general-purpose large language model (LLM), for adaptive machine translation (MT). The fine-tuning process involves utilising a combination of zero-shot and one-shot translation prompts within the medical domain. The primary objective is to enhance real-time adaptive MT capabilities of Mistral 7B, enabling it to adapt translations to the required domain at inference time. The results, particularly for Spanish-to-English MT, showcase the efficacy of the fine-tuned model, demonstrating quality improvements in both zero-shot and one-shot translation scenarios, surpassing Mistral 7B’s baseline performance. Notably, the fine-tuned Mistral outperforms ChatGPT “gpt-3.5-turbo” in zero-shot translation while achieving comparable one-shot translation quality. Moreover, the zero-shot translation of the fine-tuned Mistral matches NLLB 3.3B’s performance, and its one-shot translation quality surpasses that of NLLB 3.3B. These findings emphasise the significance of fine-tuning efficient LLMs like Mistral 7B to yield high-quality zero-shot translations comparable to task-oriented models like NLLB 3.3B. Additionally, the adaptive gains achieved in one-shot translation are comparable to those of commercial LLMs such as ChatGPT. Our experiments demonstrate that, with a relatively small dataset of 20,000 segments that incorporate a mix of zero-shot and one-shot prompts, fine-tuning significantly enhances Mistral’s in-context learning ability, especially for real-time adaptive MT.

@article{moslem-etal-2023-fine-tuning-llms,
      title={Fine-tuning Large Language Models for Adaptive Machine Translation}, 
      author={Yasmin Moslem and Rejwanul Haque and Andy Way},
      year={2023},
      eprint={2312.12740},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2312.12740}, 
}

GitHub Repository

Language Modelling Approaches to Adaptive Machine Translation

First preprint: January 2024

Published as: PhD thesis (DCU)

Abstract:

Consistency is a key requirement of high-quality translation. It is especially important to adhere to pre-approved terminology and adapt to corrected translations in domain-specific projects. Machine translation (MT) has achieved significant progress in the area of domain adaptation. However, in-domain data scarcity is common in translation settings, due to the lack of specialised datasets and terminology, or inconsistency and inaccuracy of available in-domain translations. In such scenarios where there is insufficient in-domain data to fine-tune MT models, producing translations that are consistent with the relevant context is challenging. While real-time adaptation can make use of smaller amounts of in-domain data to improve the translation on the fly, it remains challenging due to supported context limitations and efficiency constraints. Large language models (LLMs) have recently shown interesting capabilities of in-context learning, where they learn to replicate certain input-output text generation patterns, without further fine-tuning. Such capabilities have opened new horizons for domain-specific data augmentation and real-time adaptive MT. This work attempts to address two main relevant questions: 1) in scenarios involving human interaction and continuous feedback, can we employ language models to improve the quality of adaptive MT at inference time? and 2) in the absence of sufficient in-domain data, can we use pre-trained large-scale language models to improve the process of MT domain adaptation?

@article{moslem-2024-adaptive-mt-llms,
      title={Language Modelling Approaches to Adaptive Machine Translation, {PhD} thesis}, 
      author={Yasmin Moslem},
      year={2024},
      eprint={2401.14559},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2401.14559}, 
}

References

Adaptive Machine Translation with Large Language Models (Moslem et al., EAMT 2023)
Domain Terminology Integration into Machine Translation: Leveraging Large Language Models (Moslem et al., WMT 2023)
Fine-tuning Large Language Models for Adaptive Machine Translation (Moslem et al., thesis chapter 2023)
Language Modelling Approaches to Adaptive Machine Translation (Moslem, PhD thesis, DCU 2024)

Purely Synthetic Bilingual Data for Machine Translation?

Mon, 12 Dec 2022 00:00:00 +0000

In-domain data scarcity is common in translation settings, due to the lack of specialized datasets and terminology, or inconsistency and inaccuracy of available in-domain translations. You might be familiar to such situation when there is a big translation project, but there is only a tiny in-domain translation memory, or no translation memory at all. In the absence of sufficient domain-specific data required to fine-tune machine translation (MT) systems, adhering to the domain terminology and client’s style can be challenging. Recently, there has been a considerable advancement in training large language models, not only for English, but also for diverse languages. Among autoregressive language models, trained to predict the next word in a sequence, are BLOOM, GPT-3, and GPT-J. The question is: can we use these large language models to generate more domain-specific bilingual data?

Method

Interestingly, when you feed such large language models with an in-domain sentence, they can generate more synthetic sentences, that simulate the domain and linguistic characteristics of the authentic sentence. In the research “Domain-Specific Text Generation for Machine Translation” (Moslem et al., 2022), we investigated the feasibility of this domain-specific text generation technique, when either no or limited bilingual in-domain dataset is available. We proposed a novel approach to domain adaptation leveraging state-of-the-art pre-trained language models to generate huge amounts of synthetic bilingual in-domain data with the goal of improving translation of in-domain texts. The process can be summarised in three simple steps:

1. Text generation (target)

Generate target-side synthetic sentences using a large pre-trained language model.

When there is a small in-domain translation memory, you can use each target sentence as a prompt to generate text, that simulates the domain characteristics of the authentic in-domain data. If there is no translation memory at all, you can first forward-translate the source text to be translated, or a portion of it, using the baseline MT model.

2. Back-translation (source)

Back-translate the synthetic target-side sentences into source language.

Combining the idea of in-domain text generation with back-translation, you can generate huge amounts of synthetic bilingual in-domain data, for both use cases.

3. Mixed fine-tuning

Fine-tune the baseline model, on a mix of synthetic and authentic data.

Finally, the baseline MT model should be fine-tuned using a combination of the synthetic bilingual in-domain dataset and a randomly sampled section of the original generic dataset.

Target Text Generation

This code snippet shows how to load the GPT-J language model. You can use some efficient loading techniques such float16 and low_cpu_mem_usage.

from transformers import GPTJForCausalLM, AutoTokenizer
import torch


tokenizer = AutoTokenizer.from_pretrained("EleutherAI/gpt-j-6B", padding_side="left")
tokenizer.pad_token = tokenizer.eos_token

model =  GPTJForCausalLM.from_pretrained("EleutherAI/gpt-j-6B",
                                         revision="float16",
                                         torch_dtype=torch.float16,
                                         low_cpu_mem_usage=True,
                                         cache_dir = "models_cache/",
                                         pad_token_id=tokenizer.eos_token_id)
model = model.half()
model = model.to("cuda")

Afterwards, you can use each target segment in the authentic in-domain dataset, to generate synthetic in-domain text. We use top-K and top-p sampling to generate diverse text sequences. Here, we set num_return_sequences to generate 5 sequences. Each sequence might include multiple sentences. You can then split these text sequences into several sentences using any sentence splitter.

target_segment = "I am an example sentence that talks about something very specialized!"
input_ids = tokenizer(target_segment, add_special_tokens=True, return_tensors="pt").input_ids.to("cuda")

sample_outputs = model.generate(input_ids,
                                do_sample=True, 
                                max_length=300, 
                                top_k=50, 
                                top_p=0.95, 
                                num_return_sequences=5,
                                early_stopping=True)

generated_text = tokenizer.batch_decode(sample_outputs, skip_special_tokens=True)

The quality of the language model is important. Here, you can see examples of generated text, that is both linguistically correct, and factually accurate.

In March 2020, India ordered the countrywide shut down of all non-essential economic activities due to the spreading COVID-19 pandemic.

While the overall worldwide economic impact of COVID-19 will only be realized through the end of 2020 and the recovery phase in 2021, it is clear that certain parts of the world have been severely impacted.

Sometimes, the generated text can be linguistically correct; however, numbers or names might be inaccurate.

Antiviral drugs are approved for pregnant women and should be considered for children younger than XX years, although some are still being investigated.

Scientists have found some species of unicorn in Amazon rainforests.

If there are only small mistakes, the generated synthetic data can be still used. Obviously, the better the quality of the text generated by the language model, the better the quality we can expect when fine-tuning the baseline MT model on this synthetic data.

Back-Translation

Now, we have the target side of our new in-domain dataset. To generate the source side, use back-translation, into the other language direction. For back-translation, you can either train another MT model yourself, or use pre-trained models such as OPUS models. Optionally, you can convert OPUS models to the CTranslate2 formate with quantisation to enhance efficiency.

Basically, both the source and target sides of our new large in-domain dataset, consist of synthetic data. The target side is generated by a language model, while the source text is generated by back-translation, in the other language direction.

Mixed Fine-Tuning

Now, it’s time to apply mixed fine-tuning to the baseline model.

In other words, continue training our baseline model on a mix of (a) the synthetic bilingual in-domain dataset we obtained from the two previous steps, and (b) a randomly sampled portion of the original generic dataset. In our experiments, we oversampled the synthetic in-domain dataset, by a 9x ratio.

Mixed fine-tuning (Chu et al., 2017)

To apply oversampling, we employed the dataset weights feature in OpenNMT-tf. If you are using OpenNMT-py or OpenNMT-tf, you can find more details in this tutorial on mixed fine-tuning of MT models.

Results

In both scenarios, our proposed method achieves significant improvements, demonstrated by both automatic and human evaluations. As expected, Setup 1 (when there is a tiny bilingual dataset) reveals better results than Setup 2 (where there is no bilingual dataset at all). Still, models resulted from both setups outperform the baseline model.

Evaluation results on the in-domain test set, TICO-19

Conclusion

Previously, synthetic data for machine translation had been created either on the source-side only (forward-translation) or the target side only (back-translation). In some cases, researchers replaced a few words from the source and/or the target with synonyms or similar words. The assumption was that “relevant” monolingual data was available, which was not always the case!

In real-life scenarios, things can get more complex. Usually, there is insufficient human-produced data to train or fine-tune high-quality MT systems. Production-level projects can be so specialised, while mining crawled monolingual datasets is inefficient and not necessarily helpful.

This research work generates brand new synthetic data on both the source and target sides. It employs large language models to put together coherent sentences similar to those to be translated in the current project. Then, the new synthetic data can be used to fine-tune production-level MT systems for domain-specific scenarios. Feel free to check out our paper, Domain-Specific Text Generation for Machine Translation.

Download Scripts

You can download our scripts and configuration files at GitHub. If you have applied the method and/or have questions, please let me know.

Citation

@inproceedings{moslem-etal-2022-domain,
    title = "Domain-Specific Text Generation for Machine Translation",
    author = "Moslem, Yasmin  and
      Haque, Rejwanul  and
      Kelleher, John  and
      Way, Andy",
    booktitle = "Proceedings of the 15th biennial conference of the Association for Machine Translation in the Americas (Volume 1: Research Track)",
    month = sep,
    year = "2022",
    address = "Orlando, USA",
    publisher = "Association for Machine Translation in the Americas",
    url = "https://aclanthology.org/2022.amta-research.2",
    pages = "14--30",
    abstract = "Preservation of domain knowledge from the source to target is crucial in any translation workflow. It is common in the translation industry to receive highly specialized projects, where there is hardly any parallel in-domain data. In such scenarios where there is insufficient in-domain data to fine-tune Machine Translation (MT) models, producing translations that are consistent with the relevant context is challenging. In this work, we propose a novel approach to domain adaptation leveraging state-of-the-art pretrained language models (LMs) for domain-specific data augmentation for MT, simulating the domain characteristics of either (a) a small bilingual dataset, or (b) the monolingual source text to be translated. Combining this idea with back-translation, we can generate huge amounts of synthetic bilingual in-domain data for both use cases. For our investigation, we use the state-of-the-art Transformer architecture. We employ mixed fine-tuning to train models that significantly improve translation of in-domain texts. More specifically, in both scenarios, our proposed methods achieve improvements of approximately 5-6 BLEU and 2-3 BLEU, respectively, on the Arabic-to-English and English-to-Arabic language pairs. Furthermore, the outcome of human evaluation corroborates the automatic evaluation results",
}

Translation Auto-suggestions: What do Linguists Think?

Mon, 24 Oct 2022 00:00:00 +0000

Translation auto-suggestion and auto-completion are among the important features that can help translators better utilize Machine Translation (MT) systems. In a Computer-Aided Translation (CAT) environment, a translator can make use of the MT word auto-suggestion feature as follows:

typing a few words, or clicking a word in a proposed MT translation, a list of suggestions is displayed.
selecting one of the word suggestions from the list, the rest of the translation is modified accordingly.

In a user survey we designed and distributed via social media networks, we asked participants whether they thought an MT word-level auto-suggestions feature could be helpful, and provided a simple definition and an illustrative image. If their answer was “Yes”, the respondent was asked to specify a reason. By the time of writing this article, we received 41 responses to our survey. While we do not believe this survey is enough to justify introducing an auto-suggestions feature into every MT system, it can be an indicator as to why some users think such a feature could be helpful.

To answer the question, “Which of the following best describes you?” 46.3% (19) of the respondents chose “Translator/Linguist”, 31.7% (13) selected “NLP Engineer/Researcher”, and the rest 22% (9) were other “MT Users”, not included in the two aforementioned categories.

Among the respondents to the survey, 90.2% (37) answered “Yes” to the question “In general, do you believe that a word-level auto-suggestions feature is helpful?” The figure below shows the breakdown of answers to the question, “Why do you believe that a word-level auto-suggestions feature can be helpful?” taking into consideration those who answered “No” to the previous question.

Out of the 37 persons who believed a word-level auto-suggestions feature can be helpful, 40.5% (15) of the respondents specified that it can give them some inspiration. This answer is specifically interesting as it is not constrained by time-saving benefits; hence, it focusses more on effectiveness rather than efficiency. The respondent that answered with “Other” mentioned that it allows them to look for alternative senses or phrasings, especially when they suspect the initial translation is bad, and referred to this as “human in the loop”.

Respondents were allowed to give extra comments; among the notable comments were:

I think word-level suggestions can be a useful feature, particularly when the target language can have several translations of a single source word.

Word-level suggestions can be helpful, but sometimes you end up spending a lot of time figuring out if the MT suggestion is a valid translation in that context. So, I’m not really sure yet how I feel about it.

It’s useful, as long as it’s seen as a suggestion, and not inserted in the target where the translator is typing.

Among the respondents who answered “For me, it is easier or faster than typing”, comments included:

Though most of the time; the suggestions are lousy.

I don’t think it gives me inspiration as I mostly need it for structures, not single words.

Auto-suggestion does not have to come from machine translation. History is much more useful.

The last comment above might be referring to the fact that in some CAT tools, auto-suggestions can also include glossary terms, and translation memory sub-segments, which encourages further research efforts to investigate methods to enhance leveraging and interaction between various translation resources in human-in-the-loop environments.

We hope this survey will inspire future user studies to look deeper into how diverse users of MT and CAT tools prefer to utilize certain features, such as auto-suggestions, and the value they seek. More aspects should be taken into consideration such as language pairs, translation workflows, and user interfaces. This can help improve these features to better support linguists and other MT users and boost their productivity as well as translation quality.

Citation

@inproceedings{moslem2022-autosuggest,
    title = "Word-Level Auto-Completion: What can we achieve out of the box?",
    author = "Moslem, Yasmin  and
      Haque, Rejwanul  and
      Way, Andy",
    booktitle = "In Proceedings of the Seventh Conference on Machine Translation",
    month = Dec,
    year = "2022",
    address = "Abu Dhabi, UAE",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/2210.12802",
}

Machine Translation Robustness

Wed, 28 Sep 2022 00:00:00 +0000

Let’s talk briefly about the concept of “Robustness” of neural machine translation (NMT) systems. While robustness should be emphasized when building any NMT system, even high-resource languages with plenty of data can still face linguistic challenges.

What does NMT “Robustness” mean?

It simply means that a certain NMT engine can handle a specific linguistic feature, found in the input to be translated. This feature might not be naturally occurring in the training data. Among examples of linguistic features we want our NMT model to be robust while translating are: domain terminology, proper names, number formats, text case, misspellings, code switching (between two languages), and untranslatables such as tags, email addresses, etc.

How can we improve NMT robustness?

The first step to machine translation robustness is defining the issues that your model frequently encounters when translating a certain type of text. This step is underestimated, and in my opinion it is a sign of the maturity of production-level operations.

This goes beyond numerical human evaluation, and moves a step further towards defining specific types of issues. In simple words, human evaluators are asked to mention a clear reason why they think a translation should be ranked as 3 out of 5, for example. At the beginning, they might be provided with common lists, but they should also have the option to add more issues, that can be integrated into that list later. Such explanations should not be vague; they should be precise in a way allowing MT engineers to fix these issues. Problematic words should be marked; sometimes the track-changes feature is used. The main question is: What is the most critical issue in this translation?

How can we improve NMT “Robustness”?

In the findings of the WMT2020 Robustness Shared Task, under the “Common Trends” section, Specia et al. (2020) stated: “Participating systems were trained following a standard recipe, i) using big-transformer models, ii) boosting performance with tagged back-translation, iii) continued training with filtered data and in-domain data (where available), iv) ensembling different models to obtain further improvements.”

In this sense, data augmentation techniques can be helpful, and then you integrate this new data into the NMT system, via combining or fine-tuning.

As training a new system frequently might not be feasible, it is common in some companies to temporarily apply on-the-fly find-replace operations on translations, until the next training is possible. Some researchers also suggest making such on-the-fly handling easier by injecting the training data with certain placeholders, to be able to replace later. To apply this, in a portion of the training data, natural tags (html, xml, long numbers, etc.) are replaced with pseudo-tags (e.g. <t0>, <t1>, <t2>, …). These pseudo-tags should be also added as user_defined_symbols to the SentencePiece model (cf. SPM options). At inference time, it is easy to define and replace these tags with untraslatables during pre-processing and post-processing steps. On a relevant note, activating the SentencePiece option split_digits helps with copying longer numbers without intervention, while the option byte_fallback sometimes helps with irregular characters in the training data.

References

MQM - Multidimensional Quality Metrics (Lommel et al., 2013)
Training Neural Machine Translation to Apply Terminology Constraints (Dinu et al., 2019)
Improving Robustness in Real-World Neural Machine Translation Engines (Gupta et al., 2019)
How Should Markup Tags Be Translated? (Hanneman and Dinu, 2020)
Evaluating Robustness to Input Perturbations for Neural Machine Translation (Niu et al., 2020)
Findings of the WMT 2020 Shared Task on Machine Translation Robustness (Specia et al., 2020)
Business Critical Errors: A Framework for Adaptive Quality Feedback (Stewart et al., 2022)
Improve MT for Search with Selected Translation Memory using Search Signals (Zhang 2022)

Machine Translation Models: How to Build and Deploy

Tue, 15 Mar 2022 00:00:00 +0000

This is a Neural Machine Translation (NMT) tutorial with OpenNMT-py and relevant tools. It covers data preprocessing, model training, evaluation, and deployment. The tutorial was put together as part of a mentorship activity I organised in 2022.

Fundamentals

Data Processing (notebook, repository)
NMT Model Training with OpenNMT-py (notebook)
Translation/Inference with CTranslate2 (code)
MT Evaluation with BLEU (tutorial, repository)
Simple Web UI (tutorial, repository)

Advanced Topics

Running TensorBoard with OpenNMT (tutorial)
Low-Resource Neural Machine Translation (tutorial)
Domain Adaptation with Mixed Fine-tuning (tutorial)
Overview of Domain Adaptation Techniques (tutorial)
Multilingual Machine Translation (tutorial)
Using Pre-trained NMT models with CTranslate2 (tutorial)

Mixed Fine-Tuning - Domain Adaptation That Works!

Thu, 06 Jan 2022 00:00:00 +0000

Training a robust generic model is an interesting task. However, when you want to customize your Machine Translation model to observe the terminology and style of a certain domain or client, Domain Adaptation comes to life. In previous posts, we discussed several approaches to Domain Adaptation. In this post, we are going to concentrate on a very effective approach called Mixed Fine-Tuning, originally proposed by Chu et al., 2017.

Regular fine-tuning of an NMT model usually consists of two steps:

Building a baseline NMT model, e.g. a generic model.
Continuing training the baseline NMT model on an in-domain dataset.

However, fine-tuning in this way can lead to “catastrophic forgetting”, i.e. the model overfits the in-domain data, starts forgetting the information learned from the baseline data, and loses generalization. So in practice, when you compare its quality to the baseline model, the in-domain model would give a better BLEU score and human evaluation for translation of sentences very similar to the in-domain training dataset, but worse BLEU score for out-of-domain sentences or even new in-domain sentences.

Solution: Unlike plain fine-tuning, in the Mixed Fine-Tuning approach (Chu et al., 2017), you randomly sample a portion from the generic data you used to train the baseline model, and use it during the fine-tuning step along with the in-domain dataset. Over-sampling the in-domain data is the main trick.

The training procedure of the Mixed Fine-tuning approach is as follows:

Train a baseline NMT model on out-of-domain data until convergence.
Continue training the NMT baseline model on a mix of in-domain and out-of-domain data (by oversampling the in-domain data) until convergence.

In NMT tools, such as OpenNMT and MarianMT, dataset weights can be used to replicate over-sampling.

Example:

Dataset Counts:

Generic Dataset: 1,000,000 sentences
In-domain Dataset: 100,000 sentences

Use weights 1:10 so that the training takes 1 sentence from the bigger generic dataset, and 10 sentences from the smaller in-domain dataset.

Generic Dataset: 1
In-domain Dataset: 10

In this example, we sequentially sample 1 example from the “Generic Dataset” and 10 examples from the “In-domain Dataset” and so on. By giving the “In-domain Dataset” a higher weight, the model can learn the style and terminology from the in-domain dataset while still be able to generalize, i.e. output high-quality translations for out-of-domain sentences.

Setting the dataset weights differs from one tool to another. In OpenNMT-py, dataset weights are set as numbers as in the aforementioned example. In OpenNMT-tf, dataset weights are set as ratios.

Further notes on the Mixed Fine-tuning approach (feel free to experiment with something different, though!)

The approach works well for in-domain datasets between 50k and 500k. For very small in-domain datasets, this approach might not work well; for bigger in-domain datasets, you might want to try different weights; and for very big in-domain datasets, you can just use the in-domain dataset only, but enrich it with missing aspects like shorter sentences, if needed.
If your baseline training data is too big, you randomly extract 10 times the size of the in-domain data.
If both the generic and in-domain data are available before training the baseline, we build the vocabulary and SentencePiece models on all datasets, both generic and in-domain datasets.
During fine-tuning, we extract a dev/validation dataset from the in-domain dataset only.
After fine-tuning, we use two test datasets, one that we used for the out-of-domain baseline, and one extracted from the in-domain dataset, to make sure the model works in both cases.
To alleviate “catastrophic forgetting” on generic data, consider averaging the baseline model with the fine-tuned model.

Among the advantages of the Mixed Fine-tuning approach is that this fine-tuned NMT in-domain model still works well on both unseen in-domain data and general/out-of-domain data. Moreover, the approach can be fully automated (e.g. for various clients) once you verify it for your use cases.

It is worth mentioning that we have successfully applied the Mixed Fine-Tuning approach, proposed by Chu et al. (2017), in production-level scenarios in the industry. We also employed it in a number of our Domain Adaptation and Low-Resource NMT papers such as Haque et al. (2020) in combination with other approaches, through which we achieved the first place at ICON 2020 shared task, as well as Moslem et al. (2022) where we used synthetic in-domain data.

References

Axelrod, A., He, X., & Gao, J. (2011). Domain Adaptation via Pseudo In-Domain Data Selection. Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, 355–362. https://aclanthology.org/D11-1033
Chinea-Ríos, M., Peris, Á., & Casacuberta, F. (2017). Adapting Neural Machine Translation with Parallel Synthetic Data. Proceedings of the Second Conference on Machine Translation, 138–147. https://doi.org/10.18653/v1/W17-4714
Chu, C., Dabre, R., & Kurohashi, S. (2017). An Empirical Comparison of Domain Adaptation Methods for Neural Machine Translation. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 385–391. https://doi.org/10.18653/v1/P17-2061
Freitag, M., & Al-Onaizan, Y. (2016). Fast Domain Adaptation for Neural Machine Translation. In arXiv [cs.CL]. arXiv. http://arxiv.org/abs/1612.06897
Haque, R., Moslem, Y., & Way, A. (2020). Terminology-Aware Sentence Mining for NMT Domain Adaptation: ADAPT’s Submission to the Adap-MT 2020 English-to-Hindi AI Translation Shared Task. Proceedings of the 17th International Conference on Natural Language Processing (ICON): Adap-MT 2020 Shared Task, 17–23. https://aclanthology.org/2020.icon-adapmt.4
Kobus, C., Crego, J., & Senellart, J. (2017). Domain Control for Neural Machine Translation. Proceedings of Recent Advances in Natural Language Processing, 372–378. http://arxiv.org/abs/1612.06140
Luong, M.-T., & Manning, C. 2015. Stanford neural machine translation systems for spoken language domains. Proceedings of the 12th International Workshop on Spoken Language Translation: Evaluation Campaign, 76–79. https://aclanthology.org/2015.iwslt-evaluation.11
Moslem, Y., Haque, R., Kelleher, J., & Way, A. (2022). Domain-Specific Text Generation for Machine Translation. Proceedings of the 15th Biennial Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track), 14–30. https://aclanthology.org/2022.amta-research.2
Moslem, Y. (2024). Language Modelling Approaches to Adaptive Machine Translation. In arXiv [cs.CL]. arXiv. http://arxiv.org/abs/2401.14559
Saunders, D. (2022). Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Survey. Journal of Artificial Intelligence Research, 75, 351–424. https://doi.org/10.1613/jair.1.13566
Sennrich, R., Haddow, B., & Birch, A. (2016a). Controlling Politeness in Neural Machine Translation via Side Constraints. Proceedings of the 2016 Conference of the North AMerican Chapter of the Association for Computational Linguistics: Human Language Technologies, 35–40. https://doi.org/10.18653/v1/N16-1005

Sennrich, R., Haddow, B., & Birch, A. (2016b). Improving Neural Machine Translation Models with Monolingual Data. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 86–96. https://doi.org/10.18653/v1/P16-1009

Notes on Multilingual Machine Translation

Sat, 04 Dec 2021 00:00:00 +0000

Multilingual NMT is featured by its scalability between any number of languages, instead of having to build individual models. MNMT systems are also desirable because training models with data from diverse language pairs might help a low-resource language acquire extra knowledge from other languages. Moreover, MNMT systems tend to generalize better due to exposure to diverse languages, leading to improved translation quality compared to bilingual NMT systems. This particular phenomenon is known as translation Transfer Learning or Knowledge Transfer (Dabre et al., 2020).

Tips for training multilingual NMT models

Building a many-to-one MT system that translates from several languages to one language is simple: just merge all the datasets. Here is an illustration of how your data should look like. Afterwards, it is recommended to shuffle your dataset.

Source	Target
<ar> Thank you very much	شكرا جزيلا
<es> Thank you very much	Muchas gracias
<fr> Thank you very much	Merci beaucoup
<hi> Thank you very much	आपका बहुत बहुत धन्यवाद
<ar> आपका बहुत बहुत धन्यवाद	شكرا جزيلا
<en> आपका बहुत बहुत धन्यवाद	Thank you very much
<es> आपका बहुत बहुत धन्यवाद	Muchas gracias
<fr> आपका बहुत बहुत धन्यवाद	Merci beaucoup
<ar> Muchas gracias	شكرا جزيلا
<en> Muchas gracias	Thank you very much
<fr> Muchas gracias	Merci beaucoup
<hi> Muchas gracias	आपका बहुत बहुत धन्यवाद
<en> شكرا جزيلا	Thank you very much
<es> شكرا جزيلا	Muchas gracias
<fr> شكرا جزيلا	Merci beaucoup
<hi> شكرا جزيلا	आपका बहुत बहुत धन्यवाद
<ar> Merci beaucoup	شكرا جزيلا
<en> Merci beaucoup	Thank you very much
<es> Merci beaucoup	Muchas gracias
<hi> Merci beaucoup	आपका बहुत बहुत धन्यवाद

There are a few important points to take into consideration while building multilingual models:

If the data is clearly unbalanced, like you have 75 million sentences for Spanish and 15 million sentences for Portuguese, you have to balance it; otherwise, you would end up with a system that translates Spanish better than Portuguese. This technique is called over-sampling (or up-sampling). The obvious way to achieve it in NMT toolkits is through giving weights to your datasets. In this example, the Spanish dataset can take the weight of 1 while the Portuguese can take the weight of 5 because your Spanish dataset is 5 times larger than your Portuguese dataset.
Some papers suggest adding a special token to the start of each sentence. For example, you can start Spanish sentences with the token <es> and Portuguese sentences with the token <pt>. In this case, you will have to add these tokens to your SentencePiece model through the option --user_defined_symbols. However, some researchers believe this step is optional.
Multilingual NMT models are more useful for low-resource languages than they are for rich-resource languages. Still, low-resource languages that share some linguistic characteristics with other rich-resource languages can benefit from coexistence in one multilingual model. In this sense, multilingual NMT can be considered one of “Transfer Learning” approaches (Tras et al., 2021 and Ding et al., 2021).
Languages that do not share the same alphabet cannot achieve the same linguistic benefits from a multilingual NMT model. Still, researchers investigate approaches like transliteration to increase knowledge transfer between languages that belong to the same language family, but use different alphabets. For example, using this transliteration trick, my Indic-to-English multilingual NMT model can translate from 10 Indic languages to English.
Integrating other data augmentation approaches like Back-Translation can still be useful.

Using pre-trained NMT models

What about pre-trained multilingual NMT models like mBART (Liu et al., 2020) and M2M-100 (Fan et al., 2020); when to use them? The simple answer is, for low-resource languages (e.g. a few thousands to a few millions, up to 15m), using directly or fine-tuning mBART can give better results. For high-resource languages, training a baseline model from scratch can outperform mBART. Then, applying mixed fine-tuning (Chu et al., 2017) on this new baseline using in-house data can even achieve better gains in terms of Machine Translation quality. Check this code snippet if you would like to try mBART. You can also convert M2M-100 model to the CTranslate2 format for better efficiency as explained here.

References:

Low-Resource Neural Machine Translation

Sat, 25 Sep 2021 00:00:00 +0000

Developing Neural Machine Translation (NMT) models for low-resource languages is a viral topic, both in the industry and academia. In this tutorial, we are going to discuss tagged back-translation as one of the most effective and efficient approaches to training more robust models. Tagged back-translation is not only useful for low-resource languages, but also for other scenarios of data sparsity.

Table of Contents:

Tagged Back-Translation
Lower-Casing vs. True-Casing
Sub-wording to Avoid Unknowns
Shared Vocab vs. Separate Vocab
Crawled Data
Transfer Learning
References

Tagged Back-Translation

This approach aims at augmenting the available parallel training data with synthetic data that represent the purpose of the model. Several researchers, including Edunov et al. (2018) and Caswell et al. (2019), have proved that tagged back-translation is very helpful when training NMT models for low-resource languages. Moreover, it can be helpful for rich-resource languages through enriching datasets with specific linguistic features.

Assuming we want to train an English-to-Hindi NMT mode, the Tagged Back-Translation data augmentation technique depends on the following steps:

For an English-to-Hindi model, train another Hindi-to-English model (i.e. in the other direction), using publicly available data from OPUS;
Select monolingual data in Hindi publicly available (e.g. at OSCAR), which must have domains and linguistic features similar to the potential texts to be translated;
Use the Hindi-to-English model to create a synthetic dataset, by translating the Hindi monolingual data into English. Note here that only the English side (the source for EN-HI) is MTed while the Hindi side (the target for EN-HI) is human-generated text;
Consider using one the available Quality Estimation tools such as TransQuest (Ranasinghe et al., 2020) or OpenKiwi (Kepler et al., 2019) to filter out back-translations of low quality;
Add a special tag like <BT> to the start of the MTed segments;
Build the vocabulary on all the data, both the original and the synthetic datasets;
Augment the original English-to-Hindi training dataset with the synthetic dataset;
Train a new English-to-Hindi model using the dataset generated from the previous step.

For low-resource languages like Hindi, Haque et al. (2020) showed that the technique works well with 1:1 synthetic to original data. Still, you can experiment with different portions, especially for language pairs of richer resources.

As demonstrated by Hoang et al. (2018), iterative back-translation for 2-3 runs can improve the quality further. Now, as you have a better Hindi-to-English model, back-translate English monolingual data to train a new version of the English-to-Hindi model. After that, use the new English-to-Hindi model to back-translate the same Hindi monolingual dataset you used for the first run to create a new version of the Hindi-to-English model. The idea here is that you are using a better model to translate the same monolingual data, i.e. without any increase or change, which should result in a better NMT model. Interestingly, you can use both NMT and phrase-based SMT models for back-translation, and then train or fine-tune your baseline NMT system in the required language direction.

Popel et al. (2010) explored the effect of Block-Backtranslation, where the training data are presented to the neural network in blocks of authentic parallel data alternated with blocks of synthetic data.

Lower-Casing vs. True-Casing

For low-resource languages, I prefer lower-casing the data. However, in real-life scenarios or if you are submitting a paper, you are usually required to produce the translation in the true-case. so you can train a truecaser or use sacreMoses’ truecaser for English.

Sub-wording to Avoid Unknowns

To avoid out-of-vocabulary, it is recommended to train your NMT model on subwords instead whole words. Subwording (e.g. BPE or unigram model) is recommended for any type of machine translation model, regardless of whether it is for a low-resource or rich-resource language pair. Among the most popular subwording tools is SentencePiece.

If you used <BT> for example as the back-translation token, you have to add it to the SentencePiece model through using the option --user_defined_symbols during training. The same option can be useful for adding any other special tokens found in your training data, such as tags and non-Latin numbers.

Consider also using the following SentencePiece options:

--input_sentence_size to determine maximum number of sentences the trainer loads. This number must be equal to the vocab size;
--shuffle_input_sentence to shuffle the dataset;
--split_by_number to split tokens by numbers (0-9); and
--byte_fallback to decompose unknown pieces into UTF-8 byte pieces.

Shared Vocab vs. Separate Vocab

If both the source and target share some vocabulary, e.g. similar languages and code switching, using shared vocabulary might help. Using shared vocabulary involve two steps:

Training a SentencePiece model on all datasets for both languages;
Using shared vocab instead of separate vocabs while training the NMT model.

Crawled Data

Currently, OPUS includes some datasets that are crawled from bilingual websites, and then the sentences are matched using multilingual similarity tools such as LASER, LabSE, and m-USE. However, according to Kreutzer et al. (2021) crawled datasets suffer from quality issues that can affect the quality of outcome NMT models. Hence, it is important to try filtering them before using, and maybe exclude them from initial baselines.

Transfer Learning

Instead of training a model from scratch, transfer learning can be applied. In this sense, you can use a multilingual model like mBART-50, M2M-100, or NLLB-200, and fine-tune it on your dataset. Moreover, unidirectional models can be used (e.g. OPUS). If your low-resource language is similar to languages supported by such models, it can benefit from shared linguistic features. Back-translation can be used here as well to augment the authentic dataset.

References

Caswell, I., Chelba, C., & Grangier, D. (2019). Tagged Back-Translation. Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers), 53–63. https://doi.org/10.18653/v1/W19-5206
Edunov, S., Ott, M., Auli, M., & Grangier, D. Understanding Back-Translation at Scale. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 489–500. https://doi.org/10.18653/v1/D18-1045
Gebauer, P., Bojar, O., Švandelík, V., & Popel, M. (2021). CUNI Systems in WMT21: Revisiting Backtranslation Techniques for English-Czech NMT. Proceedings of the Sixth Conference on Machine Translation, 123–129. https://aclanthology.org/2021.wmt-1.7
Haque, R., Moslem, Y., & Way, A. (2020). Terminology-Aware Sentence Mining for NMT Domain Adaptation: ADAPT’s Submission to the Adap-MT 2020 English-to-Hindi AI Translation Shared Task. Proceedings of the 17th International Conference on Natural Language Processing (ICON): Adap-MT 2020 Shared Task, 17–23. https://aclanthology.org/2020.icon-adapmt.4
Hoang, V. C. D., Koehn, P., Haffari, G., & Cohn, T. (2018). Iterative Back-Translation for Neural Machine Translation. Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, 18–24. https://doi.org/10.18653/v1/W18-2703
Kreutzer, J., Caswell, I., Wang, L., Wahab, A., van Esch, D., Ulzii-Orshikh, N., Tapo, A., Subramani, N., Sokolov, A., Sikasote, C., Setyawan, M., Sarin, S., Samb, S., Sagot, B., Rivera, C., Rios, A., Papadimitriou, I., Osei, S., Suarez, P. O., … Adeyemi, M. (2022). Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets. Transactions of the Association for Computational Linguistics, 10, 50–72. https://doi.org/10.1162/tacl_a_00447
Popel, M., Tomkova, M., Tomek, J., Kaiser, Ł., Uszkoreit, J., Bojar, O., & Žabokrtský, Z. (2020). Transforming machine translation: a deep learning system reaches news translation quality comparable to human professionals. Nature Communications, 11(1), 4381. https://doi.org/10.1038/s41467-020-18073-9
Ramírez-Sánchez, G., Zaragoza-Bernabeu, J., Bañón, M., & Rojas, S. O. (2020). Bifixer and Bicleaner: two open-source tools to clean your parallel data. 291–298. https://aclanthology.org/2020.eamt-1.31/
Sennrich, R., Haddow, B., & Birch, A. (2016). Improving Neural Machine Translation Models with Monolingual Data. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 86–96. https://doi.org/10.18653/v1/P16-1009

Web Interface for Machine Translation

Sun, 25 Jul 2021 00:00:00 +0000

Today, we will create a very simple Machine Translation (MT) Web Interface for OpenNMT-py, OpenNMT-tf and FairSeq models using CTranslate2 and Streamlit.

Previously, there were other tutorials on how to use a simple server and web interface with Flask. However, today’s tutorial is for those who want to create an ultra simple, quick demo.

We also aim at highlighting that CTranslate2 is now the way to go for serving OpenNMT models due to its exceptional performance. It is completely up to you to use it in a simple way like what we will do here, or to integrate it into a REST API for advanced uses.

So let’s start…

Table of Contents:

Objective: Simple Machine Translation Web Interface
Install Requirements
- Optional: Create and Activate a Virtual Environment
- Install Required Libraries
Convert Model to CTranslate2
- CTranslate2 Example
Create Your App
Next Steps
- Streamlit Components
- Deployment

Objective: Simple Machine Translation Web Interface

Our objective is to develop a simple web interface for Machine Translation like this one.

Install Requirements

Optional: Create and Activate a Virtual Environment

Install virtualenv:
```
pip3 install virtualenv
```
Create a virtual environment, e.g. myvenv:
```
virtualenv myvenv --python=python3
```
Activate the virtual environment:
```
source myvenv/web/bin/activate
```

Install Required Libraries

pip3 install ctranslate2 sentencepiece streamlit watchdog nltk

Convert Model to CTranslate2

CTranslate2 supports both OpenNMT-py and OpenNMT-tf models. As of version 2.0, it also supports FairSeq models. However, you need to convert your model to the CTranslate2 format before using it.

The following commands are simply copied from the CTranslate2 repository, and tested to make sure they are up-to-date. This example uses pre-trained Transformer English-German models. If you trained your own model, run the same commands on it instead.

For an OpenNMT-py model:

pip3 install OpenNMT-py

wget https://s3.amazonaws.com/opennmt-models/transformer-ende-wmt-pyOnmt.tar.gz
tar xf transformer-ende-wmt-pyOnmt.tar.gz

ct2-opennmt-py-converter --model_path averaged-10-epoch.pt --output_dir ende_ctranslate2 --quantization int8

For an OpenNMT-tf model:

pip3 install OpenNMT-tf

wget https://s3.amazonaws.com/opennmt-models/averaged-ende-ckpt500k-v2.tar.gz
tar xf averaged-ende-ckpt500k-v2.tar.gz

ct2-opennmt-tf-converter --model_path averaged-ende-ckpt500k-v2 --output_dir ende_ctranslate2 \
    --src_vocab averaged-ende-ckpt500k-v2/wmtende.vocab \
    --tgt_vocab averaged-ende-ckpt500k-v2/wmtende.vocab \
    --model_type TransformerBase \
    --quantization int8

For a FairSeq model:

ct2-fairseq-converterconverter --model_path $MODEL --data_dir dict --fixed_dictionary $DICT --output_dir $OUTPUT --quantization int8

As you can see, we used the option --quantization int8 to imporve both the size and the performance of the model.

CTranslate2 Python Sample

Let’s make sure that CTranslate2 works properly in our setup by running this Python code:

import ctranslate2
translator = ctranslate2.Translator("ende_ctranslate2/")
translator.translate_batch([["▁H", "ello", "▁world", "!"]])

Note: translate_batch() can take a list of sentences and translate them in batches, which would be very efficient. Here we are using only one sentence as an example for demonstration purposes.

You can also check this detailed example that opens a file and translates it with CTranslate2.

Create Your App

Test App

Let’s first create a small app to see how Streamlit works.

Create a file called test.py for example and add the following lines to it.

import streamlit as st

st.title("Upper My Text")

user_input = st.text_input("Write something and press Enter \
    to convert it to the UPPER case.")

if len(user_input) > 0:
    output = user_input.upper()
    st.info(output)

Launch your test app by opening the Terminal and running the following command.

streamlit run test.py

If everything works as expected, you should see something like this in your browser at the URL http://localhost:8501. Once you type a text and press Enter, the text will be printed in the UPPER case.

Translation App

Let’s now develop our translation web interface. Create a file called translate.py for example, and add the following to it.

import streamlit as st
import sentencepiece as spm
import ctranslate2
from nltk import sent_tokenize


def translate(source, translator, sp_source_model, sp_target_model):
    """Use CTranslate model to translate a sentence

    Args:
        source (str): Source sentences to translate
        translator (object): Object of Translator, with the CTranslate2 model
        sp_source_model (object): Object of SentencePieceProcessor, with the SentencePiece source model
        sp_target_model (object): Object of SentencePieceProcessor, with the SentencePiece target model
    Returns:
        Translation of the source text
    """

    source_sentences = sent_tokenize(source)
    source_tokenized = sp_source_model.encode(source_sentences, out_type=str)
    translations = translator.translate_batch(source_tokenized)
    translations = [translation[0]["tokens"] for translation in translations]
    translations_detokenized = sp_target_model.decode(translations)
    translation = " ".join(translations_detokenized)

    return translation


# [Modify] File paths here to the CTranslate2 SentencePiece models.
ct_model_path = "/path/to/the/ctranslate/model/directory"
sp_source_model_path = "/path/to/the/sentencepiece/source/model/file"
sp_target_model_path = "/path/to/the/sentencepiece/target/model/file"

# Create objects of CTranslate2 Translator and SentencePieceProcessor to load the models
translator = ctranslate2.Translator(ct_model_path, "cpu")    # or "cuda" for GPU
sp_source_model = spm.SentencePieceProcessor(sp_source_model_path)
sp_target_model = spm.SentencePieceProcessor(sp_target_model_path)


# Title for the page and nice icon
st.set_page_config(page_title="NMT", page_icon="🤖")
# Header
st.title("Translate")

# Form to add your items
with st.form("my_form"):
    # Textarea to type the source text.
    user_input = st.text_area("Source Text", max_chars=200)
    # Translate with CTranslate2 model
    translation = translate(user_input, translator, sp_source_model, sp_target_model)

    # Create a button
    submitted = st.form_submit_button("Translate")
    # If the button pressed, print the translation
    # Here, we use "st.info", but you can try "st.write", "st.code", or "st.success".
    if submitted:
        st.write("Translation")
        st.info(translation)

Note: Make sure you update the variables ct_model, sp_source_model, and sp_target_model with our own paths to the CTranslate2 model, and the SentencePiece source and target models.

Let’s launch our translator. Run the following command in the Terminal.

streamlit run translate.py

If everything works fine, you should see an output like this at the URL http://localhost:8501/

Try typing a sentence (in the same source language of your model) and press the button “Translate”. The translation should be printed as you see here!

Add Language Pairs

To give your visitor the option to select between multiple language pairs, you can add a dropdown menu like this one.

You can first change the paths part into a function:

def load_models(option):
    if option == "English-to-Japanese":
        ct_model_path = "path/to/your/ct_model"
        sp_source_model_path = "path/to/your/sp_source_model"
        sp_target_model_path = "path/to/your/sp_target_model"
    elif option == "Japanese-to-English":
        ct_model_path = "path/to/your/ct_model"
        sp_source_model_path = "path/to/your/sp_source_model"
        sp_target_model_path = "path/to/your/sp_target_model"
    
    translator = ctranslate2.Translator(ct_model_path)
    sp_source_model = spm.SentencePieceProcessor(sp_source_model_path)
    sp_target_model = spm.SentencePieceProcessor(sp_target_model_path)

    return translator, sp_source_model, sp_target_model

Then, you change the form to:

with st.form("my_form"):

    # Dropdown menu to select a language pair
    option = st.selectbox(
    "Select Language Pair",
    ("English-to-Japanese", "Japanese-to-English"))
    #st.write('You selected:', option)

    # Textarea to type the source text.
    user_input = st.text_area("Source Text", max_chars=200)

    # Load models
    translator, sp_source_model, sp_target_model = load_models(option)
    
    # Translate with CTranslate2 model
    translation = translate(user_input, translator, sp_source_model, sp_target_model)

    # Create a button
    submitted = st.form_submit_button("Translate")
    # If the button pressed, print the translation
    # Here, we use "st.info", but you can try "st.write", "st.code", or "st.success".
    if submitted:
        st.write("Translation")
        st.info(translation)

Full Code

I will be updating this repository with Python samples.

Next steps

Streamlit Components

Streamlit comes with more components. One of the most interesting NLP components you might want to check is spacy-streamlit

Deployment

You can deploy your app on any service of your choice. However, if you are looking for a free and easy option, consider using Heroku. For better performance, test your app with and without Streamlit’s caching option and see if it helps.

Thanks for reading! If you have questions or suggestions, feel free to contact me.