招聘简介

路教授现需招聘研究助理完成以下任务。本科、硕士均可。需要用python 做过NLP的相关经验。因为下面提到的NLLB_200是基于深度学习的package，所以最好有相关经验，比如用过BERT。

Tasks overview: I have a dataset of text data in multiple languages, I need to translate them into English by finetuning a deep learning algorithms. I need your assistance in studying how the following existing algorithms to translate a small sample (e.g. 10,000/100,000 observations). I will then learn how to use the algorithms from you and deploy them on the large dataset myself.

Task 1: Use the Fasttext algorithm for language detection:

https://pypi.org/project/fasttext-langdetect/

Task 2: Use the following algorithm to translate language into English using the pre-trained model

Algorithm: NLLB_200:

https://github.com/facebookresearch/fairseq/tree/nllb/

Additional relevant information:

https://mr-nlp.github.io/posts/2022/07/mt-hf-pipeline/

https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes

Task 3: Use Google translate in Bulk, use some of the translation to verify the quality of translation from pre-trained model

https://cloud.google.com/translate/docs/advanced/batch-translation

Task 4: Finetune NLLB_200 algorithms using the training data generated from Task 2 and Task 3. They are translations where NLLB_200 and Google Translate agree with each other.