Hallucination, where models generate fluent text unsupported by visual evidence, remains a major flaw in vision-language models and is particularly critical in sign language translation (SLT). In SLT, meaning depends on precise grounding in video, and gloss-free models are especially vulnerable because they map continuous signer movements directly into natural language without intermediate gloss supervision that serves as alignment. We argue that hallucinations arise when models rely on language priors rather than visual input. To capture this, we propose a token-level reliability measure that quantifies how much the decoder uses visual information. Our method combines feature-based sensitivity, which measures internal changes when video is masked, with counterfactual signals, which capture probability differences between clean and altered video inputs. These signals are aggregated into a sentence-level reliability score, providing a compact and interpretable measure of visual grounding. We evaluate the proposed measure on two SLT benchmarks (PHOENIX-2014T and CSL-Daily) with both gloss-based and gloss-free models. Our results show that reliability predicts hallucination rates, generalizes across datasets and architectures, and decreases under visual degradations. Beyond these quantitative trends, we also find that reliability distinguishes grounded tokens from guessed ones, allowing risk estimation without references; when combined with text-based signals (confidence, perplexity, or entropy), it further improves hallucination risk estimation. Qualitative analysis highlights why gloss-free models are more susceptible to hallucinations. Taken together, our findings establish reliability as a practical and reusable tool for diagnosing hallucinations in SLT, and lay the groundwork for more robust hallucination detection in multimodal generation.
@inproceedings{hamidullah-2026-grounding,title={Grounding or Guessing? Visual Signals for Detecting Hallucinations in Sign Language Translation},author={Hamidullah, Yasser and Dutta Chowdhury, Koel and Al Ghussin, Yusser and Yazdani, Shakib and Oguz, Cennet and van Genabith, Josef and Espa{\~n}a-Bonet, Cristina},booktitle={The Fourteenth International Conference on Learning Representations (ICLR)},year={2026},url={https://openreview.net/forum?id=bLFW2T3UHq},}
LREC
A Critical Study of Automatic Evaluation in Sign Language Translation
Shakib Yazdani, Yasser Hamidullah, Cristina España-Bonet, and 2 more authors
In Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC), May 2026
Automatic evaluation metrics are crucial for advancing sign language translation (SLT). Current SLT evaluation metrics, such as BLEU and ROUGE, are only text-based, and it remains unclear to what extent text-based metrics can reliably capture the quality of SLT outputs. To address this gap, we investigate the limitations of text-based SLT evaluation metrics by analyzing six metrics, including BLEU, chrF, and ROUGE, as well as BLEURT on the one hand, and large language model (LLM)-based evaluators such as G-Eval and GEMBA zero-shot direct assessment on the other hand. Specifically, we assess the consistency and robustness of these metrics under three controlled conditions: paraphrasing, hallucinations in model outputs, and variations in sentence length. Our analysis highlights the limitations of lexical overlap metrics and demonstrates that while LLM-based evaluators better capture semantic equivalence often missed by conventional metrics, they can also exhibit bias toward LLM-paraphrased translations. Moreover, although all metrics are able to detect hallucinations, BLEU tends to be overly sensitive, whereas BLEURT and LLM-based evaluators are comparatively lenient toward subtle cases. This motivates the need for multimodal evaluation frameworks that extend beyond text-based metrics to enable a more holistic assessment of SLT outputs.
@inproceedings{yazdani-etal-2026-critical,title={A Critical Study of Automatic Evaluation in Sign Language Translation},author={Yazdani, Shakib and Hamidullah, Yasser and Espa{\~n}a-Bonet, Cristina and Avramidis, Eleftherios and van Genabith, Josef},booktitle={Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC)},year={2026},month=may,address={Palma, Mallorca, Spain},publisher={European Language Resources Association (ELRA)},pages={9535--9548},url={https://lrec.elra.info/lrec2026-main-749},doi={10.63317/4n2sooe4fb2i},}
LREC
A Sacred Bird Called the Phoenix: Auditing the Most-Used Parallel Corpus for German Sign Language Recognition and Translation
Vera Czehmann, Shakib Yazdani, Yasser Hamidullah, and 2 more authors
In Proceedings of the LREC 2026 12th Workshop on the Representation and Processing of Sign Languages: Language in Motion, May 2026
This paper presents an empirical audit of the widely used RWTH-PHOENIX-2014T corpus, examining its suitability as a benchmark for sign language recognition and translation. Through human annotation of the training set and extensive sign-to-text back translation of the test set, we provide detailed statistics that indicate substantial quality issues, including information loss and lexical errors. Automatic scores comparing human sign-to-text back translations to the original speech transcribed references are remarkably low, suggesting strong translationese effects and substantial paraphrasing, revealing limitations of lexical metrics in adequately scoring translation quality. Replacing the original speech-transcribed references with human sign-to-text back translations while scoring existing sign language translation systems reveals the lack of robustness of system evaluation with lexical metrics against this test set. Our findings highlight risks associated with relying on this corpus for model evaluation and call for more rigorous, linguistically grounded evaluation practices in sign language technology research. The back-translated test set and error annotations are made publicly available.
@inproceedings{czehmann-etal-2026-sacred,title={A Sacred Bird Called the Phoenix: Auditing the Most-Used Parallel Corpus for German Sign Language Recognition and Translation},author={Czehmann, Vera and Yazdani, Shakib and Hamidullah, Yasser and Nunnari, Fabrizio and Avramidis, Eleftherios},booktitle={Proceedings of the LREC 2026 12th Workshop on the Representation and Processing of Sign Languages: Language in Motion},year={2026},month=may,address={Palma, Mallorca, Spain},publisher={European Language Resources Association (ELRA)},pages={80--92},url={https://lrec.elra.info/lrec2026-ws-signlang-09},doi={10.63317/3xftynuw7i5c},}
ACL Findings
DualFact+: A Multimodal Fact Verification Framework for Procedural Video Captioning
Cennet Oguz, Yasser Hamidullah, Josef Genabith, and 1 more author
In Findings of the Association for Computational Linguistics: ACL 2026, Jul 2026
@inproceedings{oguz-etal-2026-dualfact,title={DualFact+: A Multimodal Fact Verification Framework for Procedural Video Captioning},author={Oguz, Cennet and Hamidullah, Yasser and van Genabith, Josef and Ostermann, Simon},booktitle={Findings of the Association for Computational Linguistics: ACL 2026},year={2026},month=jul,address={San Diego, California, USA},publisher={Association for Computational Linguistics},pages={38356--38371},url={https://aclanthology.org/2026.findings-acl.1912/},doi={10.18653/v1/2026.findings-acl.1912},}
SemEval
DFKI-MLT at SemEval-2026 Task 7: Steering Multilingual Models Towards Cultural Knowledge
Yusser Al Ghussin, Daniil Gurgurov, Yasser Hamidullah, and 3 more authors
In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval), Jul 2026
@inproceedings{al-ghussin-etal-2026-dfki,title={DFKI-MLT at SemEval-2026 Task 7: Steering Multilingual Models Towards Cultural Knowledge},author={Al Ghussin, Yusser and Gurgurov, Daniil and Hamidullah, Yasser and van Genabith, Josef and Espa{\~n}a-Bonet, Cristina and Ostermann, Simon},booktitle={Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval)},year={2026},month=jul,address={San Diego, California, USA},publisher={Association for Computational Linguistics},pages={2548--2563},url={https://aclanthology.org/2026.semeval-1.322/},doi={10.18653/v1/2026.semeval-1.322},}
LoResLM
A Comprehensive Evaluation of Chain-of-Thought Faithfulness in Persian Classification Tasks
Shakib Yazdani, Cristina España-Bonet, Eleftherios Avramidis, and 2 more authors
In Proceedings of the Second Workshop on Language Models for Low-Resource Languages (LoResLM), Mar 2026
@inproceedings{yazdani-etal-2026-comprehensive,title={A Comprehensive Evaluation of Chain-of-Thought Faithfulness in Persian Classification Tasks},author={Yazdani, Shakib and Espa{\~n}a-Bonet, Cristina and Avramidis, Eleftherios and Hamidullah, Yasser and van Genabith, Josef},booktitle={Proceedings of the Second Workshop on Language Models for Low-Resource Languages (LoResLM)},year={2026},month=mar,address={Rabat, Morocco},publisher={Association for Computational Linguistics},pages={311--323},url={https://aclanthology.org/2026.loreslm-1.27/},doi={10.18653/v1/2026.loreslm-1.27},}
2025
WMT
SONAR-SLT: Multilingual Sign Language Translation via Language-Agnostic Sentence Embedding Supervision
Yasser Hamidullah, Shakib Yazdani, Cennet Oguz, and 2 more authors
In Proceedings of the Tenth Conference on Machine Translation (WMT), Nov 2025
Sign language translation (SLT) is typically trained with text in a single spoken language, which limits scalability and cross-language generalization. Earlier approaches have replaced gloss supervision with text-based sentence embeddings, but up to now, these remain tied to a specific language and modality. In contrast, here we employ language-agnostic, multimodal embeddings trained on text and speech from multiple languages to supervise SLT, enabling direct multilingual translation. To address data scarcity, we propose a coupled augmentation method that combines multilingual target augmentations (i.e. translations into many languages) with video-level perturbations, improving model robustness. Experiments show consistent BLEURT gains over text-only sentence embedding supervision, with larger improvements in low-resource settings. Our results demonstrate that language-agnostic embedding supervision, combined with coupled augmentation, provides a scalable and semantically robust alternative to traditional SLT training.
@inproceedings{hamidullah-etal-2025-sonar,title={SONAR-SLT: Multilingual Sign Language Translation via Language-Agnostic Sentence Embedding Supervision},author={Hamidullah, Yasser and Yazdani, Shakib and Oguz, Cennet and van Genabith, Josef and Espa{\~n}a-Bonet, Cristina},booktitle={Proceedings of the Tenth Conference on Machine Translation (WMT)},year={2025},month=nov,address={Suzhou, China},publisher={Association for Computational Linguistics},pages={301--313},url={https://aclanthology.org/2025.wmt-1.18/},doi={10.18653/v1/2025.wmt-1.18},}
RANLP
Seeing, Signing, and Saying: A Vision-Language Model-Assisted Pipeline for Sign Language Data Acquisition and Curation from Social Media
Shakib Yazdani, Yasser Hamidullah, Cristina España-Bonet, and 1 more author
In Proceedings of the 15th International Conference on Recent Advances in Natural Language Processing (RANLP), Sep 2025
@inproceedings{yazdani-etal-2025-seeing,title={Seeing, Signing, and Saying: A Vision-Language Model-Assisted Pipeline for Sign Language Data Acquisition and Curation from Social Media},author={Yazdani, Shakib and Hamidullah, Yasser and Espa{\~n}a-Bonet, Cristina and van Genabith, Josef},booktitle={Proceedings of the 15th International Conference on Recent Advances in Natural Language Processing (RANLP)},year={2025},month=sep,address={Varna, Bulgaria},publisher={INCOMA Ltd.},pages={1374--1384},url={https://aclanthology.org/2025.ranlp-1.159/},}
ACL SRW
Sign Language Video Segmentation Using Temporal Boundary Identification
Kavu Maithri Rao, Yasser Hamidullah, and Eleftherios Avramidis
In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), Jul 2025
@inproceedings{rao-etal-2025-sign,title={Sign Language Video Segmentation Using Temporal Boundary Identification},author={Rao, Kavu Maithri and Hamidullah, Yasser and Avramidis, Eleftherios},booktitle={Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop)},year={2025},month=jul,address={Vienna, Austria},publisher={Association for Computational Linguistics},pages={1213--1224},url={https://aclanthology.org/2025.acl-srw.93/},doi={10.18653/v1/2025.acl-srw.93},}
2024
ACL
Sign Language Translation with Sentence Embedding Supervision
Yasser Hamidullah, Josef Genabith, and Cristina España-Bonet
In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Aug 2024
State-of-the-art sign language translation (SLT) systems facilitate the learning process through gloss annotations, either in an end2end manner or by involving an intermediate step. Unfortunately, gloss labelled sign language data is usually not available at scale and, when available, gloss annotations widely differ from dataset to dataset. We present a novel approach using sentence embeddings of the target sentences at training time that take the role of glosses. The new kind of supervision does not need any manual annotation but it is learned on raw textual data. As our approach easily facilitates multilinguality, we evaluate it on datasets covering German (PHOENIX-2014T) and American (How2Sign) sign languages and experiment with mono- and multilingual sentence embeddings and translation systems. Our approach significantly outperforms other gloss-free approaches, setting the new state-of-the-art for data sets where glosses are not available and when no additional SLT datasets are used for pretraining, diminishing the gap between gloss-free and gloss-dependent systems.
@inproceedings{hamidullah-etal-2024-sign,title={Sign Language Translation with Sentence Embedding Supervision},author={Hamidullah, Yasser and van Genabith, Josef and Espa{\~n}a-Bonet, Cristina},booktitle={Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},year={2024},month=aug,address={Bangkok, Thailand},publisher={Association for Computational Linguistics},pages={425--434},url={https://aclanthology.org/2024.acl-short.40/},doi={10.18653/v1/2024.acl-short.40},}
2023
ICASSPW
Towards Incorporating 3D Space-Awareness into an Augmented Reality Sign Language Interpreter
Fabrizio Nunnari, Eleftherios Avramidis, Vemburaj Yadav, and 6 more authors
In 2023 IEEE International Conference on Acoustics, Speech and Signal Processing Workshops (ICASSPW), 2023
@inproceedings{nunnari-etal-2023-towards,title={Towards Incorporating 3D Space-Awareness into an Augmented Reality Sign Language Interpreter},author={Nunnari, Fabrizio and Avramidis, Eleftherios and Yadav, Vemburaj and Pagani, Alain and Hamidullah, Yasser and Mollanorozy, Sepideh and Espa{\~n}a-Bonet, Cristina and Woop, Emil and Gebhard, Patrick},booktitle={2023 IEEE International Conference on Acoustics, Speech and Signal Processing Workshops (ICASSPW)},year={2023},pages={1--5},publisher={IEEE},doi={10.1109/ICASSPW59220.2023.10193194},}
2022
WMT
Spatio-temporal Sign Language Representation and Translation
Yasser Hamidullah, Josef Genabith, and Cristina España-Bonet
In Proceedings of the Seventh Conference on Machine Translation (WMT), Dec 2022
This paper describes the DFKI-MLT submission to the WMT-SLT 2022 sign language translation (SLT) task from Swiss German Sign Language (video) into German (text). State-of-the-art techniques for SLT use a generic seq2seq architecture with customized input embeddings. Instead of word embeddings as used in textual machine translation, SLT systems use features extracted from video frames. Standard approaches often do not benefit from temporal features. In our participation, we present a system that learns spatio-temporal feature representations and translation in a single model, resulting in a real end-to-end architecture expected to better generalize to new data sets. Our best system achieved 5 ± 1 BLEU points on the development set, but the performance on the test dropped to 0.11 ± 0.06 BLEU points.
@inproceedings{hamidullah-etal-2022-spatio,title={Spatio-temporal Sign Language Representation and Translation},author={Hamidullah, Yasser and van Genabith, Josef and Espa{\~n}a-Bonet, Cristina},booktitle={Proceedings of the Seventh Conference on Machine Translation (WMT)},year={2022},month=dec,address={Abu Dhabi, United Arab Emirates (Hybrid)},publisher={Association for Computational Linguistics},pages={977--982},url={https://aclanthology.org/2022.wmt-1.94/},}
PETRA
Towards Automated Sign Language Production: A Pipeline for Creating Inclusive Virtual Humans
Lucas Bernhard, Fabrizio Nunnari, Amelie Unger, and 20 more authors
In Proceedings of the 15th International Conference on PErvasive Technologies Related to Assistive Environments (PETRA), 2022
@inproceedings{bernhard-etal-2022-towards,title={Towards Automated Sign Language Production: A Pipeline for Creating Inclusive Virtual Humans},author={Bernhard, Lucas and Nunnari, Fabrizio and Unger, Amelie and Bauerdiek, Judith and Dold, Christian and Hauck, Marcel and Stricker, Alexander and Baur, Tobias and Heimerl, Alexander and Andr{\'e}, Elisabeth and Reinecker, Melissa and Espa{\~n}a-Bonet, Cristina and Hamidullah, Yasser and Busemann, Stephan and Gebhard, Patrick and J{\"a}ger, Corinna and Wecker, Sonja and Kossel, Yvonne and M{\"u}ller, Henrik and Waldow, Kristoffer and Fuhrmann, Arnulph and Misiak, Martin and Wallach, Dieter},booktitle={Proceedings of the 15th International Conference on PErvasive Technologies Related to Assistive Environments (PETRA)},year={2022},address={Corfu, Greece},publisher={Association for Computing Machinery},pages={260--268},doi={10.1145/3529190.3529202},}
2021
AT4SSL
AVASAG: A German Sign Language Translation System for Public Services
Fabrizio Nunnari, Judith Bauerdiek, Lucas Bernhard, and 16 more authors
In Proceedings of the 1st International Workshop on Automatic Translation for Signed and Spoken Languages (AT4SSL), Aug 2021
This paper presents an overview of AVASAG; an ongoing applied-research project developing a text-to-sign-language translation system for public services. We describe the scientific innovation points (geometry-based SL-description, 3D animation and video corpus, simplified annotation scheme, motion capture strategy) and the overall translation pipeline.
@inproceedings{nunnari-etal-2021-avasag,title={AVASAG: A German Sign Language Translation System for Public Services},author={Nunnari, Fabrizio and Bauerdiek, Judith and Bernhard, Lucas and Espa{\~n}a-Bonet, Cristina and J{\"a}ger, Corinna and Unger, Amelie and Waldow, Kristoffer and Wecker, Sonja and Andr{\'e}, Elisabeth and Busemann, Stephan and Dold, Christian and Fuhrmann, Arnulph and Gebhard, Patrick and Hamidullah, Yasser and Hauck, Marcel and Kossel, Yvonne and Misiak, Martin and Wallach, Dieter and Stricker, Alexander},booktitle={Proceedings of the 1st International Workshop on Automatic Translation for Signed and Spoken Languages (AT4SSL)},year={2021},month=aug,publisher={Association for Machine Translation in the Americas},pages={43--48},url={https://aclanthology.org/2021.mtsummit-at4ssl.5/},}