Downloads provided by UsageCountsPlease grant OpenAIRE to access and update your ORCID works.
This Research product is the result of merged Research products in OpenAIRE.
You have already added 0 works in your ORCID record related to the merged Research product.
You have already added 0 works in your ORCID record related to the merged Research product.
This Research product is the result of merged Research products in OpenAIRE.
You have already added 0 works in your ORCID record related to the merged Research product.
You have already added 0 works in your ORCID record related to the merged Research product.
Estimating post-editing effort : a study on human judgements, task-based and reference-based metrics of MT quality
Scarton, Carolina; Forcada, Mikel L.; Esplà-Gomis, Miquel; Specia, Lucia;
Scarton, Carolina; Forcada, Mikel L.; Esplà-Gomis, Miquel; Specia, Lucia;
Estimating post-editing effort : a study on human judgements, task-based and reference-based metrics of MT quality
Devising metrics to assess translation quality has always been at the core of machine translation (MT) research. Traditional automatic reference-based metrics, such as BLEU, have shown correlations with human judgements of adequacy and fluency and have been paramount for the advancement of MT system development. Crowd-sourcing has popularised and enabled the scalability of metrics based on human judgements, such as subjective direct assessments (DA) of adequacy, that are believed to be more reliable than reference-based automatic metrics. Finally, task-based measurements, such as post-editing time, are expected to provide a more detailed evaluation of the usefulness of translations for a specific task. Therefore, while DA averages adequacy judgements to obtain an appraisal of (perceived) quality independently of the task, and reference-based automatic metrics try to objectively estimate quality also in a task-independent way, task-based metrics are measurements obtained either during or after performing a specific task. In this paper we argue that, although expensive, task-based measurements are the most reliable when estimating MT quality in a specific task; in our case, this task is post-editing. To that end, we report experiments on a dataset with newly-collected post-editing indicators and show their usefulness when estimating post-editing effort. Our results show that task-based metrics comparing machine-translated and post-edited versions are the best at tracking post-editing effort, as expected. These metrics are followed by DA, and then by metrics comparing the machine-translated version and independent references. We suggest that MT practitioners should be aware of these differences and acknowledge their implications when deciding how to evaluate MT for post-editing purposes.
Comment: IWSLT 2019, Hong Kong, November 2 and 3, 2019
United Kingdom
- Department of Computing Imperial College London United Kingdom
- Imperial College London United Kingdom
- University of Alicante Spain
- University of Sheffield (Sheffield) / Department of Computer Science United Kingdom
- University of Sheffield United Kingdom
Computation and Language (cs.CL), FOS: Computer and information sciences, Computer Science - Computation and Language
Computation and Language (cs.CL), FOS: Computer and information sciences, Computer Science - Computation and Language
3314
[1] S. Nirenburg, Progress in Machine Translation. sterdam, Netherlands: IOS B. V., 1993.
[2] M. Sanchez-Torron and P. Koehn, “Machine translation quality and post-editor productivity,” in Proceedings of the Conference of the Association for Machine Translation in the Americas Vol. 1: MT Researchers' Track, Austin, TX, 2016, pp. 16-26. [Online]. Available: https://amtaweb.org/wp-content/uploads/2016/ 10/AMTA2016 Research Proceedings v7.pdf
[3] H. Hassan, A. Aue, C. Chen, V. Chowdhary, J. Clark, C. Federmann, X. Huang, M. JunczysDowmunt, W. Lewis, M. Li, S. Liu, T. Liu, R. Luo, A. Menezes, T. Qin, F. Seide, X. Tan, F. Tian, L. Wu, S. Wu, Y. Xia, D. Zhang, Z. Zhang, and M. Zhou, “Achieving human parity on automatic chinese to english news translation,” Computing Research Repository, vol. arXiv:1803.05567, 2018. [Online]. Available: http://arxiv.org/abs/1803.05567
[4] S. La¨ubli, R. Sennrich, and M. Volk, “Has machine translation achieved human parity? a case for document-level evaluation,” in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2018, pp. 4791-4796. [Online]. Available: http://aclweb.org/anthology/D18-1512 [OpenAIRE]
[5] A. Toral, S. Castilho, K. Hu, and A. Way, “Attaining the Unattainable? Reassessing Claims of Human Parity in Neural Machine Translation,” in Proceedings of the Third Conference on Machine Translation (WMT), Volume 1: Research Papers. Brussels, Belgium: Association for Computational Linguistics, 2018, pp. 113-123. [Online]. Available: http://www.statmt.org/ wmt18/pdf/WMT012.pdf
[6] H. P. Krings, Repairing texts: Empirical investigations of machine translation post-editing process. Kent, OH: The Kent State University Press, 2001.
[7] M. Snover, N. Madnani, B. Dorr, and R. Schwartz, “TERp system description,” in Proceedings of the MetricsMATR workshop, vol. 34, no. 67, 2008, p. 108.
[8] M. Snover, B. Dorr, R. Schwartz, L. Micciulla, and J. Makhoul, “A Study of Translation Edit Rate with Targeted Human Annotation,” in Proceedings of the Seventh biennial conference of the Association for Machine Translation in the Americas, Cambridge, MA, 2006, pp. 223-231.
[9] Y. Graham, T. Baldwin, M. Dowling, M. Eskevich, T. Lynn, and L. Tounsi, “Is all that glitters in machine translation quality estimation really gold?” in Proceedings of the 26th International Conference on Computational Linguistics: Technical Papers, Osaka, Japan, 2016, pp. 3124-3134. [Online]. Available: http://aclweb.org/anthology/C16-1294
[10] Y. Graham, T. Baldwin, A. Moffat, and J. Zobel, “Is machine translation getting better over time?” in Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics. Gothenburg, Sweden: Association for Computational Linguistics, 2014, pp. 443-451. [Online]. Available: http://www.aclweb.org/anthology/E14-1047
citations This is an alternative to the "Influence" indicator, which also reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).1 popularity This indicator reflects the "current" impact/attention (the "hype") of an article in the research community at large, based on the underlying citation network.Average influence This indicator reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).Average impulse This indicator reflects the initial momentum of an article directly after its publication, based on the underlying citation network.Average views 152 downloads 133 citations This is an alternative to the "Influence" indicator, which also reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).1 popularity This indicator reflects the "current" impact/attention (the "hype") of an article in the research community at large, based on the underlying citation network.Average influence This indicator reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).Average impulse This indicator reflects the initial momentum of an article directly after its publication, based on the underlying citation network.Average - 152views133downloads
citations
Citations provided by BIP!
This is an alternative to the "Influence" indicator, which also reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
Citations provided by BIP!popularityPopularity provided by BIP!
This indicator reflects the "current" impact/attention (the "hype") of an article in the research community at large, based on the underlying citation network.
Popularity provided by BIP!influenceInfluence provided by BIP!
This indicator reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
Influence provided by BIP!impulseImpulse provided by BIP!
This indicator reflects the initial momentum of an article directly after its publication, based on the underlying citation network.
Impulse provided by BIP!viewsViews provided by UsageCounts
Views provided by UsageCountsdownloadsDownloads provided by UsageCounts
Downloads provided by UsageCounts 1
Average
Average
Average
152
133
Funded by
Do the share buttons not appear? Please make sure, any blocking addon is disabled, and then reload the page.
- Department of Computing Imperial College London United Kingdom
- Imperial College London United Kingdom
- University of Alicante Spain
- University of Sheffield (Sheffield) / Department of Computer Science United Kingdom
- University of Sheffield United Kingdom
- White Rose Consortium: University of Leeds; The University of Sheffield; University of York United Kingdom
Devising metrics to assess translation quality has always been at the core of machine translation (MT) research. Traditional automatic reference-based metrics, such as BLEU, have shown correlations with human judgements of adequacy and fluency and have been paramount for the advancement of MT system development. Crowd-sourcing has popularised and enabled the scalability of metrics based on human judgements, such as subjective direct assessments (DA) of adequacy, that are believed to be more reliable than reference-based automatic metrics. Finally, task-based measurements, such as post-editing time, are expected to provide a more detailed evaluation of the usefulness of translations for a specific task. Therefore, while DA averages adequacy judgements to obtain an appraisal of (perceived) quality independently of the task, and reference-based automatic metrics try to objectively estimate quality also in a task-independent way, task-based metrics are measurements obtained either during or after performing a specific task. In this paper we argue that, although expensive, task-based measurements are the most reliable when estimating MT quality in a specific task; in our case, this task is post-editing. To that end, we report experiments on a dataset with newly-collected post-editing indicators and show their usefulness when estimating post-editing effort. Our results show that task-based metrics comparing machine-translated and post-edited versions are the best at tracking post-editing effort, as expected. These metrics are followed by DA, and then by metrics comparing the machine-translated version and independent references. We suggest that MT practitioners should be aware of these differences and acknowledge their implications when deciding how to evaluate MT for post-editing purposes.
Comment: IWSLT 2019, Hong Kong, November 2 and 3, 2019
Do the share buttons not appear? Please make sure, any blocking addon is disabled, and then reload the page.