Better translations through automated source and post edit analysis

Better translations through automated source and post-edit analysis David Landan Welocalize

Background • MT is here to stay – Better MT = less PE effort = higher throughput for less money • MT quality depends on training data quantity, quality, and relevance – Selecting in-domain data increases BLEU scores by 10-20 BLEU over generic engines • LSPs have less control over quantity, so we need to focus on quality & relevance

A data-driven approach • Analytics at each step Training MT Production • • Perplexity Evaluator • Candidate Scorer • Source Content Profiler (joint project w/CNGL) • StyleScorer • Number checking StyleScorer • UGC Normalization Post-Editing • WeScore • StyleScorer

Candidate Scorer • Uses corpus of known “difficult” text • Compares part of speech (POS) n-grams – Generates per-sentence scores

Perplexity (PPL) Evaluator • Build language models (LMs) from multiple corpora – Known “good” sentences for MT – Known “bad” sentences for MT – Client-specific in-domain data • Each document gets a PPL score against each LM

StyleScorer • Combines PPL ratio, dissimilarity score, and classification score – Each document receives a score from 0-4 – Higher score indicates better match to style established by client’s documents – Does not require parallel data • Source scored for training/tuning suitability

Source Content Profiler • CNGL project (beta) – Classification of docs into profiles – Features based on: • • • • • • • Word & sent. length Readability score Syntactic structure Terminology Tag ratios Do Not Translate lists Glossary matches

Does it work? en-USnl-NL en-USpl-PL en-UShu-HU Plain vanilla 21.26 16.88 17.31 Domain match 36.39 37.07 38.36 Plain + target 44.07 34.61 30.43 Domain + target 64.40 54.55 49.53 Engine

A data-driven approach • Analytics at each step Training MT Production • • Perplexity Evaluator • Candidate Scorer • Source Content Profiler (joint project w/CNGL) • StyleScorer • Number checking StyleScorer • UGC Normalization Post-Editing • WeScore • StyleScorer

UGC normalization • Make substitutions in source for known MT pain points before translating – Frequent misspellings – “teh”, “mroe”, etc. – Abbreviations – “imho”, “tyvm”, etc. – Missing punctuation – “cant”, “theyll”, etc. – Emoticons – Spelling variants/slang – “cuz”, “usu”, etc.

Number checking • Verify that numeric MT output is localized correctly – Currency – “$1B” vs “1 млрд. $” – Dates – “2/28/2014” vs “28/2/2014” – Time – “2pm” vs “14h00” – Separator & radix – “1,234.5” vs “1 234,5”

StyleScorer revisited • MT output is compared to client’s historical (in-domain) PE data – Treat each target segment as a document – Lower scores indicate segments likely to require greater PE effort

A data-driven approach • Analytics at each step Training MT Production • • Perplexity Evaluator • Candidate Scorer • Source Content Profiler (joint project w/CNGL) • StyleScorer • Number checking StyleScorer • UGC Normalization Post-Editing • WeScore • StyleScorer

WeScore • Dashboard for viewing MT metrics – Tokenizes input from variety of formats & runs several scoring algorithms in parallel – Exports detailed analysis to spreadsheet for sentence-by-sentence review

WeScore

WeScore

StyleScorer III • PE output is compared to client’s historical (in-domain) data – Treat each PE segment as a document – Lower score indicates possible deviation from established style

Feedback loop • Data collected and lessons learned – Update client-specific data for future engine training – Mine data for generalizable patterns in problem areas – Work with post-editors to understand how to make a better system & how to improve PE experience and throughput

Q&A Thank you!