How best to explain the uselessness of QA tools?
Thread poster: Kay-Viktor Stegemann
Kay-Viktor Stegemann
Kay-Viktor Stegemann
Germany
Local time: 17:08
English to German
In memoriam
Jan 30, 2021

Many big agencies and even end clients use QA tools (like XBench) on the translations they get and return a "QA report" to the translator with a list of inconsistencies, term/glossary violations and such. I cannot judge the usefulness of these reports in other languages, but in the pair English to German, 99.99% of the "errors" listed are false alarms, mostly due to the fact that a term can have a virtually unlimited number of correct spellings in German, due to changes of word forms for tense, ... See more
Many big agencies and even end clients use QA tools (like XBench) on the translations they get and return a "QA report" to the translator with a list of inconsistencies, term/glossary violations and such. I cannot judge the usefulness of these reports in other languages, but in the pair English to German, 99.99% of the "errors" listed are false alarms, mostly due to the fact that a term can have a virtually unlimited number of correct spellings in German, due to changes of word forms for tense, number, gender, case, and so on, and particularly to the fact that we can create compounds in German where the actual term "disappears" somewhere in the middle of the word. All that means that term and spelling "errors" from these reports can as well be trashed, it makes no sense even to look at them.

But some clients are particularly stubborn and demand explanations for every single deviation from the prescribed glossary terms, or at least an explicit "ignoring" of every single "error". This causes an enormous overhead and extra work without adding a micron of quality. Sometimes I'm at a loss how to deal with this when clients who have no grasp of the target language demand detailed explanations. Do you have similar problems sometimes? Do you explain grammar intricacies of the target language to the client in the source language?
Collapse


Elena Feriani
Matthias Brombach
Luca Colangelo
ahartje
Angie Garbarino
Kardi Kho
Stepan Konev
 
Jean Dimitriadis
Jean Dimitriadis  Identity Verified
English to French
+ ...
False positive Jan 30, 2021

Would just adding the explanation "False positive" be enough to both 1. explain the reason for ignoring or otherwise not fixing the "error" ; and 2. highlight the ratio of useless false alarms that this tool produces for you, which could also strengthen your arguments for dropping its use?

The farthest I would go is do a line by line full explanation for a single report, and agree with the agency to use "False positive" going forward (unless a short, if possible reusable, and by no
... See more
Would just adding the explanation "False positive" be enough to both 1. explain the reason for ignoring or otherwise not fixing the "error" ; and 2. highlight the ratio of useless false alarms that this tool produces for you, which could also strengthen your arguments for dropping its use?

The farthest I would go is do a line by line full explanation for a single report, and agree with the agency to use "False positive" going forward (unless a short, if possible reusable, and by no means detailed explanation is needed, like "Terminology variation to account for different in-context meaning").

Finally, maybe the tool can be set in such a way as to avoid producing so many false positives, but I guess you are using a specific set of rules in XBench?

[Edited at 2021-01-30 21:21 GMT]
Collapse


Said Abdi Ahmed
María C Turri
Clement Cheung
 
Inga Petkelyte
Inga Petkelyte  Identity Verified
Portugal
Local time: 16:08
Lithuanian to Portuguese
+ ...
Not anymore Jan 30, 2021

On several occasions, the time consumption for explanations (and finding references as an evidence) turned out to be way higher than the time for the translation itself. I ended up not doing that anymore. I tell the client to ask a specific question where they might have doubts and that's it. Normally, they don't. For preferencial options, I note that I disagree but if they/the agency are happy, the choice - and the responsability - is theirs. No more rummaging through books, internet, official ... See more
On several occasions, the time consumption for explanations (and finding references as an evidence) turned out to be way higher than the time for the translation itself. I ended up not doing that anymore. I tell the client to ask a specific question where they might have doubts and that's it. Normally, they don't. For preferencial options, I note that I disagree but if they/the agency are happy, the choice - and the responsability - is theirs. No more rummaging through books, internet, official guidelines to prove the obvious.Collapse


Victoria Britten
Sadek_A
neilmac
Philippe Etienne
Christine Andersen
Laura Kingdon
 
Kay-Viktor Stegemann
Kay-Viktor Stegemann
Germany
Local time: 17:08
English to German
TOPIC STARTER
In memoriam
I don't use these tools, the client does Jan 30, 2021

Jean Dimitriadis wrote:

Would just adding the explanation "False positive" be enough to both 1. explain the reason for ignoring or otherwise not fixing the "error" ; and 2. highlight the ratio of useless false alarms that this tool produces for you, which could also strengthen your arguments for dropping its use?

The farthest I would go is do a line by line full explanation for a single report, and agree with the agency to use "False positive" going forward (unless a short, if possible reusable, and by no means detailed explanation is needed, like "Terminology variation to account for different in-context meaning").

Finally, maybe the tool can be set in such a way as to avoid producing so many false positives, but I guess you are using a specific set of rules in XBench?

[Edited at 2021-01-30 21:21 GMT]


I don't use these tools and I don't configure them, the client does. If I had any influence on this, I would tell them to switch off term checking and spell checking in these tools since these parts are (for my pair) completely useless (for spell checking, the Word spell checker is a hundred times better because it seems to be able to recognize compounds and even grammar issues). And writing "false positive" a thousand times is a lot of useless work even if I can copy and paste it 


Matthias Brombach
neilmac
Angie Garbarino
Christine Andersen
Laura Kingdon
 
Endre Both
Endre Both  Identity Verified
Germany
Local time: 17:08
English to German
Charge them Jan 30, 2021

In one of my projects, the end client used to ask endless questions about minute details without having any target language knowledge beyond Google Translate. No amount of reasoning was enough to dissuade them. But shortly after I started billing for the extra time spent, the queries miraculously stopped, saving a lot of time for everyone involved.

Victoria Britten
Sadek_A
Susan Murphy Lamprecht
neilmac
Emanuele Vacca
Kay-Viktor Stegemann
Nikki Scott-Despaigne
 
Mario Cerutti
Mario Cerutti  Identity Verified
Japan
Local time: 01:08
Italian to Japanese
+ ...
What I do Jan 31, 2021

Kay-Viktor Stegemann wrote:
And writing "false positive" a thousand times is a lot of useless work even if I can copy and paste it.

I diligently but quickly check line by line to see if I have actually made mistakes. If really necessary, I correct those mistakes only and give a final general statement in the reply email offering plenty of apologies, then I send the revised file. For the rest, in the same email I simply state that there are no problems.

This usually works. (PMs are generally quite busy themselves and I suspect that they are happier with just receiving the revised file, that is, without having to go through translators' comments one by one.)

[Edited at 2021-01-31 02:19 GMT]

[Edited at 2021-01-31 07:26 GMT]


Kevin Fulton
 
Sadek_A
Sadek_A  Identity Verified
Local time: 20:08
English to Arabic
+ ...
..... Jan 31, 2021

Your strategy should comprise the following:
- Making 100% sure in the first place that there were no mistakes, before delivery.
- Pointing out once and for all that the tool was developed by a human brain, and not the human brain by a tool. Which serves to shake their confidence in the tool, even the slightest bit!
- Indicating beyond any doubt that your base rate doesn't cover after-sales service, and that they would need to pay extra on an hourly basis for that added service
... See more
Your strategy should comprise the following:
- Making 100% sure in the first place that there were no mistakes, before delivery.
- Pointing out once and for all that the tool was developed by a human brain, and not the human brain by a tool. Which serves to shake their confidence in the tool, even the slightest bit!
- Indicating beyond any doubt that your base rate doesn't cover after-sales service, and that they would need to pay extra on an hourly basis for that added service, as per your sole discretion! Granted, you are 100% responsible for omissions, disorders and/or verifiable mistakes (verifiable only against the source! not against the whims of client and/or their personnel) in the part(s) you got paid/POed for; but, not for explaining choices and/or clearing mere suspicions!
- Ensuring that the work coming from them is big enough to justify all the hassle.

On a different note: by "micron of quality", did you mean "scrutiny of quality"? Just checking before possibly adding to glossary!
Collapse


Susan Murphy Lamprecht
 
Elena Feriani
Elena Feriani
Italy
Local time: 17:08
Member
French to Italian
+ ...
False positives Jan 31, 2021

Two of my clients, both from China, send me Xbench reports frequently and sometimes they want me to reply immediately. I have asked them to share their checklists so I run the report before delivering and when they send me the same report back I can just reply: "All false positives". It is expensive at 99 euro per year, but it's worth it in my opinion. I use it for every client now.
In Italian, I have the same issues with false positives due to gender and number or verb form, but at least
... See more
Two of my clients, both from China, send me Xbench reports frequently and sometimes they want me to reply immediately. I have asked them to share their checklists so I run the report before delivering and when they send me the same report back I can just reply: "All false positives". It is expensive at 99 euro per year, but it's worth it in my opinion. I use it for every client now.
In Italian, I have the same issues with false positives due to gender and number or verb form, but at least the spellcheck is very good.
Collapse


Susan Murphy Lamprecht
 
Adieu
Adieu  Identity Verified
Ukrainian to English
+ ...
Wow that's obnoxious Jan 31, 2021

I thought having to copy false tags and putting a billion little check marks to ignore punctuation or confirm alphabet changes for transliteration of serial numbers and the like from Cyrillic in memoQ's QA was annoying... but written explanations? For software whose results overlook the rules of the target language? Damn.

Are these well-paying clients, at least? Have you tried billing them for QA time if not?


Christopher Schröder
 
Stepan Konev
Stepan Konev  Identity Verified
Russian Federation
Local time: 19:08
English to Russian
Picture is worth a thousand words Jan 31, 2021

I have a picture for this purpose.
It shows a list [still not exhaustive] of words that can be formed using the root 'бег' (equivalent of 'run' in English).


Elena Feriani
Elena Doroshenko
P.L.F. Persio
Adieu
WS McCallum
 
kd42
kd42
Estonia
Local time: 18:08
English to Russian
Switch off terminology check Jan 31, 2021

Kay-Viktor Stegemann wrote:
with a list of inconsistencies, term/glossary violations and such

I explained every X-th term "error" to the client, very clearly, with an example in their language.
Because the quality of their termbase is ridiculously low, all these were false alarms.

The present obsession with QA tools remind me of the early era of machine translation, when
a) the package was on two 5-inch floppies; and
b) most people were sure it would deliver flawless translations (so you can fire most of your translators).

Then I suggest solutions:
1) Pay me for the time I spend digging in the dirt (QA report);
2) Pay me for the time I spend digging in the dirt (QA report) and for trimming and optimizing their termbase;
3) Ditch the termbase and the terminology check.

All of my clients asked a couple of clever questions and chose item 3.


Adieu
Philippe Locquet
Laura Kingdon
 
Riccardo Schiaffino
Riccardo Schiaffino  Identity Verified
United States
Local time: 10:08
Member (2003)
English to Italian
+ ...
Xbench is a great tool, but... Jan 31, 2021

Xbench is a great tool, and I use it on almost every project I do.

But like any tool, it must be used properly, and the problem is that many users (especially PMs) don’t really know how to use it.


  • Spell check. The usefulness depends on the language - for some it is indeed very useful (English and Italian, among others), for other languages less so. The spell checker in Xbench is especially useful... See more
Xbench is a great tool, and I use it on almost every project I do.

But like any tool, it must be used properly, and the problem is that many users (especially PMs) don’t really know how to use it.


  • Spell check. The usefulness depends on the language - for some it is indeed very useful (English and Italian, among others), for other languages less so. The spell checker in Xbench is especially useful because, unlike the spell checker in other tools (e.g., MS Word, SDL Trados Studio, memoQ, etc.), it provides a list of misspelled words, with one occurrence only of each supposed misspelling. You can then sort the list, quickly select those that need to be further checked in context to add them to the rest of your QA, and just ignore the others. In my opinion, the spell checker should NOT in fact be used by the PMs, since they usually do not know the target language, and therefore are prone to add the entire list - this, indeed, would likely result in an enormous number of false positives.
  • Key terms. Here the usefulness depends both on the quality of the glossary used for the key terms (it should be as short as possible, with only actual technical terms included, and only one translation for each source term), and, again, also on the language. For certain languages it would in fact be better to use project or personal checklists (which allow, with some additional effort, to take into account the vagaries of most languages) instead of key term glossaries. Again, checklists and key terms glossaries are better left to the translator or to a language lead who knows the target language, rather than to a PM who doesn’t.


For a detailed introduction to the use of Xbench for terminology management and translation QA, see this page, with two online presentations on Xbench, from my blog About Translation
Collapse


texjax DDS PhD
Elena Feriani
Darius Sciuka
P.L.F. Persio
Oriana Bonan
 
Christopher Schröder
Christopher Schröder
United Kingdom
Member (2011)
Swedish to English
+ ...
Useless vs frustrating Feb 1, 2021

The trouble with declaring these tools “useless” is that I imagine, like all translators, you make mistakes, and sometimes they will pick them up, undermining your argument...

But I can see that they could be highly irritating, and it seems odd that the customer does not know this and improve them, as ultimately they are the ones who will be paying for all this wasted time.


P.L.F. Persio
Mervyn Henderson (X)
Dr. Matthias Schauen
 
Riccardo Schiaffino
Riccardo Schiaffino  Identity Verified
United States
Local time: 10:08
Member (2003)
English to Italian
+ ...
Improvements to QA tools Feb 1, 2021

Chris S wrote:
But I can see that they could be highly irritating, and it seems odd that the customer does not know this and improve them, as ultimately they are the ones who will be paying for all this wasted time.


ApSIC (the company that develops Xbench) continues to improve the tool, and in my experience they are open to adding features based on customers’ suggestions.

The problem (and the source of frustration for the original poster here) is that many PMs either don’t really know how to use these tools, or don’t care, and send out to translators QA reports that are filled with false positives.

What PMs should do:

- Make sure that the key term glossaries they use are of high quality (containing only the terms that they should)
- Do not run the spell-check test themselves (the translators should, instead)
- Be open to translators not confirming each line of a QA report when there is an unavoidable reason for all false positives

For me there is a big difference between a translator writing something like, for example “I have carefully reviewed the terminology used in this translation, and confirm that it matches the required terminology. All remaining reported mismatches are due to [linguistic reason depending on the peculiarities of source and target language]” and a translator not bothering to look at the QA report. The former is acceptable, the latter not.

But PMs who insist on the translator justifying each and every instance of a QA item left open because the PM didn’t bother to use the QA tool properly should be asked to pay for the additional time they demand from their translators.

[Edited at 2021-02-01 16:58 GMT]


Christopher Schröder
Dan Lucas
Laura Kingdon
 
Philippe Locquet
Philippe Locquet  Identity Verified
Portugal
Local time: 16:08
English to French
+ ...
1 2 3 Feb 5, 2021

kd42 wrote:

Then I suggest solutions:
1) Pay me for the time I spend digging in the dirt (QA report);
2) Pay me for the time I spend digging in the dirt (QA report) and for trimming and optimizing their termbase;
3) Ditch the termbase and the terminology check.

All of my clients asked a couple of clever questions and chose item 3.


Totally agree with this approach, makes total sense.
Many companies sometimes buy into advertisement promises and don't understand what these QA-tools do and what are their limitations. Not to mention language specific false positives, IMO, a proper QA should be performed by linguists fluent in the pair and knowing what they are doing.

Maybe there should be LSP-oriented training sessions, I haven't seen many of these online though.


kd42
 


To report site rules violations or get help, contact a site moderator:


You can also contact site staff by submitting a support request »

How best to explain the uselessness of QA tools?







Trados Business Manager Lite
Create customer quotes and invoices from within Trados Studio

Trados Business Manager Lite helps to simplify and speed up some of the daily tasks, such as invoicing and reporting, associated with running your freelance translation business.

More info »
Wordfast Pro
Translation Memory Software for Any Platform

Exclusive discount for ProZ.com users! Save over 13% when purchasing Wordfast Pro through ProZ.com. Wordfast is the world's #1 provider of platform-independent Translation Memory software. Consistently ranked the most user-friendly and highest value

Buy now! »