The Framework for Machine Translation Evaluation in ISLE is an effort to organize the various methods that are used to evaluate MT systems (machine translation systems), and to relate them to the purpose and context of the systems. Therefore, FEMTI is made of two interrelated classifications or taxonomies. The first classification helps evaluators to define an intended context of use for the MT system to evaluate. After that each feature is linked to relevant quality characteristics and metrics, defined in the second classification.
FEMTI contains: A classification of the main features defining the context of use that is: the type of user of the MT system, the type of task the system is used for, and the nature of the input to the system. Also contains a classification of the MT software quality characteristics, into hierarchies of sub-characteristics, with internal and/or external attributes (i.e., metrics) at the bottom level. The upper levels match the ISO/IEC 9126 characteristics. And finally contains a mapping from the first classification to the second, which defines or suggests the quality characteristics, sub-characteristics and attributes/metrics that are relevant to each context of use.
FEMTI helps People who want to use an MT system. They can select the quality characteristics that are most important to them and in this manner choose the MT system that best suits these characteristics. People who want to compare several MT systems are also helped by FEMTI. They can browse and select the characteristics that best reflect their circumstances, and thereby find associated evaluation measures and tests. People who want to design a new MT system or to upgrade an old one are helped too. They can learn about the requirements of users and find applications for their system.
