An Evaluation Method for MT Systems
and Its Application to ALT-J/E

Satoru IKEHARA*1,
Masahiro MIYAZAKI*2
Satoshi SHIRAI*1 and
Akio YOKOO*1

*1 NTT Communications and Information Processing
Laboratories,Take1-2356, Yokosuka-shi,Japan

*2 Niigata University, Igarasi Ninomachi8050,
Niigata-shi, Japan

Key Words: Machine Translation,
Evaluation of Translation Quality,
Translation Technology,
Japanese to English Translation

An Evaluation Method for MT System and Its Ap
plication to ALT-J/E

Summary

An evaluation method for a Japanese to English
machine translation System which estimates trans
lation quality based on translation technologies
and source text characteristics is proposed. This
method is applied to the translation of newspaper
articles by the ALT-J/E system and further
problems that need to be surmounted are discussed
based on the results.
Assuming 70% acceptability to be a practical
level of quality, it is shown that morphological
and dependency analysis technologies would require
accuracy rate per word and per "bunsetsu"(which
are Japanese phrase, in most case, consisting of a
noun and "joshi" or post-positional word) as high
as 99.8% and 99.4%, respectively. Current tech
nology has already reached this level. But the
precision of translation for individual expres
sions is lower than expected values and the ac
ceptability rate for the entire sentence drops to
between 40% and 50%. To achieve an acceptability
rate of 70%, it is necessary to achieve an average
7% improvement in 9 types of expressions and thus
achieve a successful translation ratio of 96%.
To upgrade the translation rate of these ex
pressions, it will be necessary to establish tech
niques for meaning analysis of post-positional
words, translation of expressions that combine
noun clauses and compounded words, handling con
nections and embedded clauses, and verification of
ellipses and paratactical constructions.

1. Introduction
In the field of Japanese-to-English machine
translation, numerous experimental systems(1),(2)
and commercial systems(3)-(5) have been developed.
Without exception, these systems require pre-
editing and/or post-editing to be of practical
use. Many technological problems have to be solved
in order to develop a Japanese-English translation
system capable of translating source text without
pre-editing for monolingual users(6). To clarify
these problems, an evaluation method has to be
developed.
There have been some proposal for evaluating
the quality of translation. The ALPAC Report(7)
proposed a 9 level standard. Nagao(8) proposed two
5-level standards, one each for understandability
and faithfulness, and applied it to the Mu-
Project. However these methods are concerned with
the results of translation. Translation results
are dependent on the technologies involved as well
as source text characteristics. This relation is
so complicated that there has been no evaluation
method for it.
Taking notice of the construction of the
translation system, this paper proposes an evalua
tion method which estimates translation quality
based on translation technologies and source text
characteristics. This method is applied to trans
lation of newspaper articles by the ALT-J/E
(Automatic Language Translator : Japanese to
English) system (9) ,(10) and further problems to
be surmounted are discussed based on the results.
And it is also proposed that using this evaluation
model, setting the translation acceptability rate
at 70% as the target for translation quality will
clarify the problem areas in the existing tech
nology and identify the tasks to be faced.
The translation process is divided into the
sentence analysis phase and the conversion
/generation phase. The capability of the former
phase is represented by the capability of the mor
phological and dependency analysis phases, and
that of the latter phase is derived from the
capability of English expression generation for
several characteristic expressions in language. An
evaluation method is proposed from this point of
view.
Specially, 1,000 sentences of lead texts
(which are typical introductory sentences sum
marizing a newspaper article immediately following
the headlines) were taken from the Nikkei Sangyo
Newspaper. Sentence structures and characteristic
expressions that would pose problems in machine
translation were studied. Regarding the charac
teristics of structuring, formal characteristics
of typical newspaper articles, such as length of
articles, number of "bunsetsu" and other such
statistical features are studied. As characteris
tics of expression, we consider 9 types of expres
sion may pose problems in the formulation of algo
rithms and investigate the frequency of their ap
pearance in newspaper articles.
These characteristics of newspaper articles
and the data quantity for each individual technol
ogy are used in the evaluation method. Estimates
of final translation quality are made and compared
with the actual measured values to confirm the ac
curacy of this evaluation method and examine tech
nical problems which remain for solution in the
future.

2. Statistical Characteristics Analysis of
Newspaper Articles
From the viewpoint of existing machine trans
lation technology, actual text (this excludes fic
tion, essays, poetry and such artistic literature)
for which demand is strong is to be desired for
translation. Real text could include classroom
textbooks, dissertations, manuals, and newspaper
aticles. Of these texts, newspaper articles are
read by many persons and are a popular form of
text with a premium on speed of reporting. Hopes
and expectations to be placed on translation
without pre-editing* are great indeed, and this
type of text has therefore been selected as the
object for evaluation.
Specifically, 1,000 lead texts from the
"Information, Software and AI" column and the
"Communication" column of the Nihon Keizai Shim
bun, February 1st to 28th,1988, have been selected
as object texts. The "lead text" as mentioned
previously, is the summary which follows the head
line of an article. When there is no such sum
mary, the first paragraph of the main text of the
article is regarded as the lead text. Lead texts
generally consist of anywhere from 3 to 5 sen
tences. Hereafter, the sample documents will be
referred to as newspaper article texts.
-------------------------------------------------
* Translations with pre-editing pose problems
arising from the contents of pre-editing and
evaluation becomes complicated. We shall there
fore consider work without pre-editing.
-------------------------------------------------

2.1 Characteristics of Newspaper Article Forms
A study of the characteristics of the forms
of newspaper articles in terms of average number
of characters and average number of words per sen
tence is summarized in Table 1.

2.2 Characteristics of Expression in Newspaper
Articles
From current translation technology, a sen
tence which has all of the necessary elements such
as the subject, verb and object which constitute
an English sentence and which allows application
of translated words registered in a dictionary
would pose no particular problem. Here we have
taken up 9 types of expressions which would cause
problems for current machine translation technol
ogy and tabulated the frequency with which they
appear in newspaper article sentences. The
results shown in Table 2 indicate the following.
(1) Compounds consisting of successive nouns are
numerous, appearing an average of 3 times per
sentence. These are compounded as necessary from
proper nouns, numerals and ordinary nouns. It
would be difficult to anticipate such compounds
and have them compiled and registered into a dic
tionary beforehand.
(2) There is frequent usage of functional chain
words consisting of compound words with a semantic
or syntactic role. However, their numbers are
limited when viewed from the types of compound
words.
(3) There are many predicates accompanied by
modality and compound predicates, which result
from combination of subjects or objects directly
with verbs to form new compounds. As in the case
of compounds of successive nouns, this also is a
method of compressing an expression.
(4) There are many expressions using sentences
connected within sentences, quotations, and
reporting types of expressions. With newspapers
primarily playing the role of conveying reports,
and following and reporting on events, there
frequently arise expressions where the subject and
the object are omitted and context analysis be
comes neccesary.

3. Concepts of Evaluation Model
3.1 Basic Idea of Quality Evaluation Model
(1) Structure of MT System
ALT-J/E system consists of two major
processes, source language analysis and target
language generation as shown in Fig.1. These will
be referred to as the analytical phase and the
conversion/generation phase. The conventional
translation system relying on the transfer method
usually divides the processing phase into three
phases such as the analytical, conversion and gen
eration phases. ALT-J/E is already aware of con
version to the target language at the analytic
stage, so that the conventional stage of conver
sion become a process that spans both the analytic
and generation phases. Conversion is regarded
here as a portion of the generation phase and
divides the entire translation process into two
phases. However, the conventional three phases
can also be considered as same these two phase
construction, if the conversion process can be
evaluated for each expression.
The analytical phase consists of numerous
processes. But as long as the morphological and
dependency analyses are both executed accurately,
there is little chance of failure in other
processes. So for this evaluation model, the
analytical phase is taken to consist of the mor
phological and dependency analyses.
The conversion/generation phase also consists
of numerous phases. But, if a source expression
is correctly analyzed in the analytical phase and
if the generation mechanism for each expression
works correctly, translation results should become
almost understandable, regardless of whether it is
good English or not.
(2) Remarks for Evaluation Model
The essential meaning of the above thoughts on
phase separation is as follows. There are nor
mally numerous types of expressions mixed in a
single actual sentence. These expressions have
mutual reactions that would be brought about when
they are combined together. Then the translation
quality of the entire sentence cannot be broken
down into evaluations of the expressions con
tained. This method takes these mutual reactions
into consideration and separates the evaluation of
the analytical phase from the evaluation of the
conversion/genaration phase. Various types of ex
pression are mixed in the analytical phase and
their reactions are taken into consideration.
After this has been accurately analyzed, evalua
tion of each individual expression can be con
ducted.

3.2 Target Quality for a Practical Machine Trans-
lation System
(1) Target quality for translation
The objective of machine translation is to
have a finished product in which the contents of
an entire article can be understood almost without
error. Normally, newspaper articles consist of a
number of sentences and an understanding of the
outline of an article does not require a com
pletely accurate translation of all of the sen
tences.
As the passing line for a practical system,
therefore, we shall seek as a quality target to
have 70% of all the sentences contained in a
translation be accurately understood and have
nothing particularly inaccurate with the English
language structuring.
[Remarks] With 70% as the passing grade, there
should be among the unacceptable portion some 10
to 20% whose meaning can be grasped. It is es
timated that translation which cannot be under
stood at all would be limited to about 10%. In
Saeki(11), a translation ratio of 85% is judged
from experience as being at a level that is prac
tically usable. This ratio includes the portion
of translation in which some dubious parts remain
and therefore it is judged equal to the standards
set forth in this dissertation.
(2) Technical Quality Target
In machine translation, a process error in any
one of phases will result in an inaccurate
product. In considering the probability of ac
curacy for each of the phases, the analytical
phase consists of uniform processes whereas the
conversion/generation phase consists of a variety
of processes. It is relatively difficult to im
prove the accuracy of the latter phase. With
these factors in mind, the target yield rate for
each phase needed to arrive at a translation with
an overall acceptability rate of 70% was calcu
lated (Fig.2). Converting these rates, target
processing accuracy values for each phase are as
follows.
Morphological Analysis--------- 95%
Dependency Analysis------------ 94.7%
Coversion/Generation----------- 77.8%

4. The Machine Translation Quality Evaluation
Model, and Its application to ALT-J/E
4.1 Analytical Phase Evaluation Model
(1) Accuracy of morphological analysis
Assume the average number of words in a sen
tence to be nm. Assume also that the probability
of the words of interest being accurately isolated
from the rest of the sentence and of having the
parts of speech, active code and other syntactic
attributes accurately appraised (i.e. word
accuracy) to be pm. When all of the words in a
sentence have been accurately analyzed, mor
phological analysis of the sentence can be said to
be accurate.
　　　　　　Therefore the probability of accuracy of morp
anhological analysis Pm can be expressed by the
following equation.
nm 1
Pm =pm then, pm=exp(---ln Pm) (1)
nm

Assuming from the target accuracy ratio for
practical use of the product that Pm = 0.95 (See
Fig.2), and from Table 1 that nm = 22.2, the ac
curacy per word for purposes of morphological
analysis needs to be over 99.8%.
(2) Accuracy rate of dependency analysis
In dependency analysis, dependent phrases are
determined in relationship to all "bunsetsu".
"Bunsetsu" are conceived of as pairs of dependent
and supporting phrases. All bunsetsu become a de
pendent phrase and all bunsetsu excepting the
first one but including a period becomes a sup
porting phrase. Thus, we can regard the number of
steps required for dependent and support pairing
to be the same as the number of bunsetsu.
If we assume the probability of each depen
dent and support pair determined accurately to be
pd, dependency analysis is accurate when all the
dependent and support phrases have been accurately
determined. Therefore, the probability of accurate
dependent and support pairing is
nd 1
Pd=pd then, pd=exp(--- ln Pd). (2)
nd

From Fig.2, the value Pd=0.947 and from Table
1 the value nd = 8.35 are substituted. To achieve
the practical target, the accuracy rate of the de
pendent and support pair needs to be over 99.4%.
(3) Observation
The relation between Equation (1) and (2) in
the case of newspaper articles is illustrated in
Fig.3. The technical difficulty in improving both
word accuracy in morphological analysis and pair
accuracy in dependency analysis increase sharply
in both cases above approximately the 98% level.
However, it is confirmed from the experiments that
the current level of technology has achieved this
objective in ALT-J/E.
4.2 Accuracy Rate of the Conversion/Generation
Phase
(1) Conversion/generation accuracy rate
As described before, there seem to be no
problems when all elements necessary for transla
tion are complete within one sentence and each
element is of a simplified form. We shall
evaluate the conversion/generation rate by seeing
whether the expressions taken up in Table 2 can be
accurately translated or not.
We shall assume the number of types of expres
sion to be i, and that i=1 to 9, i.e., any one of
the 9 types of expression as listed in Table 2.
Assume the maximum number of times each expression
can appear in one sentence of the original to be
Ni, and that the average number of times it ac
tually appears to be ni. The probability of the
type of expression will be translated accurately
when it appears is assumed to be pi.
With a given sentence, let us consider the
probability Pi of type i in this sentence being
translated without error. The expression of type
i has the chance of appearing Ni times and ac
tually appears ni times. Therefore, for any one
chance, it appears with the probability of ni/Ni.
When it actually appears, the probability of error
in translation is 1-pi. Thus, the probability of
not making an error in translation for every ap
pearance is:
ni
1 - --- (1-pi).
Ni

When there is no error in all Ni chances, this
sentence can be said to have cleared the type i
expression. The probability of clearing type i
expression is:
ni Ni
Pi={1 - --- (1-pi)} (3)
Ni

Thus, accuracy of processing PG of the con
version /generation phase is:
9
PG = JI Pi (4)
i = 1

(2) Application of Current Level Technology
Based on experience with the Japanese-English
translation experiment system ALT-J/E, the
feasibility of the above model is verified. The
ni and Ni values are given in Table 3. Current
rate of accuracies pi obtained from individual ex
periments on various types of expression are as
also listed in Table 3. Substituting these values
into Eq.(4), PG is obtained as:
PG =0.48 (5)
Translation experiments dealing with newspaper
articles reveal current translation acceptability
rates of about of 40% to 50%. Current analytical
phase accuracy rate is about 90%. Thus, the
translation/generation phase accuracy rate is es
timated to be about 45% to 55%. This coincides
with results of Eq.(5), which was obtained as ac
curacy rate for the individual expressions above.
(3) Relationship between average translation of
expressions under observation and the conver-
sion/generation accuracy rate
Assuming the following relationship to hold
true for translation accuracy pi for all 9 types
of expression:
p1=p2= - - - p9=p (6)
the relationship between p and PG would be as
shown in Fig.4.
Conversion/generation rate at the practical
usage level according to Section 3.2 shows PG >
0.778; therefore the translation accuracy rate for
all types of expressions necessary according to
Fig.4 shows p>_0.96. In contrast, current pi
average accuracy rate p according to the same
diagram shows p=0.89. A difference of about 7%
between the two figures remains.
(4) Bottle Neck of Translation Quality of Each Ex-
pression
In actual practice, weight of importance will
differ according to which of 9 expressions is in
volved. As a third case study, it is attempted to
identify which of the 9 expressions would cause
failure in achieving the target accuracy rate.
This results in the graph shown in Fig.5. If one
out of 9 expressions should fall below line shown
in this figure, achievement of quality required of
a practically applicable product would be impos
sible.
(5) Conversion/generation accuracy improvement
from improved individual expression transla-
tion rates
To examine the effects of translation ef
ficiency rates of various types of expression on
the entire conversion/generation rate, changes in
the accuracy rate pi of the types of expression
under observation were assumed to be dpi, and
changes in the conversion/generation accuracy rate
PG were assumed to be dPG(i). Since the value of
dPG(i) depends also on the value of pj(j=/i), the
following three cases need to be considered.
a) At current technical levels: dPG(1)(i)
When in relation to all i, pi is in the
neighborhood of the present level (Table 2).
b) At average target levels: dPG(2)(i)
　　　　　　When p1=p2= - - - p9=0.96.
c) At maximum efficiency: dPG(3)(i)
When p1=p2= - - - p9=1.0.
Assuming pi=0.01(1%) and calculating dPG(i)
for the above 3 cases results in Fig.6. This
figure reveals that the effects dPG(i) vary,
depending on the type i, of expressions. However,
in the foregoing 3 cases the effects are ap
proximately equivalent.
dPG(2)(i)/dPG(1)(i)=1.8(approximately constant)
dPG(3)(i)/dPG(1)(i)=2.2(approximatety constant)
(7)

4.3 Summary of Quality Model Evaluation Results
(1) Analytical phase technology
To achieve the practical translation quality,
a word accuracy rate of over 99.8% for morphologi
cal analysis, and a dependence and supporting pair
accuracy rate of over 99.4% for dependency
analysis is necessary. Current technical levels
for both morphological and dependency analyses
have almost achieved these rates.
(2) Conversions/generation phase technology
To achieve the target rate of over 77.8% in a
conversion/generation phase, an average transla
tion accuracy rate of over 96% for individually
observed types expression is necessary. Current
values are at about 89%, so an average of about 7%
improvement is required.
With expressions for which translation rates
are already in excess of 90%, it would be dif
ficult to achieve an additional 7% improvement.
For specially structured sentences and connections
in compound sentences, further improvement in
successful translation ratios is necessary. For
consecutive noun compounds, current accuracy rates
range in the neighborhood of 90%. The frequency
of their appearance is high and therefore quality
improvement is vital. Without such improvements,
achievement of practical utility is impossible.

5. Considerations and Tasks to be Confronted
For expressions by types, processing based on
syntactical information will enable analysis and
translation of approximately 80% to 90% of expres
sions. However, when sentences involving a com
bination of such expressions are involved, the
translation rate of each expression needs to be
improved by an average of about 7% each. Also,
the combined expressions need to be appropriately
analyzed and individual expressions decomposed and
re-compounded. To conduct analytical research of
these problems, there is need for further improve
ment of functions which relate to the meaning and
context of the sentences.
Major technical problems to be faced in
upgrading the accuracy and precision of expression
translations will be discussed in the passages to
follow.
(1) Translation of the meanings of joshi and
auxiliary verbs
Joshi are words normally positioned with nouns
and are analyzed as expressing the status of the
noun within the sentence. However, this primarily
expresses the manner in which the speaker is
grasping or handling the phenomenon which is rep
resented by the noun in question; the concept of
the status in terms of meaning and joshi do not
necessarily correspond to each other.
e.g. ﾘﾝｺﾞ　が 　食べたい。
　 Ringo ga 　tabetai.
(I wish to eat an apple)
---"ga" is used as objective. However this is
unusual. "wo" is used as an objective.
この　装置　は　寿命　が　長い
　 　　Kono　sochi wa jumyo ga nagai
(This device has a long lifetime)
---In this case "wa" is not nominative, as it
normally is.
The same can be said of auxiliary verbs. Nor
mally, auxiliary verbs are processed with verbs
and adjectives to express various modalities or
tenses. However, Japanese tenses do not correspond
exactly to English tenses. There is a major dif
ference between Japanese and English in this
regard.
e.g. この　ランプ が　ついたら、
　　 　Kono rampu 　ga　 tsuitara
(If this lamp goes on.)
---"ta", here does not represent the past tense.
Auxiliary verbs(in Japanese) are used to
express the views of the speaker regarding dynamic
attributes (expressed by verbs) or static at
tributes (expressed by adjectives). There is a
substantial difference between this and the views
that are observed or grasped from English
auxiliary verbs.
Errors in translations related to meanings of
joshi and auxiliary verbs in many instances are
caused by this difference in grasping the objec
tives in the Japanese and English languages. In
the future, there will be a need to understand the
theory of language recognition and to review the
English language expression of the meanings of ex
pressions in Japanese.
(2) Translation of noun clauses and compound words
Noun clauses or compounded words with meanings
that are clear that express time, quantity, title
and positions, personal names, or geographical
names can be analyzed in terms of structuring, and
can be translated by using the dependencies among
the semantic attributes of each word. However, if
these noun clauses and compounded words should be
in a mixture of expressions or within the context
of an entire text and the interpretation should
change, there is difficulty in extracting noun
clauses and compounded words and in selecting ap
propriate translations which match the meanings
within the context of the entire text. Especially
when many ordinary words have been combined to
form noun clauses or compounded words, it
frequently becomes difficult to make a selection
from a number of interpretations. Also, one
characteristic of newspaper articles is to
economize on space by frequently using noun
clauses and compounded words that are compressed
versions of normal sentences, which makes transla
tion difficult.
(3) Connections and embedded sentences
Texts which have connections and embedded sen
tences leave many a problem in the area of
processing for both meaning and context. Previous
translations handled this by translating each unit
sentence one by one and later connecting these by
appropriate conjunctions or relative pronouns.
From the English side however, these do not always
match each unit sentence; thus, it is frequently
necessary to use participial or gerund construc
tions to match the meanings or to substitute ad
verbial or noun clauses to complete the transla
tion. Occasionally, this type of distinction can
not be made on semantic attributes alone; it be
comes necessary to look at the relationships be
tween phenomena combining subjects, objects and
declinable words. A look at the speaker's value
judgments regarding phenomena (such as the
relationship between the main text and subordinate
clause) is required to arrive at an appropriate
translation.
(4) Ellipsis and paratactical constructions
　　　　　　When elements of texts which are felt to be
necessary in English are missing in the original
Japanese text, there is a need to supplement cer
tain elements to the text when translating. With
the current state of the art, if there should be
elements in the preceding sentence of the same
paragraph that can be referred to, translation can
be handled by such supplementation. It is also
possible to supplement ellipsis of indeclinable
parts of speech and inflexions.
Example;
Ａ社はＢを発売する。Ｂは従来の装置を改良した
ものである。五月から発売する。(Company A will
place B on sale. B is the equipment which is a
version that #1 improved the previous version.
#2 put #3 on sale from May.)
The first sentence has no need of supplemen
tation because it has both the subject and object
in place. The second sentence does not state
who(#1) improved the previous product. The third
sentence does not state clearly who(#2) will be
selling what(#3). #2 and #3 can be supplied from
the first sentence but #1 cannot be determined.

However, ellipsis of joshi and verbs are apt
to disrupt analysis of the sentence structure it
self, making translation difficult.
The same applies in the case of paratactical
construction. Expressions consisting of a series
of words in a row would be comparatively easy to
handle (Example 1 below). However, when expres
sions constituting elements of sentence structure
follow one after another in a series, analysis of
the structure itself becomes difficult (Example 2
below).
Example 1. Ａ社、Ｂ社が共同開発した。(Company A
and Company B undertook developpment jointly)
Example 2. Ａ社がＢを改良したＣ、Ｄに機能を追加
したＥならびにＧと融合させたＨなどの多くの新
製品を発売した。(Company A placed on sale
many new products, including C, an improved
version of B, E with functions added to the
conventional D, and H which is a conglutina-
tion with G)

In sentences of paratactical construction,
joshi and verbs are frequently omitted. This
problem frequently coincides with the problem of
interpretation of expressions containing ellipsis.

5. Summary
An evaluation method is proposed which es
timate translation quality based on translation
technologies and source text characteristics. Ap
plying this method to translation of lead texts in
newspaper articles with the purpose of achieving
practical translations without pre-editing,
problems which need to be solved in comparing
characteristics of text expressions and current
translation technology are identified.
Assuming 70% acceptability to be a practical
level of quality, the number of words and bun
setsu indicate that the morphological and depen
dency analysis technologies would require accuracy
rates per word and per bunsetsu as high as 99.8%
and 99.4%, respectively. Current technology has
already reached these levels. But the precision
of translation for individual expressions is lower
than expected values so acceptability rates for
the entire text drop to between 40% and 50%. To
achieve an acceptability rate of 70%, it is neces
sary to achieve an average 7% improvement in all 9
types of expressions mentioned in this study and
thus achieve a individual translation ratio of
96%.
To upgrade the translation rates of these ex
pressions, it will be necessary to establish tech
niques for analysis of the meanings of joshi,
translation of expressions that combine noun
clauses and compounded words, handling connections
and embedded phrases, and verification of ellipsis
and paratactical construction.

References
(1) Nagao, Tsujii, Nakamura, Sakamoto, Toriumi and
Sato, "Outline of Science and Technological Agency
Mechanical Translation Project", Information
Processing, Vol.26, No.10, pp.1203-1213(1985)
(2) Nagao and Tsujii, "World Selection and Process
of Structural Conversion in Machine Translation",
Information Processing, Vol.26, No.11,pp.1261-1270
(1985)
(3) Uchida et al,"Japanese/English Machine Trans
lation System ATLAS/U",Information Processing As
sociation, Natural Language Processing Research
Society Data,No.299-3(1982)
(4) K.Muraki,"Knowledge Base and the Japanese
English Machine Translation System Using Indepen
dent Intermediate Expressions for Words", Nikkei
Electronics,1984.12/17,pp.195-220(1984)
(5) J.Sigurson and R.Greatrex,"Machine Translation
of On-line Searches in Japanese Databases",The Re
search Policy Institute Lund University, Sweden,
1987
(6) X.Hung,"A Machine Translation System for
Monolingual Users",Australian-Japan Joint Sym
posium on Natural Language Processing, pp.50-56
(1989)
(7) ALPAC(Automatic Language Processing Advisory
Commitee) Report,National Research Council,1966
(8) M.Nagao and J.Tsujii,"Evaluation of the J-E
Translation Results in Mu-Project", Information
and Processing Society of Japan, Natural Language
Processing 47-11,pp.79-88(1985)
(9) S.Ikehara, M.Miyazaki, S.Shirai and Y.Hayashi,
"Recognition in Language and Multi-level Transla
tion Method", Journal of Information Processing
Vol.28,No.21,pp.1269-1279(Dec.1987)
(10) S.Ikehara, M.Miyazaki, S.Shirai and A.Yokoo,
"An Approach to Machine Translation Method Based
on Constructive Process Theory",Review of the ECL
Vol.37,No.1,pp.39-44(1989)
(11) Saeki, "Opinions from the Working Place--From
the Viewpoint of Mechanical Translation--", The
Journal of the Institute of Electronics, Informa
tion and Communication Engineers, Vol.71,No.11,
pages 1221-1224(1988)