NRLgate -
Plagiarism by Peer Reviewers
Sections 2.8 thru 2.16
This page is part of the NRLgate Web site presenting evidence of
plagiarism among scientific peer reviewers involving 9 different peer review
documents of 4 different journal and conference papers in the fields of
evolutionary computation and machine learning.
This page contains sections 2.8 through 2.16 of "Evidence of plagiarism
in reviews A and B of a paper on empirical discovery submitted to the Machine
Learning Conference."
Go to top of NRlgate Home Page
Go to Abbreviated Table of Contents
Go to Detailed Table of Contents
Go to Complaint Letter
to the Evolutionary Computation journal
Go to top of Previous Page
Go to top of Next Page
2.8. Both reviewers A and B both neglected section 5 of the paper review
form
Reviewer A's entire response to section 5 (entitled "General comments
for the author(s)") for my MLC paper on empirical discovery is as follows:
- See above.
Reviewer B's entire response the same section of the paper review form for
this paper is as follows:
- Same comments as above.
(Emphasis added).
Of course, scientific peer reviewers sometimes neglect a section of their
paper review forms. Two reviewers of the same paper may sometimes both decide
skip some section of their paper review forms. Two reviewers of the same
paper may sometimes both decide to skip the very same section of their paper
review forms.
How common is it for peer reviewers not to offer any suggestions to the
author on their paper review forms?
I was the general chair of the Genetic Programming 1996 Conference and have
access to a computer file containing the 64,109 words of the 316 paper review
forms from the 86 peer reviewers of the genetic programming papers at the
GP-96 conference.
Section 8 (entitled "Suggestions to Author") on the 16-part GP-96
paper review form was entirely blank on only 5% (19 of 316) of the review
documents. An additional 4% (14 of 316) of the reviews did not provide any
advice to the author in section 8, but nonetheless over-diligently filled
in the blank line of section 8 with a vacuous phrase (e.g., "see above").
That is, 91% of contemporary peer reviewers from this field of science affirmatively
offered some suggestions to the author.
2.9. Both reviewers A and B employed the same word while neglecting section
5 of the paper review form
Many different words can be used to express the same idea --- even the
idea of saying nothing.
As previously mentioned, 4% (14 of 316) of the reviews did not provide any
advice to the author in section 8 of the GP-96 paper review form, but nonetheless
over-diligently filled in the blank line of section 8 with a vacuous phrase.
The frequency of various vacuous phrases included the following:
- 6 said "none"
- 1 said "no"
- 1 said "see next item"
- 1 said "see #5"
- 2 reviews (the same reviewer) said "All my remarks are in the section
below"
- 1 said "see above 5 and 6."
- 1 said "see above."
- (Emphasis added).
Thus, the chance is only 2 in 316 that a reviewer would not give the author
any advice, but nonetheless over-diligently fill in the blank on the paper
review form with a phrase incorporating the word "above."
This is not the only occasion when we encounter such over-diligence in filling
in every blank line on a form. See
section 5.7. See section
7.12.
2.10. The opening sentences of both reviews A and B gratuitously provided
the same unrequested information
Question 1 of the MLC paper review form asks the reviewer to evaluate the
"significance" of the work described in the submitted paper.
- Significance: How important is the work reported?
Does it attack an important / difficult problem or a peripheral / simple
one? Does the approach offer an advance.
(Emphasis added).
Reviewer A begins his review,
- This paper reorts on a technique of learning concepts
expressed as LISP expression using genetic algorithms. This is a
topic of general interest.
(Spelling error of "reorts" in original).
(Emphasis added).
Reviewer B's begins his review,
- The author describes an interesting set of
results using GAs for empirical discovery and concept formation.
(Emphasis added).
Notice how the opening sentences of both reviews ignored the question concerning
"significance" that was actually posed by paper review form. Instead,
both of these opening sentences were unresponsive to the question
being asked. Both reviews began by gratuitously providing an unrequested
summary of the subject matter of the paper.
Is it commonplace for peer reviewers in this field of science to gratuitously
and unresponsively begin their answer to a paper review form's question
on significance on the paper with an unrequested summary of the paper?
We again make reference to the computer file containing the 64,109 words
of the 316 paper review forms from the 86 peer reviewers of the genetic
programming papers at the Genetic Programming 1996 Conference.
Question 2 on the paper review form used by the GP-96 conference asked about
the "significance" of the paper.
- Significance of the Problem: Is the subject of this paper important?.
98.4% (311) of these 316 review documents began by addressing the
question that was actually asked by the paper review form. That is, they
said something related to whether the work described in the paper was scientifically
important.
Only 1.6% (5) of these 316 review documents began by gratuitously and unresponsively
providing an unrequested summary of the subject matter of the paper.
Both reviewers A and B of my MLC paper on optimal control strategies appear
to belong to a small minority of reviewers. Isn't it improbable that two
independent-acting reviewers of the same paper would both belong this particular
small minority?
It would seem unlikely to draw 2 peer reviewers with this particular unusual
behavior on the same submitted paper. However, if one peer reviewer were
writing his review with the already written review of another review of
the same paper in front of him, the suggestive power of the wording of the
first review might cause him to pattern his opening sentence after the first
reviewer's opening sentence.
This is not the only occasion when we encounter opening sentences of two
reviews of the same paper that unresponsively provide a summary of the submitted
paper instead of providing the requested evaluation of its significance.
As will been seen elsewhere, both reviewers X and Y of my MLC paper on optimal
control strategies belong to this same small minority. See
section 3.9.
2.11. Both reviewers X and Y substituted "Genetic Algorithms"
or "GA" in lieu of the author's chosen term
Reviewer A entire response to section 1 for this paper on genetic programming
(GP) follows:
- This paper reorts on a technique of learning concepts expressed
as LISP expression using genetic algorithms. This is a topic of general
interest. The methodology adopted prevents a clear assessment of how much
over advance this approach represents.
- (Spelling error of "reorts" and grammatical error of "over
advance" in original).
(Emphasis added).
Reviewer B's entire response to section 1 (entitled "significance")
for this paper on genetic programming (GP) follows:
- The author describes an interesting set of results using GAs
for empirical discovery and concept formation.
(Emphasis added).
Notice that both reviewers A and B imposed this substitution of "genetic
algorithm" or "GA" for the author's chosen term (perhaps
offensive to both) that actually appears in the submitted paper.
This is not the only occasion when we encounter two peer reviewers of the
same paper both belonging to the small minority of peer reviewers who impose
their own term in this manner in lieu of the author's chosen term. As will
been seen elsewhere, both reviewers X and Y of my MLC paper on optimal control
strategies and reviewer T2 of the submitted TAI paper have this same propensity.
See section 3.10. See
section 5.4.
2.12. Lock-step treatment by both reviewers A and B in section 2 of the
paper review form
The joint usage of the same noun ("approach") at the beginning
of these sentences is yet another similarity between the two reviews, as
is the agreement between the two reviewers that the application described
in the paper is "original" ("new").
Reviewer B's entire response to section 2 (entitled "Originality")
of the paper review form is as follows:
- The approach is original and quite interesting.
- (Emphasis added).
Reviewer A's entire response to this section entitled "Originality"
for my MLC paper on empirical discovery is as follows:
- The approach has been reported on previously in MLW89. The
applications here are new. (Emphasis added).
2.13. Both reviewers A and B used a similar hostile and antagonistic
tone
Reviewers of scientific articles generally maintain some level of civility
in their written reviews (even when they are making negative comments and
judgments about a paper). Of course, a small minority of peer reviews drift
over the line. Both reviewers A and B seem to belong to this minority.
Reviewer A of my MLC paper on empirical discovery says,
- This is extremely suspicious ...
Reviewer A says,
- (who cares?)
Reviewer A says,
- results are claimed to appear
Reviewer A says,
- The presentation suffers from an abundance of irrelevant
details
Reviewer B of my MLC paper on empirical discovery says,
- The paper is annoyingly spotty.
Reviewer B also says,
- The reader is left with the possible interpretation that, after
weeks of trying, the GA managed to get it right once!
Keep in mind that the total size of review A is only 218 words and review
B is only 138 words.
How common is it for peer reviewers to be hostile in this way?
We again make reference to the computer file containing the 64,109 words
of the 316 paper review forms from the 86 peer reviewers of the genetic
programming papers at the Genetic Programming 1996 Conference. All of the
above highlighted words or phrases are almost always pejorative in the context
of a review (as they indeed were in the reviews in the GP-96 computer file
in which they appeared). The frequency of occurrence of the following words
is shown below:
- suspicious --- 0
- claimed (or claim or claims) --- 4
- (who cares?) --- 1
- irrelevant (or irrelevancies) --- 0
- annoyingly (or annoy or annoying) --- 0
Thus, both reviewers A and B of my MLC paper on optimal control strategies
appear to belong the small minority (about 2%) of peer reviewers with these
hostile words.
It would seem unlikely to draw 2 peer reviewers with this particular hostile
tone on one particular submitted paper. However, if one peer reviewer were
patterning his review after the words and tone established by an already
written review of the same paper, the probability of a hostile tone migrating
from one review to another is no longer a low percentage.
This is not the only occasion when we encounter two peer reviewers of the
same paper both belonging to the small minority of peer reviewers who have
a hostile tone. Reviewers #1, #2, and #3 of my paper submitted to the Evolutionary
Computation journal seem to belong to this same minority. See
section 4.8. See Section
7.8.
2.14. Both reviewers A and B offered similar advice about statistics, using
the same trio of phrases
Reviewer A also says,
- The paper suffers from a lack of data.
- ...
- The interesting data concerns the general form of learning curves over
a number of runs.
- In order to judge, it would be necessary to see the results compared
against an alternative search technquie, perhaps even random search.
- (Grammatical error of "authors" in original).
- (Emphasis added).
Reviewer B says,
- Rather, he needs to give us some graphs, charts, data, etc.
that indicate how the GAs perform on-the-average on these problems.
The reader is left with the possible interpretation that, after weeks
of trying, the GA managed to get it right once!
- (Emphasis added).
In terms of semantics, reviewers A and B cover 3 points, in the same order,
but using different words.
First, reviewer A complains about
- a lack of data
and reviewer B complains about
- he needs to give us some graphs, charts, data
Second, review A wants
- learning curves over a number of runs
while reviewer B wants
- graphs, charts, data, etc. that indicate how the GAs perform on-the-average
Third, reviewer A talks about
- random search
while reviewer B
- The reader is left with the possible interpretation that, after
weeks of trying, the GA managed to get it right once!
The issue here is not, of course, whether the paper genuinely needed any
such "data," but that two reviewers zeroed in on the same defect
in the space of two very short reviews (218 and 138 words, respectively)
using the same trio of words. It has been my experience (and probably the
reader's own experience concerning reviews of his own papers) that even
when a submitted paper has a clear deficiency, it is uncommon for two reviewers
(out of two) to specifically mention that particular deficiency. Independent
reviewers almost always make very different comments, in different words,
and in different ways --- particularly in short reviews.
2.14.1 The joint advice of reviewers A and B was at variance with the
prevailing practice concerning empirical discovery
Now let's discuss whether the question of whether this submitted paper genuinely
needed "on-the-average" "data" "over a number of
runs." Would most contemporary members of the machine learning community
would agree with reviewers A and B that this particular paper on empirical
discovery and concept formation needed "on-the-average" "data"
"over a number of runs."
The two simpler problems in my Machine Learning Conference paper were benchmark
problems taken directly from well-known and well-regarded works in the machine
learning community. Neither of these well-known papers contained any "on-the-average"
"data" "over a number of runs" for their respective
problems. One problem treated in my MLC paper was the most difficult problem
contained in the only existing book-length treatment of empirical discovery
(Langley et al. 1987). That 357-page book did not contain any such "on-the-average"
"data" "over a number of runs" for any of the several
dozen problems in the book. A second problem in my submitted MLC paper was
harder than any of the problems in Langley's book and was chosen because
the methods in Langley's book would not work on it. Pat Langley is a leading
figure in the machine learning community. Pat Langley was the founding editor
of the Machine Learning journal and has written editorials on what
good experimental papers on machine learning should contain. A third problem
treated in my MLC paper came right out of a widely-cited paper by Quinlan
on decision trees (1986). Quinlan's paper did not contain any such "on-the-average"
"data" "over a number of runs" for his particular problem.
Ross Quinlan is a leading figure in the machine learning community. The
fact is that for the particular field of empirical discovery and concept
formation covered in my MLC paper on empirical discovery, prevailing practice
did not call for such "on-the-average" "data." Needless
to say, neither Langley, Quinlan, nor I are actively opposed to collecting
and reporting such averages. However, unlike some other areas of machine
learning research, the primary question that researchers usually ask is
whether they can find any satisfactory model for the given observed data
(and, usually, whether it generalizes to other previously unseen examples
of the data). The question of the average probability of emergence of a
suitable empirical model over a number of runs is secondary. Indeed, when
one is doing empirical discovery, one is usually very satisfied to find
any one suitable model. The frequency of emergence is a perfectly legitimate
concern, but it is secondary to the main issue in empirical discovery. The
even more important point is that such averages were not contained in the
well known benchmark sources from which I obtained these 2 problems. I think
that, for the particular fields of empirical discovery and concept formation
covered in this particular paper on empirical discovery, reviewer A and
B's joint complaint is a minority point-of-view in the machine learning
community. (It would not, however, probably be a minority point of view
in the genetic algorithm community, where collection of such "on the
average" "data" is far more prevalent). Of course, the important
point here (in this discussion of plagiarism) is not whether minority point-of-view
is right or wrong. The important point is that both reviewers raised this
same complaint in their very short reviews and that their joint point-of-view
would not likely be shared by other members of the machine learning community
for the particular problems in the submitted paper. It would seem unlikely
to draw 2 peer reviewers at the Machine Learning Conference with this particular
point-of-view on one particular submitted paper.
2.15. Reviewers A and B both ignored the intellectual content of the
submitted paper
Both reviews A and B are notable in that they completely avoided any discussion
of the substance of my MLC paper on empirical discovery and concept formation.
The entire focus of the comments was directed to aspects of the paper that
could be obtained by merely browsing the paper at a high level.
What was the substance of my MLC paper?
First, it presented a new way (using genetic programming) to create decision
trees. The paper demonstrated the effectiveness of the new technique by
solving the same example problem treated in Quinlan's well-regarded and
widely cited article on decision trees (1986).
Second, the paper presented a new way (using genetic programming) to build
models from empirical data (data-mining). It solved the hardest problem
(the Kepler's law problem) contained in the only book-length treatment of
empirical discovery existing at the time (Langley et al. 1986).
Third, the paper then went on to solve a problem that was harder than any
in the book of Langley et al 1986 and that clearly could not be solved by
the technique of that book, namely a problem involving finding a mathematical
relationship between the rate of inflation in an economy and other economic
data.
Thus, the submitted 12-page paper offered a new way (i.e., genetic programming)
to tackle existing recognized problems in two recognized area of machine
learning.
There is nothing in either review A or B that deals with the actual substance
or subject matter of my paper on empirical discovery and concept formation.
2.16. Reviewer B's quotation of "on one run ..." is inaccurate
--- thereby suggesting the time sequence of the plagiarism
The joint action of reviewers A and B in elevating the same unmemorable
3-Word prepositional phrase "In one run, ..." to quotation marks,
their joint grammatical error in using the ellipsis, and their joint choice
of the same section of their paper review forms to place this quotation
together indicate that the second reviewer had the exact text of the first
review in front of him when he wrote his review.
This modus operandi of the collusion here was not of "cutting and pasting"
entire sentences of one review in order to construct a second review. Instead,
the second reviewer reworded (and freshly typed) the first review --- using
the already written review as a template for thoughts, choice of words,
punctuation, grammar, grammatical errors, and placement of items within
his paper review form.
The transformation of each already written review into its slightly altered
form required the expenditure of some effort (albeit somewhat mechanical).
The transformation of one already written review into its paraphrased form
required conscious effort expended by a person who knew he was acting improperly
at the time he did it. Of course, the transmission of an already written
review to a second peer reviewer was also a conscious activity executed
by a person who knew he was acting improperly at the time he did it.
The same modus operandi of collusion was employed by reviewers A and B of
my MLC paper on empirical discovery, by reviewers X and Y of my MLC paper
on optimal control strategies, reviewers #1, #2, and #3 of my paper submitted
to the Evolutionary Computation journal, and reviewers T2 and T3 of the
paper submitted to the Tools for Artificial Intelligence conference.
The fundamental purpose of quotation marks is to capture words that are
so significant that only the author's precise words do justice to the important
idea involved. When writers quote such memorable words, they usually carefully
check to be sure that they have correctly transcribed the quoted words.
But, notice the 3 words inside reviewer B's quotation marks for my MLC paper
on empirical discovery:
- "on one run ... "
(Quotation marks and ellipsis in the original).
(Emphasis added)
Where did reviewer B get these 3 words?
These 3 words do not appear anywhere in my submitted 3,118-word MLC paper
on empirical discovery!
In fact, the only similar phrase appearing anywhere in my paper is as follows
(as accurately quoted in review A):
- "In one run, ..."
(Quotation marks and ellipsis in the original).
(Emphasis added)
Reviewer B's error in changing "in" to "on" suggests
the sequential order in which the plagiarism took place. Reviewer A actually
extracted the 3 words "In one run" in my submitted MLC paper.
For whatever reason, reviewer A decided that these 3 words were worthy of
quotation marks. Reviewer A then made the grammatical error of using the
ellipsis. Reviewer B had a copy of reviewer A's already-written review in
front of him (either on paper, on a computer screen, or via e-mail) and
embarked on a process of slightly reworking and altering A's words so that
his review would look different from review A. In the process of hastily
paraphrasing and mechanically tweaking of reviewer A's words, reviewer B
carelessly plowed right past the onset of the quotation marks and continued
his tweaking. Reviewer B plowed past the quotation marks because he was
not the person who originally decided that the quoted works were memorable.
Reviewer B was merely engaged in a thoughtless and mechanical task of changing
words around in order to make his review look different from review A. In
his mechanical haste, reviewer B made his review look different from the
submitted MLC paper --- instead of different from review document A.
This error establishes the order of the plagiarism. If reviewer B had been
the one who actually read my paper and made the original error of converting
"in" to "on," then review A would also have contained
the erroneous word "on."
Review B is not the only occasion when we received a peer review containing
the phrase,
- "on one run ... "
(Quotation marks and ellipsis in the original).
(Emphasis added)
As will be seen later, a 31-word review (T2) of an entirely different paper
submitted to a different (but contemporaneous) scientific conference also
contained
- "on one run ... "
(Quotation marks and ellipsis in the original).
(Emphasis added)
Review T1 contained the phrase"on one run" even though this phrase
"on one run" did not appear in the paper submitted to this second
conference. This error by reviewer T2 for the TAI conference will be discussed
later because it points to the identity of the plagiarizers (since only
2 persons were reviewers for both the MLC and TAI conferences and since
these same 2 persons are the only persons from the MLC and TAI conferences
who are also among the editors and editorial board of the Evolutionary
Computation journal). See section
5 through section 5.2.
Author: John R. Koza
E-Mail: NRLgate@cris.com
Go to top of NRlgate Home Page
Go to Abbreviated Table of Contents
Go to Detailed Table of Contents
Go to Complaint Letter
to the Evolutionary Computation journal
Go to top of Previous Page
Go to top of Next Page