NRLgate -
Plagiarism by Peer Reviewers

Section 4.2

This page is part of the NRLgate Web site presenting evidence of plagiarism among scientific peer reviewers involving 9 different peer review documents of 4 different journal and conference papers in the fields of evolutionary computation and machine learning.

This page contains section 4.2 of "Evidence of plagiarism in reviews #1, #2, and #3 of a paper on electrical circuit design submitted to the Evolutionary Computation journal."


Go to top of NRlgate Home Page
Go to Abbreviated Table of Contents
Go to Detailed Table of Contents
Go to Complaint Letter to the Evolutionary Computation journal
Go to top of Previous Page
Go to top of Next Page

4.2. The telltale > e-mail symbols indicate that review #1 was transmitted to peer reviewer #2 by e-mail and that reviewer #2 had review #1 in front of him when he wrote his review. This fact is confirmed by the lock-step mirroring of thoughts, words, and phrases by reviewers #1 and #2 in ALL 5 SECTIONS of the journal's paper review form.

Upon receipt of a paper, the editorial leadership of the Evolutionary Computation journal picks three peer reviewers and sends a physical copy of the paper to each reviewer (usually by U. S. mail). At the same time, the journal's editor sends the reviewer, by electronic mail, the journal's standard 7-question paper review form. The reviewers respond by sending their review to the journal's editor by e-mail.

Reviewers ordinarily create their reviews and send them to a journal editor in one of two approaches using e-mail.

First, a reviewer could write his review as a direct e-mail "reply" to the paper review form sent to him by the editor. In that case, the standard questions on the journal's paper review form would become flagged by automatically produced >'s at the time when a recipient "replies" to the message containing the paper review form. The >'s indicate that the material (i.e., the paper review form) that was "received text" for the reviewer. In such a "reply," the reviewer's responses to the questions on the paper review form appear without the >'s because they are the reviewer's "reply" to the message that he received. (As will be seen, review #2 of my ECJ paper is an example of this commonly used approach for responding to a paper review form).

Second, a reviewer could copy the paper review form into his own word processor and use his word processor to insert his responses into the appropriate places in the form using his word processor. The reviewer then send the entire document as a direct e-mail message to the editor (without using the "reply" feature of his e-mail system). In that case, neither the questions nor answers would be flagged with >'s. (As will be seen later, review #3 of my ECJ paper is an example of this approach).

Now let's focus on the first approach, namely the "reply" method.

Most e-mail software supports the convention of flagging "received text" with a ">" symbol in the left margin at the time when a recipient "replies" to the "received text."

The automatically produced >'s appear on the responses that the recipient of message writes as a "reply" to a message that he received. That is, the >'s do not become attached to the responses written by a reviewer.

When a peer reviewer answers a question that has been e-mailed to him using the "reply" feature of their e-mail software, there is a clear distinction between the "received text" originally received by him (i.e., the 7 standard questions on the paper review form that he previously received from the journal's editor) and the reviewer's response (i.e., his evaluation of the submitted paper that he is sending to the journal's editor). The only way for the automatically produced >'s to become attached to the answers written by a particular reviewer is for the reviewer's answers to be transmitted to a second person who, in turn, types something in response and returns it by e-mail using the "reply" feature.

However, for all 5 substantive questions on the journal's paper review form, reviewer #1's responses to the standard questions on the paper review form are flagged with >'s.

This automatic electronic flagging establishes that reviewer #1's responses were "received text" to someone who then "replied" to reviewer #1's responses. Specifically, reviewer #1's responses were dispatched as outgoing e-mail to reviewer #2. Reviewer #2 then reworded each of reviewer #1's responses --- treating each of reviewer #1's responses as "received text." Reviewer #2 then sent back his responses as an e-mail "reply."

The fact that reviewer #2 had the text of review #1 in front of him while he wrote his review is further indicated by the fact that, for all 5 sections of the paper review form, there are common thoughts, words, and phrases in both reviews.

4.2.1. For section 2, the telltale electronic >'s (as well as groups of mirrored words) indicate that review #1 was supplied by e-mail to peer reviewer #2

Now let's look at the opening sentences of reviews #1 and #2 for section 2 of the journal's paper review form.
NOTE: The quoted material throughout this entire document concerning the 3 reviews of my paper submitted to the Evolutionary Compuation journal appears exactly as I received it by e-mail. There are no changes in the paragraph marks (carriage returns), spacing, >'s, or other punctuation. The only exceptions are occasional deleted material (indicated by ellipsis) and the occasional reconstruction of an isolated word that was garbled in e-mail transmission (indicated by square brackets). The reader using a WWW browser should widen his viewing window so that each line of original text appears on only one line.

Question 2 of the journal's paper review form and the first few lines of reviewer #1's response are follows.
PRESENTATION. Is the paper well-organized and well-written? Does it
use standard terminology? Does the paper describe the method(s) in
sufficient details for readers to replicate the work? Does it clearly
explain the representation and algorithms used? Does the paper include
enough examples?

> The presentation style is very uneven for a scientific journal article.
> Although there are many carefully written sections, there are parts
> which:
> - are obvious overstatements to any serious reader: E.g., p. 1
> "Genetic programming provides a way to search the space of all
> possible programs composed of ..." when, in fact, it searches
> a highly constrained space of Lisp expressions.

(Emphasis added)
Question 2 of the journal's paper review form and the first few lines of reviewer #2's response are follows.
> PRESENTATION. Is the paper well-organized and well-written? Does it
> use standard terminology? Does the paper describe the method(s) in
> sufficient details for readers to replicate the work? Does it clearly
> explain the representation and algorithms used? Does the paper include
> enough examples?

The presentation could be strengthened considerably by the elimination
of some gratuitous overstatements. Some examples:

p. 3, line 1: GP is cla)lot to search the space of *all possible*
programs defined over the specified base functions. However, in
previous papers, in particular Koza's June 1990 Tech Report on GP, it is
stated that an upper bound is usually placed in the depth of all
expressions created by crossover (usually 15). ...

(Emphasis added)
First, reviewer #1 complains of
obvious overstatements
while reviewer #2 complains of
gratuitous overstatements
Second, reviewer #1 says,
to search the space of all possible programs
while reviewer #2 modifies this 8-word phrase by adding 2 asterisks and says,
to search the space of *all possible* programs
Third, reviewer #1 rebuts the allegedly erroneous claim in the submitted paper,
when, in fact, it searches a highly constrained space of Lisp expressions

(Emphasis added)

while reviewer #2 says,
However, in previous papers, in particular Koza's June 1990 Tech Report on GP, it is stated that an upper bound is usually placed in the depth of all expressions

(Emphasis added)

Notice that all 3 criticisms were raised in the same order.

Notice also that all 3 of these criticisms could --- with equal logic --- have been reasonably raised in at least 3 other parts of the paper review form (i.e., as "evaluation," "discussion," or "general"), yet they were raised by both reviewers #1 and #2 in the same section of the paper review form (i.e., "presentation").

Reviewer #2 is mirroring the same thoughts using the same words, in the same order, and in the same spot of the paper review form that were used by reviewer #1, in his review.

The reason is that reviewer #2 had the already written words of reviewer #1 when he wrote his review. We know this not only because of the mirrored words, but because the telltale electronic >'s indicate that review #1 was supplied by e-mail to peer reviewer #2 before reviews #1 and #2 were transmitted by e-mail to the journal's editor.

4.2.2. For section 3, the telltale electronic >'s (as well as the reference by reviewer #2, in his review, to an issue raised by reviewer #1, in his review) indicate that review #1 was supplied by e-mail to peer reviewer #2

We won't repeat here all the points in the 5 parts of section 4.1 concerning the many similiarities in this section of the paper review form. However, we reshow the quotation so that the reader can see how the telltale electronic >'s confirm the analysis of section 4.1. The reason why reviewer #2 referred, in his review, to an issue of hardness raised by reviewer #1, in his review, is that reviewer #2 was looking right at the already written words of reviewer #1 when he wrote his review. We know this because the telltale electronic >'s indicate that review #1 was supplied by e-mail to peer reviewer #2 before reviewer #2 wrote his review.

Reviewer #1 starts his response to question 3 of the journal's paper review form,
EVALUATION. Does the author carefully evaluate the approach? Does the
paper include systematic experiments, a careful theoretical analysis, or
give evidence of generality?

> This is an area of concern. There is no sense of the underlying
> difficulty of the problem. Is this a hard problem for current
> engineering methods? Given the goal of finding a Lisp expression
> for the system response, how complex is the space being explored
> by GP? How would random search over the same bounded space of
> Lisp expressions perform?

(Emphasis added)
Reviewer #2 responds to question 3,
> EVALUATION. Does the author carefully evaluate the approach? Does the
> paper include systematic experiments, a careful theoretical analysis, or
> give evidence of generality?
...
p. 19, fig. 10: The figure shows that finding the correct function H(t)
is not necessary in order to give good input response. This again
raises the question: How hard is it to find some response function that
gives an adequate input response, i.e., how dense is the search space
with good sol[utions]?

(Emphasis added)

4.2.3. For section 4, the telltale electronic >'s (and similar thoughts) indicate that review #1 was supplied by e-mail to peer reviewer #2

Question 4 of the journal's paper review form and the first few lines of reviewer #1's response are follows.
DISCUSSION. Does the paper discuss relevant earlier work, noting
similarities, differences and progress? Does it discuss the limitations
of the approach along with its advantages? Does it consider the
implications of the work and outline directions for future work?

> There are relatively few other groups working explicitly in the
> area of evolving Lisp code, so the most immediate references and
> comparisons are to e!xk.Hwork by the authors. ...

(Emphasis added)
Question 4 of the journal's paper review form and the first few lines of reviewer #2's response are follows.
> DISCUSSION. Does the paper discuss relevant earlier work, noting
> similarities, differences and progress? Does it discuss the limitations
> of the approach as well as its advantages? Does it consider the
> implication of the work and outline directions for future work?

The discussion of previous work is confined to a litany of previous case
studies..

(Emphasis added)
Again, the telltale electronic >'s indicate that review #1 was supplied by e-mail to peer reviewer #2 before reviews #1 and #2 were transmitted by e-mail to the journal's editor.

4.2.4. For section 5, the telltale electronic >'s (and similar words in similar locations) indicate that review #1 was supplied by e-mail to peer reviewer #2

Question 5 of the journal's paper review form and the first few lines of reviewer #1's response are follows.
GENERAL. Does the paper make a significant, technically sound
contribution to the field of Evolutionary Computation?

> Although I'm excited and interested in the underlying core research
> going on here, I don't think the paper as it stands measures up to
> a good scientific journal paper. ...

(Emphasis added)
Question 5 of the journal's paper review form and the first few lines of reviewer #2's response are follows.
> GENERAL. Does the paper make a significant, technically sound
> contribution to the field of Evolutionary Computation?

The paper makes an interesting contribution to the further development
of the GP paradigm, but the presentation makes it seem a bit premature
for a journal article.

(Emphasis added)
Notice that the first sentence of both reviews contain 5 elements in the same order. Two of the 5 elements employ the same words in the same location. For three of the 5 elements, there is a mechanical exchange of words.

Again, the telltale electronic >'s indicate that review #1 was supplied by e-mail to peer reviewer #2 before reviews #1 and #2 were transmitted by e-mail to the journal's editor.

4.2.5. For section 1, the telltale electronic >'s indicate that review #1 was supplied by e-mail to peer reviewer #2

Question 1 of the journal's paper review form and the first few lines of reviewer #1's response are follows.
GOALS AND CONTRIBUTIONS. Does the author clearly state the research
goals of the work? Does the paper clearly indicate what the
contributions are?

> The goals and contributions are clearly stated. I have serious
> concerns about the claims made and the significance of the
> contributions as discussed below.
...
(Emphasis added)
Question 1 of the journal's paper review form and the first few lines of reviewer #2's response are follows.
> GOALS AND CONTRIBUTIONS. Does the author clearly state the research
> goals of the work? Does the paper clearly indicate what the
> contributions are?

The goals of the research on genetic programming (GP) are stated clearly
in the introduction: ...

(Emphasis added)

4.2.6. It would be difficult for 2 or more members of the geographically dispersed editorial Board of the Evolutionary Computation journal to collude on the peer reviewing of a particular paper without involvement by the editor. None of the ordinary methods of e-mail transmission by reviewer #1 to the editor would cause the >'s to become attached to the responses written by reviewer #1.

The 32 individual members of editorial board of the Evolutionary Computation journal are geographically-dispersed and institutionally-diversified.

The existence of a paper submitted to a scientific journal is ordinarily known only to the submitting authors and the journal's Editor-In-Chief and the associate editor who is designated to handle the particular paper. Unless the editors happen to appoint them to be one of the peer reviewers of the paper, individual members of the editorial board ordinarily have do not know about the existence of a submitted paper. Moreover, the editors of a journal do not ordinarily tell a reviewer the names of the other reviewers of that paper.

Although unlikely, it is possible that two peer reviewers could become aware that they were both reviewing the same paper during the limited time period for reviewing a particular paper. If that were to happen, it is conceivable that one peer reviewer could provide his already written review to the second peer reviewer so that the second reviewer could use it as a template in writing his review. However, even if two or more members of the editorial board were to collude without the knowledge of the editor, the two colluding reviewers would have to transmit their reviews to the journal's editor without arousing suspicion. The editor of the journal expects to receive each reviewer's review by e-mail separately from each reviewer --- not to receive two reviews from one reviewer and none from the other. Thus, for example, reviewer #2 could not send his review to reviewer #1 as a "reply" (the act that caused the >'s to be added to review #1) and expect reviewer #1 to send both review documents to the editor. More significantly, reviewer #1 would have to send his review to the editor --- either as an ordinary "reply" to the editor's message transmitting the paper review form or as an ordinary direct message.

However, neither of these ordinarily methods of transmission by reviewer #1 to the editor would cause the >'s to become attached to the responses written by reviewer #1.


Author: John R. Koza
E-Mail: NRLgate@cris.com
Go to top of NRlgate Home Page
Go to Abbreviated Table of Contents
Go to Detailed Table of Contents
Go to Complaint Letter to the Evolutionary Computation journal
Go to top of Previous Page
Go to top of Next Page