NRLgate -
Plagiarism by Peer Reviewers

Sections 4.9 thru 4.10

This page is part of the NRLgate Web site presenting evidence of plagiarism among scientific peer reviewers involving 9 different peer review documents of 4 different journal and conference papers in the fields of evolutionary computation and machine learning.

This page contains sections 4.9 through 4.10 of "Evidence of plagiarism in reviews #1, #2, and #3 of a paper on electrical circuit design submitted to the Evolutionary Computation journal."


Go to top of NRlgate Home Page
Go to Abbreviated Table of Contents
Go to Detailed Table of Contents
Go to Complaint Letter to the Evolutionary Computation journal
Go to top of Previous Page
Go to top of Next Page

4.9. The non-automatically-created time-and-date lines points to an unusual sequence of events in creating these 3 review documents

Ordinarily, the precise time-of-day on which a peer reviewer happened to dispatch his review by e-mail to a journal would not be noteworthy or remarkable in any way.

Before examining the issue of the time-and-date lines, let me review the mechanics of the transmission of reviews #1, #2, and #3 to me by the Evolutionary Computation journal . The journal's first "call for papers"was publicly issued in late July 1992. On September 23, 1992, I (and two co-authors, Dr. Martin Keane and James P. Rice) submitted a paper entitled, "Use of Automatic Function Definition in Genetic Programming to Facilitate Finding an Impulse Response Function." The submitted paper presented a way of enhancing the then-recently-developed technique of genetic programming (GP) with the then-very-new enhancement of automatically defined functions (ADFs) and applying these techniques to electrical circuit design.

On November 13, 1992, I received the following e-mail message from the North American Associate Editor John Grefenstette.
Date: Fri, 13 Nov 92 14:56:10 EST
From: gref@AIC.NRL.Navy.Mil
To: koza@cs.stanford.edu
Subject: EVOLUTIONARY COMPUTATION manuscript
Cc: dejong@AIC.NRL.Navy.Mil

Dear John:

I have obtained three reviews for your manuscript:

Title: Use of Automatic Function Definition in Genetic Programming to
Facilitate Finding an Impulse Response Function
Manuscript Number: EC-JG-9209-0003

which you submitted to EVOLUTIONARY COMPUTATION. As you can see from
the detailed comments below, all the reviewers liked aspects of the work
and feel that the Genetic Programming paradigm is very interesting, but
they all agree that the current manuscript has significant shortcomings.
Based on these reviews, I will accept the paper, subject to your
revising it to address the concerns of the reviewers.

Some of the major areas that require your attention are:

1. The intended focus and new contributions should be more clearly
stated.

2. The motivation and significance of the control problem chosen for the
cas%g{Uereed to be addressed.

3. The evaluation of the method is incomplete. How does GP compare on
this problem with, for example, random search, analytic methods, other
heuristic methods, other genetic algorithms?

4. The limitations of the method and topics for further research need to
be addressed.

5. A more complete discussion of related work is needed.

6. The style of the presentation needs to be improved, as suggested by
the detailed comments below.

I have enclosed the complete reviews. I'm sure you will find these
helpful in preparing a revised manuscript. I hope you can submit a
revised version for inclusion in an early issue of the journal.

John Grefenstette
Associate Editor

^L
==============================================


EVOLUTIONARY COMPUTATION

Paper Review Form


Title: Use of Automatic Function Definition in Genetic Programming ...
Author(s): Koza, Keane, and Rice
Date: 11/6/92
Manuscript Number: EC-JG-9209-0003
Reviewer: #1

(Emphasis added).

The time-stamp on this e-mail message indicates that this message was sent during normal business hours on a business day
Date: Fri, 13 Nov 92 14:56:10 EST
and that it originated,
From: gref@AIC.NRL.Navy.Mil
The sending address belongs to Code 5514 at the "Artificial Intelligence Center" of the "Naval Research Laboratory" in Washington, DC of the "Department of the Navy" which is a branch of the "Military." That is, this message originated from a government computer.

Grefenstette's message shows an electronic "carbon copy" to Editor-in-Chief Kenneth DeJong of Code 5510 of the Naval Research Laboratory.
Cc: dejong@AIC.NRL.Navy.Mil
After leaving the government computer at the government office in the District of Columbia, the e-mail message traveled to me via the Internet (which was, at the time, a government-subsidized communication network) to its destination in the state of California.

The North American Associate Editor's e-mail message consisted of four parts. The first part contained the North American Associate Editor's summarization of the reviews written by the 3 peer reviewers, his overall decision and instructions to the authors, and was signed by him. This next part of the message was separated from the remainder of the message by a dividing line consisting of 46 ='s. The North American Associate Editor's message then consisted of a heading and incorporated the "complete" written by reviewer #1. Then, there was a second dividing line consisting of 46 ='s followed by another heading and the text of the review written by reviewer #2. Finally, there was a third dividing line consisting of 46 ='s, a similar heading, and the review written by reviewer #3.

Returning now to the header of the overall e-mail message that I received from North American Associate Editor Grefenstette:
Date: Fri, 13 Nov 92 14:56:10 EST
From: gref@AIC.NRL.Navy.Mil
To: koza@cs.stanford.edu
Subject: EVOLUTIONARY COMPUTATION manuscript
Cc: dejong@AIC.NRL.Navy.Mil
This header is typical of the hundreds of e-mail messages that I receive each week from all over the world and contains information common to all e-mail messages transmitted over the Internet. Notice, in particular, the specific stylized format of the "date plus time-of-day" line:
Date: Fri, 13 Nov 92 14:56:10 EST
Peer reviewing at the Evolutionary Computation journal is anonymous in the sense that authors of submitted papers are given the texts of the reviews written by the peer reviewers, but not the names of the reviewers. After the 3 reviewers finish writing their reviews, they transmit their review to the journal (typically by e-mail over the Internet). Each of the 3 original peer review documents therefore has an e-mail header containing the familiar information contained in the header shown above.

In order to prevent identification of the 3 reviewers to the submitting author, the North American Associate Editor necessarily would delete the "from" line of each of the three e-mail messages that he received from the 3 peer reviewers when he incorporated the texts of their reviews into the his e-mail message to the submitting authors. Instead of the 3 "from" lines, we instead see the 3 uninformative phrases "reviewer #1," "reviewer #2," and "reviewer #3." And, presumably at the same time, the North American Associate Editor would have deleted the uninformative "to" line deleted (since all 3 peer reviewers presumably would have sent their reviews to the journal).

4.9.1 The absence of the time-of-day indicates that the 3 "date" lines that the submitting authors received from the North American Associate Editor were not the familiar "date plus time-of-day" lines that are automatically created when an e-mail message is transmitted on the Internet.


Ordinarily, the precise time-of-day on which a peer reviewer happened to dispatch his review by e-mail would not be noteworthy or remarkable in any way.

The "date" lines for the 3 peer reviews attached to the North American Associate Editor's November 13, 1992 e-mail message to the submitting authors are as follows (with spacing intact) for reviewers #1, #2, and #3 (in that order): Notice the absence of the time-of-day. Ordinarily, the time-of-day (right down to the second) is an automatically created timestamp that appears on every e-mail message handled by the Internet.

The absence of the time-of-day indicates that the 3 "date" lines that the submitting authors received from the North American Associate Editor were not the familiar automatically created "date plus time-of-day" lines that are routinely created when an e-mail message is transmitted on the Internet.

4.9.2 The unusual format of the 3 "date" lines were not the familiar "date plus time-of-day" lines that are automatically created when an e-mail message is transmitted on the Internet

Ordinarily, the precise date on which a reviewer happened to e-mail his review in to the North American Associate Editor would not be noteworthy or remarkable.

Note that none of the 3 dates appear to be in the familiar automatically created format. Note, in particular, that there are 11 spaces preceding the hand-created date line for reviewer #2 and there is a period after "Nov." for reviewer #3.

Whatever the three "date" lines on these 3 review documents may be, they are not the untouched, virginal, automatically created "date and time-of-day" lines that are automatically created when electronic mail is ordinarily transmitted via the Internet. Instead, they are hand-created dates.

One cannot determine the exact detailed sequence of events that took place to create each of the four parts of North American Associate Editor's Grefenstette's e-mail message merely by looking at it in a vacuum and without additional evidence. However, one can consider the possibilities.

Why would the North American Associate Editor send the submitting authors the "complete reviews" of the 3 peer review messages intact --- but not send the original, virginal, intact "date plus time-of-day" lines that were effortlessly available?

One possible explanation is that the original, virginal, intact "date plus time-of-day" lines of these 3 "independent" peer documents were unfit for the eyes of the submitting authors --- perhaps because they were embarrassingly contemporaneous with the North American Associate Editor's own summarizing message to the submitting authors at precisely
Date: Fri, 13 Nov 92 14:56:10 EST
Another possibility is that one or more of these 3 "independent" peer documents was not, in fact, ever "received" by e-mail from the outside from any of the 32 geographically dispersed members of the editorial board of the journal (none of whom, it will be recalled, are co-located in Washington, DC with the Editor-In-Chief and North American Associate Editor at the Naval Research Laboratory). That is, perhaps one or more of these 3 "independent" peer documents never arrived from outside of the NRL office and therefore never originally bore any automatically created "date plus time-of-day" line of its own. This could happen if these documents came from a computerized file residing within the very same NRL computer workstation that dispatched the e-mail message to the submitting authors (e.g., the North American Associate Editor's own workstation). To be specific: a piece of text that was scooped from one file of a computer to the e-mailer of the very same computer would not bear any "date plus time-of-day" line of its own. Similarly, a piece of text that was scooped by a direct file access (e.g., via a local area network) from another computer located under the same roof would not bear any "date plus time-of-day" line of its own. If one or more of these 3 "independent" peer documents did not bear a "date plus time-of-day" line of its own, that would explain why some kind of "date" line would have to be added.

These 3 "date" lines were not in any standard automatically created format. The question arises as to why --- in typing these 3 "date" lines (possibly typed by one hand in rapid succession) were 3 different styles employed --- namely There are 11 spaces in the second date and there is a period after "Nov" in the third date.

Perhaps the typing of these 3 dates, in rapid succession, in 3 different non-standard styles was part of an effort to make them "look different."

4.10. The 4 appearances of the submitted paper's title points to an unusual sequence of events in creating the review documents. The 2-line title line of review #3 is an exact "cut-and-paste" copy of the two-line title in the section written by the North American Associate Editor, but the abbreviated 1-line title lines of reviews #1 and #2 were not.

The title of the submitted paper is contained in the introductory portion of North American Associate Editor Grefenstette's e-mail message and it is also contained in each of the 3 attached reviews.

It would be understandable if
But the documentary evidence shows that neither of these situations prevailed.

The introductory portion of the e-mail message from North American Associate Editor Grefenstette is
Dear John:

I have obtained three reviews for your manuscript:

Title: Use of Automatic Function Definition in Genetic Programming to $
Facilitate Finding an Impulse Response Function


The title line of review #3 is
Title: Use of Automatic Function Definition in Genetic Programming to $
Facilitate Finding an Impulse Response Function


The title line of review #2 is
Title: Use of automatic function definition in Genetic Programming$


The title line of review #1 is
Title: Use of Automatic Function Definition in Genetic Programming ...$

Note that we indicate the carriage return symbol by a "$".

First consider the two-line titles appearing in North American Associate Editor in Grefenstette's introduction and review #3. Notice that both of these two-line titles consist of the complete 15-word title of the paper and that exactly the same 11 words are capitalized. Now notice the 8 spaces before the word "Facilitate" on the second lines of these two titles. Notice the single blank space after the "to" on the first line of both titles. Notice the carriage return symbol (made visible here by a "$") after the "to" on the first line of both titles. In other words, both of these two-line titles are identical (character-for-character and space-by-space).

In most computerized word processor systems, the computer software automatically drops down to the next line based on the preestablished width of the paper (or computer screen) and then continues the ongoing sentence on the new line. The person doing the typing does not normally control this breakage. Instead, the typist normally only specifically marks desired paragraph breaks (usually by hitting the "carriage return" key). Of course, if the typist wants to force the creation of a new line, he may type the "carriage return" key.

The 8 hand-introduced spaces at the beginning of the second lines of these two titles are apparently an attempt to make the masthead look more attractive by indenting "Facilitate" (the 10th word of the title) in one space with respect to "Use" (the first word of the title). To accomplish this indenting, a space and a carriage return were added after the "to" on the first line and then 8 spaces were introduced, by hand, at the beginning of the second line.

Now consider the one-line titles of review #1 and review #2. Notice that both of these titles are abbreviated to the first 8 words.

The one thing that is certain is that two-line title of review #3 --- with its 8 hand-introduced extra spaces before "Facilitate" --- is an exact "cut-and-paste" copy of the two-line title in the introductory section of North American Associate Editor Grefenstette's e-mail message to the submitting authors --- whereas the abbreviated one-line titles of reviews #1 and #2 were not.

That is, Grefenstette's introduction and review #3 contains common "cut-and-pasted" material.

It would be understandable if all four of these titles were identical and it would be understandable if all four were entirely different. However, it is difficult to visualize any consistent process that could have given rise to the four titles that we see. Apparently whatever sequence of steps gave rise to Grefenstette's introduction and review #3 was very different from the sequence of steps that gave rise to reviews #1 and #2.

When North American Associate Editor Grefenstette's began his e-mail message,
I have obtained three reviews for your manuscript...
(Emphasis added)
did we have a specific reason to carefully choose the unusual word "obtained," instead of the more commonplace "received"? Is it possible that one or more of these review documents were, in fact, never "received" from anywhere?

One cannot determine the exact detailed sequence of events that took place to create each of the four parts of North American Associate Editor Grefenstette's e-mail message merely by gazing at this message in a vacuum. Additional factual evidence is necessary to establish the precise explanation for this unusual outcome.

Author: John R. Koza
E-Mail: NRLgate@cris.com
Go to top of NRlgate Home Page
Go to Abbreviated Table of Contents
Go to Detailed Table of Contents
Go to Complaint Letter to the Evolutionary Computation journal
Go to top of Previous Page
Go to top of Next Page