NRLgate -
Plagiarism by Peer Reviewers

Sections 4.3 thru 4.5

This page is part of the NRLgate Web site presenting evidence of plagiarism among scientific peer reviewers involving 9 different peer review documents of 4 different journal and conference papers in the fields of evolutionary computation and machine learning.

This page contains sections 4.3 through 4.5 of "Evidence of plagiarism in reviews #1, #2, and #3 of a paper on electrical circuit design submitted to the Evolutionary Computation journal."

Go to top of NRlgate Home Page
Go to Abbreviated Table of Contents
Go to Detailed Table of Contents
Go to Complaint Letter to the Evolutionary Computation journal
Go to top of Previous Page
Go to top of Next Page

4.3. The unlikely coincidence that reviewers #1 and #2 would both make the mistaken assertion that the paper lacked a specific item even though the paper actually devotes almost 2 pages to the item

Most published experimental papers in the field of evolutionary computation and machine learning contain a "baseline" comparison of the performance of the paper's technique to a performance of a "blind random search" of the search space of the problem. The reason is that people legitimately and naturally wonder whether any proposed new automated technique for machine learning and artificial intelligence performs better than "blind luck" or the proverbial "monkey at the typewriter" in solving problems.

Two of the three reviewers of the submitted ECJ paper complained about the absence of the virtually obligatory "baseline" comparison to random search in my ECJ paper.

Peer reviewer #1 said,
> ... Given the goal of finding a Lisp expression
> for the system response, how complex is the space being explored
> by GP? How would random search over the same bounded space of
> Lisp expressions perform? ...

(Emphasis added)
Peer reviewer #2 said,
... Since results are
obtained so quickly (within 50 generations) it is especially important
to evaluate the density of acceptable solutions in the search space.
This usually means comparison with some baseline approach, perhaps
random search.

(Emphasis added)
Reviewer #2 was so concerned about a comparison with the absence of the obligatory comparison with blind random search that he repeated his concern later in his review.
The manuscript could be made acceptable without many further experiments if revised ... [to] ... [d]iscuss the density of acceptable solutions, and investigate how well random search performs on the chosen problems.

(Emphasis added)
However, my ECJ paper did have this very kind of "baseline" comparison!

Moreover, the comparison could not be easily-overlooked --- it occupied almost two pages of space in my submitted ECJ paper.

This "baseline" comparison could hardly be missed by a peer reviewer who was relying on his own reading of the paper to formulate his own opinion about the contents of the paper.

Indeed, the comparison with random search was covered extensively in the paper from five different perspectives:
[such and such] ... is the baseline value for a random search of the space of possible compositions of terminals and primitive functions for this problem.

Indeed, the baseline comparison in my submitted ECJ paper was so extensive that I would not have been surprised if one of the three peer reviewers had criticized my paper for devoting too much space to the baseline comparison.

Between 1988 and 1995, I have submitted about 100 papers on genetic programming to various peer-reviewed conferences, journals, and edited collections of papers. Almost 70 have now been published (or have been accepted for publication). Each of these 100 submissions were reviewed, on average, by 3 peer reviewers (sometimes by as many as 14). Thus, I have received approximately 300 peer reviews of my submitted papers on genetic programming over the years. This accumulation of peer reviews is a not insubstantial sampling of the way a broad range of anonymous scientific peer reviewers react and comment on technical papers in this field.

I know of no case in which any of approximately 300 other peer reviews missed the presence of this baseline comparison and falsely claimed that I had omitted something that was manifestly in the paper. In other words, the writers of all of these other 300 reviews had no difficulty in spotting the presence of this virtually obligatory item in the numerous other papers that I submitted to other scientific journals. conferences, and edited collections of papers.

How could two peer reviewers mistakenly assert that my submitted ECJ paper lacks a certain clearly-identified item when the paper demonstrably devotes almost two pages to that item? If, hypothetically, a peer reviewer were prejudiced against a particular technology or author, I can appreciate that he might hastily read a submitted paper and then construct a deceptive paper trail of arguable flaws to conceal the obviously illegitimate basis for his preordained negative decision about the paper. If his review were cursory, I can appreciate how one such reviewer might conceivably make the gaffe of asserting the absence of something that the paper actually contains. Indeed, the first reviewer committed just such a gaffe. If, hypothetically, a second peer reviewer were also similarly prejudiced, I can also appreciate how this second reviewer might construct his own hypocritical paper trail of negative-sounding verbiage to disguise his bias. However, two prejudiced reviewers would not make the same egregious gaffe if they were acting independently in accordance with fundamental principles of the peer review process.

4.4. The unlikely coincidence that both reviewers #1 and #3 would complain about "advertising"

There are many different things that the writer of a relatively short review document can raise about a relatively long submitted paper. There are many different words that can be used to express the same thought.

As previously mentioned, the accumulation of approximately 300 peer reviews of my submitted papers on genetic programming over the years constitutes a not insubstantial sampling of the way a broad range of anonymous scientific peer reviewers react and comment on technical papers that they review in this field. In almost 100 submitted papers on genetic programming between 1988 and 1995, I have used an almost-identical introductory paragraph to summarize genetic programming. Like many people today, I use a word processor to write my own articles and I "cut and paste" this standard "background" paragraph into each paper (while slightly customizing it). I patterned my standard paragraph after many similarly constructed paragraphs that I have seen many times in the literature. Even today, I continue to use a substantially similar standard paragraph in substantially the same style in virtually every paper that I submit for publication.

Reviewers #3 and #1 made a non-mainstream, antagonistic comment concerning forms of the pejorative word "advertising" that I have never seen (before or since) among the peer reviews of my scientific papers.

Peer reviewer #3 complained,
The authors clearly state some research goals, but I think they have
some "hidden agendas" that should be made explicit, most notably
...
to advertise the genetic
programming method (a full 1 1/2 pages of text is devoted to
describing other applications that the method has been used for), and
only slightly more subtly to advertising Koza's book and videotape. I
think this last agenda is inappropriate for a scientific journal.
(Emphasis added).
Peer reviewer #1 complains,
> The presentation style is very uneven for a scientific journal article.
...
> Although there are many carefully written sections, there are parts
> which:
...
> - are blatant advertisements for the authors' book and videotape (pp. 4-5).
>

(Emphasis added).

It is, of course, no offense --- much less a "hidden agenda" --- for the author of a paper on genetic programming
to advertise the genetic programming method
Indeed, the primary purpose of most articles in scientific publications is to "advertise" a successful method or result (using the word "advertise" in its literal and non-pejorative sense).

How common is it for a scientific peer reviewer to criticize the author of a paper because he "advertised" his "method"?

We again make reference to the computer file containing the 64,109 words of the 316 paper review forms from the 86 peer reviewers of the genetic programming papers at the Genetic Programming 1996 Conference.

The word "advertise" does not appear, in any form, among these 64,109 words written by contemporary peer reviewers in this field.

Of course, there was no "hidden agenda" to
to advertise the genetic programming method
in the my submitted ECJ paper. That was definitely one item on my "agenda." as, indeed, it is on the agenda of most authors who write papers..

Also, it is, of course, no offense --- much less a "hidden agenda" --- for the author of a scientific paper to provide
a full 1 1/2 pages of text is devoted to describing other applications that the method has been used for
Of course, references to previous applications of a technique discussed in a scientific paper are mandatory in scientific papers. A detailed recital is especially appropriate for a new technique such as genetic programming. Even today, it is not universally accepted that genetic programming can solve a wide variety of problems over a wide variety of fields (and it certainly was not universally accepted at the time of submission of this paper). There is no "hidden agenda" to "advertise the genetic programming method" by providing citations in the scientific literature to problems that genetic programming has successfully solved. Indeed, that was definitely one item on my "agenda."

As to the videotape, the idea of having a videotape to accompany a book that presents a series of two dozen segments of animated computer runs was unique at the time of submission of this paper to this journal in the fields of genetic algorithms, machine learning, and artificial intelligence. Indeed, this approach is still unique, to my knowledge, in scientific publishing, in general. My videotape contains about two dozen segments of actual computer runs and they provide a "live" visualization of material that can be difficult to understand in text form. A picture is worth a thousand words. It would be preposterous for me suppress the existence of the videotape (or to hide the existence of my own book) in a paper intended to communicate information to readers of a scientific journal. This tape (and the similar videotape accompanying my second book) are, I believe, a contribution to scientific publishing that will become increasing common in future years.

Let's return now to the common features of the reviews #1 and #3.

Both reviewers #1 and #3 expressed a thought not normally seen in the texts scientific peer reviews; they both articulated their non-mainstream interpretation using the same specific pejorative word (when many other phrases or words could have chosen to express the same thought).

Reviewer #1 complains about the reference to both
the authors' book and videotape
while reviewer #3 complains about the reference to both
Koza's book and videotape

Reviewer #1 complains that the "presentation" is
uneven for a scientific journal
while reviewer #3 complains that the "presentation" is
inappropriate for a scientific journal

Notice the symmetric "on the one hand, on the other hand" construction. Reviewer #1 said,
The authors clearly state some research goals,
but I think they have
some "hidden agendas"
Reviewer #1 said,
> Although there are many carefully written sections, there are parts
> which:
...
> - are blatant advertisements

(Emphasis added in each case above)


Of course, these particular two reviewers are entitled to their personal non-mainstream opinion about the matter of what is, or is not, "blatant" "advertising." The point here is not whether my paper really and truly is or is not "advertising." The point is that, over a six year period, these two reviews from the ECJ are the only ones (out of about 300 peer review documents) that ever criticized this standard paragraph as "advertising" and getting one (much less two) such reviews on the same paper is a rare and improbable convergence of events in the world of peer reviews.

4.5. The unlikely coincidence that both reviewers #1 and #2 would abbreviate the paper's title and that they would both abbreviate it to the same 8 words

Peer reviewer #3 refers to my submitted ECJ paper by its full 15-word title:
Use of Automatic Function Definition in Genetic Programming to Facilitate Finding an Impulse Response Function
However, reviewer #1 and reviewer #2 abbreviate the title to the same 8 words.

How common is the practice of abbreviating the author's title on a paper review form?

We again make reference to the computer file containing the 64,109 words of the 316 paper review forms from the 86 peer reviewers of the genetic programming papers at the Genetic Programming 1996 Conference.

There were 27 reviews of papers whose titles contained between 11 and 17 words. In 63% of the cases, the reviewer entered the author's full exact title onto the paper review form without alteration. Thus, the probability of randomly drawing 2 reviewers who abbreviate the author's title is about 1 in 9.

More importantly, there are many different ways to abbreviate a title. In the case of the 10 abbreviations, there was no case where two reviewers abbreviated a particular title to the same words (or even converged to the same number of words). In fact, it was somewhat surprising to see the many different ways for abbreviating titles that were used by contemporary peer reviewers in this field. For the abbreviated titles, no two styles were the same. The different styles for shortening titles included We do not have sufficient data to compute the joint probability of convergent abbreviation to identical words; however, a rough guess is that it is around 1 in 100.

Without the suggestive power of one already written review, isn't it improbable that reviewers X and Y would both abbreviate my title and that they would also converge to the identical shortened version?

This is not the only occasion when we encounter two peer reviewers of the same paper both belonging to the small minority of peer reviewers who abbreviate the author's title and who coincidentally converge on the same abbreviation. As will been seen elsewhere, reviewers X and Y abbreviated the 14-word title of my paper on optimal control strategies submitted to the Machine Learning Conference and they both converged to the same 6 words. See section 3.8.
Author: John R. Koza
E-Mail: NRLgate@cris.com
Go to top of NRlgate Home Page
Go to Abbreviated Table of Contents
Go to Detailed Table of Contents
Go to Complaint Letter to the Evolutionary Computation journal
Go to top of Previous Page
Go to top of Next Page