NRLgate -
Plagiarism by Peer Reviewers

Sections 4.6 thru 4.8

This page is part of the NRLgate Web site presenting evidence of plagiarism among scientific peer reviewers involving 9 different peer review documents of 4 different journal and conference papers in the fields of evolutionary computation and machine learning.

This page contains sections 4.6 through 4.8 of "Evidence of plagiarism in reviews #1, #2, and #3 of a paper on electrical circuit design submitted to the Evolutionary Computation journal."


Go to top of NRlgate Home Page
Go to Abbreviated Table of Contents
Go to Detailed Table of Contents
Go to Complaint Letter to the Evolutionary Computation journal
Go to top of Previous Page
Go to top of Next Page

4.6. The unlikely coincidence that both reviewers #1 and #2 would be agitated about the paper's evading what they claimed to be an important issue concerning "constraints" and "bounds"

There are many different things that the writer of a relatively short review document can raise about a relatively long submitted paper. There are many different words that can be used to express the same thought.

Peer reviewer #1 is irritated about the fact that there is a "constraint" or "bound" on the maximum overall size of the computer programs (i.e., the LISP expressions) that are produced during a run of genetic programming.
> Although there are many carefully written sections, there are parts
> which:
> - are obvious overstatements to any serious reader: E.g., p. 1
> "Genetic programming provides a way to search the space of all
> possible programs composed of ..." when, in fact, it searches
> a highly constrained space of Lisp expressions.
(Emphasis added).
...
> ... How would random search over the same bounded space of
> Lisp expressions perform? ...

(Emphasis added).
Reviewer #2 is also seems to be irritated about this same issue:
p. 3, line 1: GP is cla)lot to search the space of *all possible*
programs defined over the specified base functions. However, in
previous papers, in particular Koza's June 1990 Tech Report on GP, it is
stated that an upper bound is usually placed in the depth of all
expressions created by crossover (usually 15). In other words, if the
base functions all have a fixed number of arguments, then only a finite
space of expressions is searched. If this practice is still followed
here, it should be acknowledged, and the phase "all possible programs"
should be modified accordingly.
...
... As noted above, GP also employs an upper bound for
the depth of its expressions, and so the "shape" of the expressions is
not entirely unconstrained.

(Emphasis added).
Reviewer #2 later continues his agitated comments about this same issue,
... The use of "computer program"
seems like a deliberate exaggeration of the space being searched in the
cited publications. The actual search space is always a very restricted
class of expressions. That fact that the expressions are LISP
expressions doesn't mean that GP searches the space of general LISP
programs.

(Emphasis added).
Notice the appearance of the word "bound" and the word "constrained" by both reviewers #2 and #1. Note the appearance of the phrase "all possible programs" by both reviewers #2 and #1. There are many other words or phrases that could have been chosen to express these thoughts, so it is noteworthy that reviewers #2 and #1 use and reuse these same key words.

Notice the degree of agitation and annoyance reflected by reviewers #1 and #2 about this issue.

Of course, "bounds" are imposed and the sizes of things stored in the computer are "constrained" in actual runs of genetic programming on actual computers. They have to be. They always are in real computers. There is no terrible hidden secret here to "acknowledge." I view their objection as being comparable to complaining about statements such as "any number can be represented in the computer" or a "computer can run any sequence of instructions." Of course we all know that only numbers lying within some limited range can really be represented in any actual physical computer (e.g., between 10-38 to 1038 on some computers) and that only programs of up to a certain maximum size can actually be run on any actual piece of computing machinery. However, I know of no one in computer science who repeats these obvious, well-known, and well-understood qualifications every time they speak about these matters.

Having said that, the hair-splitting point raised by both reviewers #2 and #1 is narrowly correct.

Of course, the issue here is not which side to take in this pedantic argument over "bounds" or "constraints," but the fact that these two peer reviewers of this particular paper both happened to raise this exotic issue with such an antagonistic tone.

In thinking back over several hundred oral presentations about genetic programming and over the several thousands of questions that I have answered over the years, I cannot recall a single instance when anyone in an audience ever inquired about this minor point (much less seem offended or demanded that I "acknowledge" this terrible secret).

In thinking back over the 300 peer reviews that I have received on my papers on genetic programming over a period of years, this issue of "bounded" was raised by only one other peer reviewer between 1988 and 1995. And, it was raised there in a similar antagonistic way.

On December 10, 1991, John Grefenstette of Code 5514 of the Naval Research Laboratory in Washington submitted a signed, non-anonymous review to the MIT Press of my first book, Genetic Programming: On the Programming of Computers by Means of Natural Selection (Koza 1992). This review is the much-disputed review by which Grefenstette repeatedly delayed his review (while DeJong repeatedly delayed and never finished his review) of my first book and thereby managed to considerably delay its publication.

Before deciding whether to sign a contract to publish a particular book, book publishers often employ anonymous peer reviewers (similar to that used by the journal) to evaluate a prospective author's book proposal, a sample of the author's scientific articles, or a sample chapter. However, after the contract for the book is signed with the author, publishers frequently use a non-anonymous panel of peer reviewers to examine the actual paper in detail. At this post-contract stage, non-anonymous peer reviewing has the goal of improving the book (not deciding whether it is to be published). The non-anonymous peer reviewers at this stage typically make very detailed comments about the actual papers, often consult directly with the author about how to handle various issues, and often re-reading chapters after revisions.

Grefenstette's non-anonymous review to the MIT Press raised this same issue of the constraints (boundedness) on the size and shape of the programs that are evolved in genetic programming:
The limitation of the maximum permissible size of offspring (p. A-106)
is a significant aspect of the approach ...

[T]he author instead
dismisses the limitation of offspring size as a mere inconvenience, that
could be eliminated if parallel processing hardware were available.
This is a highly arguable point! The size limitation also belies the
author's repeated assertion that his approach makes no assumption on the
size or the shape of the required solutions.

(Emphasis added).

Note the word "shape." "Shape" is a rather nonscientific word. it refers here to the fact that genetic programming is based on bushy "trees" (i.e., computer programs) of varying "shape" in contrast to the orderly fixed-length sequence of characters used in the conventional genetic algorithm (where everything usually has a single "shape"). Note that this rather nonscientific word "shape" also appears above in reviewer #2's review (quoted above).

In any event, I really don't think that this issue of "constraints" and "bounds" is
... a significant aspect of the approach ...
nor that I am negligent by
... dismisse[ing] the limitation ...
or that this question is even
... a highly arguable point! ...
or that this hair-splitting manner is a "lie" as suggested by the word
belies
anything or that my treatment warrants being called the
... author's repeated assertion
as if I am dead wrong and know it.
(Emphasis added).
The electricity that reviewers #2 and #1 and North American Associate Editor Grefenstette attach to this issue escapes me. I just don't follow why they are so irritated and agitated about the minor undisputed fact that practical bounds are indeed imposed on the creation of programs during runs of genetic programming inside a computer.
They treat this matter as if I am trying to hide something scandalous whereas I consider the whole matter an entirely hair-splitting point.

Grefenstette says that I "dismiss" this point and it "belies" my "repeated assertion." Reviewer #1 says it is an "obvious overstatement[.]" Reviewer #2 says that I should "acknowledge" it --- as if I am hiding something and there is something terrible to "acknowledge."

But I did "acknowledge it."

These "bounds" are fully "acknowledge[d]" in detail in the listing of "minor parameters" for genetic programming found in (or cited in) all my papers. The terrible truth comes out right in my submitted paper:
Our choice of values for the various secondary parameters that control the run of genetic programming are the same default values as we have used on numerous other problems [Koza 1992a], except that we [now use] tournament selection ...
When I made my December 1991 revisions to my first book, I discussed, by phone, the specific question of whether to cater to Grefenstette's interpretation with a very prominent and well-known member of my panel of five non-anonymous peer reviewers. It was his opinion that my approach to this issue was quite acceptable and that I should proceed with it without any changes.

Neither I (nor the concurring member of my panel) say that Grefenstette and reviewers #1 and #2 are wrong in a scientific sense, but merely that one can reasonably view this hair-splitting question either way and that there is no need to clutter up every publication with such minor qualifications and quibbles that everybody fully understands.

North American Associate Editor Grefenstette, reviewer #2, and reviewer #1 (that is, the 3 or maybe 2 or maybe 1 of them) seem to be the only one(s) in the world who are so emotional about this particular issue (at least in the world of 300 received peer review documents over a period of years). The important point is not who is ultimately right or wrong about this hair-splitting issue, but that this particular issue only excites a particular very small group of people. That is, a peer reviewer voicing this particular issue is rare in the world of peer reviews.

4.7. The unlikely coincidence that both reviewers #2 and #3 would refer to "Koza," by name, in their reviews

The predominant style of scientific peer reviewers refer to "the author" or "the paper," or employ some other highly de-personalized grammatical construction to avoid using an author's name (even though the author's name appears at the top of the paper and is known to the reviewers). Even though this de-personalized grammatical style is often clumsy, it is the predominant style of scientific peer reviewers.

Reviewers #2 and #3 used the word "Koza" in their reviews. Both of the sentences containing "Koza" are highly negative in tone.

One example is the sentence where reviewer #3 criticizes "advertising,"
... only slightly more subtly to advertising Koza's book ...

(Emphasis added).
A second instance is where reviewer #2 thinks he has caught me in some kind of kind of terrible contradiction between my submitted paper and my 1990 technical report on genetic programming:
p. 3, line 1: GP is cla[imed] to search the space of *all possible*
programs defined over the specified base functions. However, in
previous papers, in particular Koza's June 1990 Tech Report on GP, it is
stated that an upper bound is usually placed in the depth of all
expressions created by crossover (usually 15). In other words, if the
base functions all have a fixed number of arguments, then only a finite
space of expressions is searched. If this practice is still followed
here, it should be acknowledged, and the phase "all possible programs"
should be modified accordingly.

(Emphasis added).
Among the 300 peer review documents, there are only a handful of instances where a reviewer refers to me by name. I can't give a precise frequency for this, but it would be a small single-digit percentage.

4.8. The unlikely coincidence that reviewers #1, #2, and #3 all shared a hostile and antagonistic tone

Reviewers of scientific articles generally maintain a high level of civility in their written reviews. This is especially true when they are making negative comments and judgments about a paper. Of course, a small minority of peer reviews drift over the line into harsh incivility.
All three reviews of this particular submission to the Evolutionary Computation journal seem to belong to this minority: What is the probability that all 3 reviewers of the same paper would belong to the minority of reviewers who exhibit such hostility in written peer review documents?

I would estimate the percentage of uncivil reviewers to lie somewhere between 2% and 5%. It would seem unlikely to draw 3 peer reviewers with this particular hostile tone on one particular submitted paper. However, if one peer reviewer were patterning his review after the words and tone established by an already written review of the same paper, the probability of a hostile tone migrating from one review to another is no longer a low percentage.

This is not the only occasion when we encounter two peer reviewers of the same paper both belonging to the small minority of peer reviewers who exhibit a hostile tone. See section 2.13. See Section 7.8.
Author: John R. Koza
E-Mail: NRLgate@cris.com
Go to top of NRlgate Home Page
Go to Abbreviated Table of Contents
Go to Detailed Table of Contents
Go to Complaint Letter to the Evolutionary Computation journal
Go to top of Previous Page
Go to top of Next Page