NRLgate -
Plagiarism by Peer Reviewers

Sections 7.9 thru 7.11

This page is part of the NRLgate Web site presening evidence of plagiarism among scientific peer reviewers involving 9 different peer review documents of 4 different journal and conference papers in the fields of evolutionary computation and machine learning.

This page contains sections 7.9 through 7.11 of "Indications that there are only 2 or 3 (as opposed to 9) different plagiarizing reviewers among the peer reviewers at the Machine Learning Conference (MLC), the editors and members of editorial board of the Evolutionary Computation journal (ECJ), and the Tools for Artificial Intelligence conference (TAI)."

Go to top of NRlgate Home Page
Go to Abbreviated Table of Contents
Go to Detailed Table of Contents
Go to Complaint Letter to the Evolutionary Computation journal
Go to top of Previous Page
Go to top of Next Page

7.9. Reviews A and X contain numerous textual similarities

This section is a comparison of reviews that I received for 2 different papers: This result of this comparison is either that

7.9.1. Two similarities in section 4 of MLC reviews A and X

Reviewer X says, in section 4 (entitled "presentation"),
The presentation suffers from the presence of irrelevancies such as the number of lines of Common Lisp code in the program (although the number seems enormous compared to other implementations of genetic algorithms), and the kinds of boards in the author's Mac II).

(Emphasis added).
Reviewer A says, in section 4,
The presentation suffers from an abundance of irrelevant details about where the author got his economic data and how he transferred to the Explorer by Ethernet (who cares?). This space could be better spent showing more data.

(Emphasis added).

7.9.2. Five similarities in section 3 of MLC reviews A and X

There is a 5-part structural similarity here. The same 5 points are made, in the same sequential order, in approximately the same words.

Reviewer X says, in section 3 (entitled "accuracy"),
The paper claims that optimal control strategies were evolved within 46 generations - extremely quickly by genetic algorithm standards. One suspects that the search space defined by the functions is dense with solutions. It would help to see comparison with another search method, even random search, on the same search space. The data provided is insufficient to judge the merits of this approach.
There is no discussion of the limitations of the method, or of directions for further research.

(Emphasis added).
Reviewer A says, in section 3,
For one experiment, excellent results are claimed to appear within the first nine generations. This is extremely suspicious, unless the choice of functions to be used in the constructions of the concepts practically guarantees success. In order to judge, it would be necessary to see the results compared against an alternative search technqiue, perhaps even random search.
The paper completely lacks any discussion of limitation of the method. This also reduces the quality of the paper.

(Spelling error in "technqiue" in original)
(Emphasis added).


7.9.3. Four similarities in section 1 of MLC reviews A and X

Reviewer A entire response in section 1 (entitled "significance") for this paper on genetic programming as follows:
This paper reorts on a technique of learning concepts expressed as LISP expression using genetic algorithms. This is a topic of general interest. The methodology adopted prevents a clear assessment of how much over advance this approach represents.

(Spelling error of "reorts" and grammatical error of "over advance" in original).
Review X entire response in section 1 is
The papers presents one example of using a genetic algorithm to learn control strategies for a version of the cart-and-pole system. The problem of learning non-linear control strategies is an important one, but the particular problem addressed here is a highly constrained case.

(Grammatical error of "papers" in original).
(Emphasis added).
There are four similarities here.

First, they both begin with the same two words.

Second, they both imposed a substitution of "genetic algorithm" for the author's chosen term (perhaps offensive to both) that actually appears in the submitted paper.

Third, the two responses are almost identical in size.

Fourth, there is a 3-part structural similarity:


7.9.4. Similarity in section 2 of MLC reviews A and X

Reviewer A says, in section 2 (entitled "originality"),
The approach has been reported on previously in MLW89. The applications here are new.

(Emphasis added).
Reviewer X says similarly in section 2.
The authors have reported similar work at last year's ML Workshop ...

(Emphasis added).
It is true that I orally presented a paper on genetic programming at MLW89 in July covering some of the material that was about to appear in my soon-to-published IJCAI-89 paper in August 1989. However, this oral presentation (to a small break-out session representing about a third of the workshop's attendance) and was not published in the printed proceedings of the MLW89 workshop. Knowledge of this unpublished, purely oral presentation is, therefore, highly limited.

By the way, none of the material in my submitted MLC papers on empirical discovery, concept formation, and optimal control strategies was contained in (or even existed at the time) of this earlier oral presentation with which both reviewers A and X seem so familiar.

7.9.5. An additional indication that between MLC reviewer X and may be reviewer A or B

Review X starts his review of my MLC paper on optimal control strategies,
The papers presents one example of using a genetic algorithm to learn control strategies for a version of the cart-and-pole system.

(Grammatical error of "papers" in original).
(Emphasis added).
The plural word "papers" is possibly a Freudian slip indicating that reviewer X was also a reviewer of my MLC paper on empirical discovery (i.e., reviewer X was also either reviewer A or B).

7.10. Similarities between reviewers A, X, and #2

This section is a comparison of reviews that I received for 3 different papers: The result of this comparison means either that

7.10.1. Reviewers A, X, and #2 employ similar words to say they are "extremely suspicious" that genetic programming worked so efficiently

Reviewer A made one additional point in section 3:
For one experiment, excellent results are claimed to appear within the first nine generations. This is extremely suspicious, unless the choice of functions to be used in the constructions of the concepts practically guarantees success. In order to judge, it would be necessary to see the results compared against an alternative search technqiue, perhaps even random search.
The paper completely lacks any discussion of limitations of the method. This also reduces the quality of the paper.

(Spelling error in "technqiue" in original)
(Emphasis added).
Normally, it is considered desirable for an automated machine learning technique to produce results quickly and efficiently. In fact, it is common to criticize techniques that consume too much computer time to produce results. It takes a distinctly non-mainstream turn-of-mind to express the point-of-view expressed here --- namely, good performance is
extremely suspicious
The reviewer's "suspicious" nature is especially surprisingly with respect to this particular paper because "the choice of functions" made by the paper's author consisted of the unremarkable operations of ordinary addition, subtraction, multiplication, and division. What is so "suspicious" about using ordinary arithmetic on numerical data?

Moreover, just how "suspicious" is achieving success at generation 9? With the population size of 300, a total of 3,000 fitness evaluations are performed between generation 0 and generation 9. This is not a particularly small number of fitness evaluations.

Between 1988 and 1995, I have submitted about 100 papers on genetic programming to various peer-reviewed conferences, journals, and edited collections of papers. Almost 70 have now been published (or have been accepted for publication). These 100 submissions were reviewed, on average, by three peer reviewers (sometimes by as many as 14). Thus, I have received approximately 300 peer reviews of my submitted papers on genetic programming over the years. This accumulation of peer reviews is a not insubstantial sampling of the way a broad range of anonymous scientific peer reviewers react and comment on technical papers in this field. Among these reviews, there is 4 cases where a peer reviewer exhibited this particular non-mainstream turn-of-mind.

7.10.2 Five similarities between reviewers A, X, and #2

There are 5 similarities between reviewers A, X, and #2

First, there is the "suspicious" matter discussed above.

Second, there is reviewer X of my MLC paper on optimal control strategies.
The papers claims that optimal control strategies were evolved within 46 generations - extremely quickly by genetic algorithm standards. One suspects that the search space defined by the functions is dense with solutions. It would help to see comparison with another search method, even random search, on the same search space. The data provided is insufficient to judge the merits of this approach.
There is no discussion of the limitations of the method, or of directions for further research.

(Emphasis added).
Third, there is reviewer #2 of my Evolutionary Computation journal paper.
Evaluation is the weak point of the paper. Since results are obtained so quickly (within 50 generations) it is especially important to evaluate the density of acceptable solutions in the search space. This usually means comparison with some baseline approach, perhaps random search. However, the comparison here doesn't do this issue justice.

(Emphasis added).
In the next 5 paragraphs, we compare, in the following order First comparison ... Second comparison ... Third comparison ... Fourth comparison ... Fifth comparison

7.11. Similarities between reviewers B and X

Reviewer B of my MLC paper on empirical discovery said. in section 4,
In the middle of a technical discussion, the author tells us how many lines of Lisp code his program is, and how he used a Mac II to pull a data set over the Ethernet!! This is not what he should be spending his precious 12 pages on.

(Emphasis added).
In the same section (4) of the paper review form, reviewer X of my MLC paper on optimal control strategies said,
The presentation suffers from the presence of irrelevancies such as the number of lines of Common Lisp code in the program (although the number seems enormous compared to other implementations of genetic algorithms), and the kinds of boards in the author's Mac II).

(Emphasis added).
Numerous sections of this document suggest that review B was plagiarized from A and that review Y was plagiarized from X.

So how then can there be similarities between reviews B and X?

One possibility is that there are 2 people involved as follows: MLC reviewers A, B, X, and Y necessarily wrote their reviews at about the same time (since the reviews were for the same MLC conference). If only 2 people were involved in writing these reviews, review B would contain thoughts, words, and phrases from both review A (from which B is heavily plagiarized) as well as thoughts, words, and phrases from review X (which would be the same person as A).

Author: John R. Koza
E-Mail: NRLgate@cris.com
Go to top of NRlgate Home Page
Go to Abbreviated Table of Contents
Go to Detailed Table of Contents
Go to Complaint Letter to the Evolutionary Computation journal
Go to top of Previous Page
Go to top of Next Page