NRLgate -
Plagiarism by Peer Reviewers

Sections 7 thru 7.8

This page is part of the NRLgate Web site presenting evidence of plagiarism among scientific peer reviewers involving 9 different peer review documents of 4 different journal and conference papers in the fields of evolutionary computation and machine learning.

This page contains sections 7 through 7.8 of "Indications that there are only 2 or 3 (as opposed to 9) different plagiarizing reviewers among the peer reviewers at the Machine Learning Conference (MLC), the editors and members of editorial board of the Evolutionary Computation journal (ECJ), and the Tools for Artificial Intelligence conference (TAI)."

Go to top of NRlgate Home Page
Go to Abbreviated Table of Contents
Go to Detailed Table of Contents
Go to Complaint Letter to the Evolutionary Computation journal
Go to top of Previous Page
Go to top of Next Page

7. Indications that there are only 3 (as opposed to 9) different plagiarizing reviewers among the peer reviewers at the Machine Learning Conference (MLC), the editors and members of editorial board of the Evolutionary Computation journal (ECJ), and the Tools for Artificial Intelligence conference (TAI)


Most people (including myself) believe that wrongdoing is relatively rare even though we know that wrongdoing occurs in every aspect of human activity. Although it is conceivable that reviewers A, B, X, Y, #1, #2, #3, T2, and T3 of the 4 different conference and journal paper involved here are 9 different people, the principle of Occam's Razor suggests that the simplest explanation for a situation warrants some extra attention.

The existence of 4 disjoint groups of plagiarizing reviewers would not make the offense of plagiarism any less serious, it would just mean that there are a surprisingly large number of disjoint groups of plagiarizing reviewers within the extraordinarily small pool of people in the fields of genetic algorithms and machine learning.

Documents A, B, X, Y, #1, #2, #3, T2, and T3 establish, on their face, that serious scientific misconduct involving collusion and plagiarism has occurred. Plagiarism among peer reviewers is an offense that goes to the heart of the integrity of the scientific peer review process. Someone created these plagiarized documents. Someone committed these offenses. Up to 9 different persons violated the trust reposed in them by the Evolutionary Computation journal, the Machine Learning Conference, and the Tools for Artificial Intelligence conference. It is conceivable that reviewers A, B, X, Y, #1, #2, #3, T2, and T3 of the 4 different conference and journal paper involved here are 9 different people. The 4 disjoint groups of plagiarizing reviewers would be I believe that a definitive identification of the wrongdoers involved in creating these plagiarized reviews should be done by an impartial person who is experience and trained in reaching findings of fact and making judgments based on the evidence. Specifically, I advocate a complaint resolution procedure involving a retired federal judge acting under the auspices of the American Arbitration Association for the task of making a definitive determination of the truth. No final judgment or opinion should be formed at this time on any of the matters herein. Instead, the truth concerning all of these matters herein should be definititvely determined in a thorough and impartial investigation and factual determination made under the proposed arbitration procedure by a retired federal judge.

However, the reader may wish to make some of his own preliminary conclusions as to who created the 9 peer reviews for these 4 different conference and journal papers. The factors below may enable the reader to reach his own preliminary conclusion concerning this matter. The discussion below falls into two main categories.

The information in the previous section (6 thru 6.3) may cause the reader to reach a preliminary conclusion for himself that the small overlap of people in the pools of reviewers for the artificial intelligence conference, the Machine Learning Conference, and the Evolutionary Computation journal along with the high level of familiarity with evolutionary computation exhibited by reviewers A, B, X, Y, #1, #2, #3, T2, and T3 may suggest the identities of the plagiarizing reviewers.

The information provided may cause the reader to reach a preliminary conclusion for himself that reviews A, X, #2, and T2 of the 4 papers are inter-linked by so many similarities that they were probably written by the same person. Similarly, for reviews B, Y, #1, and T1 and for reviewers #3 and T3. That is, The clickable diagram below presents plagiarism as horizontal dashed lines (with added arrows shown when the documents indicate the time sequence of the plagiarism) and presents likely common authorship as vertical lines for 9 different reviews of 4 different papers at the Machine Learning Conference (MLC), the Evolutionary Computation journal (ECJ), and the Tools for Artificial Intelligence (TAI) conference.

ONLY 2 COMMON
REVIEWERS FOR
MLC, ECJ, TAI
.................THIRD REVIEWER

...B..<---..A

...Y..<---..X

...#1.--->..#2..---..#3

...T1.......T2..---...T3

Of course, if multiple papers turn out to have been reviewed the same person, there is no plagiarism within that group of papers. Such reviews would be expected to resemble one another. It is bad practice, but not a crime, for a single reviewer to excessively review a single author's work.

On the other hand, if there is no common authorship among the 9 reviewers, then there must have been plagiarism between 4 separate groups of plagiarizing reviewers.

7.1. Reviews A, X, #2, the signed non-anonymous review to the MIT Press of my first book, and the North American Associate Editor's transmittal letter employ the word "limitation" with a frequency of 1-in-530; the 316 GP-96 peer reviews use it with a frequency of 1-in-12,822; and the word doesn't appear at all in reviews B, Y, #1, #3, T1, T2, and T3

Everyone has certain words that he habitually overuses and underuses.

Referring to the computer file containing the 64,109 words of the 316 paper review forms from the 86 peer reviewers of the genetic programming papers at the Genetic Programming 1996 Conference, we found only 5 occurrences of the word "limitation" (1-in-12,821 odds).

Here's the frequency of occurrence of the word "limitation" in various review documents: Now let's look at the occurrence of the word "limitation" for some other peer review documents.

Reviewer A says, in his 218-word review, says
The paper completely lacks any discussion of limitations of the method.
Reviewer X says in his 501-word review, says
There is no discussion of the limitations of the method
Reviewer #2 says, in his 1,488-word review, says
The limitations of the method and topics for further research need to
be addressed.
John Grefenstette in his 246-word transmittal letter (to which reviews #1, #2, and #3 were attached), says
The limitations of the method and topics for further research need to
be addressed.
John Grefenstette, in his 1,787-word signed, non-anonymous review to the MIT Press of my first book, says
The limitation of the maximum permissible size of offspring (p. A-106)
is a significant aspect of the approach
John Grefenstette later says,
the author instead
dismisses the limitation of offspring size as a mere inconvenience
John Grefenstette later says,
The size limitation also belies the
author's repeated assertion that his approach makes no assumption
John Grefenstette later says,
Another limitation that is lightly brushed aside is the selection of
cases

Thus, reviewers A, X, #2, and these two signed documents usesthe word "limitation" a total of 8 times in the space of a 4,240 words (1-in-530 odds).

By the way, the reader may notice that the word "limitation" appears in question 4 of the Evolutionary Computation journal's standard paper review form:
DISCUSSION. Does the paper discuss relevant earlier work, noting
similarities, differences and progress? Does it discuss the limitations
of the approach along with its advantages? Does it consider the
implications of the work and outline directions for future work?

(Emphasis added).

One might think that the appearance of a word in the question on a paper review form might create a tendency of a reviewer to use that word in his review. However, as can be seen above, ECJ reviewers #1 and #3 did not use the word at all.

By the way, it should be noted that the paper review form for the Evolutionary Computation journal was originally created by the editors of the journal at the time of its inception. Thus, the paper review form itself represents yet another appearance of this particular word.

Of course, the overuse of a particular word does not alone establish the identity of the author of any particular document.

7.2. Reviewers B, Y, and #1 consistently misspell "LISP"

The following observation is consistent with the equivalence of reviewers B, Y, and #1 (and T1).

"LISP" (an acronym) was misspelled as "Lisp" consistently 9 times out of 9 by reviewers B, Y, and #1.
(Reviewer T1 of the TAI paper did not use the word at all).

It should be noted that Kenneth DeJong has, from time to time, published a series of survey articles of the field of evolutionary computation. These lengthy articles typically contain a few sentences about genetic programming. For example, in his 11-page paper at the 1996 EvCA conference entitled
Evolutionary Computation: Recent Developments and Open Issues
In this 11-page paper, Kenneth DeJong (1996) summarizes all the activity in the field of genetic programming (including all 600 papers published by about 150 different authors) in a single paragraph (which, at the same time, purports to explain why genetic programming cannot possibly work):
Historically, much of the EA work has involved the evolution of fairly simple structures [that] could be represented in phenotypic form or be easily mapped onto genotypic representations. However, as we attempt to evolve increasingly more complex structures (e.g., Lisp code (Koza, 1992) or neural networks (de Garis, 1990)), it becomes increasingly difficult to define forms of mutation and recombination which are capable of producing structurally sound and interesting new individuals. If we look to nature for inspiration, we don't see many evolutionary operators at the phenotype level (e.g., swapping arms and legs!). Rather, changes occur at the genotype level and the effects of these changes instantiated via growth and maturation. If we hope to evolve such complexity, we may need to adopt more universal encodings coupled with a process of morphogenesis ... "

(Emphasis added).
Authors carefully check their papers prior to publication, so a spelling error appearing in a published paper probably indicates that the author thinks the word is spelled that way. The EvCA conference proceedings (like those of many conferences, including the ICGA conference mentioned below) are printed using camera-ready originals provided directly by the author. A spell checker would not ordinarily catch an error of this kind involving a technical acronym such as "LISP." Thus, there seems to be little doubt that Kenneth DeJong thinks that "LISP" is spelled "Lisp."

This is not the only published paper where we find DeJong misspelling "LISP" as "Lisp" in a published paper. For example, in their ICGA-93 conference paper entitled,
On the State of Evolutionary Computation.
Kenneth DeJong and William Spears mentioned,
... Lisp code (Koza, 1992) ... (page 621)

(Emphasis added).
Thus, all 9 occurrences of this word are misspelled in MLC reviews B and Y and ECJ review #1. And, this word is similarly misspelled in these published EvCA and ICGA papers.

In contrast, all 4 of his references to the LISP programming language by ECJ reviewer #2 are spelled correctly. Similarly, the 1 reference to LISP by MLC reviewer A is spelled correctly. Similarly, the 2 references to LISP in the signed, non-anonymous review to the MIT Press of my first book written by John Grefenstette are spelled correctly. However, MLC reviewer X and TAI reviewer T2 each used this word once and misspelled it. Thus, 7 out of 9 spellings are correct among this last group of 5 documents.

Of course, a repetitively-made spelling error does not alone establish the identity of the writer of reviews B, Y, and #1. It should be noted that I am not claiming that an individual writer can be picked out of a large group by virtue of a single idiosyncratic spelling error. However, when it gets down to grouping documents for the purpose of associating them with 1 of 2 possible persons, such personal trademarks can be useful. The following sections contain additional such trademarks.

7.3. Reviews B, Y, #1, and T1 do not contain any typographical or spelling errors (except for "Lisp"), whereas reviews A, X, #2, #3, and the signed, non-anonymous review to the MIT Press of my first book all contain numerous such errors

Some people are very accurate typists (or use a spell checker) when they write a peer review. Others are not. For example, I usually do not run my spell checker when I write a peer review.

There are no spelling or grammatical errors in reviews B, Y, #1, or T1.

Reviewer A says,
This paper reorts.

alternative search technqiue
The methodology adopted prevents a clear assessment of how much over advance this approach represents.

(Reviewer A apparently meant to type "of advance over" instead of "over advance")
(All spelling errors in original throughout this section).

(Emphasis added throughout this section).
Reviewer X says,
The papers presents one example
The technical soudness of the paper
Reviewer #2 says,
automatic function defintion
Woudln't
Reviewer #3 says,
wikch

excrutiating

too lengthly
The signed, non-anonymous review to the MIT Press of my first book written by John Grefenstette says,
mathematic formulas
Of course, being a careful typist (or using a spell checker) does not alone establish the identity of the writer of reviews.

7.4. Reviewers A, X, and T2 use the particular infrequently used word "judge"

Everybody has their favorite words.

I was surprised to learn how rarely the seemingly ordinary word "judge" is used by contemporary scientific peer reviewers. When I examined the computer file containing the 64,109 words of the 316 paper review forms from the 86 peer reviewers of the genetic programming papers at the Genetic Programming 1996 Conference, there were only 3 occurrences of this particular word in 64,109 words (1 in 21,379 odds).

Reviewer X of my MLC paper on optimal control strategies used this word twice in his 501-word review:
The technical soudness of the paper is extremely hard to judge.

...

The data provided is insufficient to judge the merits of this approach.

(Spelling of "soudness" in original)
(Emphasis added throughout this secton).
.
Reviewer T2 of my TAI paper on pursuer-evader games said,
Good solutions are achieved so rapidly that is hard to judge the difficulty of the problems (a comparison with random search or other techniques would be helpful).
Reviewer A of my MLC paper on empirical discovery said,
In order to judge, it would be necessary to see the results compared against an alternative search technqiue, perhaps even random search.
(Spelling error in "technqiue" in original)
(Emphasis added).

Here's the frequency of occurrence of the word "judge" in various review documents: Of course, the overuse of a particular word does not alone establish the identity of the plagiarizers involved here. I am not claiming that an individual writer can be picked out of a large group by virtue of frequent use of a single infrequently used word. However, when it gets down to grouping documents for the purpose of associating them with 1 of 2 possible persons, such personal trademarks can be useful.

7.5. Review X and the signed non-anonymous review to the MIT Press of my first book used the unusual self-referential style "This reader"

There are many ways to express the same thought; however, some figures of speech are rarer than others. Some are so unusual that they rarely, if ever, appear in contemporary writing by scientific peer reviewers. The self-referential style "this reader" is one such phrase.

We again make reference to the computer file containing the 64,109 words of the 316 paper review forms from the 86 peer reviewers of the genetic programming papers at the Genetic Programming 1996 Conference. There are no occurrences of the phrase "this reader" in the 64,109 words in the 316 paper review forms from the 86 peer reviewers.

Reviewer X uses this phrase in section 3 of his review:
This reader was confused by the mismatch between the phrase "Three dimensional broom balancing problem" in the title and the description of the problem in section 4.0.

(Emphasis added).
This phrase also appears in the signed, non-anonymous review to the MIT Press of my first book (December 10, 1991) written by John Grefenstette of Code 5514 of the Naval Research Laboratory in Washington:
... without making the intended point clear (at least to this reader).

Here's the frequency of occurrence of the phrase "this reader" in 7 review documents: 316 reviews by 86 GP-96 reviewers - 0 times
Of course, the use of a self-referential style that is not commonly used by scientific peer reviewers does not alone establish the identity of the writer of reviews X.

7.6. Reviews A, B, #2, and the signed non-anonymous review to the MIT Press of my first book all used the quaint and infrequently used word "Studies"

The somewhat quaint and infrequently used word "studies" appears in 4 of the review documents.

The words that are in more common usage are "work," "research," "paper," "article," etc.

This quaint and infrequently used word appeared How quaint and unusual is this word among contemporary peer reviews in this field of science?

Referring to the computer file containing the 64,109 words of the 316 paper review forms from the 86 peer reviewers of the genetic programming papers at the Genetic Programming 1996 Conference, we found that the frequency of occurrence of six alternative words contained in this corpus of writing by peer reviewers in this field: Of course, the use of a quaint and uncommon word does not alone establish the identity of the plagiarizers involved here.

This is not the only occasion when we encounter two peer reviewers of the same paper both belonging to the small minority of peer reviewers who use this particular quaint and infrequently used word. See section 3.7.

7.7. Reviews B and X and the signed non-anonymous review to the MIT Press of my first book used the forceful and infrequently used word "Strengthen"

Most scientific peer reviewers use a rather passive and bland vocabulary even though most English teachers advocate the use of forceful and colorful words.

Most writers have favorite words that they use (and overuse) in their writings.

We again make reference to the computer file containing the 64,109 words of the 316 paper review forms from the 86 peer reviewers of the genetic programming papers at the Genetic Programming 1996 Conference.

The forceful and colorful verb "strengthen" does not appear at all in the computer file containing the 64, 109 words.

However, this particular word does appear in each of the following 4 very short documents: Of course, the use of a particular forceful colorful word that is not commonly used by scientific peer reviewers does not alone establish the identity of the plagiarizers involved here.

7.8. Both reviews A, X, and the signed non-anonymous review to the MIT Press of my first book share certain antagonistic words

We previously noted that both reviewers A and B of my MLC paper on empirical discovery contained several occurrences of rarely used hostile words. See Section 2.13.

We also previously noted that reviewers #1, #2, and #3 of my paper submitted to the Evolutionary Computation journal also contained several occurrences of rarely used hostile words. See Section 4.8.

In the above cases, the appearance of the same hostile words in the short reviews for the same submitted paper indicated that these words may have migrated from one review document to another during the plagiarism process.

In this section, we suggest that the common appearance of some of these same rarely used hostile words in two reviews for different papers may suggest, to a small degree, the possibility that the same person was the reviewer for more than one paper.

When we searched the computer file containing the 64,109 words of the 316 paper review forms from the 86 peer reviewers of the genetic programming papers at the Genetic Programming 1996 Conference, we found the following frequency of occurrence for the following words: Reviewer A of my MLC paper on empirical discovery says,
The presentation suffers from an abundance of irrelevant details
Reviewer X of my MLC paper on optimal control strategies says,
The presentation suffers from the presence of irrelevancies
Reviewer A of my MLC paper on empirical discovery says,
results are claimed to appear
The word "claimed" also appears 2 times in the signed, non-anonymous review to the MIT Press of my first book (December 10, 1991) written by John Grefenstette of Code 5514 of the Naval Research Laboratory in Washington:
The claim that
John Grefenstette also mentions,
claims of significance
Of course, the use of several different hostile words that are not commonly used by scientific peer reviewers does not alone establish the identity of the plagiarizers involved here.


Author: John R. Koza
E-Mail: NRLgate@cris.com
Go to top of NRlgate Home Page
Go to Abbreviated Table of Contents
Go to Detailed Table of Contents
Go to Complaint Letter to the Evolutionary Computation journal
Go to top of Previous Page
Go to top of Next Page