NRLgate -
Plagiarism by Peer Reviewers

Sections 1.0 thru 1.8

This page is part of the NRLgate Web site presenting evidence of plagiarism among scientific peer reviewers involving 9 different peer review documents of 4 different journal and conference papers in the fields of evolutionary computation and machine learning.

This page contains sections 1.0 through 1.8 of "Background Information."

Go to NRLgate Home Page
Go to Abbreviated Table of Contents
Go to Detailed Table of Contents
Go to Complaint Letter to the Evolutionary Computation journal
Go to top of Next Page

1. Background Information

Scientific journals and conferences provide a mechanism for scientists to report research results and exchange ideas in order to advance the progress of science.

Publication in scientific journals and conferences is an important element in the careers of most scientific researchers as reflected by the well-known "publish or perish" principle. The presence or absence of publications in scientific journals and published conference proceedings plays a pivotal role in determining retention, promotion, and advancement in both the academic and governmental research communities, in determining tenure in the academic world, and in determining the funding of research projects in both the academic and governmental research communities. Thus, a decision to accept a submitted paper for publication translates directly into tangible benefits for the submitting authors while a decision to reject is a tangible loss (particularly in a journal or conference that is potentially viewed as being central to the researcher's main field of work).

Most articles in most scientific journals and conference proceedings are published after a public solicitation of papers and a competitive evaluation process. The process of selecting articles for publication in scientific journals is competitive (often highly competitive). Only a fraction (typically between a quarter and a half) of submitted papers are accepted.

1.1. Genetic Programming (GP)

Genetic programming (GP) is a recently developed method for automatically creating a computer program to solve a problem. Genetic programming addresses one of the central questions of computer science --- namely, how to get a computer to automatically solve a problem without having to explicitly write, by hand, a detailed computer program.

By way of background, I am regarded as the developer and most prominent proponent of the relatively new technique of genetic programming. My first scientific paper on genetic programming appeared in 1989 at the International Joint Conference on Artificial Intelligence (IJCAI-89). I am also author of two books from the MIT Press on this subject, namely the 819-page book Genetic Programming: On the Programming of Computers by Means of Natural Selection published by the MIT Press (Koza 1992) and and the 746-page book Genetic Programming II: Automatic Discovery of Reusable Programs (Koza 1994a). A third book on genetic programming is in the works.

Genetic programming is a form of artificial intelligence and bridges two fields: The term evolutionary computation (EC) refers to mathematical problem-solving methods that are based on the Darwinian principle of survival of the fittest and the ideas of evolution by means of natural selection. Evolutionary computation encompasses the subfields of genetic algorithms (GAs), genetic programming (GP), classifier systems, evolution strategies, and evolutionary programming. Genetic programming is a variation of the genetic algorithm. Over 5,000 scientific papers have been published in the field of evolutionary computation (mostly in the last 15 years) and there are over a thousand researchers around the world who are involved to some extent in this field.

Machine learning (ML) is a subfield of artificial intelligence (AI) that is concerned with automated methods by which computers learn to solve problems.

Genetic programming is the fastest growing segment of the field of evolutionary computation. Since 1989, there have been over 600 scientific papers published in this field (including over 70 of mine). In 1996, the first conference devoted to genetic programming was held with 292 registered attendees and with 73 regular papers contained in a proceedings book published by The MIT Press. Additional information about genetic programming and me can be obtained at John Koza's home page.

1.2. The Evolutionary Computation Journal (ECJ)

The Evolutionary Computation journal (ECJ) is currently the only scientific journal devoted entirely to the field of evolutionary computation.

Since its inception in 1992 (and until just recently), the Evolutionary Computation journal has had one editor-in-chief, three associate editors, and 32 editorial board members.

Kenneth DeJong has been Editor-In-Chief of the journal since the journal's inception in 1992 and for the entire time period involved herein. DeJong is employed at Code (department) 5510 of the Naval Research Laboratory (NRL) in Washington, D. C. He also teaches part-time in the Computer Science Department at George Mason University in Fairfax, Virginia where he supervises several PhD students in computer science (at least two of whom are full-time employees at Code 5514 and at least one of whom is a contractor of Code 5514).

John Grefenstette has been North American Associate Editor of the journal since the journal's inception. Grefenstette is Section Head of the Machine Learning research staff at Code 5514 of the Naval Research Laboratory in Washington. His "biographical sketch" at NRL states,
As NRL's Machine Learning Section Head, Dr. Grefenstette manages basic research programs in machine learning, establishes and maintains working relationships with leading academic and corporate centers of AI research, and develops applications of machine learning techniques to problems of Navy interest. Dr. Grefenstette is responsible for the evaluation of available machine learning techniques for application to problems of Navy interest, and the identification of new areas of research that may be needed to bring this technology to the application stage.
By way of background, John Grefenstette obtained his PhD from the University of Pittsburgh in 1980 under the chairmanship of Dr. Kenneth DeJong who was at the University of Pittsburgh at the time. In addition, Grefenstette is an editor of the Machine Learning journal and Kenneth DeJong is a member of its editorial board.

DeJong and Grefenstette serve (often in conjunction with one or more other employees of NRL) as peer reviewers for various scientific conferences, including the International Conference on Genetic Algorithms (ICGA), Foundations of Genetic Algorithms workshop (FOGA), IEEE International Conference on Evolutionary Computation (ICEC), Parallel Problem Solving from Nature conference (PPSN), and the recently created International Conference on Evolutionary Computation and its Applications (EvCA).

In addition, DeJong and Grefenstette serve (often in conjunction with one or two other employees of NRL) as peer reviewers for the Machine Learning Conference (next section).

Grefenstette is also on the editorial board of the Adaptive Behavior journal.

The Evolutionary Computation journal is typical of scientific journals and conferences in that its submitted papers go through a process of peer review to decide whether they will be published. The peer reviewers of the Evolutionary Computation journal are typically selected by the editors from the journal's advertised editorial board. The 32 members of the journal's editorial board are highly dispersed geographically and institutionally (ten being from outside the United States). None of the 32 members are located within the District of Columbia. The editorial board has never met as a group, but, instead, communicates by electronic mail (e-mail) and ordinary physical mail.

The relevant instructions to authors submitting papers to the Evolutionary Computation journal call for the submitting author to send five physical paper copies of his paper to the associate editor in the author's geographic area. Thus, I submitted a paper applying genetic programming to electrical circuit design (hereinafter referred to as the "ECJ paper") by sending it by U. S. mail to North American Associate Editor John Grefenstette at NRL.

For the first two years of the journal's operation, North American Associate Editor Grefenstette handled about 80% of all papers submitted to the journal. (The journal also had associate editors in Europe and Japan who handle papers originating in their designated geographic areas, but they have no known or likely connection with the subject matter of this document; accordingly, the term "North American Associate Editor" herein refers to North American Associate Editor John Grefenstette).

Upon receipt of a paper, the editors pick three peer reviewers and send one physical copy of the paper to each reviewer (usually by U. S. mail). At the same time, the journal sends the reviewer, by electronic mail, a standard 7-part paper review form that asks the reviewers for his evaluation of the paper. At the time of submission of my paper to the Evolutionary Computation journal, it was generally understood that all peer reviews were being done by members of the advertised editorial board; however, the journal occasionally appoints peer reviewers from outside the editorial board (and has recently done so with increasing frequency).

The peer reviewers normally read their physical paper copy of the submitted paper and then write a review using the journal's 7-part paper review form. The last two questions on the paper review form ask for the peer reviewer's recommendation to the editors about whether the journal should (1) accept the paper, (2) accept it on the condition that it be revised, or (3) reject it. After the peer reviewers finish writing their reviews, they transmit their review to the journal (typically by e-mail). The editorial leadership of the journal then reads the three reviews written by the peer reviewers, weighs the possibly conflicting recommendations from the three reviewers, and reaches a decision on whether to publish the submitted paper.

The most common form of peer review for papers submitted to scientific conferences and journals is "anonymous" in the sense that an author of a submitted paper is ordinarily given the text of the reviews written by the peer reviewers for his submitted paper, but not the names of the peer reviewers.

Out of 96 papers published by the Evolutionary Computation journal during its first 3 full years, 3 have been on genetic programming.

1.3. The Machine Learning Conference (MLC)

The International Conference on Machine Learning is an annual scientific conference that encompasses the field of machine learning.

The Machine Learning Conference (MLC) issues an annual Call For Papers (CFP) with a paper submission deadline that is typically in January of each year.

Papers submitted to the Machine Learning Conference go through a process of peer review to decide whether they will be published. Papers submitted to the Machine Learning Conference are typically read by two or three scientific peer reviewers drawn from the two dozen or so members of the conference's program committee. The peer reviewers write a review of each paper using the conference's standard paper review form. Finally, the program committee receives the reviews written by the peer reviewers and decides which papers will be accepted for presentation at the conference (typically held in June or July of each year) and for publication in the proceedings book of the conference (typically published at the time of the conference). The conference meets in various locations (e.g., Italy in 1996, Tahoe in 1995).

The members of the program committee are more or less the same from year to year. Most of the two dozen or so members of the program committee are highly dispersed geographically and institutionally. There have been between 2 and 4 members of the MLC program committee from the Naval Research Laboratory (NRL) since the late 1980s. For 1996, there were 4 members of the MLC program committee from NRL. For almost every year (and, specifically, for the years applicable to the particular MLC papers discussed herein), the members included Out of approximately 25 submissions since 1990, only one paper (in 1994 by Rosca) involving the use of genetic programming has been published by the Machine Learning Conference in its 13 year history. In 1995, all 7 submitted papers involving the use of genetic programming were rejected. One paper ridiculing genetic programming was published in 1995.

I have submitted papers to the Machine Learning Conference in various years (two of which will be referred to herein as the "MLC papers").

The Machine Learning Conference (MLC) regularly receives grants each year from the Office of Naval Research (ONR).

Many of the members of the program committee of the annual Machine Learning Conference are also on the editorial board of the Machine Learning journal. For example, John Grefenstette is an editor of the Machine Learning journal and Kenneth DeJong is a member of its editorial board.

No paper on genetic programming has ever been published in the Machine Learning journal.

1.4. Tools for Artificial Intelligence Conference (TAI)

The IEEE International Conference on Tools for Artificial Intelligence (TAI) is an annual scientific conference. The subject matter of the TAI conference is considerably broader than machine learning or evolutionary computation. TAI includes a wide range of subjects from the field of artificial intelligence.

In the one year when I submitted a paper to the TAI conference (hereinafter referred to as the "TAI paper"), papers on evolutionary computation papers represented only a tiny fraction (8 out of 120) of the published papers. In that year, there were only 4 reviewers on TAI's "List of Reviewers" who were involved in any known way in the field of evolutionary computation. All 4 of the EC-knowledgable reviewers at the TAI conference had the same institutional affiliation, namely the Naval Research Laboratory:

1.5. Other conferences in evolutionary computation

The International Conference on Genetic Algorithms (ICGA) is a biannual conference covering all aspects of evolutionary computation, including genetic algorithms, genetic programming, classifier systems, evolution strategies, and evolutionary programming.

The conference committee for the first International Conference on Genetic Algorithms (ICGA-85) consisted of John Grefenstette (then at Vanderbilt University), Kenneth DeJong of NRL, Lashon Booker of NRL, Stephen F. Smith of CMU in Pittsburgh and Prof. John Holland of the University of Michigan (the inventor of both the genetic algorithm and the genetic classifier system).

By way of background, Lashon Booker is a former employee of Code 5510 of the Naval Research Laboratory and is currently working for a military contractor in suburban Virginia.

David Schaffer has co-authored papers with Grefenstette during the period when both were at Vanderbilt University in the mid 1980's, and now works in upstate New York at Philips Laboratories.

Stephen F. Smith (who also received his PhD at the University of Pittsburgh under Kenneth DeJong in 1980 and has been at Carnegie Mellon University (CMU) in Pittsburgh since the early 1980's).

In 1987, the ICGA-87 program chair was John Grefenstette (then at NRL) and the program committee included Kenneth DeJong (NRL), Lashon Booker (NRL), and Stephen F. Smith of CMU in Pittsburgh and four others. In 1989, Kenneth DeJong was the conference chair; David Schaffer was the program chair; John Grefenstette was finance chair; and the conference committee included Lashon Booker. In 1991, Kenneth DeJong and David Schaffer were conference co-chairs, and Lashon Booker was one of two program co-chairs.

The Navy Research Laboratory has made financial contributions to each ICGA conference since 1985. The Office of Naval Research (ONR) has made financial contributions to ICGA in recent years.

In addition, Grefenstette, DeJong, and other NRL personnel are regularly on the program committees and are reviewers for numerous other conferences and workshops in the fields of evolutionary computation and machine learning, including

1.6. The Naval Research Laboratory (NRL)

The Naval Research Laboratory (NRL) is a well-regarded institution that does research and development work within the Department of Navy. Its commanding officer is Captain Bruce W. Buckley, USN. Dr. Timothy Coffey is the Director of Research at NRL.

Code 5510 is involved with research into artificial intelligence and is also known as the Navy Center for Applied Research in Artificial Intelligence (NCARAI). Dr. Alan L. Meyrowitz has been the director of Code 5510 since 1991. Kenneth DeJong is Chief Scientist in Code 5510.

Code 5514 is one of four research groups within Code 5510. Code 5514 specializes in machine learning. John Grefenstette is the Section Head of Code 5514.

The Office of Naval Research (ONR) funds a substantial part of the activity at the Naval Research Laboratory in Washington, DC by contracting with NRL to perform work. NRL is an internal research resource within the Department of the Navy. NRL is consistently funded by ONR from year to year. In a loose sense, ONR is the customer and NRL obtains funds by selling ONR work on particular research and development projects.

1.7. The SAMUEL system developed in-house at the Naval Research Laboratory (NRL)

One of the missions of the Naval Research Laboratory is to seek out emerging new scientific technologies and to study them for possible usefulness to the Navy.

For example, the NRL "biographical sketch" of John Grefenstette of Code 5514 of the Naval Research Laboratory states,
Dr. Grefenstette is responsible for the evaluation of available machine learning techniques for application to problems of Navy interest, and the identification of new areas of research that may be needed to bring this technology to the application stage.
In some instances, staff members of codes 5510 and 5514 go beyond "evaluation of available machine learning techniques" and have invented new technologies in-house at NRL.

For example, John Grefenstette of Code 5514 of NRL is the inventor and the most prominent proponent of a system for machine learning called SAMUEL (Grefenstette 1989, 1991, 1992). SAMUEL addresses some of the same problems that are addressed by genetic programming (of which I am regarded as the most prominent proponent). For example, SAMUEL is particularly suitable for developing control strategies for controlling autonomous robots and for developing strategies for pursuer-evader games (evasive maneuvers).

Code 5514 of the Naval Research Laboratory has expended considerable effort on the SAMUEL system since the late 1980s. On several hundred occasions, NRL personnel have traveled extensively to give talks about SAMUEL as part of a "technology transfer" effort from government laboratories to university and other scientific audiences. Numerous scientific papers have been published on the SAMUEL system at various conferences and journals covering the fields of evolutionary computation and machine learning. The authors of the papers on SAMUEL are almost always affiliated with NRL and include John Grefenstette, Kenneth DeJong, various NRL employees (e.g., Connie Ramsey and Alan Schultz of Code 5514), and various NRL contractors (e.g., Mitchell Potter of Code 5514) (Potter, DeJong, and Grefenstette, 1995; Schultz and Grefenstette 1990; Grefenstette, Ramsey, and Schultz 1990).

1.8. The connection between the scientific peer review process and the problem of measuring the internal performance at the Naval Research Laboratory (NRL)

High-level policy makers who manage the work of scientists and engineers working with numerous different technologies face a difficult problem in objectively evaluating such work.

The federal government spends a considerable amount of money on research and development in machine learning and evolutionary computation. However, nearly all of its is allocated by the government to agencies within the government (such as Code 5514).

In allocating funds to its various departments and in making decisions about the retention and advancement of scientific personnel, the high-level management of both Naval Research Laboratory and ONR has developed numerous ways to try to objectively measure internal performance.

Information from the outside scientific community has the obvious disadvantage of originating from sources that are not directly attuned to the mission of the Naval Research Laboratory; however, such outside information has the alluring advantage to the Naval Research Laboratory management of appearing to be arms-length and objective. Of course, the usefulness of the measurements provided by the peer review process by the outside scientific community depends on the extent to which these measurements are not affected by the actions of the very departments and personnel whose work is being measured.

Publication by scientific conferences and journals of the research papers written by Naval Research Laboratory authors is one measure of scientific merit. The number of citations to papers is another measure of scientific merit (that carries significant weight at Naval Research Laboratory apparently because it is thought that citations are an indicator of successful "technology transfer").

However, the affirmative appearance of publications by NRL authors is not the most important part of the publication process.

The absence of published papers in the scientific literature that address problems of interest to Naval Research Laboratory strongly supports in-house funding in such seemingly neglected problems.

Thus, the funding of a department may depend on the extent to which a department can create the impression to its own management that non-governmental scientists are neglecting a particular field.

It should be noted that the SAMUEL technique (previous section) that was invented in-house in Code 5514 of the Naval Research Laboratory and the technique of genetic programming (GP) are rival technologies within the fields of machine learning and evolutionary computation that were each developed in the late l980s to solve artificial intelligence problems.
Author: John R. Koza
E-Mail: NRLgate@cris.com
Go to NRLgate Home Page
Go to Abbreviated Table of Contents
Go to Detailed Table of Contents
Go to Complaint Letter to the Evolutionary Computation journal
Go to top of Next Page