Response to the Prometheus Society
1998/99 Membership Committee Report

Kevin Langdon

Published in Gift of Fire #103, February 1999; one paragraph has been slightly revised
for publication on this site (noted below) and a number of notes have been added.
An addendum which appeared in
GoF #104 appears at the end of this article.

The Membership Committee Report can be found at:
<http://www.eskimo.com/~miyaguch/MCReport/mcreport.html>

 

I have read the report of the Prometheus Membership Committee with great enjoyment. I'm like a kid in a candy store he's previously been kicked out of. There's a lot of good data and some well-done analysis, but there's a lack of vision in this report. Many details have been painstakingly worked over while key principles were ignored. I will provide specifics below.

This document responds directly to three sections of the report: Section IV, Committee Operating Procedures; Section V, Recommendation; and Section VI, Action Items. Material from other parts of the report are discussed under these headings.

My own recommendations appear at the end of this response.


IV. Commitee Operating Procedures

I was removed from the Membership Committee by Chairman Fred Vaughan when I would not accept the conditions outlined in this section.

The committee performed its business primarily over the Internet using e-mail messages that were routed only to other members of the Membership Committee except as authorized specifically in writing by the chairman. (This was felt to be particularly important because we would be discussing topics that could compromise the tests accepted for qualification to the Society.)

No one wants to give aid and comfort to cheaters on our admission tests by giving them hints about where to find the answers to the Mega and Titan tests, but that did not require setting up a closed circle with regard to the general operations of the Membership Committee.

Discussions of specific problems and whether or not they were to be considered compromised based on answers circulated and the specifics of where such data is available typically involved only a subset of the committee.

That's sensible.

Specific positions argued and quotations of individuals during the deliberation of the Membership Committee will remain confidential. Detailed rationale for all recommendations of the committee are provided in this final report signed by all committee members. A pledge of confidentiality for the discussions in deliberation was a prerequisite for continued appointment to this committee. It was decided that a single consensus position would be incorporated into this report if such a consensus could be obtained.

This stacks the deck against those who may have different views of the matters addressed in the MC Report, by artificially concealing real differences--and I have been told, by more than one member of the committee, that there were, in fact, significant differences on many issues.

If more than a single individual shared a position counter to the consensus, that position is summarized in the report as well subject only to the desires of those sharing the position.

Intolerance of individual dissent is one of the earmarks of totalitarianism. A minority of one in the MC could be more convincing than all the rest and produce an alternative recommendation that is voted in by the membership--and permitting him to try would give the members a choice. I have had to forego the intensely pleasurable and educational experience of working with knowledgeable colleagues on an important project in order to have the freedom to write the document you are now reading, although I had no idea at the time I refused to be bound by the Chairman's conditions that the committee's recommendation would be so far from what I believe to be sensible and prudent.

Intellectual rights to publication of material generated as a part of the deliberations of this committee belong to the individual or individuals who generated the material, but publication must be approved by the committee as expressed in writing by the chairman to assure the following: 1) All individuals who contributed to the material to be so published shall be cited if they so desire and 2) No data contained in the material to be published shall compromise Prometheus Society entry criteria.

This attempt to restrict the intellectual property rights of members of the committee is opressive. There's nothing wrong with wanting to "assure" either 1) or 2), but it should be up to each individual committee member to attend to these matters.

Agreement to these operating conditions has been a prerequisite for continued appointment to this committee. Concurrence with these conditions is tacit by a member's not having notified the chairman of a wish to resign appointment.

"Negative Option" is invalid; no one on the committee who did not explicitly agree to these conditions is bound my them (nor, of course, is anyone who comes to the conclusion that the conditions are in conflict with his or her duty to the membership of Prometheus).


V. Recommendation

I agree with the Committee only with regard to the LAIT and (with small differences) the Mega and Titan tests. It is a frightening prospect that we could soon be left with no currently-scored tests with the properties required to maintain our admission standards, but this is the real situation. The appropriate remedy is the development of new high-quality, highly g loaded, high-range tests, not to grasp at straws and begin admitting unqualified people.

We on the Membership Committee are proud to present to the Prometheus Society our proposal for revised entry requirements to the Society. We aver that it is our considered opinion that this recommendation, if adopted by the membership, will be in the best interest of this Society and its members. Our recommendation is as follows:

If I'd been responsible for these recommendations, I would not be proud.

Entry into the Prometheus Society based on a Mega or Titan score shall no longer be allowed after the date of issuance of the issue of Gift of Fire in which acceptance of this recommendation is indicated to have been ratified by the membership. Anyone having secured a raw score of 36 on either of these tests dated before that date shall be entitled to rights and privileges of the Society.

I have rather stricter instincts in this regard, because data compromising test items has been available for several years, but I agree, in any case, that acceptance of scores on the full-length Mega and Titan tests should be halted.

Anyone with a score of 164 or greater on the LAIT scored before December 31, 1993 shall be entitled to rights and privileges of the Society.

I agree with this.

Anyone with a score of 1560 on the "old" SAT (taken before April 1, 1995) shall be entitled to rights and privileges of the Society.

Figure 7, on page 25 of the report, shows that the reliability of scores on the Mega Test declines precipitously slightly above the four-sigma level, as the authors of the report pointed out. But this is a well-known property of tests in general. No test can be considered reliable very close to the extremes of its range, but the SAT score proposed for admission to Prometheus is only four IQ points below the test ceiling, and the incidence of SAT scores is anomalous above 1560 (see the tables captioned "SAT high range data distribution in 1984" and "SAT high range data distribution in 1984-1989," on page 42 of the report).

Consider Figures 5 (p. 24), 7 (p. 25), 12 (p. 31), 16 (p. 34), and 24 (p. 49). Observe how linear these distributions are in the middle. As a scientist, I look for patterns. An appropriately conservative conclusion is that departures from linearity at the extremes of a test's range may be nothing more than noise.

Figures 21 and 22 on page 41 are labeled "SAT discrimination capabilities" and they are very linear indeed--as well they might be, given that they plot "raw scores" against "scaled scores"! This clearly says nothing about the relationship of the SAT to any external criterion.

Section 8.5, in which these figures appear, is titled "The ability of the SAT to discriminate at the high end of its scale," but it does not establish this. It includes the statement, "In fact, however, in a population of 3 million there should be over 100 individuals scoring at the 1-in-30,000 level. On any given year less than ten individuals obtained a perfect score on the old SAT with on the order of 100 or less scoring 1560 or more and, therefore, it is safe to say that the 1-in-30,000 level is achieved by these individuals." This omits the possibility of factors other than ability limitations limiting the achievement of near-ceiling scores, such as careless errors (i.e., correct reasoning processes mistranslated clerically) and imperfections in the test (some of which are discovered from time to time by bright students).

[Note: The paragraph above, which is of my authorship, was incorrectly set as an indented quote in the published version of this article.]

Anyone with a score of 1610 on the "old" GRE (taken before October 1, 1981) shall be entitled to rights and privileges of the Society.

The GRE has the same problem as the SAT, a slightly lower ceiling, and less impressive correlations than the SAT with the tests we now use for admission purposes.

Anyone with a score of 98 on the MAT shall be entitled to rights and privileges of the Society.

This is only two raw score points (out of 100) below the ceiling.

Anyone with a raw score of 88 on the Cattell Culture Fair III (A+B) obtained at an age of 16 years of age or older shall be entitled to rights and privileges of the Society.

The Cattell Culture Fair III appears to have more ceiling than most other tests, but it has another, quite severe problem: it has a time limit of only 12.5 minutes (15 seconds per item).

In Chapter 5 ("Varieties of Mental Test Items") of Bias in Mental Testing, Arthur Jensen wrote: "The personal tempo factor actually has little if any correlation with intelligence. E.L. Thorndike [The Measurement of Intelligence, 1927] tried to determine the correlation between speed and altitude. To measure speed he used a large number of quite easy items and recorded the time that subjects required to complete a given number of such easy items without error. To measure altitude he gave subjects a succession of items steeply graded in difficulty; the attitude score was the difficulty level beyond which the subject failed 50 percent or more of the items. The correlations between the measure of speed (the reciprocal of time) and of altitude average about .40 in several groups (about .46 when corrected for attenuation)."

Additionally, no scatterplot data is presented showing the relationship of scores on this test to any other measure.

Anyone with a score of 160 on the WAIS-R obtained at an age of 16 years or older shall be entitled to rights and privileges of the Society.

I don't believe that the WAIS-R has a ceiling of 190 for adults, as claimed. A test that measures IQ's as low as 45 must contain items spread out to cover the range from at most 45 to at least whatever its ceiling is. This cannot allow enough very difficult items to discriminate reliably at the one-in-three-billion level. However, I might be willing to believe that the WAIS-R has a high ceiling if I were to see scatterplots--like those exhibited for the Mega, SAT, GRE, CTMM, LAIT, MAT, and even Thinkfast (though there are major problems with the Thinkfast data, as I will show below)--relating scores on the WAIS-R with an external criterion.

While there is a quotation from Jean Spruill, the author of a review of the WAIS in Test Critiques, to the effect that "the WAIS-R subtests are all relatively good measures of the general factor (g) of intelligence," no quantitative data has been provided. We need data on the g loading of the WAIS-R.

Another important problem with the WAIS is that scores are often reported in age-corrected form. If we adhere to our policy up till now in this area, we would need to have scaled scores. The table on page 51 indicates that such scores are generated before age-corrected IQ scores are computed.

At the very least, a good deal more data is needed before we can accept the WAIS for admission to Prometheus.

Anyone with a score of 21 on the Mega27 shall be entitled to rights and privileges of the Society, if a validated accompanying score on an accepted test for demonstrating a 1-in-1,000 cognitive ability according to that test is provided to the Membership Officer along with proof of the mega27 score.

The Mega27 is a praiseworthy attempt to salvage a usable psychometric instrument from the Mega Test. A similar subset of the Titan, which has been discussed in the committee, would also be helpful, though it cannot be expected that either test will have a long useful life.

I would not have chosen to discard a fairly large number (7) of "easy" items, to provide some margin against future compromised items and to retain some symmetry of item types within the test. Psychometricians generally consider 30 (or, preferably, 40) items to be a lower limit for acceptable reliability.

[Note: Editor Fred Vaughan censored the number of easy items, claiming that this information could be used to compromise the Mega27, when  this article was published in GoF (and before the Mega27 had actually been compromised). I didn't object, because there were too many other  issues on the table at the time and this was too trivial a matter to bother with.]

Extending the linear portion of Figure 12 (p. 31), on the conservative principles discussed under the Mega Test above, the qualifying score on the Mega27 should be 22, not 21.

Requiring a score at the 99.9th percentile does little to confirm a score on the Mega27, given that 97% of those with 99.9th percentile scores are not qualified at the Prometheus level.

[Note: The Mega27 was compromised by publication of answers on the Internet. According to a report by Membership Officer Bill McGaugh published in Gift of Fire #112, January 2000, Mega27 scores subsequent to November 28, 1999 are no longer accepted for Prometheus admission purposes.]

And, for a trial period of one year: Anyone with a validated score of Brain Master +11 on the chronometric battery provided by ThinkfastTM obtained at an age of 16 years of age or older shall be entitled to rights and privileges of the Society, if a validated accompanying score on an accepted test for demonstrating a 1-in-1,000 cognitive ability is provided to the Membership Officer along with proof of the ThinkfastTM score.

The data provided for Thinkfast includes one scatterplot of SAT vs. Thinkfast scores, but it shows only 28 data points, and there were only 46 scores on all tests combined. A sample of this size, composed of students personally instructed by someone as brilliant as Bill McGaugh, cannot be regarded as statistically adequate.

[Note: The paragraph above was edited to remove a misunderstood point about Bill's data, subsequently clarified.]

The information presented in Section 8.7 on the chronometric testing done in controlled laboratory conditions by Dr. Jensen and others has not been shown to apply to Thinkfast. The committee speculates that the g loading of Thinkfast should be around .7, but no data is provided to support this.

Why should a person have to be at least 16 if he or she can make a qualifying score (without age-correction)? The idea of requiring that the testee be at least 16 suggests to me a fear that video-game fanatics will make short work of Thinkfast (it seems likely that it's mainly been taken by intellectual "nerds" so far, although, admittedly, membership in these two groups is highly correlated). (This is simply my association and not a claim to have read the minds of the committee.)

The use of a 99.9th-percentile score for confirmation puposes makes very little sense, as I mentioned under the Mega27 above.

After one year, the following data will be used to determine whether to retain the test permanently, extend the trial period or discontinue this test as an entry requirement to the Society.

1. numbers of applicants to the Society who use this ThinkfastTM test criterion,

2. accompanying scores on standard tests of applicants who use this ThinkfastTM test criterion,

3. additional statistics available on high scores of ThinkfastTM participants,

4. our increased understanding of ThinkfastTM as a chronometric/ psychometric instrument.

Wouldn't it make better sense to do this kind of research before adding this strikingly different and unproven instrument to our list of qualifying scores?

Where a score of 1-in-1,000 is required on one of the following tests: ACT (32), Old SAT (1450), New SAT (1520), GRE (1460), GRE Analytical (760), MAT (85), Stanford Binet IV (149), Weschler Intelligence Scales (146), Cattell CF III (A+B) (149), Cattell Intelligence Test Scale IIIB (173 old norming), RAPM II (150), Mensa Admission Tests (149), Cognitive Abilities Test (CAT) (149)

[Note: The paragraph above, quoted from the MC Report, was incorrectly set as being of my authorship in the published article.]

These figures are in the right ballpark, though Raven Advanced Progressive Matrices scores are generally reported as raw scores, with the maximum possible score being 36.

5.2 Rejection of the recommendation

If our recommendation is rejected by a majority of voters, the Prometheus Society will retain the entry requirements established by vote in 1997.

A good idea. See the end of this critique for some alternative recommendations on our admission standards.


VI. Action Items

We have accepted the following outstanding items that we recommend for further action.

6.1 Obtain written agreement with Ron Hoeflin on Mega27

Firm up agreement in principle with Ron Hoeflin on scoring procedures and application processing for the Mega27 test. Also obtain written specifications of how profile data is to be handled by Membership Officer.

Yes, this is needed. We can't use the Mega27 without Ron's cooperation. But I still think it would be better to put back some of the "easy" items that were dropped in creating the Mega27.

6.2 Evaluate Titan test

We have agreement in principle with Ron Hoeflin to obtain data for 500 individuals who have taken the Titan test. We must perform analyses similar to those which gave rise to the Mega27 for the Titan to avoid compromised problems. Also solidify norming for the Titan.

This, too, must be done, and a short form of the Titan must be developed.

6.3 Consider the relationship of age and intelligence

There are a couple aspects of IQ variations with age that must be considered in some depth with regard to our entry requirements:

1. whether to allow test results for individuals under 16 years of age and

2. whether to consider an age profile (particularly applicable to those over 30 years of age) for intelligence criteria.

The approach that we've taken on this in the past is that what we're interested in is raw brain power, not whether one is "smart for one's age." I think that we should continue with this standard.

6.4 WAIS subtest qualification possibilities

This needs study, along with the WAIS-R itself.

6.5 Evaluate results of one-year trial period of use of Thinkfast to qualify applicants for Prometheus.

After one year, the following data will be used to determine whether to retain the test permanently, extend the trial period or discontinue this test as an entry requirement to the Society:

1. numbers of applicants to the Society who use this ThinkfastTM test criterion,

2. accompanying scores on standard tests of applicants who use this ThinkfastTM test criterion,

3. additional statistics available on high scores of ThinkfastTM participants,

4. our increased understanding of ThinkfastTM as a chronometric/ psychometric instrument.

Hopefully, this will be done before Thinkfast is adopted as an entrance test for Prometheus.

6.6 Investigate tests in other languages, including translations of English tests.

A good project after we get some of the basics worked out.


My Recommendations

First, I strongly recommend a "no" vote on the recommendation of the Membership Committee. I have outlined my reasons above.

It is essential for us to take action with regard to the compromised items on the Mega and Titan tests. Two different approaches to this have been proposed: not accepting Mega/Titan scores earned after the date the proposal is adopted, or not accepting scores earned after December 31, 1994 (that's approximately when the Internet began to seriously impact the tests and it's easy to remember that scores earned in the early 90's are O.K.).

Here are four proposals submitted for a vote of the Prometheus membership, covering all of these:

1A. Scores on the Mega Test earned after the date of adoption of this proposal shall not be accepted for Prometheus admission.

1B. Scores on the Mega Test earned after December 31, 1994 shall not be accepted for Prometheus admission.

2A. Scores on the Titan Test earned after the date of adoption of this proposal shall not be accepted for Prometheus admission.

2B. Scores on the Titan Test earned after December 31, 1994 shall not be accepted for Prometheus admission.

Vote for the alternative you prefer, for each pair. I doubt that many will choose neither, given the evidence that many Mega and Titan items have been compromised.

[Note: I withdrew these four proposals after the MC Report recommendation was ratified by the Prometheus Membership.]

I think we need more discussion on the Mega27, so I am not making a proposal regarding it at this time.

As for other tests, there are a few that might be usable if we had better data on them. (Members of the Membership Committee mentioned to me that finding data was the most difficult part of their job.)

However, at the present time, I can only recommend quick action to make short forms of the Mega and Titan tests available.

I do have another test ready to go once I resolve my differences with the California Board of Psychology or make my tests available through an out-of-state entity.

The Membership Committee has recommended a total shift in our admission criteria. It would be precipitous to make such a change without much more opportunity for discussion than the schedule on which the Membership Committee Report is being brought to a vote will allow. Vote "no" on this all-or-nothing proposition and give the Prometheus Society time to think seriously about its future.

 

A Few Additional Notes on the MC Report

[Published in Gift of Fire #104, March 1999]

[Paragraph concerning correction of my remarks about Bill McGaugh's data, corrected above, deleted.]

With regard to my advocacy of linear fitting, without bending the curve at the top and bottom, the relevant reference is from The g Factor (page 289):

Linearity of Regression. The regression line, in terms of validity, is the line that best fits the relationship between the predicted criterion and the predictor test. In nearly all studies of the predictive validity of highly g-loaded tests, this regression line is linear (i.e., a straight line). This is illustrated, for example, in Figure 9.2, from a study where the Scholastic Aptitude Test (SAT) was used to predict college grade-point average (GPA) for students just entering college. The regression line per se is not shown in this graph, but the mean GPAs at equal intervals on the scale of SAT scores all fall on a single straight line, from the lowest possible SAT score (200) to the highest possible score (800). In other words, the regression of the criterion measure (GPA) on the predictor measure (SAT score) is linear throughout the entire range of the GPAs and SAT scores. A similar picture emerges in hundreds of studies of the prediction of training success and job performance with various g-loaded tests.

In my response to the MC Report in GoF #103 I noted that, in my opinion, the four-sigma cutoff score on the Mega27 should be 22, not 20 or 21. Applying similar reasoning to the SAT data presented in Section 8.5, I conclude that 1580 would be a more accurate cutting score than 1560, strengthening my argument that the cutoff would need to be too near the test ceiling for accurate discrimination.

If we admit people who have near-ceiling scores on the standard tests, as recommended by the MC, we'll get the valedictorians, the people who dot all their i's and j's and cross all their t's and z's, not the abstract thinkers.

I once again call upon members of Prometheus to reject the recommendations of the Membership Committee and take another, and more careful, look at the problem of admission standards.

[Fred Vaughan's appended "Clarifications" omitted here.]

 

[Intelligence: Essays & Reviews]

[Consciousness and Mind]

[Polymath Systems Home Page]