STEP: The Calibration and Resolution of Confidence in Perceptual Judgements (1994) [35037]

This article applies to:

E-Prime 3.0

E-Prime 1.0

Detail

Experiment Author: Baranski, J.V., & Petrusic, W.M. Adapted from STEP and used with permission of Brian MacWhinney

Experiment Description

This experiment tests the confidence that participants have in difficult perceptual judgements. It is similar to the one in Baranski and Petrusic 1995, except that it does not include the general knowledge portion.

Experiment Instructions

In this experiment, participants encounter a single trial consisting of 32 samples. 32 sets of coordinates for the stimuli are arranged in a list and presented randomly. The coordinates are set based on pixel values. Participants use a keyboard to make responses Adjusting the experiment display size may compromise the accuracy of the data. A 640x480 pixel display is used.

Experiment Citation

Baranski, J.V., & Petrusic, W.M. (1994). The calibration and resolution of confidence in perceptual judgments. Perception & Psychophysics, 55, 412-428. (See PDF below.)

Experiment Abstract or Original Experiment Abstract

Confidence rating based calibration and resolution indices were obtained in two experiments requiring perceptual comparisons and in a third with visual gap detection. Four important results were obtained. First, as in the general knowledge domain, subjects were underconfident when judgments were easy and overconfident when they were difficult. Second, paralleling the clear dependence of calibration on decisional difficulty, resolution decreased with increases in decision difficulty arising either from decreases in discriminability or from increasing demands for speed at the expense of accuracy. Third, providing trial-by-trial response feedback on difficult tasks improved resolution but had no effect on calibration. Fourth, subjects can accurately report subjective errors (i.e., trials in which they have indicated that they made an error) with their confidence ratings. It is also shown that the properties of decision time, conditionalized on confidence category, impose a rigorous set of constraints on theories of confidence calibration.

Works Cited by the Experiment

 Adams, J.K. (1957). A confidence scale defined in terms of expected percentages. American Journal of Psychology, 70, 432-436.
 
Adams, J.K., & Adams, P.A. (1961). Realism of confidence judgments. Psychological Review, 68, 33-45.
 
Adams, P.A., & Adams, J.K. (1958). Training in confidence-judgments. American Journal of Psychology, 71, 747-751.
 
Allan, L.G. (1979). The perception of time. Perception & Psychophysics, 26, 340-354.
 
Allwood, C.M., & Montgomery, H. (1987). Response selection strategies and realism of confidence judgments. Organizational Behavior and Human Decision Processes, 39, 365-383.
 
Arkes, H.R., Christensen, C., Lai, C., & Blumer, C. (1987). Two methods of reducing overconfidence. Organizational Behavior and Human Decision Processes, 39, 133-144.
 
Audley, R.J. (1960). A stochastic model for individual choice behavior. Psychological Review, 67, 1-15.
 
Baranski, J.V. (1991). Theories of confidence calibration and experiments on the time to determine confidence. Unpublished doctoral dissertation, Carleton University, Ottawa, ON.
 
Baranski, J.V., & Petrusic, W.M. (1991). Confidence calibration and the scaling of doubt. Paper presented at the 52nd annual meeting of the Canadian Psychological Association, Calgary, AB.
 
Baranski, J.V., & Petrusic, W.M. (1992). The discriminability of remembered magnitudes. Memory & Cognition, 20, 254-270.
 
Baron, J. (1988). Thinking and deciding. Cambridge: Cambridge University Press.
 
Bjork, E.L, & Murray, J.T. (1977). On the nature of input channels in visual processing. Psychological Review, 84, 472-484.
 
Björkman, M., Juslin, P., & Winman, A. (1993). Realism of confidence in sensory discrimination: The underconfidence phenomenon. Perception & Psychophysics, 54, 75-81.
 
Brier, G.W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78, 1-3.
 
Carlson, B.W. (1993). The accuracy of future forecasts and past judgments. Organizational Behavior & Human Decision Processes, 54, 245-276.
 
Christensen-Szalanski, J., & Bushyhead, J. (1981). Physicians' use of probabilistic information in a real clinical setting. Journal of Experimental Psychology: Human Perception & Performance, 7, 928-935.
 
Coombs, C.H. (1964). A theory of data. New York: Wiley.
 
Dawes, R.M. (1980). Confidence in intellectual vs. confidence in perceptual judgments. In E.D. Lantermann & H. Feger (Eds.), Similarity and choice: Papers in honor of Clyde Coombs (pp. 327-345).
 
Bern: Hans Huber. de Finetti, B. (1937). La prevision: Ses lois logiques, ses sources subjectives. Annales de l'Institut Henri Poincaré, 7, 1-68. English translation in H.E.
 
Kyburg, Jr., & H.E. Smokler (Eds.)(1964), Studies in subjective probability (pp. 93-158). New York: Wiley.
 
Egeth, H.E., & Santee, J.L. (1981). Conceptual and perceptual components of interletter inhibition. Journal of Experimental Psychology: Human Perception & Performance, 7, 506-517.
 
Ferrell, W.R., & McGoey, P.J. (1980). A model of calibration for subjective probabilities. Organizational Behavior & Human Performance, 26, 32-53.
 
Festinger, L. (1943). Studies in decision: I. Decision=time, relative frequency of judgment and subjective confidence as related to physical stimulus difference. Journal of Experimental Psychology, 32, 291-306.
 
Fischhoff, B. (1982). Debiasing. In D. Kahneman, P. Slovic, & A. Tversky (Eds.), Judgments under uncertainty: Heuristics and biases (pp. 422-444). Cambridge: Cambridge University Press.
 
Fischhoff, B., & Macgregor, D. (1982). Subjective confidence in forecasts. Journal of Forecasting, 1, 155-172.
 
Fischhorff, B., Slovic, P., & Lichtenstein, S. (1977). Knowing with certainty: The appropriateness of extreme confidence. Journal of Experimental Psychology: Human Perception and Performance, 3, 552-564.
 
Fullerton, G.S, & Cattell, J.M. (1892). On the perception of small differences. Philadelphia: University of Pennsylvania Press.
 
Garrett, H.E. (1922). A study of the relation of accuracy to speed. Archives of Psychology, 56, 3-104.
 
Geller, E.S., & Pitz, G.F. (1968). Confidence and decision speed in the revision of opinion. Organizational Behavior & Human Performance, 3, 190-201.
 
Gigerenzer, G., Hoffrage, U., & Kleinbölting, H. (1991). Probabilistic mental models: A Brunswikian theory of confidence. Psychological Review, 98, 506-528.
 
Glenberg, A.M., & Epstein, W. (1985). Calibration of comprehension. Journal of Experimental Psychology: Learning, Memory, & Cognition, 11, 702-718.
 
Glenberg, A.M., & Epstein, W. (1987). Inexpert calibration of comprehension. Memory & Cognition, 15, 84-93.
 
Glenberg, A.M., Sanocki, T., Epstein, W., & Morris, C. (1987). Enhancing calibration of comprehension. Journal of Experimental Psychology: General, 116, 119-136.
 
Green, D.M., & Swets, J.A. (1966). Signal detection theory and psychophysics. New York: Wiley.
 
Griffin, D., & Tversky, A. (1992). The weighing of evidence and the determinants of confidence. Cognitive Psychology, 24, 411-435.
 
Heath, R.A. (1984). Random-walk and accumulator models of psychophysical discrimination: A critical evaluation. Perception, 13, 57-65.
 
Hellström, A. (1985). The time-order error and its relatives: Mirrors of cognitive processes in computing. Psychological Bulletin, 97, 35-61.
 
Henmon, V.A.C. (1911). The relation of the time of a judgement to its accuracy. Psychological Review, 18, 186-201.
 
Irwin, F.W., Smith, W.A.S., & Mayfield, J.F. (1956). Tests of two theories of decision in an "expanded judgment" situation. Journal of Experimental Psychology, 51, 261-268.
 
Jamieson, D.G., & Petrusic, W.M. (1975). Presentation order effects in duration discrimination. Perception & Psychophysics, 17, 197-202.
 
Johnson, D.M. (1939). Confidence and speed in the two-category judgment. Archives of Psychology, 34, 1-53.
 
Keren, G. (1987). Facing uncertainty in the game of bridge: A calibration study. Organizational Behavior and Human Decision Processes, 39, 98-114.
 
Keren, G. (1988). On the ability of monitoring non-veridical perceptions and uncertain knowledge: Some calibration studies. Acta Psychologica, 67, 95-119.
 
Keren, G. (1991). Calibration and probability judgments: Conceptual and methodological issues. Acta Psychologica, 77, 217-273.
 
Keren, G., & Wagenaar, W.A. (1987). Temporal aspects of probabilistic predictions. Bulleting of the Psychonomic Society, 25, 61-64.
 
Koriat, A., Lichtenstein, S., & Fischhoff, B. (1980). Reasons for confidence. Journal of Experimental Psychology: Human Learning & Memory, 6, 107-118.
 
Liberman, V., & Tversky, A. (1993). On the evaluation of probability judgments: Calibration, resolution and monotonicity. Psychological Bulletin, 114, 162-173.
 
Lichtenstein, S., & Fischhoff, B. (1977). Do those who know more also know more about how much they know? The calibration of probability judgments. Organizational Behavior & Human Performance, 20, 159-183.
 
Lichtenstein, S., & Fischhoff, B. (1980). Training for calibration. Organizational Behavior & Human Performance, 26, 149-171.
 
Lichtenstein, S., Fischhoff, B., Phillips, L.D. (1982). Calibration of probabilities: The state of the art to 1980. In D. Kahneman, P. Slovic, & A. Tversky (Eds.), Judgment under uncertainty: Heuristics and biases (pp. 306-334). Cambridge: Cambridge University Press.
 
Link, S.W. (1975). The relative judgment theory of two choice response time. Journal of Mathematical Psychology, 12, 114-135.
 
Link, S.W. (1992). The wave theory of difference and similarity. Hillsdale, NJ: Erlbaum.
 
Link, S.W., & Heath, R.A. (1975). A sequential theory of psychological discrimination. Psychometrika, 40, 77-105.
 
Luce, R.D. (1986). Response times. New York: Oxford University Press.
 
Lund, F.H. (1926). The criteria of confidence. American Journal of Psychology, 37, 372-381.
 
Maki, R.H., & Berry, S.L. (1984). Metacomprehension of text material. Journal of Experimental Psychology: Learning, Memory, & Cognition, 10, 663-679.
 
Maki, R.H., & Swett, S. (1987). Metamemory for narrative text. Memory & Cognition, 15, 72-83.
 
May, R.S. (1986). Inferences, subjective probability and frequency of correct answers: A cognitive approach to the overconfidence phenomenon. In B. Brehmer, H. Jungermann, & G. Sevon (Eds.), New directions in research on decision making (pp. 175-189). Amsterdam: Elsevier.
 
Morris, C.C. (1990). Retrieval processes underlying confidence in comprehension judgments. Journal of Experimental Psychology: Learning, Memory, & Cognition, 16, 223-232.
 
Münsterberg, H. (1894). Studies from the Harvard psychological laboratory: A psychometric investigation of the psychophysic law. Psychological review, 1, 45-51.
 
Murdock, B.B. Jr., & Dufty, P.O. (1972). Strength theory and recognition memory. Journal of Experimental Psychology, 94, 284-290.
 
Murphy, A.H. (1972). Scalar and vector partitions of the probability score: part 1. Two-state situation. Journal of Applied Meteorology, 11, 273-282.
 
Murphy, A.H. (1973). A new vector partition of the probability score. Journal of Applied Meteorology, 12, 595-600.
 
Murphy, A.H., & Winkler, R.L. (1977). Can weather forecasters formulate reliable probability forecasts of precipitation and temperature? National Weather Digest, 2, 2-9.
 
Myers, J.L., & Well, A.D. (1991). Research design & statistical analysis. New York: Harper Collins.
 
Nickerson, R.S., & McGoldrick, C.C. (1963). Confidence, correctness, and difficulty with non-psychophysical comparative judgments. Perceptual & Motor Skills, 17, 159-167.
 
Nickerson, R.S., & McGoldrick, C.C. (1965). Confidence ratings and level of performance on a judgmental task. Perceptual & Motor Skills, 20, 311-316.
 
Norman, D.A., & Wickelgren, W.A. (1969). Strength theory of decision rules and latency in short-term memory. Journal of Mathematical Psychology, 6, 192-208.
 
O'Connor, M., & Lawrence, M. (1989). An examination of the accuracy of judgmental confidence intervals in time series forecasting. Journal of Forecasting, 8, 141-155.
 
Oscamp, S. (1965). Overconfidence in case-study judgments. Journal of Consulting Psychology, 29, 261-265.
 
Peirce, C.S., & Jastrow, J. (1884). On small differences of sensation. Memoirs of the National Academy of Sciences, 3, 75-83.
 
Petrusic, W.M. (1992). Semantic congruity effects and theories of the comparison process. Journal of Experimental Psychology: Human Perception & Performance, 18, 962-986.
 
Petrusic, W.M. & Baranski, J.V. (1989a). Context, context shifts, and semantic congruity effects in comparative judgments. In D. Vickers & P. Smith (Eds.), Human information processing: Measures, mechanisms, and models (pp. 231-251). Amsterdam: Elsevier.
 
Petrusic, W.M., & Baranski, J.V. (1989b). Semantic congruity effects in perceptual comparisons. Perception & Psychophysics, 45, 439-452.
 
Petrusic, W.M., & Jamieson, D.G. (1978). The relation between probability of preferential choice and the time to choose changes with practice. Journal of Experimental Psychology: Human Perception & Performance, 4, 471-482.
 
Petrusic, W.M., & Jamieson, D.G. (1979). Resolution time and the coding of arithmetic relations on supraliminally different visual extents. Journal of Mathematical Psychology, 19, 89-107.
Phillips, L.D. (1973). Bayesian statistics for social scientists. London: Nelson.
 
Pierrel, R., & Murray, C.S. (1963). Some relationships between comparative judgment, confidence, and decision-time in weight-lifting. American Journal of Psychology, 76, 28-38.
 
Pike, A.R. (1968). Latency and relative frequency of response in psychophysical discrimination. British Journal of Mathematical and Statistical Psychology, 21, 161-182.
 
Pike, A.R. (1971). The latencies of correct and incorrect responses in discrimination and detection tasks: their interpretation in terms of a model based on simple counting. Perception & Psychophysics, 9, 455-460.
 
Pike, A.R. (1973). Response latency mechanisms for signal detection. Psychological Review, 80, 53-68.
 
Rabbitt, P.M.A., & Vyas, S.M. (1970). An elementary preliminary taxonomy for some errors in laboratory choice RT tasks. Acta Psychologica, 33, 56-76.
 
Ronis, D.L. & Yates, J.F. (1987). Components of probability judgment accuracy: individual consistency and effects of subject matter and assessment method. Organizational Behavior & Human Decision Processes, 40, 193-218.
 
Sanders, F. (1963). On subjective probability forecasting. Journal of Applied Meteorology, 2, 191-201.
 
Savage, L.J. (1954). The foundations of statistics. New York: Wiley.
 
Seward, G.H. (1928). Recognition time as a measure of confidence. Archives of Psychology, 99, 2-54.
 
Sharp, G.L., Cutler, B.L, & Penrod, S.D. (1988). Performance feedback improves the resolution of confidence judgments. Organizational Behavior & Human Decision Processes, 42, 271-283.
 
Smith, P.L, & Vickers, D. (1988). The accumulator model of two choice discrimination. Journal of Mathematical Psychology, 32, 135-168.
 
Solomon, I., Ariyo, A., & Tomassini, L.A. (1985). Contextual effects on the calibration of probabilistic judgments. Journal of Applied Psychology, 70, 528-532.
 
Tanner, W.P., & Swets, J.A. (1954). A decision making theory of visual detection. Psychological Review, 61, 401-409.
 
Thomas, E.A.C. (1971). Sufficient conditions for monotone hazard rate: An application to latency-probability curves. Journal of Mathematical Psychology, 8, 303-332.
 
Tomassini, L.A, Solomon, I., Romney, M.B., & Krogstad, J.L. (1982). Calibration of auditors' probabilistic judgments: Some empirical evidence. Organizational Behavior & Human Performance, 30, 391-406.
 
Trow, W.C. (1923). The psychology of confidence. Archives of Psychology, 67, 1-47.
 
Vickers, D. (1970). Evidence for an accumulator model of psychophysical discrimination. Ergonomics, 13, 37-58.
 
Vickers, D. (1979). Decision Processes in visual perception. New York: Academic Press.
 
Vickers, D., & Packer, J. (1982). Effects of alternating set for speed versus accuracy on response time, accuracy, and confidence in a unidimensional discrimination task. Acta Psychologica, 50, 179-197.
 
Vickers, D., Smith, P., Burt, J., & Brown, M. (1985). Experimental paradigms emphasizing state or process limitations: II. Effects on confidence. Acta Psychologica, 59, 163-193.
 
Volkmann, J. (1934). The relation of the time of judgment to the certainty of judgment. Psychological Bulletin, 31, 672-673.
 
Vreudgenhil, H., & Koele, P. (1988). Underconfidence in predicting future events. Bulletin of the Psychonomic Society, 26, 236-237.
 
Wagenaar, W.A. (1988). Calibration and the effects of knowledge and reconstruction in retrieval from memory. Cognition, 28, 277-296.
 
Wagenaar, W.A., & Keren, G. (1985). Calibration of probability assessments by professional blackjack dealers, statistical experts, and lay people. Organizational Behavior & Human Decision Processes, 36, 406-416.
 
Weaver, C.A. (1990). Constraining factors in calibration of comprehension. Journal of Experimental Psychology: Learning , Memory, & Cognition, 16, 214-222.
 
Winman, A., & Juslin, P. (1993). Calibration of sensory and cognitive judgments: Two different accounts. Scandinavian Journal of Psychology, 34, 135-148.
 
Wright, G. (1982). Changes in the realism and distribution of probability assessments as a function of question type. Acta Psychologica, 52, 165-174.
 
Wright, G., & Ayton, P. (1986). Subjective confidence in forecasts: A response to Fischhoff and MacGregor. Journal of Forecasting, 5, 117-123.
 
Wright, G., & Phillips, L.D. (1980). Cultural variation in probabilistic thinking: Alternative ways of dealing with uncertainty. International Journal of Psychology, 15, 239-257.
 
Wright, G., Wisudha, A. (1982). Distribution of probability assessment for almanac and future event questions. Scandinavian Journal of Psychology, 23, 219-224.
 
Yaniv, I., Yates, J.F., & Smith, J.E.K. (1991). Measures of discrimination skill in probabilistic judgment. Psychological Bulletin, 110, 611-617.
 
Yates, J.F. (1982). External correspondence: Decompositions of the mean probability score. Organizational Behavior & Human Decision Processes, 30, 132-156.
 
Yates, J.F. (1990). Judgment and decision making. Englewood Cliffs, NJ: Prentice-Hall.

See Also:

STEP: On the Calibration of Knowledge and Perception (1995) [35043]