Cogprints

Mathematical Principles of Reinforcement: Based on the Correlation of Behaviour with Incentives in Short-Term Memory

Killeen, Peter (1994) Mathematical Principles of Reinforcement: Based on the Correlation of Behaviour with Incentives in Short-Term Memory. [Journal (Paginated)]

Full text available as:

[img] HTML
112Kb

Abstract

Effective conditioning requires a correlation between the experimenter's definition of a response and an organism's, but an animal's perception of its behavior differs from ours. Various definitions of the response are explored experimentally using the slopes of learning curves to infer which comes closest to the organism's definition. The resulting exponentially weighted moving average provides a model of memory which grounds a quantitative theory of reinforcement in which incentives excite behavior and focus the excitement on the responses present in memory at the same time. The correlation between the organism's memory and the behavior measured by the experimenter is given by coupling coefficients derived for various schedules of reinforcement. For simple schedules these coefficients can be concatenated to predict the effects of complex schedules and can be inserted into a generic model of arousal and temporal constraint to predict response rates under any scheduling arrangement. According to the theory, the decay of memory is response-indexed rather than time-indexed. Incentives displace memory for the responses that occur before them and may truncate the representation of the response that brings them about. This contiguity-weighted correlation model bridges opposing views of the reinforcement process and can be extended in a straightforward way to the classical conditioning of stimuli. Placing the short-term memory of behavior in so central a role provides a behavioral account of a key cognitive process.

Item Type:Journal (Paginated)
Keywords:reinforcement, memory, coupling, contingency, contiguity, tuning curves, activation, schedules, trajectories, response rate
Subjects:Biology > Animal Cognition
Biology > Behavioral Biology
JOURNALS > Behavioral & Brain Sciences
Psychology > Behavioral Analysis
Biology > Animal Behavior
ID Code:591
Deposited By: Killeen, Peter
Deposited On:05 Feb 1998
Last Modified:11 Mar 2011 08:54

References in Article

Select the SEEK icon to attempt to find the referenced article. If it does not appear to be in cogprints you will be forwarded to the paracite service. Poorly formated references will probably not work.

Allen, J. D. & Kenshalo, D. R. J. (1976) Schedule-induced drinking as a function of interreinforcement interval in the rhesus monkey. Journal of the Experimental Analysis of Behavior 26:257-267.

Anger, D. (1956) The dependence of interresponse times upon the relative reinforcement of different interresponse times. Journal of the Experimental Analysis of Behavior 52:145-161.

Arbuckle, J. L. & Lattal, K. A. (1988) Changes in functional response units with briefly delayed reinforcement. Journal of the Experimental Analysis of Behavior 49:249-263.

Atkinson, R. C. & Wickens, T. D. (1971) Human memory and the concept of reinforcement. In: The nature of reinforcement, ed. R. Glaser. Academic Press.

Barofsky, I. & Hurwitz, D. (1968) Within ratio responding during fixed ratio performance. Psychonomic Science 11:263-264.

Baum, W. M. (1973) The correlation-based law of effect. Journal of the Experimental Analysis of Behavior 20:137-153.

Baum, W. M. (1992) In search of the feedback function for variable interval schedules. Journal of the Experimental Analysis of Behavior 57:365-375.

Bharucha-Reid, A. T. (1960) Elements of the theory of Markov processes and their applications. McGraw-Hill.

Bindra, D. (1972) A unified account of classical conditioning and operant training. In: Classical conditioning II: Current research and theory, ed. A. H. Black, & W. F. Prokasy. Appleton-Century-Crofts.

Bolles, R. C. (1983) The explanation of behavior. The Psychological Record 33:31-48.

Branch, M. N. (1977) On the role of memory in the analysis of behavior. Journal of the Experimental Analysis of Behavior 28:171-179.

Breland, K. & Breland, M. (1961) The misbehavior of organisms. American Psychologist 16:681-684.

Brown, J. A. (1958) Some tests of the decay theory of immediate memory. Journal of Verbal Learning and Verbal Behavior 2:34-39.

Capaldi, E. J. (1992) Levels of organized behavior in rats. In: Cognitive aspects of stimulus control, ed. W. K. Honig, & J. G. Fetterman. Lawrence Erlbaum Associates.

Catania, A. C. (1971) Reinforcement schedules: The role of responses preceding the one that produces the reinforcer. Journal of the Experimental Analysis of Behavior 15:271-287.

Catania, A. C. & Keller, K. J. (1981) Contingency, contiguity, correlation, and the concept of causation. In: Advances in Analysis of Behaviour: Predictability, Correlation, and Contiguity, ed. P. Harzem & M. D. Zeiler. John Wiley.

Catania, A. C. & Reynolds, G. S. (1968) A quantitative analysis of the responding maintained by interval schedules of reinforcement. Journal of the Experimental Analysis of Behavior 11:327-383.

Catania, A. C., Sagvolden, T. & Keller, K. J. (1988) Reinforcement schedules: retroactive and proactive effects of reinforcers inserted into fixed interval performances. Journal of the Experimental Analysis of Behavior 49:49-73.

Clark, C. W. (1976) Mathematical bioeconomics: The optimal management of renewable resources. Wiley.

Collier, G., Johnson, D., & Morgan, C. (1992) The magnitude-of reinforcement function in closed and open economies. Journal of the Experimental Analysis of Behavior 57:81-89.

Cuthill, I. C., Kacelnik, A., Krebs, J. R., Haccou, P., & Iwasa, Y. (1990) Starlings exploiting patches: The effect of recent experience on foraging decisions. Animal Behaviour 40:625-640.

Davis, D. G. S., Staddon, J. E. R., Machado, A. & Palmer, R. G. (1993) The process of recurrent choice. Unpublished manuscript

Davis, E. R. & Platt, J. R. (1983) Contiguity and contingency in the acquisition and maintenance of an operant. Learning & Motivation 14:487 512.

Davison, M. & Jenkins, P. E. (1985) Stimulus discriminability, contingency discriminability, and schedule performance. Animal Learning & Behavior 13:77-84.

Dawson, G. R., & Dickinson, A. (1990) Performance on ratio and interval schedules with matched reinforcement rates. The Quarterly Journal of Experimental Psychology 42B: 225-239.

Deutsch, D., & Deutsch, J. A. (1975) Short-term memory. New York: Academic Press.

Donahoe, J. W., Crowley, M. A., Millard, W. J., & Stickney, K. A. (1982) A unified principle of reinforcement. In: Quantitative Analysis of Behavior II: Matching and maximizing accounts, ed. M. L. Commons, R. J. Herrnstein, & H. Rachlin. Ballinger.

Donahoe, J. W. & Palmer, D. C. (1993) Learning and its implications for complex behavior. Boston: Allyn & Bacon.

Dow, S. M. & Lea, S. E. G. (1987) Foraging in a changing environment: Simulation in the operant laboratory. In: Quantitative Analysis of Behavior VI: Foraging, ed. M. L. Commons, A. Kacelnik & S. J. Shettleworth. Erlbaum.

Ettinger, R. H., Reid, A. K. & Staddon, J. E. R. (1987) Sensitivity to molar feedback functions: A test of molar optimality theory. Journal of Experimental Psychology: Animal Behavior Processes 13:366-375.

Galbicka, G. (1988) Differentiating the Behavior of Organisms. Journal of the Experimental Analysis of Behavior 50:343-354.

Galbicka, G. & Platt, J. R. (1986) Parametric manipulation of interresponse- time contingency independent of reinforcement rate. Journal of the Experimental Analysis of Behavior 12:371-380.

Gibbon, J. (1977) Scalar expectancy and Webers law in animal timing. Psychological Review, 84:279-325.

Gibbon, J. & Balsam, P. (1981) Spreading association in time. In: Autoshaping and conditioning theory, ed. J. Gibbon, C. M. Locurto, H. S. Terrace. Academic Press.

Greenwood, M. R. C., Quartermain, D., Johnson, P. R., Cruce, J. A. F. & Hirsch, J. (1974) Food motivated behavior in genetically obese and hypothalamic hyperphagic rats and mice. Physiology & Behavior 13:687-692.

Grossberg, S. (1975) A neural model of attention, reinforcement, and discrimination learning. International Review of Neurobiology, 18:263 327.

Hamilton, A. L., Stellar, J. R., & Hart, E. B. (1985) Reward, performance, and the response strength method in self-stimulating rats: Validation and neuroleptics. Physiology & Behavior, 35: 897-904.

Hayes, S. C., & Hayes, L. J. (1992) Verbal relations and the evolution of behavior analysis. American Psychologist 11:1138-1395.

Herrnstein, R. J. (1970) On the law of effect. Journal of the Experimental Analysis of Behavior 13:243-266.

Herrnstein, R. J. (1974) Formal properties of the matching law. Journal of the Experimental Analysis of Behavior 76:49-69.

Herrnstein, R. J. (1979) Derivatives of matching. Psychological Review, 86:486-495.

Heyman, G. M. & Monaghan, M. M. (1987) Effects of changes in response requirement and deprivation on the parameters of the matching law equation: New data and review. Journal of Experimental Psychology: Animal Behavior Processes 13:384-394.

Kacelnik, A., Krebs, J. R., & Ens, B. (1987) Foraging in a changing environment: an experiment with starlings (Sturnus vulgaris). In: Quantitative Analyses of Behavior. Vol. 6: Foraging, ed. M. L. Commons, A. Kacelnik, & S. J. Shettleworth. Lawrence Erlbaum.

Kelsey, J. E. & Allison, J. (1976) Fixed-ratio lever pressing by VMH rats: Work vs accessibility of sucrose reward. Physiology & Behavior 17:749 754.

Killeen, P. R. (1981) Averaging theory. In: Recent developments in the quantification of steady-state operant behavior, ed. C. M. Bradshaw, E. Szabadi & C. F. Lowe. Elsevier.

Killeen, P. R. (1982a) Incentive Theory. In: Nebraska symposium on motivation, 1981: Response structure and organization, ed. D.J. Bernstein. Lincoln: University of Nebraska Press.

Killeen, P. R. (1982b) Incentive Theory II: Models for choice. Journal of the Experimental Analysis of Behavior 38:217-232.

Killeen, P. R. (1984) Incentive theory III: Adaptive Clocks. In: Timing and time perception, ed. J. Gibbon & L. Allen. New York Academy of Sciences.

Killeen, P. R. (1985) Incentive theory IV: Magnitude of reward. Journal of the Experimental Analysis of Behavior 43:407-417.

Killeen, P. R. (1991) Behaviors time. In: The psychology of learning and motivation, ed. G. H. Bower. Academic Press.

Killeen, P. R. (1992) Mechanics of the animate. Journal of the Experimental Analysis of Behavior 57:429-463.

Killeen, P. R., Hanson, S. J. & Osborne, S. R. (1978) Arousal: Its genesis and manifestation as response rate. Psychological Review 85:571-581.

Killeen, P. R. & Smith, J. P. (1984) Perception of contingency in conditioning: Scalar timing, response bias, and the erasure of memory by reinforcement. Journal of Experimental Psychology: Animal Behavior Processes 10:333- 345.

Kintsch, W. (1965) Frequency distribution of interresponse times during VI and VR reinforcement. Journal of the Experimental Analysis of Behavior 8:347-352.

Lattal, K. A. & Gleeson, S. (1990) Response acquisition with delayed reinforcement. Journal of Experimental Psychology: Animal Behavior Processes 16:27-39.

Levine, D. (1991) Introduction to neural modelling. Lawrence Erlbaum Associates.

Lieberman, D. A., Davidson, F. H. & Thomas, G. V. (1985) Marking in pigeons: The role of memory in delayed reinforcement. Journal of Experimental Psychology: Animal Behavior Processes 11:611-624.

Mazur, J. E. (1983) Steady-state performance on fixed-, mixed-, and random-ratio schedules. Journal of the Experimental Analysis of Behavior 39:293-307.

Mazur, J. E. (1984) Tests of an equivalence rule for fixed and variable delays. Journal of Experimental Psychology: Animal Behavior Processes 10:426-436.

McDowell, J. J. (1980) An analytical comparison of Herrnsteins equations and a multivariate rate equation. Journal of the Experimental Analysis of Behavior 33:397-408.

McDowell, J. J. & Kessel, R. (1979) A multivariate rate equation for variable-interval performance. Journal of the Experimental Analysis of Behavior 31:267-283.

McDowell, J. J. & Wood, H. (1984) Confirmation of linear system theory prediction: Changes in Herrnsteins k as a function of changes in reinforcer magnitude. Journal of the Experimental Analysis of Behavior 41:183-192.

McDowell, J. J., Bass, R., & Kessel, R. (1992) Applying linear systems analysis to dynamic behavior. Journal of the Experimental Analysis of Behavior 57:377-391.

McNamara, J. M. & Houston, A. I. (1987) Memory and the efficient use of information. Journal of Theoretical Biology 125:385-395.

McSweeney, F. K. (1974) Variability of responding on a concurrent schedule as a function of body weight. Journal of the Experimental Analysis of Behavior 21:357-359.

McSweeney, F. K. (1978) Prediction of concurrent keypeck treadle-press responding from simple schedule performance. Animal Learning & Behavior 6:444-450.

McSweeney, F. K. (1992) Rate of reinforcement and session duration as determinants of within-session patterns of responding. Animal Learning & Behavior 20:160-169.

Mesterton-Gibbons, M. (1989) A concrete approach to mathematical modelling. Addison-Wesley,.

Moore, J. W. (1991) Implementing connectionist algorithms for classical conditioning in the brain. In: Neural network models of conditioning and action: A volume in the quantitative analysis of behavior series. eds. M. L. Commons, S. Grossberg, & J. E. R. Staddon. Lawrence Erlbaum.

Neuringer, A. (1992) Choosing to vary and repeat. Psychological Science, 3:246-250.

Norman, D. A. (1970) Models of human memory. Academic Press.

Palmer, D. C., & Donahoe, J. W. (1992) Essentialism and selectionism in cognitive science and behavior analysis. American Psychologist 11: 1344 1358.

Palya, W. L. (1992) Dynamics in the fine structure of schedule-controlled behavior. Journal of the Experimental Analysis of Behavior 57:267-287.

Papini, M. R., & Bitterman, M. E. (1990) The role of contingency in classical conditioning. Psychological Review, 97:396-403.

Pear, J. J. (1975) Implications of the matching law for ratio responding. Journal of the Experimental Analysis of Behavior 23:139-140.

Peterson, L. R. & Peterson, M. J. (1959) Short-term retention of individual items. Journal of Experimental Psychology 58:193-198.

Platt, J. R. (1971) Discrete trials and their relation to free-operant behavior. In: Essays in neobehaviorism: A memorial volume to Kenneth W. Spence, ed. H. H. Kendler, & J. T. Spence. Appleton-Century-Crofts.

Platt, J. R. (1973) Percentile reinforcement: Paradigms for experimental analysis of response shaping In: The psychology of learning and motivation: Advances in research and theory, ed. G. H. Bower. Academic Press.

Platt, J. R. (1979) Interresponse-time shaping by variable-interval-like interresponse-time reinforcement. Journal of the Experimental Analysis of Behavior 31:3-14.

Platt, J. R. & Day, R. B. (1979) A hierarchical response-unit analysis of resistance to extinction following fixed-number and fixed-consecutive number reinforcement. Journal of Experimental Psychology: Animal Behavior Processes 5:307-320.

Powell, R. W. (1968) The effect of small sequential changes in fixed-ratio size upon the post-reinforcement pause. Journal of the Experimental Analysis of Behavior 11:589-593.

Preston, R. A., & Fantino, E. Conditioned reinforcement value and choice. Journal of the Experimental Analysis of Behavior 55:155-175.

Rachlin, H., Raineri, A. & Cross, D. (1991) Subjective probability and delay. Journal of the Experimental Analysis of Behavior 55:233-244.

Reed, P., Schactman, T. R., & Hall, G. (1991) Effect of signaled reinforcement on the formation of behavioral units. Journal of Experimental Psychology: Animal Behavior Processes 17:475-485.

Revusky, S. & Garcia, J. (1970) Learned associations over long delays. In: The psychology of learning and motivation: Advances in research and theory, ed. G. H. Bower. Academic Press.

Reynolds, G. S., & McLeod, A. (1970) On the theory of interresponse-time reinforcement. In: The psychology of learning and motivation: Advances in research and theory, ed. G. H. Bower. Academic Press.

Robbins, T. W., & Everitt, B. J. (1992) Functions of dopamine in the dorsal and ventral striatum. Seminars in the Neurosciences, 4:119-127.

Robbins, T. W., & Sahakian, B. J. (1983) Behavioral effects of psychomotor stimulant drugs: Clinical and neuropsychological implications. In: Stimulants: Neurochemical, Behavioral, and Clinical Perspectives, ed. I. Creese. Raven Press.

Sakamoto, Y., Ishiguor, M. & Kitagawa, G. (1986) Akaike information criterion statistics. Reidel.

Shimp, C. P. (1973) Synthetic variable-interval schedules of reinforcement. Journal of the Experimental Analysis of Behavior 19:311 330.

Shimp, C. P. (1976a) Organization in memory and behavior. Journal of the Experimental Analysis of Behavior 26:113-130.

Shimp, C. P. (1976b) Short-term memory in the pigeon: Relative recency. Journal of the Experimental Analysis of Behavior 25:55-61.

Shimp, C. P. (1976c) Short-term memory in the pigeon: The previously reinforced response. Journal of the Experimental Analysis of Behavior 26:487-493.

Sizemore, O. J. & Lattal, K. A. (1978) Unsignalled delay of reinforcement in variable-interval schedules. Journal of the Experimental Analysis of Behavior 30:169-175.

Skinner, B. F. (1935) The generic nature of the concepts of stimulus and response. The Journal of General Psychology 12:40-65.

Skinner, B. F. (1938) The behavior of organisms. Appleton-Century-Crofts.

Snyderman, M. (1983) Body weight and response strength. Behaviour Analysis Letters 3:255-265.

Staddon, J. E. R. (1977) On Herrnsteins equation and related forms. Journal of the Experimental Analysis of Behavior 28:163-170.

Staddon, J. E. R., Wynne, C. D. L. & Higa, J. J. (1991) The role of timing in reinforcement schedule performance. Learning and Motivation 22:200-225.

Staddon, J. E. R. & Zhang, Y. (1989) Response selection in operant learning. Behavioural Processes 20:189-197.

Stokes, P. D., & Balsam, P. D. (1991) Effects of reinforcing preselected approximations on the topography of the rats bar press. Journal of the Experimental Analysis of Behavior 55:213-231.

Thomas, G. V. (1983) Contiguity and contingency in instrumental conditioning. Learning and Motivation 14:513-526.

Timberlake, W. (1983) Rats responses to a moving object related to food or water: A behavior-systems analysis. Animal Learning & Behavior 11:309- 320.

Timberlake, W. & Lucas, G. A. (1990) Behavior systems and learning: From misbehavior to general principles. In: Contemporary learning theories: Instrumental conditioning theory and the impact of constraints on learning, ed. S. B. Klein & R. R. Mowrer. Erlbaum.

Timberlake, W. & Peden, B. F. (1987) On the distinction between open and closed economies. Journal of the Experimental Analysis of Behavior 48:35 60.

Vaughan, W. (1985) Choice: A local analysis. Journal of the Experimental Analysis of Behavior 43:383-405.

Wearden, J. H. & Clark, R. B. (1989) Constraints on the process of interresponse-time reinforcement as the explanation of variable-interval performance. Behavioural Processes 20:151-175.

Wetherington, C. L. (1979) Schedule-induced drinking: Rate of food delivery and Herrnsteins equation. Journal of the Experimental Analysis of Behavior 32:323-333.

Wilkenfield, J. Nickel, M., Blakely, E., & Ploing, A. (1992) Acquisition of lever-press responding in rats with delayed reinforcement: A comparison of three procedures. Journal of the Experimental Analysis of Behavior 58:431- 443.

Williams, B. A. (1972) Probability learning as a function of momentary reinforcement probability. Journal of the Experimental Analysis of Behavior 17:363-368.

Williams, B. A. (1975) The blocking of reinforcement control. Journal of the Experimental Analysis of Behavior 24:215-226.

Williams, B. A. (1976) The effects of unsignalled delayed reinforcement. Journal of the Experimental Analysis of Behavior 26:441-449.

Williams, B. A. (1978) Information effects on the response-reinforcer association. Animal Learning & Behavior 6:371-379.

Williams, B. A. (1988) Reinforcement, choice, and response strength. In: Stevens Handbook of experimental psychology: Vol. 2. Learning and cognition (2nd ed), ed. R. C. Atkinson, R. J. Herrnstein, G. Lindzey, & R. D. Luce. Wiley.

Williams, D. C., & Johnston, J. M. (1992) Continuous versus discrete dimensions of reinforcement schedules: An integrative analysis. Journal of the Experimental Analysis of Behavior 58:205-228.

Zeiler, M. D. (1979) Output dynamics. In: Advances in analysis of behaviour: Vol 1. Reinforcement and the organization of behaviour, ed. M. D. Zeiler & P. Harzem. Wiley.

Zeiler, M. D. & Buchman, I. B. (1979) Response requirements as constraints on output. Journal of the Experimental Analysis of Behavior, 32:29-50.

Zeiler, M. D. & Thompson, T. (1986) Analysis and integration of behavioral units. Hillsdale, NJ: Erlbaum.

Zeiler, M. D. (1991) Ecological influences on timing. Journal of Experimental Psychology: Animal Behavior Processes 17:13-25.

Zuriff, G. E. (1970). A comparison of variable-ratio and variable-interval schedules of reinforcement. Journal of the Experimental Analysis of Behavior 12:369-374.

Metadata

Repository Staff Only: item control page

Cogprints is powered by EPrints 3 which is developed by the School of Electronics and Computer Science at the University of Southampton. More information and software credits.