Cogprints

Binding and Normalization of Binary Sparse Distributed Representations by Context-Dependent Thinning

Rachkovskij, Dmitri A. and Kussul, Ernst M. (2001) Binding and Normalization of Binary Sparse Distributed Representations by Context-Dependent Thinning. [Journal (Paginated)]

Full text available as:

[img]
Preview
 PDF
175Kb
[img] Postscript
1785Kb

Abstract

Distributed representations were often criticized as inappropriate for encoding of data with a complex structure. However Plate's Holographic Reduced Representations and Kanerva's Binary Spatter Codes are recent schemes that allow on-the-fly encoding of nested compositional structures by real-valued or dense binary vectors of fixed dimensionality. In this paper we consider procedures of the Context-Dependent Thinning which were developed for representation of complex hierarchical items in the architecture of Associative-Projective Neural Networks. These procedures provide binding of items represented by sparse binary codevectors (with low probability of 1s). Such an encoding is biologically plausible and allows a high storage capacity of distributed associative memory where the codevectors may be stored. In contrast to known binding procedures, Context-Dependent Thinning preserves the same low density (or sparseness) of the bound codevector for varied number of component codevectors. Besides, a bound codevector is not only similar to another one with similar component codevectors (as in other schemes), but it is also similar to the component codevectors themselves. This allows the similarity of structures to be estimated just by the overlap of their codevectors, without retrieval of the component codevectors. This also allows an easy retrieval of the component codevectors. Examples of algorithmic and neural-network implementations of the thinning procedures are considered. We also present representation examples for various types of nested structured data (propositions using role-filler and predicate-arguments representation schemes, trees, directed acyclic graphs) using sparse codevectors of fixed dimension. Such representations may provide a fruitful alternative to the symbolic representations of traditional AI, as well as to the localist and microfeature-based connectionist representations.

Item Type:Journal (Paginated)
Keywords:distributed representation, sparse coding, binary coding, binding, variable binding, representation of structure, structured representation, recursive representation, nested representation, compositional distributed representations, connectionist symbol processing
Subjects:Computer Science > Artificial Intelligence
Computer Science > Neural Nets
ID Code:1240
Deposited By: Rachkovskij, Dmitri
Deposited On:16 Jan 2001
Last Modified:11 Mar 2011 08:54

References in Article

Select the SEEK icon to attempt to find the referenced article. If it does not appear to be in cogprints you will be forwarded to the paracite service. Poorly formated references will probably not work.

Amari, S. (1989). Characteristics of sparsely encoded associative memory. Neural Networks, 2, 445-457.

Amosov, N. M. (1967). Modelling of thinking and the mind. New York: Spartan Books.

Amosov, N. M., Baidyk, T. N., Goltsev, A. D., Kasatkin, A. M., Kasatkina, L. M., Kussul, E. M., & Rachkovskij, D. A. (1991). Neurocomputers and intelligent robots. Kiev: Naukova dumka. (In Russian).

Artykutsa, S. Ya., Baidyk, T. N., Kussul, E. M., & Rachkovskij, D. A. (1991). Texture recognition using neurocomputer. (Preprint 91-8). Kiev, Ukraine: V. M. Glushkov Institute of Cybernetics. (In Russian).

Baidyk, T. N., Kussul, E. M., & Rachkovskij, D. A. (1990). Numerical-analytical method for neural network investigation. In Proceedings of The International Symposium on Neural Networks and Neural Computing - NEURONET'90 (pp. 217-222). Prague, Czechoslovakia.

Blair, A. D. (1997). Scaling-up RAAMs. (Technical Report CS-97-192). Brandeis University, Department of Computer Science.

Fedoseyeva, T. V. (1992). The problem of training neural network training to recognize word roots. In Neuron-like networks and neurocomputers (pp. 48-54). Kiev, Ukraine: V. M. Glushkov Institute of Cybernetics. (In Russian).

Feldman, J. A., & Ballard, D. H. (1982). Connectionist models and their properties. Cognitive Science, 6, 205-254.

Feldman, J. A. (1989). Neural Representation of Conceptual Knowledge. In L. Nadel, L. A. Cooper, P. Culicover, & R. M. Harnish (Eds.), Neural connections, mental computation (pp. 68-103). Cambridge, Massachusetts, London, England: A Bradford Book, The MIT Press.

Foldiak, P., & Young, M. P. (1995). Sparse Coding in the Primate Cortex. In M. A. Arbib (Ed.), Handbook of brain theory and neural networks (pp. 895-898). Cambridge, MA: MIT Press.

Frasconi, P., Gori, M., & Sperduti, A. (1997). A general framework for adaptive processing of data structures. Technical Report DSI-RT-15/97. Firenze, Italy: Universita degli Studi di Firenze, Dipartimento di Sistemi e Informatica.

Frolov, A. A., & Muraviev, I. P. (1987). Neural models of associative memory. Moscow: Nauka. (In Russian).

Frolov, A. A., & Muraviev, I. P. (1988). Informational characteristics of neuronal and synaptic plasticity. Biophysics, 33, 708-715.

Frolov, A. A. (1989). Information properties of bilayer neuron nets with binary plastic synapses. Biophysics, 34, 868-876.

Gayler, R. W. (1998). Multiplicative binding, representation operators, and analogy. In K. Holyoak, D. Gentner, & B. Kokinov (Eds.), Advances in analogy research: Integration of theory and data from the cognitive, computational, and neural sciences. (p. 405). Sofia, Bulgaria: New Bulgarian University. (Poster abstract. Full poster available at: http://cogprints.soton.ac.uk/abs/comp/199807020).

Halford, G. S., Wilson W. H., & Phillips S. (in press). Processing capacity defined by relational complexity: implications for comparative, developmental, and cognitive psychology. Behavioral and Brain Sciences.

Hebb, D. O. (1949). The organization of behavior. New York: Wiley.

Hinton, G. E. (1981). Implementing semantic networks in parallel hardware. In G. E. Hinton & J. A. Anderson (Eds.), Parallel models of associative memory (pp. 161-187). Hillside, NJ: Lawrence Erlbaum Associates.

Hinton, G. E. (1990). Mapping part-whole hierarchies into connectionist networks. Artificial Intelligence, 46, 47-76.

Hinton, G. E., McClelland, J. L., & Rumelhart, D. E. (1986). Distributed representations. In D. E. Rumelhart, J. L. McClelland, & the PDP research group (Eds.), Parallel distributed processing: Exploration in the microstructure of cognition 1: Foundations (pp. 77-109). Cambridge, MA: MIT Press.

Hopfield, J. J. (1982). Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences, USA, 79, 2554-2558.

Hopfield, J. J., Feinstein, D. I., & Palmer, R. G. (1983). "Unlearning" has a stabilizing effect in collective memories. Nature, 304, 158-159.

Hummel, J. E., & Holyoak K. J. (1997). Distributed representations of structure: A theory of analogical access and mapping. Psychological Review, 104, 427-466.

Kanerva, P. (1988). Sparse distributed memory. Cambridge, MA: MIT Press.

Kanerva, P. (1994). The Spatter Code for encoding concepts at many levels. In M. Marinaro and P.G. Morasso (eds.), ICANN '94, Proceedings of International Conference on Artificial Neural Networks (Sorrento, Italy), 1, pp. 226-229. London: Springer-Verlag.

Kanerva, P. (1996). Binary Spatter-Coding of Ordered K-tuples. In C. von der Malsburg, W. von Seelen, J. C. Vorbruggen, & B. Sendhoff (Eds.). Proceedings of the International Conference on Artificial Neural Networks - ICANN'96, Bochum, Germany. Lecture Notes in Computer Science, 1112, 869-873. Berlin: Springer.

Kanerva, P. (1998). Encoding structure in Boolean space. In L. Niklasson, M. Boden, and T. Ziemke (eds.), ICANN 98: Perspectives in Neural Computing (Proceedings of the 8th International Conference on Artificial Neural Networks, Skoevde, Sweden), 1, pp. 387-392. London: Springer.

Kasatkin, A. M., & Kasatkina, L. M. (1991). A neural network expert system. In Neuron-like networks and neurocomputers (pp. 18-24). Kiev, Ukraine: V. M. Glushkov Institute of Cybernetics. (In Russian).

Kussul, E. M. (1980). Tools and techniques for development of neuron-like networks for robot control. Unpublished Dr. Sci. dissertation. Kiev, Ukrainian SSR: V. M. Glushkov Institute of Cybernetics. (In Russian).

Kussul, E. M. (1988). Elements of stochastic neuron-like network theory. In Internal Report "Kareta-UN" (pp. 10-95). Kiev, Ukraine: V. M. Glushkov Institute of Cybernetics. (In Russian).

Kussul, E. M. (1992) Associative neuron-like structures. Kiev: Naukova Dumka. (In Russian).

Kussul, E. M. (1993). On some results and prospects of development of associative-projective neurocomputers. In Neuron-like networks and neurocomputers (pp. 4-11). Kiev, Ukraine: V. M. Glushkov Institute of Cybernetics. (In Russian).

Kussul, E. M., & Baidyk, T. N. (1990). Design of a neural-like network architecture for recognition of object shapes in images. Soviet Journal of Automation and Information Sciences, 23(5), 53-58.

Kussul, E. M., & Baidyk, T. N. (1993). On information encoding in associative-projective neural networks. (Preprint 93-3). Kiev, Ukraine: V. M. Glushkov Institute of Cybernetics. (In Russian).

Kussul, E. M., Baidyk, T. N., Lukovich, V. V., & Rachkovskij, D. A. (1993). Adaptive neural network classifier with multifloat input coding. In Proceedings of NeuroNimes'93, Nimes, France, Oct. 25-29, 1993. EC2-publishing.

Kussul, E. M., & Kasatkina, L. M. (1999). Neural network system for continuous handwritten words recognition. (Submitted).

Kussul, E. M., & Rachkovskij, D. A. (1991). Multilevel assembly neural architecture and processing of sequences. In A .V. Holden & V. I. Kryukov (Eds.), Neurocomputers and Attention: Vol. II. Connectionism and neurocomputers (pp. 577-590). Manchester and New York: Manchester University Press.

Kussul, E. M., Rachkovskij, D. A., & Baidyk, T. N. (1991a). Associative-Projective Neural Networks: architecture, implementation, applications. In Proceedings of the Fourth International Conference "Neural Networks & their Applications", Nimes, France, Nov. 4-8, 1991 (pp. 463-476).

Kussul, E. M., Rachkovskij, D. A., & Baidyk, T. N. (1991b). On image texture recognition by associative-projective neurocomputer. In C. H. Dagli, S. Kumara, & Y. C. Shin (Eds.), Proceedings of the ANNIE'91 conference "Intelligent engineering systems through artificial neural networks" (pp. 453-458). ASME Press.

Lansner, A., & Ekeberg, O. (1985). Reliability and speed of recall in an associative network. IEEE Trans. Pattern Analysis and Machine Intelligence, 7, 490-498.

Lavrenyuk, A. N. (1995). Application of neural networks for recognition of handwriting in drawings. In Neurocomputing: issues of theory and practice (pp. 24-31). Kiev, Ukraine: V. M. Glushkov Institute of Cybernetics. (In Russian).

Legendy, C. R. (1970). The brain and its information trapping device. In Rose J. (Ed.), Progress in cybernetics, vol. I. New York: Gordon and Breach.

Marr, D. (1969). A theory of cerebellar cortex. Journal of Physiology, 202, 437-470.

Milner, P. M. (1974). A model for visual shape recognition. Psychological Review, 81, 521-535.

Milner, P.M. (1996). Neural representations: some old problems revisited. Journal of Cognitive Neuroscience, 8, 69-77.

Palm, G. (1980). On associative memory. Biological Cybernetics, 36, 19-31.

Palm, G. (1993). The PAN system and the WINA project. In P. Spies (Ed.), Euro-Arch?93 (pp. 142-156). Springer-Verlag

Palm, G. & Bonhoeffer, T. (1984). Parallel processing for associative and neuronal networks. Biological Cybernetics, 51, 201-204.

Plate, T. A. (1991). Holographic Reduced Representations: Convolution algebra for compositional distributed representations. In J. Mylopoulos & R. Reiter (Eds.), Proceedings of the 12th International Joint Conference on Artificial Intelligence (pp. 30-35). San Mateo, CA: Morgan Kaufmann.

Plate, T. A. (1994). Distributed representations and nested compositional structure. PhD Thesis, Department of Computer Science, University of Toronto.

Plate, T. A. (1995). Holographic Reduced Representations. IEEE Transactions on Neural Networks, 6, 623-641.

Plate, T. (1997). A common framework for distributed representation schemes for compositional structure. In F. Maire, R. Hayward, & J. Diederich (Eds.), Connectionist systems for knowledge representation and deduction (pp. 15-34). Queensland University of Technology.

Plate, T.A. (to appear). Structured Operations with Vector Representations. Expert Systems: The International Journal of Knowledge Engineering and Neural Networks, Special Issue on Connectionist Symbol Processing.

Plate, T. (submitted). Randomly connected sigma-pi neurons can form associative memories.

Pollack, J. B. (1990). Recursive distributed representations. Artificial Intelligence, 46, 77-105.

Rachkovskij, D. A. (1990a). On numerical-analytical investigation of neural network characteristics. In Neuron-like networks and neurocomputers (pp. 13-23). Kiev, Ukraine: V. M. Glushkov Institute of Cybernetics. (In Russian).

Rachkovskij, D. A. (1990b). Development and investigation of multilevel assembly neural networks. Unpublished Ph.D. dissertation. Kiev, Ukrainian SSR: V. M. Glushkov Institute of Cybernetics. (In Russian).

Rachkovskij, D. A. (1996). Application of stochastic assembly neural networks in the problem of interesting text selection. In Neural network systems for information processing (pp. 52-64). Kiev, Ukraine: V. M. Glushkov Institute of Cybernetics. (In Russian).

Rachkovskij, D. A. (to appear). Representation and Processing of Structures with Binary Sparse Distributed Codes. IEEE Transactions on Knowledge and Data Engineering, Special Issue on Connectionist Models for Learning In Structured Domains.

Rachkovskij, D. A. & Fedoseyeva, T. V. (1990). On audio signals recognition by multilevel neural network. In Proceedings of The International Symposium on Neural Networks and Neural Computing - NEURONET'90 (pp. 281-283). Prague, Czechoslovakia.

Rachkovskij, D. A. & Fedoseyeva T. V. (1991). Hardware and software neurocomputer system for recognition of acoustical signals. In Neuron-like networks and neurocomputers (pp. 62-68). Kiev, Ukraine: V. M. Glushkov Institute of Cybernetics. (In Russian).

Shastri, L. & Ajjanagadde, V. (1993). From simple associations to systematic reasoning: connectionist representation of rules, variables, and dynamic bindings using temporal synchrony. Behavioral and Brain Sciences, 16, 417-494.

Sjodin, G. (1998). The Sparchunk Code: a method to build higher-level structures in a sparsely encoded SDM. In Proceedings of IJCNN'98 (pp. 1410-1415), IEEE, Piscataway, NJ: IEEE.

Sjodin, G., Kanerva, P., Levin, B., & Kristoferson, J. (1998). Holistic higher-level structure-forming algorithms. In Proceedings of 1998 Real World Computing Symposium - RWC'98 (pp. 299-304).

Smolensky, P. (1990). Tensor product variable binding and the representation of symbolic structures in connectionist systems. Artificial Intelligence, 46, 159-216.

Sperduti, A. (1994). Labeling RAAM. Connection Science, 6, 429-459.

Sperduti, A. & Starita, A. (1997). supervised neural networks for the classification of structures. IEEE Transactions on Neural Networks, 8, 714-735.

Tsodyks, M. V. (1989). Associative memory in neural networks with the Hebbian learning rule. Modern Physics Letters B, 3, 555-560.

Touretzky, D. S. (1990). BoltzCONS: Dynamic symbol structures in a connectionist network. Artificial Intelligence, 46, 5-46.

Touretzky, D. S. (1995). Connectionist and symbolic representations. In M. A. Arbib (Ed.), Handbook of brain theory and neural networks (pp. 243-247). Cambridge, MA: MIT Press.

Touretzky, D. S., & Hinton, G. E. (1988). A distributed connectionist production system. Cognitive Science, 12, 423-466.

Vedenov, A. A. (1987). "Spurious memory" in model neural networks. (Preprint IAE-4395/1). Moscow: I. V. Kurchatov Institute of Atomic Energy.

Vedenov, A. A. (1988). Modeling of thinking elements. Moscow: Science. (In Russian).

von der Malsburg, C. (1981). The correlation theory of brain function. (Internal Report 81-2). Gottingen, Germany: Max-Planck-Institute for Biophysical Chemistry, Department of Neurobiology.

von der Malsburg, C. (1985). Nervous structures with dynamical links. Ber. Bunsenges. Phys. Chem., 89, 703-710.

von der Malsburg, C. (1986) Am I thinking assemblies? In G. Palm & A. Aertsen (Eds.), Proceedings of the 1984 Trieste Meeting on Brain Theory (pp. 161-176). Heidelberg: Springer-Verlag.

Willshaw, D. (1981). Holography, associative memory, and inductive generalization. In G. E. Hinton & J. A. Anderson (Eds.), Parallel models of associative memory (pp. 83-104). Hillside, NJ: Lawrence Erlbaum Associates.

Willshaw, D. J., Buneman, O. P., & Longuet-Higgins, H. C. (1969). Non-holographic associative memory. Nature, 222, 960-962.

Metadata

Repository Staff Only: item control page

Cogprints is powered by EPrints 3 which is developed by the School of Electronics and Computer Science at the University of Southampton. More information and software credits.