Introduction

Attractor-based Neural Networks is a tool for designing real-time learning systems. For last ten years it's been one of the main areas of research at the Cybernetics Center of the Ukrainian Academy of Sciences. This research was presented at NOW'99.

The talk in Powerpoint, in Postcript, and on a handout.

The summary of the talk

Candidate of Phys.&Math. Science (Ph.D.) Dissertation

"Designing High Performance Attractor-based Neural Networks"

In the dissertation we study the problem of designing high capacity neural networks with enhanced associative capability. Attractor-based fully connected neural networks of binary neurons are considered and the pseudo-inverse learning rule is shown to be the most efficient for the memory capacity of these networks. We show that the attraction radius of the network is a function of the synaptic weight matrix of the network. We investigate the nature of the dynamic attractors of the network and derive the factors that affect their occurrence. We propose an approach based on the flood-fill neuroprocessing technique which efficiently detects the dynamic attractors. We introduce a modification to the pseudo-inverse rule based on partial reduction of the self-connection weights. This modification, termed the desaturation, is shown to practically double the attraction radius of the network. This is shown both theoretically and by simulations. In particular, we show that the desaturation increases the capacity of the autoassociative memory of the network up to 80% of the number of neurons, which is two to four times better than that of other known networks of the considered type.

The dissertation (in Russian): postscript

• Contents and Chapter 1. "Introduction"
• Chapter 2. "Fully-connected Neural Networks": overview, theorem about cycles, flood-fill neuroprocessing technique
• Chapter 3. "Pseudoinverse Learning Rule": properties, obtaining the formula for attraction radius
• Chapter 4. "Desaturated Pseudoinverse Rule": introducing the desaturating coefficient, theory of desaturation (attraction radius, cycles, energy) and simulations 
• Bibliography 
• Appendices: code and data obtained by Monte-Carlo simulations 
• Sources:
  • Code of the program which simulates the Desaturated Pseudo-Inverse Neural Network: pi_ff.c
  • Bibliography: bibdisser.tex

"The Optimal Value of Self-connection or How to Attain the Best Performance with Limited Size Memory" (Dmitry O. Gorodnichy), Proc. of IJCNN'99, Washington, July 12-17, 1999

"Best presentation" award at IJCNN'99. The paper in postscript, a succinct handout in postscript.

"Non-iterative learning rules for neural networks" (A.M. Reznik)

"Best presentation" award at IJCNN'99.

The talk given in lieu of the paper: "Designing High-Capacity Neural Networks for Storing, Retrieving and Forgetting Patterns Real-Time" (D.O. Gorodnichy, A.M. Reznik)

Abtract: In designing neural networks for pattern recognition the most challenging problems are the following.
1) How to learn a network so that a) it can retrieve as many patterns as possible, and b) it can retrieve them from as much noise as possible;
2) How to make learning fast so that patterns can be stored on-line;
3) How to make retrieval fast;
4) How to get rid of useless data, i.e. how to continuously update the memory when new data are coming.
The solutions to these problems were found at the Institute of Mathematical Machines and Systems of Ukrainian National Academy of Sciences, where a neurocomputer capable of storing and retrieving data in real-time was designed. The neurocomputer uses a non-iterative learning technique based on the Desaturated Pseudo-Inverse rule. This technique allows one to store in real-time up to 80%N patterns (as attractors with non-zero attraction basins), where N is the size of the neural network. When the number of patterns exceeds the capacity of the network, the Dynamic Desaturation rule is applied. This rule allows the neurocomputer to store patterns partially and also to remove from memory obsolete data. In retrieval, the Update Flow neuroprocessing technique is used. This technique is known to be very efficient for neural networks which evolve in time. It also automatically detects spurious dynamic attractors.
In the talk, we will describe in detail each technique contributing to the success of the project. The emphasis will be given to the description of non-iterative learning techniques which provides a valid alternative to the conventional time-consuming iterative learning methods.

Slides for the talk in Powerpoint and in Postcript


***

"Static and Dynamic Attractors of Autoassociative Neural Networks" (D.O. Gorodnichy, A.M. Reznik) Lecture Notes in Computer Science, Vol 1311 (Proc. of 9th Intern. Conf. on Image Analysis and Processing (ICIAP'97), Florence, Italy, Sept. 1997, Vol. II), pp. 238-245, Springer

Abstract: In this paper we study the problem of the occurrence of cycles in autoassociative neural networks. We call these cycles {\em dynamic attractors}, show when and why they occur and how they can be identified. Of particular interest is the pseudo-inverse network with reduced self-connection. We prove that it has dynamic attractors, which occur with a probability proportional to the number of prototypes and the degree of weight reduction. We show how to predict and avoid them.
Keywords: pattern recognition, neural network, pseudo-inverse rule, stable state.

Poster ("Neural Networks for Pattern Recognition and Computer Vision"): attractors_slides.ps 
Postscript file (100Kb): attractors.ps

***

"Increasing Attraction of Pseudo-Inverse Autoassociative Networks" (D.O. Gorodnichy, A.M. Reznik), Neural Processing Letters, volume 5, issue 2, pp. 123-127, 1997, Kluwer Academic Publishers

Abstract: We show {\em how} partial reduction of self-connections of the network designed with the pseudo-inverse learning rule increases the direct attraction radius of the network. Theoretical formula is obtained. Data obtained by simulation are presented.

Postscript file (75 Kb): npl.ps 

***

"Desaturating Coefficient for Projection Learning Rule" (Dmitry O. Gorodnichy), Lecture Notes in Computer Science, Vol. 1112 (Proc. of Intern. Conf. on Artificial Neural Networks (ICANN'96), Bochum, Germany, July 1996), pp.469-476, Springer.

Abstract: A Hopfield-like neural network designed with projection learning rule is considered. The relationship between the weight values and the number of prototypes is obtained. A coefficient of self-connection reduction, termed the desaturating coefficient, is introduced and the technique which allows the network to exhibit complete error correction for learning ratios up to 75\% is suggested. The paper presents experimental data and provides theoretical background explaining the results.

Postscript file (95Kb): ICANN'96.ps 

***

"A Way to Improve Error Correction Capability of Hopfield Associative Memory in the Case Of Saturation" (Dmitry O. Gorodnichy), HELNET International Workshop on Neural Neworks Proceedings (HELNET 94-95), Vol. I/II, pp.198-212, VU University Press, Amsterdam

Abstract: A fully connected neural network with binary self-connected neurons is considered. The properties of such networks trained by the projection learning rule are investigated. The questions of prototype attractivity are studied, especially for the case of network saturation (when $M > N/2$, where $M$ is the number of prototypes, and $N$ - the number of neurons). The formula for the attraction radius is derived and the relationship between the weight coefficients and the attraction radius is obtained. It is shown that it is possible to increase the attraction radius and improve retrieval capability of the network, by introducing a coefficient of self-connection reduction, which is termed the desaturating coefficient. For example, it is demonstrated that even for the case of $M=0.75N$ it is possible by appropriately choosing the desaturating coefficient to achieve error correction. The paper presents theoretical results and provides data obtained by simulation of the model.

Postscript file (140Kb): HELNET'95.ps 

***

"NEUTRAM - A Transputer Based Neural Network Simulator" (Dmitry O. Gorodnichy, Alexander M. Reznik), Proc. of Second Intern. Conf. on Software for Multiprocessors and Supercomputers Theory, Practice, Experience (SMS TPE'94), Sept. 1994, pp.136-142, Moscow, Russia

Abstract: Researchers in Neural Networks (NN) often require a simulation of the model under development. Conventional computer systems often do not yield the required performance for Parallel Distributed Processing even with 100 units. The problems are that available memory is not sufficient and processing iteration rate is far from desired; it is not uncommon for such simulations to take many hours to process simple network.
NEUTRAM - is a tool designed for simulation of NNs of various sizes and configurations on a net of transputers. Currently working version can process full-connected NN with up to 600 neurons (i.e. up to 360000 connections) on the net of four transputers. With its aid a lot of research dealing with investigation of NNs (such as exploration the dependence of the behavior of a NN from its size and configuration, from learning rule) has been being carried out now in the Cybernetics Center of Ukrainian Academy of Sciences in Kiev.
NEUTRAM uniformly allocates neurons and all data, upon which neurons depend, onto the available processors, and then process them using the strategy of flood data processing (also called update flow technique); that means that we keep only changes at the current iteration and process only those data that depend on these changes. It is this strategy that allows NEUTRAM to achieve high performance iteration rate. In the paper we describe it in details. We also describe the work of NEUTRAM itself: how it organizes the interaction of distributed units-neurons and data exchange among them. Our observations concerning for the above are presented.