Cognitive Memory, Part 2 Bernard Widrow Department of Electrical Engineering Stanford University Three questions are of interest: " How does human memory work? " How can one build a memory like that? " How can practical problems be solved with a chuman-like d memory?
How human memory works will remain a question for a very long time to come. Some of the main features of human memory can be understood, however, and now it is even possible to build a memory emulating these features. The purpose in doing this is twofold: " To develop a new chuman-like d memory for computers that will be adjunct to existing computer memory, thus expanding computer architecture in order to facilitate solutions to problems in artificial intelligence, pattern recognition, speech recognition, control systems, etc.
" To advance cognitive science with new insight into the working of human memory. Memory patterns are probably not stored in the brain 9s neurons, synapses, or dendrites. Why so?
They take too long to train. Also, they must be continually re- trained because (1) synaptic memory is analog and is likely to vary or drift over time, and (2) some neurons, dendrites, and synapses die out continually, and without re-training, the entire ... more. less.
neural network 9s mapping would deteriorate and become defunct. Re-training the entire network would require that the original training patterns be stored somewhere.<br><br> A permanent pattern storage system would be needed to maintain a neural network storage system, and this is a ccatch 22 d. Why would nature store patterns to train neural networks to store patterns? Human long-term memory is capable of recording patterns of interest in great detail over an entire lifetime.<br><br> When one trains a neural network, the training patterns are generally discarded when training is complete. Human memory doesn 9t discard the training patterns. Human long-term memory must be digital in order to store patterns of sight, sound, feel, smell, etc.<br><br> in detail, over a lifetime. Analog memory cannot do this. This seems to rule out long-term storage in synapses.<br><br> It is known that the functioning of the brain 9s neural networks is crucial to long- term memory. If the memory 9s patterns are not stored in the neural networks, then what is the role of the neural networks? In Part 1, I showed how neural networks trained to be autoassociative could be used in the pattern retrieval process, not in the pattern storage process.<br><br> I believe that the long term storage mechanism is digital, and that the analog neural networks associated with memory are used only in the pattern retrieval process. People suffering from age-related dementia or from Alzheimer 9s disease experience a gradual deterioration of memory. This seems to be associated with a dying off of neurons, dendrites, and synapses, slowly eating away at the neural network.<br><br> On a cbad day d, an Alzheimer 9s patient would not be able to remember very much. On a succeeding cgood day d, this person would remember many things that could not be previously recalled. This indicates that the basic digital memory remains intact, but the ability to retrieve patterns is the affected process.<br><br> In 1960, I worked with a group Ph.D. students at Stanford University, on building a machine called MADALINE I. It was a physical neural network having 6 neurons with approximately 100 weights.<br><br> The weights were electrochemical memory devices, variable resistors whose conductance was able to be varied by electroplating. The circuits were analog and completely trainable. After many months of fabrication the machine was, of course, completed just the night before it was to be demonstrated to a large group of distinguished visitors.<br><br> It worked beautifully. I was able to demonstrate learning and classification of a complex set of patterns. After the meeting was over, MADALINE was returned to the laboratory and thoroughly tested.<br><br> We found that 25% of the circuits were totally defective. There were short circuits, open circuits, cold solder joints, weights that did not adapt, and weights that were connected backwards so that they tended to diverge rather than converge. Nevertheless, the demonstration was a huge success.<br><br> With training, the machine was able to adapt around its own internal flaws. When we fixed the bugs, performance improved, but only slightly. The lesson to be learned from this is that with dementia, at the beginning, before too much damage is done, good functioning can be obtained by exercising the memory.<br><br> In spite of the loss of neurons, constant re- training of the neural networks may keep the memory retrieval process working for a long time. The cowner 9s manual d for human memory should read like this: use it, or lose it! Figure 1.<br><br> Block diagram of a highly simplified cognitive memory system In Part 1, a block diagram of a simplified artificial cognitive memory was shown and is repeated here in Figure 1. Salient features of the cognitive memory are the following: 1. It stores sensory input patterns that may be visual, auditory, tactile, radar, sonar, etc.<br><br> 2. It stores patterns wherever space is available, not in specified memory locations. 3.<br><br> It stores simultaneously received input patterns in the same memory folder. For example, simultaneous visual and auditory patterns are stored together. 4.<br><br> It retrieves patterns from memory in response to current input patterns that serve as cprompts d. These prompts could be visual, auditory, tactile, olfactory, etc. 5.<br><br> It delivers all the patterns stored in the prompted memory folder as its output. 6. It uses an autoassociative neural network to verify that there exists a pattern in one of the folders that matches the prompt input, i.e., there is a HIT.<br><br> 7. If there is a HIT, then searching the folders finds the one containing the matching pattern. 8.<br><br> It trains the neural network off-line (e.g., during non-REM sleep) by scanning through all of the patterns stored in all of the folders and uses them as training patterns. 9. The neural network plays a critical role in memory retrieval.<br><br> We have successfully experimented with several applications for the cognitive memory. One application is aircraft localization when flying over an urban area with optical sighting of the ground below. Another is location and identification of aircraft seen in satellite photos.<br><br> Yet another application is recognition of people 9s faces in photos with complicated backgrounds. The largest face-recognition exercise that we have performed used downloaded photos distributed by NIST for the Face Recognition Grand Challenge (FRGC) version 1. A set of photographs of 75 persons were selected for training.<br><br> These and variations of them (rotated, translated, zoomed, etc.) were recorded in memory. Another set of 300 photographs of people whose images were not trained-in were selected for testing. In addition, 75 photos of the persons who were trained-in were selected for testing, but these were not the trained-in photos.<br><br> Upon testing, the 75 photos of the trained-in people were correctly identified, and the 300 photos of the people not trained-in were identified as unknown people. The results were perfect, with no errors. No biometrics were used, just images of people 9s faces.<br><br> A more detailed conceptual diagram of how input patterns are stored is shown in Figure 2. All input patterns flow into short-term memory. This is like a shift register for patterns.<br><br> They are held there long enough (about 1 or 2 seconds) to make the decision to transfer them to permanent storage or to discard them. The principal basis for this decision is familiarity. If the new input patterns (with possible modifications, such as rotation, translation, scale changes, brightness, contrast, etc.<br><br> in the case of visual patterns) correlate with any of the patterns stored in any of the memory folders, the new input patterns are considered cinteresting d and the switch is closed causing all sensory inputs selected for permanent storage to be recorded over a period of time. On the other hand, if the new input patterns do not relate to anything seen before, they will be discarded. The visual images will cgo in one eye and out the other d.<br><br> The auditory images will cgo in one ear and out the other d, etc. SEGMENT N NN SENSING STRONG TRAINING MUX SEGMENT N +1 PROBLEM SOLVER HIT? HIT LINE SENSORY INPUT NN SENSING STRONG TRAINING MUX HIT?<br><br> VC AC SC SHORT TERM MEMORY cINTERESTING INPUT d SWITCH SENSORY INPUT LINE MEMORY INPUT LINE VC = VISUAL CORTEX AC = AUDITORY CORTEX SC = SENSORY CORTEX Figure 2. Sensory input pattern storage Aside from familiarity, the problem solver is also able to turn on the recording process by itself, perhaps in response to external stimuli, such as bright colors or loud noises. The output of the cognitive memory goes to the problem solver that uses stored information for problem solving.<br><br> The problem solver is somewhat like a ccognitive CPU d. We have not focused on the design of the problem solver 4that will be taken up in the future. For the moment, our attention is on the memory.<br><br> Once stored, patterns can be retrieved from long-term memory by prompting. A conceptual diagram for pattern retrieval is shown in Figure 3. Prompt patterns could come from sensory inputs or from the problem solver.<br><br> Retrieved patterns fed to the problem solver could, in turn, be used as additional prompt patterns. Thus, one step can lead to another in searching the memory. The sensory input patterns can be altered in the respective visual, auditory, etc., cortices.<br><br> Visual patterns can be, to a limited extent, rotated, translated, scaled, brightened, etc. Each visual pattern thus becomes may prompt patterns. These prompt patterns are inputted to the neural networks.<br><br> If there is a HIT in one of the autoassociative neural networks, the contents of all its folders are checked against the prompt pattern to find the selected folder, whose contents are then delivered to the problem solver. SEGMENT N NN SENSING STRONG TRAINING MUX PROBLEM SOLVER HIT? SENSORY INPUT PATTERNS PROMPT LINE VC AC SC THOUGHT PATTERNS OUTPUT BUFFER HIT?<br><br> SEGMENT N+1 NN SENSING STRONG TRAINING MUX HIT? BUFFER HIT? RETRIEVED PATTERNS LINE RETRIEVED PATTERNS PROMPT SIGNAL PROMPT PATTERNS SHORT TERM MEMORY VC AC SC PROMPT LINE VC= VISUAL CORTEX AC = AUDITORY CORTEX SC = SENSORY CORTEX Figure 3.<br><br> Pattern retrieval system Each neural network in a particular memory segment is trained off-line with all the patterns in every folder in that segment. There would be separate neural networks for each type of sensory patterns, i.e., visual and auditory patterns would not be mixed. Each neural network shown in Figure 3 represents all the neural networks for all pattern types.<br><br> The effects of sleep deprivation in humans are well known, but the purpose of sleep itself is not understood. I would speculate that a principal purpose of non-REM sleep is to systematically train the neural networks using patterns stored in long-term memory. The purpose of REM sleep (dream sleep) seems to be quite different.<br><br> I would speculate that the purpose of REM sleep is problem solving. Uninhibited thought can be highly creative. Every 90 minutes or so, during the night, the brain goes into cREM mode d.<br><br> Each episode of REM lasts for about 20 330 minutes. The body is paralyzed during REM sleep, probably to prevent the person from acting out the dream. During REM, contents are pulled from long-term memory.<br><br> I speculate that prompting is done by thought patterns from the problem solver. The memory contents provide further prompts to retrieve further related contents, thus starting a cchain reaction d. The retrieved memory contents are available to the problem solver.<br><br> The memory contents are juxtaposed and intermingled in strange ways, creating fantasies that are dreams. The dreams themselves are stored in new memory locations. Brain activity during REM is similar to that of wide-awake consciousness, according to EEG and fMRI data.<br><br> REM activity involves storing and retrieving patterns from memory, and problem solving by deductive reasoning. The difference is that during REM, the sensory inputs from the eyes, ears, etc. are shut off.<br><br> I have discussed this with sleep researchers and psychiatrists, and they think that my ideas about sleep are quite reasonable. I have also discussed with them ideas about schizophrenia as an abnormal condition under which the subject can be awake and conscious, but simultaneously in the REM sleep mode, with fantasized images superimposed on real-time visual, auditory, etc. input.<br><br> This is hallucination 4the fantasized images are drawn from memory spontaneously, without prompting. I have also discussed psychoanalysis with them, where the psychiatrist tries to prompt recall of dreams stored in long-term memory. This is difficult, because the prompt is unknown.<br><br> By encouraging the patient to talk and by asking questions, sometimes the dream or dreams of interest are prompted and then become known. Normal people have several episodes of dreaming every night, but are unaware of this unless they wake up in the middle of a dream. The portion of dream at the waking moment prompts recall of the rest of the dream.<br><br> This works like what seems to happen when listening to a familiar piece of music. The present sounds prompt a recall of what is to come. A piano soloist playing a concerto needs only to start, and then the sounds of the orchestra and the piano provide a sustained prompt so that the soloist can play the entire piece without referring to the musical score.<br><br> All animals are born with intrinsic knowledge that is essential for survival. Inborn knowledge in the form of patterns is pre-loaded in the developing brain 9s long- term memory and remains intact throughout lifetime. A baby horse is up and walking within a half hour of birth.<br><br> How to walk is inborn knowledge. The horse didn 9t learn this in half an hour. A baby bird, at the right time, jumps from the nest and flies for the first time.<br><br> How to fly is inborn knowledge. The bird couldn 9t possibly learn to fly when jumping from the nest, before hitting the ground. It is conjectured that the memory storage means and the memory retrieval means for inborn knowledge is same as for new knowledge gained during a lifetime.<br><br> The two kinds of knowledge are treated in the same way, and the animal can operate seamlessly between them. At the moment of conception, DNA is taken from the mother and father to form a new cell. That is the start of a new living animal.<br><br> The DNA of the new cell contains the information to construct the body, the internal organs, including the brain. The DNA also contains the inborn information that will be pre-loaded in the developing brain. Inborn information is stored in DNA.<br><br> The mechanism for storage and retrieval is probably the same for inborn knowledge as for all knowledge gained during a lifetime. Here are some cwild guesses d : All information stored in long-term memory is stored digitally in DNA. The amount of DNA needed to store a lifetime of acquired knowledge is very small.<br><br> (All inborn knowledge is stored originally in the DNA of a single cell, probably in its cjunk DNA d.) Having acquired knowledge over a lifetime, some DNA in the brain will be different from the DNA in the rest of the body. Since this DNA stays in the brain, the lifetime knowledge is not passed on genetically to the next generation, only the inborn knowledge can propagate. Long-term memory is not stored in the neurons, synapses, and the dendritic tree.<br><br> The neurons and the dendritic tree play a key role in association and retrieval of information that is stored long-term in DNA.