All processing at the circuit level going up to the perceptual level must synapse in the

SUMMARY

The ability to transform or manipulate visual information is a perceptual-cognitive skill of central importance in normal perceptual processing and in perceptually driven tasks requiring the use of imagery or the comparison of spatially transformed visual input to a representation of that input in memory. In most cases, image transformation occurs at a level of processing following and relying upon object identification; however, some forms of manipulation of visual information (e.g., integration and transformation of different views of an object) may be involved in the process of identification. For certain pilot performance problems (including those consisting of detection and identification), image manipulation is unlikely to be an important component of operation. For other tasks facing the pilot (including aspects of navigation, localization of a target in a visual array, and comparison of current visual input with previously available views), transformation of visual information may play a central role in performance.

A substantial body of experimental work exists on perceptual and cognitive tasks requiring the transformation of visual information and is briefly reviewed in this report. Research has, for the most part, been directed toward delineating the information-processing consequences of transforming spatial information in terms of time and accuracy constraints on performance. There is considerable evidence concerning the effects of various display and task parameters on the amount of time in which, and the accuracy with which, visual information can be transformed. Furthermore, the process of image transformation can often be shown to conform to highly regular and mathematically straightforward relationships. For cases in which errors of transformation occur frequently, the magnitude and direction of error often follow a highly predictable pattern. Yet, despite the large body of systematic experimental results, general computational models are still scarce. Those models that have been specified suffer from being stimulus or task specific, and they have generally been based on some single index of task performance.

In sum, although there has been considerable progress in understanding—at a quantitative level—the nature, time course, and limitations on the ability to manipulate spatial information, as well as the various factors that affect different aspects of performance on tasks requiring spatial transformations, as yet no set of large-scale models of image manipulation exists. A further limitation on the applicability of current data and models to pilot performance tasks is the reduced conditions under which data have been obtained in terms of both display richness and concurrent processing demands.

Effect of Changing the Stimulus Notation

The first round of experiments supports the hypothesis that much of the skilled processing in chess occurs at the perceptual front end. We have conducted some experiments to test this perceptual hypothesis against one possible artifact, and we were further interested in seeing how robust the perceptual processing is when the stimuli are subjected to a degrading transformation.

One possible alternative to our perceptual hypothesis is that the structures we are isolating actually Arise from the organization of the output at recall rather than from the perceptual process, and the pauses really represent an artifact because players need to pause in order to pick up a new set of pieces before continuing their recall. This hypothesis has trouble explaining why pieces recalled together in time are also functionally related, but it is possible that this organization is somehow imposed at recall rather than at the time of perception.

We reasoned that if our perceptual hypothesis were true, then we ought to be able to disrupt these perceptual processes by perturbing the stimuli in some way. However, if the response organization hypothesis is true, then the way to disrupt performance is by changing the response mode.

We presented the Class A player with 32 new positions taken from Reinfeld (1945), but half the positions were presented as schematic diagrams in which a piece is represented by the first letter of its name, and black pieces are circled. Figure 8 shows an example of such a letter diagram position. The other 16 positions were shown normally, as pieces on an actual board. Also, half the positions were recalled normally by placing pieces on a board, and the other half were recalled by writing letter diagrams like Fig. 8. Thus, we have a 2 × 2 design with boards vs. letter diagrams as stimuli, and boards vs. letter diagrams as responses.

The results, shown in Table 6, are straightforward. Looking at the data on the first 16 trials, it didn't make any difference, in response, whether pieces were placed on a board or a schematic diagram was drawn. On the stimulus side, however, it made a big difference whether real boards or letter diagrams were presented. The Class A player was getting almost twice as many pieces correct when real boards were presented as when diagram stimuli were shown. This result was highly significant statistically (p < 10−6), and neither the response mode nor the stimulus-response interaction was significant.

Table 6. Percent Correct Recall for Boards vs Diagrams as Stimuli and as Responses

The advantage of boards over letter diagrams was due to more chunks being recalled for boards than for diagrams (7.5 vs. 4.0, respectively); the number of pieces per chunk was relatively constant for the different stimulus conditions (2.3 vs. 3.0, respectively). It appears, therefore, that the schematic diagrams slow down the perceptual process, so that fewer perceptual structures are seen in the 5-second exposure.

However, this effect washes out very quickly with practice, so that after about an hour or so the Class A player was seeing these schematic diagrams about as well as real board positions. Neither the main effects nor the interaction was significant for the second block of 16 trials.

This experiment shows, first, that regardless of whether the player writes a letter diagram of the position or whether he picks up pieces at the side of the board and places them on the board, his performance is the same. This result eliminates the possibility that the pauses are artifacts due to picking up the pieces. Second, the fact that stimuli in the form of letter diagrams are initially disruptive suggests that performance in this task really depends upon perceptual rather than recall processes. The Class A player rapidly overcomes the difficulties of viewing the letter diagrams. Apparently some easy perceptual learning takes place so that the non-essential surface characteristics of the diagrams are ignored and the underlying invariant relations are perceived.

In a second experiment, we were interested in seeing how chess players of various strengths are affected by diagrams. In this experiment, we gave two Class A players and a beginner the same 5-second task, but this time we compared real board positions with printed (pictorial) diagrams from chess books. Figure 9 shows an example of these pictorial diagrams, selected from Reinfeld (1945), and in both cases real pieces were placed on a board at recall. Performance on pictorial diagrams is interesting because chess players spend a lot of time looking at diagrams like these when they read chess books and magazines.

Table 7 shows the basic results. Both Class A players did equally well for real boards and pictorial diagrams, but the beginner recalled real boards better than diagrams (p < .001). These limited data provide no evidence of a practice effect. These results presumably reflect the fact that the Class A players have had considerable experience with pictorial diagrams (but not with letter diagrams), whereas the beginner has had little or none.

Table 7. Percent Correct Recall of Boards and Diagrams for two Class A Players and a Beginner

9.3.2 Age-related and individual differences require user-centered engineering design

There is a further challenge for developing multimodal human–machine interfaces that could serve broad populations of users, who are of different ages or different skill levels. Accumulating empirical findings from lifespan psychology and cognitive neuroscience show that brain development and brain aging greatly impact perceptual and cognitive processes. Such age-related differences in perception and cognition challenge the one-size-fits-all assumption in technical designs and will likely break usability for users at the two ends of the lifespan.

Protracted maturation of higher-order perceptual processes The receptors in the sensory organs already undergo considerable development before birth and mature rapidly until they reach the full functioning level during early postnatal life [630,677]. In contrast, neurochemical processes and brain networks for top-down attentional modulation of subjective sensory experiences and perception develop rather gradually during childhood and adolescence [678,679]. Concerning auditory perception, although children's hearing acuity is better than adults, in situations where the direction from which auditory signals reach the ear conflicts with the direction one needs to attend to, children perform much worse than adults in detecting and identifying auditory information [680]. Regarding vision, the protracted maturation of sensory cue integration in the visual cortex until late childhood (around 11 years of age) contributes to children's lower ability relative to young adults in recognizing visual objects with multiple features [681,682]. Similarly, it is only until late childhood that the sensitivity for size discrimination by touch becomes mature [683]. As for multisensory integration, empirical evidence also indicates that the ability to combine information about visual disparity and relative motion for depth perception [681], as well as the ability to integrate visual and haptic information for size and orientation discrimination [683] do not mature until late childhood. Taken together, higher-level perceptual processing that requires top-down attention as well as processes for optimally weighting sensory cues from different modalities are still not mature during early and middle childhood. Therefore multisensory integration in children relies more on the characteristics of physical stimuli in the environment, whereas adults are more flexible in weighting the reliability of sensory signals and use prior expectations gained from past experiences to inform multisensory perception [677].

Aging and gain control of information processing Regarding effects of aging during the adult lifespan, it is well established that the acuity for auditory and visual [684,685] as well as tactile processing [406,686] all decline substantially in old age. Furthermore, brain aging entails substantial changes at the anatomical, neurochemical, and neurofunctional levels (see [625,679,687,688] for reviews). Focusing on dopamine, which also plays an important role in modulating subjective tactile sensory experiences, as discussed above (see Fig. 9.5, [653,659]), the functions of its receptors start to show age-related decline already in early adulthood, with an estimate of about 10% loss per decade in many brain regions (see [625] for review). Dopamine's role in regulating the signal-to-noise ratio of neural information transmission between neurons (i.e., neuronal gain control) has been computationally modeled as the gain parameter of the sigmoidal activation function in feed-forward multilayer neural networks [689]. The impacts of aging-related decline in dopamine-regulated neuronal gain control of uncertainty (noise) during neural information processing have also been established in computational simulations [625,690,691], which show clear negative consequences of increased processing noise and reduced processing efficiency in simulated old neural networks (Fig. 9.9).

Noisy and less efficient processing in children and older adults Accumulated empirical findings support the simulated computational effects of noisy and less efficient neural information processing with negative effects on perceptual and cognitive performance in individuals at both ends of the lifespan, whose neurotransmitter functions are either not yet mature or have already declined. When making perceptual decisions, children and old adults show lower levels of processing speed and process robustness (i.e., higher degrees of random processing fluctuations) than young adults (Fig. 9.10, [7]). Relatedly, at the brain level, the temporal precision of neuronal signals (as indicated by synchronized brain electrophysiological activities) is weaker in children and in old adults, relative to young adults (Fig. 9.10, [693]). There are also other prominent aging-related declines in frontal brain processes of attention (e.g., [694,695], executive control (e.g., [696–698]), and valuation (e.g., [699,700]) that could influence perceptual processes in the elderly. Taken together, these age-related constraints on perceptual and cognitive functions either in children or older adults need to be scrutinized and compensated when designing multimodal interfaces for human–machine interactions for these user groups in real, virtual or mixed reality environments.

Individual differences in perceptual processing Besides age-related effects, individual differences in neurobiological factors (e.g., genetic predispositions affecting neurotransmitter functions), past learning experiences, or skill levels, can all contribute to considerable between-person differences in sensory and perceptual functions, even among people within the same age groups. Take attention-guided auditory processing as an example, empirical evidence [701] shows that, even in a given age range, there are substantial differences between the listeners' ability to selectively attend to speech streams coming from different directions under conditions with no background echo (anechoic) or with a middle or high level of background reverberation (see panel a in Fig. 9.11). In general, adding background reverberation reduced performance accuracy in detecting targeted speech-sound; furthermore, large individual differences in performance were observed in all conditions.

In investigating potential neurobiological factors that might contribute to such individual differences, findings from another study [702] showed that individual differences in genetic predispositions affecting neurotransmitter functions are associated with brain psychophysiological signals that underlie attentional control of auditory perception. The DARPP-32 gene is associated with the functioning of dopamine and other neurotransmitters in the human striatum, with carriers of the AA genotype showing higher neurotransmitter function. Correspondingly, AA carriers also exhibited larger brain psychophysiological responses during attention-guided auditory processing. Specifically, the AA carriers of the DARPP-32 gene showed a larger amplitude of the N1 component during early auditory processing (see panel b in Fig. 9.11) and a larger amplitude of the N450 component during attentional regulation [702]. Neurotransmitters (e.g., dopamine) regulating neuronal gain control (Fig. 9.9) can affect the perceived uncertainty of subjective sensory experiences (see Fig. 9.5, [653,659,661]), which, in turn, could contribute to between-person differences in sensory and perceptual processing.

Individual differences in depth perception is another example. In the extreme, some individuals could not perceive 3D information in stereoscopic scenes [703]. Even within the regular functioning range, different individuals may rely on different cues for depth perception [704]. In an ongoing study of depth perception and learning, we found clear individual differences in the sensitivity to disparity cues that is reflected in the psychophysical signal detection function. Furthermore, we also observed an effect of learning, which sharpens the sensitivity to depth cues. Nevertheless, the effect of learning, which is reflected in the increased slope of the psychophysical function in the last task block, also differs substantially between individuals (see Fig. 9.12).

Taken together, development and aging affect brain functions in several regions relevant for uni- and multisensory perception and attention. Furthermore, the efficacy of neurotransmitter functions, such as dopaminergic modulation highlighted in this chapter, not only undergo clear maturation- and aging-related changes across the lifespan, but also differ between individuals. Neurotransmitter systems can affect the SNR of neural information processing in several brain networks with consequences on the precision, efficiency, and capacity of human perception and cognition. Automatic signal gain control is an important principle for information processing and dynamic process control in neurobiological sensory-perceptual systems as well as in electronic signal processing systems and digital communication networks. Advances in developing sensors and actuators, signal filtering, and compression algorithms as well as communication networks need to closely consider age-related and individual differences in neuronal gain control and their impacts on multisensory perception to develop next-generation technologies for human–machine interactions that can serve broad populations of users.

Change in ACh Metabolism (AChE and BuChE Influence)

ACh has an exclusive role as a neurotransmitter, not only because it is involved in neurotransmission and memory formation but also intervenes in the production and secretion of other neurotransmitters (e.g., glutamate, glycine, and dopamine), in the CNS. One of the theories of this pathophysiology—cholinergic theory—which was published 40 years ago [80] is still discussed and elaborated.

ACh, which is formed in the presynaptic area of neuron by action of ChAT (EC 2.3.1.6), is applied on various receptors in the body, in the brain; however, it plays an essential role of neurotransmitter [81] significantly forming memory and transmission of memory information on two main types of receptors—muscarinic acetylcholine receptor (mAChR; activation of G-proteins), nicotinic acetylcholine receptor (nAChR; form of ligand-gated ion channels in the plasma membranes of certain neurons and on the postsynaptic side of the neuromuscular junction), or more precisely their subtypes. In the synaptic cleft, ACh is degraded by AChE (and under pathological conditions in AD and dementia with Lewy bodies also BuChE) to choline and acetic acid. In AD, presynaptic section of acetylcholinergic neuron is affected—entry of precursors into neurons, ACh synthesis, and release. Attention, stimulus detection, perceptual processing, and information encoding are allowed by stimulation of the cholinergic system. Although the memory consolidation is impaired by cholinergic activation, it is not clear whether information retrieval may be improved [82]. ACh receptors are found throughout the CNS in a huge amount, especially in the cortex, thalamus, hippocampus, and various nuclei in the basal forebrain. Forebrain cholinergic systems are essential for cognitive processes [83]. It has been proved that entorhinal cortex (EC) is one of the first locations of degeneration [84]. EC acts as a hub in a large network for memory and navigation. This is the main interface between the hippocampus and neocortex. Combination of EC–hippocampus plays an important role in the existence of an autobiographical/declarative/episodic memory, but especially in spatial memory, including its formation and consolidation. This cortex is the first of the regions of disability at the onset of AD [83]. There is a link between cholinergic activation and APP metabolism: lesions of cholinergic nuclei show a rapid increase of cortical APP and CFS. Decrease of cholinergic transmission in AD leads to amyloidogenic metabolism and contributes to cognitive dysfunctions [85].

The level of ACh is regulated by ChEs (AChE and BuChE). These serine esterases are present in various tissues in the human body, in which they fulfill the role of a hydrolytic enzyme. AChE (EC 3.1.1.7) cleaves ACh to the basic components and maintains a metabolic balance. Cleavage of ACh in the CNS is normally an important factor for the regeneration of neuron. The second enzyme, which appears to be concentrated in the brain of AD patients, is BuChE (EC 3.1.1.8). It is present in the CNS in other regions than AChE, in particular in endothelial cells, neurons, and glia, and it has been proved that it is synthesized in the brain [86]. BuChE is located in neurons, glia, neuritic plaques, and tangles. When decreasing the activity of AChE, BuChE may replace it [85]. Human brain and liver BuChE and hydrophilic plasmatic G4 BuChE have an identical amino acid sequence.

Both AChE and BuChE, present mainly in the form of tetramer G4, but also dimer G2 and monomer G1, which are secreted to the tissues as hydrophilic form, occur in the tissues. The major amount of AChE in the CNS is amphiphilic, containing both hydrophilic and hydrophobic regions [86,87]. In the human brain, there are two structural types of AChE: type G4 (highly prevalent) and monomer G1 (in minor amount). Proportion of G1 is significantly increased in AD. In this case, BuChE pathologically generated in mobilized neuroglia forming inflammatory edge of Aβ plaque is also involved in the degradation of ACh. Both enzymes have the different ability to hydrolyze substrates. These differences are probably caused by changes in the arrangement of amino acids in the aromatic cavity. Altered expression of AChE in the brain of patients with AD suggests that AChE activity increases at the periphery of amyloid plaque (around the Aβ plaques) and Aβ may actually affect the levels of AChE [87]. It has been found that different forms of AChE in the brain and cerebrospinal fluid of patients with AD are changed in connection with abnormal glycosylation [88]. The role of AChE in neurodegenerative diseases is relatively well known, the role of BuChE is still not completely clarified. This pseudocholinesterase does not have natural substrates in organism [87].

ChEs fulfill other physiological roles than just degradation of ACh, especially noncholinergic trophic functions. In the brain, therefore, they are present even in regions other than the cholinergic terminals. They can modify APP metabolism and lead to increased formation of Aβ. AChE induces formation of Aβ fibrils; hydrophobic peripheral anionic site of the enzyme is responsible for this activity.

It appeared that cholinergic transmission performed other new functions. It can modulate various aspects of immune function, both innate and adaptive. Cholinergic transmission influences immune cell proliferation, cytokine production, differentiation of T-helper cells, and antigen presentation. These effects are mediated by cholinergic mAChR and nAChR and other cholinergic components present in immune cells, for example, α7 nAChR has the ability to induce anti-inflammatory activity [89]. This is probably one of the reasons why acetylcholinesterase inhibitors (AChEIs) act far broader than just to the inhibition of AChE.

2 Glutamatergic information transmission and reception

Glutamate (Glu) is the main excitatory transmitter in the brain, being largely present in cortico-cortical networks and operating both on excitatory (as pyramidal cortical) and inhibitory neurons (as GABAergic interneurons). The Glu-induced excitation (i.e., membrane depolarization) of interneurons increase their inhibitory action (i.e., GABAergic transmission inducing the flow of chloride ions to hyperpolarize their membrane) on the excitatory ones. Glu transmission is a key component in the balance of excitation and inhibition that is a necessary condition for brain function.5

Glu also operates as an information carrier to thalamocortical and cortico-cortical synapses, a role that is crucial for the understanding of perceptual processing in the brain. All perceived information, from the environment is carried by neuronal spike trains, which are transduced by Glu at each synapse, reaching the sensory cortex, where they activate specialized feature detectors. The Central Nervous System (CNS) constructs conscious episodes using feature detectors activated by the information patterns carried by a series of neurons by means of spike trains and Glu transmission at each synapse (Fig. 1).

The role of Glu as information carrier in thalamocortical and cortico-cortical synapses is crucial for the understanding of how conscious perception is possible.6 Glu is largely present in cortico-cortical networks, with a central role in the generation of conscious content in normal states, dreams and altered states. This role has been proved in experiments when the Glu NMDA receptor is transiently blocked by subanesthetic doses of an antagonist (ketamine, PCP or MK-801), thus generating perceptual distortions and hallucinations.7

Spike trains encode information by means of frequency and phase in populations of axons. The CNS constructs conscious episodes from the ensemble of neuron firing patterns received within a temporal period of approximately 2–3 s.8 The concepts of feature-detectors and population-rate coding can be combined in the idea of a sparse population code.9 In this view, the detection of real-world objects would be made by a cooperative group of neurons, forming a Hebbian cell assembly. A cell assembly is a relatively small neuronal population, located in cortical columns, with strengthened connections elicited by previous learning.10

Glu membrane receptors control intracellular signaling pathways targeting the dendritic spine, where a molecular device is able to register the relevant afferent patterns, supporting conscious perceptual learning and selective triggering of memory formation, as well as unconscious priming. The mechanism involved in such a recording of sensory patterns has been studied as the early stage of LTP.11 It involves biological molecular structures and functions, including the system of Glu receptors, and calcium-binding proteins as Calmodulin (CaM) and Calmodulin-Dependent Protein Kinase II (CamKII, a protein from the kinase family, having several receptor and effector active sites).

This mechanism operates in dendritic spines distributed over the whole neocortex. In the sensory cortex, exogenous patterns transmitted through thalamocortical glutamatergic projections are received and processed by post-synaptic mechanisms.12–14 Activation of Glu receptors combined with voltage-dependent calcium channels (VDCCs) converge to the dendritic spine, where they control CaM/CaMKII signaling mechanisms. Glu released from the presynaptic neuron's axon terminal is spread in synaptic space and bind to three different kinds of receptors (AMPA, NMDA and Metabotropic Glu Receptors—MetGR) located at the post-synaptic neuron membrane. The three kinds of receptors activate signal transduction pathways that converge into the dendritic spine.15

Calcium cations (Ca++) are largely employed biological ions with a flexible electronic structure able to encode information.16 CaM and CamKII have several receptor and effector active sites, where Ca++ ions entering through NMDA and VDCCs are trapped. The ions trapped in CaM are transferred to the kinase and trigger regulatory functions. The informational state of CaM/CaMKII is dependent on the interaction with the Ca++ population passing through NMDA and VDCCs. In normal cases, most of Ca++ entry is made through NMDA channel, which is considered to be a coincidence-detector for both bottom-up (sensory afferent) and top-down (previously learned) patterns.17 Because of this condition, the NMDA channel assures the reliability of percepts in regard to stimuli, since it is opened to Ca++ entry only if endogenous and afferent pulses reach the NMDA receptor together. When VDCCs (which are not coincidence detectors) assume the main role in glutamatergic transmission, perceptual distortions and hallucinations occur (see Table 1).

Table 1. Three different modes of functioning of the glutamatergic synapse.

The multimeric structure of CamKII, having binding sites for Ca++ and phosphatases that participate in the phosphorylation of other proteins, constitutes a micro computing device able to read quantized information from incoming Ca++, to process this information, and to activate other proteins according to the results of the information processing (Fig. 2). A model of quantum computing with calcium ions in spines supporting the formation of conscious states was presented in two publications.17,19

The multimeric structure of CaMKII contains four sites that bind to CaM, determining the conformational state of the kinase and the resulting phosphorylation functions.20 Such a micro computing device uses quantum information encoded in the electronic configuration of the ions. Besides binding with CaMKII, Ca++-activated CaM can trigger other signaling pathways in the cell, some of them exerting feedback control on the state of the membrane and others reaching the nucleus and inducing the formation of long-term memory.

The above signaling pathways can be related to cognitive processes that depend on glutamatergic mechanisms responsible for conscious perception. For instance, one of the key proteins present in the converging glutamatergic and dopaminergic pathways is DARPP-32, which is found, among other places, in the striatum, controlling thalamocortical glutamatergic and cholinergic neurons with the participation of dopaminergic modulation.21 Striatal signals processed by such converging molecular pathways convey information to the cortex as an “efferent copy” (i.e., signals sent from motor to perceptual areas when a voluntary action is initiated; for a review of such concepts, including the roles of the cerebellum and hippocampus, see22). DARPP-32 activation, as a converging route for intra-neuronal dopaminergic and Glu-activated signal transduction pathways, has been related to mental function. It is defective in schizophrenia, possibly participating in the generation of symptoms.23,24 An explanation for the role of DARPP-32 is that the feedback from motor to sensory areas would have the role of reinforcing the perceptual pattern, in order to boost its learning and memorizing process.

2.1.1 Gaps between engineering methodologies and human behavior modeling

Currently, engineered autonomous systems dwell on two ends of a spectrum. On one end, there are systems that operate based on what corresponds to rational models: The environment is sensed to produce a type of map or representation; the map, in combination with mathematical specifications of the task elements and a model of the agent’s dynamics, is then used to compute a trajectory. Most of the adaptation available in existing guidance systems is achieved either by feedback control mechanisms or simply through path re-planning. Sensor-based guidance schemes exist at the other end of the spectrum. These include strategies that exploit relatively simple control laws to steer a vehicle. Such systems depend on the availability of a sufficient explicit structure in an environment in order to operate successfully and tolerate a nominal amount of uncertainties or disturbances in conditions.

Humans use fundamentally different ways of operating than the typical guidance and navigation laws employed in robotics and other autonomous systems. Humans and other animals can extract specific pieces of information needed to perform a task at various levels of the control hierarchy (e.g., global task and environment structure, local features relevant to guiding motion, and environment and operating conditions). They can then combine these knowledge elements to form a wholistic understanding that enables managing risks and adapting to changes and uncertainties at various levels of the control hierarchy.

In contrast, engineered systems still don’t operate at knowledge level. The environment sensing is often implemented as brute force sampling and doesn’t include perceptual processing, while trajectories are computed based on some discrete representation of the environment (Goerzen, Kong, & Mettler, 2010). The engineering paradigm poses various challenges. Computational tractability is often an issue when numerical optimization is used for trajectory planning. However, there are more fundamental issues with the application of traditional control engineering methodologies to autonomous guidance. These are related to the basic language used to formulate and specify guidance problems (Mettler, Dadkhah, & Kong, 2010), and more generally, with the absence of a clearly defined unit of description to link the various functional aspect of behavior, and capture the structure of the vehicle or agent interaction with its environment. In fact, these gaps ostensibly also explain the lack of adaptability and versatility of engineered and robotic systems.

Important elements for human movement analysis are motion primitives (see Flash & Hochner, 2005 for review). Examples of versatile primitives include Schaal’s Dynamic Movement Primitives (DMP) (Schaal, 2003; Schaal, Peters, Nakanishi, & Ijspeert, 2005). These primitives are based on autonomous nonlinear differential equations whose temporal evolution creates smooth kinematic control policies. Therefore they have the flexibility to capture both discrete and rhythmic movements. However, motion primitives overall are disconnected from the environment interactions. Many skilled motion control tasks involve interactions with the environment, such as when manipulating objects or navigating between them.

Behavior-based control proposed in robotics (Arkin, 2000; Brooks, 1986) are based on a basic unit of description that is closer to the behavioral characteristics and interactions between the agent and its environment, yet the behaviors typically consider simple sensory-motor responses (see, e.g., Braitenberg, 1984), making them challenging to apply to skilled human spatial behavior.

Human and animal performance provide a dramatic contrast to engineering approaches. When considering human performance, one often tends to focus on the limitations. However, it is important to remember that these “perceived” limitations are related to a type of specialization of these systems or processes through evolution. The working principles of human sensory, control, and perceptual functions, for example, are tailored to their environment interactions and required capabilities. Therefore, before confronting the various sources of complexity with increasing amounts of computation enabled by technological progress, it is necessary to understand and incorporate principles of human factors into our methodological framework.