The New Poetics of Computer Animation: Selective Augmentation and Animated Realism

The history of animation theory has often been one of dueling poetics. On the one hand Darlene Hadricka, veteran animator at the Walt Disney Studios, said, “if you are going to animate something there should be a reason for animating it, it should be magical” (Hadricka). Her thoughts echo that of animation theorists and critics throughout history, with many emphasizing the unique capability for animated poetics to show the magical, the unreal, or the fantastic. Critic Gilbert Seldes wrote,

Something in the form itself is a satisfaction to us … it is the pleasure of magic, in seeing the impossible happen…. The animated cartoon shows us movement in something naturally inert, and it is essentially the satisfaction of magic that we get out of it (qtd. in Crafton, 1982, 12).

Furthermore, one of animation’s most influential and prolific theorists, Paul Wells, wrote,

It is as if the early animators wanted to constantly expose the limitations of representing “reality” on film and insist upon the domain of “fantasy” as: first, the most appropriate mode of expression for the cinematic form and, most specifically, the animated form; second, as the most versatile model by which to create amusement and illusion; and third, as the most expressive vocabulary by which to interrogate the complexities of the human condition. (1998, 16)

Wells’s observation that early animators, specifically, wanted to expose the limitations of representing reality on film outlines a poetic trajectory that much of early animation followed. Any student of the medium will be familiar with the madcap, zany antics of Bobby Bumps, Felix the Cat, Koko the Clown, or even the early adventures of Mickey Mouse. These and other animated shorts, made primarily before the 1930s at the outset of animation’s history, exemplified Hadricka’s conception of magic and Seldes’s injunction that the pleasure in animation was derived from seeing the impossible happen. Animation’s poetics, both visual and narrative, were centered around a locus of magic, fantasy, and depicting the impossible.

In the mid-1930s, the Disney corporation set animated poetics on a path of diegetic realism. First exemplified in their 1934 short film Flowers and Trees, and later solidified in Snow White and the Seven Dwarfs, the 1930s were a decade of technological development at the Disney studios with innovations such as the multi-plane camera, the story department, and the pencil test. Wells notes, “With each technical development, however, Disney moved further away from the plasmatic flexibility of many of the early Silly Symphonies, and coerced the animated form into a neo-realist practice” (1998, 23). A strong argument can be made that realism has become the dominant form of animated poetics today as technology has increased, especially in computer animation. For example, William Brown, discussing computer animation combined with analogue footage, writes, “we cannot tell where the computer-generated imagery begins and the non-animated footage ends” (2020, 68).

The trend towards simulating reality through digital technology has been well documented, seen when Lev Manovich writes “SIGGRAPH’s [CG’s cross-industry professional organization] research proceedings point towards realism as a common goal” (2001, 191). According to Manovich, photoreality is not only the goal of computer animation but of practically every technological implementation throughout the history of the cartoon. An article in “Science News Letter” from 1938 even states that the multiplane camera was “only one of many pieces of equipment that have been devised in recent years to enable the manufacture of realistic cartoon movies” (Wiley, 1938, 321).

How, then, is the theorist to balance these two tendencies in animated poetics, one unreal and metamorphosing and the other a technological march towards realism? I argue that, rather than being at odds with each other, animation has frequently achieved a synthesis of both poetics that I term “selective augmentation.” Selective augmentation borrows some concepts from mirror neuron theory, and also from animated caricature, particularly the stylistic propensity to accentuate certain physical characteristics over others.

Defining Animated Realism

Defining realism, in an animated context, is a problematic issue because animation, whether CG or hand-drawn, is often closely allied with the frame-by-frame construction of the non-indexical illusion of motion. Tom Gunning provides a basic definition for animation when he writes,

[Animations is] moving images that have been artificially made to move, rather than movement automatically captured through continuous-motion picture photography. Nonphotographic images are most common in animation, but still photographs can also be animated, as in Norman McLaren’s Neighbours (1952) or the collage films of Stan Vander. (2014, 86)

Given Gunning’s definition of animation as images that have been made to move artificially, addressing realism becomes difficult. What is realism, then, in the context of animation? To find a definition we can engage with Pat Power’s idea that realism is “audio-visual verisimilitude” to real life. Power’s definition narrows realism to audio-visual verisimilitude, and focuses the discussion on the formal/aesthetic realm that mimics experiential reality rather than narrative realism, though the two are not exclusive. However, Power’s definition of realism doesn’t really address the context of an unreal medium, or how we can specifically discuss “animated realism” as a concept (Power, 2008).

Luckily, scores of theorists, scholars, and practitioners have attempted to describe the complex relationship between animation and aesthetic realism. For example, Johnny Hardstaff writes, “CGI software developers have long been on a singular quest for the grail of absolute photorealism” (2007, 194). Thus, to narrow realism for the purposes of our analysis, I emphasize the difference between realism (audio-visual verisimilitude to pro-cinematic reality), and photorealism that mimics the world as photographed (what Hardstaff calls ‘misappropriating the natural’). Hardstaff’s observations work quite nicely with Stephen Prince’s “perceptual realism” (1999, 277), mimicking photographic reality through correspondences such as color, size, shape, and also lighting, cinematography, motion-blur, and lens flare.

Why make the distinction between photorealism and realism? Wells gave us a clue when he wrote on how animators wanted to “expose the limitations of representing reality” (1998, 16). Wells defined the relationship between animation and photorealism; he sees photorealism as the enemy of true animated expression, as though photoreal images were placing animation’s square peg in a round hole that was already filled by live-action cinema, which is the benchmark for photorealism, and by doing so animators were missing an opportunity to use the medium’s unique creative possibility: to portray subjective, unreal, fantastic states in a way that live-action cinema cannot. Once again, the debate between unreal and realist poetics for animation manifests itself.

Wells states the contention directly when he writes “animation is a ‘completely fake’ medium by virtue of the fact that it does not use the camera to ‘record’ reality but artificially creates and records its own” (1998, 25). Wells was describing classic, hand-drawn cartoons, though his observation works equally well in a computer animation context. Computer animation is artificial, even in such instances where motion capture has been used as a reference point for further puppet articulation, as in video games or blending live-action with animation. How real then can it be?

By Wells’s definition, and that of early animators, any attempt at representing a pro-filmic event in a photorealistic way is impossible; by virtue of its processes, animation is completely artificial and is incapable of recording indexical reality. However, to end the discussion here would be to ignore that realism (rather than photorealism) does operate in animation in conjunction with the animation apparatus’s artificiality and can provide a useful vocabulary, presenting a poetics that provides insight into both the stylistic choices of animators as well as awakening audiences to a renewed sense of the material world they inhabit. To prove this claim we must first seek to understand, apply and extend one of the newest ideas in animation discourse: mirror-neuron system (hereafter referred to as MNS) theory.

Mirror Neuron System Theory and Animated Realism

The history of MNS theory begins with Vittorio Gallese, an Italian research scientist who states that he and his team accidentally discovered mirror neurons in 1991 when working with macaque monkeys; when a researcher reached for a peanut, the same neurons fired in the monkey’s brain, as if he were performing that very action. Gallese explains the theory during an interview he gave in 2013. He says,

We are in a restaurant, somebody reaches for the salt. The classic idea of cognition is that I infer that you want the salt—that is the ‘mind-reading’ route. But there is a much faster route to understanding you want the salt. I sense it myself—my mirror neurons fire as do yours—and understand through my own body” (Badt, 2017).

While the idea did not gain immediate publicity in neuroscientific circles, it was not long before it was applied to understanding film theory. While it has been criticized for a lack of evidence scientifically, a number of its concepts have been applied to animation theory by critics and theorists.

For example, Pat Power states: “underlying perceptions of agency at the level of the brain are called mirror-neurons, known collectively as the mirror neuron system (MNS)” (2008, 30). He draws on a 2006 study by Mar and Macrae wherein participants watched two films by Richard Linklater, Waking Life and A Scanner Darkly while their brains were scanned by an fMRI (a magnetic resonance imagery) device for activity. Power reports that “the more realistic footage triggered mentalizing more strongly, while the animated footage may have been more emotionally rewarding … the more realistic the imagery, the more it triggered agency and intent…[though] the animated footage may have been more emotionally rewarding” (2008, 40). Power’s study further proved that the brain distinguishes movements that might seem to have intentions from other movements, and that movements implying agency will trigger extra responses. In animation we often see quite abstract nonliving representations as if they were intentional agents, such as the squiggles in the experimental Boogie Doodle (Norman MacLaren, 1940) and our enjoyment of animation, comics, and novels depends on an “ability to adopt the intentional stance toward clearly non-intentional (and non-biological) objects” (30). Of course, by now animation theorists are quite used to that idea, as Geldes wrote when he said that the allure of animation was perceiving intention and movement in normally inert objects.

Neuroscientific discoveries form the foundation for Power’s mirror-neuron system theory in animation, based on identifying an intentional stance in the object presented. Power writes,

These neurons fire in response to one’s own goal-related motor-actions but also respond to perceived similar actions in other entities.… these systems ‘map the actions of others to corresponding actions on one’s own body without confusing other with self (32).

Creating intention in any movement is a key aspect to MNS theory’s application in animation because intention is what separates action from motion in the brain, and it is action that causes the mirror neuron systems to engage and thereby allows the audience to mentalize, that is, to identify, simulate, or internalize that action in their own bodies.

However, the question of how purely unreal, animated characters (not rotoscoped as in Linklater’s films) create intention remains. The study above gives us valuable insight into the answer when it showed that the more realistic images produced a more powerful mentalizing response. Power surmises that intentionality is demonstrated in an animated context through “credible motion [and] physical behavior…” (35). While Power does not make this connection himself, we see that intention leading to action and mentalizing/simulation is achieved through employing realist aesthetic principles. John Lasseter demonstrates the idea when he writes, “to show that the thoughts of a character are driving his actions, a simple trick is in the anticipation; always lead with the eyes or the head” (qtd. in Power 2018, 35).

This technique shows that certain principles of behavior observed in pro-filmic reality have been translated to animation with side-effects the creators might not have fully realized; what Lasseter assumed was a way to give characters personality and motivation was also allowing the audience to become more engaged with them, as Power explains when he writes, “MNS-based simulation drives anthropomorphic-based presumptions of agency and intentionality … and enables empathy with character that in turn feeds animated narrative” (32). Power’s quote also explains why spectators can be emotionally engaged with inanimate objects; by portraying realist principles of behavior demonstrated through motion, characters express intentionality, which causes us to mentalize and then bodily simulate their experiences in a manner very similar to live-action cinema.

However useful his ideas are, Power ends his discussion leaving much to be considered. As we already briefly mentioned, he never fully explains the relationship between realism and animated style. The findings he cites seem to suggest that the more realistic the animated image, the more affective it will be and yet the more unreal it is the more emotionally rewarding it can be. Can there not be a happy medium between the two tendencies? Will only photorealistic recreations of experiential reality trigger strong, affective MNS simulation, or can the same be said of more metamorphosis-centric, unreal animation? Answering such questions will demonstrate how animation can, using the poetics of selective augmentation, present a world that is “more real” than live action by using its unique unreal capabilities, effectively combining the two tendencies of animated poetics discussed earlier.

Power takes a tentative step in that direction when he writes, “moment-to-moment multimodal perception involves proactive emotion-motivated creation of a scenario of meaning for the organism in its environment, searching out certain aspects, exaggerating some, simulating or ignoring others” (38). He posits that our perception of reality involves selection and exaggeration. I contend that in animated poetics the same process is at work and is largely completed by the animator in production, thereby creating animation’s unique brand of realism.

Selective Augmentation as Animation Poetics

Animators John Halas and Joy Batchelor wrote, “it is the live-action film’s job to present physical reality, animated film is concerned with metaphysical reality, not how things look, but what they mean” (qtd. in Wells 1998, 11). While this quote seems to plant these animators firmly in the “anti-reality” camp of the scholarly debate mentioned at the outset, this thought forms the basis of our synthesizing approach to the two tendencies of animation poetics, unreal on one the one hand, and realist on the other.

Selective augmentation occurs when animators select certain materials or behaviors to highlight through formal techniques such as movement, textural detail, or color, thereby increasing their tangible credibility while simultaneously lessening their photoreality. By selecting experientially real aspects of objects to exaggerate, selective augmentation preserves the principles of intention and action through credible movement (behavior) while demonstrating a high degree of aesthetic control over materiality, allowing the spectator to become fully engaged with the image on a mirror-neuron level while also giving the animator ample opportunity to provide valuable insight into the material spaces we inhabit by selecting which aspects to caricature.

Before delving further into this definition and its accompanying theoretical implications we can contend with a possible objection—“if MNS is triggered through realistic representations, then why not choose to augment everything and create a perfectly hyper-real world that is sure to engage the audience’s embodied simulation?” Put simply, trying to craft an entire piece indistinguishable from photographic reality would lead to what Power identifies as the problem of the uncanny, which occurs when animated films stray too far into realism.

Power cites box office failure motion-capture animated films The Polar Express and Final Fantasy: The Spirits Within as being indicative of what happens when “realism seems to elicit more engagement from viewers up to a point, until slight mismatches become uncanny and people find them strange…the brain registers the subtle mismatch resulting in a loss of emotional engagement or even repulsion” (40-41). In animation, the character is always far enough removed from reality aesthetically to avoid this feeling of the uncanny while bearing close enough resemblance to it in materiality and motion to trigger embodied simulation, thereby combining both unreal abstraction with enough reality to encourage mentalizing and internal simulation. The tightrope between abstraction and reality is the essence of caricature, wherein “the brain seems happy to accept caricature but not an uncanny mismatch” (41). Caricature is synonymous with augmentation in this sense, for it does not create something new but enhances parts of what already exists.

Take, for example, animation such as Finding Nemo and Ratatouille. Film theorist Siegfried Kracauer thought that common or transient objects/materials can gain a new and greater significance when seen through the camera’s lens (1999). Animation can accomplish the same results in the poetics of selective augmentation. Take, for example, the early experimental computer animated short Tin Toy (1988). By selectively augmenting textural materiality and explicitly demonstrating the smoothness of tin, the stiffness of plastic and the dry crinkle of paper, Tin Toy demonstrated that these materials have greater significance than just the stuff of toys: they have a beauty and life of their own. Kracauer’s redemptive reality principle states that, after seeing Tinny’s tin, one would look at all tin toys differently and mark, perhaps for the first time in one’s life, that the way the light shines off of metal is beautiful. The small act of recognizing intricacies and details in the world around us is one of the benefits Kracauer bestows on the entire cinematic apparatus, and an idea that we are extending into computer animation (1999). And, if we are looking for a deeper, more significant meaning to our entire poetics discussion, we could technically state it here: the detailed, three-dimensional recreations of materials in computer animation redeem those real-world materials from everyday obscurity and have the potential to alter the audience’s perception of the real world by instilling a newfound respect and admiration for the moral gravity of the resources around them.

In other words, reading or watching Charlotte’s Web, one might be hesitant to step on the next spider one comes across. After watching Ratatouille, one wants to go out and have a nice meal. The producers desired this—after seeing Ratatouille, critic Bob Mondello stated that “everyone will want to go out to eat afterward” (Mondello, 2007), a sentiment Calahan echoes saying “I wanted people to leave the theater feeling hungry” (qtd. in Desowitz, 2007). After watching a 3D cartoon narrative, the audience leaves the theater feeling hungry in the case of Ratatouille, wanting to go skiing after Frozen, or play a video game after Wreck it Ralph: their expectations of real-world experiences have changed. One appreciates these simple activities and materials more than one did before entering the world of the movie. This newfound appreciation of food (in Ratatouille’s case) came as a result of animation showing audiences the world of these normally “inert” or “inanimate” objects (or un-anthropomorphized in the case of Charlotte) in vivid detail.

Seeing through the lens of Kracauer’s redemptive reality, we can assume that, by choosing to augment certain material characteristics of objects and things, that is, to caricature them, the animators saved those features from obscurity and forgetful oblivion; in other words caricature is applied to materiality. Selectively augmenting the material characteristics of objects and characters allows us to experience touching those materials in a tactile, embodied way that no other medium can. Thus, selective augmentation not only presents a vision of experiential reality but has the unique opportunity of allowing us to experience these caricatured materials (impossible in photoreal cinema) in an embodied way through MNS.

However, there are a number of concerns and objections regarding the poetics of selective augmentation and its concordant theoretical applications. First, there are definite consequences to never seeing fish and rats the same way again. When audiences engage with a hyper-real world that looks and feels more engaging and magical than the one in which the audience lives, a feeling of disenchantment with the real world can occur. Selective augmentation has generally been driven towards making every day materials appear more attractive than they usually are: no rat is ever as soft and clean as Remy. After defining CGI’s obsession with photorealism cited at the beginning of our discussion, Hardstaff remarks that CGI has, in many cases, exceeded reality. He notes an emergent digital animation that “hungers for the impossibly real” (2007, 196); this is impossible not in the sense of capturing reality, but in creating an unattainable one.

Even though MNS theory ensures that viewers never confuse their bodily actions with those of the onscreen characters (which would make for a very interesting theatre-going experience indeed) there is the possibility that these images, through their combination of MNS embodiment and “attractive” selective augmentation, might cause viewers to become dissatisfied with experiential reality. Leon Gurevitch also points this out when he discussed Avatar, stating “So ‘real’ and immersive was the stereoscopic fantasy world that Cameron’s animators had built that these viewers had allegedly fallen victim to depression when faced with the reality that the world of Pandora did not and would never exist” (Gurevitch, 2010). I present these very real concerns both as suggested areas of further study and further proof that animation is, in fact, presenting a world that is somehow more real and, apparently, more desirable than the one we currently inhabit.[1]

While selective augmentation is a novel way to synthesize these various theories, it is informed by old ideas. For example, Wells coined the term “over-illusionism,” wherein “every object and environment, though recognizably ‘real’, precise in its construction, and logical in the execution of its own laws, becomes essentially over-determined, moving into a realism which is simultaneously realistic but beyond the orthodoxies of realism” (1998, 27). The difference between this theory of animated realism and my own is that Wells explicitly states “every object;” therefore, he does not make a distinction in exactly what gets over-determined and thus can fall into the trap of the uncanny. By demonstrating how credible movement such as twitching ears and leading eyes demonstrate credible motion/intention thus triggering mirror-neuron simulation and engagement along with shiny fish scales that proffer audience members a heightened version of reality in the 3D cartoon image, we have formed a viable bridge between the reality and fantasy sides of the animated debate.

One might now wonder that, giving the bridge we’ve built between reality and fantasy in animation, what is to be done with it? We’ve used examples and aesthetics to show that we can use MNS theory to form a bridge between being real enough to trigger mentalizing but not so real as to hinder identification, proving after a fashion that just because an image isn’t real doesn’t mean it can’t literally feel real. We’ve shown that computer animation can be the bridge connecting the real world and the synthetic one in a more visceral, tactile, engaging way than has ever been done before; it is a completely new viewing experience that has many implications beyond enchantment.

William Brown, for example, gives us one possibility for the uses to which animation’s more-than-real poetics can be put when he describes its ability to portray non anthropocentric points of view and depictions of the sublime. Speaking of the animated sequences of space in Life of Pi, Gravity, and Interstellar he writes

What unites [them] is their spectacular and computer-generated depictions of space … the ‘sublime’ moments in all three films suggest that while humans cannot transcend themselves, cinema can—and in these instances does—transcend humanity thereby presenting to us non-anthropocentric perspective…. Since the virtual camera has no physical existence, and since many of the images of deep space that we see in all three films are computer-generated, digital visual effects, it perhaps seems counter-intuitive to say that these films can achieve a sense of the sublime and that they are somehow ‘interactive’: we know that what we are seeing is not real and that it was created by a computer; being intangible and precisely not an index of a pre-existing and physical/material reality, there is nothing with which we can interact. However, in spite of the images lack of indexicality, and not only in spite of but also because their ‘virtual’ immaterial nature, these images can present non-anthropocentric perspectives all the more powerfully. (2020, 66)

Our poetics of selective augmentation dovetails nicely with the very effects of animation that Brown theorized above, namely how new, non-anthropocentric points of view are communicated more powerfully than in traditional cinema. Because of the animated image’s non-indexicality (unreal, fantastic, frame-by-frame construction), yet through its selective augmentation of the color, size and scope of stars and space, the viewer is treated to a perspective not tied to the auspices of any human characters. What Brown is hinting at, and which we have illuminated via our discussion of animated poetics, is that the sublime aspects of space are not only redeemed in the eyes of the viewer through animation’s selective augmentation of color, size and scope but that the viewer is able to mentalize (via MNS) the sublime aspects of those images in a new way, identifying with the transcendental, animated virtual camera and providing audiences with a window into experiences hitherto undreamt of.

Suggested Viewing

What follows is a list of short videos which exemplify how selective augmentation functions across a number of studios and styles in modern animation.

Frozen: Observe how, although Olaf is a non-indexical character, the animators have chosen to selectively augment the way light reflects off the myriad snowflakes which make up his exterior, giving him a smoother quality than the matte snow around him.

Despicable Me: Every metal surface in Gru’s lab has had its shininess selectively augmented to the point where the audience can more easily mentalize the hard textures, which are then placed in stark contrast to the organic, soft shapes of the adopted daughters.

Tin Toy: The first time Pixar attempted to selectively augment metal and plastic.

Kung Fu Panda: Compare the detail of Po’s fur, selectively augmented via small flecks of white against black to highlight its detail and softness, to the almost non-existent texture of Viper, who appears almost sketched by comparison.

Rango: The skin texture on Jake the Rattlesnake has been augmented to the point where the cracks make him appear rougher and more dangerous than a living snake of the same breed, allowing the audience to mentalize the danger in the confrontation.

Luca: In Pixar’s most recent film we can see how, although the characters have become more simplified in their design, the bumps and cracks on the walls of the town have been augmented through lighting and other means to allow the audience to more easily mentalize the setting’s history and textures.

Works Cited

Brown, William. 2020. “Knowing Not What to Believe: Digital Space and Entanglement in Life of Pi, Gravity, and Interstellar.” In Screen Space Reconfigured, edited by Susanne Ø. Sæther and Synne T. Bull, 55–76. Amsterdam: Amsterdam University Press.

Bryant, Jennings and Peter Vorderer. 2006. Psychology of Entertainment. Mahwah, NJ: Lawrence Erlbaum.

Csíkszentmihályi Mihály. 2014. Flow and the Foundations of Positive Psychology : The Collected Works of Csíkszentmihályi Mihály. Springer.

Desowitz, Bill. 2007. “Ratatouille Pixar Style: Bon Appetit.” Animation World Network.

Gunning, Tom. 2014. ”Animating the Instant: The Secret Symmetry between Animation and Photography.” Animating Film Theory, Karen Beckman (ed. and introd.), Duke UP.

Hardstaff, Johnny. 2007. “The Impossibly Real: Green Belting the Imaginary.” Animation—An Interdisciplinary Journal, 2: 187-202.

Kracauer, Seigfried. 1999. “The Establishment of Physical Existence.” Film Theory and Criticism, 6th edition, edited by Leo Braudy and Marshall Cohen, Oxford University Press, p. 303-313.

McCloud, Scott. 1994. Understanding Comics: The Invisible Art. Kitchen Sink Press.

“Multiplane Camera Developed to Give Depth Illusion.” The Science News-Letter, vol. 33, no. 20, [Wiley, Society for Science & the Public], 1938, pp. 321–321, https://doi.org/10.2307/3914300.

Power, Pat. 2008. “Character Animation and the Embodied Mind-Brain.” Animation 3 (25): 25–48.

Prince, Stephen. 1999. “True Lies: Perceptual Realism, Digital Images, and Film Theory.” In Film Theory and Criticism, 6th ed., edited by Leo Braudy and Marshall Cohen, 270–282.. Oxford: Oxford University Press.

Sherry, John L. 2004. “Flow and Media Enjoyment.” Communication Theory, no. 4: 328.

Snow White and The Seven Dwarfs. 1937. Directed by William Cottrell, David Hand and Wilfred Jackson, Walt Disney.

Torre, Dan. n.d. “Cognitive Animation Theory: A Process-Based Reading of Animation and Human Cognition.” Animation—An Interdisciplinary Journal, vol. 9 (no. 1): 47–64.

[1]. A viewer possibly becoming dissatisfied with the real world as a result of flow and mentalizing with a 3D cartoon narrative can also explain how adolescents can become addicted to the interactive, 3D cartoon world of video games.