puppets

At the NUKU theatre and puppetry museum in Tallinn, Estonia, there are puppets, automata, and masks. For now, let’s just talk about puppets.

A. puppets

Quick, how much do you think the economy of puppetry is worth in the modern day?

The answer has to be at least $1 billion, because that’s how much just e-sports is worth this year. The taut, mediating strings of the marionnette have been replaced by keyboards and mice and other controllers, but the essence is the same: a kind of faithful, kind of full mapping with negligible temporal lag from the movement space of your hands to the movement space of an avatar.

Perhaps a better descriptor for this kind of mapping, following Dessalles Why We Talk, is that these mappings are analogue. The intension of this term is best characterised by example. Say the twitch of fingers maps to the twitching arm of the marionnette, and the rotating wrist maps to a courtsey. The mapping is continuous between between the structure of the hand movement space and the marionnette movement space, in that a larger twitch of the fingers is a proportionately larger movement of the marionnettes arm. This continuity is possible because fundamentally the two spaces share the same kind of structure: just as one can combine a twitching finger and a wrist rotation, the marionette can concurrently move its arm and courtsey. Continuity of control mapping and shared structure taken together is what we mean by analog.

Puppetry becomes robotics when the mappings become digital. In a fighting game like Tekken, where one has to input a specific sequence of commands to execute a special move, the analogy to marionnettes is that the strings are mediated between the hands and the marionnette by a box that waits for essentially arbitrary input tug-sequences from the hands, and outputs essentially arbitrary output tug-sequences towards the marionnette. This arbitrariness is what Saussure identifies as the capacity for language beyond iconicity. Puppetry becomes robotics where the string is no longer taut, but mediated by a box with its own structure.

In the universe of Being John Malkovich, the puppeteer Craig Schwartz can conceivably puppet John Malkovich puppetting a marionnette. This iterated puppeteering is still plain analogue puppetry; the jump to digital strings and robotics comes from something else. The mediating command-boxes of digital mappings are mappings from control space extended by novel spatial structure: in Tekken, the characteristic logic of a fighter is a map from sequences of input commands extended in time to atomic actions in the game universe. When programming, basic functions must be arranged according to the spatial structure of the syntax of the language. In both cases, the domain of the command-box is not the control space of the user, but the control space of the user embedded into an extended space of spatdially arranged control-inputs.

I think that puppetry in its analogue and digital forms is a foundational skill for intelligent agents with embodied cognition. An appraisal or representation of one’s own capabilities in control space arises from a Yoneda Lemma style argument: what I can do is defined by the kinds of puppets I can puppet, the kinds of tools I know how to operate. But whether it’s throwing a projectile or ritualised dance with intricate wrist movements or heart surgery, the first puppet space the agent must master is the space of bodily movement in a Euclidean environment. There’s some neurological evidence that abstract concepts in our internal mental representations factor through this Euclidean spatial structure. I’d like to know what the internal concept representations look like for a person confined to an electric wheelchair from birth; would it be that their concepts factor through the control space of the wheelchair controls.

For now, depending on how much you can tolerate concept-creep, either nothing is puppetry or everything is. Maybe later we’ll try out a category-theoretic formalisation of puppetry, and see what that entails.