Tr : 21 Jan 2020 Guest : Barbara Tversky

Video: https://youtu.be/KqdlcxHbviE

Chat log: https://futuretextlab.info/2022/01/21/chat-21-jan-2022/

Frode Hegland: And here is Barbara. Bang on time. Okay, so many messages. I’m going to go all kinds of focus mode now, in case. Unless someone has a problem joining. So, I’ll look out for that a little bit. But I think we’re quite good, actually. I just hope no one goes to our normal “Zoom” weekly meetings, because that’s a different link. I decided these monthly ones should be slightly better protected.

Mark Anderson: Frode, would it help… Is the other one open? I mean, if you want I can duck off and just go check and see if anyone’s… Actually most of the faces are all here. So, yeah. 

Frode Hegland: I should get a message if someone joins, so I’ll leave it on the side. So, Barbara, who needs no introduction, we’ll get an introduction. Very brief one. But in a moment when we have a quorum. I have no idea what that means in this context, but whatever. But Barbara, before we properly start, you’re, let me guess. East Coast? West Coast?

Barbara Tversky: I am in New York. East Coast. In Manhattan, Greenwich Village.

Frode Hegland: Well, it’s a really special time in the village. This is the only time in New York where it doesn’t smell, right? I say that lovingly…

Barbara Tversky: It’s always a special time. New York has so much to offer anybody. A good friend used to call it, “The ultimate adult playground.”

Frode Hegland: Yes. I would agree with you, and you’re getting thumbs up from Alan who is just up the road from you. 

Alan Laidlaw: Yes. I would love to… Yeah. You’re around Columbia, is that correct?

Barbara Tversky: Yeah I am affiliated with Teachers College at Columbia. But at the moment, I’m living downtown. I’m teaching by “Zoom.” (INDISTINCT) has a very new meaning in these days.

Alan Laidlaw: Oh, that’s great. Once we get through all the safety, I’d love to buy you a coffee. I’m in Brooklyn right now. But I was in Chelsea in 2019.

Barbara Tversky: Well, Brooklyn is the hip place to be now. Manhattan is more and more old people. So, the young people, the new movements, they’re in Brooklyn.

Alan Laidlaw: It’s easier to have a dog in Brooklyn for sure.

Barbara Tversky: You know, there are times when I walk out at night, and there are more dogs than people in Manhattan. That’s because some people are walking two or three.

Alan Laidlaw: That’s very true.

Frode Hegland: And right there, with the entry of Ben Schneiderman, my arbitrary screen has filled up on the side here. And so, I think this is a good time to start. We’re only two minutes in. I don’t think there’s that many more people. Why don’t we do one of the nice things we sometimes do before Barbara properly starts? There maybe one or two more people. Randomly go through, and say where everyone is. Alan, Brooklyn. Barbara, New York. Bob Horn?

Bob Horn: San Francisco.

Frode Hegland: Ben?

Ben Shneiderman: Hi. I’m in Bethesda, Maryland.

Frode Hegland: Maryland, wonderful. David Lebow?

David Lebow: Tallahassee, Florida.

Frode Hegland: Karin Hibma?

Karin Hibma: I’m in Oakland California.

Frode Hegland: So cool. Oh, what the thing is moving about here. Peter, we know where you are. New York, right?

Peter Wasilko: Yes.

Frode Hegland: Dave Millard, somewhere down south in the UK. Where are you, David?

David Millard: Hi, Frode. Yeah. I’m in Wiltshire, in the UK.

Frode Hegland: Sounds so fancy when you say it. Brendan?

Brendan Langen: I am in Chicago.

Frode Hegland: Oh, I didn’t know that. Should have known that. Adam, you are in (INDISTINCT), right?

Adam Wern: No, Stockholm, Sweden.

Frode Hegland: Oh, Stockholm. Sorry, I got it backward. Dave De Roure?

David De Roure: I just changed my background to reveal my location of Oxford.

Frode Hegland: Ah, yes. Now I can see that. David (INDISTINCT)? You’re muted, David (INDISTINCT). He must be muted. Chris Gutteridge, you’re in Southampton, right?

Chris Gutteridge: Yep of mayflower fame for the Americans.

Frode Hegland: Mark Anderson? Southampton?

Mark Anderson: Portsmouth. Just along the coast. I’m from Southampton.

Frode Hegland: Pam, where are you?

Pam Drouin: I’m in Columbia, Missouri.

Frode Hegland: Very cool. Rafael?

Rafael Nepô: São Paulo, Brazil.

Frode Hegland: Brilliant. Patrick?

Patrick Lichty: Winona, Minnesota.

Frode Hegland: And Brandel?

Brandel Zachernuk: Sunnyvale, California.

Frode Hegland: And I think, finally, my list keeps moving about here, so I don’t want to waste too much time. I think we have Lorenzo. Where are you in the world? Lorenzo is muted. Okay, Barbara… 

Ben Shneiderman: Thank you, Frode, for bringing this illustrious group together. It’s quite wonderful. And thank you, Barbara, for making it possible.

Frode Hegland: Yeah, I mean. I’m just super thrilled. The world is such an interesting place now. I’m sitting in Wimbledon, South London. Lorenzo, Italy, I just saw. You know, for us to be able to meet like this, is incredible. So, without further ado, Barbara. Everybody, I’m sure, has at least read… Actually, in the chat, while Barbara does her intro, can you please write if you’ve read the book? Or if you’re currently reading it? It would be really nice to have a bit of a (INDISTINCT). I will have to write, of course.

Barbara Tversky: So, thank you for inviting me. I have far too much to tell you. And I’m trying to tell it through visuals not in the book. The talk will be like pieces of hors d’oeuvres, so a bit disjointed, but they’re meant to set up talking points so that you can ask questions, or discuss things. I should say that you’re more punctual than my students, but my students are far more geographically dispersed. Kazakhstan, China, Korea, Japan, just everywhere. And so, “Zoom” does enable that kind of interaction in one class. Okay, I’m going to share a screen and I want to, before I show pictures, I just want to say a bit, without a picture, of how I got into this field at all. I’m a bit of a contrarian. When I was a graduate student, people were reducing everything that people thought about all representations of the world to something like, language, or propositions. And my feeling, looking at that, and I did look at all the research at the time, is language is efficient, decomposable, it has all kinds of advantages. I rather like it, I’m using it right now. But it seemed to me that language couldn’t begin to describe faces, scenes, emotions, all kinds of subtleties. And then, I started thinking that space is half the cortex. So, spatial thinking must be important. And by special thinking I mean the world around us, and the things in it, including our own bodies, other people objects, scenes. And that special thinking evolved long before language, which occupies a rather tiny bit, but important place in the cortex, but came much later, and is in less connected with the rest of the cortex. And then you think, anyone who’s been a parent, or owns a dog, that babies, and other creatures think and invent so many marvellous things without language. And for that matter, so do we.

So, I got interested in spatial thinking. This is some of the early ways that we communicate. Gesture arises in children long before language. And in fact, children who gesture quite a bit, speak earlier. Games where we’re imitating each other, taking turns, alternating what we’re doing, this kind of interaction in games, rolling the ball, rolling it back, it builds trust. It sets up conversation, which is, you say something, I say something. So, it sets up cooperation, conversation, and many other things. This is done early on and communicated by action, not by actions of the body. And reciprocal expressions on the face it isn’t communicated by language. So, I’m going to jump lots of jumps, and I want to talk now, because you’re interested in text, about kinds of discourse. So, one important kind of discourse is a description, which is a state of affairs in space or time. Explanations build on descriptions, but add a layer of causality and reasons. Stories add on descriptions and explanations, they include both, and they add a point of view. The author, or authors, they add emotion, drama, there are a whole set of elements that stories can and each one isn’t necessary. And there are deep arguments about what a story is. But I wanted to make those distinctions, and check with my colleagues in discourse analysis that the distinctions reflect the field. Stories are studied in so many disciplines. Then we have directions, which build on explanations, but give you just an “A” to a “B.” And we have arguments, which bring together a set of stories, explanations, and descriptions to make a certain point. So, I’m not going to labour that. I want to jump again, I already talked about how communication begins in humans and other animals as well. Through the body, through the face, through actions. And I could talk, at this point, about mirror neurons, but I’ll skip that, just leave it as a teaser. 

But the earliest human communication and probably human includes Neanderthals and other hominids goes back at least 40.000 years. It keeps going back, as this was a discovery in the last few years. You can see hands there, there are animals there. It’s from Sulawesi. And, as I say, these are being discovered everywhere. This is the former oldest map, 6000 years. It shows two perspectives, an overview of the paths and rivers, and a frontal view of landmarks. Linguists don’t like this. Geographers don’t like two perspectives. But people seem fine with them. This is the current oldest map, it’s about two inches by one inch. A stone. It shows the surroundings around the cave where it was found, some 13.000, 14.000 years ago. And it’s tiny. So, it could be taken with you, to guide you on going back. A map of the sky going back 5000 years. This is a valley in Italy, it’s a drawing of a petroglyph. Again, two points of view. Eskimo maps. They were carved in wood, very beautiful, carried on canoes, they showed the outlines of the coasts. And they floated, in case they fell in the water. South Sea Islanders Map, probably familiar to you. Shells representing islands, bamboo strips, the ocean currents, which are like the highways of the ocean. And at least some of the people that were trained and carried these with them, 2.000 miles on the open ocean, at least some of them returned home. A map by North Coast Indians, showing the various settlements on their hands. Now I’m jumping again to depictions of scenes. Again, going back 40.000 years. Chauvet. Going back even farther in Sulawesi, although I’m not sure. This one I especially like, it is in the book. It’s a petroglyph on the left, and the drawing of it on the right. And it’s showing two suns in the sky. Quite remarkable what could account for that. An Indian astronomer did some history on it and found that, at about the time they could date the petroglyph,  there was a supernova. And it was such a remarkable event that someone inscribed it in a stone. Stones were, in a way, the newspapers of antiquity. Here’s another example from the U.S., a whole valley full of these. It’s called Newspaper Valley, and it has many of these petroglyphs showing events. So, events in making bread in a tomb  in Egypt. Events in the Trajan Column. Now we have calendars, they also go way back. Some circular. Some tabular. All these forms become important, but I won’t be able to talk about them. Then we have number. We have tallies. Again, you can find them all over the world. It’s not clear what they’re representing. But having a one-to-one correspondence from a mark, to an idea, to an object, to people whatever they were counting, moons, is a rudimentary form of arithmetic that was again, inscribed in stone. A more advanced, you know about these. So, ancient visualizations represent space. They represent people, objects. They represent time, and events in time. They represent number. These are all important concepts, and you will find them in the newspapers, journals, magazines of today. And they’re so important that the brain has specialized areas for processing them. And what’s extraordinary about all of these is, they can be spatialized. So, this is part of my argument that is, spatial thinking is foundational to all thought.

So, early communications began as pictographs. In some way, you can still find… Well, there was a civil war colonel who collected these during battles and then, Dover later printed his findings. They’re quite remarkable. This is a love letter between the two animals. On the left are her totem and her lover’s totem. And it’s a map leading him to her tepee, and she’s beckoning him there in the map. In the 18th century, the age of enlightenment, we finally get graphs. Because the early visualizations, that ones that I showed you, except for time, were more or less things that were actually in the visual spatial world. But more abstract concepts, like balance of payment and graphs, developed only in the late 18th century, and they began to blossom. So, Diderot, I would love to walk you through this, it’s a way of teaching diagrams. The top half is a scene, which would be familiar to 18th century eyes. The bottom is a diagram. It differs from the top, and things are arranged in rows and columns. There’s a key. Lighting is used not naturally, but to reflect the features of the objects. The objects are sized. So, you can see them in the diagram, not in their natural sizes. So, this is a visual way of teaching people what a diagram is. In fact, by now, we’ve diagrammed the world, and we’ve set up where different kinds of vehicles, pedestrians can go, where they can’t go, where they can park, when they can go, and it moves us through space in an organized way. But we’ve really diagrammed the world. 

So, graphics augment cognition, they record information, convey information, promote inferences, enable new ideas. This is a lot through sketches, and I won’t be able to talk about that. But can answer. They facilitate collaboration. They’re public, so we can both, or all of us, revise them, make inferences from them, enable new ideas from them, see them, and point to them. Gestures are important. So we can distinguish two kinds of graphics. Those that are inherently visual. Maps would be a prime example. They’re ancient. And visualizations of metaphorically visual. Graphs, charts, diagrams. And again, they seem to be a Western, at first, production of the age of enlightenment. Good graphics schematize. This is a prime example. They also annotate. They’re multi-modal. Graphics consist of elements and relations among them. And my argument is, these can convey meaning directly. But they don’t have to be learned. In many cases, they’re quite natural. One of my claims is this visual spatial way of communicating is much more natural. Pointing. Pointing draws a line from my finger, to the object that it’s pointing at. So, it guides your eye to that object. And that’s very natural. We saw babies use them. Chimps use them a bit, but other chimps don’t follow them. So, it shows what the chimp is thinking, but it isn’t taken as a communication by the other chimps. Those can be separate. And then again, we’ve done a lot of work on gestures, showing that many of them are really helping us think. But they can also help others think. So, graphics consist of elements, spatial relations among them, they convey meaning quite directly, and they represent thought directly, by using elements and spatial relations on a page, a virtual page, or a page in the air, as gestures do, and they represent elements and relations in the world. So, elements can resemble what they represent. Iconic. They can bear a conceptual similarity to what they represent. We call them figures of depiction. Metaphoric. And they can be schematically related. Elements can be iconic, metaphoric, abstract and symbolic. Early writing began as pictographic. And the alphabet was invented only once and took over the planet. Not everywhere, actually. Probably half the planet is still using some descendant of the pictographic language. So, meaningful marks. Our claim is that icons, dots, or depictions, lines, arrows, containers, can represent ideas quite directly, and we have some empirical evidence for it. And I’ll show you those in a minute with a detour to a minimal diagram, or could be called a minimal diagram, a line and a dot namely points representing ideas or places in a real map, and lines representing relations among them. And this minimal diagram, a link between two points, is the building block for many. This is the internet, in 1987, you could still draw it. Family trees. Social networks. These are social networks produced by some of our participants. Phylogenetic trees. Art. This is Mark Lombardi, no longer with us, who made beautiful networks representing where the money went, and other political, economic ideas. And people scrutinized them for hours. So, the meanings of these abstract forms are simple, efficient, neutral, and abstract. Some of the meanings come from Gestalt Principles, some just from gestures, some from the way we behave in the world. Like the paths on the ground are the lines that people make from place to place. So, we’ve done empirical work on each of these, showing that people produce them from verbal meanings, and they understand the graphic meaning from the verbal. And I’m not going to be able to go into that. Spatial relations, again, can be literal or metaphoric. The general inference that people make is, proximity in space represents proximity on any dimension. We use this in gesture and language. We say we’ve grown closer to people. We’ve grown far apart. Place, centre, periphery. Again, language represents that the centre is the centre, and the periphery is the periphery. Directionality is also important. The vertical. Anything that goes up is good in general, except the economist thought inflation going up, and unemployment going up. I could say they’re perverse, but it’s probably because of the numbers. We often get conflicting concepts wanting to go up. But this is gravity. Fighting gravity takes strength, health, wealth. So, anything going up is usually positive, and anything going down, like hell, is negative. 

The horizontal is pretty neutral. It’s neutral in our lives, in our motion, in our world. But there are cultural constraints on it. Writing order that are quite strong and cross-cultural. There are cultures and languages like Hebrew and Arabic written from right to left. Originally the columns in Japanese and Chinese went that way. And many of these concepts get reversed in right-to-left languages. So, just to show you, we looked at diagrams in books, all in many disciplines. And the present day, or better things go up. And I should say, in evolutionary trees it’s always man. Who gave birth to man? So, another thing we’ve looked at, and others of you have looked at going to graphs, now to information graphics is inferences. This research needs to be done more. That different displays lead to different kinds of inferences. And the reason really is the underlying visual spatial representation. Bars separate. They say there are a bunch of “A’s” in this container, and a bunch of “B’s” in another container, and therefore, encourage discrete comparisons. Lines connect showing a relationship. They say “A” and “B” share a value, but “B” is higher than “A.” And we’ve shown these effects in a number of different contexts. Despite what the statisticians would recommend, the visual form of the displays override it, and people tend to make discrete comparisons from bars and trends from lines. So, this kind of research is ongoing, of how different displays, depending on the visual spatial characteristics, lead people to make different inferences. 

Animation. People ask me about animation. Animation is something that’s relatively easy to do now. There’s tons of research showing that people don’t understand animations the way they’re intended. The Muybridge experiments looking at, whether all four legs were off the ground at any point when horses were galloping, as an instance. You can’t see it when horses are galloping. But the stop gallop photography showed that, yes, all four legs are, at some point, off the ground. But the art museums of the world are filled with horses legs incorrectly aligned when they’re galloping. Here’s an example of how hard it is to perceive. So, animations are compatible with thought, in the sense that, they use change in time to convey change in time, but they’re hard to perceive. They show but don’t explain. And most of the things that are animated, when we talk about them, chemical processes running, climate change, we talk about it in steps. So we think about these things in discrete steps, not in this continuous way. Which, as I’ve tried to show you, is difficult anyway. I’m sure good animations can be designed, but it’s trickier than some people think. And obviously, animations appeal to the eye. We’re all, in one way or another, addicted to movies and music. 

Comics. I want to jump to comics because they show all kinds of lovely ways of expressing meaning that are rarely seen in traditional graphics. Whether they’re infographics or graphs and charts. So, one thing comics artists can do, is use space to segment and connect time and space. Here you get an overview of the scene, and then you get the action superimposed on it, in frames on it. This was used by the ancient Aztecs, not just modern comic artist. Visual anaphora. You can get from one frame to another following this red book. The “New Yorker” cover is not just showing writing, but it’s a visual story and a pun. So the book ends up in a trash can being burned by homeless people to keep warm, and the verbal name is “Shelf-life.” But you can follow it because of the anaphora provided. Something from frame one is preserved in frame n plus one. And so on, so that you can follow the continuity. As for good stories and good movies, often you want to break the continuity to create suspense. I’m going to skip that, it’s lovely. Here, following the eyes, and the pointedness of the frames allows you to go back and forth and understand the David and Goliath. This one’s a little harder. It’s a beautiful book called “Signal to Noise” by Neil Gaiman and illustrated fantastically by Dave McKean. It’s showing an aging director, and he’s actually dying of cancer, and he’s got photographs from many of his productions on the wall. You can see he’s thinking. And it shifts perspective to what he’s thinking about. He’s looking, and you can see the perspective switch between the man in the blue coloured shirt and what he’s thinking about, as he watches, looks at all of these frames, and then finally, he can’t stand it. “Stop looking at me!” So, he’s both reviewing his life and haunted by it. And again, it’s conveyed visually. Steinberg, the master, a conveying peeping toms through a mirror that reflects the guy watching from the opposite apartment. More Steinberg. A pun, “Time Flies.” More comics. So here, there are so many devices, visual spatial feed metaphorical, or figures of depiction. You have puns here, polysemy, figure/ground. I want to draw your attention to the old-fashioned telephone cord, which some of you, at least, will remember. So this woman is drawing those other people into a conspiracy by calling them on the phone. So, the phone cord is a literal phone cord, it’s a metaphoric phone cord, drawing them into the same conspiracy. It also serves as the frames of the panels. So it’s doing triple duty. It’s something kids can get right away. Like gestures, you get it almost without thinking. It just pops out at you. So, a beautifully crafted device. 

Figure/ground. You’re seeing the murder, and the noise of the murder is coming through in those black figures that are superimposed on the actual scene. And the black and white drawing is emphasizing the stark brutality of the punctual murder. You light out a life in a second. More Steinberg. “Canceling Thoughts.” Again, I don’t need to tell you. Visual juxtaposition. This is another Gaiman, McKean cooperation. A child is at a birthday party. You can see on the right, they’re playing musical chairs. Here, the child is not interested in the birthday party. So, goes out, and talks with an uncle, who told the child the story of the Saint Valentine’s Massacre by Capone, where he tied his enemies up on chairs and killed them. Shot them one by one. So, you have the chairs there with the men chained to the chairs juxtaposed with it with the birthday, which is a little bit of a brutal game because one child is eliminated at each round from musical chairs. So, that juxtaposition of chairs, again, is a stark reminder of the comparison between brutality of children, and brutality of adults. Okay, metaphor pun. “Puppet Governments,” Feininger. This is Winsor McCay, a brilliant comics artist. This is from the early 1900s, New York. Parts of New York still look like that. And this is, of course, the rat race, running on a treadmill. This is a dream, another one of his where a dream transports the child and then dumps the child back in bed, the way dreams end before they should end. This is onomatopoeia rhythmicity. It’s showing a chase. And by putting the panels on a diagonal, showing the speed of the chase. “Coming out of the frame.” The first pig, whose house was blown up, comes out of the frame, and talks to this second pig inside the frame, and says, “Get out of there. It’s safe out here.” And then, the pigs all go berserk. They get out of the frames. And the frames are on the floor, and they’re stamping on them. So, this version of the three little pigs is a riot. And again, kids can get it. So I’m going to end with another Steinberg. Steinberg drawing himself. Again, a visual way of understanding drawing portraits and so forth. So, I’ve raced through a lot, and I haven’t covered everything that Frode wanted me to talk about. So, I’m open now for questions and I will stop sharing. I think you’ll moderate the questions, because it’s hard for me to read the chat and answer at the same time. 

Q&A

Frode Hegland: Yeah, absolutely. That was overwhelming, but in a very good way, of course. And I think the way we should do the questions is quite simply clicking on the reaction hands. I know quite a few of you have had questions from before we had our presentation today. So, who’s going to be the first for the digital yellow hand up in the air? Brandel, please unmute. Yeah, there we go. Perfect.

Brandel Zachernuk: Hi, Barbara. Thank you. This is brilliant. I’ve been doing a number of your talk tonight. I love the fresh ground and the emphasis on the text here. So, my question is not about animations per see, but about progressively recomposed images accompanied by illustration, via the speech of the illustrator. The actual drawing of lines along with narration. Is that something that you’ve ever studied, or that you would expect to have any particular effect from, in contrast to seeing the completed image of its entirety?

Barbara Tversky: That’s what we do in classrooms, right? I mean, that was the oldstown, I know I have many mathematician friends who still insist on going on the board as they speak. And watching it unfold, and the rhythm with which it unfolds, and the verbal accompaniment at the same time, I think is very effective. So, what you’re pointing to is one way that animations can be made more effective. They unfold in time with narration and explanation. And they add a bit of drama. What’s going to come next? So, I think that’s great. And at the back of my head, when I was thinking about this is: What can you do on text? And it will amplify it. And that is exactly the sort of thing that one can do. And it is like a comic, combining language, and symbols, and sketches, and so forth, all at once. There are beautiful examples on the web. Just lovely examples of people using that technique. And I’ve been teaching comics for probably 20 years, on and off, not quite. And I see a younger generation, growing up with that medium, and drawing and writing at the same time. So, I think people will get adept and talented at doing that, at illustrating what they think, while they’re thinking it. And I think that’s just great. It gives people an extra way of expressing themselves that’s quite poetic, or can be quite poetic, but it’s also wonderful explanations. So, yeah. I’m a real fan of that.

Brandel Zachernuk: I’m curious, have you ever seen Ken Perlin’s work at NYU around at being able to draw in Virtual Reality?

Barbara Tversky: I was an orally fan. And Ken, as a friend, and I was an early fan of his, exactly on chalk talk. And in fact he and I and Steven Feiner, whom I work with, and Hiroshi Ishii at MIT, the four of us put in grant, after grant, after grant to expand, and NSF didn’t like it, and didn’t like it, and didn’t like it. So, a real disappointment for all of us. Because classroom teaching that way is, again, natural and what Ken’s animations did is, you’re talking about a pendulum, and then it could animate the pendulum depending on the length of the string, and so forth. So, being able to speak, and use mathematical mathematics driven animations, I thought was super! Just a super way of understanding. So, yes.

Frode Hegland: Yeah, thank you very much Brandel and Barbara. Bob?

Bob Horn: Oh, hi Barbara. Of course, the question I will ask will not be a surprise to you. I’m very interested in, and wonder the degree to which you’ve done research on the textual elements intimately integrated with the spatial elements. That is most of what you’ve just presented has been the spatial aspects of the kind of visual language communication that we are all using. In addition, the diagrams and comics rely, it seems as much as maybe, 50/50 or even more sometimes, on the words, and how the words are integrated with the visual elements. And that’s been something that I’ve been very interested in, particularly in diagramming. So, I’m wondering if you’ve gotten your research to go in that direction, to analyse, and find out how text is integrated with the visual elements?

Barbara Tversky: We’ve done a lot of work that skirts around that. We’ve shown that you can go back and forth between visual descriptions of maps, or many kinds of diagrams, and the visual spatial. That the same underlying concepts are driving both of them. But that the visual form, for example, root maps is usually not for everybody. But usually a more effective way of communicating that. It’s a long story. But I agree with you that in many comics, what’s going on is in the words. I think they’re poor comics when they’re talking heads. I talk about them as talking heads. If you look at Larry Gonick’s science and history, if you look at his comics, they’re cheap, 10, 12 dollars each. They’re absolutely wonderful. His book on statistics is used as a textbook in many places. At one time, even Stanford. And he’s a neighbour in San Francisco, and his books are absolutely fabulous. He was all about dissertation mathematics at Harvard. A self-taught cartoonist. And he began doing, essentially, visual spatial textbooks on different forms of math, science, history, sex, environment. He’s got bunches of them. He always works with a domain expert. We’ve appeared together on many occasions. And once I had the temerity, stupidity to ask him,” Larry, what do you put in pictures? What do you decide to put in pictures? And what do you decide to put in words?” So, he’s very tall, I’m not, and he kind of looked down at me, with his full height and said, “Barbara, I do everything in pictures. What I can’t do in pictures, I do in words.” And he’s incredibly inventive of what he does in pictures. I have my students go through one of his books, they each choose one, and they go through looking for the visual spatial devices, and every year they come up with things that I haven’t thought of. They see things I don’t. And it’s usually the visual spatial telling the story. So he’s an excellent example of that. There are others. And Scott McCloud and his book “Understanding Comics” is a gem. It’s a gem about stories and narratives, not just about visual stories. But he does talk about the roles of language. And here you’d have to add symbols, like arrows and mathematical symbols. You have to add in comics the way the font, the size of the font, and all the squiggles that are added that give you information about movement, and mood, and smell, and sound. So, you can enrich the depictions in so many inventive ways. And what I’ve been trying to do is urge people who make charts, and graphs, and infographs for science books, to learn those techniques. And, as I say, I’m just pleased to watch younger people. I have a sample of eight grandchildren, and the grandchildren of many of my friends, and watch them latch onto graphic books, and see the graphic books that are doing so much in the depictions besides text boxes, and I’m very optimistic about people coming up with really creative ways to do visual storytelling. So, long answer. Sorry.

Frode Hegland: That was great. Thank you very much, Barbara. Ben?

Ben Shneiderman: Thank you, Barbara for a wonderful, intense, movement through all the space of these wonderful ideas. I think it aligned very well with Bob Horn’s visual language thinking, which has been an inspiration for me as well. But one of the charms of your book was that, it went beyond the spatial and the visual, to the idea of mind in motion. And could you say more about dance and body movement? You talked about gesture, you talk about hands and how people communicate, express themselves, learn by being in motion. Tell us more about that side of the story.

Barbara Tversky: So, it means speech is in motion. And speech is accompanied by prosody. I emphasize certain words, and de-emphasize others. I can give you my mood by, I can sigh, I can sigh short or long, and that’s motion, and it’s just in our voices, and it’s communicating so much more than just the words on page. Although, text again is words on page. And there are ways of amplifying text by putting “dot, dot, dot, dot,” that capture some of that. And sure, our bodies indicate, I mean, I said gravity, if I’m feeling good, I’m standing straight and strong, and if I’m in a depressed mood, I’m down, and we can pick that up in others in a second, especially people we know. We can pick it up from hearing their feet behind us. What kind of a mood they’re in. Who it is. We know these cues. Again, they’re active, motion cues to people. They’re very simple, not as complicated as dance. But really creative, and wonderful dancers, and choreographers can create absolutely amazing displays of emotion, human interaction, human non-interaction, individual feelings from the way they do dance. They’re uncanny. And you see that in theatre, they often hire choreographers to orchestrate how people are moving, and talking, moving their arm, agitated or smooth. So, yes. Huge amounts of human meaning gets conveyed through the motion of the body.

Ben Shneiderman: I do think really that deserves on a much-expanded part, just the idea of walking together, being in a forest, moving forward, sailing on an ocean, flying through the air, walking up a mountain. All of those to me, they’re not just physical experiences, they’re cognitive experiences as well. And they enrich us. And I found your book really opened my mind and thinking to the realization of, how much the body plays a role. Which your book adds so much to enrich the dimensions of analysis, which have been, as you point out, largely linguistic moving towards spatial, and visual, and maybe auditory. But the idea of moving towards body motion that was really, to me, a highlight.

Barbara Tversky: Thank you. I was limited in the book by what there was research on. This is the problem of being a scientist. You don’t want to go too much beyond research. I use a lot of examples, but the examples are all founded in research findings. But I couldn’t go off the way until now. But really, if you think about it, every organism, even a virus, needs to move in space to survive. And the basic movement is approach or avoid. And those are replete with emotion. You approach things that you’re attracted to, that might do you good, that you want. You avoid things that have negative valence. So that, from the get-go, movement is for survival. Even grasses have to move toward the sun or away from rain in order to survive. Even things rooted in the ground. So, we all have to move in space to survive. The basic movement is approach or avoid. And those come with emotions, which I think underlies some of Damasio’s claims, although he’s got brain there too, without emotion nothing happens. And emotion and motion in English and other romance languages have the same root. I don’t know about Germanic, or Chinese, or other languages. But they do have the same root. And we talk about being moved as an emotive response. So, I do think anything that has to do with life, really does derive from motion in space.

Ben Shneiderman: Exactly. But look at how they impact on design, or even the “Zoom” in front of us. Some people have just their text name. Some people have a frozen image. Others are live and animated. I like to be Zoomed standing up, so I can be freer to move around. And I think I express myself better, and I can reach out, or I feel the other person better when they are animated, as well. Anyway, thank you.

Barbara Tversky: Yeah, absolutely. I’m frozen in place in a classroom, I can’t stand. But when I move around the classroom, I’m going the whole width, and sometimes the length of the classroom. So, I can’t do that on “Zoom,” it drives people crazy. So, I plant myself in a chair. And when I’m listening to “Zoom,” it’s often on my phone, walking. So, a longer story. “Zoom” has advantages and disadvantages. Like anything.

Frode Hegland: Yeah, absolutely. You mentioned the question of other languages like Germanic languages. In Norwegian, “følese” is the word for feeling. But that can be, you can touch, it’s also touch feeling, as well as an emotional feeling. But the funny thing is that, “bevege seg,” which means movement, is also what you would say if you were emotionally moved by something.

Barbara Tversky: Yeah, nice. I should ask my students who are in Japan, or China, or Kazakhstan, or Malaysia what their languages do. Yeah, thanks.

Frode Hegland: Oh, absolutely. Peter?

Peter Wasilko: Yes. Have you given any thoughts to the evolution and interplay of note-taking? And I was recently reading “Lines of Thought,” (INDISTINCT) typesetting and textbook design. And also, do you have any thoughts about interactive fiction systems?

Barbara Tversky: I’m having trouble understanding you. Did someone understand him better and can tell me what the question is? The mic is bad, my hearing is shut. 

Frode Hegland: Well, I got the last part really clearly, Peter. You asked if Barbara has any comments or perspectives on interactive fiction? Can you please repeat the first part? I also had some problems hearing you.

Peter Wasilko: Yes, the first part was whether she had any thoughts about the evolution of note-taking and textbook design?

Frode Hegland: The evolution of note-taking and textbook design, as well as how they interplay. Thanks, Peter.

Barbara Tversky: I don’t really know much about the history of textbooks. I do know there are a huge number of experiments trying to compare text and diagrams or graphics in one way. And many of them are unsatisfactory, because you can have good text, or poor text, and you can have good graphics, or poor graphics. And I think people in the info design graphics community have been developing standards, or best practices. For good graphics is complicated, because it depends on your audience, and what you’re trying to convey. You can’t have absolute principles like you can for font size. And there’s a little bit of work on textbook design and what it should look like in good text and poor text. But the range, in both cases, seems so great. And studying it would take an historian of sorts, to know the evolution, the development of those things. And it would have to go across cultures. What happened in the East, as well as what’s happened here. So, I think that’s way beyond my expertise. but I don’t think I answered all the parts of the question. Probably I can’t answer them.

Frode Hegland: The second part was on interactive fiction.

Barbara Tversky: Ah, interactive fiction. I don’t know whether people have done research on it. I know my kids, who are now parents themselves, loved it as a kid. They weren’t around when I was a kid. But my kids loved it. And then, of course, the computer games that are built on storytelling. People get incredibly involved in. So, probably those designers know a great deal. They have a lot of heuristics and rules of thumb for that. So, the one kind of discourse I didn’t put up is conversation. And that grounds interactivity. And conversation isn’t like a lecture, it’s two or more people speaking and no one can dominate. As I’m dominating now, in a normal conversation no one can dominate. What you get in that kind of interactivity also, is little bits of information. Bite size. That you can consume, and it arouses a question, and then there’s another bit of information. And interactive graphics do that. They allow you to involve yourself in it, in little bits, where you can get background where you need it, or where you want it. Not all of us want, you know, there’s that old joke about a book about penguins that told me more about penguins than I ever wanted to know. So, different people will want different amounts of information. And that interactivity allows me to have a conversation with a graphic, where I’m asking bite-sized questions, and getting bite-sized answers that lead to something else. So, I’m building up my own knowledge that way. And I think the interactive fiction can do that. And also add the suspense to it. We started at one point trying to compare comics with traditional graphics, or traditional graphics plus text. Too many things were going on at the same time. Too many uncontrolled variables. And as a cognitive scientist, it’s those crucial variables that we’re after. When you’re a designer, or an educator, it doesn’t matter to you what’s doing it. The combination is probably doing it. It’s in the interaction, amongst those elements. So again, then finding guidelines for creating good ones becomes difficult, because there are so many moving parts. I mean, like building a city. But nevertheless, we can judge which ones are more and less effective and why. There are times when I want to lecture or a book. There are times when I want that interactivity. Again, I’m not sure I’m getting at your question, but.

Frode Hegland: I think that was very useful. Alan?

Alan Laidlaw: Sure. I’ve got so many questions. Thank you so much for giving this talk. And I had the childish desire, which I still may succumb to, of showing off every reference you made. That’s somewhere behind me because I’m a nerd. But that’s great, I used to be a cartoonist, it’s where I got started and a lot of that came from reading “Understanding Comics” and that was my first entry to like, “Oh, this person thinks like me.” And I’ve never had these thoughts. But enough of that because could go on with many questions. So instead, I’ll throw one that just popped up while you’re giving the talk. This may be out there, and feel free to dismiss it. The thinking in context of cave paintings, and sort of where we got started in scribing. The commonality, seems to me, that it’s always the physical act of resistance against a surface. And so, I’m wondering about that in context to where a theme has been trying to probe into VR, what that’ll be like? Is there any research around how that resistance, that pushing against something to create is different than, or I don’t know, is it a class? Because with VR, we could say at least, there’s nothing to push against at the moment. But in dance, there’s also nothing to… Well, there are motions, there’s creation that doesn’t involve resistance exactly, not in the same sense of pushing against something to. Does any of that make sense and is there any work on it? 

Barbara Tversky: Yeah, thank you. And I could probably learn a great deal from you, as a cartoonist. I absolutely agree with you. I don’t know about research, but it’s one of the complaints that people in architecture schools have, that people no longer know how to draw. That drawing on pixels is just very different from using a pencil, or a pen, or a brush. And artists, and calligraphers, and so forth talk about what the thing is, how it’s held in their hand, what are the motions they need to do. Cooks. Any of you who cook knows you have certain knives that work well with your hands, and others that don’t. Resistance and dance is gravity. And your own body, what it can do. will it stretch enough or not? Does it have the strength? So, that feedback to the body is huge.

Alan Laidlaw: I guess I put it… Sorry to reframe that, the aspect of us versus surface is what I was kind of trying to… The creation always seems to have a surface that’s separate from us. Anyway, continue.

Barbara Tversky: Yeah. I mean, I’m trying to generalize that to resistance, and feedback to the body, and the feel that it is when you’re dealing with a surface. And again, different surfaces. Just writing, paper makes a difference. Which kind of paper you’re writing on? Or doing charcoal on, or watercolours, all of that. And that, I think, it’s more than the resistance, it’s the subtlety of your hand movement, and wrist movement, and our movement on that surface, what it takes. And in calligraphy, they practice for years the strokes, and how they make them, and how they twist the brush, and the kind of paper. So, all that interaction with the medium, what it gives your hand. And artists, I worked with a bunch of artists interested in drawing, and some of them had done doctoral thesis, and one of them looked at professional artists, and accomplished artists, and novices on drawing, how much they’re looking, and how much they’re drawing, and what are the time spans of the interaction. And in artists, it’s much longer. They can look and draw a lot. And look and draw a lot. Novices are going back and forth. So, for artists the knowledge is already in their hand of how to translate what they see, this is life drawing, into their hands. And they talk about it as a conversation between the eye, and the hand, and the mind. And if you try to get them to talk words at the same time, they can’t do it. The words get in the way. It’s a visual, spatial, motor conversation that the words get in the way. And architects say the same. They can talk afterward. Explaining what they were doing from a video, but while they’re doing it, they’re deeply engrossed in this feedback loop. Does that align with your experience?

Alan Laidlaw: Yeah, to play off of that, that’s actually great. And got me thinking that now we have keyboards as our main interface. Which is a sad state compared to the richness of the ideal, the nostalgia, for calligraphy and whatnot. And yet, we have translated our focus into the simple clicking of buttons at a repeated pace and moving a mouse around. We can still get to that flow state, right? Coders do it, etc. So, that gives me hope in the VR space that, even though we wouldn’t have a surface to push against to create, we would still find a way to translate it through, I guess, just mainly the feedback, and the style of feedback travesty. The style of feedback would still come through, and we would still have that feeling. I was just wondering if there was something haptic, like in the way that we have gestures. I think Darwin said that, “Every culture does this.” Some version of this to say, “I don’t know.” And if there was something about the creation of man that is pushing against something, and that equals the brain does something different then?

Barbara Tversky: Sure. I mean, the feedback, and the kind of feedback, and the mode of interactivity, and some of that, I mean, VR is trying to add the kind of haptics feedback. and you certainly need it for surgery. And the VR surgery does try to add haptics, because anything you do, as a surgeon, you’re relying on that. And anything a cook is making. And it’s how it feels, you need that feedback. And the interactivity that comes from touching and moving, you need it for taking care of babies. When you pick up a baby ant the baby is tense or relaxed, you feel it in your hands right away. So, yeah. We need that level of interactivity. Smell is another thing. When I cook I’m relying on the smells to know, I got three or four pots going and I’m relying on the smells to know, “Is this butter about to burn? So, I better lower the heat.” Or “Is the rice bubbling too much? Better lower.” I’m monitoring those activities with many senses. And some of it, we become completely unaware of. We just respond. The way walking, right? Walking or running. We’re not aware of all the movements. Or typing. Once we had to be aware, but by now we don’t, it’s automatic. And there are benefits and costs to that, as well.

Alan Laidlaw: Great, thank you. I’ll cede my time to the other questions.

Frode Hegland: Yeah, I’ll hand it over to Luc in a second. But I just wanted to say, I think that interaction was really nice to hear because, for so many decades, we have had this nonsense that interaction should be invisible. They should absolutely not be invisible, depending on when you need them. If you’re walking on the ground, as you said, even with shoes you can tell what kind of ground you’re walking on. That is really useful, especially now in winter, when it may be icy. So please, let’s highlight how we use our bodies and interactions. That was wonderful. Luc, please go ahead.

Luc Beaudoin: Hi, Barbara. I’ve got a number of background projects. They’re just background projects in spatial cognition. I’m associated with Aaron Sloman in Britain. I don’t know if you know him. He has a project on spatial cognition, the evolution of spatial cognition from an AI. Aaron Sloman, you know him? There are two Aaron Sloman, one is the psychology guy, and the other one is the philosopher.

Barbara Tversky: No, I… The psychology guy was a student. 

Luc Beaudoin: No, this one is technically a philosopher, but he is an AI person. But anyway, I’ll jump to something that’s not with Aaron’s project, but another interest of mine is mnemonics. I’ve been doing visual-spatial mnemonics myself from a scientific perspective, I miss the beginning of your talk but I take it you’ve argued for the primacy of motion. Basically, motion coming before language and evolution. And there are various arguments for that. So, that makes a lot of sense. I see, as you do, the spatial cognition, spatial and movement cognition being fundamental. So, as such, I would think that for mnemonics it would be helpful. So I, myself, when I’m memorizing lists, you know that lists are the hardest thing to memorize. But if you can turn them into a visual-spatial sequence. And I’m not a dancer, so I’m not very good at the visual-spatial motion thing. But I found that if I can use a gestural mnemonic, then I can remember these lists. So I remember, Jordan Peterson has these 12 rules in his first book and I thought, Okay. Well, how do I memorize that?” I’ll turn it into a little bit of a dance and the whole thing came out within two repetitions. It was quite powerful. But I haven’t actually delved into the science of this. But it’s something I thought, “Well, if nobody’s done this, I want to do it.” Are you aware of research on using gestures for mnemonics? For remembering? Apart from drawing, I know that there’s research on drawing, how that helps remember stuff. Actually, I’m more interested in imagined gestures, because I don’t think you need to do it. We know that in sports, athletes often will imagine themselves doing things and that helps them execute the behaviour and practice. So, there’s your question. Imagine gestural mnemonics.

Barbara Tversky: So, a visual practice, or visual-spatial practice, visual motor practice for divers, golfers, or whatever does help. It helps mainly in sequencing. It doesn’t help in the fine aspects of the motion. Real practice is better than imagining practice. But imagine practice is also effective in the absence of real practice. You can do it on the train. I remember, it has happened to me several times, on the New York subway, I see singers with scores in front of them, and they’re imagining the music. So, the part of the method you’re describing is one of the oldest in the world. It’s the method of loci, that was invented by the Greeks, Romans to remember their long orations. They would imagine themselves on a walk through the Agora marketplace and put a portion of their oration at each of those places and then imagine that. So it links things together in an organized way. You still have to form that association between the place in the marketplace, and what you want to remember. The same would be true of gestures. When I was learning Latin ages ago, there was a whole set of what essentially were cheerleader exercises for remembering “amo, amas, amat,” and you could go through it for real, or you could go through it gesturally. So, those things can work for some people, and it’s usually for meaningless information. Meaningful information it’s better to link through the meaning, but images will work. This famous mnemonist beautiful book by Luria, “The Mind of a Mnemonist,” he certainly remembered himself going through walks and placing images. Again, you could place gestures in the same way. I mean, it can all be effective, what works for one person. And people rediscover these mnemonic devices. Every 10 years, write a book, it’s a bestseller, and 10 years later, the field is ripe for it again. Diet books tend to come out a lot faster. I think more people are worried about their waistlines than their memories. But there are those advice books and they would include motion and gestures as well. We’ve done a number of studies, many on people learning complex material, like in how a car break works, or an environment. And as they’re learning, they’re reading text, they’re gesturing. And the gestures are making a model of what they’re learning. So, they’re putting down dots and lines for the descriptions of the environments. And when they go to recall, they make those same movements again. So, it’s clearly helping them recall as well. And if they gesture both at learning and at recall, they remember much better. And these are spontaneous, the people aren’t even aware, really, that they’re gesturing. We don’t tell them to gesture. The gestures come from their body. Everybody learns them in different ways. Gestures, unlike words, aren’t decomposable. And you could see that with conductors. You go and watch the same concert with different conductors. They’re gesturing very differently. The orchestra can respond in similar ways. So, that visual-spatial language of the conductor can be quite different. We went to the opera two nights ago, the guy was dancing up and down and he was a joy to watch. And there’s research showing audiences respond better to conductors who jump up and down. There’s a famous video you can find of Leonard Bernstein conducting, I think Mozart, some classical piece, with his eyebrows. He had very expressive eyebrows. Nothing but his eyebrows. Now, they were well-practiced. But (INDISTINCT) and if you want to watch a really gymnastic conductor, watch him. I haven’t seen him in years but he was a master. And there were (INDISTINCT) using the motion in very complex ways to guide the music. And it makes a huge difference.

Luc Beaudoin: Okay, can I squeeze in another question? I’ve often thought people who learn pictorial languages, or languages with calligraphy, that they would basically have better memory for concrete words, as well, because they can actually go through the gesture in their head, so it kind of adds to it. Do you know of any research on that?

Barbara Tversky: There’s research on having more than one code for memory. If you have a verbal code and a visual code, you’re going to be better at remembering something, because you have more retrieval cues. And if you add a motor code, which could be gestural, you’ll have even more. If you get too many you might get confused, and it might be hard to construct them. But having more retrieval cues for the same bit of memory does work. So, drawing, imagining what something looks like, imagining how you would interact with it, all of those things can enhance memory, and there is plenty of research on that.

Luc Beaudoin: But not specifically on people who know calligraphy, or who do calligraphy?

Barbara Tversky: Some of that is going to be content-specific. Radiologists, who are trained to look for one kind of thing, like breast cancer, might not be good at broken bones. So, some of it is going to be quite content-specifically. The particular patterns of pixels that tell you that there’s cancer, are going to be different from the particular patterns of pixels that are going to tell you it’s a break. So, the movements for calligraphy are to make characters, they aren’t to make images of people. Although, plenty of calligraphers could do both. Some of it is going to be content-specific, and some of it is going to be more general. And there you need to look at the specifics to know the answer.

Luc Beaudoin: Thank you very much. It’s a pleasure meeting you. I cited you in my 1994 thesis, I counted four times.

Barbara Tversky: Okay. Thank you, thank you.

Frode Hegland: Thanks. Thanks for that. Brendan?

Brendan Langen: Hi, Barbara. Thanks so much for the talk, this is really neat. And as a funny aside, I’ve recommended your book to pretty much all of my friends who’ve recently become parents. I think there’s so much in the first few chapters, where you just lay out how children learn, and how to create trust. You mentioned some of that. I’m really curious about how some of your findings can come to life in some of our software tools? So, there’s quite a movement going on in some of the knowledge creative tool space. You can think of things like “Notion,” or maybe “Sigma,” or even “Roam” research and other notebooks. What opportunities do you see for embodied cognition and spatial thinking in our knowledge tools?

Barbara Tversky: Oh, so many. And then, they’d be specific. But thank you for the recommendation. I keep thinking and saying to publishers, “Somebody needs to write a book for new parents, and what to watch for.” From new-borns, because until children speak, I think parents aren’t aware of the huge cognitive leaps that children are… Because they’re just too subtle. And if you learn what to look for, it adds to the already thrill of having a baby. And I don’t really have the tools and the background to do that, but other people do. Yeah. I think there are so many opportunities for adding visual-spatial and embodied, what your body is doing. I mean, gestural interfaces have already done that. They’ve ruined my thumb. And I take pity on the people that have been exercising their thumbs from very young ages, because of what’s going to happen to your thumb when you get to be my age. And voice interfaces may help them, but they have other disadvantages. And sometimes people ask me, I have worked with people in HCI, and computer graphics, in AR, VR, and I’m really enthusiastic about all those media. Some of the work we did with AR was trying to make people’s interactions within finding their way in an environment, or repairing, or assembling something, as natural as finding your keys and opening a lock. So, there were ways of guiding your body to the right place. First, by having a virtual tunnel to guide your body to the right place, and then guide your head so that your eyes are looking at the right place, and then guide your hand to where you should make the motions. And then, it becomes as natural as doing something that you’ve been doing a thousand times other than doing something new. And so, that’s one example, but I think there is a huge number, and I’m really excited about what are the things you guys can do, and how they can make them more natural and comprehensible on the input side to people. Maybe you have thoughts. Because there are specifics you’re working on.

Brendan Langen: Yeah. Well, you just kind of hit on something that might make sense. There’s been some talk in the chat about these findings for education. And I can almost imagine a crossover with a tool like “Figma,” a design tool for early-stage designers. And if you can guide them through the process, that is helping them create something that’s more stimulating or sound in its interaction design. I could see that being a huge advance. Really curious to keep seeing where this research leads. Thank you so much, I appreciate the time.

Barbara Tversky: Yeah. Probably in the late 80s, early 90s, there was a Shakespeare scholar at Stanford, who was designing something that would stage Shakespeare for students. And that was prescient but close to what you’re saying. And, yeah. I think you can go a long way. One problem is scale. And in there, maybe VR is better because you can get things at scale. I mean, same with architecture. But, yeah. Tools that can allow me to imagine things that would take forever to create. And therefore, create better. Would be phenomenal, absolutely phenomenal.

Brendan Langen: That’s really interesting. Almost like bringing along a “now” sentiment into the mix, where something that takes so long to build, is often outside of the reach of what we can comprehend. That’s really neat.

Barbara Tversky: And on education, I want to just put in a small plug for some research we did with Junior High Science Students. We had them learn molecular bonding, and then half were asked to make visual explanations, and half were asked to do the normal thing you do on a test. Make or take notes someone raised at verbal explanations. And first, we tested them after they learned it, which was several days in the classroom. And the two groups were equal, we divided them into two groups. After creating it, all the groups improved without new learning. So, the process of making an explanation consolidates the material, and makes you question, “How could this have happened for an explanation?” So, both groups do better. But the group that made the visual explanation did way better than the group that made a verbal explanation. So, this is natural for science, because science is so visual-spatial, chemical bonding. But their diagrams were so different. Some had sharks grabbing electrons. Some had stick people giving them. They were adorable. And you can do it for history, you could do it for a Shakespeare play. What are the relationships of all the characters? What happens over time? I discovered my father’s old version of “Anna Karenina” and I stole it from him many years ago. He didn’t mind. The first thing it has is the family tree. He made it to understand all the familial relations amongst the characters, and then all their nicknames. Because Russians always have tons of nicknames. So that helped me reading it, and he made this. My kids doing “Dungeons & Dragons” years ago, the first thing they did was make a map. Again, from language. And that helped them with keeping track of where they were going in the game. So, education. Yeah. Creating visual-spatial representations of women’s drawings is one form, they’re easy, they’re cheap. But doing it in a computer interface might work as well. Sometimes I ask, “What does all the technology add over pencil and paper?” And I think it’s an important question to ask.

Brendan Langen: Without a doubt. Well, thank you so much for the exploration there. 

Frode Hegland: And there I go. Muted. Thank you. Peter, please go ahead.

Peter Wasilko: Yes. Do you use any mind mapping tools? And if so, how do you approach building a mind map?

Barbara Tversky: I’m sorry, what was that? How do I put what on a map?

Frode Hegland: He asked if you use any mind mapping tools and if so, how do you go about building a mind map.

Barbara Tversky: It probably depends on the content. I mean, you’re going to start with a network of sorts. The trouble with the network is usually that the lines aren’t labelled. The relationships, you’re just labelling that there is an association between “A” and “B,” or “B” and “C.” And you probably want to do something more demanding, and specify what the relationship is, and then you can cluster things. But it really, in many ways, depends on the content. And you can see those of us who remember learning sentence diagramming, which was essentially a mind map, and I loved it. Or logic. You could visualize in one way or another. So, to some extent, it depends on the concept. But I think, just making networks, you want to go beyond that and talk about what is the nature of the lines. The representations. Are they inclusion? What are they? And then, go about grouping them perhaps, clustering them along common relations. And then you can go hierarchically like a phylogenetic tree. And even a phylogenetic tree has been the basis for a great deal of controversy in biology. Where do different creatures belong? Is there another life form? And of course, one eukaryote and whatever, it was long after I learned biology. So, that particular way of visualizing really helped. Bill Bechtel did great work in an actual laboratory, I think looking at diurnal rhythms. And they were diagramming for themselves almost every day what they were finding. How did they do uncertainty? This is a big issue and a big question. They put question marks. So, they put in relations, the best they knew, and where they didn’t know things, there were question marks that meant, “That’s an open problem, let’s look at it.” It really depends a great deal on content. But certainly, there is research showing that kind of mind mapping helps people organize their thinking, and learn, and communicate.

Frode Hegland: Thank you very much. So, I have a question. And that is based on my current passion, or what I think is a realization, but I may be wrong. I feel that, within five years, we’ll be living a lot inside VR, AR, those kinds of spaces. And that’s kind of a subset of the bigger cyberspace. But a lot of this seems to be about being disembodied, walking around with an avatar that’s like a Lego situation. I know, Brandel, I see you’re going crazy there. So, my question for you, Barbara, is: How do you see VR with full-body immersion where we really use our senses to the full, in the context, not of necessarily social interaction and gaming and play, but more in the relationship of work?

Barbara Tversky: Five years seems to me, very optimistic. Partly because people get fatigued in AR situations. I get fatigued. There is an uncertainty about moving around when you know you’re not really in that space. And so, a lot of that needs to be worked through. And like “Zoom,” they’re going to be advantages and disadvantages. And we’ll see them as we go. The… I’m blocking on his name at Stanford, the guy doing VR in social situations. There are going to be, I mean, we’re going to have to do it. There are cross-national teams doing design, and you can’t fly everybody all the time to be together. So, it’s going to happen. Yeah. Jeremy Bailenson, who’s done wonderful work on social interactions, and those might be the most important for people. If we found that the internet was used to send emails to friends, children, and other people that we love, that was an early use of a massive. They’re going to be early uses of VR to be with people we love. And “Zoom” isn’t sufficient. I still can’t have a grandchild sit on my lap and feel the closeness. But I do think they’re going to be increasing uses, they’re going to be difficulties encountered and some of them will be overcome. I doubt that we’ll all be living in the metaverse, although again, I could be wrong. You need to talk to the 20-somethings that are already playing multi-person games. And it is a bit of a drug. And Yuval Harari imagines that AI is going to replace huge numbers of humans in the way that, the rest of us who are useless will exist as in this metaverse where we’ll be, and it sounds a little bit, to me, when those people talk about it, like somebody’s conception of heaven. You can have avatars of all the people you love. But then your interaction with them might not be taking place in their metaverse. How do you reconcile them? What age will they be? So, there are all kinds of cognitive and engineering ideas that need to be worked out.

Frode Hegland: I’m not going to let you get away that easily, Barbara. And first of all, Brandel is up after me, and he has an extensive, deep understanding of a lot of this. But let’s forget about the “Oculus” and that kind of current stuff. And let’s forget about timeline. Let’s say that we have a future where we can, like the “Holodeck” in “Star Trek,” we can go into it, whether we’re wearing something or not, this is very secondary. But there are two things that we can change. The external stuff, the environment, and the things we interact with. But also ourselves. So, even though we do take advantage of all this VR, with our movable hands, a movable head, and all of that good stuff. With your deep knowledge of the human body and the human mind, and completely free of technical constraints, being completely fantasy, what kind of situations, or opportunities, or issues do you see for how we work together on important problems?

Barbara Tversky: First you talked about the individual, then interpersonal. As an individual, I could imagine situations, interactions, environments, objects I’m trying to create. I can imagine them. But until I put them in the world in some way, my imagination isn’t complete. And this is why designers draw. They can’t hold the whole thing in their head. So, they put it down with tokens or a VR in the world. And that gives you feedback. It makes you see things. It expands the mind in ways that your mind can’t do. So, that power of technology is awesome as ways of expanding the mind, so that I can create better fiction, better buildings, better interactions with people. I can imagine role-play. So taking the things that we already use for augmenting our imagination, like role-playing, like creating prototypes, scripts, stage designs, whatever it is, and turning them into technology, and making it easy to do those things, and explore them, could be awesome. In molecules, combining them in just games. A deep mind has changed the game of Chess and the game of Go. People are now interacting with those machines, studying the games that AlphaGo can do. So, I think that is mind-blowing, absolutely mind-blowing. The social interactions, I don’t know how much we want to replace them. Now, there are times when I wish I had interacted with somebody differently. But I can’t redo it. I can redo it in my mind, but I can’t redo it for real. So, the social interactions, it seems to me, have to be in real-time. Space, we can change. We can all go to Machu Picchu together. Explore it together. Enjoy it together. But we can’t replay and redesign. If I had an avatar of someone I’m interacting with, and I could interact with that avatar in different ways, and try out different things, that might help me in my interactions in the future. But I can’t replay a real interaction in the way that I can replay a fiction. So, am I getting closer to what…

Frode Hegland: It’s wonderful, and very deep what you had to say. Very unexpected, which is, of course, what I was hoping for. Thank you very much. Brandel?

Brandel Zachernuk: I’m trying to decide which of the two questions I want to ask. I’d love to get you to go to both but I’ll start with just one. Have you done any work on the cognitive differences between writing script with a pen, versus typing, versus dictation for the purpose of producing text? What sort of internal cognitive impact there is in any distinctions that you would draw? Or do you see them as equivalent?

Barbara Tversky: Again, I would think it would depend on the person’s adeptness with each of those and the content. One of my former students, Danny Oppenheimer, who does very innovative research, tried to show that taking notes in classes with a computer wasn’t worse than writing. And the work didn’t replicate. Unfortunately, that happens to a great deal of our research, and I think the failure to replicate means, probably, it works sometimes for some people, and it isn’t a general phenomenon. But what I thought, at the time, is when you write it takes more time, so it makes you summarize. And when you type, the temptation to type down words in a row way is probably not the best way of learning. You want to wait, summarize, write down little telegraphic notes. And the other thing that writing allows you to do is array them in space conceptually. In that sense, I think that could help, but it depends, really, on what you want to learn. So, as a learning tool, the only research I know of is Danny Oppenheimer’s, and he did find writing was better than typing on a computer. And there, I think, it really does have to do with how you attend to the lecture. But that work didn’t quite replicate. But I have a feeling that those… I’m now in an ed school, I was in a psych department where you try to get the minimal features that are accounting for something, and in ed school, you throw the whole kitchen sink at something and you don’t care about what works. But nevertheless, people are asked, Are animations good? Is writing good versus typing? And people want a blanket answer, and then we say, “It depends.” And people don’t like that answer. But I’m afraid that is probably closer to the truth. I mean, we’re living in a Covid world now, and it’s how do you give advice, and when the target keeps changing, and the disease keeps changing, and people are left with the old ones, and then complaining they can’t give coherent, clear advice. So then, they toss everything out. Which is the wrong thing too, because there is good advice, it just keeps changing.

Brandel Zachernuk: Douglas Engelbart had a famous thought experiment of attaching a pencil to a brick and calling that a “de-augmentation” because of how much more difficult it would be to write with a whole brick on a pencil. But it occurs to me that, while it would be definitely slower, the words that you would tend to write, as a consequence, would be significantly more momentous and important for you. Only because you remember the effort that would be expended in it.

Barbara Tversky: Right. Any learning method depends on that. How much are you putting into it to learn it? And you’re going to put different things in depending on how you’re going to be tested. How you’re going to use the information? How you’re going to retrieve it? So, you want your encoding to anticipate your retrieval. What information are you going to need and when? And that’s a more subtle set of considerations. I’m afraid I’m exhausting people.

Frode Hegland: Quite the opposite. I have two questions. But first, I’d like to ask, we have a few new people here today, Karin and Lorenzo. Have you got any questions or comments?

Karin Hibma: I am just typing my goodbye now. This was brilliant, Barbara. Thank you so much. And thank you for the invite, Frode. I am a name or a language creator, and I’m always thinking forward. So, it really helps me to understand the antecedence of these kinds of understandings. And I love the aspect of mapping as a place locator for putting words together. And thank you. I am still absorbing. So, really brilliant. 

Frode Hegland: Karin, you said you are a language creator. First of all, I obviously pronounce your name completely wrong. What is your preferred way of saying your first name?

Karin Hibma: I’m Karin Hibma. People get the Himba, and there’s a tribe in Africa. But that’s not me, as you can. Hibma is a region in Northern Netherlands, a lot of last names with “ma’s” in then. I think probably means “by the ocean,” “by the sea.” But everything in the Netherlands is. I’m responsible, with my husband who’s deceased now, for naming “Kindle” and “TiVo” and a few other little things in the world. And I work with companies doing strategic identities. So, a lot of times we’re either creating names for new products or helping them define their language and their story, to get from where they are, to where they want to be. Which, of course, goes with (INDISTINCT) and the wonderful concepts you’ve done. So, I don’t have your book, but I’m certainly going to be getting it and studying it to cover the cover. And the “Babies Build Toddler’s” book that I mentioned is really brilliant. It’s a Montessori method, but very often, as I think Brendan said, “New parents don’t really understand the math.” I mean, they’re suddenly given this human being, which we don’t realize is going to come to its full awareness over a period of 25 years. And really being able to have some kind of guide rails for parents to be able to actualize that, is pretty wonderful. So, thank you.

Frode Hegland: Karin, I have to ask you with that amazing background, if you would like to consider writing a piece for The Future of Text Volume III coming out this year?

Karin Hibma: I would love to. I am the worst writer, Frode. I like to interact, but I find, sometimes, putting words down… But send me a note at karin@cronan.com.

Frode Hegland: Yeah, we met through “Twitter.” Thank you. We met through “Twitter” so we’ll continue there. But what you say there’s very interesting because Barbara was talking, just a few minutes ago, about writing in space. Yes, that’s something really worth drawing out, because, in one sense, that’s not really true, unless you’re writing on sand or a huge piece of paper. Because writing, very much, is linearizing. A sentence has to be linear to have grammar. And, of course, with software, you can write a little bit here, a little bit there. But then, at some point, you have to, and I just finished my PhD thesis, and the hardest bit was not writing, that’s easy, but kind of blocking it into a thing is impossible. So, I’m wondering if Barbara has any advice for all of us, including Karin, maybe in how to consider this? And by the way, Karin, for the book, don’t be intimidated with how you write. Please consider looking at the previous two volumes, it’s all over the place, which is a good thing. Anyway, Barbara, any thoughts on that?

Barbara Tversky: Say what that refers to again?

Frode Hegland: Yeah. What I’m referring to is, when we talk about text, there is this kind of idealized notion that you can write it down in space. But unless you’re working in a free-form mind mapping software, you’re not writing it in space as such, you’re writing it in a line. It is one single line. It happens to wrap, but it is still a linear line. And in our community here, we are trying to do many things with that. Putting it here, putting it there. I see Bob’s put his camera back on because this is, obviously, very much his field too. But from your work, and your understanding, Barbara, can you talk a little bit about, how we should be writing in space in an ideal environment?

Barbara Tversky: There’s the writing for yourself when you’re working through the ideas, and that should correspond to your ideas. Then you have to put it in a linear form for other people to understand, and organize it in a way that other people can understand it. If you want to communicate directly, like give directions for getting from my house to your house, or understanding how molecular bonding works and thereon. And there, one of the principles of InfoViz of giving a context, and then the details do go for text. And we found that a little bit in some of those experiments, where we go back and forth between a depiction and a description, that you want to give an overview, and then, fill it in in some systematic way. And the systematic way should be conformed to somebody else’s conception to make it clear. But that’s for writing clear prose. If you want to do poetry or art in drawings, then you’re free to go all over the place. And that ambiguity and openness allow many interpretations. And the ambiguity is what makes it beautiful. It’s what makes you come back to it, and come back to it. Because you see new things in the same painting or the same poem. Because you’re bringing things from you back into it and that’s a bit of the interactivity that people like and talk about in music, in art, even walking the city, you’re seeing new things, because you can’t completely structure it. And that adds. But if you really want people to grasp scientific, or historical, or arguments in law, then you have to be more systematic in getting in a way that people will understand it. And creating a context, and then relating the details back to the context it’s a general principle that goes for good writing and good diagrams at the same time. So, does that get it your question a little bit?

Frode Hegland: It really does, despite being distracted by Edgar, who just came here. Do you want to say hi?

Edgar Hegland: Hi.

Frode Hegland: So, Edgar is four and a half, and he’s learning reading and writing in school. And to watch that process is endlessly fascinating. It’s exciting.

Barbara Tversky: Yeah. Endlessly fascinating. When you think about it, reading is a cultural artifact. Cultural inventive. And one interesting fact in the brain and letters is, many letters, say in English, a small “B” and a small “D” are distinguished by their mirror images. And the visual cortex for recognizing figures, objects, whatever object like things, has many different parts to it that do slightly different computations. There’s only one tiny area that is receptive to mirror images. Otherwise, the visual cortex ignores mirror images. So, flipping faces doesn’t matter, same person. And for many objects, that’s true. Letters depend on which way they’re facing. And every culture, even cultures that read idiographic languages, like Chinese, and Japanese, use that same area of the brain to read. The one that distinguishes mirror images. And on branding which, Karin talked about earlier, we have icons. Do you want them symmetric? Not symmetric? I mean, they become extremely recognizable. Fonts become extremely recognizable. Letters are harder to discriminate. But, as anyone learning a new script knows, they can be hard to discriminate. But ideographic letters, faces were graded at millions of them. Millions may be an exaggeration, but thousands, certainly.

Frode Hegland: Thank you very much. Lorenzo?

Lorenzo Bianchi: My question has been partially answered. It was about writing in space. Because, it occurred to me, when I was learning Mandarin, so Chinese characters, what happened to me is that, even if I was using an App like “Skritter,” where you can actually trace the character with your fingers, I noticed that the movement, the range of motion wasn’t ample enough. So, I started experimenting and I noticed that, if I increased the range of motion if I started to use my whole body, instead to trace the person, the character of a person, I started to do something like that. It was incredibly more effective. But just for me. I don’t have any more data about that. So it was that curiosity. Because I’m a student of cognitive linguistics. I have an interest in body cognition. And I noticed that. And instead of reading and writing the characters, I was just actually living the characters with my whole body. It was incredibly more effective.

Barbara Tversky: Very interesting. And you know, the great calligraphers use their whole body. And it’s the motions and not what they see. It’s really the motions they practice, like the piano. And they are large motions. I don’t know quite what would happen to them, or anyone, when they get to be small hand motions instead of the whole shoulder and upper body. And it would be interesting to look at that. And if you ever get to Xi’an, which I highly recommend, there’s a calligraphy museum that has blocks of granite with calligraphy, mostly ancient. And they are just stunning. Stunningly beautiful. Without knowing you or someone that knows the characters, they will appreciate it much more. And from my understanding, people who look at calligraphy make the body motions. Miniatures of them, this is the mirroring. The mirror motor idea. So, when they see the calligraphy, there are feeling in their bodies, the motions that it would take to make them. And then your pleasure is enhanced. The same thing happens with dancers. When ballet dancers watch ballet, their motor cortex is more alive than when they’re watching capoeira. And the opposite happens to capoeira dancers. But when you know the motions well, your motor cortex is activated just from the visual motion. There’s more to say on that, and there’s a bit in my book on recognizing. If there’s time I can tell that story about the point life. But I see there’s, at least, one more question.

Frode Hegland: There is another question. But before Brandel’s question, Barbara, please, actually say this story. I’d rather go over a few minutes than lose out on something.

Barbara Tversky: Do you want me to do that before the question or after?

Frode Hegland: Yes. While it’s still fresh. I hope Brandel has written his down.

Barbara Tversky: This is a former Stanford student who did a rather brilliant work, Maggie Shafar. There was a technique that was invented by a Swede, Johansson, in the 70s, of dressing people in black, and putting lights on their joints. So then, when you take videos of the people, all you see are the joints moving. And if you look at a static display, you can find this on the web, on “YouTube,” point light. And if you look at static people you can’t even recognize that it’s a person. But once the person starts moving, you can see if it’s a male or a female. You can see if they’re happy or sad. You can see if they’re old or young. You can tell that from the body motion, from the pattern of lights. It only works for upright, upside down doesn’t work. Although I bet for gymnasts it would. I don’t know. But what Maggie did was take pairs of friends, have them come into the lab, and just walk, dance, run, play ping pong, all sorts of motions that they would do with the point light. And she had several pairs of friends. And then, three months later had them come back into the lab. And look at the point light and identify them as, “Are they my friend? A stranger? Or me?” So, they could identify friends better than chance. But what was most surprising is they could recognize themselves better than friends. Now, they’ve never seen themselves do these motions. Unless you’re a dancer, or a gymnast, or a tennis player you don’t watch yourself doing these motions. So, they’ve never seen themselves dancing, playing ping pong, and so forth. Yet, they could recognize themselves better than their friends whom they had seen doing these things. So, the explanation is that, watching it activates your motor system, and it feels right. It’s like trying on clothes, they fit me. So, you’re watching that dancing movement, or the ping pong movement, and it’s more effective for the more vigorous movements, than just the simple ones like walking, that you recognize yourself. Your body is resonating to what you’re seeing. And when it resonates to you it says, “Yeah, me!” So, that I think is fascinating. How much the human motor system or mirror motor system acts to understand the motion of others. And we’ve taken those ideas into understanding action, static pictures, and so forth, so we’ve taken those ideas further. But the basic phenomenon, I think, is fascinating. My guess is, with calligraphers would be a similar thing. They could see their own calligraphy. But as far as I know, no one’s done that.

Frode Hegland: Edgar just wanted to show he has a real bus ticket. He thought it was worth showing to the community today. Thank you. But I have to ask you, just really quickly. Who here has seen the movie “Hero?” The Chinese movie “Hero” with Jet Li? Oh, a good couple of hands. If you haven’t seen it, you have to see it. Randomly it was playing in Soho when it came out, many years ago. I was there with Ted Nelson and my brother said, “We have to see this.” We sat in the front row. Literally, after two minutes in the intro, they both went to the side and said, “Thank you.” It is basically about, I love “Hamilton” because it’s about American being written into existence, “Hero” is about China being written into existence. That’s the worst summary you could ever imagine. It’s the most beautiful movie. If you haven’t seen it, please do. Brandel?

Brandel Zachernuk: Thank you. So, the question is a little all over the place, but I’m really curious what will you do with it. So, first of all, it occurs to me that, I’m not sure whether it’s psychologically this is the case, but that there are sort of two motor systems in the sense of there being a gross motor system, and a fine motor system. Certainly, the way that I seem to sort of marshal my actions reflects that. So, I’m curious as to whether you have research on whether, the points of light sort of study is clearly about the gross motor system, people being able to understand the movement of large-scale kind of limb alternation I’d be curious whether that…

Frode Hegland: Is he frozen? Or is he just playing with us?

Barbara Tversky: I know. I think he’s frozen. He’s somewhere in the cyber space.

Frode Hegland: At least he’s frozen at a very engaged moment.

Barbara Tversky: Yeah, right. But I can answer the questions, sort of, anyway. And that is, I think people when they see handwriting, imagine how it would be written. At some point, many years ago, I needed to forge my husband’s signature on many documents. He was out of the country, and I needed to forge his signature. And I sort of went through the motor movements that it would take to make his signature. And he couldn’t tell the difference between mine and his. So, I don’t know of research that’s directly looked at fine motor. But my guess is that the same phenomenon would happen. I do know that when, this is again, years ago, more than 20, a friend was working on a pen whose writing could be recognized by a computer. And for English, at least, there were 13 strokes that underlay script writing in English. And with those 13 strokes, they could read handwriting, and you could pick it up with a pen by where people stopped and started. So even processes that we think of as continuous are often truncated. So, my guess is that… So, we missed you, Brandel. You froze at some point. But maybe you heard. Maybe I anticipated your question and answered it?

Brandel Zachernuk: Well, I’ll have to go back and watch the “YouTube.” But I look forward to doing so. The next part of the question that I can’t imagine you got to was, in linguistics, and in information theory, we have this concept of Levenshtein distance. The number of permutations that it requires to move from one word to another word. And to me, it occurs that the number of points of difference within a word are the things that make it differentiable and distinguishable from another word. The more different something is, the lower the amount of information required to distinguish it. In terms of action, what are your thoughts on the way that different motions are distinguishable and differentiable in terms of their cognitive impact? I’m thinking that when we use computers, it’s all the same stuff. You were just using a mouse and a keyboard in exactly the same way. So, browsing “Facebook” is the same as writing a thesis. At least in so far as the forms of the inputs. Do you see it as possible or beneficial to draw some of those activities apart from a physical perspective? Even if it results in individual input modalities being less optimal insofar as they then have the capacity to be cognitively separated?

Barbara Tversky: That’s again going to be a complicated answer, I think. And even your question about language, is that hearing or reading? The distinctions that you have to keep in mind. Because my hunch is, they might not be the same. And the Roman alphabet, with some variations, is used all over. And that’s visual discriminability. Fonts vary. Handwriting varies in what’s distinctive and what isn’t. What’s important to one language as distinguishable might not be important to another. Hearing would be something else. And their expertise is going to matter. And redundancy. One thing Tufte always recommends, he has contradictory recommendations, but he likes to eliminate chart junk. But ultimately doing that, eliminates redundancy. And we need redundancy to understand. Because we’re going to be missing things. And have redundancy is an error correction in part. On the visual side, similarly, what I need to watch a football game is minimal. What other people need to watch it is, again, going to be varied on the motor side. And same with dance, or music. I go to the opera a lot, and I love it. But my sophistication is at a kindergarten level. There are things I like and don’t like. And I rely on critics to tell me what to watch, what to attend to, to distinguish one singer’s… So, a lot of that is going to depend on my expertise. How much I can distinguish? A radiologist, we talked about that earlier, they’re going to see things in clouds or in points on an image that the rest of us won’t be seeing. And you need a lot of training to see. So, I don’t know if that completely addresses your question, but.

Brandel Zachernuk: I think it’s excellent context, thank you.

Frode Hegland: Aaron, have you got any comments or questions? Nice that you’re here.

Aaron Sloman: Well, since you asked. This conversation has reminded me of a strange experience I had many years ago. I always liked music, and at one point, I did play the piano, and not very well, then I learned to play the flute somewhat better. And then, I started trying to play the string quartets with friends, using a flute to play the violin. Which didn’t work very well, but I then, thought I should learn to play the violin. And I really struggled. And I remember on one occasion when I was trying to get the kind of tone quality that I knew, my wife could get out of the violin, I couldn’t do it at all. I put it down and I started watching a television program, in which, the Israeli violinist Itzhak Perlman was playing something, and I felt as if something had changed in me. It was a very peculiar experience. And the next time I picked up my violin I could do vibrato. And I’ve never heard anybody else reporting a similar experience. And I have no idea whether any neuroscientist has any idea how that works. But it seems to be relevant to what you’ve just been talking about.

Barbara Tversky: Yeah. And I’ve had that experience as well as a small child. I skated a lot without any lessons at all, and watched people twirl, and couldn’t do it, and couldn’t do it. And then I learned what you need to do, and it was a state change of competence. And I agree that sort of thing happens. And a good coach will often use metaphors to get you to do that. Telling you, for a tennis serve, how to hold the racket and how to swing. You have to have a metaphor for it. And the right coach, or right music teacher, or even the right artist, the art teacher will give you the right metaphors to set you up to do the set of actions properly. And again, it is that cycle of listening, and doing, and listening, and doing that I talked about earlier with the artist. That is a conversation of the eye, and the hand, and the page. So, for music, it would be your ears and your hands. And that cycle. And then, you could have, all of a sudden, this insight that you often can’t articulate. That changes the whole frame of reference.

Aaron Sloman: I felt it was not my eyes and hand, but some deep ancient part of my brain that I hadn’t been using, suddenly got turned on by watching paramount in a way that I don’t think anything else could have changed me, not in that space of time. It was a matter of just seconds and then I felt different, and the next time I picked up the violin, I knew I was different.

Barbara Tversky: Well, presumably you saw his arms hands bowing, or?

Aaron Sloman: Yes, I saw something. It was very abstract. I mean I could try to imitate the hands and I wouldn’t be able to do that. But there was something else about both, what he was doing, and also the sounds that were coming out, which together, drove something in me. But I may just have misremembered, or misdescribed, and I’ve never had any other experience like it.

Barbara Tversky: You know what I have, and some of how you learn a new language, and how to pronounce words, “R’s” are always a problem in different languages and all of a sudden getting the insight in how to make that sound that you’ve been hearing. And I’m not an adept linguist at all, but there, when I go to a country where, at least once I knew the language, I just listen to it. I’ll turn on the radio and just listen to the sounds and that helps me go back to that way, “maybe I can do it,” to make it sound that way. And there I think some of it is the motor resonance. From the seeing or the hearing, it transforms into motions of your body, in one way or another. But you’re absolutely right. It needs to be studied. It really needs to be studied. Yeah.

Aaron Sloman: And it has to make a permanent change in the brain. What that change is? I don’t know.

Barbara Tversky: Yeah, I wonder if you go back to the violin. I go back to try gymnastics. That was effortless when I was a kid. The muscles aren’t strong anymore. The joints don’t work. Better not.

Aaron Sloman: Semi-permanent, I should have said.

Frode Hegland: So, Aaron. I just did the thing of looking you up on “Wikipedia.” So, obviously from your voice, it’s easy to tell that you’re from the same island where we’re sitting. I’m in Wimbledon. And I’m wondering, first of all, how you came across our presentation today, our meeting? And also, if you might have perspectives around the notion of The Future of Text, which is tangentially and deeply what Barbara has been talking about today?

Aaron Sloman: I’m in Birmingham, in the United Kingdom. I was born in Southern Africa, in a little town called Kwekwe, in what was then Southern Rhodesia. And then I had a lot of my education in Cape Town, because my parents were misinformed by a teacher. They persuaded my parents that I’d get a better education in South Africa than I would in Rhodesia. I later discovered, when I had fellow students who’d done their A levels in Rhodesia, that they knew all sorts of things and had competencies that I didn’t. So, it was a struggle to catch up with them. But anyway. So, I had a collection of different backgrounds. I came to the UK in 1957. I was going to do mathematics, but I had got interested in philosophy, and then I discovered that most philosophers said things about mathematics that I thought was wrong. I thought wrong and I read that Immanuel Kant said something that I thought was right. So, I switched to philosophy to defend Kant. And I’m still trying to defend what Kant was saying in 1781 or thereabouts about the nature of mathematical discovery, which has to do with being able to see possibilities and impossibilities in structures and processes. Which is totally different from what’s currently going on in AI systems with neural nets. Where they collect lots of statistics, and then, derive probabilities. And you can never get an impossibility out of that. You can just get more probabilities. So, you’re asking me to say something about where I’m coming from, and what I’m doing, and that gives you some of a feel for it. And I now feel that there’s a whole lot going on in different disciplines, in various branches of biochemistry, microbiology, and developmental biology, which I’m trying to put together in my head in a way that will enable me to explain, first of all, how something in an egg can produce a bird that has all sorts of competences that it hasn’t learned? Like they can go and pick for food and then paddle in the water and other things. But not only birds but there are also all kinds of things that go on in eggs of different sorts, which produce different sorts of competencies. So I’m trying to see if I can assemble enough information from different sources to explain how that works. Because, at the moment, I don’t think anybody knows it. I don’t think anybody understands it. I don’t think I will be able to explain it. But I might inspire some of the very bright younger people, who are working in different sub-fields, to talk to each other, and come up with the new senses as they’ll answer my questions. That’s what I’m hoping for. Sorry, that goes a long way. Well, it’s partly related to this because I thought there might be something relevant in this. But I couldn’t get here in time. But at the end, I think, what you were talking about is relevant.

Frode Hegland: Yeah. So, thank you, Aaron, very happy to have you here. So, this talk will, of course, go up on “YouTube,” depending on my Wimbledon internet access speed. And we will also have a fellow do the transcript. A human, who is very good. He’ll make sure he gets our names and all that good stuff. Barbara, do I also have your permission to do screenshots of your slides interspersed in the transcripts?

Barbara Tversky: Yeah, it’s okay. My caveat is, I’ve been swiping slides from all kinds of sources for 25 years and I no longer know even where I’ve swiped them from. And I worry about that. I obviously don’t have copyright. And my understanding is, it’s okay to post things that have no copyright. But I’m not absolutely sure. So, that’s my only concern. And that said, there are plenty of “YouTube” recordings of my slides in different situations.

Frode Hegland: Yeah, no. That sounds fine. And that’s an interesting question. I mean, the journal we publish is non-profit, and all of that good stuff, or completely open access. So, if someone has a problem with it, that’s not a problem. We take it back. So, thank you for that.

Barbara Tversky: Yeah, I know. When I wrote the book, I had about four times more images than my publishers would let me use. So many I got Wiki creative comments. But even then, there were doubts and so forth. And I was dismayed when the Metropolitan and other museums released all their images without any demand to copyright, only a tribute or no payments. And that was too late for me because I wanted to use, instead of quotes, I wanted a depiction at each chapter. I’m glad to see, at least, some places are releasing copyright.

Frode Hegland: That’s very good. I’m just going to post them in the chat here as we wind down. futuretextlab.info, that’s where we will be putting all this data. And this is where we carry on our dialogue. Now that it’s been 2 hours and 20 minutes, which is quite poetic in terms of numbers, I’d just like to say, thank you, Barbara. Thank you, everyone, who was still here. Thank you, everyone, who was here earlier. And thank you, everyone, who will be listening in the future. And I hope we can continue the discussion. You’re all invited to our general weekly meetings, as well as of course, our forthcoming special monthly sessions. Which I hope will be even a sliver, as good as today, in order to be successful. So, thanks very much and have a wonderful weekend everyone.

Barbara Tversky: And thank you for your excellent questions and thoughts, it was a pleasure.

Frode Hegland: Yeah, it was a wonderful group. All right, take good care. Bye.