Tom Haymes: Good afternoon. You’re miming?
Frode Hegland: It’s still kind of a boxy sound, isn’t it?
Tom Haymes: Are you getting a little bit of room echo?
Frode Hegland: Let me try this microphone.
Tom Haymes: I think there’s a setting on Zoom in the audio. It’s pretty sophisticated. That will take care of room echo too, I think.
Frode Hegland: Is it better now?
Tom Haymes: A little bit. It wasn’t bad before.
Tom Haymes: Yeah. It’s great.
Frode Hegland: So I have no idea who’s going to be here today. But whoever it is, we’ll get to decide on setting some themes.
Tom Haymes: So my little experiment last week during the session led me down a tremendous rabbit hole, which I’m still in. I was hoping to have something to show today, but unfortunately I just couldn’t get through it fast enough. But I’ve basically figured out how to photograph my entire bookshelves, converted that into texts, a bibliography with a summary of each book, and added to it. And then I’ve also downloaded my entire Kindle library — not the physical library or the actual library, but the titles — and for that as well. Apparently, if you do that, it also downloads everything you’ve ever bought from Amazon in the book category, which is kind of interesting. So what I’m trying to do right now is to clean it up and mash it up, which I’m using Claude for, but it’s something like 650 titles. And actually I want to add those — those are just books. I also want to add a bunch of articles that I’ve got in PDF form. My idea is that once I get all that put together, about 1,000 pieces, I’m going to drop that into Author and maybe see if I can get it to make connections for me. And I’m working on a book. So I’m trying to pull stuff together for that.
Tom Haymes: And I’ve got a book proposal. So I’m like, which of these books do I need to put aside? Or what do I need to pay attention to from this book if I’m going to write the book that’s in my book proposal? So hopefully by next week I’ll get it to a point where it’s actually something worth showing. But right now it’s just a very long 90,000 word Word doc with some redundancy and some prompting stuck in there, because I was literally just cutting and pasting because it would only do about 20 titles at a time before it would choke. It would summarize, etc. So I had to talk to it to say, how do I get you to stop making mistakes? And it said, this is how I have to do it. So I built it in Gemini. Now I took it to Claude to see if I can get it to organize it, because Gemini seems to have a problem with that. But if somebody wants to try Claude as the first step, that might be fun and interesting to know. But I’m on a hiring committee too, so my last week was eaten up reading resumes, which was painful.
Frode Hegland: Hi, Frank. Hi, Peter.
Tom Haymes: Hi, Brandon.
Frode Hegland: Tom was just talking about taking a picture of a bookshelf, finding the books, summarizing them, analyzing them. So that’s very cool. I’m just going to put another link in for today’s session. Today is a little bit open-ended. I thought it would be nice if we talk a little bit more about what topics, but quite a few people have actually emailed me saying they can’t be here today. So that’s not very useful in that context. But it does mean we get to be bossy. And as Brendan knows very well, right now is a bit of an anxious time for me. I’m waiting for certain people to get back to me, so it’s both exciting and a little bit WTF involved. So we’ll see. But starting with MIT, which didn’t really go anywhere. It did make work happen and further work has happened now. So as we normally do, we’ll do a bit of demo in the beginning. What I suggest we do is — so I don’t just selfishly take too much time — I’m going to give you a link if I can ask you to skim the very top. Actually, I will do a screen share, walk you through briefly, and then if you watch the videos on your own, it’ll make much more sense. They’re just one-minute videos. Because it’s you.
Frode Hegland: I think it’s appropriate to just move in the chat window. Start with the research page, which is all about spatial thinking. So it’s about external and internal mental maps, how you bring them together to help people think and learn. It has a whole list of questions here at the bottom. If we go back to the product page, this is something that’s really come out of work recently. I saw this really interesting quote — believe it or not, it was from Elon Musk — and he said to understand the universe, you have to explore the universe. And I put that into knowledge terms: to understand the knowledge, you have to explore the knowledge. That means interact with it, build connections. And that maps completely straight on to how long-term memory is made in the brain, which is really, really interesting. The video is about it now. This is the bit that Tom’s going to be doing — taking whatever notes you have, turning it into notes and then looking at the concept. So please click through on your own devices and watch the first video and then the second, if you haven’t already.
Tom Haymes: That looks cool.
Frode Hegland: Thank you. What do you think of the music? I’m trying to gain perspectives rather than having just a fake voice or my horrible voice.
Tom Haymes: Initially it was a little distracting, but eventually it faded into the background as a thing.
Frode Hegland: I’ll do a share.
Peter Dimitrious: Definitely lower the volume of the music just a little bit so the voice takes precedence. But it helped flow. Made it more interesting — made you say, what’s going on here?
Frode Hegland: Thank you.
Peter Dimitrious: I’ll keep it a little bit over having the computer-generated old guy voice, or whatever voice.
Tom Haymes: Did you generate the lyrics and the music separately or together?
Frode Hegland: I put the lyrics in and the music below. It was actually quite hard work this time because it needed to match and line up. And that’s not a natural thing, but it was hard work in a good sense. It was fun.
Tom Haymes: But the same kind of problems I’m having.
Frode Hegland: Yeah, exactly. But while we’re talking, I just want to show you guys. So this is the discussion around our main topic, the whole mapping of knowledge. I told Claude right before our meeting, just make a long document out of it like it’s a student note. So now I’m going to ask AI down here and do define concepts. This is a test version. So let me just go back and ask AI to find concepts. So what it does now — this is so important. I had gym this morning and my trainer was talking about vibe coding and all of that stuff. She made this kind of statement that soon everybody will be able to make their own programs, which is in a way offensive, but also in a way true. A lot of apps and things can be vibe coded — horrible term — and it’s just a reality, not solid production-level tools for big things. But it made me realize that what we talk about in terms of data always needing to be clear is even more important with vibe-coded apps, because they need to make sure they have the right information.
Tom Haymes: One thing that I think people tend to overlook about that whole thing is that it’s only going to get better. And the vibe coding is going to become more precise and have less errors. Claude‘s already pretty amazing.
Frode Hegland: I just wanted to show — I think this is what I showed last week. I’ll click in the background while you talk.
Tom Haymes: When it comes to coding and programs and stuff, it’s not the color of the paint that matters. It’s the skill of the artist. A good program is going to be good conceptually on top and also structurally, and you kind of have to force the AI to do that most of the time. I think we’re going to end up with a very sophisticated vibe coding, but you’re not going to replace the human, you’re just going to augment, assuming you can do that higher-level work.
Frode Hegland: I hear what you’re saying. You’ll still need to ask the right questions.
Frode Hegland: So just briefly because it’s been updated — so that document that was not generated. I go to Author and I’m going to make sure it’s very clean. Here it is. Here’s the document as Author made it, and I’ve made a different color map. If I now select a person like before. Do the lines show up properly?
Brandel Zachernuk: Yeah, we can see them. It’s a little harder to read them because their Luma is in the middle of the scene, but given that you have them in stereo, it’s going to be a lot easier to discern them.
Frode Hegland: Exactly. That’s exactly the whole point. So the new layout will distribute this horizontally. Here it is. I’ll put them up here. So a lot of these gestures in the layout of this — I’ve even started arguing with Claude where to put these, which is bizarre. So let’s select “augmenting human intellect” — wasn’t anything there, which is wild. Upper Vision Pro, one item selected, do the focus thing. That’s only these two. So now I can move around with them. It just works very smoothly now. If I now select Allen Aman.k and do equal — so we’re working on the logic of what it’s like to see the equal sign, that means select everything that’s the same category. That’s a very quick way of selecting all people. So if I now go to visionOS, first deselect, then do equal, you’ll find all the products. This is so super important because if you can’t do that, it just gets really, really messy.
Frode Hegland: I’ll do persons again. We can do align in depth — which doesn’t always work that well. Distribute in depth. So all of these things are just more solid. And as you play with it, you now have things like: you select something, you have edit, hide — and like, it is now a button that used to be only a command. Hide stuff that you don’t want is really, really important. If we do this one and press — only have that selected — this means don’t select what’s selected, but what it points to. Selection goes from there to there. And I can do that again and it can be quite useful. If you now do that and then do focus, only what is related to the current topic is available. It happens to be me. That’s quite fun. Any questions before I go out of this?
Brandel Zachernuk: It’s interesting. This is definitely supportive of a deliberative process in terms of what somebody does in order to go about thinking about something. But what I’m less sure about, in terms of the specificities of that deliberative process — do you feel like you have a clear enough value system that if people don’t have one in mind, this will sort of direct them into a way of doing thinking with this space? Does the question make sense?
Frode Hegland: I think that is an absolutely valid and difficult and annoying question. That’s very important. And I honestly think there’s a Steve Jobs quote that I every once in a while have to search for, where he said it’s one thing having a product idea or making a product — most of the work is the polish. What I mean here is I’m currently working on making it so that — like today, hopefully I seemed a bit smoother and happier, less mistakes. It’s like the bar when I collapse it: there are two bars, one that is bright outline map, the other one has the map things. They too easily overlap — nothing I can do with that, it’s an OS issue. So then I have an issue of where should the darn buttons be. So in answer to your question, Brandel, until I’m at a point where I know people — they should be over there, backwards up, buildings should be alphabetical over here, can move and see — once I can do it like I’m talking like that, then maybe I can address your question.
Brandel Zachernuk: I see what you’re saying. You feel like there’s a necessary fluidity in order to imply the sort of value system that supports people being able to step into it and understand it. That makes sense. I think about that a lot in the context of — I’ve been talking to some people who know Nonny de la Peña, who’s done some amazing VR filmmaking and journalism. And you contrast that with the stuff that comes out of Jeremy Bailenson‘s lab, which is equally brilliant, but just not polished enough to really resonate with enough people. I think a lot about the psychological impact of resolution — i.e., things being resolved. In VR, it’s an exceptionally difficult thing for people to grasp that stuff has to be good enough for people to see themselves within it. So I appreciate that as an answer.
Frode Hegland: And Brandel, I’m very, very grateful for this part of the conversation because almost nobody in the whole wide world outside of our core community asks these questions, as far as I can find. I know it might be picky, and some people are interested in fonts and other details, but it’s taken me tearing out a lot of hair this weekend to decide where the word “edit” goes in the toolbar.
Brandel Zachernuk: Yeah.
Frode Hegland: Because I have two and a few in the middle — if it is off balance, but probably very few people will care. But once you’re in it, once you’re choosing commands, going in between and doing this and that, if it doesn’t feel right — this is where I differed with Doug Engelbart. He was very much about capability. Capability. And I said yes, 100%. But you have to do the next stage of it: it has to be the right kind of keyboard shortcut that’s either in your hand or in your head, and it has to be the same.
Brandel Zachernuk: I never met Engelbart, but my issue with that sort of stuff is that what design is standing in for in most instances is the centuries-long project of rounding off the edges through decades of practice of each individual person kind of passing it on and recognizing — you don’t need to have this thing sticking out here. Design is the process of trying to sand off those things that naturally come through. It’s the difference between natural and cultivated land. And it’s essential in order to make sure those things do happen. Engelbart’s view would work, but it would take a century to get there if you were really considerate of what it is that people really do. So I’m absolutely with you. One of the things I think about as well with the implicit function of it is — it’s been almost difficult for people to ascertain because it’s kind of a question of what’s water.
Brandel Zachernuk: But since Bravo — since there has been a point of view of what a word processor is there to do — it’s been a print preview, it’s been sort of for these things. And at least at the very outset of it, I read Almost Perfect recently as well, and it’s mostly about the business rather than the craft of WordPerfect — but there were sort of snapshots, echoes of people figuring out that a core part of the debate is: what do we want people to understand while they are working in this? What is this cultivating an appreciation of? And so I agree that polish is an enormous part of it. But you can polish something without necessarily — and that’s not what you’re doing — but those are both equally necessary parts. It’s like, what do we mean by it? If we succeed in polishing, what will they feel? What will they understand? So definitely do both.
Frode Hegland: I think that’s very much about what Tom is doing, because Tom has a very clear idea for one use and that’s great and it will work. I don’t have any issue with that. But the question is, will it work to a degree of smoothness or liquidity where it will be more useful than through other means? And right now I don’t know, because one of the first things you told us in the community was when you take something into a proper 3D space, the visual complexity is reduced by a factor of ten, something like that. So we have this huge long document and then you go into XR where it’s conceptualized. It looks like there’s so little there, and that’s obviously great, but it also gives a sense of — why bother? There isn’t so much here. So there’s a lot of back and forth with this.
Brandel Zachernuk: I would also say that I have done long-form reading — I read all of Understanding Media in headset — but it is a different thing to try to get the depth of reading that one experiences with paper within VR. While I totally agree that the spatial arrangement of things and the temporal sensitivity of things moving are almost incomparably enhanced in terms of how well we can intuit those things, I do think it would be harder for people to read long-form. So there are those trade-offs. But I’m very excited by trying to work out what people might glean from what might be the function and the intent and the purpose of manipulating and navigating documents like this.
Frode Hegland: I’m just going to show you here — I’m going to do a change in the prompt that turns these things into concepts, because that’s part of the experiments. At least 70 terms, which is an arbitrary number. And I’m going to set events. So what I’m telling you is that you really need to put quotes in there as well — quotes of interesting, important statements, which is very important. That’s it. I’ll just let that run in the background. See if we get something different. Tom, that’s how you would deal with your data as well. This is 4,300 words.
Brandel Zachernuk: Actually I was going to link something — it seems like some kind of engagement-farming post, but it is a reasonable kind of piece, about the differences between reading on phones and reading on paper, and making some assessments of the cognitive impacts of reading on phones and things like that. I actually think it is very hard to make — I haven’t found the study yet, but I intend to. They say basically that reading on paper activates different things, deepens connections, and people have different relationships with it. There are some comments in there that talk about the fact that paper acts as a storage of a number of parameters. And while I definitely agree with all of those things, there are undeniable benefits of digitalness in terms of cross-linkages and being able to be ubiquitously and immediately distributed. So rather than using this as an excuse not to advance past paper, we should be really cognizant of what it is that paper is specifically doing, what we get out of being able to navigate documents of the kind we’ve had for the last five centuries — although cheap wood-pulp paper and its ubiquity really needs to be considered quite different from what we had 300 years ago.
Brandel Zachernuk: When a book for a middle-class family could be considered quite an investment. So I think there’s a lot we need to do with the materiality of virtual information. There’s a lot we need to think about in how we might manipulate it, how we might alter our orientation and perspective on things — in the same way that you might hold a piece of paper up or turn it if you had one physically in your presence. So I think that would be a really interesting thing to hone in on as you’re using these fragments, comparing that to the navigation and manipulation of paper documents around your workspace and environment.
Frode Hegland: That was incredibly fruitful what you said, because it makes me come back to: this is not reading. I don’t fully know what I mean by that. Today I was playing around with Claude and having discussions around the word “understand” and “overstand” and etymology and all of that. Wow, this is the first time I realized that. Thank you for phrasing it in such a way. Because we did two years — one of reading, one of authoring. This is neither. The fact that you’re reading a word is in a way neither here nor there. It’s like when you look at a sign, you’re not necessarily doing graphic design understanding, you’re reading the sign.
Brandel Zachernuk: Yeah. I think that’s really apt. And to your question earlier, Tom — what is XR reading? — I think it’s neither. It’s not pejorative to say that what you’re proposing with an XR map is not reading. It’s having a tangible appreciation for the spatiality that is represented in the text. So it’s reading like a map rather than reading like a book. And it’s kind of up to us to get some meaningful topological and topographical representation that is a valid way of comprehending the text.
Tom Haymes: I just want to respond on the piece that you posted as well as what you just said. Understanding can mean very different things depending on the use case scenario. Understanding Shakespeare requires deep reading, and so that’s a very different experience. Maybe that’s something you do need to read on paper. But understanding, sucking the ideas out of something — you want to have perspective on that, and sometimes that’s very difficult to do. And in the spatial world, reading a map is a really good analogy. I’m not sure I’d want to read a book in an XR environment — I don’t see the value added necessarily.
Brandel Zachernuk: Did you see the Yung Chang Heavy Industries thing that I made with Lotus Blossom?
Tom Haymes: No.
Brandel Zachernuk: I’ll grab you the link. It’s this kinetic typography poetry group, Yung Chang Heavy Industries.
Tom Haymes: I think you did show that a few weeks ago in here.
Brandel Zachernuk: Yeah. What I love about it is the sparseness of the environment. Something I also did with Understanding Media a few years ago is you just put it in this vast kind of galaxy space — very apt for McLuhan. The Young Lotus Blossom is just on white because the original text is on white. But the abstract presence of the text as this thing with nothing else around it — I totally agree, it doesn’t have ubiquitous relevance. There’s a lot of stuff that’s about a world. But I’m also reading Kim Stanley Robinson‘s Ministry for the Future, and that’s quite a read in the middle of a historic —
Tom Haymes: I read that a few years back. The bits about executive and political dysfunction — that’s almost naive at this point. The thing I was going to say is that these uses are going to evolve over time. People who first did photography didn’t realize that it would help them understand motion, with Muybridge and so on. That was an unexpected thing. I think we’re going to discover that with XR — we’ll have some unexpected things. Because right now we’re trying to make sense of our current environment in an XR shell, dealing with things that used to be 2D and making them 3D. But then that next step — I love the poetry thing, because you can do a different kind of painting. Like with photography, you can do time exposures, which is something you can paint but people didn’t see it until somebody did it with a camera. I think we’re going to get insight in terms of the canvas of XR just from being in it. The biggest concern I have with XR in general is just the barriers to creation. I know those are going to come down, but that’s pretty intimidating.
Frode Hegland: I completely agree with you. On that last point, a couple of weeks ago when Ken Perlin said that he doesn’t work in XR — I do think that’s a huge part of it. And that’s why I’m doing this notes-to-notes thing. And that’s just the start. I changed the prompt. So now I have a few quotes — that’s very different. Not just concepts. I can choose to deal with that differently. But the prompt allows me to automatically have things assigned as product, project, person, or location. These are really, really important. So I can do global selections. And it is through this that it’s becoming more and more useful.
Frode Hegland: So right now, if I spend lots of time in here, it’s of no use to you guys, but it is very important that you can get to a point where anybody does anything like that, and then, boom, we send these configurations to each other.
Brandel Zachernuk: Yeah. It also depends very much on the source material. I was bafflingly invited to a very senior business strategy class at Apple that I’ll be going to tomorrow. I’ll be very interested in why, but it’s going to be fun. We were given a number of case studies about various geopolitical intertwined questions of how businesses respond to world events. It’s very much text, but then there are these enormous sidebars and supplementary information. I just didn’t print them out, but it would be great for them to be on pages next to each other. And there were some actual literal hyperlinks, as well as some things that were implicitly hypertextual — insofar as I don’t know much about India, and so I should know a little bit more about the separation of the different states and the economic conditions that obtain in each of them. Those kinds of things — you don’t have to think about them as VR, but it’s sort of self-evident what would be an appropriate workspace: a set of documents that live side by side in order to be able to really answer all of the questions that you might have at a given moment as you’re going through them.
Brandel Zachernuk: And in that sense, that’s an easier transition — rather than “what is the VR thing?” it’s “what is the workspace? What is the ideal set of things to have ready to hand? What are the ways to make marks, annotate, and include those things?” And if you want to take it further, a constructive way of being able to perform competence to another person through that workspace. That was really interesting to me because it was just like — we’ve been going through all of this for a number of years — but personally, as these things happen, I come across more and more scenarios where I can feel the pieces that I want around me for doing this. And it was a really interesting notion to have.
Frode Hegland: This stuff you’re seeing here is the result of a discussion with Claude around the neurological basis for long-term memory. I started thinking about it because of the book Behave that Andrea made me read. Thank you, Andrea — I highly recommend it to everybody. I’m bringing that up in context of what you’re saying, Brandel, because we have a poverty of language. I just noted it down here based on what we just said: reading and writing. Also what you’re saying, Tom — you may not necessarily want to read a book in XR. On Internet Archive you can read books by fake scrolling — it’s a good archival thing. But if anybody thought digital books would be like that, that would be the worst possible thing. So taking a book straight into XR is probably fun in several experimental ways, but it’s probably not going to be useful in most ways. So reading and writing is maybe a language of 2D, of substrate and inscription. Maybe we need to find a new language for what the heck this is.
Brandel Zachernuk: Well, I agree, and I would say it’s not strictly a function of 2D. It’s more a function of writing. It’s to do with the level of detail and the depth and the distance when you’re inscribing — either because you’re punching metal or plastic keys, or you’re actually scratching something down on a surface. One of the things that definitely means is that handwriting varies between 5 and 30 words a minute, and text writing when you’re typing varies between 30 and 90 for most people. But that’s still a relatively narrow band in comparison to the kind of textual creation entailed when people are doing things with a prompt. What is this going at — 250, 400 words a minute? This is fast. In terms of the extent to which and the manner in which we have the ability to attend to the output, it’s clearly a different thing. That’s past the speed of direct, literal thinking where you’re exclusively responsible for it. So rather than the dimensionality of it, the temporality and the quantity, I think, is a differentiator that may be worth dwelling on.
Frode Hegland: I completely agree with you. That’s why I’m doing this.
Tom Haymes: Are you familiar with flow theory?
Frode Hegland: I am. Please go on with that. But I just wanted to say my philosophy is that of liquid information, which seems very related. But there is one thing about flow that’s interesting: it kind of seems to be very good for almost rote tasks, even if they’re creative rote tasks. Because it has a certain level of detachment, it seems to remove a level of criticality. Which is why I think it’s really important to be able to go in and out of flow.
Tom Haymes: The reason I brought that up is that writing effectively does require a flow state. Flow is the tension between boredom and challenge — you have to meet the middle there. So if you’re messing with the tool, the challenge can often become too great. And there’s also the conceptual challenge of: okay, I’m here, what do I do? That also may knock you out of a flow state. You can probably reacquire that as you start to wrap your head around it, but the quicker you can get the user — especially the non-technical user, as we targeted last year with professors and academics — the quicker you can get the non-technical user to enter into a flow state with the tool, the more effective the tool is going to be. But what exactly that means in terms of what we have to figure out — what do we want them to do, what they can do — that needs working out.
Frode Hegland: This is also a really fantastic point for today because it relates to what we were talking about earlier. How should that button be here or there? For the edit function — the most recent thing I went through — should it be “edit,” should it be “E,” should it be “edited”? In terms of space, it’s important. And it became the one that is least likely to make you go home. So even if you don’t have a full flow, anything that makes you question what to do next takes you out of what you want to do next.
Brandel Zachernuk: Tom, I think that’s a really great point. One of the things I would level as an accusation with a lot of the LLM-mediated systems is that I’m intensely dubious that people are confluent enough in the output of the LLM at a given moment to be able to maintain a consistent flow state, based on how well they understand what’s going on. Every time I’ve tried to use LLMs, or follow other people doing this stuff with LLMs, it seems like they are spewing so much more information than could provide any kind of cognitive continuity to what it is that they are being able to surf across. I agree that there’s this sort of aspect of boredom to it, but I think that has to be underpinned by just an arch familiarity with the scope and the domain of the things they are likely to encounter. And the moment those things become surprising enough, they have to, by necessity, be pulled out of that flow state in order to properly apprehend what’s really going on. So either they’re not really paying attention, or they’re not getting a lot of information. Because every time I’ve seen this stuff, LLMs produce so much — how can somebody possibly process things in that way? So I think an LLM-mediated interface is something that is possible, but we need to figure out what it is best to output, because it might be that it just needs to output a lot less.
Brandel Zachernuk: We need to slow things down and meter it properly so that people can ratchet it up as slowly as necessary in order to properly anticipate those things. I’m not the fastest reader — my wife is faster — but I’m fast enough and I’m just completely snowed under by LLM output. So I am phenomenally skeptical that most people have the level of confidence with this sort of output and the rate at which it occurs to be able to maintain a flow state.
Tom Haymes: I will say I have definitely gotten into flow states with the LLM as I’m working problems. I think part of the trick is you don’t read everything. You skim through it, you get the key points. If you miss something, you can always go back and say, can you tell me what this is about again? So it’s like talking to a speed talker. But it’s also text, so you can apprehend that and drop it into other documents. So if I’m doing something I want to keep track of, sometimes I’ll just copy and paste all of the queries and responses straight into a Google doc, to have it in a different format.
Brandel Zachernuk: Tom, if you’re just skimming it, what is the rest of the text for? What possible function does it have if you don’t read it?
Tom Haymes: Yeah, I agree. I think one of the model evolutions is going to be to cut the fluff. I think some of them do it better than others. I can usually get Gemini to give me a pretty succinct thing, especially if I ask it to do bullet points or give me five options. If you tell it to be short to you, then it will be short to you. It depends on what your goal is and what you’re trying to use it for.
Brandel Zachernuk: It’s a relief to hear that there is at least some onus on the purveyors of these models to try to get them to exercise a little more brevity.
Tom Haymes: I’m sure that’s come up in the user testing.
Frode Hegland: I saw a quote that I shared last week — “sometimes texts should not be interactive, you should just listen.” And I really think it depends on what it is. Turns out this was a religious scholar. But sometimes you should skim and find what you want and jump on, and sometimes — no, if it’s Shakespeare or something important, print it out maybe and read it slowly. By the way, while you were talking and I know I’m sharing my screen here — based on our lack of language, I asked Claude to think of some terminology that Ted Nelson or Doug Engelbart may have come up with. These are what I came up with for thinking and working in an environment like that. Isn’t that pretty cool?
Brandel Zachernuk: Yeah. Peter, you were gesticulating. Was there something you wanted to add?
Frode Hegland: Peter, please.
Peter Dimitrious: Just talking about the fact that the skimming — you skim all this huge amount of text, it’s the same thing you do when you get a huge academic paper that has a bunch of fluff in it. But that’s on the users. Having an LLM do it to your specification is hard. It doesn’t know you. Eventually maybe there’ll be personalization and says, you’re smart on this, I can elide that information. But I think we’re surfing huge amounts of text. That’s part of our skill set coming into a more academic context, or having to consume as a consultant, having to read lots of material all the time. That was my filter.
Tom Haymes: Gemini does get to know you. And it’s actually kind of fun because the number of people in this universe that are as weird as I am is pretty small. And to have somebody to talk to who kind of gets it at that level but is different enough to give me interesting suggestions to follow through on — with Gemini, I can help it plan my day: what do I need to do first? Oh, well, your brain works this way, so you should be doing this at 8:00 and this at 10:00 and this at 2:00. I asked it this morning, am I working too hard? It said yes. Actually, it said the problem was not quantitatively that I was working too hard. It said I was switching too fast between different modes of thinking, and that was by itself exhausting. Because teaching and dealing with students, there’s an emotional mode there. Then there’s the architect mode, which is building stuff. It said: you’re just jumping back and forth between these two brains too fast, and that’s going to really make your head hurt.
Frode Hegland: So one thing I think it’s pretty clear is that at this point we are not talking about long-form reading in XR, other than for experimental reasons like Brandel does, which is cool. On Thursday I’m flying to China and then on to Japan. So I will probably be doing a little bit of reading, but that’s because the space is constrained. Also, this beautiful headset — which I used to carry around — I’ve stopped carrying it around. I keep it at home and use it here, which was a bit of a revelation because it’s not sitting in a coffee shop or something like that. It is really being home, having a safe space to walk around. It is actually the room. Here is the knowledge. The headset just gives me access. By the way, I’ve never said that — I didn’t realize it until I said it. But yeah, it really is the knowledge.
Tom Haymes: That’s good. I like it.
Frode Hegland: It’s funny — that comes from one of the songs where “you were always the knowledge.” By the way, some of the songs on some of these sessions — please listen to them. The current one for today is actually really annoyingly good. So a key issue that comes into this — I’ve brought it up many times, but it’s so important — context versus contents. You want your room to be clear and clean, but you don’t want the stuff that you already have that’s useful to be gone. That’s at least related to your bookshelf. If your bookshelf — physical or virtual or whatever — is known to the space, and if you have certain virtual objects, it should be able to relate to that. That is not straightforward.
Tom Haymes: The part I haven’t really figured out — I could tag the shelves, but if I have to go back and find the books in their physical space, and of course I’m bad about putting things back exactly where I left them. But also I want to make it clear that learning styles are pretty much crap as a theory. However, different brains do work differently in how they process information. Someone with ADHD is going to process information in a virtual space differently than someone who is very linearly focused, for instance. They’re going to have different demands of that space and be dissatisfied in different ways if those demands are not met. I don’t know if that means a mode switch or something. Like that task manager that Gemini built for me — I showed it to a friend of mine and she was like, I couldn’t possibly use that. And I said, yeah, because it’s built for me. We are entering a world of customization. We’re entering a maker world, which you said earlier. Because you can code anything, you can create anything virtually, including XR spaces. So the skill set involved in that gets away from the technical barrier to the conceptual barrier. As a teacher, my students struggle with that, and trying to teach them that is really, really hard because they’re not used to it. Other professors have kind of given up on that. It’s a very linear process for them.
Frode Hegland: That is a pain that I’m dealing with, not with students, but with almost every human being that I ever meet.
Frode Hegland: The vibe coding and all of that will happen — it’ll be better, no question. However, the conceptual level of what the tool should do is a very, very specific interest. And I do think that customization will be very, very important, but people — all of us — are lazy outside of our field of interest. So I can’t imagine many students or academics spending real time on how to develop a system. Look at how people — proper academics — deal with references. They don’t. Some use this, some use that, some use old documents. Nobody hardly ever really invests in their own workflow properly because that’s not their job. Their job is to do their job. So many things should be coded and many things should be possible to customize. For me, color is really, really important for various types of work.
Brandel Zachernuk: One thing about vibe coding as a solution there is that it might be the case that vibe coding individual customized solutions is a good scaling choice for deciding what to do. But — similar to like when we had David Chapman, the fellow who was doing the stuff at Southampton with Clarke here —
Frode Hegland: Peter Smart.
Brandel Zachernuk: Possibly. The fact of leveraging LLMs for a specific thing is really neither here nor there with regard to what it is that you’re expecting it to do. So that means that vibe coding a specific sort of task manager for you, Tom, is good that it succeeds in doing it, but the LLM involvement in it at the highest conceptual level should not be determinative of what it is. People should be able to design that. And it’s clearly beyond the scope for somebody to design 3 million versions of it. That’s where the LLM aspect of it comes in. But the thing that an LLM does for us conceptually, as we take on a subject, isn’t the thing — it’s a solution to a particular friction about the thing. So we should be able to come up with things to do without having to invoke an LLM in order to do it. And then talk about, what happens once we have it? That’s one of the challenges. It’s almost identical to a lot of the triumphalism of brain-computer interface. People say, oh, BCI will do it. Sure. But how? What are the mechanics of people’s understanding of stimulus, interpreting it, and outputting results? And if we don’t have an answer to that, neither LLM nor BCI is really going to be of any assistance to us until we understand that what we are doing is framing information for the purposes of comprehension.
Brandel Zachernuk: In that context, your task manager was bashed together, and it should be clear that a person should also have been able to do that. And as an exercise, in the event that nobody knows how to do it, it’s probably likely that a person with the appropriate resources should be able to do a better job of coming up with what that kind of thing looks like. So I’m absolutely sympathetic to the role that LLMs ought to play in people’s information management context. But I don’t think that it’s always going to be the best first port of call for understanding the shape of an ideal information space. We should be able to reason about it without having to take that step because we should have the highest-level ability to transmute information from one form into another. Hopefully that makes sense.
Tom Haymes: I agree. One thing I’ve certainly been thinking about lately is: to what extent am I — are we all, especially me — in a transitional state as far as thinking about our interactions and relationships with the LLM? Right now I’m just going, well, let’s see if it’ll do this, let’s see if it’ll do that. What is that doing to my brain? I don’t know. But I’m also already a fairly advanced thinker in the sense that I’ve got these skills from college and years of experience in terms of critical thinking, troubleshooting — to be able to critically view the output of the LLM. For instance, on the task manager, I’m still iterating it. Every few days I can run into something and I’m like, okay, this doesn’t really check that box for me, let’s figure out how to change it. That’s an ongoing process of iteration, which is something we have to teach.
Tom Haymes: It’s something we technologists know because we’ve been iterating for 40 years, but most people don’t code. Most people don’t know how to design or build stuff in maker spaces. What we’re talking about is democratizing design to such an extent that — the limiter before was you had to be able to use some sort of design software, whether it was Illustrator or InDesign or AutoCAD or whatever you were trying to do. That was a fairly high technical barrier for most people. If we start democratizing that, then the design part becomes more important and the iteration part becomes more important. So I agree that we need both: where an LLM or an AI has that option to help you, but you can turn it off. That’s the key — you’ve got to keep the human in the loop.
Frode Hegland: I’m just writing notes here. The whole point of the work that I’m doing is a kind of democratization for this. I do have one barrier, and that is: people need to give a damn. That is really, really important. Otherwise my tools are not for them. Within that, I do think it is really, really important to make an effort to learn the tool. Doug Engelbart talked about downhill skiing, but our job is to make sure that the skis are correct, that they fit your shoes properly, that they are not badly made skis. There’s a combination of that. One thing that really became clear for me this morning is my notion of visual metadata — which all it is, is: if the information is important to you, write out what it is. That’s it. It’s going to be more and more important when you have personalization and vibe coding and all of that. How many PDFs have we downloaded and there was no publication date? I mean, just madness. So we need to look at the infrastructures so that where the person can and wants to do their own coding, they can and get something reasonable out of it. But also as a community, we really need to try to design better. That means investigating alleyways. Playing around and doing it seriously. In the community, we’re supposed to be talking about topics for the next few months. I’m wondering if maybe one of the topics should be: what is interacting with knowledge in XR? Just that. Beyond reading and writing. Something like that. So April is coming up. Any thoughts?
Tom Haymes: What is knowledge in —
Brandel Zachernuk: I need to drop — I need to get into work. But I really enjoyed the topic so far.
Frode Hegland: I understand. Say hi to every single person at Apple for me.
Brandel Zachernuk: I’ll definitely do that.
Tom Haymes: By next week.
Frode Hegland: Next week I’ll be in China. I should be able to join the week after that in Japan — maybe a bit harder, but I’ll send a message beforehand.
Tom Haymes: So the week after that, we shift back to time-shift alignment. I don’t know whether Tokyo or China or how they’re aligned, but —
Frode Hegland: I’ll see you later. Thank you for today.
Tom Haymes: Thanks, Brandel.
Frode Hegland: I’ll send you the page afterwards.
Brandel Zachernuk: I also have dates for my New York trip. I will be there on the 19th and 20th. I’ll message — if I don’t see Ken, I will send a message to him to try to get him to talk to our group, but also go for a visit there.
Frode Hegland: I’m sure he will. That would be brilliant. And I really hope you guys can come here — 14th of September.
Brandel Zachernuk: I’ll see how things are panning out at that time.
Frode Hegland: All right. See you later, Brandel.
Tom Haymes: A conversation about what does knowledge look like in different media? We could do a week on how knowledge is different in — looking at McLuhan, hot media, cold media, linear media, two-dimensional media, and then three-dimensional media. Everything we’ve done so far. And the comment that Brandel made earlier really makes a lot of sense — every time I see media in the XR environment, I’m thinking map. I just have trouble getting past that. I also see that in a two-dimensional environment. I don’t consider, by the way, text on paper a one-directional environment. Whereas a concept map or something like that — that is two-directional, but that’s a map again. There are more options when it comes to two-dimensional media, from maps to illustrations to art. But three-dimensional media — that’s virgin territory in some ways, but so far I’ve only seen map. Is there something else? Has anyone ever seen anything else?
Frode Hegland: What do you mean in that sense?
Tom Haymes: Well, what you show with the tool — which I think is brilliant — it’s just a three-dimensional concept map. Maybe not.
Frode Hegland: It’s not — I know you’re not being derogatory, but it isn’t just that, because you can walk into it.
Tom Haymes: Yeah.
Frode Hegland: And the reason I’m being a bit defensive is only for the detail. So if you view a 3D concept map on a computer, even if it does eye tracking and it feels 3D — if you can’t walk into it, it’s a different thing. And of course, what I’m doing is very, very basic, which is why I’m trying to arrange the research to go way further. Everything is flat towards you and everything is just plain text. But even that is a different thing. And to try to tease out — and you mentioned McLuhan — it’s interesting to have him in the back of our minds for this.
Tom Haymes: I think it would be an interesting conversation. I don’t know who would be the person to talk about it, but it would be interesting to think about how three-dimensional spatiality changes how we interact with information, in the way that motion is another one — motion pictures, hot media as McLuhan described it, as another way of communicating knowledge. You can do different things with that than you can with still media or even text. So what’s the secret sauce for XR? What are we missing?
Frode Hegland: That’s a really worthwhile question. One of the songs that came out of this had quite a provocative start. It’s more fun if I show you. Can you hear this?
Tom Haymes: It keeps going in and out.
Frode Hegland: Please play it on your own. Just literally the first 30 seconds. The first sentence came out of a dialog with Claude. I just think it’s quite brilliant.
Tom Haymes: Is this the link?
Frode Hegland: Yeah, it’s all the way at the bottom of the page.
Peter Dimitrious: The voice on your audio is very —
Frode Hegland: I can hear you, Peter. The voice is very — I have a feeling I know the word you’re going to use. I think he’s listening. The voices can be a bit interesting. I just like the way she says “knowledge doesn’t move itself.” I think when you have these interactions with these different systems, it comes out in very different forms.
Tom Haymes: Can you get lyrics for these?
Frode Hegland: Yeah, I write the lyrics with Claude. I use Claude pretty much exclusively. I give it context — it takes the transcript and then some other context. Then I go through. Sometimes it’s absolute rubbish and I have to change it, which is totally fine. And then it does this. But it allows us to look at it in a different way, which is the point of art.
Tom Haymes: Absolutely. My suggestion though is maybe drop the lyrics down underneath it.
Frode Hegland: I have done that for some of the sessions. And I think it’s a good idea, but at the same time, if you have to listen — it’s another channel.
Peter Dimitrious: You’re forcing your brain to go through that audio thing, which is different. And that’s your choice as an author to figure out. You’re authoring little mini movies here.
Tom Haymes: Maybe make it a link then — say, here’s the lyrics somewhere else. Because sometimes music flows past you and you’re like, what was that? And in this case the text is very important, and trying to parse that is a bit of a struggle, with loving respect.
Frode Hegland: Yeah, that’s the point.
Frode Hegland: Are you both photographers as well? I have a feeling you are.
Tom Haymes: Yes. I’m a photographer.
Peter Dimitrious: Yeah, a lot of video, a lot of stills.
Frode Hegland: That’s good and interesting. So I only two days ago realized what kind of photographer I am. Because I’m the kind of person who will measure the micron size of a pixel in different cameras to compare. I found out, for instance, that the normal big Lumix I have — which is full frame — and there is a Ricoh APS-C, which is tiny. And because it’s a monochrome sensor, it has the same pixel size as the full frame sensor. So if I take a black and white picture with a big camera or the small camera, the light gathering is about the same, which is really fascinating. And the reason I’m telling you this in the middle of this discussion is: I care about the bokeh. I care about the blur. I care about the grain. I don’t really care so much about the subject. So I realized that what I care about is whatever the caption cannot say. If you have a caption that says “girl holding red balloon,” I don’t really care about the girl in the red balloon. I care about all the other stuff — that leads into the music side of it. I don’t want you, Tom, to get everything out of the song, because we have a full transcript and analysis. I want you to get little snippets of thoughts that, if you misunderstand them, is a good thing, because it’s just to provoke further thought. That’s all it’s meant to do.
Tom Haymes: One of the things I find interesting about the podcast function on Notebook LM is that in order to mash stuff up, sometimes it does some very interesting things where you’re like, yeah, okay, that makes sense, I’ve never thought of it that way before. I had to do my own stuff and I was like, wow, okay, you’re right, I never looked at it that way before, but I wrote it. And so looking at the transcript and what I say in this meeting, or what you say in this meeting, is also a bit disconnected — it’s not the same. It’s looking at the notes in the score in a sense, as opposed to listening to the music.
Frode Hegland: It’s really interesting. I’m wondering — if I upload a Zoom video straight up, the compression is really good. If I do any kind of editing in Final Cut, it balloons in size, which is why I really hate to top and tail these things. But don’t you think that maybe if I do a song, I should put it at the end of the video? So if someone actually bothers listening to us at the end, they get the musical version — or maybe even the beginning? What do you think?
Tom Haymes: Just drop a link to it in the description.
Frode Hegland: I have that kind of — it all links back to the page. But if you missed a meeting and you were so bored you decided to listen, wouldn’t you more likely just click on the YouTube channel for this? And then you have all these people talking, and suddenly somebody starts singing. Or is that not the right mix?
Tom Haymes: Go ahead.
Peter Dimitrious: When I’m listening to it, I’m looking for information. I want to get the information out of it. My listening-to-music brain is totally a different brain thought for me. I’ll flip out and not listen to the words — I’m listening to the tone of the voice, and I don’t really absorb what they’re saying because it’s a lot of technical content. The words are putting the notes — it’s a wonderful little phraseology. I think you’ve evolved these little demos. I think it’s really good. I think it becomes poetry as you’ve evolved it. But in terms of information transfer, it’s the emotional side — it’s disconcerting sometimes. I want to read the transcript and get the words of the lyrics because I don’t understand them. These are all knowable and listenable.
Frode Hegland: This is phenomenal because the point you’re making now is exactly the point that I’ve been working on over the whole weekend — to understand, you need to enter the knowledge. You have to have relationships. If you read a novel just straight through, you only get a surface. If you watch a movie, you understand it only to a certain level. I completely agree with you 100%, Peter. But I also think that a little snippet here and there and a misunderstanding to spark thinking is okay. So what I might experiment with from this meeting, which I’m extremely happy with, is: maybe a 30-second intro piece of music that is in super-summary of what you’re about to hear. Have our conversation. And then at the end, here’s a different way — here’s a full 3 to 4 minute song about what you just heard. Just mix it up.
Peter Dimitrious: That would be like abstract content summary or something like that. But the musical parts are good as an intro because you start to watch the video and say, this is really neat, and then you get a more instructional tone that says, this is what’s really going on, and then you can sit there at the end and digest it. One of the things in the Twitter link that was given by Brandel — with Anisha Moonka — I gotta go look him up and see what he does. About the kids and the EEG and figuring out that reading linearly is different from reading pages in a book, and ebooks.
Tom Haymes: I’m a little skeptical of dropping —
Peter Dimitrious: E-reading. That was some of the thoughts there that were interesting. I gotta dig down into that. That’s a whole world I haven’t seen — those studies or anything. So that’s useful for me personally.
Tom Haymes: So, first of all, as a photographer, I’m the opposite of you, Frode, in the sense that I look at everything technical as a distraction and a barrier to seeing the whole of the picture — the composition, feeling the picture. And also in terms of thinking about ideas, one of the things I really look for in the way my brain works is: what’s the space between the ideas? Where are those intersections? What’s not being said? And that’s one of the reasons why I’m building the Knowledge Navigator — it’s really about the spaces between ideas. You’ve got all those bubbles, and I don’t want to tell the bubbles where to go, but I want to understand how the bubbles might relate to each other without me telling it. Giving me a different perspective on things. That to me is really important. I’m looking at things holistically rather than granular.
Frode Hegland: I hear what you’re saying. I’m going to try to find a picture here that makes my point.
Peter Dimitrious: I think you’re trying to enter the knowledge — doing VR and walking around is fundamentally different from doing a 3D thing on a flat screen and scrolling into it, even though it’s 3D in some sense. You have that very different thing. So you walk in your library, in your home. The music, I think, is another attempt by you to get you to enter that knowledge in a different way. It’s interesting to see how that plays out for you.
Frode Hegland: I’m showing this picture because the texture of it to me is really, really important. If this was full color, everything sharp, there would be an entirely different photograph.
Tom Haymes: Oh, yeah.
Frode Hegland: So that’s what I’m talking about. If you were to describe this picture, it would lose everything of value in the picture. And I find that kind of a refreshing way — as we think about text, to also think about the opposite: what cannot be in the text, or should be in the text. Which then goes back and forth with the music in a way.
Tom Haymes: But how do we do that visually in a 3D environment?
Frode Hegland: I think what is really, really good about these discussions, and particularly today, is we can just throw out a lot of stuff and see what floats. Tom, you talk a lot of abstract, futuristic ideas. You also talk about the most obvious — a bookshelf. I think we need to do both of those things. We need to be very, very grounded about it, not throwing the baby out with the bathwater.
Tom Haymes: I think this is a really interesting space. This is Pope Gregory‘s Hall in the Vatican. The golden part is all pictures of heaven. And then along the side, you have all these maps of earth. It distinguishes between earth and heaven. But it’s also a three-dimensional space — you’re walking forward through it. This is a 16th-century XR environment.
Frode Hegland: No, continue. Sorry.
Tom Haymes: When I teach visualization, I like to show this picture because it’s a good example of visualization and how you can do different things with color, with shape, and with spatial organization.
Frode Hegland: I’m so grateful you’re showing this because the thing I keep harping on about — context versus contents. To take your office and turn it into virtually something like that, either ignoring the physical space or keeping with it, to be able to pull things out — whether it’s a movie or a text, it really doesn’t matter in this sense. But then when you need it, to fade the background away because it’s just distracting. It’s really important to be able to bring it back. We’ve looked at how to deal with context and foreground with limited resources. And let’s go even further back — if you look at cave paintings, the paintings were painted to fit the rock. The bison or whatever — the shape would have a little bulging on the body. And when you came in there with a flickering torch, it would be slightly animated. So I could imagine even then our ancestors could picture or feel that creature coming out of the wall. And we’re talking now about making that possible.
Tom Haymes: I’m not sure I want that. It depends on the creature.
Frode Hegland: Also, guys, if there are other people who should be in these calls, please invite them. We could do with more perspectives and more people. I’m going to work a little bit on the topics, and if you think of something offline, just send me a message and I’ll try to add to it. Because I’m beginning to feel that we are now truly scratching at the surface of developing a terminology and a language for working in a knowledge space. I know from my own writing I keep changing the terms as I go along. I think you guys have a similar issue — what are we trying to tell people we’re doing here?
Peter Dimitrious: You get Doug’s terms — those funny terms you use for Engelbart’s terms. As bizarre-sounding as they are, he tried to make them unique so they wouldn’t conflict with existing ones.
Frode Hegland: But there is a story that I think is actually true — one day Ted Nelson asked him, how do you pronounce H-LAM-T? Doug looked at him and said, you know what, I’ve never spoken it out loud. So of course, Ted, being Ted, said, how about “Hamlet?” So the branding there is really, really good.
Tom Haymes: Ted’s also one for inventing words, some of which are incredibly useful. I use “intertwingled” all the time, which makes people scratch their head and go, what the hell did you just say?
Peter Dimitrious: He had his technical definition for it. It’s like, no, no, we’re going to take the word and make it mean a lot more than what he was trying to promote.
Tom Haymes: It’s a fun time to be alive, guys.
Frode Hegland: Which is a fake Chinese curse, of course.
Tom Haymes: No, I don’t mean it as a curse. I’m talking about all of these possibilities of how we relate to each other and communicate with each other. To think that when we were kids, it was only text, pretty much. The visual was relegated to artists. For more normal people, you basically had text.
Frode Hegland: I’ve started doing a little bit more with AI image generation. I find music easier. But look at this. This is my beautiful son Edgar at a nice restaurant, playing with his drinking straw. So I just used Imagine and said, turn it into a flute.
Brandel Zachernuk: Yeah.
Frode Hegland: It added to his shirt, changed his eye expression to play flute. Everything else is the same. I mean, I could not tell that this was fake.
Tom Haymes: Is that Photoshop doing that?
Frode Hegland: No, I did that in Imagine AI.
Peter Dimitrious: There are so many of those deepfake kind of things you can do now. The best of times, worst of times kind of thing. It’s so cool. And then the other end, there’s all this bad stuff going on.
Tom Haymes: That’s always going to be the case.
Peter Dimitrious: That’s always what it is.
Frode Hegland: Now we have further repercussions — what the damage could be. And on that note, I’m going to stop recording.