Rob Swigart: Hello. Hello.
Peter Wasilko: Hello from brunch.
Frode Hegland: Was trying to do a quickie here with the First pro to see if I can enter what we do on Wednesdays would be it just didn’t work earlier. So just wanted to check with you guys are here. Enter VR.
Rob Swigart: Loading. Yeah, okay.
Frode Hegland: Working. We can look at that on one side. How is everyone? How is everyone? That wasn’t supposed to be a complicated question. I know. Audience.
Rob Swigart: I don’t know. Fabienne’s got.
Frode Hegland: A haircut. You know, there’s probably going to be like a A or a VR hairstyle soon where, you know, the lower back has to be really short. And I feel that’s probably what Fabian has done. Give me a headset cut. Exactly.
Fabien Benetou: It’s the spring cut.
Frode Hegland: Spring cut. Oh, okay. Yeah, I guess that’s a more normal way of saying it.
Rob Swigart: Hi, Rob. Hey, Frodo.
Frode Hegland: Did you see in the latest meeting reports? It’s taken into account the chat sessions.
Rob Swigart: No, I’ve been Head down in a book.
Frode Hegland: Oh, no. That’s fine. It’s just that it’s now referring to you. Much better.
Rob Swigart: Oh, good. Yeah, I tend to. Right. Yeah.
Frode Hegland: But yes. No. Exactly.
Rob Swigart: A sea of code. What’s this? It must be Peter’s in a sea of code.
Frode Hegland: In a sea of code?
Rob Swigart: Yes. He’s been head down in a sea of code. Which sounds like a lot.
Frode Hegland: Have you read the book? What’s his name? The guy who wrote Satanic Verses. He also wrote a book on the origins of stories, The Ocean of Notions. I think I would get a kick out of it. I can’t remember anyway. So, Andrew, you are here. That’s nice. I know Dean will not be able to be here very much today. And Adam is doing kids things. He’s going to try to join us more on this Wednesday and to be more involved in general. He’s coming out of high season kid event time.
Rob Swigart: It’s good.
Frode Hegland: So I sent out a call for part of our discussion today to be about talking about what the ultimate document might be. And this is obviously something we should, if you want to talk about it should be talked about along the lines of real use cases. We have three use cases from Deeney. One of them is reading in order to to write an article, we should probably also see if anybody has a more specific use case like I’m reading. And I want to learn this and I want to see this and that kind of stuff. But what’s the feeling on talking a bit, on throwing away all of history and reality and talking about ideal. Document. Grand silence. Well, the thing is, you know, we have to deal with realities. But I’ve been writing this document to the Apple Vision team on issues from a developer’s point of view. There are lots of little things that are clearly bugs. Like too easily it uses follows my eye rather than the trackpad and so on. But if we’re going to be thinking about How to really use the headset. How do you really use these extended things? It’s probably not going to be about reading and writing flat documents, right?
Peter Wasilko: I think one of the downsides of relying on eye tracking is, well, for one, my eyes tend to flit around to look at different objects with shifting focus. And you even get that when you’re reading. The way the eye scans on a page doesn’t track neatly with the linear order of text. So that’s probably causing a lot of issues there. Also. Also, I don’t think we should be limited. Your eyes can only be looking at one place at once. Now imagine you say I have two different objects that you want to manipulate relative to each other. You can make two fists, and your left fist could be controlling the location of one object in a 3D space, and your right fist could control the location of the second object. You could bring them together and move them apart and shift them around, manipulating two objects simultaneously. But again, that doesn’t work with eyes because unless you’re going like cross eye with each eye looking at something different, which I just shudder to think of what that would do to your long term. It doesn’t work.
Frode Hegland: Completely agree, both as an idea and in terms of the reality. I have been putting the Vision Pro on a few people’s heads lately and it’s been very disappointing. In one session, it just didn’t work very well that the lady, she had cataracts and that really threw off the eye tracking. My current experience of the eye tracking, even at the best of times, is not perfect. It’s very often I’m looking to the thing next to the thing. Exactly. And even worse, having developed author and actually writing this message to Apple in author in vision, even though I use a physical trackpad and keyboard very often, either the keyboard will pop up or the cursor will move to a different point because my eyes glance there rather than trackpad. So these are the very specific things I’m trying to write down and tell Apple. If there’s a trackpad attached, please try to reduce the influence of of the eyes. Absolutely. And I completely agree with you, Peter. We should be able to use both hands and not just sit there. As a lame person would have to, you know, if you can’t move your arms. Absolutely. This is good. But if you can, you should. And I see Rob moving his arm now to get my attention, and that worked.
Rob Swigart: I want to say that as you get older, your eyes are sometimes not properly aligned. I get some double vision since my cataract surgery and it makes it makes it very hard to select items on. For instance, if you’re trying to enter the. Password? The passcode. Into. In the vision I can see the top numbers and select them, but then if I try to go down to the next row it very often jumps back up. And it’s really not very useful. Yeah, I don’t know. I don’t know what they can do about that, but that’s that’s going to eliminate people like me.
Frode Hegland: Yeah. No, absolutely. And we’re.
Rob Swigart: Seniors. Yes.
Frode Hegland: It is an important issue because even me, who have good eyes, relatively speaking, it’s too often. I mean, one thing I’ve asked them for is on iOS we have the iOS menu, right, a context menu. I know you all know about it, but for the sake of illustration. Just going to show you something here. So opening a thing. Right. This is what we’ve managed to do in order for iOS. You see, the context menu covers the whole screen. So, you know, you can tell it’s not so bad. But on the vision it’s restricted to being quite small. So that means that that tiny, tiny little arrow to see more, you have to use it so often. And to get your eyes on that rather than next is really, really hard. So these are absolutely issues that they need to take into account. But hey, that’s why they have us, right?
Rob Swigart: Do they know they have us?
Frode Hegland: They do? Obviously. Fabian. Excuse me. Fabian. You’re just confusing me there. Brando. Is on the team. And he. Of course, he’s doing specific work with webXR. But also, Bruce Horn is now fully on the team. So Bruce is the one who I’m sending this list to. Okay, to kick start some discussion today, I just put a link in our chat. And I’m not saying we have to go on and on about this today, but if you could please have a look at the link.
Rob Swigart: We presume it should be available in vision.
Frode Hegland: Yeah. I mean, this is just a Hang on. Let me see if. Leon, are you properly here? I can’t, I’m not sure if you. You’re really here. Okay, cool. Fabian, you’re here, right? Can you guys have a look at the link? And so it’s just to spur some Some things. Okay, I’ll just give you the last I. Hopefully I won’t be speaking forever. Hopefully somebody will take over. But in the I’m just going to read something brief to you then one second. So this is the scenario that I hope we can build. This is what I’m telling Apple. So this is a small bit in the Apple email. It’s very relevant to us I think. But yeah, what I hope I can build as a community is something like this. You’re sitting at your desk at a coffee shop table or at your office. You’ve decided to use one of the wonderful environments native to the Vision Pro. So when you put it on, you have a beautiful vista stretching for most of your view with a table cut out so you can see your keyboard, trackpad, and of course, coffee and water. This is instead of or in addition to being able to twist the Digital Crown to indicate how much of your view should be artificial in the distance, you have a spatial video of yourself. Record you yourself recorded. Play in the frame on a loop, audio off of one of your favorite spots just because it feels nice to have it.
Frode Hegland: You then also have the option to tune in to live spatial video from stations playing in the background. But you’ve decided not to do that today because you are working. One of these can be your headquarters. So you can actually have like an ambient link to your office, and now it’s getting closer to relevance of us. On the other side of your table, you have an artificial item. Not unless, unlike quest augmentations, in this case it’s an owl and brackets. Do you like our owl? And you all better get that reference. This. This is where the synchronization between vision and Mac is not always perfect. It has lost some edits, so I apologize. Anyway, not much more to read. This is one of several trusted agents you give full access to your workspace, library, calendar, etc. so it can offer you perspectives on the work you’re doing. Since it’s on an Apple device, consumers feel safe that the data is processed on device. Sometimes we choose to have two of these agents and even three, so you can get a more balanced perspective. And since they share data, they can map out things for you. That’s cool right? Yeah. There’s a lot of stuff missing. The with the synchronizations.
Andrew Thompson: Frode. Are you trying to screen share something by any chance?
Frode Hegland: No I’m not. Thank you for checking. Okay, I’m just going to read something, but Okay. I’ll just have to do it from memory. Clearly, something’s gone with the synchronization. Anyway, so this is the kind of scenario in here that’s hopefully going to spur our conversation. The idea is that this person’s task is to write a report, having been to an academic conference. So they will have to do a little bit of writing, a little bit of speech to text, also probably video and images and some citations for papers from the conference. However, this conference was on a very special place. It was in Rome where Rob and I were, and therefore maybe you want to include some spatial videos and make it a bit more lively this document. So we could imagine that person sitting and having a map of where they were looking at, where they took certain pictures and models and so on, and they can fit it onto the map if they want, put it into their paper, do all these fun things so that when they share their documents, it can be read as a traditional flat document, and when opened up in some sort of a headset thing, can be opened for fully spatialized experience. That’s the thing. We want to make that possible, right? Can I, like get thumbs in some kind of direction? We want to be able to provide a means to make spatialized documents and not a cumbersome Macromedia Flash kind of way, but in a really engaging, more simple way. Right. Okay. Peter’s got all kinds of hands up.
Peter Wasilko: Okay. One of the things my ideal document would have are multiple, possibly different formalized blocks of content that are passable. And that just sort of goes back to the original Apple data detector concept, where it would recognize phone numbers and recognize addresses. But we can do that a lot more. Detail on a lot higher level of richness and fidelity. Now, you could imagine having island grammars, and the idea of an island grammar is a grammar that can recognize a structured glob of language in the middle of a large sea of unrelated content. So the idea is you basically assume that everything isn’t recognized. And as you’re looking at each character, you look ahead a arbitrary distance to see whether what’s immediately following the character you’re currently at is something that’s recognized by this island grammar. And if it is, it then goes and recognizes that grammatical construct. And it knows there’s some piece of formal language in here that is structured that I recognize, and it pulls that piece out as a data element, and then it looks for more stuff it doesn’t recognize until it finds another piece that matches that grammar. And that means that with island grammars, you can have multiple grammars applied to the same overall document. So we can have our generic document and then we can have our citation language looking for citation references.
Peter Wasilko: We can have a discourse representation language looking for structured breaks. You know now we’re talking about now we’re going to digress into whatever and recognize that and pull that out as data. Then you could take that data and have a stand off representation of it so that you can have overlaying things recognized. So we might have names recognized and you’d also have academic citations recognized. So that means for a given citation, you would both simultaneously be adding the names from the citation to your block of names that have been extracted from this document publishers, to a separate block of publishers. And then each one of those lists could then be flown out into 3D space and stuck on the wall somewhere where you’re interested. And that could be live so that as you’re typing or pulling in data, you’re constantly looking at it, parsing it for the different sub grammars that you can recognize and extracting data from that. So that would be a crucial element of my ultimate document. It wouldn’t just be a block of text for another human to read. It would also have software agents continuously monitoring it, pulling out things of value, possibly putting them into feeds that someone else could subscribe to.
Peter Wasilko: So then I could have like an ambient display and I could be looking for, say, just author names, and maybe I have 30 people’s activity that I’m monitoring and the brightness of. Michael Benedict shows up and it starts getting really bright and pulsing. Maybe. And I could tell me that a few other people just happened to be simultaneously looking at citations that are connected to that particular author, and that could, in an ambient way, draw my attention to, gee, maybe I want to pull out and see what he’s doing. And also I want to have fine grained privacy. Sometimes I actually want people to know exactly what I’m doing in a fine level of detail. When I’m researching a programing problem or something. I would love to have anyone in the world who’s working on a similar problem to be able to dial in to what I’m doing and merge what they’re doing with what I’m doing. Whereas other times, if I’m researching, say, a legal topic for a friend who asks me a law question, I’d want to make sure that absolutely no one knows what that is and have a nice big cone of privacy dropped over my head so that that activity would be completely opaque to the outside world and wouldn’t appear in the record anywhere, or would only appear in a secured record of, you know, my legal activity.
Peter Wasilko: And then I could maybe have a my legal document tracking all of my legal activity that only I’d be able to see, or another attorney that I’m collaborating with on a given project. So you have that fine grained security issue, and you want to be able to voluntarily expose stuff. That was something that always frustrated me with library catalogs, was how they just completely blew away the opportunity to get collaboration amongst the users of the catalog. Beautiful. Case in point. Many years ago, my library was discarding a cookbook, and it was a cookbook from the Revolutionary War period that had been in the collection for a long. I mean, it could be considered, you know, a treasured archive item. If the Smithsonian got their hands on it, they would want to catalog and add into a special exhibit someday on, you know. Domestic activity and culinary experiences at the time of the founding. But instead, the librarian explained to me that the decision was made because it involved having herbs, and one of the recipes was basically an herbalist approach for. A medically induced abortion, so they didn’t want any. I think the rabbit hole.
Frode Hegland: Is getting quite deep when we’re going into forever cookbooks.
Peter Wasilko: Yeah. So basically, if they were to cook that recipe and they were pregnant, it could cause a miscarriage. And therefore, based on that strength, they pulled the book, whereas they could have augmented the catalog with that information, made the book available with a medical warning that could be automatically generated by the system saying, hey, if you’re pregnant or want to be having a pregnancy develop in the near term, you don’t want to use, you know, the recipe on page 16. Everything else in the book is wonderful, but that recipe in page 16 could cause you serious issues. And instead, their choice is pull the book entirely or leave the book on and risk a patron having a disaster.
Frode Hegland: Yeah. Okay. So. Okay. That was a brain dump. But what I took from that one of the key things is you want to be able to have the data in the documents have. Programmatic and semantic meaning so that it can be used programmatically like this. Right?
Peter Wasilko: Yep. Exactly.
Frode Hegland: So one thing I want to ask you all in the context of the ideal document, is to look at a very, very traditional document. And that is the. Document I sent you of our record, because I think there’s a relevant point there. I will Give you a Dropbox link if you don’t already have it. That’ll be real quick. Now, the reason for this is this document of our meetings. It was very close to what we’ve been trying to do for the last few years. Thank God I got the link. And that is each meeting. Obviously it’s changed a little bit over time, but it’s not just one prompt, it’s several. So please tell me when you have it up on your screen and boring old PDF, and then I’ll tell you why it’s relevant to this. They all have it up. Okay. Right. So go to page 81, please. 13th of March. Wednesday. I talked a little bit to Adam with Adam about this earlier, before his family and children thing started this evening. You know, we talked many times about having it sort of a timeline thing. So this is how can we store it in this and how can we view it? So if we look at page 81, 13th of March, it has a link to the transcript and the video right there. Then it has a list of who it thinks was in the meeting. And it has a very brief intro summary. All that’s fine. But then on the next page we have Andrew update, which is copied and pasted for what Andrew has written, and a link to the demo itself.
Frode Hegland: Obviously only available for the Wednesdays and Wednesdays were initially supposed to be secret, not recorded. Most of them are recorded. None of this will be made public until we’ve all had a look and made sure there’s nothing in here that we feel shouldn’t be public. That includes the transcript, video, everything. So then we have what’s getting really interesting for our XR work. Speaker summary it’s per speaker. And again, this is all one prompt with all of these. And then we have topics discussed. That’s what I’m using now. But we could of course choose to ask it for other specific topics and then run it through again. We have then what’s really interesting names mentioned. So it’s most of them. And I really want to emphasize that after today’s meeting as going forward, I will try to do this as quickly as possible. Send it to you so you can have a look at how you are reflected and see if it disagrees with you, in which case I will change it immediately. You can also, if you have notes in the meeting that you want manually added. I’m very happy to do that. This doesn’t need to be pure, I write. So the name expansion is very listy, which is good. Then we have agreements, disagreements URLs and suggested next agenda. So the point is, doesn’t this become something we can consider having a really rich view in XR space for? At a minimum look, buy date or look buy names mentioned or look, buy this, that and the other. What do you think? So what I did as well, since you’re all being very quiet.
Frode Hegland: I asked. I took this final document and uploaded it to Claude Edeni. Okay. I just talking to the group about A little bit of we’re talking about a little bit of document formats and things. So I reshared our, our record because it’s so many headings. So we can maybe represent it in Excel in different ways, just as a way to talk about format. So as a provocation, I took that document back into Claude and I asked it, what is Fabienne’s perspective on webXR in relation to text? Not going to read all of it, but it’s forbidden. You are here. Tell me if it’s got it completely wrong. He emphasized the complexity and importance of responsive content and XR environments, and the need for accessible and semantically annotated context content that was March 4th. You suggest that exporting data from the Excel system to popular academic tools like Zotero and Mendeley to make the project more useful for the intended audience. March 27th, etc.. So it seems to me that we have all this data now and going forward we will. And just to repeat it for you, Danny this report, I will try to send it out as quickly after each meeting so we can all look and see if there is a mistake. And we should all feel free to add anything. Just email me and I’ll put it in. So we have this stuff and it’s categorized. So we should we should be able to have a very, very useful way to view this in XR. That’s all. Good. Thank you. Rob, please.
Rob Swigart: I’m looking at the document and I go to a date and I’m looking at stuff. I would like access to the table of contents. Otherwise I have to jump back to the beginning to select another option.
Frode Hegland: There should be a table of contents. Yeah.
Rob Swigart: At the very. At the beginning, I’m just in the in the sidebar.
Frode Hegland: Yeah, if you want. If you’re reading it on reader do command minus and it’ll fold out to a table of content.
Rob Swigart: Okay. Yeah. All right. It says, come back here.
Frode Hegland: The reason I think this is a bit interesting for us to think about is partly we are dealing with ACM hypertext document that is our core knowledge. Normally they are released as individual PDFs. However, last year it was also released as one compendium. So it was like a magazine of all the proceedings for that conference, which was very cool. That is quite similar to our book, which is a lot of articles by different people. So all of these things that are sexualized because a normal hypertext academic document, it doesn’t have that many pages. So the table, the table of contents are headings aren’t that useful. Relative to a major work. So if we think about in terms of a major work of two meetings every week for 6 to 9 months, which roughly will be September. That’s a real amount of data. So to make this readable in a specialized environment. Doesn’t that excite everyone? I can see gears moving and some of the foreheads. What do you think, Danny?
Dene Grigar: Sounds great. I mean, it’s going to make the report easier to write.
Frode Hegland: Yeah, the report should hopefully mostly write itself, I hope. And it’s. Yeah, I mean, I just find it absolutely gobsmacking. So as some of you, all of you probably will remember, there was a discussion, a disagreement between Brandel and me about the use of a word a few weeks ago. And that was recorded. And I looked at the transcript for that meeting. And the way Claude understood it was really frightening, almost because in the disagreements and agreements it was mentioned, as, you know, the use of this word. However, in the summary, it actually took it up a notch and said, you know, there was some disagreement on the use of language. That’s like near basic human analysis, isn’t it?
Dene Grigar: It would get Gingerbready to be just fine.
Frode Hegland: I didn’t hear what.
Dene Grigar: It would get. Gender wise, it’ll be just perfect. They keep calling me he.
Frode Hegland: Oh, it’s done a few things like that. That’s a Danny. That’s a very good example of what needs to be fixed. It also got Fabians last name wrong. Mostly. Yeah. So could you please just drop me an email? Because I’ll do all this in bulk. I will put into the prompt more of who we are, because it does use that. If you look at the last page of the document, I have written a little bit about what these meetings are about to help the summaries be more useful. So definitely not have a Deeney as anything other than Deeney.
Dene Grigar: Well, I think that’s just one example. Today I’m doing this external review of a dissertation. And these this person is working on an AI engine for for being able to discern emotion from images using, you know, a kind of a system. They’re building themselves and they’re, you know, they’re doing a lot of there’s a huge literature review on all these different AI products. And the one they’re building, he’s building, they’re building is a team. Is this one to recognize emotion in paintings, which I think is interesting. And so reading all of the kind of Problems with systems has just been on my mind for the past week as I’m reading through this document. So it’s interesting.
Frode Hegland: It is. And that’s almost a reverse of what Leon asks in the chapters as as there have been any reports of I infographic generators, as in generating images based on the text. I’ve tried a little bit of that so far. It’s been not good, but I do think that is absolutely something we should try to do to figure out how to make it more visual.
Dene Grigar: And just so you know, the reports for these kind of grants are just they’re so mind numbing and having to go back through all the archives. I’m trying to keep a running tab of what’s happening, but the reports that we’re getting from Claude are much better. But I will have to go through and clean up things. So it’s not really a done deal, but it’s just a hell of a lot easier.
Frode Hegland: Yes, absolutely agree with you. But and today’s discussion, at least for a little while, is what would if we throw away all the rules and everything? What would be the ideal document? Right. If we just think about document and we have very little preconceived notions. And one of the ideas is you’re writing up a report from the ACM hypertext conference in Rome. So in addition to writing and speaking into this thing, you should be able to include spatial videos and all kinds of things because Rome is a special place. I’m using where you and Rob and I were last year as kind of a case study. And if you take that in addition to headings and other keywords. It isn’t crazy to look at delivering this as a flat document, but also as someone puts on their headset and opens it there that this opens in a beautifully spatialized way. I really think we can do some, make some headway in that and make it much more compelling.
Dene Grigar: Well, it’s going to be helpful is once because report is very focused. Right. They’ll have categories we have to respond to. So we could take this dot. We can put into the query the different elements and then feed the document that the AI is already produced. So we can restructure it into the kind of. Process that needs to be delivered to Sloan. Yeah. So the eye. The eye on eye doctor. The eye doctor?
Frode Hegland: I imagine yes, absolutely. That’s what I did with the example in question, but the each meeting is, of course, one transcript in Claude. So if there’s anything that you think should be asked of the transcript for your report, please tell me and I’ll put it in the prompt now.
Dene Grigar: Well, some of it is things that we’re not even talking about as a group as budget expenditures. I mean, stuff that we’re not doing here. Right? So we’re not going be able to capture that information. But yeah, and I haven’t seen the report document yet. I haven’t seen it in the portal.
Frode Hegland: Okay. All right. Okay, Fabian, I’m going to pick on you. So this ultimate document, it needs to contain Fabian blocks. What would that mean in this context?
Fabien Benetou: Since you pick on me, I can say what’s what I think. To me, the ultimate document is probably an HTML page. I have a bias because that’s what I work with. But you can have video, you can have sound, you can have 3D models, you can have internal references, external ones. You can reshape its flow so that it looks a certain way on one device or another. You can have it local, you can send it to someone else, or you can host it. So I it’s a bit boring of me to say this, but and of course, one thing that I when I was whining a long time ago that books should be dead because they’re not interactive enough. And I think when the purpose is learning, interaction should be there. And again, HTML page allows this. It allows to have plain old text but also interaction thanks to code. So yeah, to me the ultimate document is an HTML page. And it’s again it’s, it’s something with that changes over time, like HTML when the web started is not HTML today. But yeah, to to me, I have a hard time imagining beyond an HTML document. I’m sorry.
Frode Hegland: That’s okay. Because with all due loving respect. And I’ll give you a hug when you’re here. That’s not what we’re talking about. What we’re talking about is ignoring the storage mechanism. And I know I’m one of the ones that talk about that a lot. So it’s really a hypothetical thing. I agree with all the attributes of Albion that you prioritize. I prioritize the same. So the question is, is really Okay, I’ll do it quickly in three parts. Number one, my advisor Wendy many years said she doesn’t have time to write anymore. She just wants to be able to put things into a document. Like a tweet or whatever, right? So I’m thinking the composing of it should be fun and engaging, like early HTML design applications like Go Live and that kind of stuff. You put things in almost like blocks and you get a document, you thread it together and so on. The storage mechanism, whether it’s PDF, HTML or database. Right now, I don’t care about at all. But the key should be that when you’re on a tablet, a computer or an XR, you can access it. And the best experience should probably be an XR.
Frode Hegland: So I could imagine you’re actually authoring this with a headset and you are manually, if you want, putting pictures here and this there and whatever, and assigning code to specific things and all of these cool things. So to be able to store that somehow. And share it is really what we’re talking about. And yeah, we can make the working assumption this is all HTML. That’s not a problem. But how amazing should it be when we open this kind of thing? Like, imagine this report from Rome because it’s got a little bit of everything in it. So there’s a little bit of Typekit, a little bit of just voice, a couple of citations because it’s academic, right? But when you do this, you want to be able to say this article, which I miniaturized or put up here. Arrow. Goes against this one. And then you pull out the data from this one and say, look at the data there. You know, this kind of theatrical stuff. We wanted to be something you go into and it is alive and active and understandable as a modern computer game.
Dene Grigar: Leon’s question is a good one. Should the ideal document of pages and unlimited canvas, I. You know what’s interesting, Leon? That’s a question I ask all the time, because in Breadcrumbing we would probably expect pages, and over time we would get rid of pages. Right. But to get people from point A to point B, it’s hard just to drop them into a space where the whole system is different. But I breadcrumbing o breadcrumbing is a term we use to that. Like for example So when we’re training students for web based writing back in the day we just didn’t throw them into HTML. We tried to show them how to read an HTML page, explain what HTML is, and then take them step by step to empty space where they start to code. So it’s leaving breadcrumbs like Hansel and Gretel and the forest so that somebody can follow. Right. So it’s following in web design. Breadcrumb is is the kind of the thing on the interface that allows a user to follow your path. Right? So what you don’t do is just throw an audience into a brand new space and go, here we go. I mean, even Apple didn’t do that. I mean, when you get into the Apple headset, you see the same kind of interface that you see with your phone and your computer, right? At some point it’s going to be different. But for the moment it has to be somewhat familiar so that people can cognitively make sense of the system. But at some point, we will let go of those prior systems because we don’t need them anymore. We don’t do page numbers and web pages, for example. But in the early days, we struggled with how to designate a page space, right? So it’s that kind of thing. But so yeah, the question is, how do you design for this new world? And I’d like to think we’re going to come up with something so different that it’s that that we’ve left the the past behind, and we’re making use of the affordance of the space and a really amazing way. We’re not there cognitively yet. Least most of us aren’t.
Frode Hegland: Product. I would say that on this issue, Addressability is really, really key. That obviously partly to cite something you need to in many cases be able to say what you’re citing inside the document. But also if you want to interact with it, you need to be able to address it somehow. So addressability should be at least to the paragraph level, maybe to the sentence, maybe to the word level. How we do that, how we interact with it, how we allow the user, the reader and the author to deal with it. And Peter saying he’d want embedded quotations to be addressed, directly addressable. Yes. Like what did Ted call it? Transclusion. Right. These these are really, really important aspects. The point is that. We need to decide on the ultimate of this at some point, at least as an exercise. But so? So imagine this report from Rome. How could we make it incredibly compelling? Let’s say that similar to our record. The person writing this report recorded all the presentations, all the meetings they were part of and went through the process. We’ve done so that there is I mean, names have been extracted, some summaries have been done. Couldn’t you imagine? Kind of. Yeah. So Rob just wrote a weird idea. Would be a memory palace. No. Absolutely. Rob. So when we were in in Rome, we were in physically different rooms depending on the conversation. So an example could be to have a model of the physical location we were in and having notes about what we saw in those two different rooms. We could make it incredibly non-virtual, incredibly physical related. So we could actually see the YouTube videos of the presentations presented in the rooms they were from. We could, you know, if somebody if we were talking to a friend in the back of the room or writing a note, we could find it there, and then we could flip it into all kinds of things. This is exactly what we’re thinking about. Fabian, please.
Fabien Benetou: Yeah. To clarify also a bit, I one of the reasons or quite a few of my prototype or it’s not Memory Palace, but it’s having trying to find spatial representation of the not necessarily the content of the document, but at the very least its structure. And and quite often when I show things like this, then people think of memory palaces. Memory palaces are usually done in order to remember unstructured data things that are I don’t want to say so dismissively, but it’s kind of like a party trick to remember things that are not really supposed to be there together. Like a kind of grocery list. And I’m not saying it’s not interesting in term of, oh, how our mind can do so much more that when we use the right process. It’s interesting. But in my opinion, it’s not as interesting as finding the structure behind the data. And once you do, then you don’t necessarily need to use the tricks of the Memory Palace. One warning, though, is that works well when your structure is stable. I don’t know. It’s like a tree, for example. And it has a certain shape, so branches go a certain way so that when you do it a couple of times, you, you go back to it and it’s easy to remember. But when you data or change every day, then maybe the structure. Also you need to reconsider what it is. And then the shape, the layout you will have then in the environment can be completely different from one day to the next. And then you basically lost in space virtually. And it’s not it becomes very difficult. And if it changes quite a bit of a time. But yeah, spatially, you going from the structure of a document to a layout we can navigate, in my opinion, is indeed quite interesting. From my experience, it has some limitation, though.
Frode Hegland: Yes yes yes yes yes. This is this is exactly the discussion. So there’s one memory palace that’s quite physical. There’s also data memory palaces, like I could imagine, literally a plant, a tree in the corner of our room. That is our discussion. And each little branch represents one of the people. And we can have keywords around this thing as kind of magnets, as little suns to draw it out. And then we can see if something changes over time. Right. That is one thing that I’ve been doing with this Claude thing. And it was a little bit generic, saying we’re more focused on specifics of implementation for slow now rather than general before, but tracking the changes, whether it’s AI or more. Crystal clear data or whatever. Absolutely. And it can be done in so many different and core ways. The little scenario that I mentioned at the beginning of having an owl on the desk, that is actually a local LLM, it each one of them could maybe if you have many, could be trained on the different people in the group, what they’ve said. So when you’re going through something new, you might get different reactions from these different ones, like the lion one, maybe, you know, cocking its head sideways when we’re saying something, worried about URLs, and then we can examine it based on that. We got to think like way beyond Harry Potter, way beyond Bladerunner, just into really cool land of how we can view our stuff.
Dene Grigar: I think it’ll be interesting to be able to say, you know, we have our goals and we put our goals into the system, and then we say, where in the document do we show we met this goal? And it designates that. Right. So we can go back and say, on this day, at this time, we discuss this goal, that we achieve this goal, and here’s how we discussed achieving it. And that will be very useful for the final report, which is usually about 20 pages long or so. So but for our own edification, like when do we when do we make a change? Like when do we realize that this juncture, this is the decision we had to make this decision then. And once we made that decision, we couldn’t go back?
Frode Hegland: Yeah.
Dene Grigar: So branching out.
Frode Hegland: Yeah. That’s that’s really perfect. Danny. Sorry. I didn’t mean to cut you off. Please continue.
Dene Grigar: That’s okay. Just branching narrative.
Frode Hegland: So one of the things I’ve asked the current thing to do is give us a suggested agenda for next week or next meeting. What we can start doing is taking our actual agendas into it and see if we, you know, how in the meeting we fulfill the agenda, but also by having explicit goals and any you’re right, we haven’t seen exactly what they are according to Sloan yet, but we should be able to put them in and we should be able to if someone, let’s say deny or deny me being Copa is maybe we’re designated in the prompt, as in people who can say whether something is done and all we need to do in a meeting is say this is now done or we now agree, and that is then put in the record, and then we can see that. Now before I turn off recording, because I see something cool here. I just wanted to mention that the record, as we have it now, and every Wednesday, there’s a link to what Andrew has done for that Wednesday. So in our amazing view of this thing, we have one click access to going into what Andrew has built. I mean, that’s freaking awesome, right? So we’re not just reading a report in the headset. It was like, what did he do? Click boom. We’re here. Can I have some massives? Yeah. Thank you. Dina, you’re the only one smiling. Come on guys, that’s amazing.
Frode Hegland: Right. And with the stuff that you’re doing, Fabian, same thing when you have something you want to have on the record. As long as it’s been in our chat or you’ve told me. People going through the record reading all this stuff. Okay. Let’s look at what Leon has written. And then we’ll do Fabian screen off is as perhaps the document should allow, hassle free editing in all three dimensions. Absolutely. But also be able to accept a grammar to store the structure of future content. Yes. Bullet points is basically hardcoded grammar and current documents. We need loosely coupled ones. Yeah, one. Leon, you’re so absolutely on my wavelength. And I look so. By the way we have booked for future Texts Social. It’s going to be the last weekend of May. Europeans are coming. The rest of you, if you can make it miraculously, please do. It’ll be awesome. So, Leon, we will discuss these points. The biggest thing that I would want to change right now in the document is to be able to choose to show a bullet point list, or just the first with ellipses to have that expandable contractible as a normal way of writing. Of course, you can make it happen in HTML, but so often you have a long list, but you don’t want to see it all the time. So that detail? Yeah. Spot on. Any other comments before we go? Secret Fabian Mission 007.
Zoom: This meeting is being recorded.
Frode Hegland: So in a generic way, what Fabian just showed us is a document in a space, a real space for the real document, which is why he couldn’t show it. It was fantastic. One of the things I really liked is you click on a document to teleport yourself to standing in front of it, which is just elegant. One thing that immediately springs to mind is the notion of having different reading rooms that we’ve talked about before. So let’s say we have an ACM hypertext Reading Room that has specific resources for those documents, so they can be presented in a similar way that we just saw on beautiful posters. And they can also have in that room, because you don’t have so much motion sickness, but, you know, clicking to jump to teleport. And also little things like these documents need to have full metadata. So if there is a page above it, it should be easy to have the headings if need be or whatever else. I could talk for an hour about this, Bobby and this is amazing. I would really like someone else to talk. Sorry just to amazing.
Dene Grigar: Can I say something? Go ahead. Rob. Rob. Talk.
Rob Swigart: I just I just wanted pop up headers as you go over the page. Because you can’t see what’s what’s what it’s about. So if you go until you go in. So.
Dene Grigar: I was going to say that the breadcrumbing is there. So, Fabian, do you see what I mean by Breadcrumbing? Because you still have the documents in succession, right? It looks like the the original document is spread out over space. But at some point we’ll probably just have text. Just floating in space like I’m imagining it for myself, right?
Fabien Benetou: Yeah. Just also in term of texting in space, it’s, it’s like the post-it note I use, like, I think two weeks ago. I think it’s same, like using a real world architecture or building or room is to me like using post-it. It’s to help people get or remember how familiar they are already with manipulating documents in space. Like we already do this, like same whiteboard and all this kind of stuff. But if you have literally the text floating in space, most people then to me are still confused or surprised or not sure what to do with it. So I think, yeah, it’s to help, maybe for more gentle transition and then, yeah, the text itself can flow, it can be different shape. Also, it doesn’t have to be flat. I mean we can go. I think Adam also did a lot of very interesting demos long time ago where, where the text was did not have to be flat. So it completely I mean, it begs for different experiences, but as a gentle transition, I find that so far that’s been helpful.
Dene Grigar: Has anyone ever read the story? I might have mentioned this before to some of us, but the arrival science fiction story turned into a movie. It was about a movie. Yeah, the aliens that thought in three dimension and the way they saw the world was totally different than the way we saw the world. And once we adopted their language, it changed the way we functioned and term. And it had to do with the way they saw time. Right. Time and language go hand in hand. And for us, time is linear. There’s a past, present, future. And for that, that world, that thought in 3D time collapsed into itself. Right. And I think that’s part of what I’m interested in with, with all of this notion of text in VR and XR. So what does it mean when we start to play with text? What’s the back of text look like? What’s when you look down at what is it, how is it structured our minds. And yes, this is navel gazing. I totally get that. But I think this is the interesting part about what we’re doing is the the intellectual part is how is it changing the way we think. And Frodo. We did talk in our grant about we, you know, the notion of how we think is associated with how we see text. And think different. So how are we going to be thinking different in this new space? Right now? We’re not thinking so different, right? We’re trying to breadcrumb things, but at some point we’re going to be moving beyond where we came from, which I think is very interesting.
Frode Hegland: Yeah, absolutely. And that’s what we’re talking about today about the what would an ultimate document be? And when I was in art school, Chelsea School of Art, a long time ago, I remember doing a thing and then I did a thing and a thing. And my teacher said, at some point you got to stop. You got to frame it, you got to end it. And that’s when I say document. That’s all. I mean, that is some boundary set on knowledge or information, whether it’s in what we call a document now or a database set or whatever. I don’t care for this discussion because I agree very much with Dean. We need to get to the rival talks about 3D texts. Apple likes the word spatial, so let’s just use the Apple word for right now and say we need to further develop spatialized thinking and communication, right? Oh, Leon. Okay. All right. We’ll see you soon. Thank you for being here. And then, Fabian, you have your hand up.
Fabien Benetou: Yeah. To me, the the equivalent or the analogy of arrival would be to, to I hope initially when I started to work in XR, it was like I said, to organize my posters and my notes all around me in space. Now I’m starting to think, well, still, that. But also more specifically in term of like weirder, I guess, or more unique again, for arrival and time and the kind of circling glyphs. It would be for me managing again grammars like the relationship between items within a domain again, being chemistry, being knowledge, being whatever because it’s so abstract. But I think if you can make it, again, not tangible, but at least give affordances to it, both visually and with hand tracking or eye tracking, whatever. But to be able to manipulate such grammars, I think it would be quite powerful. So if if again, there is a kind of way to change how we manipulate symbols and to open up maybe not a new way to think, but at least to facilitate it to the point that. That it it enables usage that were, until now, practically impossible. That’s that’s what I’m betting on basically.
Dene Grigar: I agree.
Frode Hegland: Yeah, I heartily agree as well. So. Why don’t we? This is not changing any of our Sloan stuff. But why don’t we just say that for. Barcelona presentation in September. The corpus we primarily deal with is this year’s paper papers. And we obviously will get the HTML versions of it, so we can have some standard views in our ACM hypertext library. In that library. There will be different modes of reading, including what we’re working on right now to go into a fully immersive mode for paper. And also we. This is just me being controversial. Depends on what Deeney says. We also invite everybody writing a paper to say if there’s something they would like to spatialize about it, and if so, we will try to accommodate that. And finally, in addition to having the room with the papers, different ways to read them, including Andrew’s Immersive, we could also have a thing which allows them to explore our dialog over the last many months, so that becomes another component.
Dene Grigar: We also need to do testing. So one of the things that we have some money for is is the testing, usability testing. So we want to do that and set that up and imagine we’ll do that in a demo. So the blue sky paper deadlines coming. It’s not past. So imagine what you’re talking about Frodo. For the paper is the blue sky, which is perfect for what we’re doing because we don’t really have final data, which is what a regular paper would be. And then we submit a demo, and the demo would include the testing aspect. Is that what you’re imagining?
Frode Hegland: It’s compatible with what I’m imagining, but what I’m really imagining is and again, not disagreeing with any of that, I’m just thinking how absolutely amazing would it be? And oh yeah, first of all, an important aside, I’ve been putting the Vision Pro on people’s heads lately, and it’s been relatively disastrous. First of all, it takes so many minutes to do all of that and teaching them the pinching a lot of people, they can’t pinch clearly like this once they know they don’t have to reach out, they make these minuscule things. The eye tracking is most certainly not perfect. The basic interfaces. Anyway, I do think we’re going to be using a lot of these guys as the primary demo material, and that’s absolutely fine. Using the. That Quest Pro and the quest three. But what I’m really thinking about, someone puts on the headset there and they’re in a library of sorts. Maybe we make it actually your office Dini because it’s beautiful. So for the sake of argument, we scan your office completely and then we have, you know, completely skeuomorphic to begin with. We make it as physical looking as possible, and then we start doing the kind of Fabian interactions. And then, you know, portals and so on. But just make it. This is the current proceedings. This is why you’re here. We make it so amazing and we make it public so that people at the conference will, when they’re not with us, run, go to their own headsets to read the papers because it’s better than reading it on a PDF. The one dream.
Dene Grigar: Leon, I mean Fabian.
Fabien Benetou: So one quick example. So in term of testing that I think was I mean in retrospect obvious, but the document in space I’ve shown before so initially I did what I thought was the simplest interaction for my audience. Namely, you move the pointer when you hover, you see a highlight. When you click, you get there. Honestly, I was happy with it. I thought, this works. And people who tried it also but then they showed they scrolled because a lot of people in traditional settings, they on the web or they have a PDF and and going down or reading further means scrolling down. That’s like number one interaction. And I was like, shit, that makes sense. But this is not how I move in 3D. So it’s very linear. It doesn’t it’s not even 2D. It’s 1D anymore. Like you go down a tour basically. But that’s how people wanted to interact with the space. So I did not like it initially. And then I was like, it doesn’t matter what I like. That’s what the audience wants. So then my I initially I was like, also, shit, I don’t know how to implement this and actually found a trick. Like it took literally two lines of code. So that that was actually super easy to accommodate them. And it’s a totally new mode of interaction, but it works.
Fabien Benetou: So that that was one very good thing. And I had no intelligence in that. It’s like literally testing and people telling me it does. The scroll wheel doesn’t work. And I’m like, what do you mean, the scroll? It doesn’t work. It’s not supposed to work. So testing is super valuable in such cases. And same testing ideally literally sitting behind the person. Because I had another thing like this where the person tried to experience this with a tablet and then they they tried to pinch to zoom in and I was like That also makes complete sense. Like I did not have a touch screen when I designed this demo. So I thought, just scroll wheel. And after actually somebody asked for it and clicking. But when you have the affordance, when your device pinch and to see the person do it, exactly same reaction like, whoa, shit, that makes sense. I should have done it. So I’m just trying to highlight there that testing. Can give some really good ideas. And also they can be very also easy to implement. It’s not because it looks totally different. And the way it’s not always like this. Obviously there are a lot of times it’s like yeah, no, that’s that’s a great idea.
Fabien Benetou: It’s a perfectly valid knit but never going to I don’t ever know how to do this. But those yeah, those couple of times there was a way to and again the goal there sorry to to summarize it by testing and ideally in person you get so many quality ideas for the audience, but also it doesn’t mean the initial way to interact, same as before about floating text. You go with the document that looks like a document or post-it note that looks like a post-it note, and then you depart from that one. The person has been convinced that they can actually interact with it, and there is value within that space to interact with the document. You can go crazier, let’s say. So now that let’s say scroll wheel works, then I can say at the end of the tour, hey, you can also use your mouse to click around and freely move around and your arrow keys to move around. So it’s like teaching them the audience that wanted a linear experience that sure, that works. But you can also go deeper and have generally novel ways to interact. And I think this way, rather than forcing onto them from the start a new way to interact. So far, it looks like a much easier path.
Frode Hegland: Right. Under and Fabian and Peter particularly. I uploaded a HTML version of our meeting record. So if you want to have a look and see if it’s relevant, if the. The headings are supposed to be and they look like they are properly h1 h2, which should make it more passable in the space. So with what you’ve said then, Fabian. Okay, so, Danny, I’ll tell you my dream and see, I’m sure we agree mostly, but just to make sure. How about we. Okay, first of all, the meeting record is reading in in XR. So I do think that’s Sloane related. Academics have meetings too. So if I could obviously. So if academics can see our process for dealing with our meetings, it may very well be interesting for them to use. I think that kind of reading is as important as reading papers. There’s no JSON version. No, actually, I don’t know. Hang on, maybe I can do a Jets one second here. That’s a very pertinent question.
Frode Hegland: Okay. I don’t know if this is. I’m sorry. This is just Peter’s very useful requests. Okay, I’m uploading an XML version of it. Right. Anyway So, Dania, why do you think, first of all, of having the idea of scanning, let’s say, your office and having two major things to read in there. One, the proceedings, two are meeting record. But reading it and far beyond any way you can read on platform at today.
Dene Grigar: You know what I’d like to do? I totally disagree, I totally agree with that. But I’d like to do though is offer them three. Or 2 or 3 possibilities to get a sense of what people really if you give them. If we give them one thing, then we’re testing one thing. But I’m wondering if what we’re you know, the question is, what are we testing? Right. What what is it we’re trying to test? The efficacy of reading in online spaces, the virtual spaces reacting to them. Is it trying to get a sense of where we’re going in the future? I mean, if the notion is what lies ahead and how are we going to think beyond where we are currently, which is what we argue we’re trying to do and go beyond what Apple and companies are. So why not offer maybe 2 or 3 visions like, here’s a room, here’s the text, let’s experience that. Now here’s that same text in a different scenario. Right? Here’s that same text in a different scenario and get a sense of and we can essentially breadcrumb them ourselves and get a sense of where people are currently and what they’re going to accept as we move forward. And I know that’s going to take a little longer, but I’d rather do. A more deeper dive and say, hey, put this headset on.
Dene Grigar: And I can say this because I just read through this dissertation again yesterday and reading through all their testing data. 24 people digging deep into a, you know, an AI system as opposed to hundreds of people. Not they weren’t using big data sets, using small data sets that looked really deeply at what people were thinking as they were working through the system that they were building. And I think that’s a very efficient way and a very interesting way. And since we’re not just testing once we have testing that’s ongoing, it wouldn’t be bad to try this more broader sense. And I just pick like ten people, ten really great people to put into the headset, people that might already have experienced with VR. So we’re not teaching them how to like, pinch, you know, because our audience is a little bit more savvy than someone in a coffee shop or a student that never put on a headset before. This is a great opportunity to work with really smart people, doing with a lot of talent to think about this system and who know already what ACM hypertext papers look like. And how do they really want to experience this in a different space? I just think it’s an opportunity we could take advantage of and and leverage. Broda.
Frode Hegland: Yeah, I completely agree with that. I. One of the things I’ve been talking to Adam about, and Adam is really upset because evenings now, just he just can’t do anything. On Wednesday, he hopes to be able to, but not today. So, you know, he wants to jump back in here, and he wants to start experimenting more. That’s a discussion. We’ll have more on Wednesday, I hope. But the whole point of me mentioning it now is what we talked about in our last meeting, where when you get to our experience and you have that sphere instead of one, you will have a few spheres. I’m not saying we literally have to do that, but I’m saying that for the same data, it would absolutely be worthwhile to have different experiences. So we have something to measure against. So I completely agree with that. So I can imagine in your office, in your virtual office we could maybe even do one experience that is as basic as just overlaying it on a computer screen. So, you know, your virtual computer screen then. So it’s very flat and linear. That’s the basic. And then we will look at these other things. And maybe over the summer we’ll get to an experience that is completely wild. So we’ll have like 2 or 3 experiences per data set. But the question particularly for you, but also for the rest of the group, is first of all, do you think Marius will be able to give us all the papers before the conference so we can slot them in? If they’re formatted in the normal ACM way, we shouldn’t require any time to slot them in.
Dene Grigar: He should be able to. Because I’m in charge of the awards ceremony. I have to have them in order to have people view them. Right. But whether or not we get permission to use it for this purpose is a different story, and I can talk to them about it. Okay.
Frode Hegland: So that’s great. That’s that’s all we need to know at this point.
Frode Hegland: The other question is then do we agree that we should also have our meeting record be in this room?
Speaker8: I for the.
Dene Grigar: For the conference and for the conference or just in general.
Frode Hegland: For the conference.
Speaker8: No.
Frode Hegland: Okay, let me try to sell it to you then.
Speaker8: Okay. One of the things.
Frode Hegland: In this community that we’ve had a problem with for many, many years, and I think we’re almost solved now, is going back to see what we’ve been talking about. And I think academic groups will have exactly the same problem. So I think that if in our space because it is basically text. There is a really useful way for people to view nine months worth of record. Could be really compelling for them to do it themselves. So that could be one of two experiences. Still not convinced how.
Dene Grigar: Well, I’m just wondering why they would. I mean, I mean, we could do whatever we want. I mean, that’s fine, but I’m just wondering. That wouldn’t be something I’d want to test at the conference. I’d want to test an ACM paper.
Speaker8: Yeah.
Frode Hegland: But in addition, I mean, in your academic work, you have meetings, right?
Dene Grigar: Yeah, but nobody wants to see the record of my meetings.
Speaker8: No, but.
Frode Hegland: I mean this, of course, just academic meetings in one sense. But there’s also academic meetings for specific projects where you do want to track. Right? Like like this.
Dene Grigar: Yeah, I want to track all of this, but do I want to make it available for testing at a conference? Probably not. I mean, this will be something we hold on to for ourselves. Absolutely.
Frode Hegland: Well, obviously what is released publicly will have to be vetted by all the members. We’d have to go through and make sure there isn’t anything misconstruing us or that we don’t want public. So you know that that in itself is important, let alone any other aspect of it. But, you know, I can’t imagine, for instance, the prompt thing that I just did on has what does Fabian think of this or what does Rob changed over the nine months? That kind of stuff can be really, really useful. Right. And I don’t think we have anything to be embarrassed about in this group. We’ve been learning. It may just be a really good case study system for. How to build an interaction because the technically a prompt can do anything, but that’s just a prompt. But to have a space, you know, a calendar and, you know, maybe different people. There’s all about Deeney, there’s all about the time. This is all about the subject. Slome this is about the subject symposium. These are the changes based on these people. These are the connections. It’s an incredibly interconnected network of thoughts. That can probably be more visualized in a spatial environment.
Dene Grigar: I’m not disagreeing with that. I think it’s great. I’m just saying that it’s not something I want to bring to the conference. I want to test a paper.
Speaker8: The problem I’ve been talking about.
Frode Hegland: The problem I have with papers by themselves is that, first of all, many of them are irrelevant to a specific reader. Obviously, you know, not every paper will be relevant for every reader. There’ll be different constituents. A academic paper for hypertext. It’s not that long. There aren’t that many pages. So and also in terms of the corpus of all the papers for that year to see, the interconnections may not be that useful. I do think we can augment the reading of these papers and the creation of these papers, but I don’t think it really shows off a huge volume of information presented in a more interactive manner.
Dene Grigar: So I’m imagining, I guess, what we talked about earlier and I was imagining that we have. A library. And we have a listing of papers that people can read write. Mark has gone through a lot of trouble to give us the data for that. And and we put the headset on someone and we say, okay, here’s a, here’s the library for you to choose from. Pick a paper. I mean, I can give them like thousands of paper. We pick like a a variety that may be 5 to 7. We say, now pick a paper that you want to read and let’s watch you work through this. Right. And so they they tap on one of the papers. It opens up. And then they. Hey, how are you? And opens up. And then they work through reading it, right? They swipe or pinch and do all the different actions and then they we talk to them. It’s like a talk through protocol, which is what we do in usability testing, right? A talk, a loud protocol, you know, what do you see? What are you experiencing? Okay. What are you clicking now? You know, how do you what do you think about this layout, this this way and just walk them through that. And then we say, okay, now here’s that same list. Now let’s put you in this environment. And then let them try that environment. So one would that skeumorphic which you you prefer, one that’s less skeuomorphic. And one that’s extremely, you know just spatial like, I like we just take them through three different scenarios, but.
Speaker8: Oh.
Frode Hegland: I agree. And just to summarize for Randall, who was there. Hi, Randall. We’re talking about a few different things, primarily what to show at the hypertext conference. And we’re talking Denny brought up the point that we shouldn’t show just one interaction for the same data. We should have 2 or 3 different types of interactions. So we have something to measure against, which makes complete sense. Fabian earlier showed walking around in a physical environment where there’s documents you know, on, on walls, so to speak, and you point to one and then you’re teleported to read it, which is really beautiful. It’s a memory palace thing. So what we’re talking about now is also our meeting record. So the meeting record. I have an author. It’s gone through the whole cloud thing. So there is a hierarchy of sections for it. And specifically you have to make sure you’re happy to make it public. If we make it public, it’s not public now. But one of the notions is that because we have have you had a chance to see that PDF, by the way?
Brandel Zachernuk: No, I haven’t had a look at the at the PDF.
Frode Hegland: That’s absolutely fine. Well, what I’m talking I’m going to give you. The HTML version. Shock and bold. So what we’re debating are the different values of what to show, because I’m quite excited by the process of our meeting records now. So in summary, what I do is we record here, upload to YouTube, which takes a couple hours, slow connection. Then I go to Sonics, I get a transcript and it has. Speaker one. Speaker two I just put all the right names, check if there’s something awfully wrong, save it. Then I upload it to cloud AI, where I run one prompt that has a lot of context, many, many questions, and then it gives it and I put it in an ever expanding document. Right. Point of it is I’m really shocked at how good it is, and I will be uploading as quickly as I can after every meetings now, so that everyone will have a chance to see if something is wrong based on their memory. Add manual stuff. If someone writes a note, I’ll just paste it in. It doesn’t have to be pure AI. So one of the things that can be done in addition to the ACM documents is provide an environment in XR to read our record. Because we can navigate based on dates, people, changes, topics and so on. This is something you’ve been working on, Randall? Do you have any thoughts on this? And good morning.
Brandel Zachernuk: Good morning, good morning. I’ve got to say, like I my hostility to artificial intelligence makes it very difficult for me to find any particular merit to any form of summary that comes through it. That’s not to say that it’s that there isn’t anything there. It’s that it’s not something that sort of jumps out at me. My, my preference is for things that are manipulable and that have the capacity for for, for being under the user’s control. And so to that end, you know, it’s, it’s trivially and obviously true that something that has some manner of AI for the recognition of tokens that can be kind of separated and split is, has merit. But yeah, it’s, it’s and like I said, like it’s I’m, it’s clear to me that, that there is much more to AI than I see, but I, I don’t see it.
Speaker8: Right.
Frode Hegland: So on that note, I saw Fabian had an icon for that as well. There’s two aspects that has surprised me recently. One is and this is a great example. Randall, you and I had a disagreement from a use of terminology a while back. I found that. Not just this word was blah blah blah. It wasn’t like data robot thing. It actually found it and said there was a disagreement between these people on the use of terminology and had to do this. So that to me was a bit of seriously right that that was wow. But the real use, I think you and I are completely aligned on and that is basically entity extraction. Right so that it can be manipulative, manipulated, where one of the key things I find interesting is that I ask, by the way, the prompt is on the last page of the document so you can read it there and please suggest edits. I’m very happy to. So for instance, you go through one meeting and you can see that so and so person was mentioned even in just reader. You can select that the command f and you see anywhere else it was mentioned. But I think it would be much more interesting in an XR environment to have a pile over there of people mentioned. Dates here, you know, all that kind of stuff we talked about. So that means that you can choose to do the kind of interactions you’re talking about. All the I should do is not to give us a summary from its great intelligence and wisdom. Absolutely not. It should allow for these interactions to be possible.
Speaker8: Yeah, right.
Brandel Zachernuk: So that does make sense. I’m sorry. I’m distracted. I’m trying to find the file, but it isn’t. I guess it must have been made a while ago, because I keep failing to actually find it in my downloads. What? Because the modified date is a while ago, I’m guessing.
Frode Hegland: I’m giving you both in our chat here, the HTML and the PDF now so you can download it from the chat. They’re tiny documents.
Brandel Zachernuk: Yeah. No, I have downloaded it several times. I still can’t find it.
Frode Hegland: Isn’t it exciting that the meetings dot index, dot HTML and HTML? Come on, Fabian, you should give me, like, a high five or something.
Speaker8: Right. So.
Frode Hegland: Anyway, we can those of us who want to can continue working on the how to read the record as a separate thing and then see if there’s an interesting overlap. But definitely we need to keep working on the different ways to experience the ACM documents. And I’m going to give you the mic. But just to make it clear for everyone, we’re going to try to get the ACM papers for this year. That’ll be the primary reading experience with the library of everything else, right?
Dene Grigar: Yeah. I’ll write you out to Are you sure about that today? I also want to mention to the group that you have been invited to to be the keynote speaker for the human workshop that Klaus Aizenberg puts on every year. And so my question will be, what will be the difference between your talk? And our paper. Are you thinking about what you’re going to be talking about now? I know we’re meeting with him sometimes this week. The two of us. It depends. You got three things going on at the conference, and we want to make sure that we’re. We’re writing for the right thing when we write the paper.
Frode Hegland: It depends very much on whether he’s going to fail me or not. He’s my external examiner.
Dene Grigar: I’m sorry. I’m not following. If he invites you. He’s in. He’s inviting you as he’s a chair of a conference proceeding. Are you going to hold that connected to your dissertation?
Frode Hegland: Not at all. But it would be quite annoying if he fails me and I have to go there and talk about his academic stuff. It’s just an amusing situation to be in a relatively closed academic community where, you know, people collaborate on one side and then judge each other on another side. Don’t worry, Klaus and I have a fun argument over this. I don’t know, I haven’t thought about it. Of course I’ll present. Of course I’ll do my best. And it may very well be that. Considering we’re all in this together. If you want to, we can work on making that presentation. The dialog maybe. I don’t know, Danny. What do you think?
Dene Grigar: Well, I guess my first question is who’s all going to write the paper? We have a paper to write the blue sky. Who’s all going to be part of that, and how who’s going to take the lead on it? That’s the practical stuff.
Speaker8: Yeah.
Frode Hegland: Well, I mean, for the human thing, I can take the lead on that. And I’m extremely happy for anybody who wants to join for the main paper. I’m guessing that will be you, Danny, who will take the lead. Right. Because you are the lead.
Dene Grigar: I guess so, but who else is writing on that paper with me? I’ll start a Google doc. Anybody else want to ride on it besides me?
Frode Hegland: Why don’t you put together a basic outline or something and share it with the group so that they know more of what that would mean?
Speaker8: Okay.
Dene Grigar: But we want to make sure that they’re different, that your presentation is different, and you’ll have to publish your presentation. Guys, there’s nothing you do at the ACM conference that is not a published element. When I did the exhibitions, I had to write a paper, had to publish a paper with it. And so it’s a really strange phenomena. So I just published my curatorial statement, but it’s very strange. You can’t even give a demo without a paper, as Rob and I learned last year. Right, Rob?
Speaker8: And of course.
Frode Hegland: The irony is that the discussions after the presentations of the papers and the group is not recorded, and that’s where some of the good stuff comes out. So it’s it’s a bit frustrating. Something that Mark Anderson has brought up a few times. But yeah. Okay. Why don’t we say this? The way that we record the meetings will probably be the human presentation, you see. Apparently one of the topics from the beginning today was kind of a provocation on what might be the ultimate. Hang on. I’m just going to give you another link, the ideal document. And I’m not talking about document format. I’m just talking about what should it be able to do. And clearly one of the things is specialized information. And I use the example and this is what I’m sending to Apple via Bruce Horn. As in please make it possible as part of a long list of vision issues. Is this the person the conference we were at last year? Some of us in Rome. How to write a report of it? A little bit of writing, a little bit of sighting, a little bit of pictures. But Rome is a special place. Maybe you’d want to put pictures or spatial videos on the locations where you shot them. Maybe you want to do that kind of stuff. Maybe you want to have a glossary in space. All these things. It seems like a mentally good case study for what should a spatial document be able to contain and present? So that’s partly why we’re going all these trips at the moment. And considering what you’ve done with timelines. Randall, I think it would be really fun with you to start looking at our record.
Speaker8: You know, so I.
Brandel Zachernuk: I I finally found the file. For some reason, it was Yeah, it’s modified date was old I guess. Because it was modified at some point in the past. I don’t know. We’re talking.
Speaker8: About the.
Frode Hegland: The the HTML or the PDF.
Brandel Zachernuk: Yeah. No, the HTML, it was oh, it’s modified at today at 927. I don’t know why it wasn’t turning up. As the most recent plan, I just I don’t keep a clean enough set of direct directory structure that I could find it because my download is downloads file folders, thousands long at any given time. No, this is super interesting. I also bought new AirPods, by the way. At your encouragement. So I’m still back through these, and hopefully I’m audible. Yeah, this is really interesting in terms of the it’s much closer to the tokenization that you can use to, to be able to kind of construct that at least these names were mentioned. It’s likely. It doesn’t have who was present, does it? It just has what people spoke. It says it gets a speaker summary.
Frode Hegland: If you look at Okay, just tell me what date you’re looking at. So we’re looking at same thing.
Speaker8: Let’s see.
Brandel Zachernuk: Say 3rd of April.
Frode Hegland: 3rd of April. Right. So one 3rd of April. What this means is, of course, linked to the video in the transcript, which we may or may not want to make public for everything. And then there is a list of speakers or people who were there, followed by a one basic summary.
Speaker8: Yeah. Okay.
Brandel Zachernuk: No. That’s cool.
Frode Hegland: On the next page is just copy and paste for Andrew’s updates. And then we have the names for speaker summaries. And then we have the topics discussed as now. And then the thing that I think is quite interesting is the names mentioned, because it also says who mentioned it.
Speaker8: And so on.
Frode Hegland: So if you want me, Brandel, to change the. Oh. Yeah. Danny had to go. Sorry, I didn’t even get to wave. Bye. If you want a different prompt that we can obviously change that for your analysis or visualization.
Brandel Zachernuk: Yeah. No, I mean I, I, I’m, I’m pretty distracted at the moment. I am making a tech talk and that’s kind of a meat grinder, but the but it seems like it would be really interesting to. Yeah, to iconify the regular suspects and to identify ways of pulling other people in. And turning it into a graph. And certainly, certainly a number of those moving parts are better. Better addressed by this summary than other modes. So yeah, that’ll be fun to be able to play with. Yeah, I won’t have I won’t have the capacity to play with this for another month or so at least. But it does look cool.
Frode Hegland: That’s fine. And we’re kind of slightly running out in this context. I’ll just show you two slides. Guys, you can all see this.
Speaker8: Okay.
Frode Hegland: So one of the frustrations I have with vision and any headsets is touching things. The eye tracking and pinching for a lot of stuff is fantastic. With all the stuff, it’s not very good. And I think that’s a limitation of the universe rather than necessarily Apple. But anyway, so in these two slides this is on a mac. On top there are some tabs. All they are are saved search terms. Nothing more fancy. But imagine you click on those or you type in to search for the word hypertext, for instance. And you get this. There’s two aspects to this. One is the way it’s listed. So it shows authors and then the title is a link to open the document. Below that is a one sentence AI generated summary. Then you have a list of keywords names and if there were any highlighted texts. So this is an experiment in what could be basically useful to decide what to do. But the reason I’m showing you this today, and the new thing that I’m trying to build, is that in this search result, you can select text. And if you do command F, it’ll search anew based on that text. I think that can be quite powerful. So let’s say you’ve searched for this. You come across David Orchard, for instance. You have no idea what that is or who that is. Select it. Select that text or command F and it does a new search based on that. So you can keep navigating through the information. I’m hoping that for our. Hang on. I just want to make sure I’m back with you guys for our own meeting record. We can do something like that. But more interestingly, in XR. You come across something and you pivot around the whole view based on that.
Speaker8: Right.
Frode Hegland: If, as has been asked before, if anybody has any ideas to write down a user scenario of specifics or to for specific attributes the ideal document should have, please tell me.
Speaker8: I’ll add.
Frode Hegland: It to our web.
Speaker8: Page.
Frode Hegland: And also, who here has interest in working more on the how to visualize our own record?
Speaker8: Hello.
Frode Hegland: So Peter sounds good. And Angela says not for the next few months and Fabian next month. Next. Next month. Okay. And Fabian says something. Isn’t isn’t that aspect interesting for you?
Fabien Benetou: But I’m not sure I understood. You mean our data set as like the discussions? Yes. No it’s not. I don’t, I don’t think I don’t see how to visualize it besides, like some back and forth, and I’m. I’m not sure what could be beside manually doing it. I mean and that’s like crafting a completely unique experience more than a tool, so. Can. I’m not sure how to do this. It’s not. It’s it’s a data set in the sense that collecting any information can, if it’s done regularly from the same sources, is a data set. But. I don’t see how to do this. I can put like literally the transcript back and forth and, but it would be up to the user to then. And it’s too big. Like I don’t think it’s feasible. So that’s why I stayed silent.
Frode Hegland: Okay on that topic. First of all. If we could solve the meeting records issue, we would have $1 billion company on our hands. We all agree on that. I’m half joking only. It’s a big issue for a lot of different groups. Just doing it in text is not ideal. The current way we have it is absolutely not perfect. But I can tell you that I think it can be. A tool or an environment that can be generalizable, that can imagine something basic like you have a wall of the speakers. You could even put pictures on them if you wanted to, to make it custom for yourself. There would be one fun thing. So you walk up to one speaker, and then you can interrogate the whole data set based on that person.
Speaker8: Or two persons.
Frode Hegland: You know, that kind of stuff. I could imagine having a magic carpet like Randall did for us, where different attributes mean different things, and you have that available. So in other words, doing a lot of AI, not just 1 or 2 prompts, but doing a ton for a report stage. Let’s say a report stage is every six months. So you can ask it. Based on this. How so and so changed has this and that changed and then to have waste going back up at it. But I think the key will need to be getting molecular stuff out. In other words a name or a date. So we can do lists and interactions rather than false sentences, which requires more reading. Does that make sense?
Brandel Zachernuk: Yeah. No. So I think I think it is the more I look at it and the more I think about it, the more it would be possible to do things like, you know, the very first thing is to, to take the participants and give them a, a frame you know, in a picture and encourage that to be like, like get them for anybody who’s recurrent and then for names mentioned as well, to be able to build a graph of that. And just looking at the agreements and disagreements. Yeah. I mean, the challenge, like Fabian said, is that a data set is a data set. And whatever commonality there sort of is to any given set of data sort of defines the perimeter around which you can kind of use a formalism that that is relevant to the totality of it. Like if it’s all common enough in a, in a certain form, then you can come up with some kind of solution that that visualizes it. And so if we have these, you know, if enough of this is recognizable for enough of these meetings, the suggested next agenda, the chat logs, the agreements and disagreements, what do we have in other day of other week’s new understanding? New understanding.
Speaker8: And you understanding.
Frode Hegland: Might be the weakest one, but it was fun.
Speaker8: Yeah, yeah.
Brandel Zachernuk: And to the extent that so, like, you know, I’ve done a bunch of stuff in the past with not even AI generated, but just Google searches, Google image searches of search phrases and things like that. And, and those are sort of similarly valuable for a prompt of sort of being able to conceptualize things. And so, yeah, I think those are that that would be an interesting thing to be able to To cover. And then there’s, you know, data that is more trivially like extractable from the chat video. Like the total duration or the Number of speakers may be. I mean, it would be really nice if zoom gave that to you. It’s it strikes me as though, as, as the sort of thing that zoom should be able to provide a lot more support for, and it’s frankly suspicious that they don’t. But yeah, like Yeah. I just haven’t, haven’t really put my shoulder against the problem yet. So I, I think there’s a lot that could be done with this document. There’s a lot that could be done with what zoom should, by rights, be able to provide by default. So you know, the individual speaker time, cadence and order of people it may not have a like zoom will have a perfect record as in not even in dispute of who spoke for how long and in what order. What you know, what becomes a little bit more tenuous is what they said. And more tenuous still is, for example, the relevance of one person’s statement to the next or the tenor and tone of those statements. And so if somebody is just engaged in flat out contradiction with somebody else, sorry if I’ve done that in the past then, you know, that’s a relevant kind of data point to be able to kind of aggregate the overall tone of the, of the meeting.
Brandel Zachernuk: So you can kind of go from the bottom up in that way. But some of these top down things that the I summary has are pretty cool to, to be able to pull out as well. The fact that it exists as a YouTube video means that there are potentially things that you could get out of that as well, if there’s something that happens to have been somewhat more traveled in terms of the analytics across it, then that would be fun to be able to pull out too. But yeah, no, I think I think that it’s that’s an interesting data set to be able to kind of walk over as an exemplar. I’m with Fabian in that Beyond it being a data set that happens to have some characteristics. There’s nothing. In particular that I would expect to be revelatory about doing this work. Other than doing a better job of of reading a transcript in summary, than people have done in the past. And that’s appealing. But I, I like to just make a lot of interesting half finished stuff and move on. And so I, I haven’t generally been in the business of sort of poring over the, the exhaust from that process. So I don’t know whether I would expect to get great revelations from being able to go over the transcript. But willing to try at some point, like I said, I will. I’m a little busy, but I will get get to it with whatever time appears between now and
Speaker8: Like for weeks.
Brandel Zachernuk: And and certainly we’ll have more time after.
Speaker8: So I think what you.
Frode Hegland: Said there, Brandel, is important.
Speaker8: And I spoke.
Frode Hegland: That sentence partly as a test for Claude. Because now we’ll see if in the transcripts it points out that what I think you just said was important. I’m doing it as kind of a half joke, but also we discussed quite a while ago having a grammar for a normal transcript. Like we, we use the word important. So I think that prompt engineering, as they pretentiously called it, is going to be very useful here that if we. It went to highlight. If someone expressly says, I’m not interested or this is important or whatever, we can make this I be very basic, which I think would be beneficial rather than to try to make it think, which is kind of our job. So both your and Fabian’s concern about overanalysis, which may be wrong, hallucinating or incomplete, completely agree. But I don’t think we should throw this particular baby out with any particular amount of bathwater.
Speaker8: So.
Frode Hegland: But my background is that of an artist, so to speak. Chelsea School of Art and all of that. I’m used to vagueness, you know, painting, you know, like this and getting something. I don’t necessarily need needs in this sense. So my approach to AI is that they are that they are rough tools. What you get back is literally black and white text, but they are still rough tools and we should treat them as such. You know, like try to get an impression here. But I really want to reemphasize the importance of you guys being able to email me a paragraph either saying that was wrong and I’ll delete it, or here’s what I really meant to say, and I’ll put that in, because the benefit of HTML obviously is the linking. So we already have Andrew’s demos linked from our meeting record, which I think is quite fantastic for any demo you can see it and if there’s demos being done in the in this, if we just put a link in our chat to a video of it or that demo that’s going to be on the list for that meeting to automatically. That I think is going to be useful. Fabian has done quite a few demos which have not been captured in this, because this one has no images, it’s just text. And if there’s links that will be there, so that’ll be useful. Anyway, something to experiment with. And I could imagine, Fabian, that in your job, dealing with a political reality, that this is something that you imagine half the people you work with would love, half the people would absolutely not love it at all. But for the public to be able to have summarized sessions of the Parliament that you are part of, vetted by the people who were there in the future, at some point may be useful, may be scary, but the key being able to always have access to the source.
Fabien Benetou: So, I mean, I don’t want to talk about this because I don’t think it’s feasible. I think the need for it is there. Of course I agree. The thing is to do it. Where there is no hallucination. As for my technical understanding goes, even though it is not my specialty. That’s not how the technology works. It’s even like summarization. We don’t know as far as I know, besides having a human being. And it’s exactly the question I asked you on slack a couple of days ago when you shared those transcripts and summaries, because most of it sounds right, which is already amazing, but it’s a world apart from being technically being truthful. You like all the time. Because if it’s totally different, if I read the transcript and then oh, fraud said this and you actually never said this, and I have to every time I read a single line, say, is it is it actually what was meant? And I just cannot go back and forth and ask you every time. And then every time I’m in a kind of in between states of like, maybe he said this and maybe actually you didn’t say it, but it’s still interesting.
Fabien Benetou: And my interpretation of it might lead to something useful, but it’s so there is a need for this. I’m not questioning that at all at the Parliament and elsewhere. Everywhere. The problem is that I don’t know professionally how to make that work. And my understanding of it and the discussion I have also with experts is that we’re we’re not even closer than when we were ten years ago to to this. So it’s it’s it sounds plausible and and it might even be feasible. I’m not saying it’s not feasible at all and or never will be. I personally don’t know how to do this, so I’m not going to promise this to to you or anybody else. If it does become possible that the summary is all the time, we’re like 99.9% of the time the shared understanding of how humans who did participate to the discussion and have like it would be vetoed, that that would still be interesting, but it’s still a risky situation. So, sure, the need is there. I just don’t know how to do it.
Frode Hegland: You use the word summary? I think that’s a dangerous word because that is not what this is. Some of it is summary. And I do also have a lot of grains of salt when I touch that. The two things I think are really important. One is the thing of the loop of each of us checking after each meeting, when it’s relatively fresh, to see if something egregiously wrong. The other thing is, if you look at this, a lot of it is entity extraction. And I think that’s useful. Like if you look at our Monday week ago, Ted Nelson mentioned by me in the context of Zigzag, I said that. David Bellos, right. Douglas Adams, Vint Cerf. Blah blah blah. These are really useful things. And there is not a lot of AI involved. It’s just these are entities that are listed under disagreements and new understanding. Yeah, that’s a lot more nuanced of course. But if you look at the speaker summary.
Speaker8: It’s pretty good.
Frode Hegland: I mean, look at the one for you.
Speaker8: Yeah, but okay.
Fabien Benetou: Just to go back to to repeat also what Brendel said and to be a bit more direct. I like to play with things in space. I’m not saying everything else or anything else is not interesting. It’s just my it’s not my focus at the moment. So. Sure I’m not. And I’m not suggesting to throw anything with any things water or baby’s bath, but it’s just not my focus. That’s it.
Frode Hegland: No, no, I understand that. And I’m not going to try to ram it down any further throats at all. And the only thing I was referring to was the veracity of the the record and how that has to be changed and be entities are simpler. I do, however, have a feeling that for the entity extractions that can be playing in space, and I can imagine that, for instance, maybe the topics become instantiated very similar to your knowledge blocks, that they have programmatic characteristics to them and to be interactable in that way. Right. So I’m looking at both people in space, a record, a diary in space, but also the concepts in different ways, concepts slash entities and what you’ve done. That’s a provocation for me in a good way. But we’ll see what happens in the future. Maybe we’ll record the weekend you’re here in London and you’ll say, I’m desperately need to visualize this.
Fabien Benetou: So I’d like to visualize stuff and I’d like to think that I have structure, but it’s not like I don’t have a visual manipulable summary of it in my mind. So this I do. But it’s like somebody who is not me. If you give me, let’s say, a graph of a text or some content with, for example, entities that have been extracted before, ideally validated by, let’s say, an expert or the domain but doesn’t. It’s okay without I can do something with this, but with just the text as is now and timestamps for example, it’s it’s a couple of levels to row for me to do something meaningful with it. So.
Frode Hegland: Yeah, no. That’s fine. That makes sense to an extent.
Speaker8: Well, that was under.
Frode Hegland: Leaving, right? Right. Now also, finally, I’m sending the document to Apple regarding my experience with the Vision Pro. The two main things I’ll be mentioning. One is, please, can we have some window management, please? Can we have some way of storing layouts because they get lost after a while, which kind of defeats the purpose. And the other one is when there is a trackpad attached. Please, please, please override the I, at least within that area. Too often I’m writing something and suddenly I’m writing in a different location. You know, these kinds of things. But if any of you have any other specific things you’d like to add, please do tell. And Brandel, I’ll be not be involving you because you’re already over involved in our group. I’ll be doing it via Bruce. So it has a different way in it, unless you want it done differently, of course.
Brandel Zachernuk: No, I think that’s great. I think Bruce presumably also has a fair bit of clout in terms of being able to slip things to folks. I’ve bumped into him the other day, but he. I don’t think he recognized me. And it was a I’ll say hi now that he’s actually in in our group properly.
Frode Hegland: Oh, yeah. No, you should say hi. I think he’s got a lot on his mind at the moment. So whenever I say, should we do this? Yes. Next week. So I can imagine he’s walking around with too many thoughts to see who’s in front of him.
Brandel Zachernuk: Yes. That’s believable.
Frode Hegland: So now for the record, I look forward to seeing the important bits and see if Bruce Horn was is going to be listed on our as one of the named entities. See if that works. But we are excited about the notion of the ACM library, I hope visualized. And if there’s anything else anybody wants to work on that’s related to this at all, please let’s discuss that and do that.
Speaker8: And hopefully in.
Frode Hegland: A week I’ll be able to show you a better version of Reader in Vision. Currently we can lift stuff, but it doesn’t appear right. That may address a little bit of what Dean is talking about.
Speaker8: And so on.
Frode Hegland: Yeah. Brendel to on the vision to get it into kind of fix itself mode. What do we do. Press both buttons for a long time or something. Because I had to do that this weekend and I accidentally found out how to do it. I just don’t remember how.
Brandel Zachernuk: Do you mean putting it into force quit or DFU?
Speaker8: What’s DFU?
Brandel Zachernuk: Device firmware update. It’s where you intentionally break a device in order to be able to force an OS onto it.
Speaker8: No no, no.
Frode Hegland: Basically, this weekend, when someone testing on people, they did the environment and was just black.
Frode Hegland: And it just didn’t function right. And also, the OS was filling up the entire 512 gig with nothing. So it just didn’t work. And then I pressed buttons and finally it said please remove while in maintenance mode or whatever. And it did all kinds of stuff and fixed itself.
Brandel Zachernuk: Okay. That sounds like it was resolved. I’m not familiar with that particular mode. I tend to just power cycle. On on unhitch the the cable in order to reboot. There is a way to do it. I believe without and possibly just doing some buttons, but I’m not familiar with it, so and there is a thing that is a public facing thing for all Apple devices called device firmware update, where you do some complicated sequence of buttons, and that will force it into an update where it will be able to accept an OS update. I’m not familiar with what that looks like for for Vision Pro, and I don’t know how you would do that without the developer strap. So
Frode Hegland: Yeah, I do want to develop a strap that also means faster connection when doing a virtual Mac display, right? Faster you connect.
Brandel Zachernuk: I don’t know what it means for the public.
Speaker8: Okay, that’s fair enough.
Frode Hegland: Okay, we have to close up. Fabian, considering all the discussions today, what aspect of this work, the Sloan related work do you consider most relevant and interesting to you?
Fabien Benetou: Getting Denise step. Like how she is going to grab a document and manipulate it. With her bibliographic tool, like a Zotero equivalent. And see how I don’t want to. I don’t want to presume too much, but to me, that would be the the thing where that would be most interesting for me. And I would hope I can do some most interesting prototypes.
Frode Hegland: Exactly what do you need from Dini? Just to have it really clarified.
Fabien Benetou: She sit down at 8 a.m. or whatever in her office. She has a goal for the day. So of course, everything that’s not related to research directly, like writing a grant or whatnot. I mean, that’s of course, core to her work, I imagine or same meeting students, etc. but it’s not manipulating the type of academic document I think we’re interested in. So we skip all this. But let’s say if she has to produce an academic document write an article, for example, for the upcoming conference, what are the steps and what are the steps with reading existing documents or finding those documents and then extracting or highlighting or adding some any kind of annotation? Does she do it over a single screen? Multiple screen? I think she said multiple devices. So basically. As if I was behind her shoulder to see how she was manipulating academic documents and library of academic documents like her own, how she collected papers over the years, and if she browse sciencedirect or whatever, or anything really, or physical magazines. Also, if there were publication in her work that are still in that form, and how she then put those different documents or part of documents in a space, maybe a directory and then how she go from this to an ID, how she writes the ID down like a sequence of words, maybe drafts or and then insert them one next to each other for example, to, to have the different sections and all the way to finally publishing the paper. I think that’s, that’s what I would see. And then. We would discuss to see. Are there a couple of those steps that would be the most beneficial to her to do with her headset? And and then finding the easiest, cheapest, fastest way to test this. I can be taking the cover of each paper and then displaying it on a virtual whiteboard. And then she tries it and it’s like, no, it’s not at all what I wanted. And then we try to iterate from there. That’s what I would need.
Frode Hegland: So on Wednesdays, one of the slots is exactly that. It is use cases. Will you be here at this Wednesday?
Fabien Benetou: I’ll try. I cannot say for sure. Let’s say if if you tell me that she can address that and to those questions. I would I would try even more to be there. But I can’t promise anything.
Frode Hegland: Okay, but feel free to ask her exactly this and exactly your words, because it’s very important and it should not be more generic use case. All right. Thanks guys. I look forward to seeing you Monday Wednesday whenever you can. Bye for now.
Fabien Benetou: Bye bye. Take care.
Chat Log:
16:01:47 From Peter Wasilko : Brunching in New York!
16:02:40 From Andrew Thompson : My sound isn’t working, be with you all shortly (once I fix this)
16:03:18 From Peter Wasilko : I’ve been head down in a sea of code.
16:04:19 From Hrithik’s OtterPilot : Hi, I’m an AI assistant helping Hrithik Tiwari take notes for this meeting. Follow along the transcript here: https://otter.ai/u/fQh8R73spWNk-cQRuOl_NLl4WxE?utm_source=va_chat_link_1
You’ll also be able to see screenshots of key moments, add highlights, comments, or action items to anything being said, and get an automatic summary after the meeting.
16:09:08 From Peter Wasilko : Apple needs to have more seniors involved in its development and testing.
16:10:43 From Frode Hegland : https://futuretextlab.info/ideal-document/
16:11:07 From Frode Hegland : https://futuretextlab.info/ideal-document/
16:13:54 From Peter Wasilko : https://www.youtube.com/watch?v=5GaPgYhQ8Rs
16:14:27 From Fabien Benetou : Reacted to “https://www.youtub…” with 👌
16:16:45 From Leon van Kammen : Frode, does the text make it clear whether you expect these features to be implemented in author/reader or on a (Vision) OS-level
16:16:58 From Frode Hegland : For the document?
16:17:03 From Leon van Kammen : yes
16:17:04 From Frode Hegland : Or for the email to Apple?
16:17:14 From Frode Hegland : I do not care 🙂 This is dreaming in general
16:17:24 From Leon van Kammen : ah ok no worries
16:19:37 From Leon van Kammen : Peter, you seem to be hinting at a ‘networked/multiparty document’ (when you were talking about being able to see who’s reading along)
16:22:40 From Frode Hegland : https://www.dropbox.com/scl/fi/jeel8fmksxzz0cbb6u5mq/FTL-Meetings-AI-Record-2024-Frode-Hegland-2024-04-29T15-19-25Z.pdf?rlkey=l9ryk83d75aig5n1ves8otrf3&dl=1
16:25:52 From Leon van Kammen : can you repeat the last questin?
16:26:01 From Leon van Kammen : there’s a child screaming here in the background
16:26:06 From Leon van Kammen : 🙂
16:30:26 From Leon van Kammen : has there been any reports of AI infographic generators? An index page could also be literally a ‘visual map’ (with pagenumbers)
16:36:19 From Peter Wasilko : It needs Knuth-Plass line breaks!!!!!
16:37:33 From Peter Wasilko : And micro typographic enhancement like protrusion of line ending punctuation for an “optically smooth” column edge!
16:39:16 From Leon van Kammen : should the ideal document have pages?
or an unlimited canvas?
16:39:43 From Fabien Benetou : bead crumming?
16:39:46 From Peter Wasilko : And it should have a Murder Board auxiliary display as a visual multi-clipboard for brainstorming.
16:40:10 From Frode Hegland : Every component should know what it is and how it relates…
16:40:20 From Fabien Benetou : thanks
16:41:07 From Peter Wasilko : Do we want to have “virtual highlighters”?
16:41:08 From Leon van Kammen : the power of ‘webpages’ is that you can distribute pages inside a webpage, or distribute them across different webpages..which can also be confusing for author
16:41:18 From Frode Hegland : Addressability is really important. At least paragraph addressability
16:42:03 From Peter Wasilko : I’d want all embedded quotation’s to be directly addressable.
16:42:56 From Rob Swigart : A weird idea might be a headset version of a memory palace, say Versailles, where concepts belong to a room that enhances the possibility of remembering the content.
16:46:17 From Peter Wasilko : So “Frode said that ‘I think we need to make it incredibly compelling’.” would let me grab a logical quotation object with {text: ‘I think we need to make it incredibly compelling’, attribution: ‘Frode’, id: ‘q31’, attributor: ‘Peter’,in-doc: ’transcript-4-29-24’}
16:47:00 From Fabien Benetou : I can show sth but recording off
16:47:37 From Frode Hegland : Reacted to “I can show sth but r…” with ❤️
16:49:02 From Leon van Kammen : perhaps the document should allow hasslefree free editing in all 3 dimenions…but also be able to accept a grammar to store and structure future content
‘bullet points’ is basically a hardcoded ‘grammar’ in current documents, we need a loose coupled one.
16:58:15 From Frode Hegland : Seen it, not read it
16:58:59 From Fabien Benetou : for ref https://en.wikipedia.org/wiki/Arrival_(film)
16:59:34 From Fabien Benetou : maybe https://en.wikipedia.org/wiki/Story_of_Your_Life but I didn’t read it
17:00:20 From Leon van Kammen : I have go unfortunately, but thanks for the thoughtprovoking call 👍
17:01:03 From Frode Hegland : How about ACM Hypertext ’24 Proceedings in XR be our main goal?
17:07:23 From Frode Hegland : Dene, do you think you could collaborate with Marius to get us all the papers a little bit before the conference? Even a day or two, to slot into a system
17:09:58 From Peter Wasilko : Excellent!
17:10:45 From Peter Wasilko : Is there a raw JSON version of the document?
17:24:57 From Dene Grigar : brb
17:30:32 From Peter Wasilko : Sure!
17:31:47 From Frode Hegland : https://futuretextlab.info/ideal-document/
17:34:06 From Frode Hegland : You sound perfect, I didn’t even notice
17:34:58 From Dene Grigar : I am going to get ready for the next meeting and the PhD examination taking place in 1.5 hours. Bye folks!
17:39:19 From Peter Wasilko : Sounds good
17:48:00 From Frode Hegland : ‘Important’ as keyword, as we have discussed before…
17:49:05 From Peter Wasilko : We might want to standardize on other keywords like say PCR for Popular Culture Reference eg. PCR Owl
17:52:15 From Frode Hegland : Fabien Benetou demonstrated his work on representing knowledge using manipulable objects and grammars in 3D space. He showed how different types of objects (e.g., post-it notes, code blocks) can have their own behaviors and interact with each other based on predefined rules. Fabien expressed his belief that most domains of knowledge would benefit from being represented in this way, particularly for learning purposes.
17:52:22 From Frode Hegland : That was you a week ago 🙂
17:52:38 From Frode Hegland : I think the loop of each of us checking after a meeting is important.
17:55:44 From Andrew Thompson : I’ll have to head out for today, see you all Wednesday. (I’ve got lots to work out with displaying the main document content, its been more complex than I initially hoped)
18:03:33 From Brandel Zachernuk : I’ve gotta drop for the next one – thank you, sorry for the slow start today!