Frode Hegland: Get a piece of paper and suddenly we’re stupid, right, Mark? Right. Okay. Please continue. So yeah, that’s.
Fabien Benetou: Again the movement. We move, we give affordance to things that otherwise I mean, they do have affordances, but they are the usual one. But then again what is that thing we give affordance to. Is it a paper, a paragraph, a single word. Is it. A random arrangement of those at the same level. Is it completely heterogeneous? And I think it’s a hard question. And it’s it’s Yeah. I personally don’t have an answer. Of course, the easy answer is yes, we can manipulate everything, but then if we don’t have infinite amount of resources, that seems challenging. So. And that I believe just to to give my perspective on this, is by, for example, interviewing, let’s say Dini on what’s the workflow. Then we can maybe try to find not only the unit of knowledge, but the one that is useful to her in that workflow. And then eventually we can build on this. It’s not finite. It’s not also once and done forever. But yeah, practically speaking, what do we manipulate when we’re in XOR?
Frode Hegland: One way of answering that question, and it’s also flippant but not flippant is. Knowledge is stuff put strung together. It’s a sequence that knowledge can never really be. Just one thing, not one word. So a basic thing, of course, is the grammatical sentence. So that becomes complicated. Houdini is not joining us. Hang on, so I can. Just have to give her a bit of an update. Boom. Okay. Dini dealt with Didi. Yeah. So the notion then becomes what’s stringing together? Is most useful for our use. If we look at the stringing together of papers into a journal proceedings. That is interesting and good. But what Andrew and I talked about the other day was, okay, so you have a stringing together of documents as a virtual document in a library, which is a stringing together of documents. So how to provide the affordances to the user of where they can literally pull something out and at what level that means. There’s going to be important for us. It’s like chemistry, you know, you have electrons, protons and neutrons are probably words. And then you have atoms and molecules and then you get higher forms. So something to think about. And Mark I’ll give you the mic in a second. But I just want to pre agree with you that we must be very careful not to be overly concrete on this.
Mark Anderson: Oh I, I yeah, I’ll just explain my, my comment was to say wasn’t meant to be flippant. I mean I think this is actually reflected in the comments since in terms of, you know, what Fabian said, it’s difficult at the moment. I think the one thing we can do is we can we we can certainly begin to look at some of the interactions without being too set on what they’re attached to. We know we want to sort of move things around, and we want to interact with them. Quite how far that needs to go, I think, will be a merchant. I mean, my sense is that actually it’s going to be quite dynamic too, which makes it which which fits badly with nice sort of documented structural breakdowns. Because what really happens is when I’m doing this project, as opposed to that project, the way in which I want to tease the information apart will probably differ. And what that says to me is it’s probably something I’m going to need during the session. I’m going to have to interact with the tools to sort of say this is in a sense, this is the granularity at which I want to work. Or I might, within the session, say, I now basically want to be working in a different way, because I had an interesting conversation with Adam the other day to do to do with some other visualization work.
Mark Anderson: And one of the things we were thinking about was that essentially one of the things you sort of end up doing the moment you talk about sort of pulling things apart is you’re probably making a new list, a new set of stuff in the loosest, not in the mathematical, the loosest possible terms. You’re making a group of things from from some source material and quite what that becomes. We don’t know yet because we haven’t really done enough. So again, it’s it’s hard to typify what it should be, but the underlying activity seems to be XR gives us the opportunity to tease apart information in a way we broadly had to do on a piece of paper or in our head before. And so that that poses sort of three questions is, has been raised, the granularity the second is how we interact, how we actually sort of literally get hold of the right bit. And then the third one is, in a sense, what we want to do with it. And an example of that is, is what Freud’s already mentioned, which is essentially for instance, you might be wanting to construct a new binder of information based on some notion that you have whilst working in that space at the time.
Frode Hegland: Yeah. And this is why, I mean, the notion of high resolution thinking first was a bit of a joke. You know, I thought it was a bit cutesy, but it really it really comes into what you were saying, Mark, that we should be able to. Okay? Okay. So many, many years ago when I was a student at Syracuse University in the mid 90s, yes, such a thing happened. I was a teacher for this graphic design lady. How to use the Mac. And to me it was already so intuitive, it was actually hard to figure out how to tell her how to do it. But it came down to the most fundamental and obvious thing, which is easy to forget. And that is, you got to tell the computer what you want something done to and what you want it to do. And that can be as simple as opening a file. What is the thing you want to do? This one. Well, now you need to indicate to the computer. So I think all we’re really talking about now is having a really rich set of stuff that we can tell the computer we’re working with. So that means that when Andrew is back in a second, I think we should spend a little bit of time talking about. His perspective currently on mapping in XR and Mark do not allow us to finish without settling down to go back to spend at least half an hour on the paper. But I think that really going in this direction will help us better understand what the paper might be. Yeah. Also to Mark and everyone else, they’re meeting with Dave Miller, of course, is always wonderful. He is actively trying to get some research to support working with us on this, because we like to build the things he likes more to figure out what is this thing. So it could be very interesting. Parallel research. That Mark.
Mark Anderson: I just looking back on on what you’re just saying in terms of the so the chicken egg problem we have for, in a sense, is you were noting the sort of the work that Andrew is trying to do. It is, is to sort of it would be nice if we can say these are the things we want to work with, and therefore this is what we ought to build in it. The difficulty is that most of most of what we, we, we, we can do stuff with, we’re actually at the moment almost having to hand build for perfectly good, good reasons. So I don’t have a problem with that. The sort of challenge is that it’s an iterative process, really, until we’ve. So we have to build some of these to know what will even work. And then, interestingly, I think you’re absolutely right in saying we’re going to have to find a language, even if it’s sort of facial gesture or something, or explicitly stated or typed as to what I’m doing. I. Because what I think I’m beginning to see is that our presumption is that, as we often think with new technology, well, it’s just going to work. You know, it’s going to be just be like the food replicator on on Star Trek or something. It’ll just it’ll just know how to make something terribly complicated. In fact, I suspect that’s really not quite the case. Leaving aside things like miss detection of cues, but just the computer, us signaling to the computer what our intent is. And the fact that even if we do have a, a quite well formed structure of what the various addressable units of information are, we still probably need to tell it in some way or, you know, make a selection that at this point we’re going to have we want to have an interaction with this sort of level of thing. Right.
Frode Hegland: So one thing while we’re waiting for Andrew one thing that I also talked to Mr. Millard, Professor Millard of whatever last Friday was LPs and I, and I know that some people in this community are quite reserved and their enthusiasm towards what they are, and I think I don’t mind that you are to an extent, but I also think we need to get past that. And I think Peter has solved that problem for us with the whole notion of deletable appendices. Right. So I am not somebody other than thinking it’s kind of cute when Claude does our transcripts and gets good things. I think that’s nice. That is not, in this context, important. What I think is really, really important is two things that are back to back, stuck together with glue. Entity extraction and being able to get rid of them. But so everybody here should be able to do have their own Llms or AIS or whatever the heck we want to call it on any of these data sets do all kinds of stuff when you ingest a paper or other information. And it should be ingested in a manner that if you decide that LM is rubbish or you have a new one, or you just don’t want to deal with that, you get rid of it because the affordances it gives us through having connected stuff, I think is amazing, provided it isn’t kind of baked in there. It’s okay, I’m just going to pick up. I go say hi.
Speaker5: Hi, everybody.
Frode Hegland: Hi. Yeah. So when I phrase it like that, do we agree that we should use the whole I thing, but really in a box at a distance? Fabian, over to you.
Fabien Benetou: Yeah.
Frode Hegland: Well, here’s the thing. I’m really, you know, I’m reading this book called possible.
Fabien Benetou: Okay. Sorry, sorry. Just just because I. You interrupt me a few times, so I dare do it now. Give me a useful I, for example, text in entity extraction at the end, and I happily use it. It’s like I said before, let’s say, about transforming a piece of content to another. That’s this. I’m honestly. And I do it like I don’t. I have a page on my wiki with a bunch of different I including LMS in it, so I don’t want to like, just how do you say whine about it? What I don’t want is like saying I magic and it can solve every problem because I think beside the hype in term of like epistemology, pursuit of knowledge, trying to get stuff done, it’s not useful. But again text in or whatever pdf in and then entities out and we can play with it. I’m 100% in. I’m just not going to pursue things that are too high because I don’t think it’s productive.
Frode Hegland: Oh, yeah, I think. Fabian. Where exactly? On the right on agreement. But don’t forget Alfred Sloan himself. The guy who was funding some of this. He didn’t like agreements too much because it’s lazy thinking. So for us to go through this is really important. I see how Mark saying, does the nature design of lamps realistically allow for removal of entities? Yes, Mark, of all people in the whole world. Okay. If you were sitting next to me, I’d have to ruffle your head right now. Depends how it’s stored. If it’s stored as visual metal. In a separate appendix.
Mark Anderson: Appendix no. The model. The model. The model is the sum of all the stuff it knows about. It doesn’t dynamically, each time you ask a question, go and read everything from scratch. So there is data absorbed into a model. What I was, what I was referring to in this is that I. My understanding is not, not not through anyone’s intent. So basically people have been trying to make this stuff to work, and we see the wonderful things it does at the moment, the sort of things that they, they haven’t been trying to do thus far is thinking about taking something out because one argument is, well, why would you take anything out? It’s all interesting knowledge. What we’re suggesting is, is that, yes. Well, in some circumstances you want to have you actually want to have more control over the input. But by the time it’s got into the LM, it’s sort of it’s too late for the purposes for it to work for the purposes you want. So I the question is, is open and I, I suspect it will come. But my understanding at the moment is that large LMS are not built to, to facilitate this sort of dynamic extraction of data. You know, you can’t just go to Claude and say, forget the date of the Battle of Hastings. That’s just not how it works.
Frode Hegland: Well, but also when you’re talking to a human, you wouldn’t expect that. You know, it’s interesting what you’re saying, but you wouldn’t expect that. You wouldn’t say to that human, do you know something about this topic? In which case, I can’t trust you to an extent, just to be a bit provocative. Right. But the human.
Mark Anderson: Doesn’t act like the computer. Part of the problem is we’re treating the LMS like people and they aren’t. And we I, I see a lot of I mean, and it’s not the thing is, don’t take this as a being a complete downer on the thing. I think this is the the onus is on us as the humans to to be responsible about how we use it and the thing that we’re we are as a, as a community, well, not us, but in the, you know, in the wider sense we are being irresponsible about is we’re treating it like general AI and it isn’t. And it and it, it, it, it has problems, you know, so it’s fine if it’s getting you the fastest local ramen or something cool. If it’s something like your next cancer treatment, maybe not so much. Because it doesn’t have the ability to it has doesn’t have notions of truth, doesn’t have notions of doesn’t really understand provenance fully. And that’s that’s not a canard against it. The state of the art is where it is, but it just isn’t like a human yet.
Frode Hegland: No, no, it’s not like a human, which is, of course, one of the biggest benefits of it. I mean, have you seen the protein folding? It turns out that generalized AIS are now actually better than specialized AIS. So I think that’s an interesting analogy, but we’ve been talking a lot. Peter, say something happy.
Peter Wasilko: Okay, well, there are a couple ways that you could go about performing the task. One, if it’s looking for, say, say we’re trying to do a place name extraction. If it’s. Approaching it from the problem of what place names have I seen before? Imagine it was trained on a set of Scottish geography, so it’s going to be really good at recognizing plausible names of Scottish towns, but it might blow up if given a name from New Zealand. Or it could approach it from what is the language used around a reference to a place name, in which case it would recognize patterns of, you know, I drove from to or my cousin lives in kinds of phraseology, in which case it wouldn’t matter what the location and make up of the place name is, it would be recognizing, again, basically looking for a hole in the discourse. And when it found something filling in the hole, it could plausibly conclude that it’s a place name based upon that linguistic pattern, as opposed to, again, starting from a list of names. And then it might be really good at recognizing men’s names, or it might blow up horribly at ethnic names because it was trained on Indo-European data set. And you’re now giving it a document that has names originating in Africa or some other area, and it’s going to blow up horribly and become highly sensitive to what its training set looked like. So this sort of need to tease out how it’s approaching it. And of course, all the AI companies treat it as a black box, and they won’t tell us whether it’s looking at patterns of previously identified names versus patterns of linguistic construct.
Frode Hegland: Yeah. Look, we. Absolutely worthwhile issues. Something to discuss, but I don’t think it’s that core to what we’re doing for the reason, Peter, that everybody’s getting into the weeds there.
Peter Wasilko: I just couldn’t resist.
Frode Hegland: No, no, no, I think you made very valid points. But I have you know, my first comment is experiment with them, see how you feel, see if there’s some that you feel are better than others. And to reemphasize, the ability to delete them is absolutely crucial. We do not want to pour this into a coffee. Right. And that obviously, because in the.
Peter Wasilko: Meta we want to be able to ideally to identify which I it was, you know, what model it was that this extraction was based on and some extra metadata as to how much you trust it for what kind of material it was extracting.
Frode Hegland: Yes. No. Absolutely. And individual meta. It does say that it also says the location and the time. So Andrew, the reason I asked if you were back was not for this excitement. It was for entirely different excitement. Our big, big thing now for I think most of the work we’re going to be doing is the map. So I’d like you to tell us your perspective on the data, the interactions, all of those goodies. Well, I’m not saying you should have anything prepared. I certainly didn’t expect it. I’m certainly not expecting you to have solutions, but I just really love to hear your general thinking of it now.
Andrew Thompson: Well, I’m not exactly sure what you’re asking me. I can sort of reiterate what I understand. The map to be. And that seems to be another space. It’s probably going to sit along with the library, but just render differently. So it’ll be like an alternate version or perhaps a second sphere. I’m not exactly sure what we want to do yet. And it’ll be a collection of essentially all of the the tags that we have. So it’s it’s tagging. It’s a bunch of names mentioned inside the document. It would be the different citations. So, and list of the different documents that are available and their titles. And they all kind of float there with different bits of information inside them. And based on that information, you can sort them in different ways. I assume we want the lines similar to reader. When you when you point at things, if it has two different documents, have the same tags, they’ll, they’ll draw a line or something. And then you want different ways to sort them. From what it sounds to me, this is almost the same as reader. Just. In the webXR environment which is going to add a whole new sense of depth to it. If there is something more in depth than I that I’m missing, please enlighten me.
Frode Hegland: Well, my DNA.
Dene Grigar: I just wanted to pop in and say, congratulations on your PhD.
Frode Hegland: Thank you.
Dene Grigar: You must be feeling good right now. And happy.
Frode Hegland: Yeah. No, this is nice. There’s a lot of things going on in the world, in my world at the moment, but yeah, this is definitely nice. And it’s also definitely nice in this group. Right now we’re talking about the kind of knowledge graph map thing. Also had a brief discussion on papers. We’re going to be talking more at least half an hour today on what should go in main papers and all of that. Mark, put some questions in slack so we will address that.
Dene Grigar: Okay, great. Well, I can’t stay very long. I’m just I have lots of things going on today myself. This is the last three days of my contract at the university. And then we go on summer leave. And so I’ve got a bazillion people writing me for the last three days saying, do this, do that. And then they can’t write me after Wednesday. So it’s a lot of stuff right now.
Frode Hegland: So yes, that’s absolutely fine. We’re glad that you’re here when you’re here. Andrew, to answer your question. I’m going to now do a quick screen share because that’s what I do. One second and one screenshot. I can’t see you, so please just interrupt if there is anything, okay? Because I think this is quite obvious. So this is just a mock up thing. But I think some, some of you seen it a few times. These are all the articles, papers from last year. So we have the first sentence of the abstract, a link, the names of the contributors and then the title. I’ve just made a defined thing so we can connect them. So in this we have an I summary. And then we have I extraction of keywords and names. Not all of them have names. This is not pure I, I’m very happy to go in and fiddle with it add and remove as I see fit. That is really, really important, right? We have one document. Sorry, one kind of a tag. We don’t call it tagging, we just call it. This is a. Right. And that is very, very important because when we go to the map view now. Well, I’ve never been crushed. Oh, yeah. By the way, as you can see here on Facebook, it’s Emily’s birthday. Now that’s all very nice. Right. So go to the map view. And it starts as a complete mess.
Frode Hegland: And you know, we’re not going to be spending the time doing all of this stuff. So that’s why it’s so important we have this opportunity here with hide and reveal categories. For instance, if I now bring back people, they’re all selected. I really want you as you are, to pay attention to this, because these are the kinds of interactions that I think are very useful. If you don’t think they are, we need to really think about that, because now currently I have to do this via keyboard shortcuts. But this should be on the prism control panel. I think now we have all the people selected. We can do this. Just align left. That will be absolutely crucial because otherwise, how are you going to make sense out of this? And now I select all the people again and line left again. And then I do vertical layout. So to be able to do volume operations on the same kind of thing is just absolutely crucial. I’ll do one more thing. Moving everything across. And then I do. As you can see here. Hide all the documents and then show the documents again. So here they are. If I now align them and now do this layouts. Of course they’re too big. They’re overlapping. But at least now we can begin to deal with it. Yeah. Leo, please.
Leon Van Kammen: Can you hear me?
Frode Hegland: Yeah, I can.
Leon Van Kammen: Okay, so about this you know, being able to do these multi selects is crucial. My question is are you sure. Because there’s also I also feel that certain actions are better in 2D than to translate them to 3D. And you know, are you sure this is not a use case or action which might be better in 2D?
Frode Hegland: What I’m thinking about is we need to dream our way into XR, and we need to also inch our way. So first of all, this is of course right now in 2D and I would easily say I need a bigger monitor. Right. So I think initially from this perspective and this is entirely up for discussion, we should build something that works on a very large virtual monitor. And then also under. I’m not saying explicitly in the time sequence here, but we can look at curvature versus a flat monitor. We can look at, you know, more cylinder design like we already have because like.
Andrew Thompson: Consistency, we’re going to want it to be curved. So the whole environment feels like it’s wrapping around you. I’m not sure why we would ditch that at this point. But it is worth looking into.
Frode Hegland: Yeah. I’m not saying we should ditch it at all either. What I’m saying is that I’m surprised at what seems to work and not work. So just as an interesting aside, on the iPad mini, I’ve been looking at the pencil. So the pencil works in author right now it’s just there. It works really well until it doesn’t. Right? Like how do you undo? Right. There are many things. I’m working on it for my own software. Well, what I’m saying is that it’s easy to think of a new interaction as giving more opportunities like XR, but it also really narrows it like the act of selecting by volume. Do we really want a pop up menu on the prism, or do we want something more elegant? I just think we need to build and test and work.
Mark Anderson: I’m thinking of that sort of that moment from Devil Wears Prada about, you know, where’s that piece of paper I had in my hand yesterday? Because I’m. Forgive me. Because I just don’t know the current state of the app, but can I make an ad hoc selection because my from the I mean, I use maps a lot every day in Tinder box have been doing it for, gosh, almost 20 years now. And most of the time I’m working with an ad hoc selection and so that a way to signal that now it might be a pre-applied keyword. So what it might be, in other words, before you even go into the map, you have some way in which you select the things you’re really interested in, because trying to do it on the map once it gets past a few items is not really tractable. I think the other lesson from using sort of 2D mapping spaces for a long time is less is more. So you have a problem here. You’re starting with way too much stuff, and just those top level filters don’t actually help you enough because they are unnatural constructs placed across the top. So the fact that something is an organization versus a person is very definable and understood, and it’s not wrong that you can identify them as such, but it’s not helpful because probably you’ll want organizations and people just not all not all of them. So trying to take a sort of a reduction out of that. So the challenge is how do you select pre-select the things that you actually want to work with, and then the rest can either not be shown at all or can be way off somewhere to the side. And because you might want it still visible because you might, you might in the moment say, oh, right, I actually need to I need to demote this thing out of the current working set, or I need to pull this, this one in.
Frode Hegland: Yeah. Andrew, can I take that or are you ready to bounce? But okay, you can do a few things already like this. Just shift, click and stuff. But your bigger point is very, very important. And I think that comes under the notion of being able to save views. That’s definitely one of them. And and on the more pedestrian side, which is really, really important, it should be possible to do some interaction where you hold down the equivalent of a shift key in XR. So I’m going to keep selecting. Now we may do that as a gesture. We have one hand doing this and another I don’t know. But key key is Mark that the concerns you’re showing there are super important. One more thing before I give it over to Andrew. And that is I got a prompt from Chris Gutteridge whom I saw on Friday as well. He wrote a prompt to take all of these titles and summarize them into shorter titles. There was quite a clever little thing, and that’s a view thing that I think we should definitely look at. But how can we use techniques of whatever sorts to make these things more scannable? Leon, you can wait till after Andrew Yao. Or is it really on this point? Okay. Andrew, go.
Andrew Thompson: Yeah. I just want to give a real quick reality check. Which I hate being that guy, but I’m the developer for this troika is super efficient with large chunks of text and not that great with a bunch of little pieces. And since we need every single piece of text here, just just like a word or two, because it just titles and names and tags, they all need to be able to move independently. I really don’t see a good way to optimize this so I can still make this. We can still test it. Great. But I already know it’s going to have a frame drop whenever this loads in. If that’s something that we’re okay with, that’s fine. Just putting it out there ahead of time.
Frode Hegland: Okay. That’s really, really important to know because And I’m very glad you’re still here, Denny, on that very, very important point with Andrew. We do want at least all the papers from one. Conference to be visible on screen. And the authors names. So I’m wondering if if this because of what you just said, this might be a case where the names should be pre-rendered into images, so they become like tiny little black and white cards, almost better. Illustrating this is a paper. Maybe we should start looking at techniques like them. I don’t know, but this is definitely one of those important details that we could die on. By Brandel. We’re just talking about doing math type of things and XR and lots of other things, but we’ll we’ll return to that later. Leon.
Leon Van Kammen: Yes. So I was I had another thought. I’m trying to synthesize what we started with basically the units of knowledge which are going to be very.
Frode Hegland: Very quickly, very, very quickly. So.
Leon Van Kammen: Even worse.
Frode Hegland: Let’s see. So do you have controls? Okay. Okay. Why don’t you just pause and write a note and then Peter talks. We try again. Okay. Okay. Peter?
Peter Wasilko: Yeah. I was just wondering, are there any alternative libraries to troika? Maybe some sort of a vector based font that would be more amenable to being done in small chunks?
Andrew Thompson: There’s definitely other rendering options. Tracker right now is kind of the the top that everyone talks about working in three.js. But there’s definitely alternatives for sure.
Frode Hegland: So just to elaborate on what we’re talking about, to where do you go? Yeah, Brandel and Dini, we’re talking about the project today, specifically this because it’s so much fun and it’s crunch time because we have to get our ACM hypertext paper in by the end of the month. And we’re now talking about the project a little bit more. We’re going to talk about the papers at the end of the session today, because the notion of putting stuff in space has become so crucial to us that it’s we’re way past generalizations, and that’s why we’re digging into it. So I’m very excited with your yellow handout. Brandel, please go ahead. I was just going to make.
Fabien Benetou: A pragmatic.
Brandel Zachernuk: Suggestion in terms of the kind of implementation that might be able to support stuff that you need to do to have independently addressable regions. Okay. Instead of having.
Frode Hegland: Sorry, sorry.
Brandel Zachernuk: So instead of having individual troika elements, it may it may be possible to use troika. Maybe. Maybe not. But the way you could do it is using a skin mesh geometry where you apply skin weights associated with bones, where you have one bone. Do you know what you know? Are you familiar with skinned geometries and the idea of skin animations and things like that?
Andrew Thompson: Yeah, for sure. Just not in the sense of text. I’m not exactly sure where you’re going with that yet.
Brandel Zachernuk: Right. So. So normally what you do with skin weights is that you have transitional weights where you have an elbow and you have the vertices in them so that you have a gradient of of weights from zero here to one here to be able to kind of deform those things and those relative spaces. But there’s nothing that prevents you from just using 0 or 1 for bones. And so you can have a, a bone, which is for a paragraph of, and you have all of the vertices bound with 100% to that one. Then it becomes then what you’re doing is actually just moving bones around with a single geometry where you have independently addressable things. So there’s a little bit of reckoning. It’s a possibility that you could hack troika to do it, but what that would mean is that you could have one geometry. That’s all of those those texts, or you could have a few that would be a large number of things that you’re managing with each one. So instead of having one troika object or one geometry object per per line, you would have one per maybe 100 lines or something like that, and then have 100 bones that you’re then managing the manipulators on.
Andrew Thompson: That’s, that’s super clever.
Brandel Zachernuk: There’s a I.
Andrew Thompson: Don’t think I can select them though, if that’s the case, because they would all register the same. Sort of a hit test for my selection. I think they’d all returned the.
Frode Hegland: Same result, so.
Brandel Zachernuk: Yeah. In that instance, you may need to manage what constitutes hit boxes differently in order to do it. But yeah, there are there are sort of it really depends on how hard it is and what optimizations you need to go through in order to, in order to be able to make it happen. So one of the things you could do there is Given that you know that it’s only going to be these things, you could have a an x, y insertion sorted list of things based on their, their y position. And so then you can have, you know, like n log n like access into those things by figuring out where on that region it is like or near linear, where like if you’re there and you know that all of them are in order, like collision the collision detection is something that can be done. Much more efficiently. If you know something about the relative order, or you have confidence about about the structure of those things. So, for example, maintaining an insertion sorted list of the vertical positions of all of your all your guys is going to be really easy because it’s always going to be almost always sorted and they’re not sort of undergoing random perturbations. And so from that you can link in pretty much exactly to where, where a hit is and stuff like that. So you can use much more expensive computation that doesn’t have to have, have to have to come through it. But like the point to your, to your point like. How far you go down that track depends on how efficient it has to be based relative to the. Yeah. But like, if you must, if you must do all of those things, those are some pretty good starts. Hopefully that makes sense.
Andrew Thompson: Yeah. And I think in general just working with like a smaller library or catalog, whatever we’re going to call it at this point should still be okay. But once we get into like, the hundreds of troika elements rendering, you can sort of feel the frame rate and frame rate drop in a headset is going to be rough. But you know it. We can still have the functionality and stuff. And for the demo, maybe that’s okay. I don’t know. We can. I just wanted to give people a warning ahead of time. And if it’s getting bad, I can really look into some of these alternative rendering solutions.
Brandel Zachernuk: I am really interested in whether you could pull troika into leveraging a skinned mesh geometry off the bat. Just hack it minimally to do that. Certainly. So I wrote for the timeline VR, the Wikipedia thing. I wrote a full, you know, WebGL text serializer for HTML so that you have the ability to get all of the the geometry. What I didn’t do at that point was, and on reflection, would have been a good idea is to to do this, this skin mesh geometry system where you have independently addressable pieces. You know, that, you know, I feel any regrets about it, but it’s something that you could expand it to be able to do, and then that would be able to give you the work for it. It’s only alpha thresholded rather than a signed distance field based text rendering. But beyond that, it’s it’s functionally equivalent to what troika offers you. And you would be able to integrate this skinning and boning. I don’t want to say trivially, but like in an afternoon of concerted effort. Sorry. I’m all done. Sorry.
Leon Van Kammen: All right. Can you hear? Can everybody hear me? I will do another attempt. Okay, so that is really interesting stuff. These projects, I think that afternoon of work is probably going to be a two, two evenings for me, but it’s it’s it’s a pretty cool idea. There’s one thing I. And Ted Nelson is probably going to hate me for saying this, but do we really need these basically combination of everything on the screen and visible links, like. To me, it sounds a bit like that might be a lot of optimizations. For something which might be a bit overwhelming for the average user. And I’m thinking of ways to sort of mitigate this visible link stuff by perhaps introducing a good old school, old fashioned tag filters. So basically you have your tags in front of you and on the side you have like tags, tags which are generated or from visual meta or a combination. And basically if you click a tag, then it would basically filter out what is not you know, tagged with that tag and then yeah, to basically filter before you really do dragging, dragging stuff out or because a document is a pretty big piece of data. And I was just thinking of maybe adding a step before you know, doing. Really. Gpu or 3D intensive intense stuff. So that’s just my my thought I had Leon out.
Mark Anderson: Oh, I’m just going to pick up circle back to. When we were talking about curved space. I thought one one thing to consider is a hyperbolic view. If I just share very quickly share. Oops. Get right thing our number. So hopefully. Are you seeing a map? But are you seeing. Okay. Right. Okay. So this is unashamedly hype. Tinderbox is just something I happen to be used to using. And it’s been a long time in gestation. I know it’s 2D, but I think it is perhaps pertinent here to this overload thing. So currently. And sorry, this is just a screen grab, and I won’t go down the rabbit hole trying to show you it live. The main point being that it’s sort of it’s it’s this, to me, speaks to the noise problem of having too much. So one of the things that’s actually not shown in this picture is what’s being done here is only linked items are being shown of the overall corpus. Well, that’s akin to what in effect is happening in the reader and author map, because I’m going to put the people over here. So you’re not looking inside of my head when I try and talk to the camera. They talk to the people. So things that you can do is you, for instance, can cut out the cross links so you can have it so that only things that are direct linear link through from the the chosen item in the middle, which becomes a sort of pivot item of the display are shown.
Mark Anderson: Or you can put in the cross links, but that tends to make things much busier and goes back to the point that Leon makes that, you know despite Ted’s point, that actually it turns out that a lot of links just is an awful lot of links. So, you know, what worked in his 1960s type demo on a screen doesn’t actually necessarily translate in scale. It’s just unfortunate, but that’s the way it rolls. The other interesting aspect is trying to work out how much information you need for something to actually be usable. So you’ll notice here that the text isn’t being drawn on the smaller items. And in fact, in the current implementation, I think you’re looking at it actually cuts off after seven items. So you can do, you know, six degrees of affinity. But the and this is pragmatic. Don’t 70 is just a figure that’s there at the moment. But what it what it addresses is after a certain point you also just get too much. So separate to the point raised when Fred was showing the original map of to, well, how do you select what you want to see? Is this issue basically of how do you avoid the, the overload? Because one of the things I think you want to do in a map is explore. And one of the things the map, if it’s if it’s responsive to you, is really good for serendipitous discovery.
Mark Anderson: In a way that you don’t force it. You literally do serendipitously discover points, connections that you didn’t really think about. And I think so that in also in the exile space is is one of the things it can deliver. So our challenge is to put the information into the space in a way that allows that. And just a last point, Ray. The text, I was just thinking back to the the very point that, you know, Andrew Brendel has just been going through the difficulty of actually handling potentially this much text is to say, well, do we actually always need all the text? The more I use maps to find, I find that I’m constantly trying to turn down the amount of visual. So abstract shapes and colors can be a really useful proxy, because if I’ve, if I’ve decided that as a person driving it, I know what it means, I don’t have to reinterpret it. I appreciate that somebody else perhaps looking at it in the moment, I would have to explain it to them. But I think at this stage, a lot of stuff we’re doing for the the individual. So there is something to say in terms of the labeling and stuff that less is more. Thanks. Should I stop sharing? Okay. I can do it again if you want. Fabian.
Fabien Benetou: So regarding the the implementation itself, I don’t want to give advice, but I’d rather suggest not to go back to, let’s say, tag or to D. Let’s say if we flatten on a plane to have more text, if we lose the spatial aspect in the end by flattening or abstracting it away, I think we lose the potential of the medium. Of course, if we had like ten fps and we have to puke, it’s not good. But I would suggest then to benchmark it and say, for example, on a quest three today after 50 elements troika is not good enough. And then we saw it in term of display, how many items are important for the task at hand. And then we display 49 and with an asterisk on the code and the paper or however it’s being shared to the world saying that today in 2024 with this device, the usability threshold is that. But again, I it’s a little bit more argument every time in this kind of question is okay, even if software has problem like hardware keeps on getting better, the CPU, GPU, whatever chipset we use. So I think it’s a safe assumption to say, okay, in couple of months, years, the order of magnitude is will be sufficient. So I think having a maximum threshold of item and an asterisk, let’s say in term of exploring like novel interaction, that might make sense. I think we don’t have to display a thousands of items to manipulate. And I know it’s frustrating, and we want to just not have to think in terms of sorting like what’s important. But I think it’s I mean, it happened to me also in the past, and I found that knowing what to display and having a threshold basically. And after this, just saying it, communicating to users is is enough at least to try, if it makes sense.
Frode Hegland: Better go ahead.
Peter Wasilko: Okay. Another possibility might be a semantic zoom. Maybe when a cluster gets more than, say, 17 elements or whatever is going to break down under our current rendering ability, we replace it with counts of categories along whichever dimensions can best be used to cluster the elements in that clump. So it might be, you know 30 professors, 17 grad students, 21 practitioners. And then if you selected one of those, then. It could expand into the full list for each one of the little sub clusters, but the semantic zoom approach seems like an alternative to doing a gunshot splatter. The other possibility might be to pre-render representations of clusters with different numbers of elements, and then just sort of drop it into a bin, assuming that once the count is at a certain level, the user isn’t going to be able to distinguish that it’s 47 versus 87. You could just sort of have one cluster representing a centroid for that number of elements arranged. So you’d get basically stock representations of clumps at different sizes, based upon the fact that people aren’t going to be able to do the count anyhow, and that might be able to serve as a substitute and be rendered very quickly, because, of course, it would have been pre-rendered for a cluster within this certain range of sizes.
Frode Hegland: Yeah, it sounds like an approach. Randall.
Brandel Zachernuk: Yeah, I’m going to try to have it both ways. On the one hand I totally agree. Like, the function of rendering is, is to be legible. And so to that end, what you need is not necessarily like a, like the absolute variety and truth of what it is, but something that allows you to be able to make the appropriate decisions. And so to that end, stylization is a necessary sort of component of being able to kind of pull back from the level of level zero level of detail that text tends to represent. So, you know, something that has has been sort of a central motivator for me is like, how do you stylize text? How do you how do you come up with characterizations that are are appropriate to the degree of remove? But I think one of the things that Mark mentioned about the structure and the shape of things is, is really wonderful. So even in my Wikipedia article thing it was for quest one at the time. So a much, much less powerful to, to to Fabian’s point. But it was something it’s something that is there to manage an ungodly quantity of text, and it succeeds in doing so with already its own form of stylization in the form of greeking, which is where you it’s the process that Adobe Illustrator and others use, where you just use black lines that are appropriate to the layout of the actual words and word spacing, paragraphs and stuff like that.
Brandel Zachernuk: So it gives you some of the structural information that’s appropriate there. And so if you have the ability to lean on color then that’s a really great mechanism for that sort of degree of remove stylization. If you have sort of persistent shapes. I think that Peter’s concept of pre-rendered notions of higher order, of complexity that you can use to stand in for themselves until you have the real estate for them is is a reasonable one as well. But the, the, the, the reason why I want to have it both ways is because I think that it is and maybe this isn’t the time for it, but but I think that there would be something really instructive about being able to deal with just a stupid amount of data and, and to feel what it’s like to be in that space without being encumbered by these, these these rising but slowly rising ceilings, like Fabian was saying of, of the complexity that you can manage without doing any more work.
Brandel Zachernuk: So again I think that on balance, it’s probably better to go. Let’s just not solve that problem. Let’s let’s stylize let’s work within the ceilings and characterize them as being transient. But also at the same time, if you’re willing to I think that there should be really interesting things from inventing a system that gives you a ceiling, those orders of magnitude beyond, even if it only represents a momentary sort of piece that it doesn’t integrate back into the whole in order to go like, well, what do we get when we when we live here? What can we get away with? So that’s hopefully that doesn’t sound too contradictory. It’s just yeah. In terms of the goals for the immediate goals, I think that finding ways around it is reasonable. But in the event that you really want to do it, I think that there is there’s real merit to to getting stuck in and then then just being being able to give yourself a system that is That that just has so much capacity that you are presented with and sort of confronted with the manifestly different problems that would sort of obtain as a, as a result.
Frode Hegland: So I have a question to the technical people at what needs when we’re talking about slowing down because there’s too much stuff happening. What does that mean? Does that mean lots of things floating in space, or does it mean when they’re being interacted with? But what I mean, since all of the above. All of the above. Okay. Because.
Brandel Zachernuk: Yeah. So it means the frame rate drops functionally that you would end up dropping below your six year 72 or your 60 or whatever. And the problem with that is, yeah, like, interaction looks bad. To some extent quest has the ability hopefully it happens for you by default. Andrew. But you’re getting space. You’re getting time warp and space warp interpolated frames out of your webXR. Right? So that if you just move your head, then you’re getting those interpolated frames. So you get that. But that’s still not everything. And if you’re animating, then you don’t get to see the consequences of it. It’s just for if you have a static world that you’re just moving your head through. So it’s just a You begin hitting the degraded experience of bad frame rates, which affect different people differently. But they’re not good for anybody.
Frode Hegland: Right because. The. Sorry. And I’m just some important, confusing messages coming in, right? So Mark made the point that it’s also very important to hide stuff, something we’ve discussed for quite a while. So that’s not necessarily a bad thing, that there is a limit to what can be shown. But, you know, in the beginning of the call today, Randall, we discussed the kind of jokey thing about whether perspective gives us perspective or not. I mean, in theory, it’s easy to see around something. But the example that Mark showed us with that kind of a space, there was occlusion and occlusion is not necessarily a good thing. So what we might do is take this as an opportunity for nesting. What I mean is having one map slash graph inside another. So we have we’ll go back to the discussion of what level of resolution are we talking. Is that a library? Is it everything or is it a proceedings or is it a document? And show them relatively separately. It may actually turn out to be a blessing for the user. Oh, and one more thing, Brandon, before I forget. Last week the the work that Andrew did looked amazing on the Quest Pro, but in the vision it was worse than unreadable. It actually looked like there was a moiré, whatever it’s called pattern on it. It was very, very odd. I don’t know if you have any perspectives on that. Oh, by the way, just quickly, Andrew, have you had a chance to look at it in the Vision Pro since last Wednesday?
Andrew Thompson: No. I can only go in on Wednesday, so I’ll be going in this Wednesday. Okay.
Brandel Zachernuk: Well visionOS is only providing webXR as a developer preview at this stage, so there’s that aspect of it. I don’t know. So the color gamma is wrong. That’s a known issue. It’s one of those things that that Apple is required to exhort people not to fix, because it’s the sort of thing that should be fixed. On on the implementation side.
Brandel Zachernuk: Yeah. I mean, anything about webXR as it’s currently provided is for developer purposes only. It’s not. It’s not on for customers. It’s on for developers. Yeah.
Frode Hegland: So do you have any magic buttons or commands that Andrew can write into the code to make the vision look at least as good as it does on the quest?
Brandel Zachernuk: If you want to come work for Apple and work on our webXR implementation, and I’m sure you’ll be able to drag the standard forward. No, I don’t. There’s nothing that I that springs to mind. I mean, the color gamut is different. So if you have the ability to square all of the 0 to 1 values. What I’ve done in three.js is inject a little bit of code into the on before compile of the shaders to, to do that. That’s the sort of thing that Apple is asking not to for people not to do, because in the event that it becomes fixed, then it will break. But yeah, like no, I don’t think that I have any particularly actionable things for how it’s presented in visionOS one.
Frode Hegland: Okay, Leon.
Leon Van Kammen: Yeah. Just just for the video recording purposes and transcripts, I would like to still add one more sort of you know, have have it both our ways. As Brendel also described it, I have seen I have witnessed many situations where there is this point of, no, we can’t do it in WebEx or it’s going to be it’s going to be too, too heavy. And then at some point there’s always some person or group of persons who who do the impossible. And they usually do that by coming up with some very clever The concept of translating the data to a shader. And sorry for being a bit technical, but it is. It’s basically also what LMS do is that they, they basically there is there’s a lot of early prototypes which actually used shaders to basically store data in textures. So basically not using textures for graphic imagery, but just to store data. And then they would basically have an incredible smart way of using that texture data to finally render something visually in another shader. And this sort of communication between shaders is incredibly fast. It all happens on the GPU. And sorry, this might be a bit of a sloppy explanation for this, but there is maybe. And this. I don’t think it fits in this project, but there might be a way of basically translating a huge amount of documents into a huge texture.
Leon Van Kammen: And I would imagine a, a combination of some opcodes which, which maybe hints of this is a header, this is a paragraph, this is a quotation, etc.. Perhaps even a visual meta could be the opcodes and and then we would have another shader basically be able to render all that text in one shader, which means that it’s like super high quality. And so instead of looking at this problem through the lens of troika, which is basically also a shader solution for rendering a string of text, let’s say one one word or a paragraph, like instead of that, we would we could make our own sort of troika utility, which basically converts a whole document to a geometry. And so I don’t want to bore everybody with this abracadabra. But yeah, there might be some very clever solutions in the shader side of things. And that might be interesting at some point to visit and to explore. But I, I have to stress and that’s my closing statement that first we need to be able to know what kind of operations we would like to do in that shader side of things. Okay.
Frode Hegland: Yeah. We have to get a better mic. That was solar storm. As Peter was writing. I think we got it. Mark, over to you. But a quick question, Brandel. I’m trying to to use your dynamic view thing that you made various sticky paste, which got me it doesn’t seem to be up. Can you have a quick look? Because I just want to check if the amount of space we had at our test thing actually renders in your space. All right thank you. And while I go on the headset for a bit. Okay.
Mark Anderson: Just quickly, I’m just going to where’s my where’s my screen share gone? Boo! Come on. Zoom. Are you doing zoom? Nope. I do wish they wouldn’t keep redesigning the.
Frode Hegland: So you lose things.
Mark Anderson: Sorry. And.
Frode Hegland: What are the designers.
Brandel Zachernuk: Going to do for money? So if it has been asleep for a while, then it takes a while to wake up. When I go to the.
Frode Hegland: Just woke up. Thank you. Yeah.
Mark Anderson: Just trying to. Oh. There we go. Right.
Frode Hegland: Oh, by the way, since you’re waiting, Mark, look, I have this accessory for my vision that protects the inside. You know, it’s I wonder why it wasn’t shipped with that. I mean, it’s just a lens cleaning cloth, but isn’t that the really important side to clean? It is shipped with it. Not on the inside. Really? Or the lens cleaning cloth. Yeah, but that doesn’t really. That’s not big enough, though, is it? Anyway, just a detail, Mark, please go. Well, okay.
Mark Anderson: Well, I was trying I, I, I don’t know, zoom has removed any ability. Oh. There are found it now. Right just at the last moment. So right I take my words back. User error. So can you see a rather messy map. Yeah, yeah. Okay, so this was just a reflecting on I think it was Fred’s point about some nesting material, and I was reflecting on the fact this is something that’s bugged the certainly the tinderbox community for some while, because the picture on the screen is basically a map doesn’t really matter whether it’s in 2D or 3D. But going to the visible lines, this is this is Ted’s visible lines, and it’s horrid and it’s just hard to work with. And yes, you can, you can drop out visualizations of certain links or link types, but what it turns out is your mind tends to work faster than the controls can go. So you know, you you know, the ones you don’t want to see. But by the time you’ve managed to make those not seeable then you’ve already moved on. You’re trying to do something different. And that was, funnily enough, what led to this, because in in a tinderbox document and again, I just an example, but one that I know a bit about because I’ve been doing some while a map is always a map view is a view spec of one container in the outline that is the canonical storage form of the document. So the problem comes that if you want to see things on the same map, they have to be in the same container, which both means you have to have you can end up with very large containers, and also you just end up with very busy maps.
Mark Anderson: And often what people want is a small, you know, they don’t want enormous containers and stuff, but they want to see relationships. And this is this was is a solution to do it. Now, in this case, it has to have links explicitly made. But in reality if we think forward a bit, those could perhaps be being computed on the fly for you. So and so I think that I don’t think that nesting things except in this sort of semantic zoom that, that Peter mentioned where, where basically you aggregate you aggregate bits towards the outside of the, the graph. So in a sense, in this sense, what the little group at the bottom might, might cluster and show itself as a different shape, or it might show it as itself as a number or a shape or something indicating that you’ve effectively aggregated things. So without, without sort of, you know, saying to naysaying sense, I, I, I’m not sure that nesting things, explicitly nesting things helps because you also get back to the, the old problem from the early days of visual metadata, which is, you know, does the brown dog go in the brown folder and the dog folder or does it go in both? So hierarchies are outline. Hierarchies normally don’t really fit real world data very well. Thanks.
Frode Hegland: Yeah. I mean, one.
Brandel Zachernuk: Thing I would be really curious about with that mark is whether in an environment where parallax is a usable component of a view manipulation, that sort of that graph becomes a easier to be able to follow. What’s definitely the case is that so so my contention is my suspicion is that would be that that it would that said, there’s no such thing as a free lunch, that eventually something becomes overwhelming. And three dimensions is no is no perfect panacea against that?
Mark Anderson: But I do agree with you having the ability to. Yeah, I think you’re. I think you’re well made about seeing it all.
Brandel Zachernuk: Yeah. Because if you can look around it, if you can, if you can kind of prioritize and structure your kind of your viewing to be able to make selections and focus in on things. I was staring out the window the other day, looking at somebody’s spinning little fan on the top of their their house that got damaged during a storm. And it was remarkable how differently large it looked, depending on how hard I was concentrating on it. And just and I imagine from your Navy time, you will have you experienced that magnification that sort of that attentional magnification when you’re really focusing on something and it completely belies the actual size of the visual feature in comparison to, to the amount of attention that you’re able to pay to it. That said, I still agree that nothing no no structure, no system like that gets you completely out of jail. And that’s where I think that the idea, you know, you were talking about the the dangers and the downsides of of containment. And I also agree with that. And that’s where I think stylization is a more comprehensive and inclusive term for what it is you want to do to the structure of nodes, to the relationships that you see. Because all you need to know is some stuff about them.
Brandel Zachernuk: And it could be that it’s related to hard containment. It could be to do with there being some conceptual containment where you go like, oh, here are ten authors or ten ish authors or ten author ish things, or some, some sort of stylization like that, because the amount of detail that you’re able to attend to, it’s not so much that you need to know that these are strict categories that those things are under, but that that’s approximately what it is that you’ll get if you go back to attending to that thing. And so trying to sort of work out, that’s awesome. Trying to go back, trying to work out what come on, what is sort of discernible about those things what can be done to, to construct those stylizations in a, in a manner that’s shovel ready, I guess, is the challenge. But I think a better and more fruitful one to pursue than, than strict containment or showing all of the edges or anything like that because, yeah, things get features get too small, regions get too cluttered, all of those kinds of things. And so being able to have heuristics for what are reasonable and discernible kind of characterizations to, to, to privilege the stylized view with at that point are, is is the thing to get off.
Mark Anderson: Yeah. Because very quickly for Fred takes over because he sees a hand up. But to just draw on your point, the thing I showed with the hyperbolic view, so the colors there and the colors and the shapes are all due to the prototypes being used within the, the document. So that’s how that’s done. So you just decide. So it goes back to these ad hoc effectively these ad hoc groups you want to make are actually quite easily done because you can you can ask for a thing, you can give it a all a prototype, and they’re all of a type now and then you can quite fluidly do this. Fred. Sorry.
Frode Hegland: No, no, don’t be sorry. I am so grateful for this discussion. We spent five minutes. Five minutes. There you go. We spent five months to get to this point. You know, we’ve done a lot of walking around the issue. Andrew has done a lot of building foundations. What you what Mark and Brandel kind of disagreed on is, I believe, the whole point. And we need to really dig into this. I mean, that’s why I’m happy to now have the ACM hypertext paper be high resolution thinking, because we need to figure out what that means in this context. And that means how to hide stuff, how to manipulate stuff, what should be on screen, what shouldn’t be, how do you connect things or nest them? It’s to me. Okay, I don’t know if you heard Randall, because I mentioned in the very beginning I got my PhD today. And the very first work that I did. Randall’s not listening. Randall is not, I heard.
Brandel Zachernuk: So do you mean you received it? You were awarded or. Oh, that’s.
Frode Hegland: Funny that your video was extremely delayed there. Yes. No, I mean, I got the confirmation email that they said, okay, forget it. This is taken too long. Just bugger off. Slightly better.
Brandel Zachernuk: Congratulations.
Frode Hegland: Thank you. So the. Very wonderful. Yeah. It’s nice. So the very first work that I did when I started this PhD was on what Mark and Adam made live. The beige version was one of my tests in 2016. Right. And now we’ve come to the point where we can do things like this. And it’s not just, pardon my honesty bullshit demos, right? We’ve looked at the infrastructures, we looked at the metadata available from ACM. We have various ways we can deal with it. So to now discuss and disagree on how to display stuff in XR is, to me, absolute heaven. I couldn’t imagine anything nicer today. One more thing to mention. I just took Randall’s rough and ready saying that humorously thing what he did two years ago for us in the headset, now loading up the same view that you saw in author that had all the papers it rendered with absolutely no delay. It was absolutely perfect. There are lots of design issues that needed to be changed for it, but in terms of performance, absolutely no issues. So that may be something, Brandel, if you don’t mind sharing with Andrew. So what happened there? Sharing with Andrew so that he can look at it in parallel with the techniques he’s using now? That would be really lovely if you could do that.
Brandel Zachernuk: Oh, yeah. So? So, Andrew, the thing that’s at various sneaky pace, I can see if I can give you the when the code is, is is plain text there. None of it’s a built system or anything like that. It’s the all of the stuff that’s linked off of here. And I just write it in the text editor on glitch, but it’s just using that WebGL text serialization system that I, that I mentioned that the timeline VR was built for. Yeah. Albeit in this one it’s a much more simplified thing, actually. Is it even or is this just. No, this must be because I think, oh no, no, no, I think these are just alpha threshold of textures. I don’t think I’m even using the WebGL laser. So then, yeah, this is just the process of you know, rendering, rendering text to canvas here. So this SDF shader material is probably doing a lot of heavy work, but if we back out of then we can get a lot more stuff on screen.
Frode Hegland: You know, for those who want to try that, I can upload all of this. Now if you want to have a look. It’s actually really quite impressive what Randall has done here. I’m going to expand the. Works. Yeah.
Brandel Zachernuk: I had I happen to have it up while you were, while you loaded it in. So I have it. Have the view of it all and it’s like, wow, that’s that’s coping. That’s that’s fun.
Frode Hegland: Yeah. This is special. And just to share my screen with you guys, because now that I’m just preparing it to to put it up there. These names are obviously too long. So that’s an example of. Having to do these. The transformations using a keyboard shortcut for layouts. So, you know, it should be at least as fast as this. To to do these things and XR. Right. So, you know, I’m just taking some high up, some low down pressing V for vertical layouts and then align left. That’s all okay. So I’m not going to save this. The actual. I’m going to take this guy. This is a test document. It doesn’t matter what I really do. They take the locations and try to move them way a little further off. Save close and then I go to their old desktop. But why is that? And this is. Go to the chrome. Drop the file. Oh, I should probably reload before, shouldn’t I? So here we are. So if you go to that page in your headset now. You will see this.
Brandel Zachernuk: You need to. You need to be. You need to already be on the page in order for it to load. Yeah.
Frode Hegland: If you want to.
Brandel Zachernuk: Just. It has no memory.
Frode Hegland: Yeah. If you want to try it, I will very happily do that. Huge amount of work of re dragging it. Mark, do you want to try this?
Mark Anderson: Go to a sneaky place.
Brandel Zachernuk: There there are occasional circumstances in which visionOS outperforms. Question webXR. I don’t know if this will be one of them. Visionos one you know, one would hope that things improve in the future, but but yeah, apparently Gaussian splats are already better in visionOS, which is gratifying.
Mark Anderson: If I go to this page, do I see what’s loaded? At the moment? I don’t have to load anything into it.
Frode Hegland: No, it’s the opposite.
Brandel Zachernuk: So what you would need to do is be on the page when froda drops it on so it doesn’t have any memory. But as soon as you’re on the page.
Frode Hegland: I’ll drag it over again. Under. Do you want to go in as well? Shall we all dive into this and. Yeah.
Brandel Zachernuk: For. And for anybody with the content already up on their screen, they’ll get a second copy of it, unfortunately. So you would want to reload your page there and.
Frode Hegland: Please tell me when you’re ready.
Mark Anderson: Oh, it’s going to take me half an hour to type the URL in. No no.
Frode Hegland: No. No. It shouldn’t. If you go to future texts.
Mark Anderson: Yeah.
Frode Hegland: So this is what I try to do. Hang on. Zoom is interfering with my life. But there it is. Yeah, it.
Andrew Thompson: Runs quite well.
Frode Hegland: That’s. Yeah. It’s good. Right. Yeah. So if you go mark to Future Text Lab and then you go to XR experiments. Yeah. Scroll down to XR experiments prior to 2024. And then scroll down to constellation. It’s inside constellation, right?
Mark Anderson: So one thing text handling in XR is rubbish. Just, you know, it thinks every time I touch text that I want to select it rather than edit it.
Frode Hegland: Yeah. So, you know, I sent that email finally to Sean and he forwarded it that it is mostly, you know, even with the keyboard, it suddenly wants to look at things. And whether it’s WebEx or native, I mean, an author. I’m having a very interesting design time. I.
Mark Anderson: Should be fair in saying that it’s very much a first world problem. So I’m on the 2024 page and I go down to where.
Frode Hegland: Now you go down to VR experiences. Oh yeah, you are in that. Go down to constellation.
Mark Anderson: Constellation. I can’t see that on page. I’m on 2024. I’m on the dates on 2024. Is that wrong?
Frode Hegland: Go. Go from the first page again, please. Okay.
Mark Anderson: Hold on. I’ll just go to the start page, which I can’t do with a handset. Interestingly, I can do it with a. Go on. Just know. So it’s three clicks to tell it. You want to actually insert the insertion point.
Rob Swigart: Interesting.
Frode Hegland: All right, so you’re on the main page. Right? Front page almost.
Mark Anderson: Right. So I’m on dot info. Yeah. Go to XR experiments.
Frode Hegland: Scroll down to quite far down previous XR experience. Experiences, XR experiment, blah blah blah prior to 2024. Click on that.
Rob Swigart: Yeah.
Frode Hegland: And then scroll down to constellation and click on the title constellation. And that should open it for you. Got it, got it.
Mark Anderson: Sorry. There at last.
Frode Hegland: And I’m going to now drag it over. Tell me when it’s a gray screen rather than white.
Mark Anderson: I’ve got a black screen at the moment, so I need to say enter VR. Probably.
Frode Hegland: No no no no no. Now enter VR.
Mark Anderson: Okay. Now I can see it now.
Rob Swigart: Yeah. Okay. Ooh.
Brandel Zachernuk: It’s such a such a silly, silly. But it’s, you know.
Frode Hegland: It.
Brandel Zachernuk: Was, it was, it was hurled together. And it seems to work, so.
Frode Hegland: No.
Mark Anderson: It’s great. I used the main trigger and pull and it rotated everything round, which was all the fun. And so the. I’m getting zoom on the little mushroom hat on the on the controller. This is great.
Rob Swigart: This is great. Oh, yeah.
Brandel Zachernuk: And then in the 2D view of the 3D world. Yeah, yeah, yeah.
Rob Swigart: Sorry. Yeah, yeah yeah, yeah.
Mark Anderson: No, no, it is interesting. And of course, what it reveals is the sense of. So, you know, we get all this way and this is really good. And then you think, no, but now what am I really doing? Because you’re probably going to end up unless you want to, unless you basically want to explore a computed graph for you, i.e. it’s been laid out for you. It’s at this point you normally start doing hand selection. Yeah.
Rob Swigart: Yeah. So well, that.
Brandel Zachernuk: Works in webXR. So you can get in there and then you either point at or something.
Frode Hegland: The pinch, I can’t remember.
Rob Swigart: It’s pinch, you say.
Frode Hegland: Under. This is something you know, we have our distance thing, and we have our laser pointer, which has been discussed with much emotion. One thing we may want to do, I don’t know if you all remember a soft space. It had a make a fist, and you moved the whole room kind of a thing. And Randall had in the mural pinch and moved the whole thing. So it was seeing how lovely it is to touch these. You know, I’m not. This is. I’m touching straight on them, and things happen, but things are too far away on the sides. So to be able to toggle between touching and manipulating one thing versus doing either scaling or moving the whole thing would be a very interesting. Further interaction. What do you guys think? What do you think of the touching notion that under.
Andrew Thompson: Well, I know we had originally touching text and nobody liked it at the time. We can look back into reimplementing it of some sort.
Frode Hegland: How did you.
Andrew Thompson: Feel about the.
Frode Hegland: Way the touching thing works right now in this?
Andrew Thompson: I like the pinch to interact.
Rob Swigart: Yeah.
Frode Hegland: So?
Andrew Thompson: Yeah. I mean, it’s they’re going to expect it to do the same as the remote pinch effect. Yeah. Right now I think that the pinch effect is the hover effect.
Frode Hegland: So you can see my screen, right?
Andrew Thompson: Yeah. You’ve got a box over the top of it, but we can see it.
Rob Swigart: Okay.
Frode Hegland: Sorry. So I’m just wondering for everyone putting a lot of our work together into something. How about we do a test in your environment? Underwear pinch.
Brandel Zachernuk: Would you be able to predict? Can you look at your screen? Can you minimize the the chrome chrome window?
Frode Hegland: You should be seeing my vision of who.
Mark Anderson: We are, but I.
Rob Swigart: Do, but we.
Peter Wasilko: We see a chrome window in.
Rob Swigart: Front, so there’s a.
Frode Hegland: Okay. Hang on as I can. Do now. Do you see the chrome window? No, I know you don’t. But do you see the chrome window? No.
Brandel Zachernuk: No, no. Now we just see your desktop.
Frode Hegland: Now let’s see how this works.
Brandel Zachernuk: I. I got a good feeling about it. I think it was a bug on visionOS part that was showing the chrome window still, but.
Rob Swigart: Out of interest does some.
Mark Anderson: Is there a control in the Oculus that controls the sensitivity? I find hand tracking is much more shaky in three than two, so unless I’ve suddenly my my my hand shakiness has increased markedly, it’s virtually impossible. I find it virtually impossible to point at something in in three, because at the moment my hand stops the the the suddenly the pointer just starts going haywire. In other words, it starts processing seemingly of its own accord. I think it’s taking micro tremors from the stop position and magnifying them, but of course it makes it virtually impossible to do something like putting an insertion cursor into text at the right place.
Rob Swigart: Yeah.
Brandel Zachernuk: Yeah, I that’s interesting. I mean, I, I think I have observed that as well. And of course, three it could be to do with the illumination conditions in your, in your environment. One of the challenges with any of these things is that the, the, the processing, like, we don’t normally think about the fact that a, you know, that a that a mouse these days is sort of facilitated by a camera, a very small, very high fidelity, very high frame rate camera that is optically looking at the surface underneath it in order to be able to do it, like most of our reckoning about the internal machinations of basic pointing and touching and stuff like that are are done well enough for us to avoid needing to think about them at all. That’s not the case with hand tracking on a mobile chipset at this point, unfortunately.
Frode Hegland: Let’s say.
Rob Swigart: You could attend.
Frode Hegland: To the to the quest line too much. Is that okay? I mean, please finish Brandel, but let’s just go back to this afterwards.
Rob Swigart: Yeah. Sorry, I just Well. It’s hard.
Brandel Zachernuk: Maybe being in better lighting, but definitely, like, if you can use controllers, then they’re better at it because it’s cheaper for it. And it might the performance may attenuate because of the system performance as well. If you move your hands out of your view and then show your left one first. Then that may help.
Rob Swigart: Oh, I lost.
Frode Hegland: Everything, though. Okay, right. You can all see this now, right? Right. So what I want to know from the team as a way of combining things is obviously this needs to be at least slightly spherical. And if we can keep it so it’s this close, obviously the person can move back and forth for interactions like this. And other interactions may be and this is what I really want to hear, what you guys have to say. Maybe the other hand, let’s not say left because we can reverse it for left handed, but the other hand can do action, such as if they do it at a gun thing, they move the whole thing or scale it. So we have, you know, two handed interactions. So this these mean different things. So I can do this and move the space like the mural. Something like that, I don’t know. What do you think?
Rob Swigart: I was going to move the whole thing.
Frode Hegland: Have you all tried Brandle’s mural? For those of you who may not have tried Randall’s mural, it’s the simplest thing ever an XR, and it’s very powerful. This was when Bruce Horn was more of an active part of our community, and he does these wall sized murals. So all Randall did in terms of the user experience, not the technical expertise, all he did was took it in as one giant image on the floor. And we first did the whole joystick walking. We felt a bit queasy, so he changed it. So if you pinch and move, it moves the entire mural. Which is visually the same, but you don’t get queasy. You know, the human brain is the oddest thing. So to be able to do this on a massive thing is actually very powerful. And usually in soft space, you had something. Remember in the previous version you grab and now you move the entire workspace. So combining that with being able to pinch and move things might be very powerful over.
Mark Anderson: I’m reminded that I ended up watching because it just happened to be on Minority Report, and I was grinding my teeth because I kept saying to myself, it’s not that easy. Every time they were doing their wonderful gestural stuff.
Frode Hegland: Exactly. Reality is a funny thing.
Rob Swigart: And that sells.
Frode Hegland: Okay, Andrew, I said something controversial. You’re the one with the most important opinion at the moment. What do you feel about that kind of an approach?
Andrew Thompson: I mean, the the pinch to drag the I’m pretty sure Adam put together some demos that actually test that. And of course, Randall’s got his demo right here. I would say it. I mean, it works well in those tests. It’s very much not something that we’ve implemented in my environment. So suddenly having it only work in one space would be really strange. I can still implement it if you want. Well, I just feel like if we have that test already working, why redevelop it? Why not put the time towards something else? But.
Rob Swigart: Yeah, I.
Frode Hegland: Agree. And under one of the things I really like about working with you is you’re really trying to be flexible. So thank you. I’m trying now to, to really look at compromises for further testing. And once you’re, you know, in the environment, you’re built so far as a cylinder, right. So, you know, the same kind of grabbing wouldn’t make sense in a cylinder. And we have agreed that curvature is really, really useful. So maybe what we want to do is quite simply Live with the cylinder. Positively live with it, right? But have a thing which allows you to move the whole display, not just one item.
Andrew Thompson: That’s good. That makes sense. Yeah. Have a global turn. The entire cylinder around. That’s good.
Rob Swigart: Yeah.
Frode Hegland: So as I move camera in a sense. Right.
Andrew Thompson: Right.
Rob Swigart: I threw it back.
Frode Hegland: In the 90s. But. Okay. We have ten minutes left, and we didn’t do the papers talk. So let’s just do a few major points now and see if we should continue that on Wednesday. Number one. I’ve seen already that Peter would like to work on the high resolution thinking. Do we need a more plain paper just explaining what this is? Or should we put this work as the main bit in the high resolution thinking paper?
Mark Anderson: I’d suggest doing that. Otherwise, I think they’re going to be too many papers floating around in this. The mate, I think the useful, the thing to bear in mind is that we need to have something in the proceedings in order for this to be quotable and usable beyond. That’s that’s to a certain extent, that’s one of the points of the paper. If we want, we can do it as a secretary. It’s a two page extended abstract. That means that the activities that will show will be something of record within the proceedings. But we don’t have to write as much. Or we can do a six page six page minus references thing if we want. We don’t have to do six pages. What I mean, but that’s the other option is, is essentially as a paper in the practitioner track, in which case where essentially what we’d be reporting on is what we’re doing. I don’t think it, I don’t think it makes sense to put in the blue skies track because and start talking unless you want to do another paper about high resolution tracking. But with a week to go, I think that’s I think that’s a not not a great idea. I would suggest we make sure we have by close of play on the 26 or probably a day before, and then I can try and put into Overleaf for you. So we get a nice clean input, but we just have basically a summary partly explaining what this project is doing why we’re doing it. And, and essentially some of the headline things are what we found and why we’re demonstrating them in Poznan.
Frode Hegland: Yes, obviously, I agree with my colleague Doctor Anderson. And what I think we should do is all of that. But to further elaborate a little bit, I would really like it if all of you wrote a paragraph. For this paper. Now, whether it’s going to make the final cut or not, I don’t know, because we only have six pages, but I think a bigger version of this paper will be the future of text paper that we write. So if you write a lot and there’s only a little bit now that goes for what I’m writing too. That should be relatively okay. You will all obviously be credited. So what I would request then is starting with you, Rob, that you write a little bit of what this environment might be in portal. And that also means you can site yours and Denise paper. Does that make sense? Great.
Rob Swigart: Yeah. I’ve been thinking about metaphors for this process, and since I have some archeology background, I’m really thinking about this in terms of what’s the process in archeology? How do you excavate? How do you plan to excavate? What do you need to have in mind before you start? How do you do a test trench? Stuff like that.
Frode Hegland: That’s wonderful. I’m going to email. I really will make an effort to email what I have by end of close tomorrow, so at least you guys can look at it. Fabian I’m officially tasking you. Not asking you. I’m tasking you with the notion of your connected, intelligent blocks, how they might work in a high resolution thinking environment. It seems like a natural fit, but I don’t know what that means. That’s cool. Right? Excellent.
Mark Anderson: Just a quick thing is to work out there are two ways and also which which we can recognize what’s going on here. There are I mean there’s a who’s gonna be the lead author that’s for you and you and Dean to sort of sort out between you. But who wants who wants to be, in a sense, one of the authors of the paper? But even if they don’t, we probably need to have some acknowledgments of what people have done. So also, if you’re writing a few notes, if there’s some way that you would like to be you know, sort of memorialized in that if anything, that you get to say what you do rather than somebody else saying it that will probably be useful for whoever puts the paper. I would suggest that it might be useful for certainly for Andrew to be a credited author on the paper because looking around the room, most of us are a bit further along in things and probably don’t have a great need for an extra. An extra publication.
Frode Hegland: Just writing that down.
Brandel Zachernuk: So my next one will be my first, but I’m not in a hurry. Well that’s.
Mark Anderson: Super. No, no, I was having discussion with Adam because I’m going to do a small demo on the side with him about something else. But I said to him, it’s entirely up to you. And he’s saying, well, I don’t I don’t have a need. I said, that’s fine, but but you can do it. You can be a publisher.
Rob Swigart: Yeah. No.
Brandel Zachernuk: Totally, I appreciate it.
Frode Hegland: Yeah, we can see that. It’s Emily’s birthday, so I can’t be too many minutes over. Leon. But you can write anything you want. But one thing I would like to see is basically taking this entire world and compressing it into URLs, because that notion you’ve been working on is just amazing. What is what would a the love child of Earl and Jason be?
Rob Swigart: Jason Lldi. Yeah.
Chat Log:
16:02:52 From Mark Anderson : The CHI’24 Explorer Frode mentioned is here: https://johnguerra.co/viz/chi2024Papers/
16:03:37 From Fabien Benetou : I think it’s happening now
16:03:54 From Mark Anderson : The CHI’24 Explorer Frode mentioned is here: https://johnguerra.co/viz/chi2024Papers/
16:04:04 From Fabien Benetou : CHI 2024 – May 11-16
16:04:39 From Fabien Benetou : UMAP is a popular layout technique but it’s a bit opaque, not easy to understand yet kind of works
16:06:50 From Fabien Benetou : please change your Zoom name to Dr Frode.
16:07:15 From Mark Anderson : https://memes.getyarn.io/yarn-clip/708bfb5e-61d8-4ba7-a2d4-dbf4628f9514/gif
16:07:30 From Fabien Benetou : Reacted to “https://memes.gety…” with 👌
16:07:40 From Fabien Benetou : guilty
16:08:35 From Frode Hegland : The digital documents which support our scholarly communication do not allow for digital affordances beyond the very basic text selection and link following. What follows is a proposal for a method to enable true digital literature in academia, without incurring incompatibility with legacy systems.
The aim is to support what we call ‘high resolution’ thinking by providing more fine grained interactions and clearer views of the user’s information.
16:14:01 From Fabien Benetou : (brb)
16:15:30 From Leon van Kammen : stupid question: what are units of knowledge?
16:16:25 From Fabien Benetou : the hard questoin
16:17:13 From Mark Anderson : Whatever we want them to be – and that’s not to be flippant. It’s as yet too early to put concrete edge to this.
16:18:10 From Frode Hegland : Reacted to “Whatever we want the…” with 🔥
16:19:55 From Fabien Benetou : and then to mess with your mind https://en.wikipedia.org/wiki/Mereology
16:21:36 From Peter Wasilko : I like the notion of content being semi-structured, which is why I find the notion of multiple island grammars so powerful. One might extract a discourse structure, another might generate a list of referents, another might extract blocks of code, etc. without the author having been required to provide manual annotations.
16:22:14 From Andrew Thompson : Be back shortly
16:22:41 From Fabien Benetou : (brb)
16:25:01 From Peter Wasilko : Description Logics are worth looking into, I think it might have been Wendy Hall (60% confidence) who introduced them to me at a conference ages ago.
16:27:01 From Frode Hegland : Andrew, are you back?
16:27:18 From Peter Wasilko : Note the above is the sort of meta data that I wouldn’t want to put in a paper proper, but I might want to express in Visual Meta for its potential to give readers usual research threads to chase down.
16:28:45 From Andrew Thompson : Replying to “Andrew, are you back…”
I am now
16:29:50 From Frode Hegland : Reacted to “I am now” with 👍
16:30:07 From Mark Anderson : Does the nature/design of LLMs realistically allow for removal of entities. IOW, we might want it, but is it believably achievable today?
16:30:12 From Peter Wasilko : I might trust an AI to pull place names, but distrust it at building summarizations.
16:35:05 From Leon van KAmmen : brb
16:36:04 From Mark Anderson : A challenge in context us our ability to signal the ‘envelope’ for what we want excluded/deleted.
16:41:19 From Peter Wasilko : One of the need aspects of putting those messy leads in VM is that a collection aggregator like ACM could surface references to oneself in the collection and perhaps use an AI to generate a query letter to nail it down, e.g. It might add a filterable note in Wendy’s feed that Peter Wasilko thought she might have told him about Descriptions Logics to which she could respond with ‘yes, no, maybe at some %, or can’t remember one way or the other’, allow the system to update the VM with a “Confirmed by on date”.
16:41:36 From Peter Wasilko : Or on the TouchBar!!!
16:41:49 From Mark Anderson : Reacted to “One of the need aspe…” with 👍
16:42:10 From Peter Wasilko : Replying to “One of the need aspe…”
*nice aspects
16:43:57 From Mark Anderson : How do I select an ad hoc group, i.e. things I’m interested in at the moment. This might be a (pre-applied) keyword.
16:44:15 From Mark Anderson : Re ‘curved – consider hyperbolic surfaces.
16:45:27 From Peter Wasilko : Replying to “One of the need aspe…”
*allowing the system
16:50:06 From Fabien Benetou : bad sound from mic
16:50:18 From Mark Anderson : So, might mapping vs. close grained text be different environments within the whole to allow shifting between more relevant frameworks (re troika, etc.)
16:50:18 From Leon van KAmmen : I’ll skip this one
16:50:20 From Leon van KAmmen : 😀
16:55:34 From Fabien Benetou : FWIW if a good solution is found or even some practical ideas could be good to participate to https://github.com/protectwise/troika/issues/117
16:59:23 From Peter Wasilko : How about Leap Keys via a wireless keyboard?
17:00:57 From Peter Wasilko To Frode Hegland(privately) : I would like to join you in the paper you sketched out at the start.
17:04:45 From Frode Hegland To Peter Wasilko(privately) : Reacted to “I would like to join…” with ❤️
17:12:15 From Mark Anderson : Also, seeing ‘all’ and seeing there is ‘too much’ is also useful in terms of understanding the context of the workspace to the wider problem space.
17:15:11 From Peter Wasilko : https://babylonjs.medium.com/rendering-text-bf3bc53f7c2e
17:17:28 From Fabien Benetou : (same bad audio)
17:17:43 From Frode Hegland : Very bad audio…
17:17:53 From Peter Wasilko : Tortured audio
17:18:05 From Peter Wasilko : Must be the solar storms
17:18:46 From Peter Wasilko : Or Leon has been experimenting with Cold Fusion.
17:21:06 From Leon van Kammen : sorry for the bad audio.
Zoom is tanking my browser (again)
17:29:55 From Mark Anderson : Reacted to “sorry for the bad au…” with 😞
17:30:43 From Fabien Benetou : close call, good recovery, nearly busted
17:30:53 From Peter Wasilko : So super happy for Frode!!!!!!!!!!!!
17:32:22 From Frode Hegland : https://various-sneaky-paste.glitch.me/author
17:32:24 From Brandel Zachernuk : https://various-sneaky-paste.glitch.me/author
17:37:39 From Fabien Benetou : so… need to get the .author file on the HMD first?
17:38:06 From Fabien Benetou : no query parameter pointing to a .author file? would make testing easier
17:40:27 From Peter Wasilko : Looks good in Firefox’s faux 3-D
17:43:48 From Peter Wasilko : Great minds have messy desktops.
17:45:59 From Peter Wasilko : I wonder if we will see a MacBook Pro with binocular web cams for gesture recognition.
17:46:18 From Peter Wasilko : ( Hint, Hint to Apple )
17:47:37 From Peter Wasilko : ( And maybe super light weight LCD 3-D shutter glasses for budget 3-D )
17:47:56 From Peter Wasilko : ( or an auto-stereoscopic display )
17:54:12 From Peter Wasilko : Breadcrumbs to go back and Clues to lead you forward!
17:54:49 From Fabien Benetou : tried to share my screen but somehow buggy, rejoining
17:56:16 From Andrew Thompson : I need to get going for my next meeting.
17:56:22 From Rob Swigart : 5/13/24 Archaeology of text
Thinking of text as dirt
General survey with broad tools, lidar, ground survey, ground penetrating radar, geology, historical references etc
selection of site (document, communication artifact like email or text that operate between ‘sites’)
Kind of site (house, outbuildings, barn, etc, or kind of document)
test trench for general characteristics of the site, likely finds (ideas, bibliography, outline, references, research objectives, conclusions)
test excavation of small section to see what things are there that can be pulled out and examined in depth, perhaps in the lab, or a separate location
evaluation of finds for importance, relevance, etc.
DIFFERENCES:
Sites are fixed, texts can be dynamic
Close up vs. wide view
17:56:38 From Fabien Benetou : not sure if the HMD sharing worked but the idea was reparenting to camera
17:56:45 From Fabien Benetou : well not idea, demo
17:56:54 From Rob Swigart : Future headset is a major element in the book I just finished.