Frode Hegland: Hey, you.
Dene Grigar: Good morning. How are.
Frode Hegland: You? I’m good. Let me just turn off my background. Chill out. Trying to work music. There we go. I am good. And Andrew’s here. Yeah. I should have been a bit earlier. Sorry. I was just stressing with a little PowerPoint or keynote or whatever for us, but we are here now. So, Danny, the big question. Okay. How was the vision? How was the vision Pro?
Dene Grigar: I like it a lot. What I did yesterday, well, I got it all set up, and then I had my workshop with the Apple people, and what? For some reason, it was set up. So that I had to use my right hand and a pointer. And so we got that fixed. And then I worked out how to get my keyboard to come up, which is helpful. The virtual keyboard. So I was I was working with it yesterday and what I, what I was doing, Andrew was going through the next with the headset on. To see what that looked like as a large, immersive environment, as a 2D space in this 3D environment, to see what could be gleaned from that experience from for our project, but also moving forward with the kind of stuff I want to do in the future. And my guess my big question is right now, webXR can’t really is not really that clear and crisp. And according to what I learned on Friday from you, and I think also from Adam, is that the Meta Quest three does a better job with with webXR than the Apple does. Is that correct?
Andrew Thompson: Yeah. For now, at least, that’s what I was told. Like, eventually they’ll be on par, or maybe the vision will even be better. Right now because web isn’t supported by the vision. It’s one of those things that we’re kind of working in, in a beta environment, so it won’t be as polished as the quest would be just in the webXR standard.
Frode Hegland: Okay, just a brief comment.
Dene Grigar: I’m wondering, because if we can.
Frode Hegland: I just want to say Joel is here. We’re just finishing a little random discussion. Joel, I look forward to you introducing yourself when you When. Yeah, when we’re all here. Sorry.
Dene Grigar: Okay. Yeah. Hi. How are you? Hello? Yeah. But yeah, so what we’re thinking about, what I was thinking about is what parts of the next could be optimized for webXR. So that we can actually experience some of this environment. We can’t. The works can’t ever be fixed for it. I mean, the works are going to stay where they are, but what can the environment do? And I think the first place to start is a visualization page. And I’m writing that. I’m in the middle of writing that Warhol grant right now. And I’ve put money in for the visualization space yesterday morning. And to optimize it for the headsets, like if somebody really wanted to exceed those, those pieces. So just explain folks that are new here. Andrew and I and and everybody from my lab has built a virtual museum and library called The Next. And it holds currently 2371 works that are participatory, interactive and experiential. So multimedia. Dating back to the late 70s onward and the, the we’re getting awards and grants and things like that. And what I want to do is optimize parts of it for the, the headset. And the part I want to optimize are the physical objects that we have 3D turned in the 3D models and made available in the space. So for example, if an artist created a. A pinwheel, the kind that you blow and turn it into a little poetry machine. We have that in our physical archives, and we’ve modeled that so that anybody going to the next can actually see that 3D model and watch it spin. So the next step is to be able to pull that pull that out of the flat surface. Andrew. And be like, you’re actually holding it, right? And so that’s what I’m thinking, is we could start with the visualization space so that that’s your that’s my long answer to you. Proto about what I think of the of the headset. It’s got a lot of potential.
Frode Hegland: It’s been really, really interesting. The quality of the native apps is just so much better. So what is that weird sound outside? I apologize, so, yeah. No, it’s That’s why I think we should consider. You know, for the short period. I don’t know for what period of time the native apps will be better. Consider offloading some of our effort to to do it. I mean, it’s kind of crazy. Currently I look at something one of the native apps in vision and it’s just it’s literally retina quality. It’s as good as a monitor. And let me go into one of our experiences and it looks very, very, very, very rough indeed. And well.
Dene Grigar: We just just to answer that, just I saw your email this morning, but I haven’t had a chance to really digest it. But this, this grant money can only be used for nonproprietary open source. So if we’re going to make an app. It has to be. It can’t be in a proprietary system like Apple. We can do that, but it won’t be covered by the grant. We can do whatever we want. I’m writing a grant. I’m writing a grant right now so I can support the stuff I want to do. Right. That’s outside the scope. I would never I would never use our funds to do anything for the next. Are you sure about that?
Frode Hegland: Because yes, I.
Dene Grigar: Am sure about that.
Frode Hegland: I look through our documentation again and also the comments from the reviewers. So I’m not saying the whole thing should be. And if we do, of course it would have to be open sourced, but okay. Anyway, that’s something we can discuss. That’s more of a one state discussion. It’s not an urgent thing here. And I see we’re we’re filling up a bit. Andrew, you had your hand up.
Andrew Thompson: Yeah. I just wanted to briefly throw out the idea. We could always use what we’re currently working on as a prototype of sorts for that. We always knew that having an actual standalone application would run better. And it would be. It would be able to do more and it would have higher resolution. But it wouldn’t be available on all devices. So we could consider what we’re making here as a test for mechanics. Since we’re less worried about having a final product anyways, we just want to we want to get a final product if possible. But it’s the journey along the way and the things we learn along the way. Maybe by the end of this grant when, like, my part of the project wraps up. Then all the research notes and all the prototypes we’ve made get passed on to an app developer, and then it continues in a different form. That’s also possible.
Frode Hegland: Yeah. No, I think that’s completely, utterly the right perspective, Andrew. And also it visionOS is, of course, a bit of a mess at the moment. For instance, if you open the notes app, which is the one I’ve been using to test because it’s included, I can’t download any other apps and it’s really nice to see the typography. It’s so crisp. However, the sidebar, I managed to collapse it away and there’s absolutely no way for me to get it back out. It may be with a keyboard shortcut with a physical keyboard, but it just will not restore, you know? So I just see that as one amusing aspect where Apple is showing that, no, they’re not really investing in tech stuff, so that’s what we’re doing. So that’s good. Let me just check on Adam, see if he’s joining us. He should be. I know that Leon is not joining us. He’s doing something with Rock and Roll band, he said. And also, Mark is overloaded with absolute serious amounts of work. So he will join us on Wednesday and not today.
Speaker4: Right.
Frode Hegland: Well, let’s just kick off with six minutes passed. Joel, I got an email from you not too long ago, and you explained you guys are doing related work. We’re very happy that you can join us here today. So if you want to do a brief intro, both of you, that will be lovely.
Speaker5: Okay.
Speaker6: So my name is Joel Tran. I’m an assistant professor in the University of Maryland. We are, I guess, siblings in the Sloan world. So we have some some work also funded by Sloan looking at cognate topics around kind of the future of scholarly communication. And our focus is on understanding kind of the new tools that people are using and kind of new practices and institutional structures and see how that could be joined together into kind of new scholarly communication ecosystems and infrastructures. So we think a bunch of interviews with people to understand, like the kind of new tools and workflows they’re using. And I guess our complement to what you all are doing is, you know, are inventing new ways of of doing knowledge work. And the specific reason that we have been following your progress is also my students is is with us. He’s interested specifically in intersection of what we’re thinking about with XR. So, so some obvious intersections here. We’re just getting started on our end, so we’re mostly here to sort of learn, but we’ll share. We’ve got some results that we can share. It’ll be interesting. It might be useful for fodder for thinking about new interactions in the space. So I’ll hand off to you to introduce yourself as well.
Speaker4: Hi everyone. My name is Sue and I’m a PhD student at the University of Maryland. Obviously working with Joel. I was a previously a user experience designer, so I’m pretty pretty interested in how XR and can bring that new interactions to people. Affordances, especially the visual affordances of bring to people and how people can interacting with knowledge and information particularly so especially creating a, a collaborative kind of situations environments for people to work together and using XR as a kind of like an additional layer that none above the physical world to help people and more information, more ways to interacting with information especially doing larger sensors works.
Frode Hegland: Thank you. Oh, that sounds very good. Thank you for your email and thank you for being here. Today is our general day. Wednesdays I Peter Wednesdays is more specific project day. But today we do talk a little bit about the project to update the team. What’s going on? So we’re all involved in different ways. Hi, Peter. We were just earlier talking about the apple Vision Pro and how the webXR experience is really quite rough, and that is, you know, fine. It’s annoying, but it is what prototyping is. But what we’re doing have you had a chance to go to an Apple Store to try the vision yourself? Not yet. It is on my agenda. Yeah, I think you need to book ahead. So just go to the website and I really look forward to your perspective on on actually using the thing. Sorry. It’s a weird message. Let me just go on the privacy here. The weirdest sperm ever. Right. Here we go. Anyone else have any introductory updates or comments or questions today?
Dene Grigar: I love Beat Saber. My favorite. My favorite thing. So I’m playing that on on the quest. Right. But there’s this big joke that Andrew and I saw on YouTube how to play Beat Saber on the Apple Vision Pro so it says step one. Step one put on your Apple Vision Pro. Step two put on your quest over over the top of your Vision Pro now you can play Beat Saber.
Frode Hegland: No, that’s
Dene Grigar: I was like, yes.
Frode Hegland: Someone needs to do a cartoon on that. That’s just too awful.
Dene Grigar: Oh it is, it’s a joke. It’s a huge joke. It’s a hysterical. Andrew and I were like, cracking up about it. But so yeah, I do have to say I love Beat Saber. It’s almost as good as DDR. So you may remember.
Andrew Thompson: You might be happy to know. There’s something called I want to say it’s moonrider. It’s basically Beat Saber, but a punching version, and it’s made in webXR. So you could probably play that in the vision.
Speaker5: Yeah, but don’t send me that information.
Frode Hegland: Don’t do that. Just don’t do that.
Dene Grigar: One more thing I’ll say is that Well, now I forgot you had me thinking of something else. Thank you.
Frode Hegland: Yeah.
Speaker7: So it actually I didn’t try it with moonrider. But I imagine it works because it has hand tracking. So you can punch the boxes. And it’s, I want to say it’s functionally equivalent to Beat Saber. The whole point of that demo moonrider was to say, hey, WebEx R is good enough to do like some good looking stuff. It doesn’t have all the IP and everything, but in terms of if you want to move your body in XR and you don’t really know what to do or how to do it with the Vision Pro, that might indeed be a good escape hatch, which I think is a good showcase for for for web XR in that context. Now, yeah, it’s not exactly the same. My my hand was raised for something else, which is term of updates. Kind of a sad one. Maybe some of you have saw it Mozilla hubs, which is a social XR experience. Social VR, I think actually only is shutting down. I used it quite a bit. It was the only project for Mozilla that was still about XR after they fired some of their staff, I want to say two years ago. I think it’s a good project. I think it still is a good project. But without the funding from Mozilla, I think after March or April, I forgot it’s going to be a project funded community, which I have no idea how it’s going to go. Maybe it’s going to get better because there will be more sovereignty and self-governance from the community, and maybe it’s going to actually shut down because there’ll be a dozen of shitty forks in a month or two. I have. Yeah, I think it’s all about coordination. I think it’s a fork in the road for this project. Yeah. It’s not, it’s not ideal. But yeah, that’s a little bit the the sad WebEx update of the of the week. I want to say.
Dene Grigar: That sounds great. I did I did find a few apps that I liked and one of them was the The the one that the constellations. I don’t know if you’ve seen that or not, but you can actually see the. Yeah, you’re shaking your head and you could pull the constellations down. It gives you information about the stars. It’s that I’m a real nut about the sky. And it was. That was a really good one. The dinosaur one I thought was supposed to be supposed to be really cool, but it’s, you know, it’s What’s the word? Very Pixar. You know, starts off a little tiny dinosaur, you know, beeping across the terrain, runs into a hole in the big monster, comes and gets you, and it comes right at you. Right? So there’s that kind of stuff like that. But but I think some of the more productivity apps are really interesting.
Frode Hegland: Yeah. On Mozilla hubs. That’s where I first you know, for our previous back, fugitive text did some rooms settings. So was the only way we could easily as a non-programmer just set up things and be in the space. It was absolutely awful to use, but it was a lot better than any other alternative. So it’s really, really Yeah, that’s. That’s sad. That’s a problem.
Speaker7: Quick, quick word on one of the reasons it’s awful. I want to say, despite being, as you mentioned, slightly less awful than the rest is, I think it’s because it’s tries to be properly responsive, namely, if you want to use it on the state of the art hardware being a quest three or Quest Pro even the Vision Pro, even though you don’t have hand tracking, it does actually work. It also works on a phone. It also works on a tablet. It also works on a desktop. It works on pretty much everything. But in terms of how pleasant the interface and the flow, how you can actually get in there, and how you can try to do everything, like writing in space with your mouse just doesn’t work. So, yeah, it’s it’s I don’t want to justify the imperfection of it, but I want to say being genuinely responsive across non-immersive and immersive device and even mobile phone, I think it’s it’s a real challenge. I think it’s a good one. And I think in term of, let’s say, accessibility, as we discussed in a couple of the meetings in the past, it’s definitely worthwhile. But I just want to say it’s also really, really hard.
Frode Hegland: Oh yeah. I was reflecting on the use when something was built. I was talking about the building. I even bought a fancy 3D mouse that you recommended, which helped a little bit, but it’s just you know, you move here. Just I’ve been doing 3D quite a lot in the 90s and the early 2000 and, you know, I can actually build stuff, but in here it was how you get to the thing. But anyway, it’s it’s sad. We need that kind of a possibility to build environments, to test stuff.
Frode Hegland: So we were away this weekend. We stayed in a railway carriage. The family. It was a big adventure. Steam train, the whole thing. Absolutely. Quite exciting. I had two brief conversations with Adam who will be here on Wednesday. Maybe today. I’m not sure. And we both shared. Frustrations about the general work, and I’m talking about our project. For a minute we want to have notes in there. We want to have all kinds of things. But you know what is actually feasible, what is useful, especially within augmenting academics. And we decided and this is where I need to talk to you guys, because us deciding doesn’t mean anything until we all decide. The thing that makes academic documents unique, above all else, seems to be the reference section. And the reference section is quite normally listed. So my professor at Southampton, Les Carr, he went through and made a script where you take a reference section and this is primarily for a teacher checking a student’s work, but it’s also any academic reading work and it allows you to see such things as The headings and the document. While you look at the references, you can see where they actually are. You can see if they are duplicated, if they used more places, you can see how they’re chunked. You can choose to view them by date, author and all these things. So it seems that considering the relative crapness of webXR texts, this is actually okay because it’s not expecting you to read long sentences. It’s more finding things and moving them about. So I’m wondering what the feeling with you guys is. To focus on references as one of our main directions.
Dene Grigar: Well, if, since Rob and I are the only academics in the room, are or have been on this team now that we have other folks here, thank you for being here. The thing I talked about last week with all of us is we’re talking about the project, right? Is that academics don’t read from the start to the end. We pull things up. So references are the second thing or third thing I look at, right? I start with the abstract, the conclusion, and then I go to references. And what I want with the reference is to pull the references out, and I want to pull each reference out that I want, I want to be able to to search for what I’m looking for if possible, but I want to pull out what I want. I want it to be modular. That’s the word I was thinking about last night is modularity, which is a hallmark of what digital is supposed to be, according to Lev Manovich. And Rob, you’re familiar with that work. That’s the language of new media. That’s one of the four characteristics of the digital world. And so if we can’t be modular. There were not really applying to the notion of what digital text and digital textuality means. I see you’re shaking your head. Joel, you must be agreeing with me to some, some extent.
Speaker6: Yeah, I’m also learning. Yeah. No, the modular nature of reading is Yeah, something we see a lot in our our data as well. Very strategic. Depends on what goal you have at any given moment. Something that I think to make it concrete, like what is a modular thing you might get out of a reference section, is placing it in terms of your prior knowledge. Right. Do I recognize anybody or it could be identifying an exciting the right people like if they claim to be studying hypertext, right. They should probably cite some of the people in this room. Right. And so that helped me that that shows up at different points of the sense making cycle. So yeah, yeah.
Dene Grigar: So yesterday and I was working on this Andy Warhol grant, I was looking at the, a set of references and I was copying it, moving one over to a new page and putting it there, finding another one on another, you know, document, copying it, putting it on another page. So I was doing this physically. And that is not a very effective way of doing things in the 21st century. Seems to me there’s got to be an easier way. So the copying and pasting onto a new document that, you know, and then the tediousness of that, it’s got to be simpler.
Frode Hegland: But then can you elaborate a little bit on this? I think I only understood half of that exact work.
Dene Grigar: So writing a grant and I’m trying to add the theory part. The grant requires theory, right. So I’m going through a stack of books. Finding those books online because I have the books physical, but I want to see if they had the the PDFs of those books so I wouldn’t have to like, type the information I needed. And I did find some of the PDFs, so I’ll pull the PDFs from the web, had it on, and then downloaded it on my desktop, found the had to go through and search for that information in the book, copied it, put it on a another document, empty sheet and then do that tediously through the six different PDFs I was working with so that I can construct the theoretical aspect of my grant.
Frode Hegland: I’m talking about quotes or just citations.
Dene Grigar: Quotes and citations, because I had to then follow up with a reference list. Right? You can’t just have a grant without a references. So now I have a page of references that I had to do, but I had to find that had to find the quotes first. There’s so.
Speaker5: This.
Frode Hegland: Is not an answer to what you’re doing, but it reminds me of something really cool and kind of hidden. And that is if you take a picture of a page of a physical book and then you go into your photos app, you can select the text, of course. And when you then go into something like author, you can use that as of course, the quote, but you can also use it to automatically sign the citation.
Speaker5: Okay.
Frode Hegland: So that means that you when you find it it because it searches Google books based on your text, you don’t have to enter the title of the book or anything like that. So there are some interesting infrastructures and APIs available that aren’t always That obvious. And just to be able to take a picture of a page in a book you’re reading and then it’s digital text is just wonderful, isn’t it? Yeah.
Dene Grigar: I was taking pictures like crazy yesterday to Vision Pro.
Frode Hegland: Yeah, it’s a gobsmackingly impressive technology that is lacking some serious stuff that hopefully we can provide some of it.
Dene Grigar: Well, you know, it’s funny the also the atmosphere. So when you put the quest 2 or 3 on there’s a fogginess. Right. There’s a bit of a I call it fog, but it’s pixilation of the atmosphere. Right? And and to me, it kind of puts a veil over everything you’re looking at. And you put the Vision Pro on. There’s none of that. It’s crisp. So I thought that was another important distinction between the two.
Frode Hegland: I Brundle good timing. We’re just reverted a little back to talking about the the gear. I mean, I have to say, the number one thing for me is no setup. None of this. Create your boundary and all of that nonsense. But however, there is something we do need to take into consideration in the community for our Polish presentation of hypertext is that it takes quite a few minutes to set someone up on the vision, because they have to take it on. They have to do the eye level adjustment. They have to do all of that. It’s at least 2 or 3 minutes. And then once they’re in the environment to be taught where to, you know, the pinching, you don’t need to do this. And that’s a few minutes. So the whole thing we talked about presenting our work, and we have a few minutes to impress upon them what we’re doing. And that’s going to be cut into the cost of this nonsense. Of course, at one point everybody will know. But remember, it’s not that long ago, people didn’t know what a mouse was.
Speaker5: Like what?
Frode Hegland: Why are you moving that thing?
Speaker8: Better is that every time you don the headset, or just the first time when you’re getting initialized as a user of it.
Speaker5: It’s always the first.
Frode Hegland: Time as a user and owner of the headset. There is a guest mode and to to. Let’s say I own the headset and you are here with me. I would have to put it on first, go into guest mode, and then I have a few minutes where you can put it on your head. Then you have to go through all that setup, measuring this, that and the other. And if I’m then going to give it to, let’s say Fabian, I have to put it back on my head, go into guest mode, take it off again, put it on Fabian’s head. He has to go through the setup. It’s very clunky, but let’s remember obviously version one. So. It’ll get better, right?
Dene Grigar: So if you have insets. Also I have inserts for my eyes. You have to take those out because somebody else may not need them. And mine are set pretty high. So I don’t have to wear glasses or contacts or anything.
Speaker5: Yeah.
Frode Hegland: Did they work perfectly? Those inserts?
Speaker5: Oh, God. Yeah.
Dene Grigar: Yeah, they just fit right in. And it’s. I wear like, two point glasses. So I got two point over.
Speaker5: The.
Frode Hegland: Prescription you gave Zeiss. Was it for reading distance or normal? Walking around? Distance.
Dene Grigar: I don’t need it for distance. I just need it for reading. Okay. That’s perfect. I mean, it just it’s like not needing glasses again. Ever.
Speaker5: Oh. Very good.
Dene Grigar: But it’s really it’s a it’s a very interesting I think the apps are going to continue building. And the focus right now is on productivity. There’s not a lot of, you know, fun, you know, playful gaming apps, although you can access some of them. But it’s I don’t really care. It’s I’m not there to play games with the Apple Pro. It’s a quest.
Frode Hegland: So Yeah, I mean, yeah. Sorry. Peter.
Speaker8: Yeah. Has Apple put out a formal statement of what? Distance we want to have our prescription optimized for? If we went to an optometrist to get a reading for inserts for an AVP. May, Brenda would know.
Speaker5: In terms of the. I don’t I.
Speaker9: Don’t think that there’s been explicit advice to optometrists by
Speaker5: Yeah, it.
Speaker9: Sounds like it’s it’s more around the rating prescription. But, you know, I’m not sure if there’s anything explicit that has been put about, but I, I can I can also ask around because it may be that.
Speaker8: I’d really like that information that would be really useful. And everybody else who’s nearsighted would also like to have an official statement from I mean, Apple really needs to do this. It has to have on the page that if you’re going to an optometrist to figure out what prescription to order the inserts for, here’s exactly how many meters out you need to have them calibrate for an optimal experience. And so maybe you can talk to somebody. Somebody has to be listening at Apple to you. So I don’t really loud up the line and get them to make an official statement.
Frode Hegland: Yeah I have to second what Peter is saying because as Hussein well knows, I have two sets of contact lenses that I’ve been testing over the last few days at slightly different reading distances. But still, when I take a document in the vision towards me, quite close, it is actually sharper, you know, and it’s a bit. Wow. Moments like, it’s like I can hold up a printed document and see all the details, but Deeney has sent my Zeiss inserts, which of course had to be delivered to America over. I have no idea if they’re going to work. They are set for my normal prescription. So if they don’t work, then I’ve spent another 50, $200, whatever it is on another one. And even with the two that Hussein and his brother provided for me, I’m not really sure which one it is. So it is really baffling not to know the exact distance. So, yes. That would be a good thing to.
Speaker5: Let’s see.
Speaker9: A support page that I linked has fairly substantial breakdown of what it what it what it says you need a legal, comprehensive prescription. I presume that’s a meaningful statement within American optician product.
Frode Hegland: Can can you say that again? Randall? You broke up a little bit. You need what? I couldn’t hear you.
Speaker5: I. I sent the link there.
Speaker9: It says that you need a comprehensive prescription. And that sounds like it’s a meaningful sort of entity. A concept in American parlance. If your competition, your if your prescription is comprehensive. And it also says that.
Speaker5: Oh. Sorry. Brundle.
Frode Hegland: Hussein, does that phrase ring a bell to you? Comprehensive. You may be busy now, so you may not be hearing me, but Hussein is at work, so. Hi, guys.
Speaker10: Comprehensive. Yeah, I mean, comprehensive over here would just mean that we will check for your far distance vision, but would also provide you with a up close and your intermediate prescription. From the reading that I’ve done on the Apple website, and when I’ve pretended to buy one and I’m looking at the Zeiss insert bar it says, I mean, from from from what I remember, it said that you have to order it as your distance prescription. I don’t know if I heard correctly, but I think you said you ordered yours for your reading prescription. And they worked great for you. So, I mean, it’ll be interesting once your ones arrive, because I think you’ve ordered yours for your long for your for your normal distance prescription. It’ll be interesting to see how they will work for you when they do arrive. But comprehensive just means I’ve been having a comprehensive prescription. If someone asked me for a comprehensive prescription, it’s just a valid prescription that’s been done over the last year. And would give the distance, the intermediate and the reading prescription.
Speaker5: You know. Thank you.
Frode Hegland: I see Randall highlighted. Should not be. Anyway, this. We’ll find out. Hopefully Dean sent that express super expensive hyper mail, so I’ll get it soon. I can update you guys how that will work. Fabian.
Speaker7: It’s not really a constructive point, but I want to vent a bit. I guess because I’m hearing a lot, especially I guess in podcasts and comments everywhere about the Vision Pro. Like a lot of people wanted to do everything. A lot of people wanted cheaper. It’s a $3,500 piece of device with Pro in the name. Yeah, 4000 bucks doesn’t really matter. It’s not. It’s not 500 bucks is my point. It’s not a phone. It’s not a consumer VR headset. I’m not saying it shouldn’t be cheaper or it shouldn’t be more this and that, but my point is, it’s for professional early adopters. It’s version one. Again, I wish it would perfect from the start and cheap from the start. But I think when people complain about a lot of detail about how it should be both cheaper and more everything, it’s like. Yeah, okay. That’s true. Everything should be better and cheaper. But in practice it’s it’s an early stage version where the target audience or professional, I mean, if you have to shelf this amount of money and it actually makes a hole in your bank account and you’re not sure about your use case, you’re probably not the target market. Like it’s probably going to make Apple money, which is fine for Apple. But in term of you as an individual, I want to say not yet. Again, you should. It’s good if you’re excited about it and you project yourself and you start to build something. But if you only build expectation and frustration for device, which is not yet for you, I think you should hit yourself in the foot. I think that’s it. Psychologically I don’t think it’s a healthy attitude. Well, that was just venting.
Speaker5: I think we all.
Frode Hegland: Agree very, very strongly. And that’s why it’s kind of fun to find bugs. So Brundle, here’s a funny bug. The notes app envision. I managed to get rid of the sidebar, so it’s just text. I cannot figure out how to get the sidebar back. That’s a version one bug and it’s kind of exciting to find them.
Speaker5: I haven’t found.
Frode Hegland: Anything online or in clicking on it, but a really interesting issue and this may be a good topic for discussion. Here is how we interact with things that are beyond hand’s reach, which is something humanity hasn’t really had to do before. Of course, with the quest and so on. We have lasers beaming and controllers and all these things, and with a vision we have look and tap. I’m wondering if there are any thoughts on the limitations and maybe how to get beyond that, because it’s it seems quite ridiculous if you’re thinking five years in the future, a knowledge worker sitting doing real work very quickly, interacting with things, we can’t keep doing that, can we?
Speaker5: It’s only.
Frode Hegland: Will it be more hard work? Tools like controllers that are more interesting, maybe better?
Speaker8: Yeah, I really think we need to have some notion of being able to project a three dimensional cursor through the space and control its location, as opposed to having to get your arm all the way out to that location. So maybe my hand could be staying here. And, you know, slight movements of the hand could control the vector that the 3D cursor is moving at in the space.
Frode Hegland: But Peter, that’s what these lasers do. This is why it’s so cool that you’re going to be checking out the Apple Vision Pro soon, because that’s one of the ways we do do it. So got to get that headset on. Gotta get it. Baby.
Speaker7: So so I think, well, first of all, I don’t think that’s feasible to explore in webXR which is problematic in terms of prototyping because eye tracking, as far as I know, and correct me if it’s wrong but is not available so we can think about it and think about tracking the target or a laser through only eyes. But that would not be implementable at the moment in webXR. Still in term of, for example, for Peter in term of experience. Now with eye tracking, you can still, if you pinch let’s say vertically and you have selected a list, it’s going to scroll up and down. Same if you have a horizontal list. If you look at it, it’s going to highlight that element. And by pinching to the side in small pinch not like I don’t know, it’s like maybe ten x or 100th of it. So that that I think is quite efficient in term of small motion for something quite precise. And we can still imagine that. So you look at it, you target or you lock on a target with only eye tracking, and then instead of moving to the side or up and down, you can also move in depth. So you could have on the ray selecting an intersection of the different targets that it’s so a lasso selection in depth basically through eye tracking. But again, all that I think is technically feasible, but not yet in webXR.
Frode Hegland: Yeah, that’s a fair point. I do think.
Speaker5: Have you
Speaker9: Fabian, have you paid? Have you have been have you been looking at the webXR? I guess probably not. The webXR. Competing agenda and updates to the specification that we put through last year.
Speaker7: For the yes, for the last bit, but for very recently, I basically see the weekly meeting and skim through the agenda sometimes, but not always.
Speaker5: Okay.
Speaker9: Last year we passed something called transient Tracks input.
Speaker5: Now.
Speaker9: Which do you have a visionOS device?
Speaker5: Okay, great.
Speaker9: So if you are you in the beta.
Speaker7: I think not, no.
Speaker9: Okay. So if you’re if you’re if you’re in the developer program, you can join the beta. Program, and then you’ll be able to see the visionOS 1.1 beta.
Speaker9: And. There’s a mode of input that we should be getting release notes out for soon. But yeah. It’s it’s. One of the things about the eye tracking that folks fail to remember is that Apple will not let people track files. That’s not ever going to be part of a.
Speaker5: In any case.
Speaker9: But you do need to be able to get some rays, some ways of being able to see where things point. And.
Speaker5: Yeah.
Speaker9: So the transient track input as specified within the WebEx specification is a thing that. We drafted for that benefit.
Speaker5: But. Rob.
Frode Hegland: Oh, sorry. Peter, please go on then. I have a question for Rob.
Speaker8: Yeah. So, Randall, will you be able to get a. This particular object is being looked at at the moment. We can’t track eyes, but can we at least know whether this webXR object is the current focus of the gaze of the user? Will it give us that much information?
Speaker9: Not in webXR. That’s the the highlight mechanism that happens in the native system is one that happens out of the purview of the objects under investigation. So if you look at something it highlights, but that’s unbeknownst to the application that’s providing it.
Speaker8: Well, how do we let the object know that it’s being looked at. Is there any mechanism.
Speaker5: Excellent question.
Speaker8: Otherwise it’s pretty useless isn’t it?
Speaker9: That’s an excellent question, Peter.
Frode Hegland: We have hands in webXR. So as Andrew has built, you can either grab something directly or you can use a pointer. It’s far from ideal, which is why we need to keep working on it. But he’s done a great job with what’s available so far. So, Rob, question for you. How are you now feeling about your Vision Pro particularly selecting and interacting? Is it better than what you felt in the beginning?
Speaker5: I’m getting used to it. I thought.
Speaker11: I’d discovered that you could move things in the. Is it the Z axis towards you and away from you? I managed to move. I think maybe it was the notes application. I didn’t find the problem with this with the sidebar, but I didn’t try it. I tried, I turned on FaceTime and I didn’t see any place to enter a number. Or a name. So I don’t have anybody to FaceTime with anyway, so Yeah.
Frode Hegland: Do you have access to your messages up? Sorry if you have access to your messages app.
Speaker5: Yes.
Frode Hegland: So that’s one way you can go into a message to deny. Or me or Brandon or Fabian, and then you can choose from there to send a message, or you can speak to Siri and say, Siri, make a call to stop. I’m not talking to you, Siri, stop. Right. And that works surprisingly well. I’ve been doing that quite a lot lately.
Speaker11: Okay. I did watch avatar yesterday in 3D and after, you know, starting and stopping several times, I sat down and ended up staying in it for how many ever hours it was, because it was nine at night when I finished. It’s a very compelling in 3D because you’re in it. It’s not out there. So there’s, there’s stuff that I think is going to be wonderful with this.
Speaker5: Yeah.
Speaker9: And even with stereoscopic content.
Speaker11: Peter posted a. Oh, sorry. Now, Peter posted a thing about somebody who’s done exactly what I wanted, which is to take a photograph of your bookshelf. Outline all the book spines. Ocr the titles and then go to Google Books and get all the information about it I want it. And I only have about 200 shelves full of books, so it’ll take me probably twice what’s left of my life to do it. But I think I think it’s it’s a great step if it works. Yeah. So, no, I’m still. I’m quite intrigued.
Speaker11: And I think that it’s going to be very useful. I’ll tell you something I like is that you can take the San Francisco airport or any airport and place it anywhere in your vicinity and watch traffic in real time. In the air in 3D. The ground is not very much 3D, but the planes are in the air.
Frode Hegland: That’s amazing. What app is that?
Speaker11: I forget what it’s called, but it’s Watch. Now that your airport in real time.
Speaker5: And the and the.
Speaker11: Planes are identified like they were on radar. So it’s it’s awesome.
Frode Hegland: We can’t access that outside of the US, unfortunately, but that sounds very cool.
Speaker11: Oh, that’s too bad.
Speaker9: That the notes thing that you experienced wrote. It just sounds like a bug. If you can file a information in the feedback assistant, then I’m sure people would be really thrilled to hear about it.
Speaker5: I will pop in the.
Frode Hegland: Context later and try and deny the insert. You sent them super high speed, right?
Speaker5: Cool.
Frode Hegland: Good. Yeah. Sending mail from here is slow. Yeah, I will slow up.
Speaker11: Will arrive yesterday.
Dene Grigar: Well, also, there’s a federal holiday today, so everything’s closed here in the States. Things are going to be slow. I had to mail some things off Saturday and.
Speaker5: So
Speaker9: Another thing that got people’s attention recently is the the possibility and opportunities of AR quicklook and I don’t know if you noticed, Fabian, but those are the same usdz files that can actually be generated dynamically as well. So the benefits there being that you can, you can construct something within three.js or anywhere else and have that exist as a, as a full object rather than, rather than just existing monoscopic within a window. And they can look really good. I don’t think I’ve set my shelf. I don’t think I’ve sent a model of my shelves around, but
Speaker5: But they’re cool.
Speaker9: And if you have a do you do you have a mac problem? You don’t.
Speaker7: I will have to find a way, so I will. Let’s imagine I have an M2.
Speaker5: Okay. Cool.
Speaker9: So there’s a there’s an application called Reality Composer Pro and it allows you to build it’s not an amazing like digital content creation tool, but what it does, what it is amazing at is it allows you to construct material shader graphs and with material actuator graphs. I don’t know if I send these to you, Frodo. Did I show you the thing with the numbers and the distance?
Frode Hegland: You showed me text on a circle.
Speaker9: Did I show you a thing? Which if you if you move your head closer to it or further away from it, it shows you the distance in centimeters. It is.
Speaker5: Don’t think.
Speaker9: So. That’s right. So shader graphs are just the way that you build things in blender and everything else these days. But you can, you can, you can take the combination of all of the inputs and then create actually a numerical digital readout of how far away you are from the object and what angle you’re viewing it at and things like that. As a combination of all of the inputs. And yeah. So it’s just it’s an incredibly powerful thing. I haven’t, I haven’t built
Speaker5: I haven’t built.
Speaker9: A programmatic way of doing that. But I have sort of hacked programs. Like done programmatic hacks based on their awareness of how to wrangle certain shader graphs. And it’s pretty exciting as well. So yeah, like definitely check it out and play with it. There’s a website beautiful things.xyz that a lot of people.
Speaker5: Around. Yeah. Rob.
Speaker11: Is there a way to share? The vision space. Can two people be in their. How do you how do you share what you’re doing with somebody?
Dene Grigar: I don’t we? I don’t think we can right now. I mean, Andrew has been doing video capture of things, but I don’t think there’s a way to get in there. The FaceTime, we have an experiment. I have an experiment with that. But I’d like to see what we can do in FaceTime together. Rob you too?
Speaker11: Yeah, I’d like that.
Speaker5: I don’t know.
Frode Hegland: I’ll be right back. Sorry.
Speaker11: Well, it seems it seems vital to me that you be able to to share that space. I mean, otherwise, how do you show people what you’re doing?
Dene Grigar: Well, the video capture and photographs.
Speaker5: Yeah.
Dene Grigar: But I think that’s the next step, right? I mean, I don’t want this is what I said last week. I mean, we’re talking in one of the things that bothers me about the the whole headset experience is being alone in it, right? And not being able to share it, because I was as I was working with the constellation app, I wanted to show John, and the only way to do that is for him to, you know, for me to take it off and let him do it. But it’d be nice if we had two headsets, you know, and were able to see the same thing and have a discussion about it. So if there’s a way to do that in the future or even now, I’d love to know. Yeah. I’m sorry. Fabian.
Speaker7: Well the way I do it, and I share the link on the the chat interm of sharing experience. It’s obviously not the same, but at least you can mirror or you can screencast to your iPhone and then you can, for example, cast your iPhone to a TV or a screen. Let’s say if it’s a 15 Pro, you have USB-C. So if your TV or screen has USB-C or converting to HDMI, obviously it’s not volumetric. So people who are not who haven’t tried it at least once, they might not really get what’s going on. But I think in terms of sharing a live experience, I was positively surprised by how efficient that was. And I think I didn’t have much time to explore that, let’s say for live interaction to see a bit more but that so far that was quite positive. Not the same as sharing the space itself, especially for somebody who is not there. But in terms of at least sharing the point of view recorded or live. For me, that that was pretty convenient, actually.
Dene Grigar: That’s good to know. Thank you for that.
Frode Hegland: In the earlier kind of introductions. Less bad for the vision. There was this neat little sequence of someone unpacking in a hotel room while talking to someone via FaceTime. The person in the hotel room had the headset. And it gave the impression that the person on the other side could see what’s going on in the room. But of course they can’t. You know, it wasn’t deceptive advertising by any means, but, you know, you were in the room and you had that friend of yours over there that worked really, really well. But it will be an interesting development when people can share the space better. I see a lot of YouTubers saying I want to have my people in the space, so great. Kind of an update very different from what you were saying, of course. Fabiano showing a representation of what you’re seeing which works quite well. So it’s almost five as in five UK time anyway. So if you don’t mind, I think it would be really nice to spend 5 to 10 minutes on some ideas around our Wednesday project as a little bit of an update, but also to have something to think about. And then we stopped that discussion again exactly on the hour and eight minutes. Is that cool with everyone?
Speaker5: Good.
Frode Hegland: I want to show you something that I built because I like to do that. But everybody can see the screen, right?
Speaker5: Yeah. Cool.
Frode Hegland: So as some of you have seen, there’s been all kinds of design things for layouts or where things should be in, in our space. And the discussion on so this is just earlier mess. I’m not expecting you to actually properly look at this. So then we’ve talked about modularity at the beginning of this discussion, which goes exactly to this kind of stuff. So here we have the abstract table of contents and references. What if we look a little bit again at how we can interact with that in a different way? So is it large enough that you can kind of see what the text is? It’s a bit small, but okay. Fair enough. So what we have here is like the card, the top level card for a document seven Hypertexts. And then we have the table of contents. So now if the user interacts with the abstract it looks and pinches or laser or whatever. And the abstract appears and it appears this what I’m showing you now is based on either having a perfect HTML or TF rendering of the document, or having it encoded in visual media. This is not based on having just a PDF where extractions are problematic. So if the user then interacts with a heading.
Frode Hegland: Exactly the same. You get the text showing up like this, which you probably wouldn’t do very much, at least currently, because it’s not that sharp in webXR. So this is the point that I’d like us to think about, if you don’t mind. Let’s say they go to references. This document has many hundreds of references. It’s Dave Millard and Mark Anderson’s documents, which I was surprised to find some mistakes in the references, despite Mark having done it, which highlights the importance of that being automatic, not manually made. So you see the little kind of pill around the word references. So imagine someone interacts with that and that goes out to show options for that section. So we’re not going to choose the option focus. So obviously it would be much faster in real life. But because we’re sharing so the control prism that Andrew’s working on, imagine that having just a few view controllers in this context. Right. And because you’re now focused on the references, you don’t need this artificial frame. And this kind of text renders quite well in webXR. It’s quite plain text and quite a plain background. So now what would be interesting for all of us to think about is what kind of controls to put down here.
Frode Hegland: So for instance, you highlight one of the items and depending on what you highlight within that, you can choose to view all papers by one of the authors. You can choose to open that document either to take over the view or on the side. Yeah. Oops. Sorry about this rendering. You can choose by clicking on the number the year to view it chronologically, so you don’t have to go down to the control panel to change the view. And then under the second column here. You can choose to have the headings. So you can see where in the document it is. And also you can choose to define what should be shown as bold. Maybe it’s a specific name. Maybe a specific category. And when you then interact with the title again, you have the option to return to the full document. So that was kind of basic. But what I’m hoping we can have is a little bit of a discussion on what kind of interactions we may want to have for a list of references, which is essentially a list of documents, a reference list could be a mini library in a sense.
Speaker5: Over to you.
Frode Hegland: It’s got to be some thoughts on how to have a reference system in an XR.
Speaker9: Yeah, I’ve been thinking about I’ve been reading Understanding Media, and it’s 33 chapters of about just 12 pages each. One of the things that I’ve added is that it’s they’re pretty self contained. They’re, they’re pretty sort of independent sort of documents. It wouldn’t make sense, I think, to be able to deal with individual pages as separable entities easily within the context of that. But but you could you could easily, I think pull out each chapter and definitely the finding device, like having an entity that’s sort of enduring and distinct with with the with the table of contents is something that I’ve been thinking a lot about. There’s somebody promised, and hopefully I’ll get to go to it. A I sell on and on on sort of user interface computing in San Francisco and a couple of weeks and yeah, one of the things that I was thinking about is like, what? What are the things that are sort of all contained within the same linear like organization of a book for the experience of being able to sell a single unit versus what are the benefits? What are the things that would be beneficial to be able to kind of separate and have manageable as independent objects? I do think that a finding device would be really good to be able to exist entirely independently, such that you would be able to kind of refer to it and and manage it independent of the actual corpus, because you basically do different things with them. So I’m for having that. I would actually say that, I mean, it depends on what you’re doing with it, the size of the document. So if it’s academic papers, then there. Presumably not frequently more than 20 pages. And so you may not need the finding devices there, but but for, for a textbook or for a, for a book of the size of, you know, like McLuhan’s books all weigh in sort of around 400 pages. It’s pretty useful to be able to have that and to be able to kind of deal with it independent of the main corpus.
Speaker5: I think you.
Dene Grigar: Just answer your question. What what would be useful? And we talked about this last week. I don’t think you’re at the meeting, but the notion of going back to a hypertextual environments, right, where everything is seen as Alexia. And right now, if you look at a PDF doc from ACM, there’s there’s like sections and subsections. You can imagine that each of those sub sections and subsections would be discrete elements that are modular nature. Right. Right. Now we’re reading it linearly. But we could invariably set this up as modular elements so that you could pull out this section like Alexia, pull out this element, put them back, mix them up whatever you want. But if we set this up with the notion of nodes and links and lexias, then we’re going back to hypertextual environments, which I think are very fruitful environments to work in. Thanks.
Speaker5: I think.
Frode Hegland: Danny, that is exactly what we’re approaching, and that’ll be very interesting to get more dialog on what that actually means in practice. So yeah, very cool. Peter.
Speaker8: Yeah, I think since it can be a challenge to code things in XR, maybe the most flexible interface would be to just have a command line field at the bottom of the prison where I could type something, have the system just pass that as a command line interaction event to the back end application. And then I could describe textually whatever kinds of elaborate filtering I want just by typing and not taking my hands off of my Bluetooth keyboard, talking to the headset, have the back end system apply that filter, and then update the visualization, rather than having to worry about changing my gaze and look at a whole bunch of buttons and navigate through a visual interface to effectively create what would be a text prompt for the back end filtering system. And then I could ask it queries I could describe, you know, temporary variables associated with search groups so that I could let’s say I said, you know show me. I could type in textually, show me all books after 1977, okay. And then say, call these recent works. Then I could use the phrase recent works in another command in order to use that pre-filtered set. And when I did a name like that, the system could even put up a visualization of the different sets that I had named and add that in 3D. But I think just the fastest way for interaction would be through text on the command line, and just put the command line at the bottom of the prison, type it in, hit return, and then the system can do all kinds of exotic things without having to mess too much with the interface affordances in any particular headset.
Frode Hegland: Yeah, I think they’re very different kinds of users who would find the different modes appropriate. So if you want to work on that Absolutely. Interesting to put that in a in a, in a spot on the control bar. Robin.
Speaker7: Well, to me, the both for code and for citations or for pieces of knowledge, they of course, they raise the minimum efficiency threshold below which it becomes so tedious. I can’t actually think I’m just doing the task so. But above the third and threshold, I don’t think pure efficiency is the right criterium. I think at some point it has like, does it actually generally help you to think like, do you get new ideas basically of it? Do you get deeper into the actual content? And I don’t think speed alone is the only metric. I think at some point, if you can like get new affordance to that thing and maybe typing is the fastest, but it doesn’t help change you the way you manipulate that object, which might be a 3D object, which might be a series of citations, whatever that is. But in practice, I think it’s more speed. Of course, again, above a certain threshold is important. But after that, once that has been met, in my opinion, it’s like, is it a qualitatively different way to manipulate that thing, either being a piece of information, a 3D model? And I think that’s more interesting. I say this again candidly, I don’t necessarily have a solution to this.
Speaker7: So I think, yes, if you want to have a command line for sure if you want to directly manipulate us, although I think if you are, if you went through the trouble of buying a headset that is volumetric, at the very least it should be an affordance that is available, in my opinion. But it just it’s still something I’m tinkering with. But I think the intersection of both, like being able to have those 3D objects in space being either a proper form, let’s say, or shape or 3D model, or something that is represented as, let’s say, a citation or an entire book or a piece of knowledge, the intersection of both that you can program it or manipulate programmatically and directly is the most interesting. But I don’t think speed alone is sufficient. It’s like, is it? Am I manipulating this? And whatever that might be can be bookmarking, can be reading, can be coding for how to change its position in space. Again, I think that’s the more interesting thing when you basically don’t know how to use the tool at the beginning, because, yeah, you actually have to think in order to use it. Like consider what you’re manipulating.
Frode Hegland: Yeah, exactly. Fabian. So, Peter, when you’re talking about a command line interface, that could work also very well with a large display, a large monitor. You know, we’re trying to find out how this also fits in space, which is what I showed now only partly addresses, but that is a key thing. And before I hand you back the mic, just an observation that maybe we’ve talked a lot about having interactions with one document or a few and a library. Of course, this definition of a library is up for discussion itself, but it’s interesting, I think, to think of a reference section as being a library that is owned by a document. So we can use probably many of the affordances to interact with the reference section as we can interacting with a library. So it may be spatially we can we can start to build something there. Peter.
Speaker8: Yeah, one of the favorite interaction affordances that the current macOS lets me have is grabbing an item in the finder and dragging that item’s icon onto the terminal window, and then having the path appear in the terminal. And I’d love to see that in 3D. We grab the item in 3D, and we pull it in, and then drop it onto the command line in the prism so that a reference and it could just be shown as like a little glowing marble embedded in the prison, so that we know that that’s a reference to an actual object in the world. And then you can get that nice interplay of the text talking about the selection and the visualization, and then the text driving a modification of the visualization. So you can then grab a new thing in the visualization and throw it back into the text world. So two supporting each other I think is key.
Frode Hegland: For clarification though for all of you. And now we’re five minutes over. So please feel free to to change the discussion into either something else or something more general. Our, our goal with the project right now is when we present at Hypertext in Poland in September. Right. They need.
Speaker5: Yes.
Frode Hegland: Yeah, it’s very soon. It’s half of its theater. The person putting it on must get the feeling like, oh, I could work here.
Speaker5: And all I.
Frode Hegland: Can work here does not mean holding up a PDF and reading it. Obviously it is about using the spatial nature of the environment. So we really need to think of what are the key things an academic wants to do. And one of them uniquely of course, is references. And then we got to think of the spatiality. So whatever thoughts there are on that are just extremely welcome. What what is uniquely special that we can do? Well, one thing that I really wanted to mention is that Adam was kind of bullying me last week in the best possible way. Because he’s very upset he doesn’t have a apple Vision Pro yet. Of course. So he wanted to watch me at work in, you know, just the room. And he found it useful to see that. So then the second thing he made me do, which was extremely useful. And I’ll suggest all of you do it as well if I make a panorama in Photoshop. Put it in photos. When I’m then in the Apple Vision, I click on that panorama. It becomes a fully wrapping panorama. It is an incredibly useful way to check sizes, relationships, and layouts with absolutely no programing whatsoever. It’s a very simple and powerful prototyping tool. It doesn’t use the Z space. It doesn’t use depth, obviously, but the fact that you can move all the way around is mesmerizing. So, Fabian, since you unfortunately don’t have the time resources to be Wednesday with us, what are the key research things you are looking at now and how might it overlap with what we’re doing?
Speaker7: Well, right now it’s What can I do with the Vision Pro? That’s the immediate that’s the immediate check. Knowing the limitation. That’s why I was asking also a bit in the chat about the model, like manipulating USD or Usds because some of you might know, but you can do VR. And I had the good experience, I think this morning I tried some old code, let’s say that I made for teaching a bit of math, or at least number two kids through direct manipulation in Excel or in VR. Precisely. And I basically tried on the Vision Pro and I did not change anything and it just worked. So that’s again I mean, I mentioned this every couple of months because, yeah, this is what I do, why I do webXR rather, it’s like I get a new device. Doesn’t really matter if I like it or not. If I’ve done this right and if the device supports webXR. Boom, my content works on it. It doesn’t necessarily take the full capability of the device, but that’s already in term of onboarding or being able to use the device and even resilience really, really powerful. So that was a very good news. But of course, that’s the let’s say that becomes the lowest threshold. Now I need to what can I do with it that is different that I cannot do with anything else.
Speaker7: And this. Yeah, that’s what I’m exploring at the moment. In term of manipulating documents, that also means some of the stuff I was doing, for example, until now on the quest three. I can do take care. I can do with the Apple Vision Pro. And it also means in term of competitive advantage or differentiating aspect. I got a much more powerful device, meaning in term of manipulating 3D models or number of documents or interaction. I can go a little bit crazy. So I didn’t benchmark it yet. But yeah, I’m going to try this basically to do as much as I can with it, both in term of yeah, size and complexity of content. And yeah, that’s, that’s what I’m going to to be working on. So for this, for manipulating documents. Yeah. I have a colleague who sent me Like 20 documents in different types videos, text, PDF, HTML. So I’m going to kind of make a whatever cloud wall, whatever with it and then give them a couple of tools like linking them visually, for example, opening them fully simultaneously or bookmarking something like this on it and then see what sticks.
Frode Hegland: Yeah. That’s great. And I’m going to have a discussion further with Dean to see if we really are constrained that the systems we develop for have to be open, not just the software itself, because the difference between native and WebEx are right now is a bit massive. And I’m desperately trying to get some updates, even to reader, to have multiple documents open at the same time. I don’t have the availability of the developers I should at the moment, so it’s kind of frustrating because that should take care of Dean’s requirement of reading side by side. Just that, you know, because one of the key things here is we’re not doing one thing with the Sloan thing. We’re doing a myriad of different tests and make them all available. And just maybe some of them will be in a menu. You want to use this, this and this, maybe most of them will be thrown away. It doesn’t really matter. That some of them can be native for sure.
Speaker5: A little bit of it. I don’t, so.
Speaker7: I’ll let her interpret it because I don’t know if she knows it better. I think maybe what they mean by proprietary is rather that if there is only a single platform that can use it. The then it’s not compatible with the requirement basically. So if you can only use that code on a single device, then it’s not okay. But again, this is just me speculating. I don’t know it, I’m not a lawyer, blah blah blah.
Frode Hegland: We’re trying to find out. And also for me, what is more important than that is that the data should be transferable. So we could maybe make a native app reader, for instance, obviously open source, that if you click a button or whatever it opens in webXR I know in Safari you still have to approve. So I’m not saying automatically the whole thing, but these things need to be. We talked about modularity at the beginning of the call today. We need to do that with this two and one thing that has become completely clear every single day is the importance of the metadata. I was amused in a fun way to find a couple of errors and Mark Anderson’s reference section. He is the best reference section person I have ever met. So the fact that he even can make mistakes because he is a human is, you know, really highlights the need for a lot of this to just be kind of baked into documents. And don’t forget, infrastructure is a third of our Sloan thing. Yeah. Fabian.
Speaker7: In term of Let’s see what toolset or toolkit both software and hardware should be used and if it’s compatible or not. I mean, I think it’s important again, to To keep in mind that Vision Pro quest three. Whatever. No. Sorry, sir. It’s still like couple of steps down the long lineage of devices, both before them and after them. So I think every time one wants I don’t know, faster refresh rate or higher resolution. Again, I agree with this. I’m not going to ask for lower resolution, but at the same time, like, it’s not is it literally blocking the exploration of the idea, the prototype validating if the idea is good or not. And again, most situations I would say probably not. There are some cases where it might not be, but then again, if the goal is to learn, then just saying, oh, it’s not feasible right now because of either software, let’s say WebEx or or hardware, let’s say resolution or the responsiveness of eye tracking. I don’t know, just saying random stuff here. I think it’s already valuable. And then saying, okay, it’s not ready yet or we don’t know how to do this yet, and they are still like an entire ocean of things we don’t know that might be valuable, and we just move on to the next thing. To me, that’s okay. Might be a little bit frustrating or look like an easy way out. And I don’t want to say, oh, there are so many low hanging fruits, that doesn’t really matter. But I still think there were so, so many interesting things that some of the limitation I see right now. Yeah, sure, not perfect, but long term doesn’t matter much.
Frode Hegland: I agree with provisions. Provisions are that there are some stages where everything changes. Like before Macintosh, you could not have desktop publishing, obviously. That made it possible. There was no such thing as desktop publishing before the Mac. Before you could display an image in a sequence relatively fast, you couldn’t have video. You just had still images, one after the other. I do think that the Vision Pro is such a step change, because the quality of the display is of course, it’s going to be improved, especially with the camera for the room and all of those good things. But it is of a quality where it is. Just visually, you can’t read in there. Clearly, it’s clean and amazing. I mean, there are some times I use my normal computer with a vision without virtual, because I don’t have time to set it up. I can still read. It is an enabler of a whole different level of of interaction. So that’s why I agree very much, Fabian, with your first statement that now it’s time to figure out how to prototype and stretch that platform.
Speaker5: And of course it’s.
Frode Hegland: Going to improve. Of course there’s going to be other platforms. Now, the the kind of compromise of looking at reference sections, I think is a good compromise because it’s basically some text in space. Hopefully we can use the text to space more, you know, as we figure out more.
Frode Hegland: But yeah, would also be interesting to go go full native.
Speaker5: I think we’re agreeing on everything.
Frode Hegland: Fabian, I look forward to hearing what you have to say.
Speaker7: No, it’s not to disagree this time, even though I will soon, but it’s for text benchmarking. Have you used anything like this? Do you have like a native and or WebEx or page where you display a page, a thousand pages? One line, 10,000 lines? One font, 10,000 fonts, whatever, in different position, close, far away, etc.. And I see you, I mean, anybody, not just fraud.
Frode Hegland: In terms of having lots of stuff up.
Speaker7: Yeah, lots of turf or just not a lot of stuff, but like the font or perfectly drawn regardless of distance or whatever, because I’m thinking that I think of course it depends on the use case. Like some people might want to read an entire 500 pages book in it. Some might want to just manipulate references, but if there is such a page that allows somebody to just quickly benchmark, they say, okay, for me, with my vision at the moment and this specific hardware, it’s a goal or it’s a no go. And I can do this, for example, manipulate annotation. That are one line long, but I can’t manipulate or I can’t enjoyably read a full page or 500 pages. So I’m just wondering if anybody has this, like I did, not, for example, myself take a proper book, let’s say, or a PDF or whatnot in the Vision Pro and try to read it. I did not have the time to do that yet. So I’m wondering, yeah, if anybody here who because it’s the future of text group. So we manipulate mostly text. So I’m wondering if we do have such a, such a thing.
Frode Hegland: Oh yeah. I’m happy you got more to say, but I just want to say I have used the Apple books and reading and in the vision, it’s amazing. It’s flawless. It’s just unfreaking believable. Job done. We don’t need to do that. We need to work on space and interactions is how I feel. Brendel.
Speaker9: I. Yeah. You can also read your Kindle ebook, even though the app on Red dot amazon.com, which is really neat. If you if you have those I then you can obviously read any web pages that you have things. And so if there’s a caliber page then then you’d be able to kind of load that up as well. If you have a sort of a local cloud based system for reading them and it’s it’s gorgeous.
Speaker5: I was going to say.
Speaker9: That one of the challenges with ECS right now is that we mostly read it at a certain scale, a certain expected fixed scale, and so that it’s not as directly amenable to being kind of portrayed at different sizes or distances, because we mostly. Get up to it at the same, at the same distance in order to relate to it. In contrast, you so like, you don’t read a book from across the room. You and so. We need to come up with representations of texts, kind of like what books are to what text is. But maybe more intermediate representations in order to be able to make use of the space as it exists within these spatial computing devices. And so, you know, there have been especially when you have a ragged right. Rather than a fully justified piece of text, it is a little easier to be able to kind of parse what the structure of text is. But I think that one of the things that will be really important for us to do is figure out whether there are. Intermediate representations or ways of being able to kind of discern what the. What the function of a piece of text is because, yeah, like you can you can glean the structure. Like once you’ve read a piece like up in the close mode, then you can kind of see it at a distance at a degree of remove and know this is that chapter, that is that section there, or these are these paragraphs of delineations. But because we have had this fixed relationship that we always do the sort of the ground truth processing of information at this arm’s length or shorter distance. We haven’t had to have ways of being able to kind of summarize things that have come before it. I, I think that. Artificial intelligence, like in terms of text summarization, might be able to help. In terms of being able to give you indications of what paragraphs will be ahead of the time that you’ve read them.
Speaker5: But yeah.
Speaker9: Like I text in space needs a range of intermediate representations of text that amenable to being able to kind of comprehend in these different ranges and representations that that are sort of maybe orientation invariant or, or other things. One of the things that I would like to do with McLuhan is come up with a, you know, a couple of emoji size or, you know, a couple of Unicode size icons or chapter that can be used to convey the function of those chapters. And I think that those kind of landmarks and signposts will be an important part of other sort of larger scale textual navigations as well, in the future. But I don’t know that there’s a sort of a cheap answer for like, what are these icons? What are these, these coarse representations that can be kind of passed around for these purposes? I mean, I see it as pretty valuable as a as a function.
Speaker5: Two things.
Frode Hegland: In terms of reading pleasantness. For me, sharp, sharp, sharp focus is the key. Take off my glasses. I’m short sighted. Read on paper, you know, move the book to super sharp. That makes the best same in the headset. Now I have to move it a little too close, but. Oh, my God, you know, not having that blurriness. I think there is a huge cognitive load, at least for me to process. The blurriness is sharp. Outline seems to take less mental effort, even if it’s slightly, slightly sharp. So that’s an extremely important feature which we may or may not be able to relate to. The other thing is, Randall, you were talking about the kind of dimensionality. I’ve been shooting a ton of iPhone spatial videos and it is amazing. Some people are saying that what you shoot with the headset is better than the phone. Come on, the phone is incredible, right? We mark this place having lunch yesterday and I’d go on it to play with a toy. I made sure there was good separation. He is there when you watch it. We I really the stuff that I showed you with the references. I want us to implement something like that as soon as possible in a modular sense, so that we can move stuff back and forth.
Frode Hegland: You know, it may very well be that we don’t need to hide the table of contents, just move it at a certain space, maybe even closer than what you’re reading. Actually, you know, we haven’t had the opportunity to do that testing, which is partly why it’s so sad. Mozilla hubs is gone because there were prototype, that kind of stuff really, really simply. So to bring. A level of specialty into reading will be incredibly interesting and important. I don’t think it’s going to be really via emojis though. Brandel, so I’m very much looking forward to fighting you on that. See who comes up with different solutions. But I do think that recall, excuse me, recognition is easier for humans than recall. So to be able to have visualized areas, things and knowledge that when you see them again, you recognize them, rather than having to just do a find to find them. It’s going to be a huge thing. So to make text more visual, as we’ve talked about before, will be really key to this. Peter.
Speaker8: Yeah. About six months ago, I recall seeing some project that said that it would. Automatically generate unique looking icons for you, and it didn’t seem important at the time, so I didn’t bookmark it and I have no idea where I saw it. But, you know, roughly maybe six, seven months back, I did see something cross my desk along those lines. I’ve always had a real strong esthetic preference for fully justified text. And I like to have, you know, protrusion applied all the micro typography nuances to optimize the gray level of the text overall. And I find that text much easier to read than text with a ragged right. And I don’t know whether some brains are wired to prefer ragged right text. And other brains are wired for fully justified mind’s wired for fully justified. Not sure why but we might want to see whether we could. Bring the new flat line break algorithm over into three dimensional space somehow, and maybe see if we could take that forward and add an extra dimension to that kind of logic. So you’re talking in terms of elements with stretchable glue in between them that can be expanded or shrunk and. It’s something to think about. Certainly we should be able to, in XR, get the same quality of line breaks and justification that you can get in traditional media. There is one JavaScript project already out there that takes the Gnuplot line break algorithm and will apply it to text on a web page. So you just do this one little manipulation to run the script and automatically all the justification lines up, looking just as good as if it had been done by the latex rendering engine only it’s just with the regular browser engine itself. So that’s something that we might want to try to play with in the XR environment. And finally, I’m going to have to drop off at the end of the hour to give mom a little help with a couple of things. So I’ll see you all on Wednesday.
Frode Hegland: Yeah, yeah. We’re all going to drop off on on the our.
Speaker5: But
Frode Hegland: I’m just wondering. Okay, so let’s say that we’ve decided one of our focuses is going to be on references because it is a key academic thing. What is the best way to work on the dimensionality of that? What is the best way to tell Andrew to build it in a way that we can test specialness? One of the great things about the Vision Pro that Apple really has done is when you move things, everything is billboarded when you’re in your default space. At least that’s really, really well done. It currently webXR it’s too easy to move things all over the place and it becomes really, really hard to orient things the way you want. Any thoughts on that? We can experiment with that.
Speaker7: My my very limited thought. There is again, take all the low hanging fruit you can. Namely, if the OS already allows to manipulate certain document a certain way, obviously try the way they’ve done it, not necessarily because you believe in it or whatnot, but just because it exists. So at the very worst, you learn what you like and what you don’t like. So I really like I did not I don’t have a Kindle, but if whether since Kindle is supported with some books and yeah, just do this and then note what you in record, for example, because it’s easier to communicate to others that don’t have such a device. Yeah. What what works and what doesn’t work. And there is no point in not doing it like it already exists. Just let’s do this. And after this, indeed, then it becomes like, okay, let’s let’s try crazier information, things that might be overwhelming or too free, let’s say to to looks manageable. But initially I’m always taking all the low hanging fruit that someone else have done before. Like, there is no point of not trying it.
Frode Hegland: I just put a link in probably into the tests of how we’re doing it now. So when Andrew does something, it’s all listed here, and people are welcome to comment, ideally comment in the comments because it seems to be easier. We haven’t really started doing that. Somebody did slack, and it’s a bit disjointed, so to speak. But it’s very, very strange for me now, having to put on contacts and not being perfect. I don’t do the longest sessions, but it is still very, very useful to go in and out to webXR. The opportunity is just so far beyond anything we’re doing now. And maybe Brandel, we should have a section on you and Gutenberg on that document. You know, you show us how you’re reading it and we talk about different ways of doing that. I also think that if we think of a reference section as being a library owned by a documents, we can have many designs around how to interact with that. Because some of it, of course, is constrained by the fact that it’s a reference section of the library, but other things are not. I don’t know what you guys think. To move ahead. So, Andrew, the way you’ve been implementing what we talked about last week, I believe Adam said that these rectangles are being implemented in a kind of modular fashion. Right.
Andrew Thompson: Probably nothing’s been implemented yet. I got the stuff on Friday and haven’t worked over the weekend, so I’m going to start working on that today.
Frode Hegland: But that’s fine. I’m just going to share a screen here for everybody’s delectation. So we were looking at this kind of stuff. Where we have references on the sides, we have a title. Basically. This kind of a layout. I am now wondering if. We’re not throwing that away. Please don’t think we’re going all crazy, but maybe just. We should start with this. Just throwing up a list of texts of a real reference section. I’ll give you that. And then we start.
Andrew Thompson: Send me the JSON for that.
Speaker5: I’ll send you.
Frode Hegland: A plain text, if that’s okay.
Andrew Thompson: Sure. Yeah. I mean, if you have a JSON export, that’s great. But if you don’t, don’t worry about it.
Speaker5: I have this.
Frode Hegland: So what I’ve done is I’ve gone to dear old Reader. So this is Mark’s document. It’s got 205 references. I copied the first 105 and cleaned them up. And some of them missed out a few things. I’m going to go through it again. Because it shouldn’t have the in. This is supposed to be just the number it appears on this document. Author’s title and year. That’s it. But do you guys feel that we should try to work on a basic interaction just because it’ll be a huge, long document and then we can
Andrew Thompson: Probably a vertical scroll?
Frode Hegland: Yeah, a vertical scroll to begin with, because even positioning that pleasantly will be an issue. And this is something that could pop in and out of the those five rectangles. It.
Speaker9: Yeah. So Andrew if you not played with it you can use clipping frames. Clipping planes in jazz and they work with stroke attacks where you scroll with that if you want.
Andrew Thompson: Does that just affect it visually? Like can you still raycast onto the object when it’s hidden? Or is it does it actually affect ray casting as well as a proper mask?
Speaker9: My expectation is that it would that the ray like. It, it would be good for it to respect the clip. But it probably doesn’t because it will be doing it based on the geometry bounding boxes.
Andrew Thompson: Okay, that’s what I.
Speaker9: Was worried about. But in general. But but in general, I would expect actually to be I wouldn’t raycast on to the text per se, because it’s likely to be resolved in the vertex shader. So I would set up explicit planes that represent the sort of the geometry boundaries that you want to be doing your raycasting against in general. Another thing I wanted to check with is like, do you have a do you have a PR connection set up for your setups between a local computer and the headset, or are you exclusively working on the headset by itself at this point?
Andrew Thompson: I have the. Yeah, I have a local server. I run on the computer and then just run that onto the headset. Right.
Speaker9: Do you have your ability to hit a real time, like how a real time connection between the the computer as client and the and the headset as client?
Andrew Thompson: Oh, like wirelessly? No, I don’t have that setup.
Speaker5: Yeah, yeah.
Speaker9: So, like using a peer connection or something. I’ve got a CodePen up and running, but I’m, I haven’t keep on building for.
Andrew Thompson: I haven’t seen the need for that yet with the testing. Do you think it would give me something I don’t currently have?
Speaker9: So one of the things that is really interesting and fun about. Vacation hours or safari and visionOS is the ability to be able to do speech speech synthesis and speech recognition or Zimbabwean. You need to lifted a little higher. There we go.
Frode Hegland: It’s pretty cool, right?
Speaker5: Yeah.
Speaker9: So unfortunately, at this, at this moment in time, quest does not have the ability to do either speech synthesis to my to my recollection or speech recognition. And so if you want to if you want to leverage those, then you know, your, your Mac or your PC or whatever is right there and it has those full supports. But what you would need to do is do the speech recognition and synthesis on your Mac and then have headphones. So in order to do that, you will need to have a real time connection between your between your, your Mac as client rather than merely a server in order to be able to send that stuff back and forth. But it’s a it’s a way of being able to back it would be a way of being able to backfill the gap in functionality and plus browser so that you have the ability to kind of lean on those, because on vision you’ll be able to do like on Apple Vision Pro you’ll you’ll be able to do that exclusively inside the headset. But you, you can’t you don’t have all of those capabilities right now. On on quest. So if you have a framework that allows you to kind of backfill that, then you’d be able to play with those because, you know, one of the things that if you look back at all of the all of the marketing material for, for Vision Pro as it was initially initially announced, then they said, you can use that. You can interact using your eyes, hands and voice and that and voice is not like a, just a, you know, an also ran it’s a really interesting and important and valuable part of it. So having the ability to get something that’s relatively flexible up and running, to be able to like, manipulate things in web with voice is pretty often.
Andrew Thompson: Yeah, I like that. Definitely look into it.
Frode Hegland: Sorry. Hang on. Let me mute her.
Andrew Thompson: And I have seen I haven’t looked into it because we haven’t been working on the voice to text prompting yet, but I have seen some stuff in webXR that supports verbal components. And I wonder if they’re working on some like, their own way because I the quest has a microphone, so I wonder if they’re kind of trying to make their own implementation of it. It’s probably not as good, but there’s a chance we could.
Speaker5: So I mean.
Andrew Thompson: They’ll have it work.
Speaker9: It totally depends that like quest has a microphone so you can do VR chat, all that kind of stuff. All it would take is that you would need to have some kind of voice recognition. But but it’s not going to be happening on device or the device isn’t managing the recognition for you per se. What’s what it’s doing is shipping the voice samples off to someone else, whereas on.
Speaker5: Just one quick second.
Frode Hegland: While you and Andrew are talking, which is fantastic. Rob, can you put on your headset? I’m going to call you and FaceTime. Okay? We’ll do a bit of testing on that side of things. Sorry, Randall. Please continue. Obviously.
Speaker5: Yeah. No problem.
Speaker9: Do you have. Oh, no. You can’t get apps at all on your on your device yet because you’re in the UK. But you should be able to get zoom, I think.
Speaker5: Well, you know, that’s.
Speaker9: What happens when you jump the queue. The jump in the national queue. If you if you’re really serious about it, you should have migrated to the United States.
Speaker5: I
Frode Hegland: I am a developer, you know. I should be able to have a developer thing, shouldn’t I? Because before it was public, if you were a developer, you could, but I don’t know.
Speaker5: Yeah, no, I’m.
Speaker9: I so that’s something that I’m in the details of the decisions for.
Frode Hegland: Yeah. No don’t worry. Okay. Yeah.
Speaker5: So.
Speaker9: Sorry. Go ahead. You don’t. Brokering with Rob.
Speaker5: No no, no.
Frode Hegland: You want Andrew, please. I’m sorry. I’m going to call Rob now.
Speaker9: Okay. So yeah, the quest definitely has all of the hardware required so far, but it’s got a microphone and the ability to, to, to deal with it. What people must be doing there is sending the voiceprint off to some kind of recognition server. And so it’s kind of up to you as to how complicated you want to go with that. What you could do, for example, is send it to your Mac, send a voice to your Mac, and then run sort of recognition through that somehow. But what would probably be easier is to not connect to your headphones, to your your vision, to your quest at all and actually just have like, I have AirPods, they’re connected to my Mac right now. And you could just do that. You could just have AirPods or other headphones that are connected to your computer. And as long as your computer is set up to send those things over the network, then you’re not you’re not sending voiceprint data anywhere. And at some point, it might be the case that quest, you know, gets gets their speech synthesis and their speech recognition. And then you could you’d be able to deal with it entirely locally on device. But until such time, you know, unless you have a visionOS device to be able to test on, then you won’t you wouldn’t be able to do that. So it may be I’m, you know, not your not your boss, but it may be worth investigating a way to be able to kind of imagine and play with those things because voice is pretty cool.
Andrew Thompson: Right. I just a little bit of a Google search where we were chatting. Because I vaguely remember seeing something in, like, the experimental settings inside the quest. And it does look like they’re working with voice commands. Yeah. Last updated 18 weeks ago. So it’s fairly new, but I don’t know if we have access to that information inside webXR. That might just be a something to use to navigate their settings. We’ll see. I have to tinker with that. It looks like they’re working on it, though.
Speaker5: Well. So.
Speaker9: So quite a Not quite what, what do you call it, I guess. They haven’t given the OS a name, I don’t think, but the quest OS definitely does have sort of nascent, I would say speech recognition capabilities. What I don’t have a lot of hope for. Not out of disrespect for the folks, but just that I think they haven’t gotten around to it is plumbing that into the web speech recognition API. And I will I’m actually going to be going to meadow just to visit for the webXR face to face in March. So if it doesn’t come up before then, maybe I’ll talk to them tomorrow about it. I’ll check where they are with it.
Andrew Thompson: Yeah, that’d be really neat to learn. All right, I want to know.
Speaker7: I want to say, you know, in the context of prototyping, I, I think saying I’m going to polyfill whatever I can do right now because I believe it’s a key break or key component for the workflow is fine. And if in the future I don’t need to polyfill it because the operating system or WebEx or whatever down the stack does it for me. Brilliant. But if it does not as long, for example, as you don’t like, literally need an entire data center to run the thing for one person. To me, that’s not cheating. It’s like, oh, it’s not there yet. I delegate this to another normal, I want to say machine because I want to test the usage. I think it’s fine. And for the anecdote, this is also some of the stuff I tried with text to. Speech to text, rather as commands for a frame like. That’s one of the very earliest demos I’ve done. Don’t ever know. Seven, eight years ago or something where, like, you were moving a cube up and down or turning it using your voice. And I’m pretty sure I was cheating by using, like, the text to speech of maybe even windows on a desktop or something, and then sending back the result. It doesn’t make like the most amazing experience again, both in terms of doing the setup itself and obviously latency. But in terms of exploration, being able to like move a 3D object in space. More than five years ago, I think that was that was pretty cool. So in terms of being able to poke at what the future of interactions they look like, I would argue that polyfilling whatever is not there yet is a is a fair practice.
Andrew Thompson: Right. I definitely time honored tradition. Yeah, I think it’s it’s totally fine to use, say, the laptop or the computer microphone or whatever for that. The question just is like a development priority. If it’s already something that’s being implemented on a, in a better way and we just have to wait like a couple months and there’s other more important things we need to develop anyways. No sense in like wasting time implementing that. Unless it’s like, really easy to do. I have no idea. That’s. I haven’t looked into it yet.
Speaker5: So.
Speaker7: In term of speech to text and text to speech, for example, there are kokui that I or others that provide Docker images. If you’re familiar with Docker, they even do a server version. So you basically have something running on your desktop in minutes. Basically, you send it a wave, for example, the outcome of the microphone, and you get some text out. Again, it’s not like, well, I mean, if you have a fast machine, it’s relatively fast. But so it’s relatively convenient to have your own do it yourself basically version at home. I’ll put some link also in the chat.
Frode Hegland: So in terms of development priority under sorry I was muted then. That you’ve tried the Vision Pro now, right? Are you going to try it today?
Andrew Thompson: Holidays. So I’m not going to be on campus. I’ll be working from home. I’ll probably get a chance to test it Wednesday at the earliest, I think.
Frode Hegland: Okay, now that’s fine. But the key thing is the billboarding. So when you move things, they always face you. It’s beautifully done. Beautifully, beautifully done. We, as we agree, we have too much freedom of movement in our environment. So I’ll put now on our websites, the most important thing we need to do is have a mechanism whereby we can move things and they move basically in a sphere, whatever level, the way they are from us.
Andrew Thompson: Yeah, that was the cylinder swipe. We actually have that working.
Frode Hegland: We have the cylinder swipe for the. But we also need a way to to move things back and forth.
Andrew Thompson: Oh, yeah, that’s a separate thing. We need to discuss how we want to do that. But I believe we said at least at the time it was several different cylinders, sort of like at different preset distances. And you can change them in the settings to move them around to a better spot. But that’s where everything snaps to. So it just relatively larger circular distances around you.
Frode Hegland: So this question is largely for Brandel. Of course, the way it’s implemented at vision is you. Touch through looking and pinching a bar underneath the window. Right. And then you have the option to do x, y and z. And it works phenomenally well. So Brendel has for the mural that he did a long time ago similar. You move the mural and it you know, it moves on one plane, but you move it towards you. But it doesn’t wobble. It doesn’t do any roll on any axis except for.
Speaker5: Good.
Frode Hegland: Except for making sure it stays billboarded. Right? So from the center out, it doesn’t do any roll. Does that make sense? Yeah, it makes sense. Right.
Andrew Thompson: So I’m wondering yeah, it’s only rotating on two of the axes I believe. Depends on what type of angular system you’re using. But yeah it’s not flipping around.
Speaker5: Yeah.
Frode Hegland: So the thing that we’ve discussed that was annoying with immersed with Virtual Monitor was too easy. Once you set it up really well to accidentally move the monitor. Utterbach. Right. So here’s a question for everyone. Let’s say that we have a special mode that now I’m moving the documents rather than interacting with it, because that is one issue with the vision right now. It’s too easy to select text. In fact, authors. Highlight menu. The first item is going to be deselect because it happens way too much. But to separate interacting with whatever objects and moving the object in space, should we have something along the lines of a keyboard shortcut?
Speaker5: Obviously. I mean.
Andrew Thompson: As in most software on computers would have like you’ve got the hand grab tool to move the canvas around and then you’ve got the mouse to click and edit. And if we think along the same lines, those are different tools. You said keyboard shortcuts. The equivalent for that is our prism menu. So you’d have an easy tool you can just select to switch between like drag and interact I suppose.
Frode Hegland: Okay, I would. This is really interesting. I need your attention here because this is your territory. Because I see you’re posting interesting things. So the notion of doing that we have for the control thing, which makes sense, right? I would prefer not to tap somewhere. Then you’re in move mode and you tap out of move mode. That just seems very clunky. It’s very I know that’s not what you’re saying, but I’m just wondering, is there the equivalent of holding down the command key? So now your movement. Right. So what should we consider doing.
Andrew Thompson: Should we. You’re saying do a gesture on the other hand. And then now your hand can do stuff because you have like a fist. This one now drags and then it’s, it’s like.
Frode Hegland: Soft space uses the fist. Once you’re in fist, you’re moving the object or the environment.
Andrew Thompson: Okay. What I discovered when I was testing with a bunch of the gestures is that so many of the things we think is an intentional pose, we do unintentionally all the time, like, say like a fist grab, right? I was thinking that would be fine. And then I realized when I was testing, I kept triggering it because I’d set my hands down on my table and I’d kind of have them like this just resting. And that’s a fist. So many little things you don’t assume is an intentional gesture trigger all the time.
Frode Hegland: This is the interesting
Speaker5: New thing.
Frode Hegland: We didn’t realize before that. I mean, could we? One second. Probably. Could we? Maybe. Logically within the system. Are we able to say the hands must be held out? Can’t be down. Both hands must be first. And that’s how we move.
Andrew Thompson: Yes, but also that kind of kills any of the ease of use that becomes super rigid, which maybe that’s what we have to do. I think it’s better to be too rigid than have everything triggering by accident. Ideally, we’d find a compromise. Okay.
Frode Hegland: Thank you. Fabian, please.
Speaker7: Kind of lost my train of thought there, to be honest. In term of yeah gesture. Basically, one indeed has to try and has to try legitimately because indeed, if if you only try for half a second for the gesture and you start your normal flow and then indeed you don’t have some kind of collision with poses you do naturally or, I don’t know, you hold your chin or whatever gesture that are not gesture but still end up being recognized. So I think it is very important to try and not just for five seconds in term of what would be the right one, let’s say quote unquote. I think it’s actually interesting to switch, for example, from the quest three to the Vision Pro like everybody, I mean Randall would know best, but I imagine that everybody that comes from the quest world to the Vision Pro can’t help but reach forward and try to touch things rather than, quote unquote, just look and pinch. We just started to get trained for the last five or so years to move our hands to, to reach and touch, and now that’s not needed anymore. So I think it’s also if if the target user is either quest three or Vision Pro. Who who masters the device? Well, versus a newcomer versus an either or, namely that it should work on both quest three and Vision Pro not just technically, but also that the gestures will be relatively fast to use. Then it’s it’s not the same question, basically that if we say, oh, it’s just for Vision Pro user or just for quest user because some of the gestures are just not the I don’t want to say culture, but the, the expectation in terms of what the OS taught users.
Frode Hegland: That makes a lot of sense. Andrew, how was currently how was the prism rotated? What’s the interaction?
Andrew Thompson: It’s the wrist tap. Oh, wait. Like how it appears or how it once you’re in the menu? You just tap the menu buttons because it’s supposed to be like main menu and then like submenus.
Frode Hegland: Maybe if we have on the sides of the we have a circle, spheres. Not spheres, cylinders, flat cylinders like coins. So if you touch them, that indicates you want to spin it. You know, this is a detail, but the point.
Speaker5: Is then.
Andrew Thompson: You have of running thing of you have to find which menu has the tool you’re looking for. And I don’t think we should have the user do that.
Speaker5: Okay.
Frode Hegland: This is the the question is maybe the prism rotates and one of the three sides is movement.
Andrew Thompson: I just realized we may have a fundamental misunderstanding on how the submenus work. Okay, here’s how I’m picturing it. Right? The main menu is always the same. That’s got all of your main settings. And if you have, say, like tools that are frequently used, there’s like a little like side area that’s easy to tap, the stuff that’s used all the time. Now some things need to be adjusted. So you’re like, okay, well here’s settings for layout. So you tap layout and it spins. And now you have all the layout options on one of the sides, right. Which will give you options for like moving the distance of the reading plane and stuff like that. Right. Maybe default font size. That’s that’s the submenu, right? If you go back to the main menu, maybe you pick a different submenu. It switches back to the same submenu side of the prism, but the buttons are now different. The only side of the prison that’s always the same is the main menu side. The others are just like like the right click menu, essentially.
Speaker5: Yeah, I agree.
Frode Hegland: With most of that. That’s very good. My only reservation is when you’re in a kind of a library slash references view, and when you’re reading a specific document you will want different controls. So I think that on this prism we need to design maybe on the left hand side in a box or the standards like go to go home and all that stuff. And then we have a section that is context sensitive. But I agree with the notion of just tapping a button there and it flips and you’re now in rearrange your environment or view or something. So that’s good. We’ll talk more about that on Wednesday. I’m glad for that. We had that clarification.
Andrew Thompson: Yeah. I don’t want it to be seeming like we only have three sides of the prism. Conceptually, we have three layers to the menu. There can be multiple different menus in there.
Frode Hegland: We have three sides. One is contextual for what you’re viewing with a little sidebar to go home to go up a level. And then we have layout slash environments. And then the third one is when you press the button here, the controls for it. And these here are the controls for it will change and the contextual will change. Right.
Andrew Thompson: Yeah, pretty much it’s we can decide what we want on those other two sides, right? Right now I have it just set up as like settings and then like sub settings, but it can be whatever.
Speaker5: Cool, cool.
Frode Hegland: Thank you. We’ll go further with that on Wednesday. Fabian.
Speaker7: I’ll try to share my screen very briefly because it’s 57 plus, I see that my you have some visual glitch, so let me know please, if you can see that window.
Frode Hegland: Can say.
Speaker7: So do you see, like a green cube on the top, right? Yes. Okay. Thank you. So I’m just going to play it. It’s using the quest three last week or so. So you can see on the left or robot arm. And when I hold the cube, the color of the robot arm matches it. And when I turn the cube, the robot arm is controlled to move with the same rotation of that cube.
Frode Hegland: That’s crazy.
Speaker7: And so the hope there is to show a little bit of physicality of XR to say, yes, you can manipulate virtual objects, but you can also manipulate physical objects that manipulate other physical objects, like kind of expanding your dexterity. And it works. Not perfect, but it’s relatively reliable and relatively precise.
Speaker5: Very, very cool.
Frode Hegland: Very cool.
Speaker5: Yeah.
Frode Hegland: I think Fabian, I know I can’t rope you into this loan thing quite yet, but the notion of lists of libraries maybe related to some of your other work. So if you have any thoughts on views or displays or anything, don’t be shy. Obviously. And also without trying to rope you in. If you do have some time on a Wednesday, nothing wrong with just coming by for a few minutes.
Speaker7: I sincerely appreciate and I genuinely want to be there. Just like at the moment, it’s Soon it will be.
Frode Hegland: We’re making a lot of mistakes. Right now. We’re making a lot of learning. So, you know, we’ll get to a good point. But I am glad that in September, one of the things people will see right up front is stuff. And then a focus on the library slash references. They can view it in useful ways as an academic. And they say, oh wow, this is different. So once we get the basic mechanics we can work on spatializing it. Having two 300 references in a space is going to be really, really interesting to see how we can procedurally move them around. So I think we have 20s left, roughly.
Speaker5: Any.
Frode Hegland: Other thoughts for today? Look forward to Wednesday, Friday and all of you on Monday. See you later, everyone.
Speaker7: Take care.