Themed discussion with a presentation by Frode Hegland, who also moderates today.
AI: Summary
This meeting focused on spatial thinking in XR environments, specifically addressing how to manage and manipulate knowledge objects in three-dimensional space. Frode Hegland presented a framework distinguishing between knowledge objects (useful in themselves) and volumes (collections that can be broken down into subunits), emphasizing the challenges of building XR interfaces without traditional GUI elements like toolbars or keyboard shortcuts. The discussion covered fundamental interaction paradigms including selection methods, billboarding text orientation, scaling, grouping, and environmental design choices. Participants explored the philosophical and practical implications of creating spatial interfaces, debating whether virtual environments should follow physical laws or embrace new interaction paradigms. The conversation also touched on broader themes of human discourse, the democratization of technology, and the importance of building open, modifiable systems that serve diverse user needs.
Frode Hegland, Tess Rafferty, Ken Perlin, Fabian Benetou, Rob Swigart, Peter Wasilko, Tom Haymes, Ayaskant Panigrahi, Edgar Hegland, Jimmy Sixdof
AI: Speaker Summary
Frode Hegland served as moderator and presented the main framework for the discussion. He emphasized the distinction between knowledge objects and volumes, outlined the fundamental challenges of XR interface design without traditional GUI elements, and stressed the importance of building tools for “thoughts in space.” He shared personal anecdotes about weddings and teaching experiences to illustrate points about spatial memory and human nature. Frode expressed concerns about current global discourse and hoped their work could contribute to better human understanding.
Tess Rafferty focused on user experience considerations and the importance of customizable, appealing interfaces for different user types. She discussed her upcoming move to London using a global talent visa and emphasized the need to consider commercial users who want inviting, personalized experiences versus researchers who need functional data manipulation tools. She advocated for involving creative professionals in technology development and stressed the human agency aspect of tool creation.
Ken Perlin demonstrated his technical expertise while maintaining intellectual humility, repeatedly emphasizing that despite extensive technical work, real understanding only comes through practical application. He described his 6×6 meter XR room and ongoing experiments with webXR rendering optimization. Ken shared insights about billboarding text for multiple users and the psychological differences between personal versus architectural information placement. He also offered philosophical perspective on human nature and tool-building.
Fabian Benetou provided technical insights throughout the discussion, clarifying current XR capabilities like keyboard shortcuts and modifier support. He shared links to demonstrations of vector manipulation and probe concepts, emphasizing the open-source philosophy of making technology modifiable by all users. His contributions were often practical, addressing implementation feasibility and existing technical solutions.
Rob Swigart contributed creative perspectives, suggesting randomization as a tool for sparking new configurations in writing, similar to the New York Times spelling bee game. He recommended London neighborhoods based on literary history and proposed voice activation as a simple solution for text input in VR environments where typing is difficult.
Peter Wasilko presented highly technical concepts including reified view specifications, multi-dimensional data exploration through probe interfaces, and end-user programming capabilities. He advocated strongly for open-ended tools that allow infinite combinations rather than narrow, specialized applications. His contributions included detailed descriptions of interaction paradigms and references to historical computing concepts like HyperCard’s multi-level interface approach.
Tom Haymes focused on philosophical and pedagogical aspects, emphasizing the importance of perspective in both photography and information visualization. He discussed teaching experiences across different modalities and the value of recording conversations for persistence. Tom stressed that all technology should augment human intellect and expressed optimism about progressive augmented enlightenment while acknowledging the limitations of expecting technology to solve fundamental human nature issues.
Ayaskant Panigrahi explored questions about shortening the learning curve for XR interfaces and suggested using AI techniques like RAG (retrieval augmented generation) to capture contextual information such as audio notes about spatial arrangements. He emphasized the need for consistency in virtual element behavior to manage user expectations effectively.
Edgar Hegland made a brief appearance as Frode’s son, sharing about his camping trip and the animals he saw, adding a personal family moment to the technical discussion.
AI: Topics Discussed
What was discussed regarding WebXR? Ken Perlin described his extensive work optimizing webXR rendering to achieve fast performance for line drawings, text, and editable content using single mesh rendering on Android devices. Fabian mentioned that webXR environments have built-in utilities for vector computations and 3D manipulations, making probe-like interactions feasible to implement. The discussion touched on webXR’s limitations compared to native VisionOS software, particularly regarding keyboard visibility and input methods.
What was discussed regarding gestures? The conversation covered various gesture paradigms including pinching for selection (current method in Fabian’s environment), laser pointing (mentioned by Frode as previously used but not preferred), one-handed and two-handed gestures, and probe-based interactions controlled by hand position relative to a center point. Peter Wasilko described a detailed probe interaction system where hand displacement from center would control movement vectors, with nested spheres providing different sensitivity levels. The discussion emphasized that gesture systems need to be self-consistent within each environment.
Were other topics discussed? The meeting covered billboarding (text orientation that follows user perspective), spatial memory and its importance for information retrieval, the distinction between neutral backgrounds and realistic room environments, virtual graffiti and annotation systems, end-user programming capabilities, the philosophy of tool building, global discourse and political tensions, the democratization of technology, open-source development principles, voice input methods, AI integration with RAG techniques, and the historical context of computing pioneers. Personal topics included London neighborhoods, Tess’s immigration process, and family moments.
Were there any interesting anecdotes? Ken Perlin described having a dedicated 6×6 meter VR room in his house designed specifically for XR work. Frode shared experiences from a lavish wedding at the Four Seasons by Tower Bridge, where he took 1783 photographs and had political discussions with privileged individuals who showed surprising empathy. He also recounted a “Woody Allen moment” when introducing students to internet history and unexpectedly having Vint Cerf take over the explanation. Tom Haymes mentioned teaching modalities and the value of recorded classes over in-person sessions. Rob Swigart shared working on a game at Activision that became “Hacker” and started as a 3D tunnel racing concept.
Did anyone seem to change their position during the call? No clear position changes were evident during the call. The discussion was more exploratory and collaborative rather than featuring strong disagreements that led to mind changes. Participants built on each other’s ideas and offered different perspectives, but there weren’t obvious shifts in fundamental positions.
What were the major outcomes of this session? The session established a clear focus on “thoughts in space” for the remaining months of the project, with emphasis on basic interactions like creating, moving, and organizing text snippets in XR. Frode decided to prioritize small, focused tests over comprehensive integration, accepting that the project might end with multiple separate URL demonstrations rather than one unified experience. The group clarified the distinction between building for expert “race car driver” users versus eventual mainstream adoption, with current focus on demonstrating the impressive capabilities of spatial thinking tools.
AI: Concepts Introduced
Billboarding was discussed extensively with multiple definitions. Ken Perlin explained it as text planes that follow the user’s orientation, with his system offering both user-following and fixed-orientation modes. Fabian’s implementation involves text that twists when grabbed but billboards to the user’s head position when released, maintaining that angle during head movement.
Knowledge Objects vs. Volumes was Frode’s central framework distinction. Knowledge objects are useful in themselves and important for generating new content, while volumes (books, journals, collections, libraries, the web) are things that can be broken down into subunits and are more relevant for existing collections.
Probe Interface was Peter Wasilko’s concept for navigating high-dimensional data spaces using hand position to control a virtual probe’s movement vector, with nested spheres around the hand providing different sensitivity levels for navigation speed and direction.
Reified View Specifications was Peter’s concept of representing configuration preferences as objects within the environment itself, allowing users to select different spatial mappings of data attributes and potentially combine multiple view specifications.
AI: People Mentioned
Doug Engelbart (mentioned by Frode and Tom as pioneer of augmentation), Vint Cerf (mentioned by Frode as helping with metadata implementation and teaching story), Ted Nelson (mentioned by Frode and Ken regarding reluctance to adopt XR despite document connection relevance), Bill Atkinson (mentioned by Frode as HyperCard creator who wasn’t interested in current developments), Mark Anderson (mentioned by Frode as giving advice to go back to basics), Dene (mentioned as lead PI on the Sloan project), Mark Anderson (mentioned by Peter as creator of ATB reference for Tinderbox), Mark Bernstein (mentioned by Peter as Tinderbox creator), Dene Grigar (mentioned by Frode as project co-PI), Alan Kay (mentioned by Tom and others in chat as visionary), Oscar Wilde (quoted by Ken about standing in mud but looking at stars), Allan Poe (mentioned by Rob regarding Stoke Newington grammar school)
AI: Product or Company Names Mentioned
HyperCard (discussed by Peter and Frode as example of multi-level user interface), Tinderbox (extensively discussed by Peter as example of open-ended tool with run command capabilities), Author (mentioned by Frode as his productivity app ranking #5 on App Store), Google (mentioned regarding privacy concerns and dictation features), Microsoft (mentioned in context of major tech companies), Apple (mentioned regarding HyperCard and innovation limitations), Photoshop (mentioned by Frode regarding different user mentalities), Final Cut (mentioned by Frode regarding different user mentalities), New York Times Spelling Bee (mentioned by Rob as inspiration for random text arrangements), Siri (mentioned by Frode as poor conversational AI), OpenAI (mentioned regarding Ken Perlin’s background), Google Brain (mentioned regarding Ken Perlin’s background), NYU (mentioned as Ken Perlin’s teaching institution), Activision (mentioned by Rob regarding game development), Anthropic (referenced in context of Claude), ACM Hypertext (mentioned as recent conference), Four Seasons (mentioned as wedding venue), Sloan Foundation (mentioned as project funder), WebXR (discussed as technical platform), VisionOS (mentioned regarding native capabilities), Notebook LM (mentioned by Tom for analysis), Steyer.net (mentioned by Tess for AI group)
AI: Other
The meeting demonstrated the interdisciplinary nature of the group, combining technical developers, academics, writers, and philosophers working together on spatial computing challenges. The discussion revealed ongoing tension between creating impressive demonstration capabilities versus building user-friendly tools, with current focus on the former. The group’s commitment to open-source development and democratization of technology was evident throughout, as was concern about broader societal discourse and the role their work might play in improving human communication. The personal elements (family appearances, life changes, wedding stories) added warmth to what could have been purely technical discussions, reflecting the community aspect of their collaboration.
Chat Log URLs
https://mymodernmet.com/phd-infographic-matt-might/ https://docs.ultraleap.com/ultralab/virtual-elements-in-vr.html https://sites.ualberta.ca/~michaelf/IWPASH.pdf https://atbref.com/atbref11/index.html https://www.steyer.net/events/weekly-ai-chat/ https://www.researchgate.net/publication/236899967_VRSearch_An_Architectural_Approach_to_Cyberspace https://git.benetou.fr/utopiah/spasca-fot-sloan-q3/src/branch/main/data/filters/tinderbox.js https://en.wiktionary.org/wiki/yak_shaving https://video.benetou.fr/w/i6v3nB7gFWSPPb7QChuxW7 https://threejs.org/docs/?q=math#api/en/math/MathUtils
Chat Log Summary
The chat log reveals active engagement from participants sharing resources and reactions throughout the meeting. Key contributions included Fabien sharing technical implementation details and demo links, Peter Wasilko providing academic references and expressing appreciation for Mark Anderson’s Tinderbox documentation, Tom Haymes sharing resources about Sun’s Wonderland data visualization environment, and Tess Rafferty sharing her AI discussion group link. The chat showed real-time reactions to speakers’ points, with participants using emoji responses and brief comments to show agreement or appreciation. Technical discussions continued in chat with Ayaskant sharing interaction technique references and Fabien providing specific code examples. The informal nature of the chat complemented the formal discussion, with participants making jokes about corporations, technology history, and personal observations about the group dynamic and broader technology landscape.