XR to Think & Create & Being Creative
Presentation on Authoring in XR for the Alfred P. Sloan Foundation. Introduction and moderation by Dene Grigar. Presentation by Dene Grigar, Frode Hegland and Fabien Bénétou.
Live XR Demos
- Annotated bibliography
(Dene Grigar) - Spatial Knowledge – Thinking Space
(Frode Hegland) - Experiments for Experiences
(Fabien Bénétou), including:
- Live XR experiences: (main) (experimental ring interaction) (environment test) (wrist testing) (hand menu testing) (ring keyboard) (tap keyboard) (Map JSON with keyboard)
- Video Presentations
Dene Grigar, Frode Hegland, Fabien Benetou, Mark Anderson, Andrew Thompson, Peter Wasilko, Brandel Zachernuk, Ken Pfeuffer, Tom Haymes, Jonathan Finn, Ken Perlin, Karl Hebenstreit Jr., Jimmy Sixdof
AI: Summary
This was a midway presentation for the Sloan Foundation-funded Future of Text project, focusing on XR (Extended Reality) authoring and reading environments. The presentation was structured around three main themes:
- XR as an authoring space for scholarly text (Dene Grigar)
- Spatial knowledge and thinking space (Frode Hegland)
- Experiments for experiences with technical demonstrations (Fabien Benetou).
The project spans two years (January 2024 – December 2025) with year one focused on reading and year two on authoring. The team demonstrated various prototypes including text highlighting, annotation systems, virtual keyboards, and spatial organization tools, all built using WebXR technologies.
Key challenges discussed included transitioning between physical and virtual spaces, adapting to non-Euclidean virtual environments, and cognitive ‘code switching’ between different interaction paradigms. The presentation emphasized creativity over productivity, open-source development, and the importance of extending human capabilities rather than just replicating existing tools in virtual space.
AI: Speaker Summary
Dene Grigar served as the moderator and presented on XR as an authoring space for scholarly text. She focused on three scholarly activities: preparing academic articles, editing review articles, and assessing graduate papers, with a primary focus on annotation as their primary case study. Grigar emphasized the importance of transitional spaces in VR, drawing parallels to physical vestibules and praising environments like Supernatural VR for their thoughtful onboarding. She introduced the concept of ‘code switching’ in VR contexts and stressed the cognitive challenges users face when adapting to virtual environments that don’t follow physical world laws. Her vision positioned XR as a creative medium rather than just a productivity tool, citing research linking creativity to happiness and cognitive health.
Frode Hegland presented on spatial knowledge and thinking maps, grounding his work in his ‘liquid information’ philosophy from the 1990s. He demonstrated the workflow from desktop software (Author on macOS) to XR environments, emphasizing the importance of open data formats like JSON and decentralized hosting solutions. Hegland stressed the human mission behind their technical work, quoting Doug Engelbart about complex problems requiring better tools for human augmentation. He advocated for maximum experimentation and creative freedom, explicitly opposing premature standardization. His presentation highlighted the seamless transition from 2D mapping to 3D spatial manipulation, where users can pick up and move text elements freely in space.
Fabien Benetou provided extensive technical demonstrations of their WebXR prototypes, showing over 30 different experimental features including virtual keyboards, text highlighting, JSON manipulation, and programmable “rings” for applying functions to objects. He emphasized the ‘Lego brick’ philosophy where different components can be combined and remixed by users. Benetou demonstrated live coding in VR, real-time customization through URL parameters, and various file format filters (CSV, PDF, DocX, etc.). His approach balanced experimental innovation with practical usability, maintaining an open-source codebase that allows for extensive customization without requiring deep technical knowledge from end users.
Mark Anderson contributed insights about information triage and the importance of abstraction in spatial environments. He emphasized that less can be more when organizing information spatially, noting that visual abstraction (like color coding) can be more effective than literal text labels. Anderson discussed the challenges of premature sorting and the value of emergent organizational structures that develop through human interaction rather than predetermined systems. He supported the experimental approach while noting the practical challenges of gesture recognition and spatial organization.
Ken Perlin offered perspective from his mixed reality research, noting significant differences between VR-optimized and AR-optimized approaches. He emphasized the value of video passthrough mixed reality for development work, where users can maintain connection to the physical world while working with virtual objects. Perlin discussed the evolution from realistic 3D graphics toward pure information objects that annotate rather than replace the real world. His comments highlighted the different design considerations needed when optimizing for immersive VR versus augmented reality applications.
Brandel Zachernuk provided important perspectives on trust, safety, and privacy considerations in spatial computing environments. He advocated for declarative approaches over imperative WebXR for broader deployment, citing his work on HTML model elements and web backdrops. Zachernuk emphasized the importance of considering who controls data and decision-making in spatial environments, particularly regarding AI-assisted organization systems. He supported experimental prototyping while stressing the need to evaluate trust implications of different approaches.
Ken Pfeuffer raised important questions about visual clutter and spatial organization in VR environments. He noted that many VR applications struggle with feeling chaotic when multiple objects and interfaces are present simultaneously. Pfeuffer also contributed insights about haptic feedback and suggested connections to object-oriented drawing research, referencing specific academic papers on the topic. His questions sparked broader discussion about managing complexity in spatial environments.
Jonathan Finn advocated for standardized semantic spaces to improve familiarity and usability in XR environments. He drew parallels to traditional organizational systems like library catalogs and thesauruses, suggesting that common spatial arrangements could help users navigate between different virtual environments. Finn proposed using AI to automatically suggest organizational axes for large document collections, emphasizing the potential for creative and surprising results from automated categorization.
Tom Haymes contributed philosophical perspectives on innovation networks, language translation challenges, and the nature of virtual versus physical reality. He emphasized that innovation emerges from networks rather than individual eureka moments and drew connections between code switching in language and interface paradigms. Haymes also noted the evolution from application-centric to tool-centric computing paradigms, referencing historical examples like PARC GUI and OpenDoc architecture.
Peter Wasilko suggested various technical enhancements including stenography in XR, machine vision for reading physical Lego configurations, and haptic feedback integration. He contributed perspectives on memory organization and the importance of contextual information for recall. Wasilko also made connections to polyglot programming and the potential for curry-like URL parameter systems.
AI: Topics Discussed
WebXR The team extensively discussed WebXR as their primary development platform, emphasizing its advantages for rapid prototyping and experimentation. Fabien demonstrated how WebXR enables live coding and real-time updates without compilation steps, allowing developers and designers to iterate quickly while wearing headsets. The discussion covered URL-based parameter passing for customization, the ability to work across different devices and platforms, and the importance of maintaining open, standards-based approaches. They noted WebXR’s limitations for production deployment while celebrating its power for research and exploration.
Gestures Gesture interaction was a central topic, with demonstrations of pinching, grabbing, and direct manipulation of text and objects in 3D space. The team discussed the evolution from controller-based to hand-based interaction, noting both advantages (natural feeling) and challenges (lack of tactile feedback). They addressed issues with gesture recognition ambiguity, since humans naturally gesture while speaking, making it challenging for systems to distinguish intentional commands from conversational gestures. The discussion included considerations of transparency effects when grabbing objects and the importance of visual feedback for successful interactions.
Other Topics The conversation covered cognitive challenges in VR adoption, including transitional spaces and onboarding experiences. They discussed the concept of “code switching” between different interaction paradigms and the importance of non-Euclidean design possibilities in virtual spaces. Memory palaces, information organization, and the tension between standardization and creative experimentation were major themes. The group explored the balance between productivity and creativity in VR applications, with strong emphasis on XR as a creative medium rather than just a digital productivity tool.
Interesting Anecdotes Dene Grigar shared a personal story about learning English as a second language, specifically discovering that “belly button” was the English term for what she knew as “poop” in Czech, illustrating code switching between languages. Tom Haymes mentioned difficulties translating “You’re pulling my leg” into German and the useful German phrase “Da bin ich überfragt” meaning “I am over asked.” Frode Hegland described his wife taking a photo of him pretending to use a headset at a coffee shop before the Vision Pro was released, which led to their current workflow demonstration.
AI: Concepts Introduced
Code Switching – Defined by Dene Grigar as a term originally from Roman Jakobson (1953) describing language shifts, but applied to VR contexts where users switch between physical and virtual interaction paradigms, often causing cognitive overload and frustration.
Liquid Information – Frode Hegland referenced his 1990s philosophy with two key principles: 1) You don’t learn to use the computer, you learn to use tools that happen to live in the medium of the computer, and 2) The end of isolated information, emphasizing open information and rich metadata.
Non-Euclidean Characteristics – Dene Grigar described how VR spaces don’t need to follow physical world laws regarding force, velocity, acceleration, action and reaction, mass, and scale, allowing for novel spatial arrangements impossible in physical reality.
COIK (Clear Only If Known) – Dene Grigar introduced this concept to describe how their demonstrations might seem clear to regular participants but could be confusing to newcomers unfamiliar with the project.
Information Triage – Mark Anderson emphasized this concept as a key application for spatial environments, suggesting that XR excels at organizing and sorting information in ways difficult to achieve on traditional screens or physical desks.
Lego Brick Philosophy – Fabien Benetou used this metaphor throughout his presentation to describe modular, composable components that can be combined, modified, and remixed by users to create new functionality.
AI: People Mentioned
Doug Engelbart, Ed Leahy, Sarah Walton, Bill English, Jeff Rulifson, Roman Jakobson, Mark Anderson (in context of JSON contributions), Peter Wasilko (data contributions), Vint Cerf, Ismael Ghalimi, Andrew Thompson, Apple (company context), Ken Perlin (object-oriented drawing research), Prince (purple color reference), Descartes (VR philosophy context), Plato (dialogue reference), Jackson Pollock (anti-sort visualization)
AI: Product or Company Names Mentioned
Sloan Foundation, Washington State University Vancouver, Apple Vision Pro, Meta Quest, Supernatural VR, Beat Saber, WebXR, WordPress, Docker, Storyspace, Bernstein’s software, PARC, OpenDoc, Internet Archive, New York Times, NLS, JSON, CSV, PDF, DocX, PMWiki, GLB, Lego, Machu Picchu, Tetons, Giza Pyramids, JSON, JavaScript, A-Frame, GitHub, Total Perspective Vortex, DynamicLand, HTML, CSS
Chat Log URLs
https://video.benetou.fr/videos/overview https://futuretextlab.info/2025-schedule/ https://companion.benetou.fr/index.html https://git.benetou.fr/utopiah/spasca-fot-sloan-q2 https://www.openstenoproject.org https://companion.benetou.fr/found_set_param.html https://dl.acm.org/doi/abs/10.1145/2858036.2858075 https://dl.acm.org/doi/abs/10.1145/3242587.3242597 https://www.youtube.com/watch?v=aE-I69l5yBQ https://video.benetou.fr/w/x2PaLFLyHgpMaAjfhB79T2
Chat Log Summary
The chat log reveals active engagement from participants throughout the presentation, with technical links shared by Fabien Benetou to video demonstrations and code repositories. Notable interactions included reactions to demonstrations (particularly the Lego brick metaphor and haptic feedback concepts), discussions about purple color themes, and references to related research. Participants shared academic papers on object-oriented drawing and connections to Ken Perlin’s VR work. The chat shows enthusiasm for the demonstrations while also revealing some technical difficulties with screen sharing and camera issues. Several participants had to leave early for other commitments, and there were discussions about the upcoming November symposium date (28th-29th, Friday after Thanksgiving). The chat also captured side conversations about language translation, innovation networks, and philosophical questions about virtual reality, demonstrating the interdisciplinary nature of the group’s interests.
1 comment