AI Summary
This meeting, moderated by Dene Grigar with Frode Hegland as the main presenter, focused on “Shapes and Transitions” in XR environments, particularly exploring how knowledge objects can transform between different representations. The discussion centered on moving beyond the metaphor of folding paper to more fluid transformations of information that allow for different views, connections, and analysis. Key topics included the cube as a fundamental shape that can unfold into different views like matrices or spreadsheets, the use of gestures to manipulate these objects, and how different shapes might serve different purposes for knowledge representation. Bob Horn shared his extensive work on visual scaffoldings for thought, presenting 25 different ways to organize ideas visually. There was also significant discussion about how to balance freedom and constraints in these environments, the importance of developing intuitive gestures, and considering how different users might want different ways to interact with information. The meeting touched on the challenges of creating tools that others can use in WebXR, the need to document ideas and approaches for future reference, and the philosophical underpinnings of how we approach knowledge representation in spatial environments.
Frode Hegland, Dene Grigar, Bob Horn, Jim Strahorn, Mark Anderson, Tom Haymes, Peter Wasilko, Alan Laidlaw, Aspasia, Astral_Druid, Brandel Zachernuk, Karl Arthur Smink, and Rob Swigart.
AI: Speaker Summary
Frode Hegland: Presented the main concept of “Shapes and Transitions” in XR, focusing on how knowledge objects can transform between different representations. He introduced the idea of the cube as a basic unit that can be unfolded into matrices or spreadsheets, revealing different aspects of information. Frode emphasized how stretching and manipulating these objects could reveal headings, references, annotations, and other metadata. He was particularly interested in the gestures that could facilitate these transformations, such as putting one’s hand inside a cube and opening it to expand the information. Frode stressed the importance of documenting these ideas for future reference and invited participants to share their thoughts on what shapes and transitions might be useful.
Dene Grigar: Moderated the discussion and emphasized that each object should have its own indigenous shape and behaviors rather than forcing all objects to conform to a single model. She suggested that different shapes like accordions, fans, cubes, or prisms could be used depending on the nature of the information. Dene highlighted how traditional paper-based formats with two-sided pages can disrupt thought flow, especially for current students, and how new spatial formats could provide more continuous experiences. She was interested in how to make these tools accessible to users who aren’t programmers and suggested that Bob Horn’s visual scaffolding work could be presented at a future event.
Bob Horn: Shared his extensive work on “platforms for thought” or “scaffoldings” – 25 different ways to organize ideas visually. He showed examples of these scaffolds, including argumentation mapping and other structured formats that academics, businesses, and organizations use to organize thought. Bob suggested that each of these scaffoldings could be considered as a “cube” or knowledge unit in the XR environment. He was interested in how matrices could be applied to any of his idea chunks and requested a list of gestures being considered for the environment so he could think about them more.
Jim Strahorn: Questioned whether “cube” was the right term, suggesting that “rectangular prism” might be more appropriate for many knowledge objects like books or videos, while reserving “cube” for truly special applications. He also emphasized how Frode’s matrix concept was similar to Bob Horn’s 2050 infomercial grid from 15 years ago. Jim highlighted the importance of developing “new, better plural ways of thinking” as an objective for the group, connecting learning, inventing, absorbing, relating, and contextualizing as different cognitive processes that could be enhanced through these spatial tools.
Mark Anderson: Raised concerns about losing context when expanding one object in a spatial environment, suggesting that the power of spatial hypertext comes from seeing relationships between multiple objects. He noted the importance of selective information display and raised concerns about data that is “lumpy” rather than evenly distributed, questioning how their display mechanisms would handle this reality. Mark also suggested writing in hypertext first and linearizing only when necessary, rather than trying to convert linear text into hypertext after the fact.
Tom Haymes: Cautioned against letting metaphors become prisons, emphasizing that the focus should be on propagating ideas rather than getting stuck in technical metaphors. He argued for the need to easily slide between metaphors and for having interconnected information spaces rather than isolated ones. Tom suggested that participants could “decorate their cube homes” to create idea spaces that others could tour, with breadcrumbs connecting back to their own spaces – visualizing academic discourse. He also noted that buttons are merely facilitators for action that might not be necessary in an XR environment.
Peter Wasilko: Suggested abstracting out concepts like space configuration and command types, giving different commands distinct looks to indicate their functionality. He proposed exploring physics-based simulation with gravitational fields to make selection easier, and emphasized the need for layers of abstraction beyond modeling concrete things. Peter also highlighted the importance of being able to linearize and extract content from 3D environments so it can be transmitted through 2D channels, suggesting a markdown-like format for describing spatial structures.
Alan Laidlaw: Shared an example from the HTML review that demonstrated innovative scroll-based navigation where text would change and rearrange as the user scrolled, creating a unique reading experience with peripheral annotations that appeared and disappeared. This example showed how basic interaction models could be reimagined, creating multiple reading pathways through a text without the need for explicit linking structures.
Karl Arthur Smink: Described his work on developing a “discretized gesture library” for hand tracking, treating each finger as an indexed element with binary states (curled or extended). This approach allows users to define hand gestures that correlate to specific postures and link them to particular code actions, which could be saved as preference documents. Karl also commented on the challenges of input in XR environments and the need for context or dimensionality to provide more options.
Brandel Zachernuk: Made technical contributions about hand pose recognition, noting that full specification would require an orientation vector for the palm and consideration of multiple degrees of freedom for the thumb and forefinger. He confirmed that Karl’s representation was similar to one he had independently conceived, suggesting convergent thinking in this area.
Astral_Druid: Introduced themselves as coming from VRChat with a community called the “Transplanter Ecological Society” that discusses how the metaverse should be shaped for human good. They described their vision of hosting the metaverse in public libraries as a “global commonwealth” where everyone owns the assets and resources are distributed through the metaverse. They emphasized the universal nature of play and games as a foundation for cooperation.
AI: Topics Discussed
WebXR Discussion
WebXR was mentioned primarily in the context of the challenges of creating tools that others can use. Dene Grigar noted that it’s very difficult to code in WebXR, raising the question of how to make it possible for users to author their own content in these environments. She contrasted this with tools like Unity and Unreal Engine, which would make development easier but wouldn’t have the web accessibility of WebXR. Peter Wasilko mentioned that he was exploring Babylon JS as a rendering engine with more functionality than A-Frame for his own side project.
Gestures Discussion
Gestures were a significant topic throughout the meeting. Frode Hegland introduced several gesture concepts, including putting one’s hand inside a cube and opening it to transform the cube into a spreadsheet, or closing one’s hand to collapse a spreadsheet back into a cube. He emphasized that different gestures could be used to pull out specific information types.
Bob Horn requested a list of gestures being considered so he could think about them offline. Frode acknowledged that this information was scattered across various documents and discussions, highlighting the challenge of documenting a developing system.
Karl Arthur Smink described his work on a “discretized gesture library” for hand tracking, treating each finger as an indexed element with binary states (curled or extended). This approach allows users to define custom gestures linked to specific actions, which could be saved as preference documents.
Frode suggested that users might be able to switch between different gesture sets, similar to putting on different gloves, allowing for personalization and flexibility. He noted that touching one’s own body, particularly hands, is unlikely to be accidental and therefore useful for gesture recognition, though technical limitations might still apply.
Brandel Zachernuk added technical details about hand pose recognition, noting the need for an orientation vector for the palm and consideration of multiple degrees of freedom for certain fingers.
Book Discussion
The book was mentioned briefly. Frode referenced a document where he was trying to write a user guide for someone to author for the next volume of their book in their XR environment. He noted the difficulty of explaining how the actions and views should work, suggesting that if you can’t explain something in a user guide, you can’t expect users to figure it out.
Mark Anderson observed that Frode seemed to be “trying to make a book out of a hypertext” and suggested instead to “make the hypertext and then you can serialize it if you have to” – writing as hypertext first and linearizing only when necessary.
Peter Wasilko referenced “The New Media Reader” book as an example of preserving hypertextuality in a linear format through margin annotations and suggested the importance of being able to extract content from 3D environments into linear formats.
Other Topics Discussed
Visual Scaffolding: Bob Horn shared his extensive work on 25 different “scaffoldings” or “platforms for thought” – visual organizations of ideas used by academics, businesses, and organizations. These included argumentation mapping and other structured formats.
Knowledge Objects and Their Shapes: Extensive discussion about what shapes information could take in XR beyond cubes – including prisms, accordions, fans, and other forms. Dene Grigar suggested that each object should have its own indigenous shape and behaviors.
Matrix/Grid Views: Discussion of how information could be organized in matrix or grid formats, with columns for different aspects like outline, references, annotations, and extracted metadata.
Transitions Between Views: How objects could transform between compact (cube) and expanded (matrix) views, and how these transitions could be made fluid and meaningful.
User Interface and Experience: Discussions about how to create intuitive interactions and avoid getting stuck in limiting metaphors like buttons or rigid boxes.
Metaverse Philosophy: Astral_Druid shared their vision of a metaverse as a “global commonwealth” hosted in public libraries, where everyone owns the assets and the library distributes both real world and digital resources.
Learning vs. Inventing: Frode highlighted the difference between learning (where a coherent model exists) and inventing (where no model exists), explaining why developing these new systems is challenging.
Interesting Anecdotes
Bob Horn mentioned that they had created around 50,000 “idea chunks” over the past 30-40 years, presenting a page containing 25 different “scaffoldings” or “platforms for thought” that he had developed for clients.
Dene Grigar mentioned being at the Bloomsbury Hotel in London and asked Bob Horn about where Steve Toulmin (who invented argumentation mapping) might have been in England when developing his ideas.
Frode described a moment where he was crumpling paper on the subway and realized it was relevant to their discussion of moving beyond rigid folding patterns.
Dene mentioned showing the movie “Ready Player One” to her students at a “Geek Movie Night” and described it as “the worst possible iteration of the metaverse.”
Frode referenced a “monster movie” where a character says important knowledge is “hidden in books,” noting that this reflects how physical books literally hide information behind pages.
AI: Concepts Introduced
Knowledge Objects (Dene Grigar): Physical or conceptual items in the XR environment that have their own indigenous shapes, behaviors, and properties. They replace the concept of “documents” to move beyond paper metaphors.
Shapes and Transitions (Frode Hegland): The different forms that information can take in XR and how they transform between each other, moving beyond rigid folding to more fluid transformations.
Matrix/Grid View (Frode Hegland): An expanded view of a knowledge object showing different aspects in columns, such as outline, references, annotations, and extracted metadata.
Scaffoldings/Platforms for Thought (Bob Horn): 25 different visual organizations of ideas used by academics, businesses, and organizations to structure thinking, including argumentation mapping and other formats.
Spatial Hypertext (Mark Anderson): A system where the position and appearance of objects in space has semantic meaning, not just a random grid.
Discretized Gesture Library (Karl Arthur Smink): A system for hand tracking that treats each finger as an indexed element with binary states (curled or extended), allowing users to define custom gestures.
Indigenous Behaviors (Dene Grigar): The idea that each object in XR should have its own natural set of interactions rather than applying the same interactions to all objects.
Learning vs. Inventing (Frode Hegland): The distinction between learning (where a coherent model exists that can be gradually understood) and inventing (where no model exists, making it more challenging).
Gesture Clash (Frode Hegland): The problem of accidental gestures being triggered when users are trying to perform different actions.
AI: People Mentioned
Steve Toulmin, mentioned by Bob Horn as the inventor of argumentation mapping.
Walter Benjamin, quoted by Tom Haymes regarding books as “an outdated mediation between two different filing systems.”
John Berger, mentioned by Dene Grigar as the author of “Ways of Thinking,” an influential book on art history and media.
Doug Engelbart, mentioned by Frode Hegland as his “mentor for a while.”
Sarah Walton, mentioned by Frode Hegland as a friend who teaches fiction writing using spreadsheets for organization.
Ted Nelson, mentioned by Tom Haymes and Dene Grigar in reference to hypertext concepts and Xanadu.
Mark Bernstein, mentioned by Dene Grigar in relation to hypertext history.
James Michael Joyce (should be Michael Joyce), mentioned by Dene Grigar as the author of “afternoon, a story,” an early hypertext work.
Wendy, mentioned by Dene Grigar in relation to creating Microcosm (likely Wendy Hall).
Nick Montfort, mentioned by Dene Grigar as co-editor of “The New Media Reader” with Noah Waldrip-Fruin.
Noah Waldrip-Fruin, mentioned by Dene Grigar as co-editor of “The New Media Reader” with Nick Montfort.
Chris Klimas, mentioned by Dene Grigar as the creator of Twine.
Emily Short, mentioned by Dene Grigar as the creator of “First Draft of the Revolution.”
Nick Sousanis, referenced by Tom Haymes regarding his book “Unflattening.”
Donatello Debris, mentioned by Alan Laidlaw as the designer/developer of the HTML review interface he showed.
Andreea, mentioned by Frode Hegland as someone who was supposed to be at the meeting but had an emergency.
Fabian, mentioned by Frode Hegland multiple times regarding gesture concepts and compiling information about which limbs work well for gesture recognition.
Andrew, mentioned by Dene Grigar as having done the programming on their visualization space.
Ge Li, (Jessi)mentioned in chat by Frode Hegland in reference to a YouTube video he shared with Peter. Also mentioned by Frode Hegland as someone who couldn’t attend due to Monday classes and who had a discussion about learning vs. inventing.
Jenny Nicholson, mentioned by Peter Wasilko in the chat as “queen of extraordinary well researched snark.”
Marilyn Monroe and Einstein, vaguely referenced by Frode Hegland regarding a famous movie scene with a manuscript thrown out a window.
Bob Stein, mentioned by Jim Strahorn in reference to tapestry and showing books diagonally.
Huizinga (referred to as “Sega” in transcript), mentioned by Dene Grigar as someone who wrote about play (Homo Ludens).
AI: Product or Company Names Mentioned
Babylon JS, mentioned by Peter Wasilko as a rendering engine with more functionality than A-Frame.
Unity and Unreal Engine, mentioned by Dene Grigar as tools that make development easier compared to WebXR.
Microsoft Word, mentioned by Frode Hegland as something they don’t want to emulate with “200 buttons.”
Photoshop and Illustrator, mentioned by Frode Hegland as tools that allow complex layouts that don’t scale to what they’re trying to achieve.
Twine, mentioned by Dene Grigar and Alan Laidlaw as a tool that allows for creative text manipulation in hypertext.
VRChat, mentioned by Astral_Druid as the platform where they have their community.
Godot engine, mentioned by Astral_Druid as the engine used by a project called “Sky All Built.”
Basis, mentioned by Astral_Druid as an upcoming platform.
Bark VR, mentioned by Astral_Druid as another platform.
Waterstones, mentioned by Dene Grigar as a bookstore she visited.
Sega, incorrectly transcribed reference to Johan Huizinga (author of Homo Ludens) by Dene Grigar.
PDF viewer, mentioned by Frode Hegland in comparison to how their system should feel different from having separate applications open.
Aqua Net and Vicki, mentioned by Mark Anderson as early spatial hypertext systems.
Factorio, mentioned by Karl Arthur Smink in chat as a game that showcases automation and complexity.
Zuckerberg, mentioned by Dene Grigar in reference to his vision of the metaverse which she disagrees with.
iPad, mentioned by Brandel Zachernuk regarding scroll-redirection used in iPad Pro marketing, and by Randall (in chat) regarding early promises of comic book navigation.
Ready Player One, mentioned by Dene Grigar as a movie she showed her students that represents “the worst possible iteration of the metaverse.”
Google Docs, mentioned by Tom Haymes as something some users find intimidating.
AI: Other
The meeting had an interesting dynamic between discussing highly theoretical concepts about knowledge representation and the pragmatic concerns of implementing these ideas in current XR environments. There was a clear tension between wanting to break free from limiting metaphors like books and buttons while still creating systems that users can intuitively understand.
Several participants emphasized the historical context of their work, from ancient matrices 3000 years ago (Bob Horn) to more recent hypertext pioneers (Ted Nelson). This suggests an awareness that while their XR explorations may seem revolutionary, they are part of a longer continuum of human attempts to organize and visualize knowledge.
The meeting exemplified the challenge of collaborative invention versus documentation – as ideas evolved rapidly through the discussion, participants struggled with how to capture and organize them for future reference. Frode’s comment about the difference between learning (where a model exists) and inventing (where no model exists) seemed particularly relevant to the group’s own process.
Chat Log URLs
https://thefutureoftext.org/future-of-text-25/
https://futuretextlab.info
https://www.youtube.com/watch?v=aJFVk7H757E https://the-next.eliterature.org/works/757/0/0/
https://www.youtube.com/watch?v=-GV814cWiAw&ab_channel=GameMaker%27sToolkit http://www.newmediareader.com/index.html http://www.newmediareader.com/book_samples/nmr-key.pdf https://mitpress.mit.edu/9780262535793/design-unbound-designing-for-emergence-in-a-white-water-world-volume-1/ https://dl.acm.org/doi/10.1145/3506469.3506483 https://www.academymuseum.org/en/CyberpunkVirtualProductionExperience
https://aeon.co/essays/how-evolution-favoured-costly-and-frivolous-animal-play https://galaxy.opensyllabus.org https://monoskop.org/images/9/9e/Berger_John_Ways_of_Seeing.pdf https://www.youtube.com/watch?v=0pDE4VX_9Kk https://aousd.org/community/interest-groups/
Chat Log Summary
The chat log complemented the meeting with additional resources, clarifications, and side discussions. Karl Arthur Smink emphasized that the cube is just a metaphor for simplicity, while Tom Haymes quoted Walter Benjamin about books being “an outdated mediation between two different filing systems.” Peter Wasilko expressed concern about students losing the ability to digest long-form content, leading to a discussion about what constitutes “long-form” in different media.
Several participants shared links to relevant resources, including John Berger’s “Ways of Seeing,” the New Media Reader’s design for preserving hypertextuality in linear form, examples of data visualization like galaxy.opensyllabus.org, and articles about play in evolution. Tom Haymes suggested setting up a Wiki to organize their ideas and creating an “automatic transmission creator tool” since “the audience of digital mechanics is very small.”
The chat also included technical discussions about gesture recognition, with Brandel Zachernuk confirming Karl’s approach to hand tracking and adding technical details. Karl noted the cognitive limitations of complex dimensional spaces and how recognition memory allows us to work within contextual windows of larger structures.
Frode used the chat to share working documents and highlight core questions like “What shapes do you want your knowledge to have?” and “What shapes do you want to ‘think on’?” Tom Haymes advocated for avoiding buttons in XR environments since they’re just “facilitators for action” when more natural interactions are possible.
Overall, the chat enhanced the verbal discussion by providing references, visual examples, and allowing quieter participants to contribute their perspectives.