Co-PI Frode Hegland would like to expand the work the WebXR work done in the Future Text Lab to provide the user with ‘Spatial Thinking Spaces’. This will be in concert with next year’s Future of Text theme of ‘Tools for Thought’ based on the perspective that in an age of emerging AI super-intelligence we need to also build tools to augment human super-intelligence. Sports have athletes. Music has virtuoso performers. Hegland makes the point that we should help cultivate ‘athletes of the mind’ who are virtuosos with their knowledge ‘instruments’, expertly working their knowledge with the finesse of a goldsmith, the situational awareness of a fighter pilot and the focus of a race car driver. The work next year will therefore focus on developing and championing knowledge interaction tools. We will be inviting more people to show knowledge tools and we will expand our efforts to present to new audiences, including universities. In order to do this, continuing to build our own XR knowledge interaction experience will help us developer deeper understanding and give us credibility to reach new communities.
Summary: Create powerfully interactive Spatial Thinking Spaces to augment how academic and scientific users will be able to think and work, drawing on experiences from our own research, the wider community’s research into knowledge representation, as well as more artistic perspectives of TV ‘murder walls’ and movie fiction like ‘Minority Report’. In these spaces the capabilities will be focused on giving the user powerful abilities to change their view and to create views, to configure knowledge in useful ways for deeper understanding and to increase the bandwidth of dialog. The resulting spaces will be easily shareable through standard academic papers, which can be reconstituted into spatial XR configurations.
AI Integration. It is important that this is not an AI focused project, it is an interaction augmentation project which utilizes AI where appropriate. The AI integration Hegland is pursuing and will continue to pursue, is based on the notion that AI can do someone’s work or homework, but not think or learn for them. AI integration will not author for the user, only support authoring. My personal work with AI has been focused on that of his ‘Author’ word processor for macOS. The work the group is doing for XR has been focused on interactivity, where AI can act as an external support.
Key Findings: Basic work in XR works. A key finding of the work over the last two years is that not only is working in a headset for part of the day already a useful for specific activities, the pace of improvement makes it more clear that lightweight, very high quality devices are on the closer, not farther horizon. Already, traditional working on 2D displays projected in XR are at very high quality which is useful when physical large displays are not available. The quality of the displays are such that reading even a traditional academic paper in XR can be more pleasant on the eye than reading on a laptop simply because it can be enlarged to comfortable size and headset display resolution has increased to allow this with sufficient resolution.
Key Finding: Rich interactions enriches work. There is nothing of note available for specifically interacting with academic papers or knowledge in XR on the market. There are scientific experiences, such as for anatomy and physics, but these are not presented as findings or papers. Our work has been to develop interactions with spatial knowledge, where the virtual space serves as substrate for thought, allowing the user to not only write information everywhere in space, but also add links, multimedia and embedded spaces.
The spatial configurations be shaped into different configurations as well as reveal different background environments–or both–giving the user access to a myriad of perspectives at very low mental load. The instant change of the users entire environment has proven to be a surprising benefit, far beyond changing the background desktop on a laptop since the background can act both as a purely visual backdrop to contribute to different user states of mind, and also as active substrate for information presentation and sense making. Our experiments on reading have included reading multiple papers in a minimalist environment, with the ability for the user to interact freely, including instantly access References, something which requires rich data to enable.
Key Finding: Rich data enables rich interactions. In order for the user to experience rich interactions with their information in XR, to ‘literally’ see more and different, the information available needs to be commensurate with the experience desired. This is why the effort for how to deal with data and metadata has been tested, re-tested and refined to include more robust formatting in document appendices and the implementation of a ‘sidecar’ JSON file with is connected to the primary file. If the user’s data cannot flow freely, neither can the user’s mind nor academic or scientific discourse.
Since the great innovations in interactivity during the from the 1960s to the 1990s, where most of the current user interaction paradigms and document formats were established, inertia has meant that improving the data flow has been a very heavy lift. For example .word documents for authoring are still not fully open and .pdf documents still do not contain useful metadata, even though there is technical potential to an extent. There is simply not the institutional will to make the changes. With XR, documents will have to change, in order to deal with the added third dimension, which opens an opportunity to also deal with further dimensions of knowledge in stored and shared form.
Key Development: Sharing of rich spatial knowledge. Working with information in spatial configurations requires new thinking and new perspectives on how to store and share data, and we have developed an approach to openly and cleanly achieve this, in a way that can even be embedded in traditional academic papers for instant re-spatialization when the user opens the document in a headset, allowing the flattened information to bloom into space again. The group is still experimenting with this approach and we are making it as easy as copy and paste to embed spatial configurations of knowledge into traditional documents for sharing, something we believe can become a vital piece of open information sharing of spatialized knowledge. Hegland calls this approach ‘Visual-Meta’.
Proof of concept. Demonstrations of the Visual-Meta concept for XR has been implemented in Hegland’s independent software development company’s product ‘Author’ and ‘Reader’, as described in original Alfred P. Sloan Foundation pitch. This allows for ‘Maps’ of knowledge to be easily transferred from the user’s Author running on a Mac to the XR ‘Knowledge Space’ developed for this. This is done through easy to understand JSON (JavaScript Object Notation, a very basic and easy to parse format developed by early Future of Text contributor Douglas Crockford). The result is that anyone can create such spatial configurations manually if they wish, and any software developer which features spatial knowledge in their work can integrate with minimal effort. Author and Reader acts as demonstrators of the concept. It is important to note that no funds have been spent on Author from the Sloan Foundation work, Author and Reader have been used as proof of concept and testing applications. The spatial configurations can be imported this way or entirely created in XR.
As Vint Cerf says about this progress, it is ‘existence proof’, making it harder to argue that it cannot exist. It now does exist and it now becomes important to open up development of XR infrastructures so that they are user and community owned. This work will need further work, testing and dialog to be truly deployable. The result will be enabling views of user’s knowledge far beyond what we expected to be possible when we started this project, far beyond what is possible to embed in documents through traditional means.
Enabled & implemented experiences. We have just begun to scratch the surface of interactions in XR, where in an instant the entire view can change–something which has been shown to be a major benefit of working in XR beyond even what a single environment can provide. In our project we have made it possible to tap your wrist and instantly you are in a library or in plain space, in an office or a forest, whatever suits the information space you need. Since the environment can serve as a passive background or an active element, configurations of your knowledge will morph to fit, giving you new ways to see and think. Using the Mana Menu you use your fingers as controls, where a simple tap on a finger, as though your fingers are all magic wands, to issue commands in a more intuitive and quick manner than could ever be possible by reaching out and clicking on a mouse.
Continued financial support. Continued support will enable us to build on the infrastructure developed over the last two years, in terms of WebXR environment, information flow and end user experiences, in order to further grow the community and dialog, to then further grow the work, to enable academics and scientists to don a headset, access their work, share their work in real time with collaborators and later as solid documents, including as much or as little of the elements in the space as they want to, publish their work in traditional channels and have readers access their spatial knowledge perspectives.
Work Products. Continued work would include:
- Interfaces. Refining and building interactions via different methods for testing (Mana/Hand menu, Ring menu and more) in order to optimize for a user who is motivated to go past the surface introduction to XR and really ‘dive into’ their knowledge.
- Space Management. Interactions for the user to manage their thinking space to select multiple items and to easily arrange them into configurations and contract them into sections. This work will utilize the experience gained building similar interactions in the Map view in Author, extending it into 3D. Our testing so far has shown the arranging knowledge in 3D space has significant challenges beyond a 3D view on a 2D display.
- Document Sharing. Modes and methods for adding spatial scenes to documents (‘Volumes’).
- Dynamic Substrates. Background/Substrate interactions for purely visual effect and for dynamic layouts of information based on the environments.
- Multi-User. Multi-user experiences with in-XR and out-of-XR connectivity for collaboration.
We have started delivering on the interactions and the data methods (working on robustness and usability) and we are ready to go to continue this work, with the reality that the ideas and work presented will one day compete with how the big corporations will want to also own this work with their own closed formats and closed applications as well as the interaction paradigms, which we believe should remain open.