The Letter
Concern 1: Most Academic Text is Frozen and Non-Interactive
The free and fluid flow of ideas through citations are crucial to maximize academic and scientific discourse. In my PhD research at the University of Southampton I was surprised when I learnt the state of academic documents. These are either ‘frozen’ and archival PDFs or HTML where links are not guaranteed to work, and in both cases interaction is sparse. Furthermore, even when citations are followed, the original authors have often not even fully read what they cite, where the cause is related to the amount of effort dealing with citations (Agarwal, Arafa, Avidor-Reiss, Rupin 2023). With the situation that freshmen college students are no longer as literate as they once were (R.H 2024), the we need to consider how text can deliver more satisfying interactive experiences.
The result of the lack of flexibility in how we interact with citations has been reported widely, including in The MIT Press Reader (G.L, R.L) (This citation highlights the problem of manually crated citations, where I was unable to ascertain the publication date of the article. The approach below, includes a solution for such missing information and user-added errors).
Approach to Solution 1: Parseable Appendices
My PhD thesis ended up with the simple notion that “if it’s important, write it down” and “if it should be used, make it clear and easy to find”. This resulted in the ‘Visual-Meta’ approach where the academic standard ‘BibTeX’ for the document is added as an appendix at the back of the document, easily available for any system to parse.
This information initially held the document’s title, author, date etc., and has subsequently been extended to include further information, including the references it contains. It has been demonstrated to work and is fully based on standard open formats. Vint wrote about this as an editorial in ACM Communications better than I ever could, that this approach adds “an exploitable self-contained self-awareness within some of the objects in this universe and increase their enduring referenceability” (V.G.C 2021).
Visual-Meta has of course been automatically appended to the document you are looking at now.
The result is that LLMs can easily and correctly extract information. If you like, try a prompt in an LLM such as “Find whatever metadata you can in this PDF and present it to me in JSON” to see the results of this process, which only extracts information, makes no inferences and does not need to consult online sources which may have broken since the document was produced.
In addition, custom software (such as my own ‘Reader’ for macOS and visionOS) can provide richer interactions, including with citations such the ones above, which you can double-click to see the reference information of, and if you have it in your personal library (as can be expected if the papers are within the user’s field) you can click on the title and the PDF will open up immediately. It further means that users can copy from a paper and paste to cite, removing the chance of errors occurring.
Concern 2: Interacting with & Sharing of Knowledge ‘Sculptures’
The age of AI is upon us, with all the benefits and issues that entails, one of which is the linear nature of AI chatbot dialog. Additionally, I think it has become clear that the age of XR is right around the corner. I believe in the utility of working part of the day in XR having tremendous benefits, including being able to interact not only with our information, but in our information, where the full environment is active, not only the document ‘brought into XR’ as a floating artifact.
The information spatialized could include everything from citations trees to chemical compounds, planets and graphs, engines and networks. The key presented here is to allow for the spatial positioning to be small and shareable, separate from what is positioned, to allow for large datasets to be presented in myriad ways, while still fitting in traditional academic documents where required.
‘Being in the information also relates to AI dialog, where results are currently primarily presented as a linear ‘authoritative’ steam, and can be presented as configurations, or ‘sculptures’ of knowledge for the user to interact with in XR.
This then really presents two related concerns: How can we best interact with our knowledge as a true ‘volume’ of spatial knowledge, and how we can share this?
Approach to Solution 2: Simple Spatial Knowledge Sculptures
Concern 1 addresses how we can get robust, accurate and useful information into the XR environment (as well as in traditional, framed, digital environments) to build such sculptures of knowledge. Associated with this is how we can share the spatialized knowledge. There are many computer formats for displaying rich 3D contents but none so far are focused on ‘knowledge’ rather than appearance. While there are for 2D knowledge graphs and so on, 3D in open format has proven elusive. The approach here is therefore to include as little information as possible about the 3D knowledge sculpture for easy transfer between systems. This is based on separating information about the knowledge objects from their positions, making it possible to embed useful 3D knowledge shapes even in standard scientific and academic papers. The PDF you are reading now is a demonstration of this. When viewed in my software ‘Reader’ in an Apple Vision Pro, you can click/tap on <1> to see all the citations and defined concepts in this letter in 3D space which you can then walk around in and interact with.
The Overriding Concern: Paradigm Freeze
We are looking at a once–in–a–species opportunity. There has never been, and never will be again, a first time we ‘enter into’ knowledge at scale, as is about to happen. When our entire field of view can be brought to bear to unleash knowledge, we have a serious responsibility to take this seriously. Walking around a knowledge space is a very different experience and interaction than looking at our information framed in displays and will have very different potentials for unlocking new aspects of thought.
Future generations will not have our naivety and previous generations have not had the powerful devices we have today. The opportunity is to really take ownership of what working in a headset–and other implementations of XR–can be, and not leave it up to only market forces, but to truly experiment, and to experiment with robust information, in deep and dialog with the user community. We can neither afford to have our knowledge in XR be owned, such as Microsoft owned the Office formats for decades, neither can we afford to have our imagination limited, as happened with desktop computers with the graphical user interface which, once generalized, become the ‘truth’ of how to interact with computers. Fully immersive knowledge spaces must be developed in myriad directions and not become beholden to any one single paradigm. Only the open flow of robust spatial data can enable this. Our first concern here is the frozen academic documents of the past and how to allow them to contain rich information. The following concern is the freezing of the very way we look at the future, once big business interaction and consumer market paradigms for how we should interact take hold.
The Desired Result: Richer Knowledge Interactions
The research we have done so far has truly only scratched the surface of what working inside knowledge can enable. So far we have ‘seen’ how organizing and clustering large volumes of knowledge interacts with our brains in more effective and natural ways than trying to extrapolate from what we can glean from screens. These are early days and it is important to be honest and state that we truly do not know what the potential for knowledge in space can be, but experiment to experience what it can be will be crucial to develop this understanding.
The Request: Collaboration
If you feel this approach fits within, and extends what you are already doing, I would be honored to figure out how to make it work. I am available as a remote researcher (I live in London) as a collaborator with any of your team if they find the work interesting. I can imagine Pattie Maes who is a legend in the field, or Hiroshi Ishii, whom I was honored to have been introduced to by Doug Engelbart, a long time ago.
I see the issues around connected knowledge and spatial extension of our minds to be of the utmost importance to improve and I will continue to do what it takes to address what can be done to continue to ‘augment human intellect’ and communication, with text at the center.
What I bring to the table: Proven Dedication, Insight & Results
What I bring to this work is a proven dedication with 14 years of hosting the annual Future of Text Symposium, the most recent at the Royal Society, co-hosted with Dene Grigar, PI for the Sloan Foundation work, and Ismail Serageldin, founder of the Modern Library of Alexandria: https://thefutureoftext.org/future-of-text-25/. Out of these Symposia I have published 6 collected works, the latest one which just dropped December 2025: https://futuretextpublishing.com/v6/. The Alfred P. Sloan Foundation project, under Josh Greenberg, resulted in a multitude of insights accrued from experiments to experience working with text in XR, including weekly recorded Open Office sessions, available, along with the experiences, at: https://futuretextlab.info.
The active, professional, community I have built around this work includes Bruce Horn, creator of the original Macintosh Finder, Barbara Tversky, author of Mind in Motion (B.T 2019), Ken Perlin, academy award winning computer graphs pioneer and NYU professor, along with many others from different countries, professions and perspectives. Doug Engelbart, my mentor, wrote that he did not know anyone as dedicated to unleashing the power of text than me, an honor I will be able to realize further with an association with you. The last two years of working with the Sloan Foundation opened doors and grew the community.
My software ‘Author’ and ‘Reader’ was available day-one in the visionOS App Store, allowing me to experiment with what authoring in XR can be like, with the resources of a small, independent software developer.
This Document
Interactive PDF
If you read this paper in Reader for macOS, further interactions will be available because of the Visual-Meta, including the ability to double-click on citations to see the full reference information and to jump to the sources.
https://apps.apple.com/app/liquid-reader/id1179373118?mt=12
XR
This paper can also be read in XR. If you use the ‘Reader’ app in Apple’s Vision Pro, this document will be presented as a ‘flat’ PDF, which you can then interact with to ‘enter’ the information presented.
If you click on <1> you will get one view of the knowledge in this document, and if you then click on <2> for a different layout.
(Search ‘Reader’ in the visionOS App Store should you wish to experience this view)
Audio
In the interest of being open to experiment with multiple media for thought and expression tied together with text, I also produced a music spoken word AI co-orchestrated version of this pitch to you, available privately. This is also a new media, in terms of music production though AI being available with such ease, helping inspire further the written word by the thought being presented very differently.
https://youtu.be/QfJHfEu3cSA
And in more poetic structure at: https://youtu.be/NNcKpvBalgc