Document Interactions

Document Interactions refer to reading documents in XR, either individually or multiple in the same space. Our user group of academics uses PDF predominately but in this project we make the assumption that we have perfect origin text and metadata so we will use HTML where available, such as from ACM over the recent years, and Visual-Meta transmitted in JSON. This is to support the Use Cases and is in conjunction with Library Interactions.

We aim to support reading styles for different aspects of academic reading, consistent the Use Case interactions of Find, See Connections, Read and Annotate.


  • Find. Select any text or element to search in the document itself or in the Library or external.
  • Scan/Quick Read. A quick scan to get an overview to determine the paper’s relevance and interest for the user.

See Connections

  • ‘Pull Out’ to view in context with the user’s own articles and external sources in the user’s Library or external document collections.
  • Explode is more unique to XR, though a printed paper could have been cut in ‘Ted Nelson Style’ and pinned on a cork board for a similar effect. 3D representations of the elements of the document will be a key exploration in XR. This is to see connections and structure within the document.


  • Read. Traditional reading of going through the paper.
  • Deep Reading. Reading the paper thoroughly, spending time on different sections and seeing how they connect with other sections.
  • ‘Re-Implement’. The key is to attempt to virtually re-implement the paper: that is, making the same assumptions as the authors, re-create the work.


  • Annotate the paper with annotations/highlights/strike throughs and notes for the user or another person, such as when preparing to write an academic article, editing review articles or assessing a graduate paper. The notes and annotations should be available for the user in the Library.

Enabling Affordances


  • Pages. Change how many pages are shown, presented as PDF (Fx: 1, 2 page spread, all horizontal, all as Wall)
  • As text scroll. Full text of document rendered as though HTML page
  • Sections. Choose what sections are shown
  • Cuttings. Pull out sections or text
  • Connections. See connections (Reference links to where it appears in the document and to source, URLs to source etc.)

Interactive Elements

  • Any Text. Select arbitrary text for contextual actions, such as in Reader iOS/macOS/visionOS, Fx: Copy, Copy as Quote, Copy as Citation, Look Up, Search Web.
  • Citations. Citation in body of document (to see Reference information etc.)
  • Reference section.
    • View sources
    • View in document
    • Sort by author, title, date etc.
  • Abstract/Summary
  • Entities (can be AI extracted) such as names of persons, institutions, technologies etc.
  • Glossary viewed from context menu, in list or on Map

Future work

  • Expand contents of document into XR space. This will include the means to annotate what sections of a document contain extended data, either automatically or manually, and the means to extract this information in XR space.
  • More than one document in the same reading space, out of the Library so to speak.