Ken Pfeuffer, Associate Professor at the computer science department at Aarhus University, Denmark, will present on ‘Evolution of XR input’ with moderation by Dene Grigar.
AI: Summary
The May 19, 2025 Future of Text meeting featured a presentation by Ken Pfeuffer on the “Evolution of XR Input,” focusing on how interaction methods for extended reality have developed over time.
Pfeuffer, an Associate Professor in Computer Science from Denmark, discussed the progression from traditional input devices to eye-tracking and hand gesture interactions in XR environments. His research particularly focuses on combining eye-tracking with hand gestures for more intuitive interfaces. The presentation covered three main categories: general computer input evolution (keyboard to mouse to touchscreen), XR interaction paradigms (virtual hand vs. pointer metaphors), and eye-tracking integration.
Pfeuffer demonstrated several examples of his research, including gaze-enhanced tablet interaction and eye-and-pinch controls for VR that are now being implemented in commercial devices like Apple Vision Pro. The discussion expanded to include concepts of embodiment, the balance between direct and indirect interactions, and the future of input methods.
Dene Grigar, Frode Hegland, Ken Pfeuffer, Mark Anderson, Fabien Bénétou, Brandel Zachernuk, Jonathan Finn, Peter Wasilko, Rob Swigart
AI: Speaker Summary
Dene Grigar served as moderator for the presentation and contributed insights about the importance of direct interaction with digital objects, noting how the introduction of touchscreens with iPhone and iPad created a sense of intimacy that influenced both user experience and digital art. She discussed embodiment issues and how we connect with technology, offering examples like how people experience music and vibrations in disco environments. Grigar also highlighted the historical context of technologies, noting that many “new” innovations actually appeared earlier but weren’t adopted until the right confluence of factors came together. She mentioned her work in motion tracking labs with voice control and referenced the Logos Doctrine about speaking things into existence.
Frode Hegland discussed his preference for trackpads over mice, describing how he experiences a more direct connection with trackpads despite them being technically indirect input devices. He compared this to how driving a car extends one’s sense of self to include the vehicle. Hegland highlighted the importance of keyboard shortcuts for efficiency and discussed tool intimacy, where some users maintain distance from technology while others embrace and incorporate it. He commented on how the rifle becomes an extension of self when properly trained, showing how physical tools can be embodied. He also raised points about evolutionary pressures in technology development and how commercial companies don’t always have incentives to develop advanced text interactions.
Ken Pfeuffer presented his research on XR input evolution, showcasing his work on combining eye-tracking with hand gestures for more intuitive interfaces. He explained how his research has influenced commercial products like Apple Vision Pro, which uses gaze+pinch interaction similar to his 2017 research. Pfeuffer categorized input devices as direct (touchscreen) or indirect (mouse), and demonstrated how eye-tracking can bridge these approaches. His examples included a tablet interface where users look at elements and tap anywhere to interact, a stylus interface where gaze changes interaction modes, and a VR environment using eye-tracking with pinch gestures. He emphasized that effective interfaces should minimize learning effort and be discoverable, and expressed his vision that phones might someday incorporate eye-tracking for more efficient interactions.
Fabien Bénétou shared insights about his participation in a symposium on anarchy and complex systems, noting how participants organized ideas spatially using paper sheets and post-it notes, connecting this to XR knowledge spaces. He referenced Bret Victor’s “Dynamic Land” project as an interesting alternative approach that eliminates devices from collaboration, allowing people to interact directly with content and each other. Bénétou discussed tools for interoperability between devices and shared links to projects like “offline-octopus” and KDE Connect that facilitate device communication, noting that these could potentially work with remarkable tablets since they run Linux.
Brandel Zachernuk raised technical questions about the comfort of working at perceptual distances in modern head-mounted displays, noting that virtual image distances are typically set to 1-1.5m while arms reach about 70cm. He suggested that this technical limitation creates a bias against arm’s-length interaction in VR and proposed the idea of hacking an HMD to have a closer virtual image distance to better study close-range interactions. He also discussed challenges with facial expression tracking as input, noting issues with false positives and the fidelity of signals, while acknowledging that such interfaces are used in accessibility contexts.
Jonathan Finn referenced Wittgenstein’s Philosophical Investigations regarding touching things with a stick, noting how we feel at the end of the stick rather than at our fingertips. He suggested this has implications for how we think about virtual interactions, challenging the prejudice that we need to feel things at our fingertips. Finn also mentioned Austin’s book “How to do things with words” in relation to Dene’s point about voice control and speaking things into existence.
Peter Wasilko asked about virtual hands that could project beyond arm’s length, inquiring if research had been done on non-linear mapping of hand movements to reach distant objects. He also asked whether anyone had experimented with eyebrow gestures or pursed lips as additional control modalities.
Mark Anderson commented on trackpad usage, suggesting that touch typists might prefer trackpads due to keyboard proximity, while his own experience favored mice for fine-grained control. He also mentioned Engelbart’s “chording” interface with five keys, noting how pointer input (eye or hand) plus modifier input could operate without a QWERTY keyboard.
Rob Swigart contributed a reference to Gregory Bateson’s idea that a blind person’s sense of self extends to the end of their walking stick because it provides environmental information similar to built-in senses.
AI: Topics Discussed
WebXR
Fabien Bénétou mentioned WebXR briefly in the context of teleportation mechanics, stating “in WebXR you ‘just’ have to teleport without putting your virtual hands on a rig, and voila, they move but your body is away from them.” He cautioned that this approach might not be ideal for XR newcomers or precise work without additional adjustments like temporary vision teleporting or AR HUD overlays for hands.
Gestures
Gestures were a central topic throughout Ken Pfeuffer’s presentation. He demonstrated various gesture interactions including:
- Gaze+tap: Looking at an element on a tablet and tapping anywhere to interact with it
- Gaze+pinch: Looking at objects in VR and using pinch gestures to select and manipulate them
- Two-handed pinching: Using both hands for zooming and manipulation in VR
- Stylus interactions combined with gaze
- Asynchronous bimanual interaction: Using dominant and non-dominant hands for different tasks
Peter Wasilko asked about using facial gestures like eyebrow movements or pursed lips as additional control modalities. Brandel Zachernuk responded that while possible (he had made a game controlled by eyebrows), such interfaces face challenges with false positives and signal fidelity, though they are used in accessibility contexts.
Frode Hegland suggested using secondary hand gestures as modifiers, such as making a fist with one hand to change what a pinch gesture does with the other hand.
Input Device Evolution
Ken Pfeuffer presented on the evolution of input devices from keyboards to mice to touchscreens, categorizing them as either direct (touchscreen) or indirect (mouse) input. He discussed how eye tracking has consistently lagged behind other input technologies, with researchers typically trying to adapt it to existing paradigms rather than developing entirely new interaction models.
Embodiment and Tool Extension
Several participants discussed how tools become extensions of our bodies. Jonathan Finn referenced Wittgenstein’s description of feeling at the end of a stick rather than at the fingertips. Rob Swigart mentioned Gregory Bateson’s observation that a blind person’s sense of self extends to the end of their cane. Frode Hegland described how a rifle becomes an extension of self with training, and how driving a car extends one’s perception to include the vehicle.
Direct vs. Indirect Interaction
The distinction between direct and indirect interaction was a major theme. Ken Pfeuffer categorized input devices along this spectrum, with mice being indirect (input happens in a different location than output) and touchscreens being direct (input and output surfaces merged). His research attempts to bridge these approaches by using eye tracking to enable direct-like interaction at a distance.
Symposium Planning
Before the presentation, the group discussed plans for their annual symposium. Originally planned for Seattle, it’s now being relocated to either Egypt or London due to travel concerns into the US. If held in Egypt, it would include events at the Grand Museum in Giza and the Modern Library of Alexandria, with a possible Nile cruise. The symposium theme will be “a multitude of perspectives” focusing on text being readable in many ways, tentatively scheduled for the Monday before Thanksgiving 2025.
AI: Concepts Introduced
Direct vs. Indirect Input (Ken Pfeuffer)
Ken defined direct input devices as those where input and output surfaces merge together (like touchscreens), while indirect input devices have input separated from output (like mouse). He explained how direct input affords immediate interaction but only works in close range, while indirect input allows interaction over distance but requires moving a cursor before performing the main action.
Virtual Hand vs. Pointer Metaphors (Ken Pfeuffer)
Ken described two fundamental categories of XR interaction: the virtual hand metaphor (direct manipulation in 3D space) and the pointer metaphor (interaction over distance). He cited Mark Mine’s observation that VR interfaces must balance natural forms of interaction with extensions that aren’t possible in the real world.
Asynchronous Bimanual Interaction (Ken Pfeuffer)
Ken explained this concept as how we naturally use our dominant and non-dominant hands together in complementary ways, such as holding an object with the non-dominant hand while manipulating it with the dominant hand. He referenced Jihad Eve’s work on this topic and explored how it translates to VR environments.
Go-Go Interaction Technique (Ken Pfeuffer, responding to Peter Wasilko)
In response to Peter’s question about extending virtual hands beyond arm’s length, Ken described the “go-go interaction technique” from research in 1995, inspired by Inspector Gadget. This technique uses non-linear mapping of hand movements so small physical movements translate to larger virtual ones, allowing users to reach farther in virtual space.
Tool Embodiment (Various speakers)
Multiple participants discussed how tools become extensions of our perception and sense of self. Jonathan Finn cited Wittgenstein’s example of feeling at the end of a stick rather than at the fingertips. Rob Swigart mentioned Gregory Bateson’s similar concept regarding blind people and canes. Frode Hegland described how rifles and cars become embodied with practice.
Logos Doctrine (Dene Grigar)
Dene explained this as a Western philosophical/theological concept about “speaking the word and the thing happens,” creating through speech. She connected this to voice control interfaces and the idea of speaking things into existence, referencing the biblical concept “the word was God, and God is the word.”
AI: People Mentioned
Richard Balch, mentioned by Ken Pfeuffer as an MIT researcher who created early gaze input systems in the 1980s with his “put that there” work.
Mark Mine, mentioned by Ken Pfeuffer for his report on VR interaction stating that interfaces must balance natural interaction with extensions not possible in the real world.
Steve Jobs, mentioned by Frode Hegland for introducing the iPhone without a stylus, contrasting with the discussion about stylus utility.
Gregory Bateson, mentioned by Rob Swigart for his observation that a blind person’s sense of self extends to the end of their walking stick.
Jihad Eve (or Yves), mentioned by Ken Pfeuffer for work on asynchronous bimanual interaction.
Ken Hinkley and Bill Buxton, mentioned by Ken Pfeuffer as colleagues he worked with at Microsoft Research.
Doug Rushkoff, mentioned by Dene Grigar as someone she’d like to invite to the symposium, noting his early work on VR discussions.
Evan (surname not given), mentioned by Ken Pfeuffer as researcher who worked on the go-go interaction technique around 1995.
Mark Weiser, mentioned by Ken Pfeuffer for his prediction about ubiquitous computing and mobile devices.
Doug Engelbart, mentioned by Mark Anderson for his “chording” interface with five keys.
Bret Victor, mentioned by Fabien Bénétou regarding the “Dynamic land” project that rejects XR approaches.
Wittgenstein, mentioned by Jonathan Finn regarding his philosophical observations about touching with a stick.
Austin, mentioned by Jonathan Finn for his book “How to do things with words” about performative speech.
Baron, mentioned by Dene Grigar as the author of “A Better Pencil,” a book about writing technologies.
Ismail, mentioned by Frode Hegland as the founder of the Modern Library of Alexandria who is helping organize the symposium in Egypt.
AI: Product or Company Names Mentioned
Microsoft Research, mentioned by Ken Pfeuffer as a place where he worked, alongside Ken Hinkley and Bill Buxton.
Google, mentioned by Ken Pfeuffer as a place where he had an internship, and Dene Grigar asked if he was considered a “Googler.” He mentioned enjoying the ramen restaurant there.
Apple Vision Pro, mentioned by Ken Pfeuffer as a commercial device implementing eye-gaze and pinch interactions similar to his research. Also referenced later in discussions about interaction models.
Microsoft HoloLens, mentioned by Ken Pfeuffer as one of several VR/AR devices demonstrating different input methods.
Meta Quest, mentioned by Ken Pfeuffer as one of several VR/AR devices demonstrating different input methods. Also mentioned by Dene Grigar when comparing the disconnection she feels using Quest controllers versus the direct manipulation in Apple Vision Pro.
HTC Vive, mentioned by Ken Pfeuffer as one of several VR/AR devices demonstrating different input methods. Also mentioned as the hardware he used in his demonstrations with attached eye tracking.
iPhone and iPad, mentioned by Dene Grigar as turning points in technology due to their touch interfaces creating direct connection with digital objects.
Photoshop, mentioned by Frode Hegland as an application he was “brought up on” that trained him to use keyboard shortcuts efficiently.
Pupil eye tracker, mentioned by Ken Pfeuffer as equipment used in his research.
Leap Motion tracker, mentioned by Ken Pfeuffer as equipment used in his research.
Meta Quest Pro, mentioned by Ken Pfeuffer as a current device that combines the sensors he previously had to assemble separately.
Unity, mentioned by Ken Pfeuffer as the software used to create his demonstrations.
Vive Pro, briefly mentioned by Ken Pfeuffer when describing his equipment setup.
Dynamic Land, mentioned by Fabien Bénétou as Bret Victor’s project that takes an alternative approach to collaboration without devices.
Remarkable tablets, mentioned by Frode Hegland as a device he’d like to integrate with his Mac ecosystem.
BBC, mentioned by Dene Grigar regarding an article featuring her lab.
Beat Saber, mentioned by Dene Grigar as a VR application she uses.
VR Fit, mentioned by Dene Grigar as a VR application she uses.
KDE Connect, mentioned by Fabien Bénétou as a popular alternative for device interoperability.
AI: Other
The meeting began with informal conversation and an update about the annual Future of Text symposium. Original plans to hold it in Seattle had to be changed due to travel concerns, with Egypt and London being considered as alternative locations. The symposium theme will be “a multitude of perspectives” with a focus on text being readable in many ways, tentatively scheduled for the Monday before Thanksgiving 2025.
There was an interesting thread throughout the conversation about how technological innovations often appear earlier than their widespread adoption, waiting for the right context or “zeitgeist” to become mainstream. This applies to many of the eye-tracking and gesture interactions Ken Pfeuffer has researched, some of which are now appearing in commercial products like Apple Vision Pro.
The discussion frequently touched on philosophical aspects of human-computer interaction, including embodiment, tool extension, and how different interaction paradigms affect our relationship with technology. The group demonstrated a multidisciplinary approach, drawing connections to philosophy, linguistics, art, and cultural practices.
Chat Log URLs
https://clea.research.vub.be/complex-anarchism-symposium-19-23-may-2025-in-brussels
https://www.youtube.com/watch?v=WhA8n4IXeoY https://www.billbuxton.com/input04.Taxonomies.pdf https://github.com/tridactyl/tridactyl?tab=readme-ov-file#replace-firefoxs-default-control-mechanism-with-one-modelled-on-the-one-true-editor-vim https://folk.computer
https://www.youtube.com/watch?v=PixPSNRDNMU https://bsky.app/profile/did:plc:5sfd4v4dnrtd5ogec2idv7sl/post/3lpcbfiwdpk2r
https://www.bbc.com/future/article/20250516-the-people-stuck-using-ancient-windows-computers
https://git.benetou.fr/utopiah/offline-octopus https://kdeconnect.kde.org
Chat Log Summary
The chat log complemented the spoken discussion with additional references, links, and side conversations. Fabien Bénétou shared information about a symposium on anarchy and complex systems he was attending, noting how participants organized ideas spatially. Several participants contributed links to relevant resources, including Buxton’s taxonomy of input devices, accessibility tools for browsers, and projects for device interoperability.
The chat featured discussions about embodiment (Jonathan Finn referencing Wittgenstein, Rob Swigart mentioning Gregory Bateson), ergonomics (Fabien lamenting “terrible rectangle” keyboards), and alternative input methods (Mark Anderson noting Engelbart’s chording interface). Several participants shared personal preferences about input devices – Frode embracing trackpads, Mark preferring mice for fine control.
Brandel Zachernuk raised technical questions about perceptual distances in VR headsets, suggesting research using modified hardware. Fabien discussed alternative approaches like Bret Victor’s “Dynamic Land” and shared links to projects enabling device interoperability. Dene Grigar shared a BBC article featuring her lab that paradoxically showed her discussing Windows computers.
The chat provided a richer context to the verbal discussion, with participants expanding on points, sharing resources, and adding humor and personal reflections that complemented the formal presentation.