Cognitive Context AI

In order to really get AI to support intellectual augmentation and not simply think on behalf of the user, it will be necessary to work on what extraction and/or analysis the AI should perform when the user looks at a paper to give the them an Overview, particularly when reading something outside of their field.

The next step is to get the AI to consider who the user is; what the user already knows and what the user is interested in learning.

  

AI Overview (for general users)

The approach I am taking in Reader (macOS, iOS & visionOS) is to give the user an ‘AI Overview’ option which will take the paper and run a prompt to extract a very short summary and a list of key terms. This should help the user better understand the paper when they read it themselves.

I have posted the AI Overview Prompt in case you are interested in look at it and perhaps further experiment with it.

 

AI Overview with Cognitive Context (for specific users)

The next step could simply be to allow the user to enter information about who they are, what the prior knowledge is and what they are interested in learning more about., for the AI to take into account when submitting the prompt. I have mocked this up in Reader’s ‘Settings’ and intend to develop this soon:

The result can then be something like this, with the modified prompt sent to ChatGPT (or other LLM provider, including local), customized for the user. Note that these have been tested manually using ChatGPT in Reader:

Interested to learn more about : Brain

User who is knowledgeable about tech, would like to learn about brain.

User Profile provided: I am: A university student majoring in IT. I know a lot about: IT, technology, programming, hypertext, computers and history of technology. I would like to learn more about: biology, human brain, human computer interaction, knowledge work, tools for thought, culture, sociology 

Keywords presented by AI: Cultural Heritage, Semantic Analysis, Manuscript, Digitization, Ontology, Corpus, Visualization, Lemmas, Cognitive Science, Sociology, Knowledge Graph, Clustering (Prompt and Result)

(click image for full size)

Interested to learn more about : Tech

User who is knowledgeable about the brain, would like to learn about technology.

User Profile provided: I am: A university student majoring in biology. I know a lot about: biology, human brain, human computer interaction, knowledge work, tools for thought, culture, sociology. I would like to learn more about:  IT, technology, programming, hypertext, computers and history of technology

Keywords presented by AI: Automated Analysis, Document Classification, Optical Character Recognition (OCR), Handwritten Text Recognition (HTR), Semantic Analysis, Knowledge Graph, IIIF (International Image Interoperability Framework), UMAP (Uniform Manifold Approximation and Projection), TF-IDF (Term Frequency-Inverse Document Frequency), Cartographic Visualization (Prompt and Result)

(click image for full size)

   

XR Extension

This personalization can go much further when the results are not just presented in a rectangle on a relatively small screen, but spatially in an XR environment, where different results can appear in different spaces, based on user preference. For example, an overview of key terms above the document, a full timeline behind you and people referred to on your right etc.

Illustrated very crudely below, with expectations that we can go much further than this. Imagine this done spatially with more meaning than simply a few lists in a few places.

(click image for full size)

4 comments

Leave a comment

Your email address will not be published. Required fields are marked *