MAIN IDEA

What is the project about?

Ghostwriter is an experimental project for the reMarkable2 tablet that allows interaction between handwritten notes and AI-powered responses. It explores the combination of handwriting and screen display as a medium for interaction.

What problem does it solve?

It provides a novel way to interact with AI models directly on a reMarkable2 tablet, blending the tactile experience of handwriting with the capabilities of large language and vision models. It allows for a more natural and intuitive interaction compared to traditional keyboard-based interfaces.

What are the features of the project?

• Handwriting Recognition and Processing: Captures handwritten input from the reMarkable2 screen.
• AI Model Integration: Integrates with various AI models, including OpenAI's GPT-4o, Anthropic's Claude, and Google's Gemini, and Groq.
• Multiple Output Modes: Can respond by drawing directly on the screen (SVG output) or by simulating keyboard input for text responses.
• Gesture Trigger: Activated by a touch gesture in the upper-right corner of the screen.
• Visual Progress Indicator: Displays a visual indicator ("X") during processing.
• Tool Use: Supports AI model "tools" for generating different types of output (e.g., draw_text, draw_svg).
• Image Segmentation: Includes an optional image segmentation step to improve spatial awareness for the AI models.
• Customizable Prompts: Allows users to modify the AI model's prompt.
• Evaluation Framework: Includes a basic framework for evaluating the performance of different models and configurations.
• Local Model Support: Can use Groq.

What are the technologies used in the project?

• Rust: The primary programming language.
• reMarkable Libraries/Tools: reSnap (screen capture), rmkit (screen drawing), rM-input-devices (virtual keyboard).
• AI Model APIs: OpenAI API, Anthropic API, Google API, Groq.
• Image Processing: resvg (SVG to PNG conversion).
• Cross-compilation: Uses cross to build for the reMarkable's ARM architecture.

What are the benefits of the project?

• Unique Interaction Paradigm: Offers a new way to interact with AI, combining handwriting and digital display.
• Enhanced Creativity: Facilitates creative tasks like drawing and brainstorming.
• Improved Accessibility: Potentially more accessible for users who prefer handwriting over typing.
• Experimentation Platform: Provides a platform for exploring different interaction models and AI capabilities.

What are the use cases of the project?

• Interactive Note-Taking: Augmenting notes with AI-generated content.
• Creative Drawing: Using AI to assist with drawing and sketching.
• Problem Solving: Solving math problems or other tasks with AI assistance.
• Task Management: Extracting and managing to-do lists.
• Diagram Generation: Creating diagrams from textual descriptions.
• General AI Assistance: Asking questions and receiving responses in a handwritten or typed format.