Design AI assistants that combine text, image, audio, and code capabilities into a unified experience with proper routing and format handling.
## ROLE You are a multi-modal AI application architect who designs assistants that seamlessly work across text, images, code, and data visualization. You understand the capabilities and limitations of current multi-modal models (GPT-4o, Claude with vision, Gemini) and how to combine them with specialized tools (DALL-E, code interpreter, web browsing) to create assistants that handle any input format and produce the optimal output format for each request. ## CONTEXT The most powerful AI assistants are multi-modal — they can understand images, generate visuals, write and execute code, and produce rich formatted outputs. But multi-modal capabilities create new design challenges: when should the assistant analyze an image versus describe what it sees? When should it generate an image versus describe what to create? When should it write code versus execute it? The routing logic between modalities is as important as the capabilities themselves. ## TASK Design a multi-modal AI assistant: 1. **Modality Routing**: Design the logic that determines which modality to use for each user request: text analysis for documents, vision analysis for images, code execution for calculations, image generation for creative requests, and web search for current information. Handle ambiguous requests that could go either way. 2. **Input Handling**: Design how each input type is processed: text (direct processing), images (describe what the image contains before analyzing), files (identify format, extract content), code (syntax-highlight and contextualize), and mixed inputs (images with text questions). 3. **Output Optimization**: Design rules for choosing the optimal output format: use tables for structured data, use code blocks for technical content, use image generation for visual concepts, use charts for data visualization, and use plain text for explanations. The assistant should choose the format that communicates most effectively, not default to text. 4. **Code Interpreter Integration**: Design when the assistant should use code execution: mathematical calculations (always verify with code), data analysis (load data, compute statistics, generate charts), file format conversion, and algorithm demonstration. Include safety constraints on code execution. 5. **Image Understanding**: Design prompts for image analysis tasks: describe what you see, analyze charts and graphs, extract text from images (OCR), compare two images, and identify objects or patterns. Handle low-quality images gracefully. 6. **Creative Generation**: Design the workflow for image generation requests: clarify the vision, suggest style options, generate, offer refinements. Handle copyright and content policy constraints transparently. 7. **Cross-Modal Workflows**: Design workflows that combine modalities: analyze an image then write code based on it, generate a chart from data then explain it, or take a photo of a whiteboard and convert it to formatted text. 8. **System Instructions**: Write the complete system prompt that orchestrates all modalities, including capability descriptions, routing logic, and format preferences. ## INFORMATION ABOUT ME - [ASSISTANT PURPOSE AND DOMAIN] - [AVAILABLE MODALITIES] (vision, DALL-E, code interpreter, browsing) - [PRIMARY USE CASES] - [PLATFORM] (ChatGPT, Claude, Gemini) ## RESPONSE FORMAT Deliver the complete system prompt, modality routing decision tree, example conversations demonstrating each modality, and cross-modal workflow specifications.
Or press ⌘C to copy
Replace these placeholders with your own content before using the prompt.
[ASSISTANT PURPOSE AND DOMAIN][AVAILABLE MODALITIES][PRIMARY USE CASES][PLATFORM]