Building JARVIS in 3 Ways

Tutorial: Building a No-Code AI Personal Assistant for Administrative Tasks

This tutorial will guide you step-by-step in creating a no-code AI personal assistant using platforms like n8n, Twilio, and Google Workspace. The assistant will handle tasks like retrieving emails, scheduling meetings, managing tasks in Notion, and updating calendars—all through simple text or voice commands.

Step 1: Setting Up Your Workflow

1. Choose a No-Code Platform:
• Use n8n, Make.com, or Zapier to build and automate your workflow.
• For this tutorial, we’re using n8n.
2. Add a Trigger:
• Integrate Twilio to receive SMS messages.
• Purchase a number from Twilio (costs around $0.30/month).
• Configure the Twilio Trigger Node in n8n:
• Set it to monitor new incoming SMS messages.
3. Connect AI for Processing:
• Add an OpenAI Node:
• Prompt it to analyze the incoming message and determine actions based on the text.
• Example actions: “Get Emails,” “Send Email,” “Get Calendar,” “Create Task,” or “Create Meeting.”

Step 2: Handling Emails

1. Retrieve Emails:
• Filter the actions from OpenAI to detect if the user requested email retrieval (e.g., "Get Emails").
• Use Google Gmail Node:
• Configure it to retrieve messages based on specific dates or conditions.
• Example filter:
• Label: Inbox
• Unread: True
• Date: Provided by OpenAI (e.g., “yesterday”).
2. Send Emails:
• Add another filter for "Send Email."
• Use Google Sheets Node to look up email addresses in your database.
• Use Gmail Draft Node:
• Draft the email using a prompt that formats the message body and subject.
• Optionally, send the email directly or prompt for confirmation.

Step 3: Managing Your Calendar

1. Retrieve Calendar Events:
• Add a filter for "Get Calendar."
• Use Google Calendar Node to retrieve events based on date ranges provided by OpenAI.
2. Create Meetings:
• Add a filter for "Create Meeting."
• Use Google Sheets to retrieve attendees’ email addresses from your CRM.
• Use Google Calendar Node to create an event:
• Include start and end times (calculate end time by adding 1 hour to the start time).
• Add attendees and include a meeting description.

Step 4: Creating and Managing Tasks

1. Create Tasks in Notion:
• Add a filter for "Create Task."
• Use Notion API Integration:
• Connect it to your project management database.
• Set task details like name, execution date, and priority.
• Example:
• Task: “Buy a whiteboard.”
• Execution Date: Extracted from the command (e.g., “tomorrow”).

Step 5: Building Interactivity

1. Voice Commands with Siri:
• Configure Siri to text your Twilio number for seamless voice activation.
• Example commands:
• “Hey Siri, text my AI agent: Can you get my emails from yesterday?”
• “Hey Siri, schedule a meeting with John for next Tuesday at 11 a.m.”
2. SMS Responses:
• Use a Twilio Node to send SMS responses:
• Example: “Here’s your calendar for the next two weeks.”

Step 6: Improving Efficiency

1. Filters for Actions:
• Use If Nodes to route tasks based on specific actions (e.g., "Get Calendar" vs. "Send Email").
• Example: If the message contains “Get Emails,” route it to the Gmail retrieval workflow.
2. Optimized Prompts:
• Create detailed prompts for OpenAI to ensure accurate interpretation.
• Example prompt for retrieving emails:
• “If the message asks for emails from yesterday, retrieve only those emails.”
3. Standardize Dates:
• Use the Date & Time Node to calculate and format dates (e.g., “yesterday,” “next Tuesday”).

Step 7: Testing and Debugging

1. Run Tests:
• Test each workflow component (e.g., retrieving emails, sending emails, creating tasks) separately.
• Debug any issues by checking logs and ensuring nodes are correctly linked.
2. Refine Outputs:
• Ensure outputs are concise and easy to understand.
• Example email draft:
• Subject: "Meeting Reminder"
• Body: “Hey John, this is a reminder about our meeting tomorrow at 11 a.m. Best, [Your Name]”

Step 8: Expanding Functionality

1. Add CRM Integration:
• Connect your CRM (e.g., Airtable, Google Sheets) to manage client details dynamically.
2. Enable Cross-Platform Access:
• Integrate other messaging platforms like WhatsApp or Messenger for broader accessibility.
3. Custom Automation:
• Use AI to summarize emails, prioritize tasks, or even auto-generate project updates.

Tutorial: Building Smithi, Your AI Assistant

This step-by-step tutorial will guide you in creating Smithi, an AI assistant with capabilities beyond typical assistants like Siri or Alexa. Smithi is tailored for technical tasks, robotics, and engineering support, making it perfect for creators and tinkerers.

Step 1: Choosing the AI Foundation

1. Select a Language Model:
• Use OpenAI's ChatGPT as the core for Smithi. It's highly customizable and easy to integrate into your software or systems.
2. Set Up the Integration:
• Follow OpenAI's API documentation to integrate the model. Within hours, you can get the assistant replying in the tone and personality you want.
3. Create a Prompt for Personality:
• Tailor the assistant’s behavior:
• E.g., “You are Smithi, a snarky, intelligent assistant for Hacksmith Industries, with a dry British wit. Provide concise, helpful answers with a touch of humor.”
• Test and refine the prompt to ensure it captures the tone and behavior you want.

Step 2: Adding Voice Interaction

1. Speech-to-Text Conversion:
• Use tools like Google Speech-to-Text or Whisper for converting verbal commands into text that Smithi can process.
2. Wake Word Detection:
• Implement wake word detection similar to “Hey Siri” or “Alexa.”
• Tools like Porcupine by Picovoice or open-source solutions can enable this feature.
3. Text-to-Speech for Voice Responses:
• Integrate with 11 Labs for natural-sounding voice generation.
• Use API calls to generate voice outputs based on Smithi’s responses. Customize the voice to sound British or any other style you prefer.

Step 3: Enabling Task-Specific Features

1. General Knowledge & Engineering Support:
• Train Smithi to answer specific technical questions by priming it with documentation, FAQs, or relevant knowledge bases.
• Example: “What fuel did we use in the Mandalorian flamethrower?” → Answer: “Propane and cornstarch.”
2. Home Automation and Control:
• Create commands to control lights, devices, or machinery in your workshop.
• Example:
• Command: "Smithi, turn on the lights."
• Response: API call to your smart light system.
3. Advanced Robotics Integration:
• Use Smithi to command robots in your workspace. Program waypoints or tasks into your robots and connect them to Smithi.
• Example:
• “Smithi, send the robot dog to the office.”
• Smithi calculates the path and instructs the robot via API.
4. Unique Features for Your Projects:
• Use Smithi for specialized queries, like generating engineering blueprints or finding project-specific data.

Step 4: Adding Fun and Interactivity

1. Interactive Jokes and Snark:
• Program Smithi to include humor and personality in its responses, like JARVIS from Iron Man.
• Example:
• Question: "How much wood could a woodchuck chuck?"
• Response: “Captain, a woodchuck could chuck as much wood as a woodchuck could chuck, if a woodchuck could chuck wood.”
2. Custom Visualizer:
• Build a visualizer for Smithi, such as swirling arcs that react to Smithi’s states (listening, thinking, or speaking). Tools like Processing or p5.js can help.

Step 5: Expanding with Cool Features

1. Pizza Ordering System:
• Leverage a pizza chain’s public API to automate ordering:
• Steps:
• Register your address.
• Download the menu via the API.
• Use Smithi to create and place an order.
• Example:
• “Smithi, order me a large pepperoni pizza.”
• Response: “Pizza ordered for $28.23, expected in 25-35 minutes.”
2. Image Generation:
• Connect to an image-generation tool (e.g., OpenAI’s DALL-E).
• Example:
• Command: "Smithi, generate an image of a cute cat."
• Response: An AI-generated cat image.
3. Multi-Lingual Support:
• Enable Smithi to respond in different languages for fun or practical uses.
• Example:
• “Smithi, reply in French.”
• Response: “Oui, bien sûr, monsieur.”

Step 6: Troubleshooting and Iteration

1. Improving Question Handling:
• Use fine-tuned models to improve Smithi’s ability to handle specific queries (e.g., Hacksmith project details).
2. Streamlining Responses:
• Refine prompts to prevent Smithi from “forgetting” capabilities or misinterpreting commands.
3. Testing and Debugging:
• Regularly test Smithi’s features to ensure smooth functionality and address bugs (e.g., ordering unintended pizzas due to misunderstood commands).

Step 7: Bringing It All Together

1. Integration into a Community:
• Deploy Smithi in your Discord or other platforms for real-time interaction with users.
2. Practical Applications:
• Use Smithi for assisting in workshops, robotics competitions, or even helping team members troubleshoot problems.

Tutorial: Building Your JARVIS AI on n8n

This step-by-step guide will show you how to create a JARVIS-inspired AI assistant using n8n, including voice interactions, task management, and integrations with various tools like Airtable, Google Calendar, and 11 Labs for text-to-speech generation.

Step 1: Setting Up Telegram as a Trigger

1. Create a Telegram Bot:
• Use Telegram's BotFather to create your bot and get the API token.
2. Add a Telegram Trigger Node:
• Add a Telegram Trigger node in n8n and configure it to trigger on messages.
3. Add a Switch Node:
• Add a Switch node to separate incoming text messages, voice messages, and errors.
• Set rules to route audio messages to voice processing and text messages to text processing.

Step 2: Processing Audio Messages

1. Get Voice File:
• Use the Telegram API to retrieve the voice file ID.
• Download the voice file using the Telegram File node.
2. Transcribe Voice to Text:
• Add an OpenAI Transcribe node or a similar speech-to-text service to convert the voice message into text.
3. Standardize Data:
• Use an Edit Fields node to format all text into JSON format, ensuring consistent input for the AI agent.

Step 3: Integrating the AI Agent

1. Add a Tools Agent Node:
• Create a central AI agent node (e.g., J.A.R.V.I.S Tools Agent) and configure it with:
• A comprehensive prompt for tool selection based on user input.
• Integrations with tools like Airtable, Google Calendar, Gmail, SerpAPI, or custom APIs.
2. Add Memory Support:
• Integrate a Window Buffer Memory to allow JARVIS to maintain context in conversations.

Step 4: Adding Personality

1. Basic LLM Chain:
• Add a secondary LLM node (e.g., Basic LLM Chain) to capture JARVIS's personality.
• Use prompts to generate quick-witted, humorous, or sarcastic responses based on user queries.
• Ensure the personality node does not repeat information already provided by the AI agent but instead adds commentary.

Step 5: Generating Voice Responses

1. Integrate with 11 Labs:
• Use the HTTP Request node to call 11 Labs API for text-to-speech conversion.
• Configure the API call with your 11 Labs API key and voice settings.
• Ensure the text is properly formatted and stringified to avoid errors.
2. Send Voice Messages:
• Use the Telegram Send Audio node to send the generated audio file back to the user.

Step 6: Building Additional Features

1. Project Management with Airtable:
• Add Airtable nodes to manage tasks, retrieve contacts, and store to-do lists.
• Query data dynamically based on user input (e.g., “Get my tasks for tomorrow”).
2. News and Weather Updates:
• Use APIs like SerpAPI for internet search and Hacker News for news updates.
• Integrate weather APIs to respond to queries like “Do I need an umbrella today?”
3. Custom Use Cases:
• Add more tools (e.g., CRM systems, email automation, or calendar integrations) depending on your needs.
• Separate workflows for better modularity if adding many tools.

Step 7: Testing and Debugging

1. Test each route separately:
• Voice messages: Ensure audio files are transcribed correctly and the response is accurate.
• Text messages: Confirm tools are invoked correctly based on queries.
2. Debug errors:
• Use n8n’s logs to troubleshoot issues like API misconfigurations or JSON formatting errors.

Step 8: Deploying and Sharing

1. Export Workflow:
• Export your workflow as a template to share or reuse.
2. Community Support:
• If you’re part of a learning community, share your build for feedback and improvements.

Step 6: Improving Efficiency

1. Filters for Actions:
• Use If Nodes to route tasks based on specific actions (e.g., "Get Calendar" vs. "Send Email").
• Example: If the message contains “Get Emails,” route it to the Gmail retrieval workflow.
2. Optimized Prompts:
• Create detailed prompts for OpenAI to ensure accurate interpretation.
• Example prompt for retrieving emails:
• “If the message asks for emails from yesterday, retrieve only those emails.”
3. Standardize Dates:
• Use the Date & Time Node to calculate and format dates (e.g., “yesterday,” “next Tuesday”).

Step 7: Testing and Debugging

1. Run Tests:
• Test each workflow component (e.g., retrieving emails, sending emails, creating tasks) separately.
• Debug any issues by checking logs and ensuring nodes are correctly linked.
2. Refine Outputs:
• Ensure outputs are concise and easy to understand.
• Example email draft:
• Subject: "Meeting Reminder"
• Body: “Hey John, this is a reminder about our meeting tomorrow at 11 a.m. Best, [Your Name]”