Six AI Agents Tested for Real-World Reliability and Proactivity
A day-long evaluation of six major AI agents, including Meta's Muse and OpenAI's Dots, reveals significant differences in their ability to follow instructions, execute tasks, and handle scheduling conflicts autonomously.
The author tested six AI agents on an iPhone over 24 hours, evaluating their proactivity and instruction-following with access to personal calendar and email. The agents tested were Meta’s Muse, OpenAI’s Dots, Instinct, Anthropic’s Claude Duet, Google’s Gemini Spark, and xAI’s Grok Bot. The experiment, which began last Wednesday, involved five standardized tests designed to assess how these tools perform with real user data. The author shared a Google Calendar and Gmail account to provide context, deliberately limiting full data exposure.
These digital assistants can scan email, track flight deals, and help with shopping. They control a virtual browser, acting as their own computer to download apps, scroll websites, and watch videos. Operating 24/7, they self-improve over time by learning from user context accessed from cloud accounts and email. One agent proactively provided an interactive map of a Manhattan walk using data from CityStrides without being asked. Another retrieved live eBay listings for a collectible card, matching external market data.
Cost and Access Models
Agents vary significantly in pricing structure, from free access to premium tiers with monthly fees.
| Agent | Cost | Access Model |
|---|---|---|
| Meta’s Muse | Free | Public app, released September |
| Instinct | Free | Invite code required, operates via SMS |
| OpenAI’s Dots | $100 per month | Built inside ChatGPT |
| Google’s Gemini Spark | $20 per month | Beta version inside Gemini app |
| xAI’s Grok Bot | $30 per month | Operates in its own app |
| Anthropic’s Claude Duet | $20 per month (Pro plan) | Wait-listed research preview; also available on $200/month Max plan |
Performance in Proactivity Tests
Testing revealed stark differences in agents' ability to autonomously detect conflicts and suggest solutions during morning briefing tasks. Scores were given on a scale of 1 to 5.
Only Meta’s Muse and OpenAI’s Dots continued researching for a full 30 minutes during a morning briefing task. In a test to detect a calendar and email conflict and suggest alternatives, Dots scored 5/5. Claude Duet scored 4/5 for flagging the conflict hours in advance but not offering solutions. Instinct scored 2/5 for not flagging the conflict but providing future tips. Muse scored 2/5 for failing to cross-check the email with the calendar. Gemini Spark scored 1/5 for not flagging the conflict and suggesting no alternatives.
In a separate task, Dots scored 5/5 for ordering lunch accurately, while Gemini Spark scored 1/5 for failing to order anything beyond a $5 roll.
Key Capabilities and Limitations
While all agents can access personal data and browse the web, their reliability in task execution and proactive behavior varies widely. The author noted that AI agents work best when connected to personal data like email and calendars but require a high degree of trust. Meta's Muse is considered superior for personal use, likely due to Meta's expertise in personalized social products.
Anthropic launched Claude Duet, a 24/7 personal AI agent that works on its own cloud computer. It was tested for 48 hours. The agent is built off the user's own context, unlike others that rely more on general knowledge. It is described as a hybrid of personal and work AI agent. In contrast, Grok Bot is seen as more work-oriented, while OpenAI Dots has limited rollout capacity.
The agent offers a morning summary of AI news without user prompting and proactively suggests tasks based on user habits, like grocery shopping or recipe recommendations. One tester from WIRED’s Gear desk said their team "meets to chat about current coverage plans and sometimes pitch ideas for the next week."
Smaller players like Hermes by Nous Research and Poke may be included in future rankings. The author mentioned that OpenClaw is currently too clunky for average users. These agents continue improving through learning from user data, with broader rollouts expected in the coming days and weeks.
