ASO Skills
Focus: aso, app store optimization and app marketing. Shares mcp and open source with Kitaru. Same 4.7 rating.
WebsiteFreeWeb, Cross-Platform
4.7Visit
Website
Kitaru
Replay-based evals for AI agents — traces you can run, not just read.
Rating
4.7 / 5
Pricing
Free
Difficulty
Intermediate
Works on
Web, macOS, Windows
Built with love by Shushant Lakhyani. Building bestproductivityresources.com
What is Kitaru?
Kitaru turns production agent traces into reproducible regression tests. Wrap your agent once, or import existing traces from Langfuse, LangSmith, Braintrust, Logfire or Arize Phoenix, then replay any session against your next model, prompt or code change. Replays re-execute your real code with recorded tool-call responses, so an unchanged run gives a faithful baseline and a forked run shows exactly what changed. Human reviewers can pin judgments to exact trace locations, calibrate evaluators, freeze cohorts and run experiments before shipping. It is open source, self-hosted, ships with a Python SDK and TypeScript client, an MCP server, and coding-agent skills for Claude Code and Cursor.
Third-party open-source projects. We are not affiliated with their authors; repositories are provided as-is by their maintainers. Review each project's license, code and security before installing. See our Disclaimer.
Pros: what Kitaru does well
- Replays real agent code against recorded tool responses for faithful baselines
- Imports traces from most popular observability platforms
- Open source and self-hosted with Python SDK, TypeScript client and MCP server
Cons: where it falls short
- Self-hosting requires infrastructure setup and a worker next to your agent's dependencies
- Most valuable once you already have production traces to replay
Who Kitaru is for
AI engineering teams and agent builders who want regression testing and evals tied to real production traces.
Who can skip it
Teams without existing agent traces or those that only need simple prompt-unit tests.
Kitaru alternatives
Not sold on Kitaru? These do a similar job, and each card says what it has in common.
Startup Skill
Focus: claude code, claude skill and startup. Shares ai agent and open source with Kitaru. Same 4.7 rating.
WebsiteFreeWeb, Cross-Platform
4.7Visit
Transformers
Focus: transformers, hugging face and ml. Shares open source and python with Kitaru. A framework, not a website. Rated 4.9 (Kitaru: 4.7).
FrameworkFreemacOS, Windows
4.9Visit
FFmpeg Skill
Focus: ffmpeg, video editing and ai agent skill. Shares open source and mcp with Kitaru. An automation tool, not a website. Same 4.7 rating.
Automation ToolFreemacOS, Windows
4.7Visit
hol-guard
Focus: ai agents, security and prompt injection. Shares mcp and open source with Kitaru. An AI tool, not a website. Same 4.7 rating.
AI ToolFreemacOS, Windows
4.7Visit
OpenSquilla
Focus: token efficiency, cost optimization and llm. Shares ai agent and open source with Kitaru. An AI tool, not a website. Rated 4.6 (Kitaru: 4.7).
AI ToolFreemacOS, Windows
4.6Visit
Free alternatives to Kitaru
Want to spend nothing? These free picks cover similar ground.
Agent LSP
Focus: lsp, ai agents and language server. Shares mcp and open source with Kitaru. A framework, not a website. Rated 4.5 (Kitaru: 4.7).
FrameworkFreemacOS, Windows
4.5Visit
toolport
Focus: gateway, tools and local first. Shares mcp and open source with Kitaru. An AI tool, not a website. Rated 4.5 (Kitaru: 4.7).
AI ToolFreemacOS, Windows
4.5Visit
AgentPipe
Focus: agents, task execution and multithreading. Shares python and open source with Kitaru. A framework, not a website. Rated 4.4 (Kitaru: 4.7).
FrameworkFreeCross-Platform
4.4Visit
arifOS
Focus: ai governance, agents and safety. Shares mcp and open source with Kitaru. A framework, not a website. Rated 4.4 (Kitaru: 4.7).
FrameworkFreeCross-Platform
4.4Visit
Guardana
Focus: ai security, verification and agents. Shares mcp and open source with Kitaru. A framework, not a website. Rated 4.4 (Kitaru: 4.7).
FrameworkFreemacOS, Windows
4.4Visit
Kiln
Focus: 3d printing, ai agents and hardware. Shares mcp and open source with Kitaru. An AI tool, not a website. Rated 4.4 (Kitaru: 4.7).
AI ToolFreeCross-Platform
4.4Visit
Are you the maker of Kitaru?Get a free Featured badge for your site, a share image and ready-to-paste posts. No sign-up.
New · updated daily
What people are actually doing with AI
Real use cases shared on Reddit, Hacker News, X, GitHub and blogs, summarised in plain words with a link to every original post.
Free Notion database
1,100+ ways people are making money online
Real online businesses, creators and projects, broken down by how they make money: apps, online stores, AI tools and more. Free on Gumroad.
Get the free databaseFree Notion database
399+ ways to make money online
Hundreds of online income methods in one place, so you can compare options and pick one to try. Free on Gumroad.
Get the free listFor research and ideas only. Results vary and nothing here is a promise of income. See our Disclaimer.
Weekly newsletter
Everything new, every Monday.
One short email every Monday with everything shipped that week: new free tools and websites, new guides, new free browser tools and what's new in my free Gumroad libraries, plus how the site is growing. Free, and you can unsubscribe anytime.
Join the newsletterYou sign up on Gumroad, which stores your email to send the newsletter. See our Privacy Policy.