Senior Software Engineer (Agentic Intelligence)

* AI agents at Honeycomb investigate, reason, and act on real observability data. They live in Canvas: the agentic workspace where engineers go to understand their systems * The Agentic Intelligence team has shipped Canvas, the Honeycomb MCP server, and the Canvas Agent and Canvas Skills surfaces * Honeycomb’s data store is fast and accepts high cardinality data; that’s what makes agents built on top of it different from anything built on a conventional observability backend * This role is about taking advantage of that building agents that can do things no other observability product can do because the underlying data makes it possible * Some of this work will start as a prototype. The expectation is that the code makes it through the full arc, from the rough first version through to something that holds up in production * Design and deliver production-grade agents. Build agents that investigate, reason, and act on live observability data inside Canvas. These agents must be trustworthy to engineers in high pressure situations, including mid-incident. Take one from rough first version to something that holds up under production traffic * Own the agent work; support the whole product. Scope, build, ship, and maintain the agents including the evals that tell you whether they got better or are just different. This role is agent-focused and also includes some fullstack development * Build agents only Honeycomb can build. Use a data store that returns high-cardinality queries in seconds to reason over signal a conventional backend can’t serve at this fidelity correlating across services, drilling into a single trace, comparing before and after a deploy * Extend the surface, and decide what’s next. Ship new capability into Canvas, the MCP server, and Canvas Skills memory, spatial awareness, a faster Bedrock loop and make the case for what comes after with working code. Distinguish hype from signal in a field with plenty of both * Define what “good” means for agents here. Set the bar: measurable against real evals, maintainable, and honest about their limits * Example projects: * Multiple agents collaborating on one shared Canvas investigation each claiming a hypothesis, publishing findings, and narrowing the search space for the others so it resolves faster (blog) * Auto-investigation the moment an SLO burn alert fires the agent forms hypotheses and prepares visualizations before a human looks, cutting mean-time-to-insight for on-call (o11ycon 2026) * Skills that encode a team’s domain expertise e.g. Kubernetes thresholds so agents and human colleagues can lean on them (o11ycon 2026) Benefits -------- * 100% covered medical, dental, vision, HSA, FSA, disability, and more * Unlimited PTO, synchronous company break, paid sabbatical, and expressed minimum for time to recharge * 401K retirement plans and fully paid life insurance * Up to 16 weeks paid leave regardless of path to parenthood; flexible hours for returning parents and caregivers * $10K to cover gaps in insurance, in support of adoption, surrogacy, preservation, and family-planning costs * $600 per year to spend on the perks you care for most * A stake in our success * Generous new hire equity, and refresh grants * Annual $1,500 development stipend; funding for cognitive coaching; and a focus on career pathing * $500 new hire home setup stipend and $200/mo for wifi, phone and co-working * What we’re looking for today is someone who brings deep agent expertise and uses it to expand what the team can build: new agents, new surface area in Canvas, memory, spatial awareness, improved performance on our Bedrock loop * Product judgment. You can look at what the agent layer does today and see what it should do next and make that case with a prototype, not a deck * Agent architecture depth. You understand how a fast, high-cardinality data store changes what an agent can reason about, and how to design for that * End-to-end ownership. On a small team there’s no handoff queue. You can take something from rough prototype to production-grade without needing someone behind you to do the durable engineering * Current judgment, not just past experience. You have informed opinions about what’s shifted in agent design in the last six to twelve months that would change how you’d build today * AI and agent engineering experience. You’ve shipped LLM-based systems people relied on in production not demos, not fine-tuned models in a research context. You know where agent systems break and how to design around it * Observability or developer-tools background. Engineers are your users; you’ll ramp faster with fluency in that world, and the work is better * Familiarity with eval frameworks, agent tooling, RAG, and prompt engineering