AgentOps: Monitor and Debug Your Local Business AI Agents

What AgentOps Does

• Session replay: watch every conversation step by step
• Cost tracking: exactly how much each agent run costs in API tokens
• Error detection: automatic flagging of failed LLM calls, tool errors, and unexpected outputs
• Latency monitoring: which steps are slow and why
• LLM comparison: run the same task with two different models and compare quality + cost
• Team dashboard: share access with a developer or VA who manages your agents

Installation and Setup

pip install agentops

Adding AgentOps to an Existing Agent

import agentops
import anthropic

# Initialize AgentOps at the start of your script
agentops.init("YOUR_AGENTOPS_API_KEY")

client = anthropic.Anthropic(api_key="YOUR_ANTHROPIC_API_KEY")

def generate_review_response(review: str, stars: int, reviewer_name: str) -> str:
    """Generate a Google review response with full AgentOps tracking."""
    
    # AgentOps automatically tracks this LLM call
    response = client.messages.create(
        model="claude-haiku-4-5-20251001",
        max_tokens=300,
        system="""You write professional Google review responses for The Loft Hair Studio.
Be warm, specific, and 3-5 sentences. Always thank the reviewer by name.""",
        messages=[{
            "role": "user",
            "content": f"Write a response to this {stars}-star review from {reviewer_name}: '{review}'"
        }]
    )
    
    return response.content[0].text

def process_weekly_reviews(reviews: list) -> dict:
    """Process all reviews for the week — every call tracked in AgentOps."""
    
    responses = {}
    
    for review in reviews:
        try:
            response = generate_review_response(
                review["text"],
                review["stars"],
                review["name"]
            )
            responses[review["id"]] = {"status": "success", "response": response}
            
            # Log custom events for business metrics
            agentops.record(agentops.ActionEvent(
                action_type="review_processed",
                params={"stars": review["stars"], "review_id": review["id"]},
                returns={"response_length": len(response)}
            ))
            
        except Exception as e:
            # AgentOps captures this error with full context
            agentops.record(agentops.ErrorEvent(exception=e))
            responses[review["id"]] = {"status": "error", "error": str(e)}
    
    return responses

# End the session — AgentOps uploads all data
agentops.end_session("Success")

Using AgentOps with CrewAI

import agentops
from crewai import Agent, Task, Crew

# AgentOps automatically instruments all CrewAI agents
agentops.init("YOUR_AGENTOPS_API_KEY")

review_writer = Agent(
    role="Review Response Writer",
    goal="Write warm, specific Google review responses",
    backstory="You've been managing customer relations for The Loft Hair Studio for years.",
    verbose=True
)

review_task = Task(
    description="Write responses to these reviews: {reviews}",
    agent=review_writer,
    expected_output="A response for each review, formatted as a list"
)

crew = Crew(agents=[review_writer], tasks=[review_task])

# AgentOps tracks every agent conversation, tool call, and LLM request
result = crew.kickoff(inputs={"reviews": sample_reviews})
agentops.end_session("Success")

What to Monitor

1. Cost Per Task

• Your review response bot costs $0.003 per review (acceptable)
• Your weekly marketing report costs $0.12 to generate (acceptable)
• One broken prompt is causing the FAQ bot to make 8 LLM calls per question instead of 1 (fix: tighten your prompt)

2. Error Rate

• max_tokens exceeded → increase max_tokens or summarize input first
• rate_limit_error → add exponential backoff retry logic
• context_length_exceeded → your conversation history is growing too long, add a summarization step

3. Latency Outliers

• Trimming the system prompt
• Limiting conversation history to last N turns
• Switching to a faster model (Claude Haiku vs Sonnet)

4. Output Quality with Tags

# After generating a review response, ask for a quick quality check
agentops.end_session(
    end_state="Success",
    end_state_reason="review_responded",
    tags=["review_response", f"stars_{stars}", "production"]
)

Debugging a Broken Agent

1. Find the session: Search by timestamp or customer ID in the dashboard
2. Replay the conversation: See every message exactly as sent and received
3. Identify the failure point: Was it the wrong prompt? A tool returning bad data? A hallucination?
4. Compare to good sessions: Filter to find similar successful sessions and spot the difference
5. Fix and verify: Update your prompt, redeploy, and watch the next sessions to confirm the fix

Comparison Table

FAQ

Related Articles

• AutoGen Multi-Agent: Automate Your Marketing Reports and Review Responses
• CrewAI for Local Business: Build a Team of AI Agents
• LangGraph: Build an AI Booking Agent with Memory and State
• LangChain RAG: AI That Answers Customer FAQs from Your Own Data
• OpenRouter Guide: Access 100+ AI Models for Local Business Automation