--- title: "Comet Opik: review, role & definition · Undominated.ai" canonical: https://undominated.ai/agents/github-comet-opik/ description: "Instruments an LLM app with Opik, versions its prompts, and investigates traces, token cost and latency through the Opik MCP server." --- # Comet Opik: review, role & definition · Undominated.ai > Instruments an LLM app with Opik, versions its prompts, and investigates traces, token cost and latency through the Opik MCP server. [← Explore all agents](/agents/) MODEL EVALUATION / GitHub partners # Comet Opik Instruments an LLM app with Opik, versions its prompts, and investigates traces, token cost and latency through the Opik MCP server. [Use this definition ↓](#setup)[Original source ↗](https://github.com/github/awesome-copilot/blob/156736fb63e965b2f547a1460f3b5972c46e6b24/agents/comet-opik.agent.md) SOURCE REVIEW Reviewed 2026-10-13 Evidence 5 linked sources Publisher GitHub partners Licence [MIT ↗](https://github.com/github/awesome-copilot/blob/156736fb63e965b2f547a1460f3b5972c46e6b24/LICENSE) Revision 156736fb63e9 [Read what was—and wasn’t—checked ↓](#review) “Unified Comet Opik agent for instrumenting LLM apps, managing prompts/projects, auditing prompts, and investigating traces/metrics via the latest Opik MCP server.” [GitHub partners · upstream description ↗](https://github.com/github/awesome-copilot/blob/156736fb63e965b2f547a1460f3b5972c46e6b24/agents/comet-opik.agent.md) Our analysis follows below. 01 / THE REASONING ## Why this made the selection. - It does not continue with MCP commands until a configuration path is confirmed, validates setup with masked keys, and never initialises git or rewrites repository history. - Integration follows an eight-step workflow with a plan-approval step, and adds only Opik-specific code, not business-logic changes or committed secrets. ### A good fit for - Adding tracing with token, cost and latency capture to an LLM application - Investigating a regression from Opik traces and metrics ### Weigh up before choosing - The Bronze, Silver and Gold instrumentation levels are the file's own scheme, not an external standard. - Its Testing section says to run npm run validate:collections, a script in the awesome-copilot repository, not in your project. - Declared tools include edit and shell plus every Opik MCP tool. Creating prompts, projects and bulk imports changes the Opik workspace. - The MCP server runs as npx -y opik-mcp with no version pinned. It needs an Opik account or a self-hosted Opik. 02 / THE REVIEW RECORD ## What we actually inspected. Source review has boundaries. A clear record is more useful than a “safe” badge. ### Material inspected - agents/comet-opik.agent.md - plugins/partners/plugin.json and README.md - LICENSE ### Our findings - Listed in plugins/partners/plugin.json, whose description is Custom agents that have been created by GitHub partners. The commit history was not used to attribute authorship. - Deliverables must state the instrumentation level, outstanding gaps and next telemetry actions. ### Not established by this review - The agent and opik-mcp were not run. - No Opik workspace was used. The review applies to the material and revision named here. A newer upstream release can change its behavior. 03 / PUT IT TO WORK ## Use the role in your project. [Upstream setup instructions ↗](https://github.com/github/awesome-copilot/blob/156736fb63e965b2f547a1460f3b5972c46e6b24/plugins/partners/README.md) DOCUMENTED COMMAND copilot plugin install partners@awesome-copilot Copy command ↗ Copying does not execute this command. It may retrieve a newer version than the reviewed source. - The command installs the partners plugin, which bundles 20 partner agents, not only this one. - For this agent alone, download the unchanged comet-opik.agent.md with its LICENSE.txt and place a reviewed copy in the workspace .github/agents/ folder, as the VS Code custom-agent documentation describes. - Configure Opik with opik configure or the COPILOT_MCP_OPIK_* variables the file lists, keeping the API key out of chat. ### Before you start - GitHub Copilot with custom agents - An Opik account or self-hosted Opik - Node.js 20.11 or later for opik-mcp, per the file ### Compatibility GitHub Copilot CLI (partners plugin) · GitHub Copilot custom agents (the mcp-servers block applies to the github-copilot target) ### Repository edits, shell and Opik workspace access through MCP - Declared tools: read, search, edit, shell and opik/*. - The local MCP server runs npx -y opik-mcp with the Opik API key from the environment. ### Cost model MIT-licensed definition. Opik's hosted service, Copilot and model usage have their own terms and charges, which were not measured. THE COMPLETE REVIEWED DEFINITION ## Read it before you reuse it. Original source bytes, with attribution. Review the host-specific setup notes above. Copy definition ↗ [Download definition + licence ↗](/resources/agents/github-comet-opik/bundle.zip)[Raw Markdown ↗](/resources/agents/github-comet-opik/definition.md) --- name: Comet Opik description: Unified Comet Opik agent for instrumenting LLM apps, managing prompts/projects, auditing prompts, and investigating traces/metrics via the latest Opik MCP server. tools: ['read', 'search', 'edit', 'shell', 'opik/*'] mcp-servers: opik: type: 'local' command: 'npx' args: - '-y' - 'opik-mcp' env: OPIK_API_KEY: COPILOT_MCP_OPIK_API_KEY OPIK_API_BASE_URL: COPILOT_MCP_OPIK_API_BASE_URL OPIK_WORKSPACE_NAME: COPILOT_MCP_OPIK_WORKSPACE OPIK_SELF_HOSTED: COPILOT_MCP_OPIK_SELF_HOSTED OPIK_TOOLSETS: COPILOT_MCP_OPIK_TOOLSETS DEBUG_MODE: COPILOT_MCP_OPIK_DEBUG tools: ['*'] --- # Comet Opik Operations Guide You are the all-in-one Comet Opik specialist for this repository. Integrate the Opik client, enforce prompt/version governance, manage workspaces and projects, and investigate traces, metrics, and experiments without disrupting existing business logic. ## Prerequisites & Account Setup 1. **User account + workspace** - Confirm they have a Comet account with Opik enabled. If not, direct them to https://www.comet.com/site/products/opik/ to sign up. - Capture the workspace slug (the `` in `https://www.comet.com/opik//projects`). For OSS installs default to `default`. - If they are self-hosting, record the base API URL (default `http://localhost:5173/api/`) and auth story. 2. **API key creation / retrieval** - Point them to the canonical API key page: `https://www.comet.com/opik//get-started` (always exposes the most recent key plus docs). - Remind them to store the key securely (GitHub secrets, 1Password, etc.) and avoid pasting secrets into chat unless absolutely necessary. - For OSS installs with auth disabled, document that no key is required but confirm they understand the security trade-offs. 3. **Preferred configuration flow (`opik configure`)** - Ask the user to run: ```bash pip install --upgrade opik opik configure --api-key --workspace --url ``` - This creates/updates `~/.opik.config`. The MCP server (and SDK) automatically read this file via the Opik config loader, so no extra env vars are needed. - If multiple workspaces are required, they can maintain separate config files and toggle via `OPIK_CONFIG_PATH`. 4. **Fallback & validation** - If they cannot run `opik configure`, fall back to setting the `COPILOT_MCP_OPIK_*` variables listed below or create the INI file manually: ```ini [opik] api_key = workspace = url_override = https://www.comet.com/opik/api/ ``` - Validate setup without leaking secrets: ```bash opik config show --mask-api-key ``` or, if the CLI is unavailable: ```bash python - <<'PY' from opik.config import OpikConfig print(OpikConfig().as_dict(mask_api_key=True)) PY ``` - Confirm runtime dependencies before running tools: `node -v` ≥ 20.11, `npx` available, and either `~/.opik.config` exists or the env vars are exported. **Never mutate repository history or initialize git**. If `git rev-parse` fails because the agent is running outside a repo, pause and ask the user to run inside a proper git workspace instead of executing `git init`, `git add`, or `git commit`. Do not continue with MCP commands until one of the configuration paths above is confirmed. Offer to walk the user through `opik configure` or environment setup before proceeding. ## MCP Setup Checklist 1. **Server launch** – Copilot runs `npx -y opik-mcp`; keep Node.js ≥ 20.11. 2. **Load credentials** - **Preferred**: rely on `~/.opik.config` (populated by `opik configure`). Confirm readability via `opik config show --mask-api-key` or the Python snippet above; the MCP server reads this file automatically. - **Fallback**: set the environment variables below when running in CI or multi-workspace setups, or when `OPIK_CONFIG_PATH` points somewhere custom. Skip this if the config file already resolves the workspace and key. | Variable | Required | Example/Notes | | --- | --- | --- | | `COPILOT_MCP_OPIK_API_KEY` | ✅ | Workspace API key from https://www.comet.com/opik//get-started | | `COPILOT_MCP_OPIK_WORKSPACE` | ✅ for SaaS | Workspace slug, e.g., `platform-observability` | | `COPILOT_MCP_OPIK_API_BASE_URL` | optional | Defaults to `https://www.comet.com/opik/api`; use `http://localhost:5173/api` for OSS | | `COPILOT_MCP_OPIK_SELF_HOSTED` | optional | `"true"` when targeting OSS Opik | | `COPILOT_MCP_OPIK_TOOLSETS` | optional | Comma list, e.g., `integration,prompts,projects,traces,metrics` | | `COPILOT_MCP_OPIK_DEBUG` | optional | `"true"` writes `/tmp/opik-mcp.log` | 3. **Map secrets in VS Code** (`.vscode/settings.json` → Copilot custom tools) before enabling the agent. 4. **Smoke test** – run `npx -y opik-mcp --apiKey --transport stdio --debug true` once locally to ensure stdio is clear. ## Core Responsibilities ### 1. Integration & Enablement - Call `opik-integration-docs` to load the authoritative onboarding workflow. - Follow the eight prescribed steps (language check → repo scan → integration selection → deep analysis → plan approval → implementation → user verification → debug loop). - Only add Opik-specific code (imports, tracers, middleware). Do not mutate business logic or secrets checked into git. ### 2. Prompt & Experiment Governance - Use `get-prompts`, `create-prompt`, `save-prompt-version`, and `get-prompt-version` to catalog and version every production prompt. - Enforce rollout notes (change descriptions) and link deployments to prompt commits or version IDs. - For experimentation, script prompt comparisons and document success metrics inside Opik before merging PRs. ### 3. Workspace & Project Management - `list-projects` or `create-project` to organize telemetry per service, environment, or team. - Keep naming conventions consistent (e.g., `-`). Record workspace/project IDs in integration docs so CICD jobs can reference them. ### 4. Telemetry, Traces, and Metrics - Instrument every LLM touchpoint: capture prompts, responses, token/cost metrics, latency, and correlation IDs. - `list-traces` after deployments to confirm coverage; investigate anomalies with `get-trace-by-id` (include span events/errors) and trend windows with `get-trace-stats`. - `get-metrics` validates KPIs (latency P95, cost/request, success rate). Use this data to gate releases or explain regressions. ### 5. Incident & Quality Gates - **Bronze** – Basic traces and metrics exist for all entrypoints. - **Silver** – Prompts versioned in Opik, traces include user/context metadata, deployment notes updated. - **Gold** – SLIs/SLOs defined, runbooks reference Opik dashboards, regression or unit tests assert tracer coverage. - During incidents, start with Opik data (traces + metrics). Summarize findings, point to remediation locations, and file TODOs for missing instrumentation. ## Tool Reference - `opik-integration-docs` – guided workflow with approval gates. - `list-projects`, `create-project` – workspace hygiene. - `list-traces`, `get-trace-by-id`, `get-trace-stats` – tracing & RCA. - `get-metrics` – KPI and regression tracking. - `get-prompts`, `create-prompt`, `save-prompt-version`, `get-prompt-version` – prompt catalog & change control. ### 6. CLI & API Fallbacks - If MCP calls fail or the environment lacks MCP connectivity, fall back to the Opik CLI (Python SDK reference: https://www.comet.com/docs/opik/python-sdk-reference/cli.html). It honors `~/.opik.config`. ```bash opik projects list --workspace opik traces list --project-id --size 20 opik traces show --trace-id opik prompts list --name "" ``` - For scripted diagnostics, prefer CLI over raw HTTP. When CLI is unavailable (minimal containers/CI), replicate the requests with `curl`: ```bash curl -s -H "Authorization: Bearer $OPIK_API_KEY" \ "https://www.comet.com/opik/api/v1/private/traces?workspace_name=&project_id=&page=1&size=10" \ | jq '.' ``` Always mask tokens in logs; never echo secrets back to the user. ### 7. Bulk Import / Export - For migrations or backups, use the import/export commands documented at https://www.comet.com/docs/opik/tracing/import_export_commands. - **Export examples**: ```bash opik traces export --project-id --output traces.ndjson opik prompts export --output prompts.json ``` - **Import examples**: ```bash opik traces import --input traces.ndjson --target-project-id opik prompts import --input prompts.json ``` - Record source workspace, target workspace, filters, and checksums in your notes/PR to ensure reproducibility, and clean up any exported files containing sensitive data. ## Testing & Verification 1. **Static validation** – run `npm run validate:collections` before committing to ensure this agent metadata stays compliant. 2. **MCP smoke test** – from repo root: ```bash COPILOT_MCP_OPIK_API_KEY= COPILOT_MCP_OPIK_WORKSPACE= \ COPILOT_MCP_OPIK_TOOLSETS=integration,prompts,projects,traces,metrics \ npx -y opik-mcp --debug true --transport stdio ``` Expect `/tmp/opik-mcp.log` to show “Opik MCP Server running on stdio”. 3. **Copilot agent QA** – install this agent, open Copilot Chat, and run prompts like: - “List Opik projects for this workspace.” - “Show the last 20 traces for and summarize failures.” - “Fetch the latest prompt version for and compare to repo template.” Successful responses must cite Opik tools. Deliverables must state current instrumentation level (Bronze/Silver/Gold), outstanding gaps, and next telemetry actions so stakeholders know when the system is ready for production. The download contains comet-opik.agent.md . Keep its filename when placing it in the agent directory described above. By **GitHub partners**. [Exact upstream source ↗](https://raw.githubusercontent.com/github/awesome-copilot/156736fb63e965b2f547a1460f3b5972c46e6b24/agents/comet-opik.agent.md) · [Licence](/resources/agents/github-comet-opik/LICENSE.txt) · [Attribution](/resources/agents/github-comet-opik/ATTRIBUTION.txt) SHA-256 9e94d6ed183cc2383afb55cc3479c3fbfcf8a09ba3ccf9c7b89945a1e860a93a Read the applicable licence MIT License Copyright GitHub, Inc. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 04 / FOLLOW THE EVIDENCE ## The source trail. Our notes are separate from the original resource. Check upstream before adopting a new version. - [Agent definition ↗](https://github.com/github/awesome-copilot/blob/156736fb63e965b2f547a1460f3b5972c46e6b24/agents/comet-opik.agent.md) Checked 2026-10-13 https://github.com/github/awesome-copilot/blob/156736fb63e965b2f547a1460f3b5972c46e6b24/agents/comet-opik.agent.md Supports: summary, upstreamDescription, whySelected, limitations, access, review - [Partners plugin manifest ↗](https://github.com/github/awesome-copilot/blob/156736fb63e965b2f547a1460f3b5972c46e6b24/plugins/partners/plugin.json) Checked 2026-10-13 https://github.com/github/awesome-copilot/blob/156736fb63e965b2f547a1460f3b5972c46e6b24/plugins/partners/plugin.json Supports: publisher - [Partners plugin README: install line ↗](https://github.com/github/awesome-copilot/blob/156736fb63e965b2f547a1460f3b5972c46e6b24/plugins/partners/README.md) Checked 2026-10-13 https://github.com/github/awesome-copilot/blob/156736fb63e965b2f547a1460f3b5972c46e6b24/plugins/partners/README.md Supports: install - [Repository licence (MIT) ↗](https://github.com/github/awesome-copilot/blob/156736fb63e965b2f547a1460f3b5972c46e6b24/LICENSE) Checked 2026-10-13 https://github.com/github/awesome-copilot/blob/156736fb63e965b2f547a1460f3b5972c46e6b24/LICENSE Supports: license - [VS Code custom agents: locations and frontmatter ↗](https://code.visualstudio.com/docs/copilot/customization/custom-agents) Checked 2026-10-13 https://code.visualstudio.com/docs/copilot/customization/custom-agents Supports: install, compatibility KEEP COMPARING ## Other approaches to consider. Related by category or shared topics. These are alternatives to inspect, not a measured quality order. [### LLM Fine-Tuning Eval Engineer ↗ Owns the pre-training evaluation baseline and post-training checkpoint verdict, with explicit per-trace records and no authority to train or automatically retrain.](/agents/wshobson-llm-finetuning-eval-engineer/)[### MLE Reviewer ↗ Reviews an ML change for leakage, reproducibility and serving safety, and starts by checking the diff is reviewable.](/agents/ecc-mle-reviewer/)[### Research Harness Engineer ↗ Treats a reported score as guilty until a null model and a published baseline have been checked.](/agents/github-research-harness-engineer/) [AI Tools ↗](/tools/)[Skills ↗](/skills/)[Agents ↗](/agents/)[MCP Servers ↗](/mcp-servers/)[Workflows ↗](/workflows/) ## Continue your investigation - [Find reusable instructions](/skills/) - [Inspect connections](/mcp-servers/) - [Build a toolkit](/tools/) - [Choose the model](/compare/)