Alerting
Watch agent runs for loops, failures, and cost, and get notified when something breaks.
keelwave evaluates Alert Rules against your agent telemetry and opens an
Alert when a rule's condition holds. Aggregate alerts move through
pending → firing → recovering → resolved; event alerts (like loop) fire
once and suppress re-notifying until their cooldown elapses. Every notification
attempt is recorded so you can tell "nothing fired" apart from "the email
never arrived".
Creating a rule
Open Alerts → Rules → New rule in the dashboard. Pick the signal you want to watch; the form shows only the fields that signal needs.
| Signal | What it measures | Class |
|---|---|---|
loop | An agent repeated the same tool call | Event only |
run_failure | Runs finishing in a failed state | Event, or aggregate over a window |
cost_burn | USD spent over the window | Aggregate |
termination_shift | Share of runs ending in error, timeout, or max steps | Aggregate |
tool_failure | Share of failing tool calls | Aggregate |
duration_p95 | 95th percentile run duration | Aggregate |
eval_regression | Average evaluation score | Aggregate |
loop fires per run, on run finish, with no threshold to configure.
run_failure can go either way: left alone it's an event rule that fires the
moment a run fails, or you can flip it to evaluate as a completion rate over a
window like the other aggregate signals. Every other signal is aggregate-only —
it's evaluated on a schedule over a window, so it needs a comparator, a
threshold, and a window length. As you type a threshold, the form shows the
metric's current value and whether the rule would breach right now.
Comparator direction is fixed per signal
The comparator isn't a free choice — each aggregate signal only accepts the direction that makes sense for it, and the server rejects the other one:
- Lower-is-bad (
run_failure,eval_regression): only<or<=. These track a rate you want to stay up — completion rate, correctness — so the alert fires when it drops too low. - Higher-is-bad (everything else —
cost_burn,termination_shift,tool_failure,duration_p95): only>or>=.
The rule form only offers the valid comparator for whichever signal is selected, so you can't build a rule the API would reject.
Alert lifecycle
An aggregate alert that clears before for_seconds elapses never notifies —
it never reaches firing. The dashboard shows it as Cleared before firing
(a resolved alert with no fired_at) so you can see the damping working
rather than wondering where the alert went.
Delivery
Every alert with a notification attempt carries the status of its most recent delivery job:
- Sent — the job reached
done. - Retrying (n) — the job is still
pendingorprocessing;nis the attempt count so far. - Failed — the job went
deadafter exhausting retries. Hovering the badge shows the underlying error.
A firing alert with a failed delivery means the rule worked and the email did not — check your SMTP configuration in Configuration.
API
Alerts are readable programmatically with a project API key. state=active
covers pending, firing, and recovering:
curl -H "Authorization: Bearer $KEELWAVE_API_KEY" \
"http://localhost:8080/v1/projects/$PROJECT_ID/alerts/events?state=active"Alert rules themselves — creating, listing, updating, deleting, and previewing a draft rule's current value — are org/admin-scoped rather than project-key scoped, since a preview runs an arbitrary aggregate query and rule changes affect who gets paged:
curl -H "Authorization: Bearer $KEELWAVE_ADMIN_TOKEN" \
"http://localhost:8080/v1/admin/orgs/$ORG_ID/projects/$PROJECT_ID/alert-rules"See the API reference for the full response shape.