What pages
Emergency alerts repeat until acknowledged. High alerts bypass Pushover’s
quiet hours. Normal alerts respect them.
What deliberately stays quiet
Routine pipeline events, every successful API call, and — importantly —access_denied. Refused requests are frequent by design; a channel
that fires on each one gets muted by its owner within a week, and then the
alerts that matter go unread too.
notification_failed never raises an alert of its own, so a broken channel
cannot feed itself.
The gate-closure watchdog
On every orchestration pass the service checks the day-ahead delivery day: if it is not submitted and accepted, and gate closure is inside the warning window, it raises an emergency alert. Two details that matter:- It runs regardless of whether anything changed. Change detection drives the pipeline, but “nothing changed” is exactly the symptom of a day nobody delivered data for.
- A rejected submission counts as not submitted. The submit step having run is not the same fact as the schedules being in.
Channels
Three, and any combination of them can be active. Each has a threshold: the priority at which it starts caring. That is what keeps the loud channels for the alerts that deserve them.
Channels are independent: one failing does not stop the others, and the
recorded failure names which one. Every credential below is editable from
the console, so the on-call target moves without a deployment, and the
secrets are masked when the configuration is read back.
Slack
Create an incoming webhook for the channel. The URL is the credential — anyone holding it can post — so it is treated as a secret.Pushover
Create an application at pushover.net for the application token, and take the user key from your dashboard. Both are needed before the channel exists.
Use a group key, not a personal one: an alert that only reaches someone
on holiday has not been delivered. With a group, an emergency alert stops
repeating as soon as any member acknowledges it.
Pushover delivers only to its own clients — iOS, Android and a desktop
browser client — each a separate one-time purchase per person.
SMS
SMS is the channel that survives a dead data connection and an uninstalled app. It costs per message, so it defaults to emergencies only.
Germany and Switzerland both allow alphanumeric sender IDs (e.g.
NLEAD), which avoids renting a number — the trade-off is that the
recipient cannot reply. All four values are required before the channel is
built: a half-configured channel that fails on every alert is worse than no
channel.
Bodies are truncated so one alert cannot become a long chain of billed
segments, and a bad number does not stop delivery to the others.
iMessage is not offered, deliberately. Apple publishes no API for it;
the only routes are a Mac running AppleScript or a third-party bridge,
both of which mean maintaining a Mac as production infrastructure that
can stop delivering silently. SMS reaches the same iPhone.
Adding another channel
Notifier is a Protocol with one method. Telegram, PagerDuty or a
webhook of your own is a class that POSTs a different payload, plus one
entry in build_notifier. The rules in alert_for do not change, because
they describe what is worth an alert, not where it goes.
The watchdog outside the service
Everything above runs inside the service, so none of it can report that the service is not running: if the container is down, so is the code that would tell you.infra/monitoring.bicep deploys four Azure Monitor rules
that read the Log Analytics workspace and therefore survive the thing they
are watching.
The signals come from what the orchestrator job actually logs. Every 15
minutes it calls the orchestrate endpoint and prints the JSON response —
one line that proves the schedule fired, the app was reachable, the
credential worked and the response was 2xx.
Each query is written to return a row even when nothing matched, so no
data is a value the rule compares rather than an absence it ignores. That
is the difference between a watchdog and a decoration.
Deployed separately from
main.bicep on purpose. A full deployment of the
application template reconciles the container app and would discard
secrets set outside it; watching a service should never be a reason to
touch it.