/ README.md
README.md
1 This project is no longer supported by AcalephStorage. Please use https://github.com/fusiondog/consul-alerts for active development going forward. 2 3 consul-alerts 4 ============= 5 6 [](https://gitter.im/AcalephStorage/consul-alerts?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) 7 8 A highly available daemon for sending notifications and reminders based on Consul health checks. 9 10 Under the covers, consul-alerts leverages Consul's own leadership election and KV store to provide automatic failover and seamless operation in the case of a consul-alerts node failure and ensures that your notifications are still sent. 11 12 consul-alerts provides a high degree of configuration including: 13 14 - Several built-in [Notifiers](#notifiers) for distribution of health check alerts (email, sns, pagerduty, etc.) 15 - The ability to create Notification Profiles, sets of Notifiers which will respond to the given alert when a configurable threshold is exceeded 16 - Multiple degrees of customization for Notifiers and Blacklisting of alerts (service, check id or host) 17 18 19 ## Requirement 20 21 1. Consul 0.4+. Get it [here](http://consul.io). 22 2. Configured `GOPATH`. 23 24 Releases 25 -------- 26 27 Stable release are [here](https://github.com/AcalephStorage/consul-alerts/releases). 28 29 Latest release are found here: 30 - [darwin-amd64](https://bintray.com/artifact/download/darkcrux/generic/consul-alerts-latest-darwin-amd64.tar) 31 - [FreeBSD-amd64](https://bintray.com/artifact/download/darkcrux/generic/consul-alerts-latest-FreeBSD-amd64.tar) 32 - [linux-386](https://bintray.com/artifact/download/darkcrux/generic/consul-alerts-latest-linux-386.tar) 33 - [linux-amd64](https://bintray.com/artifact/download/darkcrux/generic/consul-alerts-latest-linux-amd64.tar) 34 35 Installation 36 ------------ 37 38 ``` 39 $ go get github.com/AcalephStorage/consul-alerts 40 $ go install 41 ``` 42 43 This should install consul-alerts to `$GOPATH/bin` 44 45 or pull the image from `docker`: 46 47 ``` 48 $ docker pull acaleph/consul-alerts 49 50 ``` 51 52 Usage 53 ----- 54 55 ``` 56 $ consul-alerts start 57 ``` 58 59 By default, this runs the daemon and API at localhost:9000 and connects to the local consul agent (localhost:8500) and default datacenter (dc1). These can be overriden by the following flags: 60 61 ``` 62 $ consul-alerts start --alert-addr=localhost:9000 --consul-addr=localhost:8500 --consul-dc=dc1 --consul-acl-token="" 63 ``` 64 65 Once the daemon is running, it can act as a handler for consul watches. At the moment only checks and events are supported. 66 67 ``` 68 $ consul watch -type checks consul-alerts watch checks --alert-addr=localhost:9000 69 $ consul watch -type event consul-alerts watch event --alert-addr=localhost:9000 70 ``` 71 72 or run the watchers on the agent the daemon connects by adding the following flags during consul-alerts run: 73 74 ``` 75 $ consul-alerts start --watch-events --watch-checks 76 ``` 77 78 Usage - Docker 79 -------------- 80 81 There are a few options for running in docker. 82 83 First option is using the consul agent built into the container. This option requires overriding the default entry point and running an exec to launch consul alerts. 84 85 Start consul: 86 87 ``` 88 docker run -ti \ 89 --rm -p 9000:9000 \ 90 --hostname consul-alerts \ 91 --name consul-alerts \ 92 --entrypoint=/bin/consul \ 93 acaleph/consul-alerts \ 94 agent -data-dir /data -server -bootstrap -client=0.0.0.0 95 ``` 96 Then in a separate terminal start consul-alerts: 97 98 ``` 99 $ docker exec -ti consul-alerts /bin/consul-alerts start --alert-addr=0.0.0.0:9000 --log-level=info --watch-events --watch-checks 100 ``` 101 102 The second option is to link to an existing consul container through docker networking and --link option. This method can more easily 103 share the consul instance with other containers such as vault. 104 105 First launch consul container: 106 107 ``` 108 $ docker run \ 109 -p 8400:8400 \ 110 -p 8500:8500 \ 111 -p 8600:53/udp \ 112 --hostname consul \ 113 --name consul \ 114 progrium/consul \ 115 -server -bootstrap -ui-dir /ui 116 ``` 117 118 Then run consul alerts container: 119 120 ``` 121 $ docker run -ti \ 122 -p 9000:9000 \ 123 --hostname consul-alerts \ 124 --name consul-alerts \ 125 --link consul:consul \ 126 acaleph/consul-alerts start \ 127 --consul-addr=consul:8500 \ 128 --log-level=info --watch-events --watch-checks 129 ``` 130 131 Last option is to launch the container and point at a remote consul instance: 132 133 ``` 134 $ docker run -ti \ 135 -p 9000:9000 \ 136 --hostname consul-alerts \ 137 --name consul-alerts \ 138 acaleph/consul-alerts start \ 139 --consul-addr=remote-consul-server.domain.tdl:8500 \ 140 --log-level=info --watch-events --watch-checks 141 ``` 142 143 **NOTE:** Don't change --alert-addr when using the docker container. 144 145 Configuration 146 ------------- 147 148 To assure consistency between instances, configuration is stored in Consul's KV with the prefix: `consul-alerts/config/`. consul-alerts works out of the box without any customizations by using the defaults documented below and leverages the KV settings as overrides. 149 150 A few suggestions on operating and bootstrapping your consul-alerts configuration via the KV store are located in the [Operations](#operations) section below. 151 152 If **ACL**s are enabled the folowing policy should be configured for consul-alerts token: 153 154 ``` 155 key "consul-alerts" { 156 policy = "write" 157 } 158 159 service "" { 160 policy = "read" 161 } 162 163 event "" { 164 policy = "read" 165 } 166 167 session "" { 168 policy = "write" 169 } 170 ``` 171 172 ### Health Checks 173 174 Health checking is enabled by default and is at the core what consul-alerts provides. The Health Check functionality is responsible for triggering a notification when the given consul check has changed status. To prevent flapping, notifications are only sent when a check status has been consistent for a specific time in seconds (60 by default). The threshold can be set globally or for a particular node, check, service and/or all of them. 175 176 **Configuration Options:** 177 The default Health Check configuration can be customized by setting kv with the prefix: `consul-alerts/config/checks/` 178 179 | key | description | 180 |------------------|----------------------------------------------------------------------------------------------------| 181 | enabled | Globally enable the Health Check functionality. [Default: true] | 182 | change-threshold | The time, in seconds, that a check must be in a given status before an alert is sent [Default: 60] | 183 | single/{{ node }}/{{ serviceID }}/{{ checkID }}/change-threshold | Overrides `change-threshold` for a specific check associated with a particular service running on a particular node | 184 | check/{{ checkID }}/change-threshold | Overrides `change-threshold` for a specific check | 185 | service/{{ serviceID }}/change-threshold | Overrides `change-threshold` for a specific service | 186 | node/{{ node }}/change-threshold | Overrides `change-threshold` for a specific node | 187 188 When `change-threshold` is overridden multiple times, the most specific condition will be used based on the following order: (most specific) `single` > `check` > `service` > `node` > `global settings` > `default settings` (least specific). 189 190 ### Notification Profiles 191 192 Notification Profiles allow the operator the ability to customize how often and to which Notifiers alerts will be sent via the Interval and NotifList attributes described below. 193 194 Profiles are configured as keys with the prefix: `consul-alerts/config/notif-profiles/`. 195 196 #### Notification Profile Specification 197 198 **Key:** The name of the Notification Profile 199 200 Ex. `emailer_only` would be located at `consul-alerts/config/notif-profiles/emailer_only` 201 202 **Value:** A JSON object adhering to the schema shown below. 203 204 ``` 205 { 206 "$schema": "http://json-schema.org/draft-04/schema#", 207 "type": "object", 208 "title": "Notifier Profile Schema.", 209 "description": "Defines a given Notifier Profile's configuration", 210 "properties": { 211 "Interval": { 212 "type": "integer", 213 "title": "Reminder Interval.", 214 "description": "Defines the Interval (in minutes) which Reminders should be sent to the given Notifiers. Should be a multiple of 5." 215 }, 216 "NotifList": { 217 "type": "object", 218 "title": "Hash of Notifiers to configure.", 219 "description": "A listing of Notifier names with a boolean value indicating if it should be enabled or not.", 220 "patternProperties" : { 221 ".{1,}" : { "type" : "string" } 222 } 223 }, 224 "VarOverrides": { 225 "type": "object", 226 "title": "Hash of Notifier variables to override.", 227 "description": "A listing of Notifier names with hash values containing the parameters to be overridden", 228 "patternProperties" : { 229 ".{1,}" : { "type" : "object" } 230 } 231 } 232 }, 233 "required": [ 234 "Interval", 235 "NotifList" 236 ] 237 } 238 ``` 239 240 #### Notification Profile Examples 241 **Notification Profile to only send Emails with reminders every 10 minutes:** 242 243 **Key:** `consul-alerts/config/notif-profiles/emailer_only` 244 245 **Value:** 246 ``` 247 { 248 "Interval": 10, 249 "NotifList": { 250 "log":false, 251 "email":true 252 } 253 } 254 ``` 255 256 **NOTE:** While it is completely valid to explicitly disable a Notifier in a Notifier Profile, it is not necessary. In the event that a Notification Profile is used, only Notifiers which are explicitly defined and enabled will be used. In the example above then, we could have omitted the `"log": false` in the `NotifList` and achieved the same results. 257 258 **Example - Notification Profile to only send to PagerDuty but never send reminders:** 259 260 **Key:** `consul-alerts/config/notif-profiles/pagerduty_no_reminders` 261 262 **Value:** 263 ``` 264 { 265 "Interval": 0, 266 "NotifList": { 267 "pagerduty":true 268 } 269 } 270 ``` 271 272 **NOTE:** The Interval being set to 0 **disables** Reminders from being sent for a given alert. If the service stays in a critical status for an extended period, only that first notification will be sent. 273 274 **Example - Notification Profile to only send Emails to the overridden receivers:** 275 276 **Key:** `consul-alerts/config/notif-profiles/emailer_overridden` 277 278 **Value:** 279 ``` 280 { 281 "Interval": 10, 282 "NotifList": { 283 "email":true 284 }, 285 "VarOverrides": { 286 "email": { 287 "receivers": ["my-team@company.com"] 288 } 289 } 290 } 291 ``` 292 293 **Example - Notification Profile to disable Slack:** 294 295 **Key:** `consul-alerts/config/notif-profiles/slack_off` 296 297 **Value:** 298 ``` 299 { 300 "Interval": 0, 301 "NotifList": { 302 "slack":false 303 } 304 } 305 ``` 306 307 #### Notification Profile Activation 308 309 It is possible to activate Notification Profiles in 2 ways - for a specific entity or for a set of entities matching a regular expression. 310 For a specific item the selection is done by setting keys in `consul-alerts/config/notif-selection/services/`, `consul-alerts/config/notif-selection/checks/`, `consul-alerts/config/notif-selection/hosts/`, or `consul-alerts/config/notif-selection/status/`, with the appropriate service, check, or host name as the key and the desired Notification Profile name as the value. 311 To activate a Notification Profile for a set of entities matching a regular expression, create a json map of type `regexp->notification-profile` as a value for the keys `consul-alerts/config/notif-selection/services`, `consul-alerts/config/notif-selection/checks`, `consul-alerts/config/notif-selection/hosts`, or 312 `consul-alerts/config/notif-selection/status`. 313 314 **Example - Notification Profile activated for all the services which names start with infra-** 315 316 **Key:** `consul-alerts/config/notif-selection/services` 317 318 **Value:** 319 ``` 320 { 321 "^infra-.*$": "infra-support-profile" 322 } 323 ``` 324 325 **Example - Disable slack notifications when status is passing-** 326 327 **Key:** `consul-alerts/config/notif-selection/status/passing` 328 329 **Value:** `slack_off` 330 331 In addition to the service, check and host specific Notification Profiles, the operator can setup a default Notification Profile by creating a Notification Profile kv `consul-alerts/config/notif-profiles/default`, which acts as a fallback in the event a specific Notification Profile is not found. If there are no Notification Profiles matching the criteria, consul-alerts will send the notification to the full list of enabled Notifiers and no reminders will be sent. 332 333 As consul-alerts attempts to process a given notification, it has a series of lookups it does to associate an event with a given Notification Profile by matching on: 334 335 - Service 336 - Check 337 - Host 338 - Status 339 - Default 340 341 **NOTE:** An event will only trigger notification for the FIRST Notification Profile that meets it's criteria. 342 343 Reminders resend the notifications at programmable intervals until they are resolved or added to the blacklist. Reminders are processed every five minutes therefore Interval values should be a multiple of five. If the Interval value is 0 or not set then reminders will not be sent. 344 345 #### Enable/Disable Specific Health Checks 346 347 There are multiple ways to enable/disable health check notifications: mark them by node, serviceID, checkID, regular expression, or mark individually by node/serviceID/checkID. This is done by adding a KV entry in `consul-alerts/config/checks/blacklist/...`. Removing the entry will re-enable the check notifications. 348 349 ##### Disable all notification by node 350 351 Add a KV entry with the key `consul-alerts/config/checks/blacklist/nodes/{{ nodeName }}`. This will disable notifications for the specified `nodeName`. 352 353 ##### Disable all notifications for the nodes matching regular expressions 354 355 Add a KV entry with the key `consul-alerts/config/checks/blacklist/nodes` and the value containing a list of regular expressions. This will disable notifications for all the nodes, which names match at least one of the regular expressions. 356 357 ##### Disable all notification by service 358 359 Add a KV entry with the key `consul-alerts/config/checks/blacklist/services/{{ serviceId }}`. This will disable notifications for the specified `serviceId`. 360 361 ##### Disable all notifications for the services matching regular expressions 362 363 Add a KV entry with the key `consul-alerts/config/checks/blacklist/services` and the value containing a list of regular expressions. This will disable notifications for all the services, which names match at least one of the regular expressions. 364 365 ##### Disable all notification by healthCheck 366 367 Add a KV entry with the key `consul-alerts/config/checks/blacklist/checks/{{ checkId }}`. This will disable notifications for the specified `checkId`. 368 369 ##### Disable all notifications for the healthChecks matching regular expressions 370 371 Add a KV entry with the key `consul-alerts/config/checks/blacklist/checks` and the value containing a list of regular expressions. This will disable notifications for all the healthchecks, which names match at least one of the regular expressions. 372 373 ##### Disable a single health check 374 375 Add a KV entry with the key `consul-alerts/config/checks/blacklist/single/{{ node }}/{{ serviceId }}/{{ checkId }}`. This will disable the specific health check. If the health check is not associated with a service, use the `_` as the serviceId. 376 377 ### Events 378 379 Event handling is enabled by default. This delegates any consul event received by the agent to the list of handlers configured. To disable event handling, set `consul-alerts/config/events/enabled` to `false`. 380 381 Handlers can be configured by adding them to `consul-alerts/config/events/handlers`. This should be a JSON array of string. Each string should point to any executable. The event data should be read from `stdin`. 382 383 ### Notifiers 384 385 There are several built-in notifiers. Only the *Log* notifier is enabled by default. Details on enabling and configuration these are documented for each Notifier. 386 387 #### Custom Notifiers 388 It is also possible to add custom notifiers similar to custom event handlers. Custom notifiers can be added as keys with command path string values in `consul-alerts/config/notifiers/custom/`. The keys will be used as notifier names in the profiles. 389 390 #### Logger 391 392 This logs any health check notification to a file. To disable this notifier, set `consul-alerts/config/notifiers/log/enabled` to `false`. 393 394 The log file is set to `/tmp/consul-notifications.log` by default. This can be changed by changing `consul-alerts/config/notifiers/log/path`. 395 396 #### Email 397 398 This emails the health notifications. To enable this, set `consul-alerts/config/notifiers/email/enabled` to `true`. 399 400 The email and smtp details needs to be configured: 401 402 prefix: `consul-alerts/config/notifiers/email/` 403 404 | key | description | 405 |--------------|----------------------------------------------------------------------------------| 406 | enabled | Enable the email notifier. [Default: false] | 407 | cluster-name | The name of the cluster. [Default: "Consul Alerts"] | 408 | url | The SMTP server url | 409 | port | The SMTP server port | 410 | username | The SMTP username | 411 | password | The SMTP password | 412 | sender-alias | The sender alias. [Default: "Consul Alerts"] | 413 | sender-email | The sender email | 414 | receivers | The emails of the receivers. JSON array of string | 415 | template | Path to custom email template. [Default: internal template] | 416 | one-per-alert| Whether to send one email per alert [Default: false] | 417 | one-per-node | Whether to send one email per node [Default: false] (overriden by one-per-alert) | 418 419 The template can be any go html template. An `TemplateData` instance will be passed to the template. 420 421 #### InfluxDB 422 423 This sends the notifications as series points in influxdb. Set `consul-alerts/config/notifiers/influxdb/enabled` to `true` to enabled. InfluxDB details need to be set too. 424 425 prefix: `consul-alerts/config/notifiers/influxdb/` 426 427 | key | description | 428 |-------------|------------------------------------------------| 429 | enabled | Enable the influxdb notifier. [Default: false] | 430 | host | The influxdb host. (eg. localhost:8086) | 431 | username | The influxdb username | 432 | password | The influxdb password | 433 | database | The influxdb database name | 434 | series-name | The series name for the points | 435 436 #### Slack 437 438 Slack integration is also supported. To enable, set 439 `consul-alerts/config/notifiers/slack/enabled` to `true`. Slack details needs to 440 be configured. 441 442 prefix: `consul-alerts/config/notifiers/slack/` 443 444 | key | description | 445 |--------------|----------------------------------------------------- | 446 | enabled | Enable the Slack notifier. [Default: false] | 447 | cluster-name | The name of the cluster. [Default: `Consul Alerts`] | 448 | url | The incoming-webhook url (mandatory) [eg: `https://hooks.slack.com...`] | 449 | channel | The channel to post the notification (mandatory) [eg: `#consul-alerts` or `@consul-alerts`]| 450 | username | The username to appear on the post [eg: `Consul Alerts`] | 451 | icon-url | URL of a custom image for the notification [eg: `http://someimage.com/someimage.png`] | 452 | icon-emoji | Emoji (if not using icon-url) for the notification [eg: `:ghost:`] | 453 | detailed | Enable "pretty" Slack notifications [Default: false] | 454 455 In order to enable slack integration, you have to create a new 456 [_Incoming WebHooks_](https://my.slack.com/services/new/incoming-webhook). Then use the 457 token created by the previous action. 458 459 #### Mattermost 460 461 Mattermost integration is also supported. To enable, set 462 `consul-alerts/config/notifiers/mattermost/enabled` to `true`. Mattermost details needs to 463 be configured. 464 465 prefix: `consul-alerts/config/notifiers/mattermost/` 466 467 | key | description | 468 |--------------|-----------------------------------------------------| 469 | enabled | Enable the Mattermost notifier. [Default: false] | 470 | cluster-name | The name of the cluster. [Default: "Consul Alerts"] | 471 | url | The mattermost url (mandatory) | 472 | username | The mattermost username (mandatory) | 473 | password | The mattermost password (mandatory) | 474 | team | The mattermost team (mandatory) | 475 | channel | The channel to post the notification (mandatory) | 476 477 Notifications can also be sent to [Incoming 478 Webhooks](https://docs.mattermost.com/developer/webhooks-incoming.html). To enable, set 479 `consul-alerts/config/notifiers/mattermost-webhook/enabled` to `true` and set 480 `consul-alerts/config/notifiers/mattermost-webhook/url` to URL of the webhook created on 481 the previous step. 482 483 prefix: `consul-alerts/config/notifiers/mattermost-webhook/` 484 485 | key | description | 486 |--------------|----------------------------------------------------- | 487 | enabled | Enable the Mattermost Webhook notifier. [Default: false] | 488 | cluster-name | The name of the cluster. [Default: `Consul Alerts`] | 489 | url | The incoming-webhook url (mandatory) [eg: `https://mattermost.com/hooks/...`] | 490 | channel | The channel to post the notification (mandatory) [eg: `consul-alerts`] | 491 | username | The username to appear on the post [eg: `Consul Alerts`] | 492 | icon-url | URL of a custom image for the notification [eg: `http://someimage.com/someimage.png`] | 493 494 #### PagerDuty 495 496 To enable PagerDuty built-in notifier, set `consul-alerts/config/notifiers/pagerduty/enabled` to `true`. This is disabled by default. Service key and client details also needs to be configured. 497 498 prefix: `consul-alerts/config/notifiers/pagerduty/` 499 500 | key | description | 501 |---------------------|----------------------------------------------------------------------| 502 | enabled | Enable the PagerDuty notifier. [Default: false] | 503 | service-key | Service key to access PagerDuty | 504 | client-name | The monitoring client name | 505 | client-url | The monitoring client url | 506 | max-retry | The upper limit of retries on failure. [Default: `0` for no retries] | 507 | retry-base-interval | The base delay in seconds before a retry. [Default: `30` seconds ] | 508 509 #### HipChat 510 511 To enable HipChat built-in notifier, set 512 `consul-alerts/config/notifiers/hipchat/enabled` to `true`. Hipchat details 513 needs to be configured. 514 515 prefix: `consul-alerts/config/notifiers/hipchat/` 516 517 | key | description | 518 |--------------|-----------------------------------------------------------| 519 | enabled | Enable the HipChat notifier. [Default: false] | 520 | from | The name to send notifications as (optional) | 521 | cluster-name | The name of the cluster. [Default: "Consul Alerts"] | 522 | base-url | HipChat base url [Default: `https://api.hipchat.com/v2/`] | 523 | room-id | The room to post to (mandatory) | 524 | auth-token | Authentication token (mandatory) | 525 526 The `auth-token` needs to be a room notification token for the `room-id` 527 being posted to. 528 See [HipChat API docs](https://developer.atlassian.com/hipchat/guide/hipchat-rest-api). 529 530 The default `base-url` works for HipChat-hosted rooms. You only need to 531 override it if you are running your own server. 532 533 #### OpsGenie 534 535 To enable OpsGenie built-in notifier, set 536 `consul-alerts/config/notifiers/opsgenie/enabled` to `true`. OpsGenie details 537 needs to be configured. 538 539 prefix: `consul-alerts/config/notifiers/opsgenie/` 540 541 | key | description | 542 |--------------|-----------------------------------------------------| 543 | enabled | Enable the OpsGenie notifier. [Default: false] | 544 | cluster-name | The name of the cluster. [Default: "Consul Alerts"] | 545 | api-key | API Key (mandatory) | 546 547 #### Amazon Web Services Simple Notification Service ("SNS") 548 549 To enable AWS SNS built-in notifier, set 550 `consul-alerts/config/notifiers/awssns/enabled` to `true`. AWS SNS details 551 needs to be configured. 552 553 prefix: `consul-alerts/config/notifiers/awssns/` 554 555 | key | description | 556 |--------------|--------------------------------------------------------------| 557 | enabled | Enable the AWS SNS notifier. [Default: false] | 558 | cluster-name | The name of the cluster. [Default: "Consul Alerts"] | 559 | region | AWS Region (mandatory) | 560 | topic-arn | Topic ARN to publish to. (mandatory) | 561 | template | Path to custom template. [Default: internal template] | 562 #### VictorOps 563 564 To enable the VictorOps built-in notifier, set 565 `consul-alerts/config/notifiers/victorops/enabled` to `true`. VictorOps details 566 needs to be configured. 567 568 prefix: `consul-alerts/config/notifiers/victorops/` 569 570 | key | description | 571 |--------------|-----------------------------------------------------| 572 | enabled | Enable the VictorOps notifier. [Default: false] | 573 | api-key | API Key (mandatory) | 574 | routing-key | Routing Key (mandatory) | 575 576 #### HTTP Endpoint 577 578 To enable the HTTP endpoint built-in notifier, set 579 `consul-alerts/config/notifiers/http-endpoint/enabled` to `true`. HTTP endpoint details 580 needs to be configured. 581 582 prefix: `consul-alerts/config/notifiers/http-endpoint/` 583 584 | key | description | 585 |--------------|-----------------------------------------------------------------------| 586 | enabled | Enable the http-endpoint notifier. [Default: false] | 587 | cluster-name | The name of the cluster. [Default: "Consul Alerts"] | 588 | base-url | Base URL of the HTTP endpoint (mandatory) | 589 | endpoint | The endpoint to append to the end of base-url | 590 | payload | The payload to send to the HTTP endpoint (mandatory) | 591 592 The value of 'payload' must be a json map of type string. Value will be rendered as a template. 593 ``` 594 { 595 "message": "{{ range $name, $checks := .Nodes }}{{ range $check := $checks }}{{ $name }}:{{$check.Service}}:{{$check.Check}} is {{$check.Status}}.{{ end }}{{ end }}" 596 } 597 ``` 598 #### iLert 599 600 To enable iLert built-in notifier, set 601 `consul-alerts/config/notifiers/ilert/enabled` to `true`. Service API 602 key needs to be configured. 603 604 prefix: `consul-alerts/config/notifiers/ilert/` 605 606 | key | description | 607 |-----------------------|---------------------------------------------------------------------------| 608 | enabled | Enable the iLert notifier. [Default: false] | 609 | api-key | The API key of the alert source. (mandatory) | 610 | incident-key-template | Format of the incident key. [Default: `{{.Node}}:{{.Service}}:{{.Check}}` | 611 612 Health Check via API 613 -------------------- 614 615 Health status can also be queried via the API. This can be used for compatibility with nagios, sensu, or other monitoring tools. To get the status of a specific check, use the following entrypoint. 616 617 `http://consul-alerts:9000/v1/health?node=<node>&service=<serviceId>&check=<checkId>` 618 619 This will return the output of the check and the following HTTP codes: 620 621 | Status | Code | 622 |----------|------| 623 | passing | 200 | 624 | warning | 503 | 625 | critical | 503 | 626 | unknown | 404 | 627 628 `http://consul-alerts:9000/v1/health/wildcard?node=<node>&service=<serviceId>&check=<checkId>&status=<status>&alwaysOk=true&ignoreBlacklist=true` 629 `v1/health/wildcard` is similar to `v1/health` but returns all matched checks (omitted service/node/check params assumed as any) . Values returned in JSON form, status code 503 if one of services in critical state. 630 631 Additional params are ignoreBlacklist and alwaysOk which forces status code to 200 regardless of checks status. 632 633 ## Operations 634 Configuration may be set manually through consul UI or API, using configuration management tools such as [chef](https://github.com/dpetzel/consul_alerts-cookbook), puppet or Ansible, or backed up and restored using [consulate](https://github.com/gmr/consulate). 635 636 Consulate Example: 637 ``` 638 consulate kv backup consul-alerts/config -f consul-alerts-config.json 639 consulate kv restore consul-alerts/config -f consul-alerts-config.json --prune 640 ``` 641 642 Contribution 643 ------------ 644 645 PRs are more than welcome. Just fork, create a feature branch, and open a PR. We love PRs. :) 646 647 TODO 648 ---- 649 650 Needs better doc and some cleanup too. :)