About SRGTechnical articleRecruitment Information
About SRGTechnical articleRecruitment Information
HOME/Technical article/Easily send Slack notifications for GKE cluster upgrades.

Easily send Slack notifications for GKE cluster upgrades.

This is Kobayashi (@berlinbytes) from the Service Reliability Group (SRG) of the Media Division.
#SRGThe Service Reliability Group primarily provides comprehensive support for the infrastructure surrounding our media services, focusing on improving existing services, launching new ones, and contributing to open-source software (OSS).
This article is a report on our testing of the new feature that allows GKE cluster-level events to be easily notified to Slack.
 

Introduction

Until now, GKE cluster-level events were only posted to Cloud Pub/Sub, so if you wanted to receive notifications in Slack,
  • GKE -> Cloud Pub/Sub -> Cloud Functions -> Slack
  • GKE -> Cloud Pub/Sub -> Datadog Logs -> Slack
We needed a mechanism to trigger events based on topics received from Pub/Sub, etc.
 
That will start around February 2025.GKE UpgradeEvent NotificationSince it seems that these notifications are now also being sent to Cloud Logging, it should be easier to send notifications to Slack than before, so I'm going to test it out.
 

Set up notifications

This could be easily achieved by triggering a log-based alert policy when logs are sent from GKE to Cloud Logging, and then directing the alert notifications to Slack.
 
 

Add Slack to your notification channels

First, set up a notification channel for Slack. The documentation says you can add it when creating an alert, but I encountered an error during the redirection process, so I'll explain the procedure that avoids this by preparing it in advance.
  1. Edit notification channels
  1. Add New
  1. 許可する
  1. If you do not have administrator privileges, an additional request will likely be sent to the administrator. If you have the necessary privileges, proceed to the next step.
  1. Slack Channel Name
  1. テスト環境通知用
  1. テスト通知を送信
 

Create an alert policy

We will now configure a log-based alert policy.
  1. Log ExplorerAccess
  1. In the query pane, create a query that is appropriate for your environment.
    1. クエリを実行
    1. If there is no output, broaden the log period and check to see if output is generated.
    1. ログアラートの作成
    1. ログベースのアラート ポリシーの作成
      1. GKE UpgradeEvent Notification
      2. No Severity
      3. 次へ
      4. Preview logs
      5. 次へ
    1. Slack
    1. Save
     

    Test the alerts.

    We will manually write and publish log entries and verify that alerts are triggered.
    1. Method: entries.write  |  Cloud Logging  |  Google CloudAccess to
    1. Request body
      1. Execute
      1. Once the authentication flow is displayed, follow the instructions.
      1. 200
       
      With that, we were able to notify Slack of the GKE upgrade.
      That was easy, wasn't it!

      price

      Although there was much talk this year about the service becoming a paid service, as of March 2025, the alert policy is still free.
      The page below states that billing will begin in April 2026.
      • $1.50 per month per alert policy condition.
      • The query for the metric alert policy conditions returns $0.35 per 1,000,000 time series.
      月額 $1.50
       

      In conclusion

      The ability to easily send notifications to Slack without having to process Pub/Sub was very convenient and easy to use.
      Other cluster-level notifications include:
      It appears that this data is now also being output to Cloud Logging.
      It's also useful to set up notifications in Slack using the same procedure.
      And you can easily send notifications via email, SMS, and PagerDuty!
       
      SRG is looking for new team members. If you are interested, please contact us here.