Making capacity planning easier with Datadog + Dify + Slack
This is Kataoka from the Service Reliability Group (SRG) of the Media Division.
#SRGThe Service Reliability Group primarily provides comprehensive support for the infrastructure surrounding our media services, focusing on improving existing services, launching new ones, and contributing to open-source software (OSS).
This article is about the SRG TechFest hackathon event held within SRG. This is an introduction to a Proof of Concept (PoC) designed to simplify capacity planning.
SRG TechFest ？
Before getting into the main topic, let me briefly explain SRG TechFest. It's a hackathon event held quarterly by SRG. Each time, a theme is chosen, and individuals or teams work on it for a full day.
This time, we'll be working in teams of 3-4 people on the theme of "Utilizing Generative AI from an SRE Perspective."
* In addition to the above, SRG also holds various other activities such as workshops (study sessions) every quarter and weekly reading groups organized by volunteers.
Datadog + Dify +Slack
This time, our team decided to try entrusting capacity planning to AI, with the goal of "making capacity planning easier."
SRG has a Dify environment that members can use freely, so we explored ways to leverage this and integrate it with Datadog, which is used in many services. We also set up notifications for capacity planning results to be sent to Slack.
Creations
The structure is as follows:
Step
- Specify the Datadog Dashboard ID and TimeRange.
- Retrieve all dashboard information using the Datadog API.
- LLM: Generate relevant Metrics queries from the results obtained via the API.
- Extract data as an array of queries.
- Iterate over the query (execute the Datadog QueryMetrics API).
- Data conversion (Array to String)
- LLM: Creating a capacity planning report (adjusting the format to suit Slack)
- Analyze the iteration results
Difficulties
- Using the example "Please output it like this" as the data in the answer (providing the example is surprisingly bad).
- The results change frequently each time I run the command prompt.
- 2, 7: What data is there, and how can the prompt be modified until the desired process is performed?
Improvement points
- Variable conversion within a query will send the data as is without replacing the information, even if it's already present in the data, so you need to add more detailed instructions.
- Prompting the system to provide detailed information about the data in the context improves accuracy.
- The API response passed to LLM lacked Query/Metrics information, which caused low accuracy. Therefore, we implemented a fix to include Query information in the API response.
result
They organized a summary of the dashboard data for the given period and pointed out areas of concern.
However, the suggested improvements seem to be limited to general suggestions. Actually, the output was completely off the mark at first, so this is a significant improvement. (Looking back, I think most of the time was spent battling the prompts.)
In conclusion
Ultimately, it wasn't something I could use as a capacity planner, but even with the short amount of time I had to actually work on it, I felt that the accuracy was gradually improving through repeated refinements, so I'd like to make time to try again.
SRG is looking for new team members.
If you are interested, please contact us here.