About SRGTechnical articleRecruitment Information
About SRGTechnical articleRecruitment Information
HOME/Technical article/Vector searching using n8n and Qdrant was easy.

Vector searching using n8n and Qdrant was easy.

This is Hasegawa (@rarirureluis) from the Service Reliability Group (SRG) of the Media Division.
#SRGThe Service Reliability Group primarily provides comprehensive support for the infrastructure surrounding our media services, focusing on improving existing services, launching new ones, and contributing to open-source software (OSS).
This article shares our findings from conducting a "Proof of Concept (PoC) for vector search for internal systems/ticket management" using the workflow automation tool n8n and the vector search engine Qdrant. We found that it was surprisingly easy and versatile to apply.

Introduction

In handling internal inquiries and managing tickets, it's often time-consuming and costly because past knowledge cannot be retrieved immediately.
This time, we verified the effectiveness of "instant search for similar tickets," "automatic FAQ responses," and "relevant document recommendations" through a proof-of-concept (PoC) for vector search.
The biggest benefit of combining n8n and Qdrant was the ability to quickly build workflows that could be used not only by engineers but also by operators.
I myself have absolutely no knowledge of vectors, but I proceeded while receiving various advice from AI.

Tools used this time

This system was built primarily using a combination of the following tools.

n8n

n8n is a platform for integrating various applications and APIs to automate workflows.
It features a node-based visual editor, allowing users with little programming knowledge to intuitively assemble processing flows.
With over 400 integration options available and the ability to self-host, you can handle your data flexibly and securely.

Qdrant

Qdrant is a database engine specifically designed for vector searches.
We excel at rapidly searching and managing AI-generated vector data (embedding).
Developed in Rust, it boasts high performance and reliability.
In addition to similarity searches, it can also store JSON-formatted payloads linked to vectors, allowing for filtering based on a variety of conditions, such as keywords and numerical ranges.
This allows you to implement complex business logic that goes beyond simple similarity searches.

GitHub and CSV

The question and answer pairs that form the basis of the Q&A data were created in CSV (Comma-Separated Values) format.
CSV is a simple text file with comma-separated values, making it easy to handle with many tools.
We decided to manage this CSV file in a GitHub repository, considering the ease of version control and collaborative editing.

Building a simple QA system

The overall system is very simple.
  1. Upload the CSV file containing the QA data to GitHub.
  1. First, set up the Qdrant environment and create the collection (the location where the data will be stored).
  1. We create a workflow using n8n, retrieve CSV data from GitHub, and feed it into Qdrant.
  1. When n8n receives a question from a user, it vectorizes the question content and queries Qdrant to find the most similar answer.
  1. The search results are returned to the user.
We implemented this entire process as an n8n workflow.
By simply connecting n8n nodes, we were able to easily automate processes such as data acquisition, processing, and querying Qdrant.

Results of the implementation and future applications

We found that by combining n8n and Qdrant, it's possible to build an AI-powered QA system very easily.
This mechanism can be applied to a variety of uses.

Internal reviews such as Project Readiness Checklists

When launching a new service, you can easily perform checks by placing checklists and requirements on GitHub and providing Terrafrom code and tfstate as input.

Automating the internal help desk

It can also be used as a contact point (help desk) for inquiries to the company's IT department, etc.
It will be possible to automatically answer standard questions such as "How to set up a PC" and "How to apply to use software" 24 hours a day, 365 days a year.

summary

Through a proof-of-concept (PoC) combining n8n and Qdrant for vector search, we gained confidence that we could easily solve challenges in internal and external ticket management and help desk operations.
The workflows created in the Proof of Concept (PoC) can be directly used in production, and in the future, we envision expanding them to knowledge-sharing portals and incident analysis support tools.
SRG is looking for new team members.
If you are interested, please contact us here.