About SRGTechnical ArticlesRecruitment Information
About SRGTechnical ArticlesRecruitment Information
HOME/Technical Articles/Vector search using n8n and Qdrant was easy

Vector search using n8n and Qdrant was easy

This is Hasegawa (@rarirureluis) from the Service Reliability Group (SRG) of the Media Headquarters.
#SRG(Service Reliability Group) is a group that mainly provides cross-sectional support for the infrastructure of our media services, improving existing services, launching new ones, and contributing to OSS.
In this article, we will share our findings from conducting a "vector search PoC (Proof of Concept) for in-house systems/ticket management" using the workflow automation tool n8n and the vector search engine Qdrant, which revealed that it appears to be easier and more versatile than we had imagined.

Introduction

When responding to internal inquiries or managing tickets, it is often time-consuming and costly to not be able to instantly access past knowledge.
This time, through the PoC of vector search, we verified the effectiveness of "instant search for similar tickets," "automatic FAQ answers," and "recommendation of related documents."
The biggest benefit of combining n8n and Qdrant was that it allowed us to quickly build a workflow that could be used by both engineers and operators.
Although I have no knowledge of vectors, I proceeded with the help of various advice from the AI.

Tools used this time

This system was built mainly using the following tools:

n8n

n8n is a platform for automating workflows by connecting various applications and APIs.
It features a node-based visual editor that allows you to intuitively assemble processing flows even if you have little programming knowledge.
With over 400 integrations available and the option to self-host, you can handle your data flexibly and securely.

Qdrant

Qdrant is a database engine specialized for vector searching.
We specialize in high-speed search and management of AI-generated vector data (embedding).
Developed in Rust, it boasts high performance and reliability.
In addition to similarity searches, you can also associate and save JSON-formatted payloads as vectors, and filter them using a variety of conditions, such as keywords or numerical ranges.
This allows you to implement complex business logic beyond simple similarity searches.

GitHub and CSV

The question and answer pairs that served as the source data for the QA were created in CSV (Comma-Separated Values) format.
CSV is a simple text file with values separated by commas, which can be easily handled by many tools.
We decided to manage this CSV file in a GitHub repository to facilitate version control and collaborative editing.

Building a simple QA system

The overall system is very simple.
  1. Upload the CSV file containing the QA data to GitHub.
  1. Build the Qdrant environment and create a collection (a location to store data).
  1. Create a workflow in n8n to retrieve CSV data from GitHub and input it into Qdrant.
  1. When a user asks a question, n8n vectorizes the question and queries Qdrant to search for answers with high similarity.
  1. The search results are returned to the user.
This series of steps was implemented as the n8n workflow.
By simply connecting n8n nodes, we were able to easily automate processes such as data acquisition, processing, and queries to Qdrant.

Results of the construction and future use

We found that by combining n8n and Qdrant, it is very easy to build an AI-based QA system.
This mechanism can be applied to a variety of purposes.

Internal reviews such as production readiness checklists

When launching a new service, you can put a checklist and the things that must be complied with on GitHub, and then simply provide the Terrafrom code or tfstate as input to easily check them.

Internal help desk automation

It can also be used as a help desk for inquiries to the company's information systems department, etc.
It will be possible to automatically answer routine questions such as "How to set up a PC" and "How to apply to use software" 24 hours a day, 365 days a year.

summary

The vector search PoC, which combined n8n and Qdrant, demonstrated its potential to easily solve issues related to ticket management and help desks both inside and outside the company.
The workflow created in the PoC can be used in production as is, and in the future it could also be expanded into a knowledge sharing portal or a fault analysis support tool.
SRG is looking for people to work with us.
If you are interested, please contact us here.