About SRGArticlesRecruitment Information
About SRGArticlesRecruitment Information
HOME/Articles/Vector search with n8n and Qdrant was easy

Vector search with n8n and Qdrant was easy

This is Hasegawa (@rarirureluis) from the Service Reliability Group (SRG) of the Media Headquarters.
#SRG(Service Reliability Group) mainly provides cross-sectional support for the infrastructure of our media services, improving existing services, launching new ones, contributing to OSS, etc.
In this article, we will share our findings from conducting a "vector search PoC (Proof of Concept) for in-house systems/ticket management" using the workflow automation tool n8n and the vector search engine Qdrant, which revealed that it appears to be easier and more versatile than we had imagined.

Introduction

When responding to internal inquiries or managing tickets, it is often difficult to instantly access past knowledge, which can be time-consuming and costly.
Through this PoC of vector search, we verified the effectiveness of "instant search for similar tickets," "automatic FAQ answers," and "related document recommendations."
The biggest benefit of using the combination of n8n and Qdrant was that it enabled us to quickly build a workflow that could be used by both engineers and operators.
Although I have no knowledge of vectors, I proceeded with the project with the help of various advice from the AI.

Tools used this time

This system was built mainly using the following tools:

n8n

n8n is a platform for automating workflows by linking various applications and APIs.
It features a node-based visual editor that allows you to intuitively create processing flows even if you have little programming knowledge.
With over 400 integrations available and the option to self-host, you can handle your data flexibly and securely.

Qdrant

Qdrant is a database engine specialized for vector searching.
We specialize in quickly searching and managing AI-generated vector data (embedding).
Developed in Rust, it boasts high performance and reliability.
In addition to similarity searches, you can also associate and save JSON-formatted payloads as vectors and filter using a variety of conditions, such as keywords and numerical ranges.
This allows you to implement complex business logic beyond simple similarity searches.

GitHub and CSV

The question and answer pairs that formed the source data for the QA were created in CSV (Comma-Separated Values) format.
CSV is a simple text file with values separated by commas, which can be easily handled by many tools.
We decided to manage this CSV file in a GitHub repository to facilitate version control and collaborative editing.

Building a simple QA system

The overall system is very simple.
  1. Upload the CSV file containing the QA data to GitHub.
  1. Build the Qdrant environment and create a collection (a place to store data).
  1. Create a workflow in n8n to pull CSV data from GitHub and ingest it into Qdrant.
  1. When a user submits a question, n8n vectorizes the question and queries Qdrant to search for answers with similarities.
  1. The search results are returned to the user.
This series of steps was implemented as the n8n workflow.
By simply connecting n8n nodes, we were able to easily automate processes such as data acquisition, processing, and queries to Qdrant.

Results of the construction and future use

We found that by combining n8n and Qdrant, it is very easy to build an AI-based QA system.
This mechanism can be applied to a variety of purposes.

Internal reviews such as production readiness checklists

When launching a new service, you can put a checklist or things that need to be complied with on GitHub, and then just provide the Terraform code and tfstate as input to easily run the checks.

Internal help desk automation

It can also be used as a help desk for inquiries to the company's information systems department, etc.
It will be possible to automatically answer routine questions such as "How to set up a PC" or "How to apply to use software" 24 hours a day, 365 days a year.

summary

By combining n8n and Qdrant in our vector search PoC, we were able to see that it would be possible to easily solve issues related to ticket management and help desks both inside and outside the company.
The workflow created in the PoC can be used as is in production, and in the future it could also be expanded into a knowledge sharing portal or a fault analysis support tool.
SRG is looking for people to work with us.
If you are interested, please contact us here.