About SRGTechnical articleRecruitment Information
About SRGTechnical articleRecruitment Information
HOME/Technical article/My first local LLM

My first local LLM

This is Aoyama from the Service Reliability Group (SRG) of the Media Division.
#SRGThe Service Reliability Group primarily provides comprehensive support for the infrastructure surrounding our media services, focusing on improving existing services, launching new ones, and contributing to open-source software (OSS).
 
This article is aboutCyberAgent Group SRE Advent Calendar 2024This is the eighth article in this series. It summarizes questions I had, starting with "What is a local LLM?". I hope it will be helpful for those considering implementing a local LLM.
 

What is a local LLM?

A local LLM is a large-scale language model (LLM) that can run in a local environment, such as on your own computer or server. While LLMs like ChatGPT typically run on the cloud, local LLMs have the following characteristics:
  1. Runnable in a local environment You can download the model and run it on your PC or server. It may even work without an internet connection.
  1. Privacy Protection Because we do not send data to the cloud, you can rest assured when handling confidential and personal information.
  1. Customizable You can fine-tune the model to suit your needs and train it with specific data.
  1. Cost savings By not using commercial cloud services, you can avoid usage-based billing. However, if high-performance hardware is required, that cost will apply.
 

Examples of using a local LLM

  • AI app development as a personal project
  • Data analysis and tool development within companies
  • AI use in offline environments
While some technical knowledge is required for implementation, local LLM is a viable option for those who want to leverage advanced AI technology while respecting privacy.
 

Model types

Some representative models that can be used when building an LLM (Large-Scale Language Model) locally include the following:
 
  1. LLaMA（Large Language Model Meta AI）Developed by Meta (formerly Facebook), the latest AI model, "Llama3," was released in April 2024. Its use is recommended for research purposes.
  1. BERT（Bidirectional Encoder Representations from Transformers）BERT is a deep learning model for natural language processing developed by Google. BERT can perform language comprehension tasks and understand the meaning and context of sentences.
  1. BLOOM:
      2. This is an open-source, multilingual, large-scale language model developed by BigScience, a collaborative AI research workshop.
  1. phi-3Phi-3 is a high-performance, open-source SLM (Small-Scale Language Model) developed by Microsoft. It is said to offer significantly better performance than language models with the same parameter size.
 
These models can be easily used through libraries such as Hugging Face's Transformers library. Local deployment and customization are also relatively easy.
 

What is hallucination in LLM?

 
 
One problem in Large Language Models (LLMs) is "hallucination." This refers to the phenomenon where a model generates facts or information that do not actually exist. This means that the model provides incorrect answers with a high degree of confidence for a given input.
 
For example, if you ask microsoft/Phi-3-mini-4k-instruct-gguf about the recipe for "pain d'épices," a French Christmas pastry, you will get the following response:
 
 
☆Example of an actual pain d'épices recipe
 

RAG (Retrieval-Augmented Generation)

One technique to avoid this "hallucination" is called RAG (Retrieval-Augmented Generation).
This refers to an approach that combines generative AI (especially large-scale language models, LLMs) with information retrieval capabilities. RAG is a mechanism in which a model retrieves information from external knowledge sources (e.g., databases, documents, web pages, etc.) and uses that information to generate responses and text.
 

Trying out a local LLM (Microcosm beta version)

I decided to try out the beta version of "Microcosm," a local AI application that was just released in November 2024.
 
Thanks to the comprehensive Japanese manual, you can complete the installation and model setup without any difficulty.
 
 
Selecting model microsoft/Phi-3-mini-4k-instruct-gguf and asking a question about "Pandepis" will cause "hallucination."
 
To resolve this, please register your RAG. You can register your data from the menu on the right side.
Since it's possible to select a directory, this time I'll register some PDF files of recipes I found on the web and some PDF files of content from Wikipedia.
 
This is the RAG data created after importing.
You can delete, edit, or add new data from the screen.
 
 
Apply RAG from the side menu on the home screen.
 
When I ask the same question about "pain d'épices," I now get much more accurate answers.
 
 
 

In conclusion

I did a quick search on local LLMs and RAGs, which can be used in situations where you want to use an LLM for business purposes but don't want to send data to the cloud.
 
We would like to express our sincere gratitude to Number One Solutions Co., Ltd. for kindly granting us permission to use images from the beta version of "Microcosm" and to link to its manual in the creation of this article.
 
SRG is looking for new team members. If you are interested, please contact us here.