How To: Running the Small Language Model ‘Phi3’ Locally

I had been getting requests from people looking to understand how or if language models could be run locally.

As far back as 5 years ago, during scoping calls, clients would ask if services such as LUIS (Language Understanding Intelligence Service) could be run on-prem in areas with poor cell phone signal.  The use case was a manufacturing plant where tablets were in use, and they wanted to use voice and natural language to interact with a model.

The technology at the time didn’t really support it.

There are many reasons why you might want to run a language model on your own infrastructure or devices.

Some of these might include:

• Data protection
• IP concerns
• Performance

 

With the advent of small language models (SLM’s), the technology now exists to support the running of language models on local devices, edge devices or IoT devices, and your local infrastructure.  It’s economical too.

 

Bruno Capuano recently published a blog post that detailed how to use the SLM Phi-3 on your local machine using the ONNX Runtime.  I’m adding this code here for my own personal reference.

~

What is the ONNX Runtime

The ONNX Runtime is an open-source scoring engine for machine learning models in the Open Neural Network Exchange (ONNX) format.

ONNX is a format for representing machine learning models that allows them to be easily transferred between different frameworks.

The ONNX Runtime offers cross platform support. Recently support for the runtime has been made available to .NET developers via a new API Microsoft.ML.OnnxRuntimeGenAI.

In short, this lets you run SLM, or LLM on a device such as your local machine or edge device.

Learn more about the Microsoft.ML.OnnxRuntimeGenAI API here.

Learn more about the ONNX Runtime here.

~

Downloading the Model

Various flavours of Phi-3 are available. I was most interested in the mobile and cpu versions.  At the time of writing, support for cuda, and directml is not supported by the Microsoft.ML.OnnxRuntimeGenAI API.

The model is optimised for the ONNX runtime.

 

Within the cpu_and_mobile folder, there are 2 sub folders.

• cpu_and_mobile/cpu-int4-rtn-block-32-acc-level-4
• cpu_and_mobile/cpu-int4-rtn-block-32

 

You might be wondering what this mean.

ONNX model for CPU and mobile using int4 quantization via RTN. There are two versions uploaded to balance latency vs. accuracy. With accuracy level = 1 and accuracy level = 4. If better performance with a minor trade-off in accuracy (for example on mobile devices), we recommend using the model with acc-level-4.

 

My laptop has limited resources, so I opted for the model that uses acc-level-4.

~

Loading the Model

After you have downloaded the mode, you can load it in code.  Here we point the path to the root folder for the model and load it:

var modelPath = "E:\\Source\\Models\\Phi-3-mini-4k-instruct-onnx\\cpu_and_mobile\\cpu-int4-rtn-block-32-acc-level-4";

var model = new Model(modelPath);

var tokenizer = new Tokenizer(model);

~

Setting the Meta Prompt

It can be helpful to prime the agent with a system prompt.  We can see this here:

var metaPrompt = "You are an AI assistant that helps people find information. Answer questions using a direct style. Do not share more information than the requested by the human.";

~

Invoking the Agent

The following code invokes the agent (thanks Bruno!):

while (true)
{

    // Get user question
    Console.Write("How can I help you?: ");

    var humanQuestion = Console.ReadLine();
    if (string.IsNullOrEmpty(humanQuestion))
    {
         break;
     }


      // show phi3 response
      Console.Write("Agent: ");

      var fullPrompt = $"<|system|>{systemPrompt}<|end|><|user|>{humanQuestion}<|end|><|assistant|>";
      var tokens = tokenizer.Encode(fullPrompt);
      var generatorParams = new GeneratorParams(model);

      generatorParams.SetSearchOption("max_length", 2048);
      generatorParams.SetSearchOption("past_present_share_buffer", false);
      generatorParams.SetInputSequences(tokens);

      var generator = new Generator(model, generatorParams);

      while (!generator.IsDone())
      {
          generator.ComputeLogits();
          generator.GenerateNextToken();

          var outputTokens = generator.GetSequence(0);
          var newToken = outputTokens.Slice(outputTokens.Length - 1, 1);
          var output = tokenizer.Decode(newToken);

          Console.Write(output);
      }

      Console.WriteLine();
}

 

 

After running the console application, we can interact with the Phi-3 agent:

That’s it.  A simple agent, in less than 100 lines of code.

~

Summary

The main thing I like about this approach is that you don’t need use an additional tool such as Ollama or LMStudio to interact with the model.  You could easily stick this behind and API or web service.

In a future blog post, we will see how to implement a 100% local RAG solution.  This is a very hot topic right now.

~

Further Reading and Resources

You can learn more about Phi-3 and the ONNX Runtime here:

• ONNX Runtime – ONNX Runtime | Home
• Phi-3 Download via Huggingface – microsoft/Phi-3-mini-4k-instruct-onnx
• Phi-3 Stats – Enjoy the Power of Phi-3 with ONNX Runtime on your device

 

Enjoy what you’ve read, have questions about this content, or would like to see another topic? Drop me a note below.

You can schedule a call using my Calendly link to discuss consulting and development services.