Run Llama3.1 locally with Ollama

Install Ollama
Download and run the model
Adding a custom Provider
Basic Demo
Building a custom Provider

Install Ollama

Visit https://ollama.com/ and download ollama

Download and run llama3.1

ollama run llama3.1

Adding a Ollama Llama3.1 Provider

To add a provider to an existing project:

Add to Gemfile:
```
 gem 'sublayer', '~>0.2.0'
```
Run:
```
 bundle install
```

Add the provider:

 module Sublayer
   module Providers
     class OllamaLlama31
       def self.call(prompt:, output_adapter:)
         response = HTTParty.post(
           "http://localhost:11434/api/chat",
           body: {
             "model": 'llama3.1',
             "messages": [
               {
                 "role": "user",
                 "content": prompt
               }
             ],
             "stream": false,
             "tools": [
               {
                 "type": "function",
                 "function": {
                   "name": 'response',
                   "parameters": {
                     "type": "object",
                     "properties": output_adapter.format_properties,
                     "required": output_adapter.format_required
                   }
                 }
               }
             ]
           }.to_json
         )

         message = response['message']

         raise "No function called" unless message["tool_calls"].length > 0

         function_body = message.dig("tool_calls", 0, "function", "arguments")
         function_body[output_adapter.name]
       end
     end
   end
 end

Add to your configuration file:

 Sublayer.configuration.ai_provider = Sublayer::Providers::OllamaLlama31

Build a sublayer generator:

Use in your code:
```
 MyGenerator.new(attributes).generate
```

Full Demo

Let’s make a ruby project to generate Dr. Seuss styled short poems

# bash
mkdir dr_seuss_poem_maker
cd dr_seuss_poem_maker
touch Gemfile
touch ollama_llama_3_1.rb
touch dr_seuss_poem_generator.rb
touch dr_seuss_poem_maker.rb

# Gemfile
source 'https://rubygems.org'
gem 'sublayer', '~>0.2.0'

# bash
bundle install

Build a sublayer generator with the following description:
- “generator that writes poems in the style of Dr. Seuss”

Paste the result from above into dr_seuss_poem_generator.rb (rename if desired)

Paste the following Provider into your project (to see how to build this or any custom provider go to Build a Custom Provider)

class OllamaLlama31
  def self.call(prompt:, output_adapter:)
    response = HTTParty.post(
      "http://localhost:11434/api/chat",
      body: {
        "model": 'llama3.1',
        "messages": [
          {
            "role": "user",
            "content": prompt
          }
        ],
        "stream": false,
        "tools": [
          {
            "type": "function",
            "function": {
              "name": 'response',
              "parameters": {
                "type": "object",
                "properties": output_adapter.format_properties,
                "required": output_adapter.format_required
              }
            }
          }
        ]
      }.to_json
    )

    message = response['message']

    raise "No function called" unless message["tool_calls"].length > 0

    function_body = message.dig("tool_calls", 0, "function", "arguments")
    function_body[output_adapter.name]
  end
end

Write the following code in dr_seuss_poem_maker.rb:

# dr_seuss_poem_maker.rb
require 'sublayer'
require_relative 'dr_seuss_poem_generator'
require_relative 'ollama_llama_3_1'

Sublayer.configuration.ai_provider = OllamaLlama31

puts DrSeussPoemGenerator.new.generate

run your code:
```
ruby dr_seuss_poem_maker.rb
```

Building a Custom Provider

Sublayer Gem comes with httparty
Add a class method .call with two named parameters: prompt: and output_adapter:
Find the url endpoint you need for llm chat completions
Find the request format for function calling
If no function calling exists click here to find an example provider of custom response formatting
Take the response and parse for the desired value