Run LLM Models Locally with Llamafile

  1. Install Llamfile
  2. Download the model
  3. Run the model with Llamafile
  4. Use with Sublayer
  5. Basic Demo

Install Llamafile 

  1. git clone git@github.com:Mozilla-Ocho/llamafile.git

  2. brew install make

  3. cd llamafile
gmake -j8
sudo gmake install PREFIX=/usr/local

Download the model

Run the model 

llamafile -ngl 9999 -m path/to/model.gguf --host 0.0.0.0 -c 2048
  • Recommended settings for Apple M1 users: 
    llamafile -ngl 9999 -m path/to/model.gguf --host 0.0.0.0 -c 2048 --gpu APPLE -t 12
  • visit localhost:8080

Use with Sublayer (skip to Basic Demo if you don’t have a project)

  1. Add to Gemfile: 
     gem 'sublayer', '~>0.0.7'
  2. Run: 
     bundle install
  3. Add to your configuration file: 
     Sublayer.configuration.ai_provider = Sublayer::Providers::Local
 Sublayer.configuration.ai_model = "LLaMA_CPP"
  4. Build a sublayer generator:
  5. Use in your code: 
     MyGenerator.new(attributes).generate

Basic Demo

Let’s make a ruby project to find a past historical event on today’s date

# bash
mkdir historical_event_finder
cd historical_event_finder
touch Gemfile
touch historical_event_finder.rb

# Gemfile
source 'https://rubygems.org'
gem 'sublayer', '~>0.0.7'

# bash
bundle install
  • Build a sublayer generator with the following description:
    • “generator that uses Time.now and finds a fun historical event from the past that occurred on the same month/day as a value”

  • Paste the result from above into historical_event_generator.rb (rename if needed)

  • Write the following code in historical_event_finder.rb:

    # historical_event_finder.rb
require 'sublayer'
require_relative 'historical_event_generator'

Sublayer.configuration.ai_provider = Sublayer::Providers::Local
Sublayer.configuration.ai_model = "LLaMA_CPP"

puts HistoricalEventGenerator.new.generate

  • run your code:

    ruby historical_event_finder.rb