Guides
Build an LLM TDD Bot with Sublayer
Introduction
In this guide, we'll walk through step by step on how to create a bot with the Sublayer gem that takes your tests, any failures of those tests, and gets an LLM to write code to pass those tests.
If you'd like to follow along on the guide and with the video you can grab the code here on the "starting_point" branch: TDD Bot
The final code for this guide is available on the "main" branch: TDD Bot
Step 0 - The MakeTestsPass Command and includes
We're using Shopify's cli-kit
for this tutorial, and the main entry point for this program is the MakeTestsPass command.
The file is located at lib/tddbot/commands/make_tests_pass.rb
require 'tddbot'
module Tddbot
module Commands
class MakeTestsPass < Tddbot::Command
def call(_args, _name)
Sublayer::Tasks::MakeRspecTestsPassTask.new(implementation_file_path: _args[0], test_command: _args[1]).run
end
def self.help
"Have an LLM continually modify the implementation file until the test command passes successfully.\n
Usage: {{command:#{Tddbot::TOOL_NAME} make_tests_pass <implementation_file_path> \"<test_command>\"}}\n
Example: {{command:#{Tddbot::TOOL_NAME} make_tests_pass lib/my_class.rb \"rspec spec/my_class_spec.rb\"}}"
end
end
end
end
We've also included the folders and libraries needed in /lib/tddbot.rb
Most importantly in lines 3 and 4:
require 'sublayer'
require 'open3'
and lines 16-20
['generators', 'tasks', 'actions'].each do |subfolder|
Dir[File.join(ROOT, 'lib', 'tddbot', 'sublayer', subfolder, '*.rb')].each do |file|
require file
end
end
Step 1 - MakeRspecTestsPassTask
The first step is to create the Sublayer Task that's used in the MakeTestsPass command. It takes the implementation\_file\_path
and the test\_command
which correspond to the first and second arguments to the command line command.
What we do is perform a loop of:
- check if the tests pass
- if they do, we're done
- If they aren't, generate a new implementation to try to pass the tests
- Save that new implementation to the file
- Go back to step 1
This code is located at /lib/tddbot/sublayer/tasks/make_rspec_tests_pass_task.rb
module Sublayer
module Tasks
class MakeRspecTestsPassTask < Base
def initialize(implementation_file_path:, test_command:)
@implementation_file_path = implementation_file_path
@test_command = test_command
end
def run
loop do
stdout, stderr, status = Sublayer::Actions::RunTestCommandAction.new(test_command: @test_command).call
puts stdout
puts stderr
if status.exitstatus == 0
puts "All tests pass!"
return
end
modified_implementation = Sublayer::Generators::ModifiedImplementationToPassTestsGenerator.new(
implementation_file_contents: File.read(@implementation_file_path),
test_file_contents: File.read(@test_command.split(" ")[1]),
test_output: stdout
).generate
Sublayer::Actions::WriteFileAction.new(
file_contents: modified_implementation,
file_path: @implementation_file_path
).call
end
end
end
end
end
Step 2 - The Actions
The task uses two different Sublayer actions that we need to create: RunTestCommandAction
and WriteFileAction
.
lib/tddbot/sublayer/actions/run_test_command_action.rb
module Sublayer
module Actions
class RunTestCommandAction < Base
def initialize(test_command:)
@test_command = test_command
end
def call
stdout, stderr, status = Open3.capture3(@test_command)
[stdout, stderr, status]
end
end
end
end
lib/tddbot/sublayer/actions/write_file_action.rb
module Sublayer
module Actions
class WriteFileAction < Base
def initialize(file_contents:, file_path:)
@file_contents = file_contents
@file_path = file_path
end
def call
File.open(@file_path, 'wb') do |file|
file.write(@file_contents)
end
end
end
end
end
Step 3 - The Generator
Finally, we need to take the output of the test run, the test code, and the implementation file and generate the new code to try to pass the tests.
That's accomplished with this generator below:
lib/tddbot/sublayer/generators/modified_implementation_to_pass_tests_generator.rb
module Sublayer
module Generators
class ModifiedImplementationToPassTestsGenerator < Base
llm_output_adapter type: :single_string,
name: "modified_implementation",
description: "The modified implementation that will pass the tests"
def initialize(implementation_file_contents:, test_file_contents:, test_output:)
@implementation_file_contents = implementation_file_contents
@test_file_contents = test_file_contents
@test_output = test_output
end
def generate
super
end
def prompt
<<-PROMPT
You are an expert in debugging and test resolution.
You have the current implementation, the tests, and the latest failure information at your disposal.
Your task is to modify the existing implementation using the implementation file content: #{@implementation_file_contents},
the test file content: #{@test_file_contents},
and the latest test output: #{@test_output},
to ensure that the tests will pass.
Approach this task with careful analysis and methodical thinking.
PROMPT
end
end
end
end
Step 4 - Run the bot!
After all this you should be where we are in the video when we run the bot.
The only thing to do now is to install the gem you've created and try to run the bot:
$ bundle install
$ gem build tddbot.gemspec
$ gem install ./tddbot-0.0.1.gem
$ tddbot make_tests_pass {YOUR IMPLEMENTATION FILE} {YOUR TEST COMMAND}