Testing CockroachDB

Learn how to run tests, understand test infrastructure, and write effective tests

CockroachDB has a comprehensive test suite covering unit tests, integration tests, logic tests, and end-to-end tests. The ./dev test command provides a unified interface for running tests.

Running Tests

Basic Test Commands

Use the ./dev test command to run tests:

Package Tests

# Run all tests in a package
./dev test pkg/sql

When filtering tests with -f, always include the -v flag. This warns you if your filter didn’t match anything. Look for testing: warning: no tests to run in the output.

Common Test Scenarios

Running Fast Unit Tests

# Run tests in a specific package (faster than full package)
./dev test pkg/util/log -f=TestFormat -v

# Run tests in multiple packages
./dev test pkg/util/log pkg/util/timeutil

Running Slow Integration Tests

Some packages have slow tests (SQL tests can take 10+ minutes):

# Be patient - SQL tests are comprehensive
./dev test pkg/sql

# Use filters to run subset
./dev test pkg/sql -f=TestLogic/local/select -v

The KV and SQL packages have particularly comprehensive test suites. Use filters to run specific tests during development.

Running Logic Tests

Logic tests are SQL correctness tests:

# Run all logic tests
./dev test pkg/sql/logictest

# Run specific logic test file
./dev test pkg/sql/logictest -f=TestLogic/local/select

Stress Testing

Detect race conditions and flaky tests:

# Run test 100 times looking for failures
./dev test pkg/kv --count=100 -f=TestRangefeed

# Use stress tool for more control
stress ./dev test pkg/kv -f=TestRangefeed --count=1

Test Options

View all available options:

./dev test --help

Verbose Output

./dev test pkg/sql -v

Shows detailed test output including test names and logs.

Short Mode

./dev test pkg/sql --short

Skips slow tests (those marked with testing.Short()).

Race Detector

./dev test pkg/kv --race

Enables Go’s race detector (much slower but catches concurrency bugs).

Test Coverage

./dev test pkg/util/log --cover

Generates test coverage reports.

Test Infrastructure

Test Types

CockroachDB uses several types of tests:

1. Unit Tests

   Standard Go tests in *_test.go files:

   func TestFeature(t *testing.T) {
       // Test implementation
   }
   

   Located alongside the code they test.

2. Logic Tests

   SQL correctness tests in pkg/sql/logictest/testdata/:

   • Test SQL semantics and correctness
   • Compare output against expected results
   • Run in various configurations (local, distributed, multi-region)
3. Data-Driven Tests

   Tests using the datadriven framework:

   • Test data in testdata/ directories
   • Allows easy addition of test cases
   • Common in SQL optimizer and KV tests
4. Roachtests

   Large-scale integration tests in pkg/cmd/roachtest/:

   • Run on real clusters
   • Test distributed scenarios
   • Performance and correctness at scale
   • Typically run in CI, not locally

Test Utilities

CockroachDB provides testing utilities in pkg/testutils/:

import (
    "github.com/cockroachdb/cockroach/pkg/testutils"
    "github.com/cockroachdb/cockroach/pkg/testutils/sqlutils"
    "github.com/cockroachdb/cockroach/pkg/testutils/serverutils"
)

Test Server

Create lightweight test clusters:

s, sqlDB, _ := serverutils.StartServer(t, base.TestServerArgs{})
defer s.Stopper().Stop(ctx)

SQL Utilities

Execute SQL in tests:

sqlutils.MakeSQLRunner(sqlDB).Exec(t, "CREATE TABLE t (id INT)")

Assertions

Common test assertions:

testutils.SucceedsSoon(t, func() error {
    // Check condition
})

Cleanup

Automatic cleanup:

tempDir := testutils.TempDir(t)
// Automatically cleaned up

Code Generation

Many CockroachDB components rely on generated code:

What Gets Generated?

Protocol Buffers

.proto files → .pb.go files

Used for serialization and RPC.

SQL Parser

Grammar files → Parser code

Generates the SQL parser.

SQL Optimizer

Optgen rules → Optimizer code

Generates optimizer transformations.

Stringer

Type definitions → String methods

Generates String() methods for enums.

Generating Code

Use ./dev generate to regenerate code:

# Generate all code (SLOW - can take 10+ minutes)
./dev generate

# Generate only Go code
./dev generate go

# Generate only protocol buffers (relatively fast)
./dev generate protobuf

# Update BUILD.bazel files when dependencies change
./dev generate bazel

Code generation is slow. Only regenerate what you actually need.

./dev test and ./dev build automatically generate dependencies but don’t lift them into the worktree. If you need to see generated code, run ./dev generate explicitly.

When to Regenerate

Regenerate code when you modify:

• .proto files (protobuf definitions)
• sql.y or sql.bnf (SQL grammar)
• *.opt files (optimizer rules)
• Go dependencies (run ./dev generate bazel)
• Stringer annotations

Writing Effective Tests

Best Practices

1. Test at the Right Level

   • Use unit tests for logic and edge cases
   • Use integration tests for component interaction
   • Use logic tests for SQL correctness
   • Use roachtests for distributed scenarios
2. Make Tests Deterministic

   Avoid flaky tests:

   // Bad: timing-dependent
   time.Sleep(100 * time.Millisecond)
   
   // Good: condition-based
   testutils.SucceedsSoon(t, func() error {
       if !conditionMet() {
           return errors.New("condition not met")
       }
       return nil
   })
   
3. Use Testdata Files

   For complex test cases, use data-driven tests:

   testdata/
     my_feature
     another_case
   

   Easier to add cases and review changes.

4. Clean Up Resources

   Always clean up:

   s := serverutils.StartServer(t, base.TestServerArgs{})
   defer s.Stopper().Stop(ctx)
   
5. Test Error Cases

   Don’t just test the happy path:

   // Test error conditions
   _, err := fn(invalidInput)
   require.Error(t, err)
   require.Contains(t, err.Error(), "expected error message")
   

SQL Logic Tests

When adding SQL features, add logic tests:

# testdata/logic_test/my_feature

statement ok
CREATE TABLE t (id INT PRIMARY KEY, name STRING)

query IT rowsort
SELECT * FROM t
----

statement ok
INSERT INTO t VALUES (1, 'alice'), (2, 'bob')

query IT rowsort
SELECT * FROM t
----
1  alice
2  bob

Continuous Integration

CI Test Runs

When you open a PR, several CI checks run:

• Unit Tests: All package tests
• Logic Tests: SQL correctness tests
• Roachtests: Distributed integration tests (subset)
• Linters: Code style and quality checks
• Build Tests: Ensure code builds on all platforms

Handling CI Failures

Test Failures

If tests fail in CI:

1. Reproduce locally: ./dev test pkg/failing/test -f=TestName -v
2. Fix the issue
3. Verify fix: ./dev test pkg/failing/test --count=10
4. Push the fix

Flaky Tests

If a test fails intermittently:

1. Report it by opening an issue
2. Tag with A-flaky-test
3. Include CI logs and failure details
4. Consider adding stress testing

Lint Failures

Fix formatting and style issues:

# Format your code
crlfmt -w -tab 2 your_file.go

# Regenerate if needed
./dev generate bazel

Debugging Tests

Common Debugging Techniques

Verbose Output

# See detailed test output
./dev test pkg/kv -f=TestRangefeed -v

Next Steps

Building from Source

Learn how to build CockroachDB

Code Structure

Understand the codebase organization

Contributing Guide

Read the full contribution guidelines

Test Engineering

Advanced testing documentation on wiki