Younes Bouchbouk | Blog

Why Golang Is Ideal for Microservices Architecture (2026 and Beyond)

Sat, 07 Feb 2026 00:00:00 GMT

Why Golang Is Ideal for Microservices Architecture (2026 and Beyond)

As we navigate the tech landscape of 2026, microservices remain the dominant architectural pattern for scalable applications. Among the languages vying for dominance in this space, Go (Golang) stands out as the pragmatic, high-performance choice.

Here’s why Go continues to be the best language for building microservices today.

1. Unmatched Performance and Efficiency

Microservices often run in containerized environments where resources directly translate to cost. Go binaries are statically linked and extremely lightweight—often just a few megabytes.

Fast Startup: Go applications start almost instantly. This is critical for auto-scaling environments (like Kubernetes or Knative) where services need to scale from zero to handle traffic spikes in milliseconds.
Low Memory Footprint: Compared to JVM-based languages, Go services consume a fraction of the memory, allowing you to pack more services onto the same hardware.

2. Built-in Concurrency That actually Works

Concurrency isn't an afterthought in Go; it's a core feature.

Goroutines: Instead of heavy OS threads, Go uses lightweight goroutines. You can spawn thousands of them to handle concurrent requests without exhausting system resources.
Channels: Go’s channel primitives make it safe and easy to communicate between concurrent processes, avoiding the "callback hell" or complex locking mechanisms found in other languages.

In a microservices world where a single user request might fan out to call ten other services, this efficient concurrency model is a game-changer.

3. Simplicity at Scale

One of Go’s biggest strengths is what it doesn't have. The language is intentionally simple.

Readability: Go code is uniform and easy to read. A new developer can jump into a microservice written by someone else and understand it quickly.
Maintainability: In a microservices architecture with hundreds of repositories, consistency is key. Go's strict formatting (gofmt) and lack of "magic" ensure that all your services look and behave similarly.

4. Cloud-Native DNA

Go is the language of the cloud. Kubernetes, Docker, Prometheus, Terraform, and many other foundational cloud-native tools are written in Go.

First-Class Support: The ecosystem for cloud-native development in Go is mature and battle-tested.
Excellent Tooling: Libraries for gRPC, metrics, tracing (OpenTelemetry), and logging are robust and performant.

5. The Future (2026 and Beyond)

Go hasn't stood still. With the addition of generics and continuous runtime optimizations, it has become even more powerful without losing its simplicity.

As edge computing and serverless platforms continue to grow in 2026, Go's characteristics—fast startup, small binaries, and high performance—make it the perfect fit for the next generation of distributed systems.

Conclusion

For teams building scalable, maintainable, and cost-effective microservices, Go offers an unbeatable combination of performance and developer productivity. It strips away the complexity of traditional enterprise languages and provides exactly what you need to build robust distributed systems.

If you're architecting for the future, Go is the solid foundation you should build on.

Disclaimer: The views expressed in this article are my own and stem from my personal experience working with fast-growing startups that rely on Go as their primary backend language for microservices architecture.

Happy coding! 🚀

Mastering Context in Go: From Basics to Production-Grade Patterns

Wed, 31 Dec 2025 00:00:00 GMT

Mastering Context in Go: From Basics to Production-Grade Patterns

If you've worked with Go for more than a week, you've seen context.Context everywhere: HTTP handlers, database calls, gRPC methods, and pretty much any I/O operation. But why does context exist, and how do you use it effectively in production microservices?

In this post, I'll walk you through:

What problem context solves (and what it doesn't)
The three core use cases: cancellation, timeouts, and value propagation
A practical implementation with real code and tests
Production patterns I've used in Go microservices

By the end, you'll understand not just how to pass context around, but when and why it matters.

The problem: coordinating lifecycle across goroutines

Imagine a typical web request in a microservice:

HTTP handler receives request
Spawns goroutines to query multiple services in parallel
Aggregates results and returns response

What happens if:

The client disconnects before the response is ready?
The request times out before all goroutines finish?
You need to pass request-scoped data (trace ID, user ID) across function boundaries?

Without context, you'd need to:

Manually pass cancellation signals through channels
Track timeouts with separate timers
Thread metadata through every function signature

Context solves all three problems with a single, idiomatic interface.

What is context.Context?

context.Context is an interface that carries:

Cancellation signals: When work should stop
Deadlines: When work must stop
Request-scoped values: Metadata that travels with the request

type Context interface {
    Deadline() (deadline time.Time, ok bool)
    Done() <-chan struct{}
    Err() error
    Value(key interface{}) interface{}
}

Key insights:

Immutable: Once created, a context never changes (you derive new contexts)
Safe for concurrent use: Multiple goroutines can read from the same context
Forms a tree: Child contexts inherit from parents but can have tighter constraints

Use Case 1: Cancellation (stop work when it's no longer needed)

When a client disconnects or upstream work is cancelled, you want downstream work to stop immediately—not waste CPU and keep connections open.

Example: Simulate cancellable work

func SimulateWork(ctx context.Context, workDuration time.Duration, description string) error {
    timer := time.NewTimer(workDuration)
    defer timer.Stop()

    select {
    case <-ctx.Done():
        // Context was cancelled—stop immediately
        return ctx.Err()
    case <-timer.C:
        // Work completed successfully
        return nil
    }
}

Why this matters in production

In a real HTTP handler:

func SearchHandler(w http.ResponseWriter, r *http.Request) {
    ctx := r.Context() // Inherits cancellation from HTTP server

    // If the client closes the connection, ctx.Done() fires automatically
    results, err := performExpensiveSearch(ctx, query)
    if err != nil {
        // Could be context.Canceled if client disconnected
        http.Error(w, err.Error(), http.StatusInternalServerError)
        return
    }
    
    json.NewEncoder(w).Encode(results)
}

Without context, the expensive search would run to completion even if the client already left.

Use Case 2: Timeouts (fail fast instead of hanging forever)

Database calls, HTTP requests, and RPCs can hang. Timeouts prevent one slow dependency from blocking your entire service.

Example: Context with timeout

func FetchWithTimeout(url string, timeout time.Duration) (string, error) {
    ctx, cancel := context.WithTimeout(context.Background(), timeout)
    defer cancel()

    req, err := http.NewRequestWithContext(ctx, "GET", url, nil)
    if err != nil {
        return "", err
    }

    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        // Could be context.DeadlineExceeded
        return "", err
    }
    defer resp.Body.Close()

    body, _ := io.ReadAll(resp.Body)
    return string(body), nil
}

Why `defer cancel()` matters

Even if the request completes successfully, you must call cancel() to release resources (timers, goroutines) associated with the context.

Best practice: Always defer cancel() immediately after creating a cancellable or timeout context.

Use Case 3: Value propagation (request-scoped metadata)

Sometimes you need to pass data across function boundaries without polluting every function signature: trace IDs, user IDs, request IDs, authentication tokens.

Example: Adding and retrieving values

func AddValue(parent context.Context, key, value interface{}) context.Context {
    return context.WithValue(parent, key, value)
}

func GetValue(ctx context.Context, key interface{}) (interface{}, bool) {
    val := ctx.Value(key)
    if val == nil {
        return nil, false
    }
    return val, true
}

In production: structured logging with trace IDs

type contextKey string

const (
    traceIDKey   contextKey = "traceID"
    requestIDKey contextKey = "requestID"
)

func WithTraceID(ctx context.Context, traceID string) context.Context {
    return context.WithValue(ctx, traceIDKey, traceID)
}

func GetTraceID(ctx context.Context) (string, bool) {
    val := ctx.Value(traceIDKey)
    if val == nil {
        return "", false
    }
    traceID, ok := val.(string)
    return traceID, ok
}

Warning: Don't abuse context.WithValue for things that should be function parameters. Use it for request-scoped data that crosses abstraction boundaries (like observability metadata).

Practical implementation: ContextManager

Let me show you a simple ContextManager interface that wraps the common patterns. This is useful when building reusable libraries or when you want to standardize context handling across your codebase.

type ContextManager interface {
    // Create a cancellable context from a parent context
    CreateCancellableContext(parent context.Context) (context.Context, context.CancelFunc)

    // Create a context with timeout
    CreateTimeoutContext(parent context.Context, timeout time.Duration) (context.Context, context.CancelFunc)

    // Add a value to context
    AddValue(parent context.Context, key, value interface{}) context.Context

    // Get a value from context
    GetValue(ctx context.Context, key interface{}) (interface{}, bool)

    // Execute a task with context cancellation support
    ExecuteWithContext(ctx context.Context, task func() error) error

    // Wait for context cancellation or completion
    WaitForCompletion(ctx context.Context, duration time.Duration) error
}

Implementation

type simpleContextManager struct{}

func NewContextManager() ContextManager {
    return &simpleContextManager{}
}

func (cm *simpleContextManager) CreateCancellableContext(parent context.Context) (context.Context, context.CancelFunc) {
    return context.WithCancel(parent)
}

func (cm *simpleContextManager) CreateTimeoutContext(parent context.Context, timeout time.Duration) (context.Context, context.CancelFunc) {
    return context.WithTimeout(parent, timeout)
}

func (cm *simpleContextManager) AddValue(parent context.Context, key, value interface{}) context.Context {
    return context.WithValue(parent, key, value)
}

func (cm *simpleContextManager) GetValue(ctx context.Context, key interface{}) (interface{}, bool) {
    val := ctx.Value(key)
    if val == nil {
        return nil, false
    }
    return val, true
}

The tricky one: ExecuteWithContext

This is where context cancellation gets interesting. You want to run a task in a goroutine, but stop it if the context is cancelled:

func (cm *simpleContextManager) ExecuteWithContext(ctx context.Context, task func() error) error {
    ctx, cancel := cm.CreateCancellableContext(ctx)
    defer cancel()
    
    done := make(chan error, 1)
    
    // Run task in goroutine
    go func() {
        done <- task()
    }()

    // Race: who finishes first?
    select {
    case err := <-done:
        return err // Task completed first
    case <-ctx.Done():
        return ctx.Err() // Context cancelled/timeout first
    }
}

Key insight: The select statement races the task completion against context cancellation. Whichever happens first determines the outcome.

WaitForCompletion: A context-aware sleep

func (cm *simpleContextManager) WaitForCompletion(ctx context.Context, duration time.Duration) error {
    timer := time.NewTimer(duration)
    defer timer.Stop()

    select {
    case <-ctx.Done():
        return ctx.Err() // Context cancelled
    case <-timer.C:
        return nil // Duration elapsed successfully
    }
}

This is useful when you need to wait for a fixed duration, but want to respect cancellation (e.g., graceful shutdown, backoff delays).

Real-world example: Processing items with cancellation

This pattern shows up constantly in batch processing, ETL pipelines, and background workers:

func ProcessItems(ctx context.Context, items []string) ([]string, error) {
    const perItemWork = 40 * time.Millisecond

    results := make([]string, 0, len(items))
    
    for _, item := range items {
        // Check cancellation before processing each item
        if err := ctx.Err(); err != nil {
            return results, err // Return partial results
        }

        timer := time.NewTimer(perItemWork)
        select {
        case <-ctx.Done():
            timer.Stop()
            return results, ctx.Err()
        case <-timer.C:
            // Process the item
            results = append(results, "processed_"+item)
        }
    }

    return results, nil
}

Production note: This pattern ensures that if a batch job is cancelled (e.g., during deployment or shutdown), you get partial results and a clean exit instead of orphaned goroutines.

Testing context behavior

Context behavior is subtle, so testing is critical. Here are the high-value tests:

Test 1: Cancellation propagates correctly

func TestExecuteWithContext_Cancellation(t *testing.T) {
    cm := NewContextManager()
    ctx, cancel := context.WithCancel(context.Background())

    // Long running task
    task := func() error {
        time.Sleep(200 * time.Millisecond)
        return nil
    }

    // Cancel after a short delay
    go func() {
        time.Sleep(50 * time.Millisecond)
        cancel()
    }()

    err := cm.ExecuteWithContext(ctx, task)
    if err != context.Canceled {
        t.Errorf("Expected context.Canceled, got %v", err)
    }
}

Test 2: Timeout fires correctly

func TestCreateTimeoutContext(t *testing.T) {
    cm := NewContextManager()
    timeout := 50 * time.Millisecond

    ctx, cancel := cm.CreateTimeoutContext(context.Background(), timeout)
    defer cancel()

    select {
    case <-ctx.Done():
        if ctx.Err() != context.DeadlineExceeded {
            t.Errorf("Expected context.DeadlineExceeded, got %v", ctx.Err())
        }
    case <-time.After(100 * time.Millisecond):
        t.Fatal("Context should timeout after specified duration")
    }
}

Test 3: Values survive context derivation

func TestAddAndGetValue(t *testing.T) {
    cm := NewContextManager()
    ctx := context.Background()

    ctx = cm.AddValue(ctx, "user", "alice")
    ctx = cm.AddValue(ctx, "requestID", "12345")

    value, exists := cm.GetValue(ctx, "user")
    if !exists || value != "alice" {
        t.Fatal("Expected user value to exist and equal 'alice'")
    }

    value, exists = cm.GetValue(ctx, "nonexistent")
    if exists {
        t.Error("Expected nonexistent value to not exist")
    }
}

Production patterns I actually use

After shipping context-heavy Go services to production, here are the patterns that stuck:

1. Always derive from request context in HTTP handlers

func MyHandler(w http.ResponseWriter, r *http.Request) {
    ctx := r.Context() // Already has cancellation + deadlines from server
    // Use ctx for all downstream calls
}

2. Add timeouts at service boundaries

func CallExternalAPI(ctx context.Context, url string) ([]byte, error) {
    // Enforce a tight timeout for external calls
    ctx, cancel := context.WithTimeout(ctx, 2*time.Second)
    defer cancel()

    req, _ := http.NewRequestWithContext(ctx, "GET", url, nil)
    resp, err := http.DefaultClient.Do(req)
    // ...
}

3. Use typed keys for context values

type contextKey string

const (
    userIDKey    contextKey = "userID"
    requestIDKey contextKey = "requestID"
)

// Prevents collisions with other packages using string keys

4. Propagate cancellation to all goroutines

func FanOut(ctx context.Context, tasks []func() error) error {
    ctx, cancel := context.WithCancel(ctx)
    defer cancel()

    errCh := make(chan error, len(tasks))

    for _, task := range tasks {
        task := task // Capture loop variable
        go func() {
            errCh <- task()
        }()
    }

    // Wait for first error or all completions
    for range tasks {
        if err := <-errCh; err != nil {
            cancel() // Cancel all other goroutines
            return err
        }
    }

    return nil
}

5. Graceful shutdown with context

func (s *Server) Shutdown(timeout time.Duration) error {
    ctx, cancel := context.WithTimeout(context.Background(), timeout)
    defer cancel()

    // Shut down HTTP server with context
    return s.httpServer.Shutdown(ctx)
}

Common pitfalls (and how to avoid them)

Pitfall 1: Not checking context errors

// BAD: Ignoring context cancellation
func Process(ctx context.Context) {
    for {
        doWork()
        time.Sleep(1 * time.Second)
    }
}

// GOOD: Checking ctx.Done()
func Process(ctx context.Context) {
    ticker := time.NewTicker(1 * time.Second)
    defer ticker.Stop()

    for {
        select {
        case <-ctx.Done():
            return // Clean exit
        case <-ticker.C:
            doWork()
        }
    }
}

Pitfall 2: Using context for optional parameters

// BAD: Using context for business logic
func CreateUser(ctx context.Context) error {
    email := ctx.Value("email").(string) // Fragile, not obvious
    // ...
}

// GOOD: Use function parameters
func CreateUser(ctx context.Context, email string) error {
    // ...
}

Pitfall 3: Not calling cancel()

// BAD: Leaking resources
func BadTimeout() {
    ctx, _ := context.WithTimeout(context.Background(), 5*time.Second)
    // Forgot to call cancel() - timer leaks
    doWork(ctx)
}

// GOOD: Always defer cancel()
func GoodTimeout() {
    ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
    defer cancel() // Releases timer even if doWork returns early
    doWork(ctx)
}

Pitfall 4: Passing nil context

// BAD: Causes panics
func Process(ctx context.Context) {
    // If ctx is nil, this panics
    select {
    case <-ctx.Done():
    }
}

// GOOD: Use context.Background() or context.TODO()
func Process() {
    ctx := context.Background()
    // Or context.TODO() if you plan to replace it later
}

When NOT to use context

Context is not a silver bullet. Don't use it for:

Configuration: Use config structs or dependency injection
Business logic state: Use function parameters or struct fields
Database transactions: Use *sql.Tx explicitly
Authentication/authorization objects: Pass them explicitly or use middleware

Performance considerations

Context operations are designed to be fast, but not free:

Value lookups are O(n): Context values form a linked list, so deep chains are slow. Keep context chains shallow.
Cancellation is O(n): Broadcasting to (n) child contexts takes (O(n)) time.
Every WithValue allocates: If you're in a super hot path (millions of QPS), consider alternatives.

For 99% of Go services, context overhead is negligible. Profile before optimizing.

Conclusion: Context is about coordination, not configuration

The Go context package is elegant because it solves three hard problems (cancellation, timeouts, value propagation) with one simple interface.

The key mental model:

Context flows downstream with your request, carrying deadlines and metadata, and fires cancellation signals when work should stop.

If you internalize that model, you'll write Go code that's:

Resilient: Fails fast instead of hanging
Efficient: Cancels work when it's no longer needed
Observable: Propagates trace IDs and request metadata
Idiomatic: Uses the patterns every Go developer expects

This blog was inspired by Go Interview Challenge 30, which tasks you to dig into real-world Go context management across microservices. The patterns, examples, and code in this post reflect practical lessons learned from tackling those scenarios. If you're preparing for Go interviews or want to strengthen your understanding of context in production, the original challenge is a great way to get hands-on.

Happy coding! 🚀

Benchmarking in Go: Measure Performance with Real Examples (Sorting + strings.Builder)

Sun, 28 Dec 2025 00:00:00 GMT

Benchmarking in Go: Measure Performance with Real Examples (Sorting + strings.Builder)

Benchmarking in Go isn’t a “nice to have” — it’s how you avoid guessing. The fastest way to waste time in performance work is to optimize blindly, then ship code that “feels faster” but isn’t.

In this post, I’ll explain what Go benchmarks are, how to run them, how to read the output, and I’ll use two concrete examples:

Sorting: bubble sort vs Go’s optimized sort (slices.Sort)
Strings: += concatenation vs strings.Builder

What is benchmarking in Go?

A benchmark measures how long a piece of code takes to run (and optionally how much memory it allocates). In Go, benchmarks live next to tests in _test.go files and use the testing package.

A benchmark function:

Starts with Benchmark...
Accepts a *testing.B
Runs the target operation b.N times (Go automatically chooses b.N to stabilize the timing)

Example skeleton:

func BenchmarkMyFunction(b *testing.B) {
	for i := 0; i < b.N; i++ {
		MyFunction()
	}
}

Example 1: Sorting (bubble sort vs `slices.Sort`)

The “slow” version (bubble sort)

This version is intentionally inefficient: (O(n^2)).

// SlowSort sorts a slice of integers using a very inefficient algorithm (bubble sort)
func SlowSort(data []int) []int {
	// Make a copy to avoid modifying the original
	result := make([]int, len(data))
	copy(result, data)

	// Bubble sort implementation
	for i := 0; i < len(result); i++ {
		for j := 0; j < len(result)-1; j++ {
			if result[j] > result[j+1] {
				result[j], result[j+1] = result[j+1], result[j]
			}
		}
	}

	return result
}

The optimized version (standard library sort)

Same output, better algorithmic complexity in practice ((O(n \log n)) average).

import "slices"

// OptimizedSort is an optimized version of SlowSort
func OptimizedSort(data []int) []int {
	sorted := make([]int, len(data))
	copy(sorted, data)
	slices.Sort(sorted)
	return sorted
}

Correctness test (optimized must match slow)

Before caring about speed, we ensure both functions return identical results:

func TestOptimizedSort(t *testing.T) {
	testCases := []struct {
		name  string
		input []int
	}{
		{"Empty", []int{}},
		{"One Element", []int{42}},
		{"Already Sorted", []int{1, 2, 3, 4, 5}},
		{"Reverse Sorted", []int{5, 4, 3, 2, 1}},
		{"Random Order", []int{3, 1, 4, 1, 5, 9, 2, 6}},
		{"Random Large", generateRandomSlice(100)},
	}

	for _, tc := range testCases {
		t.Run(tc.name, func(t *testing.T) {
			slowResult := SlowSort(tc.input)
			optimizedResult := OptimizedSort(tc.input)

			if !slicesEqual(slowResult, optimizedResult) {
				t.Errorf("OptimizedSort gave different results than SlowSort. Got %v, expected %v", optimizedResult, slowResult)
			}
		})
	}
}

Benchmark code (the important part)

Benchmarks are just code that runs your function b.N times:

func BenchmarkSlowSort(b *testing.B) {
	sizes := []int{10, 100, 1000}

	for _, size := range sizes {
		data := generateRandomSlice(size)
		b.Run(fmt.Sprint(size), func(b *testing.B) {
			for i := 0; i < b.N; i++ {
				SlowSort(data)
			}
		})
	}
}

func BenchmarkOptimizedSort(b *testing.B) {
	sizes := []int{10, 100, 1000}

	for _, size := range sizes {
		data := generateRandomSlice(size)
		b.Run(fmt.Sprint(size), func(b *testing.B) {
			for i := 0; i < b.N; i++ {
				OptimizedSort(data)
			}
		})
	}
}

Real results (my run)

Environment:

OS/Arch: darwin/arm64
Go: go1.25.1
Command: go test -run '^$' -bench '.' -benchmem -count=3

For 1000 elements:

SlowSort: ~580–588 µs/op
OptimizedSort: ~9.7 µs/op
Speedup: ~60× faster

Representative lines:

BenchmarkSlowSort/1000-8            587459 ns/op   8192 B/op   1 allocs/op
BenchmarkOptimizedSort/1000-8         9740 ns/op   8192 B/op   1 allocs/op

Example 2: Building strings (`+=` vs `strings.Builder`)

The “slow” version (repeated `+=`)

This looks innocent, but it repeatedly allocates and copies as the string grows.

// InefficientStringBuilder builds a string by repeatedly concatenating
func InefficientStringBuilder(parts []string, repeatCount int) string {
	result := ""

	for i := 0; i < repeatCount; i++ {
		for _, part := range parts {
			result += part
		}
	}

	return result
}

The optimized version (`strings.Builder`)

strings.Builder is the idiomatic way to build strings efficiently.

import "strings"

// OptimizedStringBuilder is an optimized version of InefficientStringBuilder
func OptimizedStringBuilder(parts []string, repeatCount int) string {
	var builder strings.Builder

	for i := 0; i < repeatCount; i++ {
		for _, part := range parts {
			builder.WriteString(part)
		}
	}

	return builder.String()
}

Correctness test (optimized must match slow)

func TestOptimizedStringBuilder(t *testing.T) {
	testCases := []struct {
		name        string
		parts       []string
		repeatCount int
	}{
		{"Empty", []string{}, 10},
		{"Single Part", []string{"Hello"}, 5},
		{"Multiple Parts", []string{"Hello", " ", "World", "!"}, 3},
		{"Large Build", []string{"This", " ", "is", " ", "a", " ", "test"}, 100},
	}

	for _, tc := range testCases {
		t.Run(tc.name, func(t *testing.T) {
			inefficientResult := InefficientStringBuilder(tc.parts, tc.repeatCount)
			optimizedResult := OptimizedStringBuilder(tc.parts, tc.repeatCount)

			if optimizedResult != inefficientResult {
				t.Errorf("OptimizedStringBuilder gave different results than InefficientStringBuilder. Got %q, expected %q", optimizedResult, inefficientResult)
			}
		})
	}
}

Real results (my run)

The “Large” case is where the difference becomes obvious:

Inefficient: ~4.28–4.38 ms/op, ~70 MB/op, ~7003 allocs/op
Optimized: ~26.7–27.5 µs/op, ~84 KB/op, 18 allocs/op
Speedup: ~160× faster
Alloc reduction: ~7003 → 18 allocs/op

Representative lines:

BenchmarkInefficientStringBuilder/Large-8   4277351 ns/op   70153813 B/op   7004 allocs/op
BenchmarkOptimizedStringBuilder/Large-8       27495 ns/op      84729 B/op     18 allocs/op

How to run benchmarks (exact commands)

You can run benchmarks locally without any special tooling — it’s just go test with -bench.

If you want to benchmark a single file submission (like in this challenge), copy your code + the _test.go file into a temp folder:

cd /Users/younesbouchbouk/dev/go-interview-practice/challenge-16

tmp="$(mktemp -d)"
cp "submissions/YounesBouchbouk/solution-template.go" "solution-template_test.go" "$tmp/"
cd "$tmp"
go mod init challenge

# Run only benchmarks, skip tests
go test -run '^$' -bench '.' -benchmem

# More stable results
go test -run '^$' -bench '.' -benchmem -count=3

Running one benchmark (or a subset)

# Sorting only
go test -run '^$' -bench 'OptimizedSort|SlowSort' -benchmem

# String building only
go test -run '^$' -bench 'OptimizedStringBuilder|InefficientStringBuilder' -benchmem

How to read Go benchmark results (ns/op, B/op, allocs/op)

You’ll typically see output like:

BenchmarkSomething-8   1000000   123 ns/op   16 B/op   1 allocs/op

Here’s what matters:

ns/op: nanoseconds per operation (lower is better)
B/op: bytes allocated per operation (lower is better)
allocs/op: number of heap allocations per operation (lower is better)
-8: the GOMAXPROCS value (how many logical CPUs the benchmark used)

Tip: Always run with -benchmem when optimizing. It’s common to make code faster while accidentally allocating more, which can hurt real production latency under GC pressure.

The full raw output from my run is stored next to this post in benchmarks.txt.

Best practices (what I actually do in real Go services)

Benchmark the real hot path: benchmark the function at the boundary where it matters (e.g., request handler, serialization step, DB mapping).
Use -benchmem: allocations are latency in disguise.
Run multiple times: -count=3 (or more) helps avoid noise.
Compare runs with benchstat:

go test -run '^$' -bench '.' -benchmem -count=5 > old.txt
# change code
go test -run '^$' -bench '.' -benchmem -count=5 > new.txt
benchstat old.txt new.txt

Profile when you’re stuck: if the benchmark shows you’re slow but you don’t know why, use pprof (CPU + heap profiles).

Conclusion

Go makes benchmarking ridiculously accessible. The workflow is simple:

Write benchmarks for the code you care about
Run go test -bench ... -benchmem
Change one thing
Measure again

That “measure → optimize → measure” loop is exactly how you get fast, reliable Go code without guessing.

This blog was inspired by Go Interview Challenge 16, which tasks you to dig into real-world Go benchmarking for sorting and string building. The practical lessons, examples, and code here draw directly from working through that challenge. If you're prepping for Go interviews, or just want to sharpen your performance intuition, you’ll find the original challenge a great hands-on exercise.

Happy benchmarking! 🚀

Circuit Breaker Pattern in Go (with a Practical Implementation)

Mon, 22 Dec 2025 00:00:00 GMT

Circuit Breaker Pattern in Go (with a Practical Implementation)

In distributed systems, we inevitably depend on things we don't control: external APIs, databases, queues, and networks. When one dependency becomes slow or unavailable, naive retries can turn a single failure into cascading failure.

The Circuit Breaker pattern protects your system by failing fast when a dependency is unhealthy, and by probing for recovery after a cooldown.

The problem it solves

When a downstream service is degraded, your application can:

Waste CPU and goroutines retrying doomed requests
Exhaust connection pools and timeouts
Increase latency for healthy requests
Accidentally DoS an already struggling dependency

Circuit breakers add a lightweight state machine in front of the risky call to prevent repeated failures.

How a circuit breaker works (3 states)

1) Closed (normal)

Requests are allowed through.
Failures are monitored.
When failures exceed a threshold, the breaker trips to Open.

2) Open (fail fast)

Requests fail immediately without calling the dependency.
After a timeout, the breaker transitions to Half-Open.

3) Half-Open (probe recovery)

A limited number of requests are allowed as probes.
If probes succeed, the breaker returns to Closed.
If a probe fails, the breaker goes back to Open.

State transition diagram

    [Closed] --failure threshold--> [Open]
        ^                             |
        |                             |
     success                      timeout elapsed
        |                             |
        v                             v
    [Half-Open] --failure--> [Open]

Implementation concepts in Go

These are the non-negotiables for a circuit breaker in a Go service:

Thread safety: the breaker is shared across goroutines.
Metrics: you need counters to decide when to trip.
Configurable behavior: thresholds/timeouts vary per dependency.
Context-aware calls: respect cancellation and deadlines.

Reference implementation (Go)

Below is the implementation I use as a learning-friendly baseline. It exposes a small interface, tracks metrics, and implements the Closed/Open/Half-Open state machine with locks.

Note: This is intentionally kept readable. In the “Production hardening” section, I’ll call out the upgrades I’d do before shipping to a high-throughput system.

// Package challenge20 contains the implementation for Challenge 20: Circuit Breaker Pattern
package main

import (
	"context"
	"errors"
	"fmt"
	"sync"
	"time"
)

// State represents the current state of the circuit breaker
type State int

const (
	StateClosed State = iota
	StateOpen
	StateHalfOpen
)

// String returns the string representation of the state
func (s State) String() string {
	switch s {
	case StateClosed:
		return "Closed"
	case StateOpen:
		return "Open"
	case StateHalfOpen:
		return "Half-Open"
	default:
		return "Unknown"
	}
}

// Metrics represents the circuit breaker metrics
type Metrics struct {
	Requests            int64
	Successes           int64
	Failures            int64
	ConsecutiveFailures int64
	LastFailureTime     time.Time
}

// Config represents the configuration for the circuit breaker
type Config struct {
	MaxRequests   uint32                                  // Max requests allowed in half-open state
	Interval      time.Duration                           // Statistical window for closed state
	Timeout       time.Duration                           // Time to wait before half-open
	ReadyToTrip   func(Metrics) bool                      // Function to determine when to trip
	OnStateChange func(name string, from State, to State) // State change callback
}

// CircuitBreaker interface defines the operations for a circuit breaker
type CircuitBreaker interface {
	Call(ctx context.Context, operation func() (interface{}, error)) (interface{}, error)
	GetState() State
	GetMetrics() Metrics
}

// circuitBreakerImpl is the concrete implementation of CircuitBreaker
type circuitBreakerImpl struct {
	name             string
	config           Config
	state            State
	metrics          Metrics
	lastStateChange  time.Time
	halfOpenRequests uint32
	mutex            sync.RWMutex
}

// Error definitions
var (
	ErrCircuitBreakerOpen = errors.New("circuit breaker is open")
	ErrTooManyRequests    = errors.New("too many requests in half-open state")
)

// NewCircuitBreaker creates a new circuit breaker with the given configuration
func NewCircuitBreaker(config Config) CircuitBreaker {
	// Set default values if not provided
	if config.MaxRequests == 0 {
		config.MaxRequests = 1
	}
	if config.Interval == 0 {
		config.Interval = time.Minute
	}
	if config.Timeout == 0 {
		config.Timeout = 30 * time.Second
	}
	if config.ReadyToTrip == nil {
		config.ReadyToTrip = func(m Metrics) bool {
			return m.ConsecutiveFailures >= 5
		}
	}

	return &circuitBreakerImpl{
		name:            "circuit-breaker",
		config:          config,
		state:           StateClosed,
		lastStateChange: time.Now(),
	}
}

// Call executes the given operation through the circuit breaker
func (cb *circuitBreakerImpl) Call(ctx context.Context, operation func() (interface{}, error)) (interface{}, error) {
	if err := ctx.Err(); err != nil {
		return nil, err
	}

	if err := cb.canExecute(); err != nil {
		return nil, err
	}

	result, err := operation()

	cb.mutex.Lock()
	defer cb.mutex.Unlock()

	if err == nil {
		cb.recordSuccess()
	} else {
		cb.recordFailure()
	}
	return result, err
}

// GetState returns the current state of the circuit breaker
func (cb *circuitBreakerImpl) GetState() State {
	cb.mutex.RLock()
	defer cb.mutex.RUnlock()
	return cb.state
}

// GetMetrics returns the current metrics of the circuit breaker
func (cb *circuitBreakerImpl) GetMetrics() Metrics {
	cb.mutex.RLock()
	defer cb.mutex.RUnlock()
	return cb.metrics
}

// setState changes the circuit breaker state and triggers callbacks
// NOTE: Caller MUST hold write lock (mutex.Lock)
func (cb *circuitBreakerImpl) setState(newState State) {
	// Check if state actually changed
	if cb.state == newState {
		return
	}

	oldState := cb.state
	cb.state = newState
	cb.lastStateChange = time.Now()

	// Reset appropriate metrics based on new state
	if newState == StateClosed {
		cb.metrics = Metrics{}
	}
	if newState == StateHalfOpen {
		cb.halfOpenRequests = 0
	}

	// Call OnStateChange callback if configured
	if cb.config.OnStateChange != nil {
		cb.config.OnStateChange(cb.name, oldState, newState)
	}
}

// canExecute determines if a request can be executed in the current state
func (cb *circuitBreakerImpl) canExecute() error {
	cb.mutex.RLock()
	state := cb.state
	lastChange := cb.lastStateChange
	cb.mutex.RUnlock()

	// For StateClosed: always allow
	if state == StateClosed {
		return nil
	}

	// For StateOpen: check if timeout has passed for transition to half-open
	if state == StateOpen {
		if time.Since(lastChange) > cb.config.Timeout {
			// Need to transition to half-open - acquire write lock
			cb.mutex.Lock()
			// Double-check state hasn't changed while we were waiting for lock
			if cb.state == StateOpen {
				cb.setState(StateHalfOpen)
				cb.halfOpenRequests = 1 // Count this request
			} else if cb.state == StateHalfOpen {
				// Another goroutine already transitioned, increment counter
				if cb.halfOpenRequests >= cb.config.MaxRequests {
					cb.mutex.Unlock()
					return ErrTooManyRequests
				}
				cb.halfOpenRequests++
			}
			cb.mutex.Unlock()
			return nil
		}
		return ErrCircuitBreakerOpen
	}

	// For StateHalfOpen: check if we've exceeded MaxRequests
	if state == StateHalfOpen {
		cb.mutex.Lock()
		defer cb.mutex.Unlock()

		if cb.halfOpenRequests >= cb.config.MaxRequests {
			return ErrTooManyRequests
		}
		cb.halfOpenRequests++
		return nil
	}

	return nil
}

// recordSuccess records a successful operation
// NOTE: Caller MUST hold write lock (mutex.Lock)
func (cb *circuitBreakerImpl) recordSuccess() {
	// Increment success and request counters
	cb.metrics.Requests++
	cb.metrics.Successes++

	// Reset consecutive failures
	cb.metrics.ConsecutiveFailures = 0

	// In half-open state, consider transitioning to closed
	if cb.state == StateHalfOpen {
		cb.setState(StateClosed)
	}
}

// recordFailure records a failed operation
// NOTE: Caller MUST hold write lock (mutex.Lock)
func (cb *circuitBreakerImpl) recordFailure() {
	// Increment failure and request counters
	cb.metrics.Requests++
	cb.metrics.Failures++

	// Increment consecutive failures
	cb.metrics.ConsecutiveFailures++

	// Update last failure time
	cb.metrics.LastFailureTime = time.Now()

	// In half-open state, transition back to open immediately
	if cb.state == StateHalfOpen {
		cb.setState(StateOpen)
	} else if cb.config.ReadyToTrip(cb.metrics) {
		// Check if circuit should trip (ReadyToTrip function)
		cb.setState(StateOpen)
	}
}

// shouldTrip determines if the circuit breaker should trip to open state
func (cb *circuitBreakerImpl) shouldTrip() bool {
	cb.mutex.RLock()
	defer cb.mutex.RUnlock()
	return cb.config.ReadyToTrip(cb.metrics)
}

// isReady checks if the circuit breaker is ready to transition from open to half-open
func (cb *circuitBreakerImpl) isReady() bool {
	cb.mutex.RLock()
	defer cb.mutex.RUnlock()
	return time.Since(cb.lastStateChange) > cb.config.Timeout
}

// Example usage and testing helper functions
func main() {
	// Example usage of the circuit breaker
	fmt.Println("Circuit Breaker Pattern Example")

	// Create a circuit breaker configuration
	config := Config{
		MaxRequests: 3,
		Interval:    time.Minute,
		Timeout:     10 * time.Second,
		ReadyToTrip: func(m Metrics) bool {
			return m.ConsecutiveFailures >= 3
		},
		OnStateChange: func(name string, from State, to State) {
			fmt.Printf("Circuit breaker %s: %s -> %s\n", name, from, to)
		},
	}

	cb := NewCircuitBreaker(config)

	// Simulate some operations
	ctx := context.Background()

	// Successful operation
	result, err := cb.Call(ctx, func() (interface{}, error) {
		return "success", nil
	})
	fmt.Printf("Result: %v, Error: %v\n", result, err)

	// Failing operation
	result, err = cb.Call(ctx, func() (interface{}, error) {
		return nil, errors.New("simulated failure")
	})
	fmt.Printf("Result: %v, Error: %v\n", result, err)

	// Print current state and metrics
	fmt.Printf("Current state: %v\n", cb.GetState())
	fmt.Printf("Current metrics: %+v\n", cb.GetMetrics())
}

Production hardening (what I’d improve before shipping)

This reference implementation is a great learning base, but for production workloads I usually harden it in a few ways:

Actually use Interval: implement a rolling window (or periodic reset) so “old failures” don’t poison the breaker forever.
Classify errors: don’t count context.Canceled, context.DeadlineExceeded, or “client errors” (e.g., HTTP 4xx) as breaker failures.
Half-open close criteria: instead of closing after one success, require (N) successful probe calls to reduce flapping.
Avoid lock-heavy hot paths: consider atomics for counters + a single CAS state machine if the breaker sits on a very hot route.
Expose metrics to Prometheus: state, total, failures, consecutive failures, failure rate, last failure time.
Add fallback strategy: when Open, return cached data, degrade response, or route to an alternate dependency.

Testing strategy (high-value tests)

If you add this breaker to a real service, the tests that matter most are:

State transitions: Closed → Open, Open → Half-Open after timeout, Half-Open → Closed on success, Half-Open → Open on failure
Concurrency: many goroutines calling Call() simultaneously (run with the race detector)
Half-open request limit: ensure MaxRequests is enforced under contention

Where to use it

Common uses in a portfolio-grade backend:

HTTP client calls to third-party APIs
Database calls when the DB is intermittently failing
Calls to internal microservices (especially during deploys/incidents)

Conclusion

Circuit breakers are one of the simplest (and highest ROI) resilience patterns you can add to a backend system. They help you fail fast, protect your dependencies, and recover automatically—without turning every outage into a cascade.

If you want, I can also refactor the implementation into a reusable package (breaker), add unit tests + race tests, and show a real integration example (e.g., wrapping http.Client).

This blog was inspired by Go Interview Challenge 20, which tasks you to implement and production-harden a real-world circuit breaker in Go. The concepts and code here are shaped directly by that hands-on challenge—if you're preparing for Go interviews or aiming to upgrade your system resilience, I highly recommend giving the original challenge a try!

Data Serialization and Deserialization in Golang

Sat, 30 Mar 2024 00:00:00 GMT

Data Serialization and Deserialization in Golang

Background

Serialization and deserialization are essential processes in modern software development, enabling data to be efficiently transported and stored.

In Go (Golang), these processes are straightforward, thanks to its robust standard library and the availability of third-party packages.

In this blog post, we will delve into the basics of serialization and deserialization in Go, covering popular formats such as JSON, XML, Protocol Buffers (protobuf), and YAML.

What are Serialization and Deserialization?

Serialization is the process of converting an in-memory data structure into a format that can be easily saved to a file or transmitted over a network.

Deserialization is the reverse process, where the serialized data is converted back into an in-memory data structure.

Common formats for serialization include JSON, XML, binary formats like Gob, Protocol Buffers, MessagePack, and YAML.

Each format has its own use cases and advantages:

JSON: Great for web APIs and interoperability
XML: Useful when a strict schema is needed
Gob: Ideal for Go-specific, high-performance serialization
Protocol Buffers: Efficient and versatile, especially for large-scale systems
MessagePack: Compact and efficient for bandwidth-sensitive applications
YAML: Human-readable, often used for configuration files

JSON Serialization and Deserialization

JSON is one of the most commonly used serialization formats, especially for web APIs. Go's standard library provides excellent support for JSON through the encoding/json package.

Basic Example

package main

import (
	"encoding/json"
	"fmt"
	"log"
)

// User defines our structure for JSON serialization
type User struct {
	ID        int    `json:"id"`
	Username  string `json:"username"`
	Email     string `json:"email"`
	IsActive  bool   `json:"is_active"`
	CreatedAt string `json:"created_at,omitempty"`
}

func main() {
	// Create a user
	user := User{
		ID:       1,
		Username: "johndoe",
		Email:    "john@example.com",
		IsActive: true,
	}

	// Serialize (Marshal) to JSON
	jsonData, err := json.Marshal(user)
	if err != nil {
		log.Fatalf("Error marshaling to JSON: %v", err)
	}

	fmt.Printf("Serialized JSON: %s\n", jsonData)

	// Deserialize (Unmarshal) from JSON
	var newUser User
	err = json.Unmarshal(jsonData, &newUser)
	if err != nil {
		log.Fatalf("Error unmarshaling from JSON: %v", err)
	}

	fmt.Printf("Deserialized User: %+v\n", newUser)
}

Custom JSON Marshaling

Sometimes you need more control over how your data is serialized. Go allows you to implement the json.Marshaler and json.Unmarshaler interfaces for custom serialization logic:

package main

import (
	"encoding/json"
	"fmt"
	"log"
	"time"
)

// CustomTime showcases custom marshaling
type CustomTime struct {
	time.Time
}

// MarshalJSON implements json.Marshaler
func (ct CustomTime) MarshalJSON() ([]byte, error) {
	return json.Marshal(ct.Time.Format("2006-01-02 15:04:05"))
}

// UnmarshalJSON implements json.Unmarshaler
func (ct *CustomTime) UnmarshalJSON(data []byte) error {
	var timeStr string
	if err := json.Unmarshal(data, &timeStr); err != nil {
		return err
	}

	parsedTime, err := time.Parse("2006-01-02 15:04:05", timeStr)
	if err != nil {
		return err
	}

	ct.Time = parsedTime
	return nil
}

// Event uses our custom time type
type Event struct {
	Name      string     `json:"name"`
	Timestamp CustomTime `json:"timestamp"`
}

func main() {
	event := Event{
		Name:      "Conference",
		Timestamp: CustomTime{time.Now()},
	}

	// Serialize with custom marshaling
	jsonData, err := json.Marshal(event)
	if err != nil {
		log.Fatalf("Error marshaling to JSON: %v", err)
	}

	fmt.Printf("Serialized Event: %s\n", jsonData)

	// Deserialize with custom unmarshaling
	var newEvent Event
	err = json.Unmarshal(jsonData, &newEvent)
	if err != nil {
		log.Fatalf("Error unmarshaling from JSON: %v", err)
	}

	fmt.Printf("Deserialized Event: %+v\n", newEvent)
	fmt.Printf("Timestamp: %v\n", newEvent.Timestamp.Time)
}

XML Serialization and Deserialization

XML is another popular format, especially for SOAP APIs and configuration files. Go's standard library provides the encoding/xml package for XML processing.

package main

import (
	"encoding/xml"
	"fmt"
	"log"
)

// Book demonstrates XML serialization
type Book struct {
	XMLName  xml.Name `xml:"book"`
	ID       int      `xml:"id,attr"`
	Title    string   `xml:"title"`
	Author   string   `xml:"author"`
	ISBN     string   `xml:"isbn"`
	Price    float64  `xml:"price"`
	InStock  bool     `xml:"in_stock"`
	Category string   `xml:"category,omitempty"`
}

func main() {
	// Create a book
	book := Book{
		ID:      101,
		Title:   "Go Programming",
		Author:  "Jane Smith",
		ISBN:    "978-1234567890",
		Price:   29.99,
		InStock: true,
	}

	// Serialize to XML
	xmlData, err := xml.MarshalIndent(book, "", "  ")
	if err != nil {
		log.Fatalf("Error marshaling to XML: %v", err)
	}

	// Add XML header
	xmlData = append([]byte(xml.Header), xmlData...)
	fmt.Printf("Serialized XML:\n%s\n", xmlData)

	// Deserialize from XML
	var newBook Book
	err = xml.Unmarshal(xmlData, &newBook)
	if err != nil {
		log.Fatalf("Error unmarshaling from XML: %v", err)
	}

	fmt.Printf("Deserialized Book: %+v\n", newBook)
}

Protocol Buffers (protobuf)

Protocol Buffers is a binary serialization format developed by Google. It's known for its efficiency, cross-language compatibility, and strong typing.

First, install the Protocol Buffers compiler and Go plugin:

go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest

Next, create a .proto file:

// person.proto
syntax = "proto3";
package person;

option go_package = "example/person";

message Person {
  int32 id = 1;
  string name = 2;
  string email = 3;
  
  enum PhoneType {
    MOBILE = 0;
    HOME = 1;
    WORK = 2;
  }
  
  message PhoneNumber {
    string number = 1;
    PhoneType type = 2;
  }
  
  repeated PhoneNumber phones = 4;
}

Generate Go code from the proto file:

protoc --go_out=. person.proto

Now use the generated code:

package main

import (
	"fmt"
	"log"

	"google.golang.org/protobuf/proto"
	"example/person" // This path depends on your go_package option
)

func main() {
	// Create a new Person
	p := &person.Person{
		Id:    123,
		Name:  "Alice",
		Email: "alice@example.com",
		Phones: []*person.Person_PhoneNumber{
			{Number: "555-1234", Type: person.Person_MOBILE},
			{Number: "555-4321", Type: person.Person_HOME},
		},
	}

	// Serialize
	data, err := proto.Marshal(p)
	if err != nil {
		log.Fatalf("Marshaling error: %v", err)
	}
	fmt.Printf("Serialized size: %d bytes\n", len(data))

	// Deserialize
	newPerson := &person.Person{}
	err = proto.Unmarshal(data, newPerson)
	if err != nil {
		log.Fatalf("Unmarshaling error: %v", err)
	}

	fmt.Printf("Deserialized Person: %+v\n", newPerson)
	fmt.Printf("Name: %s\n", newPerson.Name)
	fmt.Printf("First Phone: %s (%s)\n", 
		newPerson.Phones[0].Number,
		newPerson.Phones[0].Type)
}

YAML Serialization and Deserialization

YAML is a human-friendly data serialization format often used for configuration files. We'll use the popular gopkg.in/yaml.v3 package.

First, install the YAML package:

go get gopkg.in/yaml.v3

Now let's implement YAML serialization and deserialization:

package main

import (
	"fmt"
	"log"
	"os"

	"gopkg.in/yaml.v3"
)

// Config demonstrates YAML serialization
type Config struct {
	Server struct {
		Host string `yaml:"host"`
		Port int    `yaml:"port"`
		TLS  bool   `yaml:"tls"`
	} `yaml:"server"`
	Database struct {
		Host     string `yaml:"host"`
		Port     int    `yaml:"port"`
		Username string `yaml:"username"`
		Password string `yaml:"password"`
		Name     string `yaml:"name"`
	} `yaml:"database"`
	Logging struct {
		Level  string `yaml:"level"`
		Format string `yaml:"format"`
		File   string `yaml:"file,omitempty"`
	} `yaml:"logging"`
}

func main() {
	// Create a config
	var config Config
	config.Server.Host = "localhost"
	config.Server.Port = 8080
	config.Server.TLS = true
	
	config.Database.Host = "db.example.com"
	config.Database.Port = 5432
	config.Database.Username = "admin"
	config.Database.Password = "securepass"
	config.Database.Name = "appdb"
	
	config.Logging.Level = "info"
	config.Logging.Format = "json"

	// Serialize to YAML
	yamlData, err := yaml.Marshal(&config)
	if err != nil {
		log.Fatalf("Error marshaling to YAML: %v", err)
	}

	fmt.Printf("Serialized YAML:\n%s\n", yamlData)

	// Write to file
	err = os.WriteFile("config.yaml", yamlData, 0644)
	if err != nil {
		log.Fatalf("Error writing YAML to file: %v", err)
	}

	// Deserialize from YAML
	var newConfig Config
	err = yaml.Unmarshal(yamlData, &newConfig)
	if err != nil {
		log.Fatalf("Error unmarshaling from YAML: %v", err)
	}

	fmt.Printf("Deserialized Config: %+v\n", newConfig)
	fmt.Printf("Server Port: %d\n", newConfig.Server.Port)
	fmt.Printf("Database Name: %s\n", newConfig.Database.Name)
}

Performance Considerations

When choosing a serialization format, consider these factors:

JSON:
- Pros: Human-readable, widely supported
- Cons: Relatively verbose, slower than binary formats
XML:
- Pros: Supports complex structures, widely used in enterprise systems
- Cons: Very verbose, slower than most other formats
Protocol Buffers:
- Pros: Very efficient binary format, strong typing, schema evolution
- Cons: Not human-readable, requires schema definition
YAML:
- Pros: Very human-readable, good for config files
- Cons: Slower than JSON, more complex parser

Benchmarking Example

Here's a simple benchmark comparing JSON and Protocol Buffers:

package main

import (
	"encoding/json"
	"fmt"
	"log"
	"time"

	"google.golang.org/protobuf/proto"
	"example/person" // Your generated protobuf package
)

type PersonJSON struct {
	ID     int32    `json:"id"`
	Name   string   `json:"name"`
	Email  string   `json:"email"`
	Phones []Phone  `json:"phones"`
}

type Phone struct {
	Number string `json:"number"`
	Type   int    `json:"type"`
}

func main() {
	// Create test data
	personJSON := PersonJSON{
		ID:    123,
		Name:  "Alice",
		Email: "alice@example.com",
		Phones: []Phone{
			{Number: "555-1234", Type: 0},
			{Number: "555-4321", Type: 1},
		},
	}
	
	personProto := &person.Person{
		Id:    123,
		Name:  "Alice",
		Email: "alice@example.com",
		Phones: []*person.Person_PhoneNumber{
			{Number: "555-1234", Type: person.Person_MOBILE},
			{Number: "555-4321", Type: person.Person_HOME},
		},
	}

	// Benchmark JSON
	jsonStart := time.Now()
	iterations := 100000
	
	for i := 0; i < iterations; i++ {
		data, err := json.Marshal(&personJSON)
		if err != nil {
			log.Fatal(err)
		}
		
		var p PersonJSON
		err = json.Unmarshal(data, &p)
		if err != nil {
			log.Fatal(err)
		}
	}
	
	jsonDuration := time.Since(jsonStart)

	// Benchmark Protocol Buffers
	protoStart := time.Now()
	
	for i := 0; i < iterations; i++ {
		data, err := proto.Marshal(personProto)
		if err != nil {
			log.Fatal(err)
		}
		
		var p person.Person
		err = proto.Unmarshal(data, &p)
		if err != nil {
			log.Fatal(err)
		}
	}
	
	protoDuration := time.Since(protoStart)

	fmt.Printf("JSON: %v for %d iterations\n", jsonDuration, iterations)
	fmt.Printf("Protocol Buffers: %v for %d iterations\n", protoDuration, iterations)
	fmt.Printf("Protocol Buffers is %.2f times faster\n", float64(jsonDuration)/float64(protoDuration))
}

Real-World Challenges and Solutions

In my projects, I've faced several challenges when working with serialization:

Challenge 1: Large JSON Payloads

When dealing with very large JSON responses from an API, unmarshaling the entire payload could consume significant memory.

Solution: Use JSON streaming with json.Decoder for incremental processing:

func processLargeJSON(r io.Reader) error {
	decoder := json.NewDecoder(r)
	
	// Read opening brace
	if _, err := decoder.Token(); err != nil {
		return err
	}
	
	// Process each object in an array
	for decoder.More() {
		var item map[string]interface{}
		if err := decoder.Decode(&item); err != nil {
			return err
		}
		
		// Process each item individually
		processItem(item)
	}
	
	return nil
}

Challenge 2: Backward Compatibility with Protocol Buffers

When evolving our APIs, we needed to ensure backward compatibility while adding new fields.

Solution: Follow Protocol Buffers' compatibility rules:

Never change field numbers
Only add optional fields
Use reserved fields for removed fields

message User {
  int32 id = 1;
  string name = 2;
  reserved 3;  // Previously 'address', now removed
  reserved "address";
  string email = 4;
  optional string phone = 5;  // New field
}

Challenge 3: YAML Configuration with Environment Variables

We needed to support environment variable substitution in YAML config files.

Solution: Implemented a custom unmarshaler that processes environment variables:

func processEnvVars(in []byte) []byte {
	r := regexp.MustCompile(`\${([^}:]+)(:([^}]*))?}`)
	return r.ReplaceAllFunc(in, func(match []byte) []byte {
		parts := r.FindSubmatch(match)
		envVar := string(parts[1])
		
		// Get environment variable
		val := os.Getenv(envVar)
		if val == "" && len(parts) > 2 {
			// Use default value if provided
			val = string(parts[3])
		}
		
		return []byte(val)
	})
}

func loadConfig(path string) (*Config, error) {
	data, err := ioutil.ReadFile(path)
	if err != nil {
		return nil, err
	}
	
	// Process environment variables
	processed := processEnvVars(data)
	
	var config Config
	if err := yaml.Unmarshal(processed, &config); err != nil {
		return nil, err
	}
	
	return &config, nil
}

Conclusion

Serialization and deserialization are fundamental operations in modern software development. Go provides excellent built-in support for common formats like JSON and XML, while the ecosystem offers robust libraries for formats like Protocol Buffers and YAML.

When choosing a serialization format, consider your specific requirements:

For web APIs and broad compatibility, JSON is often the best choice
For configuration files, YAML offers better readability
For high-performance microservices, Protocol Buffers provides excellent efficiency
For Go-specific internal data transfer, Gob can be a good option

By understanding the strengths and weaknesses of each format, you can make informed decisions that lead to more efficient and maintainable systems.

What serialization formats do you use in your Go projects? Share your experiences in the comments!

MongoDB Performance Tips for Node.js Applications

Tue, 26 Mar 2024 00:00:00 GMT

MongoDB Performance Tips for Node.js Applications

MongoDB is a popular choice for Node.js applications due to its flexible document model and JavaScript-friendly query language. However, as your application grows, you might encounter performance issues that can impact user experience. In this post, I'll share some practical tips for optimizing MongoDB performance in Node.js applications.

Understanding Indexes

Indexes are one of the most powerful tools for improving query performance. Without proper indexes, MongoDB has to scan every document in a collection to find matches for your query.

// Creating a simple index
await db.collection('users').createIndex({ email: 1 });

// Creating a compound index
await db.collection('products').createIndex({ category: 1, created_at: -1 });

// Creating a text index for search
await db.collection('articles').createIndex({ title: 'text', content: 'text' });

To see which indexes exist on a collection:

const indexes = await db.collection('users').indexes();
console.log(indexes);

Always check if your queries are using indexes with explain():

const explanation = await db.collection('users')
  .find({ email: 'user@example.com' })
  .explain('executionStats');

console.log(explanation.executionStats.executionTimeMillis);
console.log(explanation.executionStats.totalDocsExamined);

Efficient Query Patterns

Use Projections to Return Only Required Fields

// Bad: Returns all fields
const user = await db.collection('users').findOne({ _id: userId });

// Good: Returns only needed fields
const user = await db.collection('users').findOne(
  { _id: userId },
  { projection: { name: 1, email: 1, role: 1 } }
);

Avoid Negation Operators

Negation operators like $ne, $nin, and $not often can't use indexes efficiently.

// Less efficient - may require full collection scan
const users = await db.collection('users').find({ status: { $ne: 'inactive' } }).toArray();

// More efficient - can use index on status
const users = await db.collection('users').find({ status: 'active' }).toArray();

Use Aggregation for Complex Queries

const result = await db.collection('orders').aggregate([
  // Match stage - use indexes
  { $match: { status: 'completed', created_at: { $gte: new Date('2023-01-01') } } },
  
  // Group stage
  { $group: {
    _id: '$customer_id',
    totalSpent: { $sum: '$total' },
    orderCount: { $sum: 1 }
  }},
  
  // Sort stage
  { $sort: { totalSpent: -1 } },
  
  // Limit stage
  { $limit: 10 }
]).toArray();

Connection Management with Mongoose

If you're using Mongoose, make sure to set up connection pooling correctly:

const mongoose = require('mongoose');

mongoose.connect('mongodb://localhost:27017/myapp', {
  // Maintain up to 10 socket connections
  poolSize: 10,
  // If a connection is idle for more than 30 seconds, close it
  socketTimeoutMS: 30000,
  // Give up connecting to MongoDB after 5 seconds
  connectTimeoutMS: 5000,
  // Keep trying to reconnect for 60 seconds
  serverSelectionTimeoutMS: 60000,
  // Never stop trying to reconnect
  useUnifiedTopology: true
});

const db = mongoose.connection;
db.on('error', console.error.bind(console, 'connection error:'));
db.once('open', function() {
  console.log('Connected to MongoDB');
});

Pagination Best Practices

Pagination is essential for performance when dealing with large datasets.

Offset-Based Pagination

const pageSize = 20;
const pageNumber = req.query.page || 1;
const skip = (pageNumber - 1) * pageSize;

const products = await db.collection('products')
  .find({})
  .sort({ created_at: -1 })
  .skip(skip)
  .limit(pageSize)
  .toArray();

Cursor-Based Pagination (More Efficient)

const pageSize = 20;
const lastId = req.query.lastId;

const query = lastId 
  ? { _id: { $lt: new ObjectId(lastId) } } 
  : {};

const products = await db.collection('products')
  .find(query)
  .sort({ _id: -1 })
  .limit(pageSize)
  .toArray();

// Return the last ID for the next page
const nextLastId = products.length > 0 
  ? products[products.length - 1]._id 
  : null;

Handling Large Write Operations

For bulk operations, use the bulk operation API:

const bulkOp = db.collection('products').initializeUnorderedBulkOp();

// Add operations to the bulk
for (const product of productsToUpdate) {
  bulkOp.find({ _id: product._id }).updateOne({
    $set: { price: product.price, stock: product.stock }
  });
}

// Execute all operations
const result = await bulkOp.execute();
console.log(`Modified ${result.nModified} documents`);

Avoiding N+1 Query Problems

N+1 query problems occur when you fetch a list of items and then make a separate query for each item's related data.

// Bad approach (N+1 queries)
const orders = await Order.find({ status: 'processing' });

for (const order of orders) {
  // This makes a separate query for each order!
  const customer = await Customer.findById(order.customer_id);
  order.customerName = customer.name;
}

// Better approach (2 queries)
const orders = await Order.find({ status: 'processing' });
const customerIds = orders.map(order => order.customer_id);
const customers = await Customer.find({ _id: { $in: customerIds } });

// Create a lookup map
const customerMap = {};
customers.forEach(customer => {
  customerMap[customer._id] = customer;
});

// Attach customer data to orders
orders.forEach(order => {
  order.customerData = customerMap[order.customer_id];
});

Using MongoDB Change Streams for Real-time Updates

Change streams allow applications to access real-time data changes:

const changeStream = db.collection('orders').watch();

changeStream.on('change', (change) => {
  console.log('Received change:', change);
  
  // Handle different operations
  switch (change.operationType) {
    case 'insert':
      notifyNewOrder(change.fullDocument);
      break;
    case 'update':
      updateOrderStatus(change.documentKey._id, change.updateDescription);
      break;
    case 'delete':
      removeOrderFromUI(change.documentKey._id);
      break;
  }
});

Schema Design Considerations

MongoDB's flexible schema can be both a blessing and a curse. Here are some guidelines:

Embed related data when possible, especially for one-to-one or one-to-few relationships
Use references for one-to-many relationships where the "many" side grows unbounded
Denormalize when needed for performance, but be careful about keeping data in sync
Use appropriate data types (e.g., Date for dates, not strings)
Consider document size limits (16MB per document)

// Example of embedding (one-to-few)
const user = {
  _id: ObjectId("..."),
  name: "John Doe",
  email: "john@example.com",
  addresses: [
    { type: "home", street: "123 Main St", city: "Boston", zip: "02101" },
    { type: "work", street: "456 Corp Ave", city: "Boston", zip: "02101" }
  ]
};

// Example of referencing (one-to-many)
const customer = {
  _id: ObjectId("..."),
  name: "Jane Smith",
  email: "jane@example.com"
};

const orders = [
  { _id: ObjectId("..."), customer_id: customer._id, items: [...], total: 99.99 },
  { _id: ObjectId("..."), customer_id: customer._id, items: [...], total: 149.99 }
];

Monitoring and Profiling

Always monitor your MongoDB performance in production:

Enable database profiling to identify slow queries:

db.setProfilingLevel(1, { slowms: 100 }); // Log operations that take more than 100ms

Use MongoDB Atlas monitoring or set up your own monitoring with tools like Prometheus and Grafana
Check slow query logs regularly

Conclusion

Optimizing MongoDB performance is a continuous process that involves proper schema design, indexing, query optimization, and monitoring. By following these best practices, you can ensure your Node.js applications remain responsive and efficient even as your data grows.

In my next post, I'll dive deeper into MongoDB's aggregation framework and show how you can use it to build powerful analytics features for your applications.

What MongoDB performance challenges have you faced in your projects? Share your experiences in the comments!

Younes Bouchbouk | Blog

Why Golang Is Ideal for Microservices Architecture (2026 and Beyond)

Why Golang Is Ideal for Microservices Architecture (2026 and Beyond)

1. Unmatched Performance and Efficiency

2. Built-in Concurrency That actually Works

3. Simplicity at Scale

4. Cloud-Native DNA

5. The Future (2026 and Beyond)

Conclusion

Mastering Context in Go: From Basics to Production-Grade Patterns

Mastering Context in Go: From Basics to Production-Grade Patterns

The problem: coordinating lifecycle across goroutines

What is context.Context?

Use Case 1: Cancellation (stop work when it's no longer needed)

Example: Simulate cancellable work

Why this matters in production

Use Case 2: Timeouts (fail fast instead of hanging forever)

Example: Context with timeout

Why defer cancel() matters

Use Case 3: Value propagation (request-scoped metadata)

Example: Adding and retrieving values

In production: structured logging with trace IDs

Practical implementation: ContextManager

Implementation

The tricky one: ExecuteWithContext

WaitForCompletion: A context-aware sleep

Real-world example: Processing items with cancellation

Testing context behavior

Test 1: Cancellation propagates correctly

Test 2: Timeout fires correctly

Test 3: Values survive context derivation

Production patterns I actually use

1. Always derive from request context in HTTP handlers

2. Add timeouts at service boundaries

3. Use typed keys for context values

4. Propagate cancellation to all goroutines

5. Graceful shutdown with context

Common pitfalls (and how to avoid them)

Pitfall 1: Not checking context errors

Pitfall 2: Using context for optional parameters

Pitfall 3: Not calling cancel()

Pitfall 4: Passing nil context

When NOT to use context

Performance considerations

Conclusion: Context is about coordination, not configuration

Benchmarking in Go: Measure Performance with Real Examples (Sorting + strings.Builder)

Benchmarking in Go: Measure Performance with Real Examples (Sorting + strings.Builder)

What is benchmarking in Go?

Example 1: Sorting (bubble sort vs slices.Sort)

The “slow” version (bubble sort)

The optimized version (standard library sort)

Correctness test (optimized must match slow)

Benchmark code (the important part)

Real results (my run)

Example 2: Building strings (+= vs strings.Builder)

The “slow” version (repeated +=)

The optimized version (strings.Builder)

Correctness test (optimized must match slow)

Real results (my run)

How to run benchmarks (exact commands)

Running one benchmark (or a subset)

How to read Go benchmark results (ns/op, B/op, allocs/op)

Best practices (what I actually do in real Go services)

Conclusion

Circuit Breaker Pattern in Go (with a Practical Implementation)

Circuit Breaker Pattern in Go (with a Practical Implementation)

The problem it solves

How a circuit breaker works (3 states)

1) Closed (normal)

2) Open (fail fast)

3) Half-Open (probe recovery)

State transition diagram

Implementation concepts in Go

Reference implementation (Go)

Production hardening (what I’d improve before shipping)

Testing strategy (high-value tests)

Where to use it

Conclusion

Data Serialization and Deserialization in Golang

Data Serialization and Deserialization in Golang

Why `defer cancel()` matters

Example 1: Sorting (bubble sort vs `slices.Sort`)

Example 2: Building strings (`+=` vs `strings.Builder`)

The “slow” version (repeated `+=`)

The optimized version (`strings.Builder`)