Insomnia Technologies

Introduction: Why We Migrated (And What Went Wrong)

Three years ago, our team managed a 2-million-line .NET monolith that took 45 minutes to build, required 4-hour maintenance windows for deployments, and crashed whenever the marketing team ran a promotion. We decided to migrate to microservices. What followed was 18 months of distributed transaction failures, Kafka consumer lag spikes, and a 3 AM page when our Saga compensations created infinite loops.

This guide isn't theoretical. It's the distilled experience of running 23 .NET microservices in Kubernetes, processing 50M+ Kafka messages daily. I'll show you the patterns that work, the traps that will cost you weeks of debugging, and when you should just keep your monolith.

Production .NET microservices architecture with Kafka and Kubernetes — Our current production architecture: 23 services, 3 Kafka clusters, 150+ Kubernetes pods. It took 18 months to get here safely.

Monolith vs Microservices: The Real Trade-Offs

Consultants love microservices. Developers often hate them. The truth is nuanced: microservices solve organizational scaling problems, not just technical ones. But they introduce distributed system complexity that will haunt you if you're not prepared.

Factor	Monolith	Microservices	Reality Check
Development Speed	Fast initially	Slower per feature	Microservices win at 50+ developers
Deployment Risk	High (all or nothing)	Low (isolated failures)	But you need solid CI/CD
Debugging	Easy (single process)	Hard (distributed tracing required)	Expect 3x longer MTTR initially
Database Scaling	Vertical only	Horizontal per service	Finally escape shared DB bottlenecks
Team Autonomy	Low (coordination hell)	High (own your stack)	Requires strong API contracts

💡 When to Stay Monolithic

If you have <10 developers, <50k daily users, and deployment frequency < once/day, microservices will slow you down. We learned this the hard way—our first microservice migration failed because we solved problems we didn't have yet.

Strategic Service Boundaries: Lessons from DDD

The most common microservices mistake is creating boundaries based on technical layers (API, Service, Repository) instead of business capabilities. After three failed attempts, we finally succeeded using Domain-Driven Design:

●
Identify Bounded Contexts: Map business domains (Orders, Inventory, Payments), not CRUD operations.
●
Find the Ubiquitous Language: If "Customer" means different things to Sales vs Billing, they're separate contexts.
●
Analyze Data Coupling: Services sharing database tables will become a distributed monolith.
●
Study Team Structure: Conway's Law is real—services align with communication patterns.

Our Breaking Point: We initially split by "Orders API" and "Orders Database." Six months later, we couldn't deploy either independently. The fix? Separate "Order Placement" (customer-facing) from "Order Fulfillment" (warehouse-facing) based on different business workflows.

ASP.NET Core Microservice Template: Production-Ready Structure

Every microservice should be independently deployable with its own CI/CD pipeline. Here's the structure we standardized across 23 services after countless refactoring cycles:

Production Service Structure

text

              OrderService/
├── src/
│   ├── OrderService.API/           # HTTP/gRPC entry points
│   │   ├── Controllers/
│   │   ├── HealthChecks/
│   │   └── Program.cs
│   │
│   ├── OrderService.Application/   # Use cases, commands, queries
│   │   ├── Commands/
│   │   ├── Queries/
│   │   ├── EventHandlers/
│   │   └── Interfaces/
│   │
│   ├── OrderService.Domain/        # Entities, value objects, events
│   │   ├── Entities/
│   │   ├── ValueObjects/
│   │   └── Events/
│   │
│   └── OrderService.Infrastructure/ # External concerns
│       ├── Persistence/
│       ├── Kafka/
│       ├── Redis/
│       └── ExternalServices/
│
├── tests/
── Dockerfile
└── kubernetes/

💡 Template Repository Pro Tip

Create a dotnet new template for your service structure. We reduced new service setup from 3 days to 20 minutes. Include pre-configured health checks, logging, and Kafka consumers.

Communication Patterns: Choosing the Right Tool

Not all communication is equal. We learned this when our synchronous HTTP calls created cascading failures during a Kafka outage. Here's our decision framework:

Pattern	Use When	Avoid When	Our Experience
REST/HTTP	External APIs, simple queries	High-volume internal calls	Great for BFF pattern, terrible for chatty internal communication
gRPC	Internal service-to-service	External clients (browser)	5-10x faster than REST, but debugging is harder
Kafka Events	Async workflows, eventual consistency	Need immediate response	Saved us from cascading failures
Redis Pub/Sub	Real-time notifications	Message durability required	Fast but messages lost on restart

The 80/20 Rule: 80% of our inter-service communication is async via Kafka. 20% is gRPC for real-time queries. We eliminated synchronous HTTP between services after a 2-hour outage caused by one slow service.

Kafka Event Streaming: Beyond Basic Pub/Sub

Kafka is powerful but dangerous. Misconfigured, it will silently drop messages or create infinite processing loops. Here are the patterns that keep our 50M daily messages reliable:

ReliableKafkaProducer.cs

csharp

              public class ReliableEventPublisher
{
    public async Task Publish<T>(
        string topic,
        T eventData,
        string correlationId,
        CancellationToken ct = default)
    {
        // CRITICAL: Include correlation ID for tracing
        var headers = new Headers
        {
            { "correlation-id", Encoding.UTF8.GetBytes(correlationId) },
            { "event-type", Encoding.UTF8.GetBytes(typeof(T).Name) },
            { "timestamp", Encoding.UTF8.GetBytes(DateTime.UtcNow.ToString("O")) }
        };

        var message = new Message<string, string>
        {
            Key = eventData.AggregateId.ToString(), // Partition by aggregate
            Value = JsonSerializer.Serialize(eventData),
            Headers = headers
        };

        try
        {
            var result = await _producer.ProduceAsync(
                topic,
                message,
                ct);

            _logger.LogInformation(
                "Event published: {EventType} to {Topic} partition {Partition}",
                typeof(T).Name, topic, result.Partition.Value);
        }
        catch (ProduceException<string, string> ex)
        {
            // CRITICAL: Don't lose events—store in outbox
            await _outboxStore.SaveFailedEvent(
                topic, message, ex);
            throw;
        }
    }
}

💡 Partition Strategy Warning

Always partition by aggregate ID (Order ID, Customer ID), never round-robin. We learned this when order events arrived out of sequence, causing inventory mismatches. Kafka guarantees order ONLY within a partition.

Kafka Consumer Groups: Scaling Without Chaos

Running multiple Kubernetes pods with Kafka consumers seems simple until you encounter rebalancing storms. Here's how we handle it:

ResilientConsumer.cs

csharp

              public class OrderEventConsumer : BackgroundService
{
    protected override async Task ExecuteAsync(
        CancellationToken stoppingToken)
    {
        _consumer.Subscribe("orders");

        while (!stoppingToken.IsCancellationRequested)
        {
            try
            {
                var consumeResult = _consumer.Consume(
                    stoppingToken);

                // CRITICAL: Process with idempotency
                if (await IsDuplicate(
                    consumeResult.Message.Offset))
                {
                    _consumer.Commit(consumeResult);
                    continue;
                }

                await ProcessMessage(
                    consumeResult.Message.Value,
                    stoppingToken);

                _consumer.Commit(consumeResult);
            }
            catch (ConsumeException ex)
            {
                // CRITICAL: Log offset to avoid infinite loop
                _logger.LogError(ex,
                    "Error consuming offset {Offset}",
                    ex.ConsumerRecord?.Offset.Value);

                // Don't commit—retry on next poll
            }
            catch (OperationCanceledException)
            {
                // Graceful shutdown
                break;
            }
        }

        _consumer.Close();
    }

    private async Task ProcessMessage(
        string message,
        CancellationToken ct)
    {
        // Use Polly for retries with backoff
        await _policyRegistry
            .Get<AsyncPolicy>("KafkaConsumer")
            .ExecuteAsync(
                () => HandleEvent(message, ct),
                ct);
    }
}

Consumer Lag Monitoring: We alert when lag exceeds 10,000 messages or 5 minutes. Use Prometheus metrics (kafka_consumer_lag) and Grafana dashboards. Our worst incident: 2M message backlog from a deadlocked database—took 6 hours to recover.

Saga Pattern: Managing Distributed Transactions

Distributed transactions are the hardest problem in microservices. We tried two-phase commit (don't do this) before settling on Sagas. Here are the patterns that prevent data corruption:

Saga Step	Forward Action	Compensation	Idempotency Key
1. Create Order	INSERT Order (Status=Pending)	UPDATE Order (Status=Cancelled)	Order ID
2. Reserve Inventory	UPDATE Inventory (Qty -= N)	UPDATE Inventory (Qty += N)	Order ID + Product ID
3. Process Payment	Charge Credit Card	Issue Refund	Order ID + Payment ID
4. Confirm Order	UPDATE Order (Status=Confirmed)	N/A (final state)	Order ID

ChoreographySaga.cs

csharp

              // EVENT-DRIVEN SAGA (Choreography)
// Each service publishes events that trigger next steps

public class OrderSagaOrchestrator
{
    // Step 1: Order Created
    [KafkaConsumer("order-created")]
    public async Task Handle(OrderCreatedEvent evt)
    {
        try
        {
            await _inventoryService.Reserve(
                evt.OrderId,
                evt.Items);

            await _eventPublisher.Publish(
                "inventory-reserved",
                new InventoryReservedEvent
                {
                    OrderId = evt.OrderId
                });
        }
        catch (InsufficientInventoryException)
        {
            // CRITICAL: Trigger compensation
            await _eventPublisher.Publish(
                "order-failed",
                new OrderFailedEvent
                {
                    OrderId = evt.OrderId,
                    Reason = "Out of stock"
                });
        }
    }

    // Step 2: Payment Failed → Compensate
    [KafkaConsumer("payment-failed")]
    public async Task Handle(PaymentFailedEvent evt)
    {
        // Release inventory (compensation)
        await _inventoryService.Release(evt.OrderId);

        // Cancel order
        await _orderService.Cancel(
            evt.OrderId,
            "Payment failed");
    }
}

💡 Saga Debugging Nightmare

Implement saga state tracking. We store each step in a SagaState table with timestamps. When things break (and they will), you can replay or manually compensate. Without this, debugging is impossible.

Outbox Pattern: Preventing Dual-Write Failures

The dual-write problem killed our data consistency for 3 weeks: database commit succeeds, but Kafka publish fails (or vice versa). The Outbox Pattern is the only reliable solution:

OutboxPattern.cs

csharp

              // STEP 1: Save entity AND outbox message in SAME transaction
public async Task CreateOrder(CreateOrderCommand cmd)
{
    await using var transaction = await _dbContext
        .Database.BeginTransactionAsync();

    try
    {
        var order = new Order(cmd);
        _dbContext.Orders.Add(order);
        await _dbContext.SaveChangesAsync();

        // CRITICAL: Store event in outbox table
        var outboxMessage = new OutboxMessage
        {
            Id = Guid.NewGuid(),
            EventType = "OrderCreated",
            Payload = JsonSerializer.Serialize(
                new OrderCreatedEvent
                {
                    OrderId = order.Id,
                    CustomerId = cmd.CustomerId
                }),
            CreatedAt = DateTime.UtcNow,
            Processed = false // Flag for publisher
        };

        _dbContext.OutboxMessages.Add(outboxMessage);
        await _dbContext.SaveChangesAsync();

        await transaction.CommitAsync();
    }
    catch
    {
        await transaction.RollbackAsync();
        throw; // Neither order nor event saved
    }
}

// STEP 2: Separate background worker publishes events
public class OutboxPublisher : BackgroundService
{
    protected override async Task ExecuteAsync(
        CancellationToken ct)
    {
        while (!ct.IsCancellationRequested)
        {
            var messages = await _dbContext
                .OutboxMessages
                .Where(m => !m.Processed)
                .Take(100)
                .ToListAsync(ct);

            foreach (var msg in messages)
            {
                try
                {
                    await _kafkaProducer.Publish(
                        "orders",
                        msg.Payload);

                    msg.Processed = true;
                    msg.ProcessedAt = DateTime.UtcNow;
                }
                catch (Exception ex)
                {
                    _logger.LogError(ex,
                        "Failed to publish outbox message {Id}",
                        msg.Id);
                    // Don't mark as processed—retry later
                }
            }

            await _dbContext.SaveChangesAsync(ct);
            await Task.Delay(TimeSpan.FromSeconds(5), ct);
        }
    }
}

CQRS: When Separation Actually Helps

CQRS isn't always necessary, but it saved us when read/write ratios hit 50:1. Here's our pragmatic approach (not the academic purity):

PragmaticCQRS.cs

csharp

              // WRITE SIDE (Command Handler)
public class CreateOrderCommandHandler
{
    public async Task<Guid> Handle(
        CreateOrderCommand cmd,
        CancellationToken ct)
    {
        var order = Order.Create(cmd);

        await _repository.Add(order, ct);

        // Domain events automatically published
        await _eventPublisher.PublishDomainEvents(
            order.DomainEvents);

        return order.Id;
    }
}

// READ SIDE (Query Handler with Dapper for performance)
public class GetOrderDetailsQueryHandler
{
    public async Task<OrderDetailsDto> Handle(
        GetOrderDetailsQuery query,
        CancellationToken ct)
    {
        // Use Dapper for complex joins
        return await _connection.QueryFirstOrDefaultAsync<
            OrderDetailsDto>(@"
            SELECT o.*, c.Name as CustomerName,
                   p.Name as ProductName
            FROM Orders o
            JOIN Customers c ON o.CustomerId = c.Id
            JOIN OrderItems oi ON o.Id = oi.OrderId
            JOIN Products p ON oi.ProductId = p.Id
            WHERE o.Id = @OrderId",
            new { query.OrderId });
    }
}

// SEPARATE READ DATABASE (eventually consistent)
// Updated via Kafka event handlers
public class OrderReadModelHandler
{
    [KafkaConsumer("order-created")]
    public async Task Handle(OrderCreatedEvent evt)
    {
        // Denormalize into read-optimized schema
        await _readDb.ExecuteAsync(@"
            INSERT INTO OrderReadModel
            (Id, CustomerName, Total, CreatedAt)
            VALUES (@Id, @CustomerName, @Total, @CreatedAt)",
            new
            {
                evt.OrderId,
                evt.CustomerName,
                evt.Total,
                evt.CreatedAt
            });
    }
}

💡 CQRS Complexity Warning

Start with CQRS only for reads. We wasted 2 months implementing full CQRS with separate write databases before realizing 90% of our services didn't need it. Use it when read/write patterns differ significantly.

Redis Caching: Distributed Cache Done Right

Redis is essential for microservices, but cache invalidation in distributed systems is brutal. Here are our battle-tested patterns:

CacheAsideWithInvalidation.cs

csharp

              public class CachedProductService
{
    public async Task<ProductDto> GetProduct(
        Guid id,
        CancellationToken ct)
    {
        var cacheKey = $"product:{id}";

        // Try cache first
        var cached = await _cache.GetAsync<ProductDto>(
            cacheKey, ct);

        if (cached != null)
            return cached;

        // Cache miss - load from DB
        var product = await _repository.GetById(id, ct);

        if (product == null)
            return null;

        // CRITICAL: Use distributed lock to prevent cache stampede
        var lockKey = $"lock:product:{id}";
        if (await _lock.TryAcquire(lockKey, TimeSpan.FromSeconds(5)))
        {
            try
            {
                // Double-check after acquiring lock
                cached = await _cache.GetAsync<ProductDto>(
                    cacheKey, ct);

                if (cached != null)
                    return cached;

                // Set cache with reasonable TTL
                await _cache.SetAsync(
                    cacheKey,
                    product.ToDto(),
                    TimeSpan.FromMinutes(15), // Sliding expiration
                    ct);
            }
            finally
            {
                await _lock.Release(lockKey);
            }
        }
        else
        {
            // Another pod is loading - wait briefly
            await Task.Delay(100, ct);
            return await _cache.GetAsync<ProductDto>(
                cacheKey, ct);
        }

        return product.ToDto();
    }

    // CRITICAL: Invalidate on updates
    public async Task UpdateProduct(
        Guid id,
        UpdateProductCommand cmd)
    {
        await _repository.Update(id, cmd);

        // Invalidate cache immediately
        await _cache.RemoveAsync($"product:{id}");

        // Publish event for other services to invalidate
        await _eventPublisher.Publish(
            "product-updated",
            new ProductUpdatedEvent { ProductId = id });
    }
}

Kubernetes Deployment: From YAML to Production

Kubernetes is powerful but complex. We went from manual YAML to GitOps with ArgoCD. Here's our production deployment strategy:

production-deployment.yaml

yaml

              apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-service
  namespace: production
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0  # Zero downtime
  selector:
    matchLabels:
      app: order-service
  template:
    metadata:
      labels:
        app: order-service
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8080"
    spec:
      # CRITICAL: Resource limits prevent noisy neighbor
      containers:
      - name: order-service
        image: registry/order-service:v2.3.1
        ports:
        - containerPort: 8080
        resources:
          requests:
            memory: "256Mi"
            cpu: "250m"
          limits:
            memory: "512Mi"
            cpu: "500m"
        env:
        - name: KAFKA_BOOTSTRAP_SERVERS
          valueFrom:
            configMapKeyRef:
              name: kafka-config
              key: bootstrap-servers
        - name: ConnectionStrings__Default
          valueFrom:
            secretKeyRef:
              name: db-secrets
              key: connection-string
        # CRITICAL: Health checks for reliability
        livenessProbe:
          httpGet:
            path: /health/live
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /health/ready
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5
        # Graceful shutdown
        lifecycle:
          preStop:
            exec:
              command: ["/bin/sh", "-c", "sleep 10"]
      # Anti-affinity prevents single point of failure
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchLabels:
                  app: order-service
              topologyKey: kubernetes.io/hostname

💡 GitOps Pro Tip

Use ArgoCD or Flux for GitOps. We store all K8s manifests in Git, and ArgoCD automatically syncs to the cluster. Rollbacks are one command. No more kubectl apply -f from someone's laptop.

Observability: Debugging Distributed Systems

Without observability, microservices are a black box. We use the three pillars: logs, metrics, traces. Here's what actually works:

●
Distributed Tracing: OpenTelemetry + Jaeger. Every request gets a correlation ID that flows through Kafka, HTTP, and databases. Essential for finding bottlenecks.
●
Structured Logging: Serilog + Elasticsearch. Never use Console.WriteLine. Include correlation IDs, user IDs, and request IDs in every log.
●
Metrics: Prometheus + Grafana. Track: request rate, error rate, latency percentiles (P50, P95, P99), Kafka consumer lag, database connection pool usage.
●
Health Checks: /health/live (is process running?) and /health/ready (can handle traffic?). K8s uses these for auto-healing.

Our Worst Debugging Session: A payment timeout took 6 hours to diagnose. The issue: Kafka consumer lag → delayed event processing → payment service waiting for order confirmation. Without distributed tracing, we were blind.

Production Best Practices: Hard-Won Wisdom

●
✅ Implement idempotent consumers: Kafka delivers messages at-least-once. Duplicate detection is mandatory.
●
✅ Use circuit breakers: Polly prevents cascading failures. We use 50% error threshold over 10 requests.
●
✅ Version everything: API endpoints (/api/v1/orders), Kafka topics (orders.v2), database schemas.
●
✅ Set resource limits: Without CPU/memory limits, one service can starve the entire node.
●
✅ Test failure scenarios: Use Chaos Engineering (Chaos Mesh) to simulate pod kills, network partitions.
●
✅ Monitor consumer lag: Alert when lag > 10K messages or > 5 minutes.
●
✅ Use blue-green deployments: Test new version with 1% traffic before full rollout.
●
✅ Implement retry budgets: Don't retry infinitely—use exponential backoff with jitter.
●
✅ Document service dependencies: Maintain a service mesh diagram. Update it religiously.

"Microservices don't make problems disappear—they just make them distributed. Invest heavily in observability, or you'll be debugging in the dark."

Common Mistakes That Caused Our Outages

●
❌ Chatty services: 50 HTTP calls to fulfill one order. Fixed with API composition and denormalization.
●
❌ Shared databases: Two services writing to the same table. Result: schema change coordination hell.
●
❌ Ignoring Kafka partitioning: Round-robin partitioning caused out-of-order events. Lost $50K in inventory mismatches.
●
❌ No circuit breakers: One slow database query took down 5 dependent services.
●
❌ Caching everything: Cached user-specific data with wrong key. Users saw other users' orders.
●
❌ Synchronous event publishing: HTTP event publishing blocked main transaction. Switched to Outbox Pattern.
●
❌ Deploying all services together: Defeated the purpose of microservices. Now each service deploys independently.

Frequently Asked Questions

How many microservices should I start with?

Start with 3-5 services max. We began with 8 and couldn't manage the complexity. Grow organically: split when a service has >3 distinct responsibilities or when team size requires it. Remember: you can't go wrong by starting with a modular monolith.

Should I use Kafka or RabbitMQ for event streaming?

Use Kafka for: event sourcing, replay capability, high throughput (>10K msg/sec), stream processing. Use RabbitMQ for: simple pub/sub, work queues, low latency requirements. We use both—Kafka for domain events, RabbitMQ for background jobs.

How do you handle database migrations across services?

Each service owns its database migrations. We use Flyway for versioning. For breaking changes: 1) Deploy backward-compatible version, 2) Run migration, 3) Deploy version that uses new schema. Never break consumers.

What's your Kubernetes cluster size for 23 services?

3 control plane nodes (HA), 8 worker nodes (16 vCPU, 64GB RAM each). Runs ~150 pods total. We use cluster autoscaler to scale 3-12 nodes based on load. Cost: ~$2K/month on AWS EKS.

How do you manage configuration across services?

Azure App Configuration (or AWS AppConfig) for feature flags and dynamic config. Kubernetes ConfigMaps for static config. HashiCorp Vault for secrets. Never commit secrets to Git, even in private repos.

When should I use gRPC vs REST?

Use gRPC for: internal service-to-service, mobile clients (small payload), streaming. Use REST for: public APIs, browser clients, simple integrations. We use gRPC for 70% of internal communication, REST for external APIs.

Migration Strategy: From Monolith to Microservices

Don't rewrite—strangle. The Strangler Fig Pattern saved us from a 2-year rewrite disaster:

●
Phase 1 (Months 1-3): Extract low-risk, independent features (notifications, reporting). Build confidence.
●
Phase 2 (Months 4-9): Extract core domains one at a time (Orders, Inventory). Use API gateway to route traffic.
●
Phase 3 (Months 10-15): Extract complex domains (Payments, Pricing). Implement Saga Pattern for transactions.
●
Phase 4 (Months 16-18): Decommission monolith modules. Migrate data. Train teams on new architecture.

💡 Migration Reality Check

Expect 30% slower feature development during migration. We had to justify this to stakeholders upfront. The payoff: after 18 months, deployment frequency increased from weekly to 50+ per day.

Conclusion: Microservices Are a Journey, Not a Destination

Building production-ready .NET microservices isn't about adopting the latest frameworks—it's about managing complexity through proven patterns: event-driven architecture with Kafka, distributed transactions with Sagas, reliability through the Outbox Pattern, and scalability via Kubernetes. But more importantly, it's about knowing when NOT to use microservices, investing heavily in observability, and learning from failures. Our 23-service architecture took 18 months and countless outages to stabilize. Start small, automate everything, and never stop monitoring.

Ready to dive deeper? Check out our guides on [Kafka Consumer Best Practices], [Kubernetes Cost Optimization], and [Distributed Tracing with OpenTelemetry] to continue your microservices journey.