Building Custom CI Runners for Flutter with Kubernetes

The Hybrid Reality of Flutter CI/CD

When you are managing a fleet of containerized applications in Panama, you quickly learn that the 'cloud-only' philosophy is a luxury you cannot afford. In our shipping operations, we deal with a hybrid topology: AWS provides our public-facing APIs, but our heavy processing and local logistics database live on bare-metal servers in our regional data center. When we started building a Flutter-based mobile dashboard for our dock logistics team, we hit a wall. Flutter builds are notoriously resource-heavy. Compiling iOS and Android artifacts for a complex logistics app involves fetching massive SDKs, running intensive build tasks, and managing stateful cache volumes.

If you use managed CI runners, you are at the mercy of their availability and their egress costs. When you pull a 2GB Flutter engine build through a NAT gateway into an AWS VPC, and then push that build artifact across AZs to sync with an on-premises testing farm, you aren't just paying for compute; you are paying a 'convenience tax' that can balloon your monthly bill into the thousands. We needed a custom solution: CI runners that sit inside our own Kubernetes clusters, respecting our network topology and minimizing latency.

Designing the Infrastructure Strategy

Standard CI runners are often "dumb" nodes—they don't care where they land, and they don't care how much data they pull. In a hybrid environment, this is a recipe for disaster. Our Kubernetes clusters are spread across two AWS zones and one physical location. If a CI runner boots up in AWS but needs to pull data from our on-prem storage array, we are effectively burning money on every packet.

To solve this, we architected a custom runner solution based on Kubernetes Pod orchestration that uses specific node affinity and anti-affinity rules to ensure that builds occur as close to the required data as possible. We opted to use custom Docker images containing the Flutter SDK, the Android NDK, and the necessary Java versions, all pre-warmed on the local nodes to avoid redundant pulls during the pipeline execution.

Step-by-Step Implementation of Custom Runners

Building your own runner system requires three components: a trigger mechanism, the orchestration controller, and the runner image itself. Here is how we implemented the core logic.

1. Define the Runner Image: We built a hardened image that includes the Flutter stable channel, fastlane, and required build tools. We pin the versions to avoid 'it worked yesterday' syndrome.
2. Custom Namespace Isolation: Every pipeline execution gets its own dedicated namespace with resource quotas to prevent a single build from starving our production API services.
3. Node Affinity Rules: We use node labels to ensure that builds targeting on-premises physical assets stay on physical nodes, while general testing artifacts are offloaded to spot instances in AWS.
4. Configuring the K8s manifest: Below is a template for the runner deployment that manages resource constraints and affinity.

apiVersion: v1
kind: Pod
metadata:
  name: flutter-build-runner
  labels:
    app: flutter-ci
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: 'topology.kubernetes.io/zone'
            operator: In
            values: ['us-east-1a']
  containers:
  - name: builder
    image: registry.internal/flutter-builder:2.10.0
    resources:
      requests:
        cpu: "4"
        memory: "8Gi"
      limits:
        cpu: "8"
        memory: "16Gi"
    volumeMounts:
    - name: flutter-cache
      mountPath: /root/.pub-cache
  volumes:
  - name: flutter-cache
    persistentVolumeClaim:
      claimName: flutter-cache-pvc

Managing Cross-AZ Data Traffic

One of the biggest hurdles when building mobile apps in Kubernetes is the caching layer. Flutter's .pub-cache can easily reach several gigabytes. If your runner is scheduled on a random node, it will spend the first six minutes of every pipeline downloading dependencies.

We solved this by using Local Persistent Volumes (LPV) on our on-premises nodes. By pinning the build to specific hardware, the disk cache persists across container restarts. This turns an 8-minute build into a 2-minute build. The key is setting the schedulerName in your pod spec if you are using a custom scheduler, or simply relying on nodeAffinity if you are using the default scheduler but with strict topology constraints.

Furthermore, for our AWS-resident runners, we utilize EFS with Provisioned Throughput to ensure that high-IOPS demands during the Gradle sync don't choke the network performance. While EFS is more expensive than standard storage, it is infinitely cheaper than the cross-AZ traffic charges we incurred when our runners were trying to sync data from the primary RDS instance in a different availability zone.

Troubleshooting and Operational Best Practices

Even with a custom setup, things break. The most frequent issue we see is the "Orphaned Build" scenario. When a pod is evicted by the K8s scheduler due to a sudden spike in production load, the build fails midway, leaving behind a locked .pub-cache directory.

Pro Tips for the Hybrid Operator:

1. Always implement a pre-build cleanup script: Add an initContainer that checks for stale locks in your cache directory before the main Flutter build starts.
2. Use PriorityClasses: Set your production APIs to system-node-critical and your CI runners to a lower priority. This ensures that when the cluster gets crowded, Kubernetes kills your CI builds first, not your customer-facing APIs.
3. Monitor Cache Hit Rates: Use Prometheus to track the duration of the 'dependency fetch' stage of your builds. If this time starts trending upwards, it means your cache volume is being detached or rotated too frequently.
4. Tainting and Tolerations: Keep your build nodes tainted. You don't want a stray microservice pod deciding to land on your high-spec build node just because it had a little extra RAM available.

Integrating with CI Tools

Whether you are using GitLab CI, GitHub Actions (Self-Hosted), or a custom webhook listener, the principle remains the same: the CI orchestrator should interact with the Kubernetes API to spin up a job, wait for the exit code, and then tear the pod down.

In our environment, we use a small Golang controller that watches for new pipeline events. It doesn't use the standard GitLab runner daemon because it's too opaque. Instead, our controller reads the job requirements, checks the current load on our on-premises bare metal, and decides whether to schedule the pod there or in the cloud. This 'intelligence' is what bridges the gap between a pure cloud setup and our hybrid reality. It’s not just about running code; it’s about understanding the cost of infrastructure and the physical location of the data the build needs to process.

Conclusion

Building custom CI runners for Flutter is not about over-engineering. It is about taking back control of your infrastructure costs and reliability. The default Kubernetes scheduler is designed for high-availability microservices, not for compute-heavy, cache-dependent build pipelines. By manually managing your node affinity, leveraging local storage, and strictly controlling resource quotas, you can transform your CI/CD pipeline from a source of frustration into a stable, cost-efficient engine. In a hybrid environment, your infrastructure is your product. Treat your CI runners with the same architectural rigor as your production APIs, and you will see the impact immediately—not just in build times, but in your AWS invoice at the end of the month. Don't settle for the default configuration when your business logic demands a specialized, topology-aware approach.