Evaluating serverless Container-as-a-Service platforms: Knative, Cloud Run and AWS Fargate

Serverless became popular through Function-as-a-Service, but the function programming model constrains running general-purpose applications. Platforms implementing Container-as-a-Service (CaaS) deliver software as container images: they ease migration of traditional applications, give control over the runtime, remove hard execution-time limits and reduce lock-in. The dissertation evaluates these platforms on two fronts — qualitative and quantitative.

Qualitative evaluation

Seven platforms were characterised against execution criteria (memory, CPU, storage, GPU, execution time limit, concurrent executions), scalability (autoscaling, autoscaling metric, scale to zero, warming), triggers, security and integration: Knative, OpenWhisk, OpenFaaS, Azure Container Instances, Google Cloud Run, AWS Lambda and AWS Fargate.

• Open source frameworks (Knative, OpenWhisk, OpenFaaS) run on top of orchestrators and give full control, but require management — and several rely on custom image templates, which pushes them away from pure CaaS.
• Public cloud services (Cloud Run, ACI, Fargate, Lambda) shift management to the provider in exchange for restrictions.
• Azure Container Instances and AWS Fargate keep containers continuously active: no scale to zero, but no cold starts either.
• Knative, OpenFaaS and Cloud Run allow a minimum container count — a hybrid model that mitigates cold starts.
• Selection criterion for the quantitative phase: only Knative, Cloud Run and AWS Fargate. OpenWhisk, OpenFaaS and Lambda require custom image templates; ACI has no autoscaling.

Experiment architecture

All infrastructure, benchmark images and load scripts are in the master-lab repository. Locally, the project spins up a Kubernetes cluster via Terraform for Knative; for Cloud Run and AWS Fargate, the same container set and scripts can be reused on each platform.

• Benchmark machines (BM) in two geographically close European regions: AWS Stockholm (eu-north-1) and GCP Hamina (europe-north1).
• Each BM boots via docker-compose with two containers: InfluxDB (time-series store for metrics, centralised on GCP) and a custom Jupyter Notebook image holding the automation scripts.
• The load agent is k6.io, launched by a bash script parameterised with URL, platform, test id, run id and the third independent variable.
• k6 streams per-request metrics into InfluxDB; Python scripts in Jupyter query, aggregate and plot them.
• Knative ran on a 3-node GKE cluster of e2-standard-4 VMs (16 GiB, 4 vCPUs), exposed through an Ingress. Platforms were configured without scale to zero, to avoid cold starts, and with a single container per instance.
• Benchmark images for C#, Go, Java, Node.js, PHP and Python, with yarn commands to build, run locally and push to GCP/AWS.
• Terraform and Docker Compose in the repository allow the test environment to be reproduced: local cluster, metrics collector and Grafana dashboard.

The seven tests

• T1 — Load: a sequence of low-cost GET requests (a Node.js “Hello World” endpoint) over 60 s with a single virtual user, measuring end-to-end latency.
• T2 — Concurrent load: latency, request count and success rate with virtual users (vus) ramping in parallel. It is the base for T4 to T7.
• T3 — Spike: elasticity, with a ramp-up and ramp-down phase to force scale-out and scale-in.
• T4 — Payload size: T2 with payload size as the third variable (I/O impact).
• T5 — Programming language: T2 varying the container language/image.
• T6 — Memory and CPU: T2 across proportional memory and CPU settings.
• T7 — Compute intensive: recursive Fibonacci, correlating the Fibonacci number with increasing vus.

Results

• Low-load latency (T1): Knative and AWS Fargate are practically tied; Cloud Run shows latencies around five times higher.
• Scalability (T2/T3): the order flips under stress — Cloud Run improves as vus grow, followed by Knative and last AWS Fargate. Success rates stayed at 100% (99.99% in the spike test).
• Stability: despite the best behaviour at scale, Cloud Run was unstable across every experiment, with high latency spikes — an argument against it for critical systems that need predictable latency.
• Payload (T4): AWS Fargate is indifferent to payload size; Knative and Cloud Run improve response times, likely because heavier I/O resource usage triggers scale-out earlier.
• Language (T5): Go and JavaScript perform best under parallel load; Java and especially Python degrade response times sharply.
• Memory and CPU (T6): more resources improve performance, but the gain is limited and depends on application efficiency and the configured scaling rules.
• Compute intensive (T7): Knative is the most efficient, with more computations and lower average latency; AWS Fargate failed to compute large Fibonacci numbers and showed high latency.

Cost and implementation effort

• Cloud Run and AWS Fargate show similar estimated and actual costs, but Cloud Run allocates more CPU time, supports scale to zero and bills only for request execution time — making it cheaper.
• Knative is the most expensive option, since it depends on a VM pool for the Kubernetes cluster. It is justified when private cloud or avoiding vendor lock-in matters.
• Learning curve: Cloud Run is the simplest (one CLI command creates a scalable service); Knative comes second if cluster setup is excluded; AWS Fargate demands more components and specific AWS networking knowledge.

Conclusion: CaaS platforms respond well to increasing traffic and offer concrete mechanisms for serverless's classic problems — cold starts and execution limits. The choice comes down to the trade-off between scalability (Cloud Run), predictability and portability (Knative), and integration with existing infrastructure (AWS Fargate).