Application-Centric Benchmarking of Distributed FaaS Platforms using BeFaaS
0
🤷
Sign in to get full access
Overview
- The paper examines the need for a comprehensive benchmarking framework for Function-as-a-Service (FaaS) systems, which are popular serverless computing platforms.
- It presents BeFaaS, an extensible application-centric benchmarking framework that evaluates FaaS platforms using realistic and typical FaaS application examples.
- The paper enhances BeFaaS with features for distributed FaaS setups, designs application benchmarks for common FaaS use cases, and runs extensive experiments on commercial cloud FaaS platforms and an edge serverless platform.
Plain English Explanation
Serverless computing, where you run your code on a cloud provider's servers without managing the underlying infrastructure, has become increasingly popular. This has led to the development of various Function-as-a-Service (FaaS) platforms, such as AWS Lambda, Azure Functions, and Google Cloud Functions, that allow you to run your code in response to events or HTTP requests.
To compare the performance and capabilities of these different FaaS platforms, developers need benchmarking frameworks. However, existing frameworks often only evaluate isolated aspects of FaaS systems. The authors of this paper wanted to create a more comprehensive, application-centric benchmarking framework called BeFaaS.
BeFaaS allows you to run realistic and typical FaaS application examples, such as event-driven data processing workflows, on different FaaS platforms and collect detailed performance measurements. This helps developers understand how the platforms would handle real-world FaaS applications, not just synthetic tests.
The paper enhances BeFaaS with features for distributed FaaS setups, where the application is spread across multiple FaaS systems. This is important for federated analytics and edge computing use cases. The authors also designed specific application benchmarks to reflect common FaaS use cases and ran extensive experiments on commercial cloud FaaS platforms and an edge serverless platform.
Technical Explanation
The paper presents an enhanced version of the BeFaaS benchmarking framework for FaaS systems. BeFaaS is designed to evaluate FaaS platforms using realistic and typical examples of FaaS applications, rather than just synthetic tests.
The key features added to BeFaaS in this work include:
- Support for distributed FaaS setups, where the benchmark application is spread across multiple FaaS systems. This allows for the evaluation of federated FaaS deployments.
- The design of four application benchmarks that reflect common FaaS use cases, such as event-driven data processing.
- Extensive experiments conducted on commercial cloud FaaS platforms (AWS Lambda, Azure Functions, Google Cloud Functions) and the tinyFaaS edge serverless platform.
The experiments revealed several insights:
- Network transmission is a major contributor to response latency for function chains, and this effect is exacerbated in hybrid edge-cloud deployments.
- The trigger delay between a published event and the start of the triggered function ranges from about 100ms for AWS Lambda to 800ms for Google Cloud Functions.
- Azure Functions showed the best cold start behavior for the tested workloads.
Critical Analysis
The paper provides a comprehensive and practical benchmarking framework for FaaS systems, addressing the need for a more holistic approach compared to existing frameworks. The authors' focus on realistic application scenarios and support for distributed FaaS setups are particularly valuable, as they better reflect real-world usage patterns.
However, the paper does not address potential limitations or caveats of the BeFaaS framework. For example, it would be helpful to understand the scope and representativeness of the selected application benchmarks, as well as any challenges or biases that may arise from the experimental setup.
Additionally, the paper could benefit from a more critical analysis of the performance differences observed across the tested FaaS platforms. While the results provide useful insights, a deeper discussion on the underlying architectural and implementation factors that contribute to these differences would enhance the value of the research.
Further work could also explore the extensibility and adaptability of BeFaaS to accommodate emerging FaaS use cases, such as machine learning workloads, or investigate the framework's applicability to other serverless computing paradigms beyond FaaS.
Conclusion
The paper presents an enhanced version of the BeFaaS benchmarking framework, which aims to provide a more comprehensive and realistic evaluation of FaaS platforms. By designing application benchmarks that reflect common FaaS use cases and supporting distributed FaaS setups, the authors have made significant progress in addressing the limitations of existing FaaS benchmarking approaches.
The extensive experiments conducted on commercial cloud FaaS platforms and an edge serverless platform offer valuable insights into the performance characteristics of these systems, particularly regarding network latency, trigger delays, and cold start behavior. These findings can inform the design and deployment of FaaS applications, helping developers make more informed decisions when choosing the appropriate FaaS platform for their needs.
Overall, the BeFaaS framework and the insights gained from this research contribute to a better understanding of the FaaS landscape and provide a foundation for further advancements in serverless computing benchmarking and optimization.
This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!
Related Papers
🤷
0
Application-Centric Benchmarking of Distributed FaaS Platforms using BeFaaS
Martin Grambow, Tobias Pfandzelter, David Bermbach
Due to the popularity of the FaaS programming model, there is now a wide variety of commercial and open-source FaaS systems. Hence, for comparison of different FaaS systems and their configuration options, FaaS application developers rely on FaaS benchmarking frameworks. Existing frameworks, however, tend to evaluate only single isolated aspects, a more holistic application-centric benchmarking framework is still missing. In previous work, we proposed BeFaaS, an extensible application-centric benchmarking framework for FaaS environments that focuses on the evaluation of FaaS platforms through realistic and typical examples of FaaS applications. In this extended paper, we (i) enhance our benchmarking framework with additional features for distributed FaaS setups, (ii) design application benchmarks reflecting typical FaaS use cases, and (iii) use them to run extensive experiments with commercial cloud FaaS platforms (AWS Lambda, Azure Functions, Google Cloud Functions) and the tinyFaaS edge serverless platform. BeFaaS now includes four FaaS application-centric benchmarks, is extensible for additional workload profiles and platforms, and supports federated benchmark runs in which the benchmark application is distributed over multiple FaaS systems while collecting fine-grained measurement results for drill-down analysis. Our experiment results show that (i) network transmission is a major contributor to response latency for function chains, (ii) this effect is exacerbated in hybrid edge-cloud deployments, (iii) the trigger delay between a published event and the start of the triggered function ranges from about 100ms for AWS Lambda to 800ms for Google Cloud Functions, and (iv) Azure Functions shows the best cold start behavior for our workloads.
Read more4/29/2024
0
Increasing Efficiency and Result Reliability of Continuous Benchmarking for FaaS Applications
Tim C. Rese, Nils Japke, Sebastian Koch, Tobias Pfandzelter, David Bermbach
In a continuous deployment setting, Function-as-a-Service (FaaS) applications frequently receive updated releases, each of which can cause a performance regression. While continuous benchmarking, i.e., comparing benchmark results of the updated and the previous version, can detect such regressions, performance variability of FaaS platforms necessitates thousands of function calls, thus, making continuous benchmarking time-intensive and expensive. In this paper, we propose DuetFaaS, an approach which adapts duet benchmarking to FaaS applications. With DuetFaaS, we deploy two versions of FaaS function in a single cloud function instance and execute them in parallel to reduce the impact of platform variability. We evaluate our approach against state-of-the-art approaches, running on AWS Lambda. Overall, DuetFaaS requires fewer invocations to accurately detect performance regressions than other state-of-the-art approaches. In 98.41% of evaluated cases, our approach provides equal or smaller confidence interval size. DuetFaaS achieves an interval size reduction in 59.06% of all evaluated sample sizes when compared to the competitive approaches.
Read more8/20/2024
🏋️
0
ElastiBench: Scalable Continuous Benchmarking on Cloud FaaS Platforms
Trever Schirmer, Tobias Pfandzelter, David Bermbach
Running microbenchmark suites often and early in the development process enables developers to identify performance issues in their application. Microbenchmark suites of complex applications can comprise hundreds of individual benchmarks and take multiple hours to evaluate meaningfully, making running those benchmarks as part of CI/CD pipelines infeasible. In this paper, we reduce the total execution time of microbenchmark suites by leveraging the massive scalability and elasticity of FaaS (Function-as-a-Service) platforms. While using FaaS enables users to quickly scale up to thousands of parallel function instances to speed up microbenchmarking, the performance variation and low control over the underlying computing resources complicate reliable benchmarking. We demonstrate an architecture for executing microbenchmark suites on cloud FaaS platforms and evaluate it on code changes from an open-source time series database. Our evaluation shows that our prototype can produce reliable results (~95% of performance changes accurately detected) in a quarter of the time (<=15min vs.~4h) and at lower cost ($0.49 vs. ~$1.18) compared to cloud-based virtual machines.
Read more7/30/2024
👀
0
GeoFaaS: An Edge-to-Cloud FaaS Platform
Mohammadreza Malekabbasi, Tobias Pfandzelter, Trever Schirmer, David Bermbach
The massive growth of mobile and IoT devices demands geographically distributed computing systems for optimal performance, privacy, and scalability. However, existing edge-to-cloud serverless platforms lack location awareness, resulting in inefficient network usage and increased latency. In this paper, we propose GeoFaaS, a novel edge-to-cloud Function-as-a-Service (FaaS) platform that leverages real-time client location information for transparent request execution on the nearest available FaaS node. If needed, GeoFaaS transparently offloads requests to the cloud when edge resources are overloaded, thus, ensuring consistent execution without user intervention. GeoFaaS has a modular and decentralized architecture: building on the single-node FaaS system tinyFaaS, GeoFaaS works as a stand-alone edge-to-cloud FaaS platform but can also integrate and act as a routing layer for existing FaaS services, e.g., in the cloud. To evaluate our approach, we implemented an open-source proof-of-concept prototype and studied performance and fault-tolerance behavior in experiments.
Read more7/26/2024