Hello everyone! I’m Yash Israni, an open-source enthusiast passionate about automation, DevOps practices, and building tools that eliminate repetitive manual work.

This summer, I had the privilege of participating in the Open-Source Promotion Plan (OSPP) 2025, where I collaborated with the Kmesh community to automate documentation and release workflows. Over the course of three months, I designed and implemented GitHub Actions pipelines that keep the Kmesh website always up-to-date, properly versioned, and reviewed for language quality.

In this blog, I’ll share my journey—from acceptance to project execution, the technical decisions I made, and the lessons I learned along the way.

OSPP Program – Overview​

The Open-Source Promotion Plan (OSPP), organized by the Institute of Software, Chinese Academy of Sciences (ISCAS), gives students and early-career contributors the opportunity to gain hands-on experience by working on impactful open-source projects under the guidance of mentors.

Each term runs for about three months (1 July – 30 September in my case). Contributors not only deliver real-world features but also learn how large open-source communities operate.

My Acceptance​

I have always enjoyed contributing to open source, and my interests naturally align with automation and cloud-native tooling. When I saw that Kmesh was offering projects under OSPP 2025, I was immediately drawn to their proposal for automating documentation workflows.

The project addressed a clear pain point: documentation updates and versioning were being done manually, often lagging behind releases. The opportunity to replace repetitive tasks with reliable automation felt both impactful and challenging.

I received my acceptance email on 28 June 2025, and the program officially ran from 1 July to 30 September.

Interestingly, I was able to complete the majority of my project work before the mid-term evaluation, so that checkpoint was skipped, giving me extra time to refine the workflows and write proper usage guidelines.

Project Workthrough​

1. Doc-Sync Workflow​

• Trigger: on every push to the main branch
• Action: opens a pull request in the website repository with the latest documentation updates
• Enhancements: automatically labels the PR for triage and runs the site’s CI pipeline to validate changes

2. Release Versioning Workflow​

• Trigger: when a new Git tag is pushed (release event)
• Action: generates a versioned snapshot of the documentation in the website repository
• Enhancements: automatically opens a PR for any versioning-related changes

3. Chinese Grammar Checker Workflow​

• Trigger: on pull requests that modify Chinese documentation
• Action: uses the LanguageTool API to detect grammar and style issues
• Enhancements: posts line-level review comments as warnings (non-blocking) so contributors receive suggestions without being blocked from merging

Results​

These workflows have effectively eliminated delays and manual errors, ensuring Kmesh documentation stays accurate and up-to-date.

All three workflows are now live in both the Kmesh main repository and website repository under .github/workflows.

Key Technical Decisions​

• Adopted repository dispatch for secure cross-repo communication, eliminating the need for long-lived personal tokens
• Granted the GitHub Actions token read & write permissions only where necessary, while delegating other operations to a scoped bot account for better security
• Implemented Docusaurus-compatible versioning by dynamically generating versions.json, keeping navigation in sync with releases
• Added robust error handling in the doc-sync workflow to gracefully manage missing folders or files, preventing workflow crashes

Mentorship Experience​

My mentors, Li Zhencheng and Zhonghu Xu, along with the Kmesh maintainers, were consistently supportive—whether through GitHub reviews or quick clarifications on Slack. Even though I delivered my main workflows ahead of schedule, their feedback helped me refine edge cases and improve overall reliability.

As a recognition of my contributions and active involvement, the Kmesh community welcomed me as a member of the organization. This acknowledgment was both humbling and motivating, and it strengthened my commitment to continue contributing to Kmesh and supporting its growth.

Lessons Learned​

1. Automation empowers humans – the goal isn’t to replace contributors but to free them from repetitive tasks so they can focus on meaningful reviews and design.
2. Start small and iterate – building workflows in incremental, testable steps made debugging and maintenance far easier than deploying everything at once.
3. Security matters – applying the principle of least privilege to tokens and permissions reduced risk while keeping automation safe.
4. Expect edge cases – workflows behave differently across environments; testing on forks and multiple platforms prevented surprises in production.
5. Documentation is part of the code – writing clear workflow descriptions and PR comments ensured maintainers trusted and understood what the automation was doing.

Acknowledgements​

I would like to sincerely thank my mentors Li Zhencheng and Zhonghu Xu for their guidance, quick reviews, and encouragement. Thanks also to the OSPP program staff for ensuring smooth operations throughout the term.

Links​

• Project issue & Pull requests
• OSPP website
• Yash Israni's github

Hello everyone! I'm Wu Xi, an open source enthusiast with deep interests in kernel networking, eBPF, and test engineering.

This summer, I had the privilege to participate in Open Source Promotion Plan (OSPP) 2025 and collaborate with the Kmesh community, focusing on eBPF program UT enhancement. Over three months, I primarily completed unit testing work for Kmesh eBPF programs. I wrote and successfully ran UT test code for sendMsg and cgroup programs, and supplemented testing documentation based on this work. Kmesh community developers can now verify eBPF program logic without depending on real kernel mounting and traffic simulation, significantly improving development efficiency. In this blog, I'll share my complete experience—from acceptance to project execution, technical choices, and lessons learned along the way.

OSPP Project Overview​

Open Source Promotion Plan (OSPP) is organized by the Institute of Software, Chinese Academy of Sciences (ISCAS), providing students and early-career developers with opportunities to collaborate on real open source projects under the guidance of experienced mentors.

Each session lasts approximately three months (my session was July 1st – September 30th). Participants not only deliver functional features but also experience firsthand how large open source communities operate.

My Acceptance Experience​

I've always enjoyed contributing to open source, and my interests happen to focus on network kernels and cloud-native tools. When I saw the "eBPF" and "unit testing" related topics offered by Kmesh in OSPP 2025, I was immediately attracted.

The pain points this project aimed to solve were very clear: eBPF program verification has long relied on black-box testing, which is not only inefficient but also has coverage that depends on testers' experience. By introducing a unit testing framework and supplementing key use cases, functional verification can be completed without requiring real kernel mounting, which is both valuable and challenging.

I received my acceptance email on June 28, 2025, with the official project cycle running from July 1st to September 30th.

Interestingly, I completed the main work of the project before the mid-term evaluation, so that stage was skipped. This gave me more time to refine the workflow and write usage documentation.

Project Work Content​

1. eBPF Unit Testing Framework Construction​

• Core Technology: eBPF kernel function simulation based on #define mock macro replacement
• Test Coverage: Covers sendmsg TLV encoding, cgroup sock connection management, cgroup skb traffic processing
• Innovation: Embedding test infrastructure in production code through conditional compilation #ifdef KMESH_UNIT_TEST

2. sendmsg TLV Encoding Verification​

• Test Objective: Verify correctness of TLV metadata encoding in waypoint scenarios
• Test Data: IPv4 (8.8.8.8:53) and IPv6 (fc00:dead:beef🔢🔡53) simulation data
• Verification Mechanism: Real-time parsing of TLV message format, verifying integrity of type, length, IP, and port

3. cgroup Lifecycle Management Testing​

• Hook Coverage: cgroup/connect4, cgroup/connect6, cgroup/sendmsg4, cgroup/recvmsg4
• Test Scenarios: kmesh management process registration/deregistration, backend connections without waypoint, tail call mechanism
• Verification Method: Verify netns cookie management correctness through km_manage map state changes

Project Results​

These testing frameworks effectively eliminated blind spots in eBPF program testing, ensuring the stability and correctness of Kmesh's data plane.

Currently, the testing framework has been integrated into the CI/CD pipeline, allowing execution of the complete eBPF unit test suite through the make run command, covering core components like workload, XDP, sockops, sendmsg, cgroup_skb, and cgroup_sock.

Key Technical Decisions​

• Used define mock for function replacement, replacing eBPF kernel functions at compile time through macro definitions like #define bpf_sk_storage_get mock_bpf_sk_storage_get, achieving dependency isolation in unit tests
• Adopted conditional compilation test infrastructure, embedding test-specific map definitions and data structures in production code through #ifdef KMESH_UNIT_TEST macros, ensuring consistency between test and production code
• Used Go + eBPF hybrid testing framework, combining C language eBPF program compilation with Go language test execution, implementing automated testing workflow through go test -v ./...

Mentor Guidance Experience​

My mentors Li Zhencheng and Xu Zhonghu, along with other Kmesh maintainers, provided tremendous support throughout the UT testing framework development process.

They not only patiently pointed out improvements in test design during GitHub reviews but also quickly answered my questions about bpf helper mocking and map validation on Slack.

Although I completed the core UT for sendMsg and cgroup programs relatively early, mentor feedback helped me notice more edge cases and pushed me to further improve test coverage and documentation.

Finally, the Kmesh community invited me to become an organization member as recognition of my contributions and active participation. This not only made me feel humble but also strengthened my determination to continue participating in and supporting Kmesh's development.

Lessons Learned​

1. Unit testing is a tool to enhance development efficiency — It doesn't replace black-box testing but complements and frees developers, allowing them to focus faster on feature implementation and optimization.
2. Start small and iterate gradually — First supplement UT for core eBPF programs (like sendMsg, cgroup_skb), then gradually expand to more scenarios, which is more stable than covering all logic at once.
3. Anticipate edge cases — eBPF programs may behave differently across kernel versions or environments; simulating various inputs and exceptions in UT in advance helps avoid production environment surprises.
4. Communication can accelerate learning progress — Every time I submitted a PR, mentors would comment with better solutions or questions, which taught me a lot in a short time.
5. Facing challenges head-on is the recipe for progress — When learning fields you're interested in but haven't had much exposure to, setbacks are inevitable. Don't give up at these times; believe in yourself and keep trying—you'll eventually find solutions to problems.

Acknowledgments​

I want to sincerely thank my mentors Li Zhencheng and Xu Zhonghu throughout the process. In every community meeting, they would actively solve my current problems and understand my progress. Whenever I submitted a PR, they would share their insights in the comments, making my thinking clearer and improving my problem-solving abilities. I also want to thank the OSPP organizing committee for providing us with a smoothly running environment. This open source participation was an extraordinary experience for me, and I will continue to dedicate myself to open source and love open source!

Related Links​

• Project Issue & Pull Requests
• OSPP Official Website
• Wu Xi's GitHub

Hello readers, I am Yash, a final Year student from India. I love building cool stuffs and solving real world problems. I’ve been working in the cloud-native space for the past three years, exploring technologies like Kubernetes, Cilium, Istio, and more.

I successfully completed my mentorship with Kmesh during the LFX 2025 Term-1 program, which was an enriching and invaluable experience. Over the past three months, I gained significant knowledge and hands-on experience while contributing to the project. In this blog, I’ve documented my mentorship journey and the work I accomplished as a mentee.

LFX Mentorship Program – Overview​

The LFX Mentorship Program, run by the Linux Foundation, is designed to help students and early-career professionals gain hands-on experience in open source development by working on real-world projects under the guidance of experienced mentors

Participants contribute to high-impact projects hosted by foundations like CNCF, LF AI, LF Edge, and more. The program typically runs in 3 terms throughout the year, each lasting about three months.

More-info

My Acceptance​

I am a regular opensource contributor and loves contributing to opensource. My interests heavily aligned with clound-native technologies. I was familiar with popular mentorship programs like LFX and GSoC, which are designed to help students get started in the open source world. Based on my work the Kmesh community also promoted for the member of Kmesh I had made up my mind to apply for LFX 2025 Term-1 and began exploring projects in early February. The projects under CNCF for LFX are listed in the cncf/mentoring GitHub repository. I came across the Kmesh project, a newly added CNCF sandbox project participating in LFX for the first time. I found the Kmesh project particularly exciting because of the problem it addresses—providing a sidecarless service mesh data plane. This approach can greatly benefit the community by improving performance and reducing overhead.

Kmesh came up with 4 projects in term-1, i selected long-connection-metrics projects as it allows me to works with eBPF a already have a prior experience on working with eBPF.

I began exploring the Kmesh project by reading the documentation and contributing to Good First Issues. As I became more involved, the mentors started to take notice. I also submitted a proposal for the long connection metrics project.

In late February, I received an email from LFX notifying me of my selection.

Project Workthrough​

The tcp long connection metrics project aims to implement access logs and metrics for TCP long connections, developing a continuous monitoring and reporting mechanisms that captures detailed, real-time data throughout the lifetime of long-lived TCP connections.

Ebpf hooks are used to collect connection stats such as send/received bytes, packets losts, retransmissions etc.

More-information

Mentorship Experience​

The Kmesh maintainers were always available to help me with any doubts, whether on Slack or GitHub. Additionally, there is a community meeting held regularly every Thursday, where I could ask questions and discuss various topics. I’ve learned a lot from them, including how to approach problems effectively and consider edge cases during development in these three months.

Based on my contributions and active involvement, the Kmesh community recognized my efforts and promoted me to a member of the organization. This acknowledgment was truly encouraging and motivated me to continue contributing to Kmesh and help the project grow.

Building on the foundation of v1.0.0, this release introduces significant enhancements to Kmesh’s architecture, observability, and ecosystem integration. The official Kmesh website has undergone a comprehensive redesign, offering an intuitive interface and streamlined documentation to empower both users and developers. Under the hood, we’ve refactored the DNS module and added metrics for long connections, providing deeper insights into more traffic patterns.

In Kernel-Native mode, we’ve reduced invasive kernel modifications. Also, we use global variables to replace the BPF config map to simplify the underlying complexity. Compatibility with ​​Istio 1.25​​ has been rigorously validated, ensuring seamless interoperability with the latest Istio version. Notably, the persistent TestKmeshRestart E2E test case flaky—a long-standing issue—has been resolved through long-term investigation and reconstruction of the underlying BPF program, marking a leap forward in runtime reliability.

Main Features​

Website overhaul​

The Kmesh official website has undergone a complete redesign, offering an intuitive user experience with improved documentation, reorganized content hierarchy and streamlined navigation. In addressing feedback from the previous iteration, we focused on key areas where user experience could be enhanced. The original interface presented some usability challenges that occasionally led to navigation difficulties. Our blog module in particular required attention, as its content organization and visual hierarchy impacted content discoverability and readability. From an engineering perspective, we recognized opportunities to improve the code structure through better component organization and more systematic styling approaches, as the existing implementation had grown complex to maintain over time.

To address these problems, we shifted to React with Docusaurus, a modern documentation framework that's much more developer-friendly. This allowed us to create modular components, eliminating redundant code through reusability. Docusaurus provides built-in navigation systems specifically designed for documentation and blogs, plus version-controlled documentation features. We've implemented multilingual support with both English and Chinese documentation, added advanced search functionality, and completely reorganized the content structure. The result is a dramatically improved experience that makes the Kmesh site more accessible and valuable for all users.

Long connection metrics​

Before this release, Kmesh provides access logs during termination and establishment of a TCP connection with more detailed information about the connection, such as bytes sent, received, packet lost, rtt and retransmits. Kmesh also provides workload and service specific metrics such as bytes sent and received, lost packets, minimum rtt, total connection opened and closed by a pod. These metrics are only updated after a connection is closed.

In this release, we implement access logs and metrics for TCP long connections, developing a continuous monitoring and reporting mechanism that captures detailed, real-time data throughout the lifetime of long-lived TCP connections. Access logs are reported periodically with information such as reporting time, connection establishment time, bytes sent, received, packet loss, rtt, retransmits and state. Metrics such as bytes sent and received, packet loss, retransmits are also reported periodically for long connections.

DNS refactor​

The current DNS process includes the CDS refresh process. As a result, DNS is deeply coupled with kernel-native mode and cannot be used in dual-engine mode.

In release 1.1 we refactored the DNS module of Kmesh. Instead of a structure containing cds, the data looped through the refresh queue in the Dns is now a domain, so that the Dns module no longer cares about the Kmesh mode, only providing the hostname to be resolved.

BPF config map optimization​

Kmesh has eliminated the dedicated kmesh_config_map BPF map, which previously stored global runtime configurations such as BPF logging level and monitoring toggle. These settings are now managed through global variables. Leveraging global variables simplifies BPF configuration management, enhancing runtime efficiency and maintainability.

Optimise Kernel Native mode to reduce intrusive modifications to the kernel The kernel-native mode requires a large number of intrusive kernel reconstructions to implement HTTP-based traffic control. Some of these modifications may have a significant impact on the kernel, which makes the kernel-native mode difficult to deploy and use in a real production environment. To resolve this problem, we have modified the kernel in kernel-native mode and the involved ko and eBPF synchronously. Through the optimization of this release. In kernel 5.10, the kernel modification is limited to four, and in kernel 6.6, the kernel modification is reduced to only one. This last one will be eliminated as much as possible, with the goal of eventually running kernel-native mode on native version 6.6 and above.

Adopt istio 1.25​

Kmesh has verified compatibility with istio 1.25 and has added the corresponding E2E test to CI. The Kmesh community maintains verification of the three istio versions in CI, so the E2E test of istio 1.22 has been removed from CI.

Critical Bug Fix​

kmeshctl install waypoint error (#1287)

root analysis:

Remove the extra v before the version number when building the waypoint image.

TestKmeshRestart flaky (#1192)

root analysis:

This issue is actually not related Kmesh restart, and it can also be produced in non-restart scenario.

The root case is that it's not appropriate to use sk as the key of map map_of_orig_dst, because it is reused and the value of map will be incorrectly overwritten, resulting in the metadata is not being encoded when it should be encoded in the connection sent to the waypoint, resulting the reset error in this issue.

TestServiceEntrySelectsWorkloadEntry flaky (#1352)

root analysis:

before this test case, there is a test TestServiceEntryInlinedWorkloadEntry which will generate two workload objects, for example, Kubernetes/networking.istio.io/ServiceEntry/echo-1-21618/test-se-v4/10.244.1.103 and ServiceEntry/echo-1-21618/test-se-v6/10.244.1.103.

In the current use case, WorkloadEntry will generate the workload object Kubernetes/networking.istio.io/WorkloadEntry/echo-1-21618/test-we.

If the test case runs fast enough, the removal operation of the first two workload objects will be aggregated with the creation operation of the latter object.

Kmesh will process the new object first and then remove the old resources, reference.

The IP addresses of these three objects are the same, which will eventually lead to the inability to find the IP address in the Kmesh workload cache, which will cause auth failure and connection timeout.

Acknowledgment​

Kmesh v1.1.0 includes 118 commits from 14 contributors. We would like to express our sincere gratitude to all contributors:

We have always developed Kmesh with an open and neutral attitude, and continue to build a benchmark solution for the Sidecarless service mesh industry, serving thousands of industries and promoting the healthy and orderly development of service mesh. Kmesh is currently in a stage of rapid development, and we sincerely invite people with lofty ideals to join us!

Reference Links​

• Kmesh Release v1.1.0
• Kmesh GitHub
• Kmesh Website

Hi everyone! I'm Jayesh Savaliya, a B.Tech student at IIIT Pune passionate about backend technologies and open source. Over the last two years, I've been selected for the C4GT program twice (2024 & 2025) - yes, they let me back in - and recently completed LFX Mentorship 2025 (Term 1), where I somehow went from fixing typos to being responsible for reviewing other people's code at Kmesh.

In this blog, I'll share my journey and the strategies that actually worked (no generic "just be passionate" advice, I promise).

My Background​

When I applied to LFX, I wasn't starting from scratch. I had already battle-tested myself with:

• Sunbird (EkStep Foundation) via C4GT, where I learned that education tech is harder than it looks
• Mifos, a GSoC organization focused on financial services (because debugging payment systems at 2 AM builds character)
• Various backend projects where I definitely didn't break production. Much.

Choosing Kmesh​

I shortlisted projects from the LFX portal based on three key criteria:

1. Tech stack relevance - Technologies I wanted to master
2. Learning potential - Projects that would challenge and grow my skills
3. Active maintainers - Communities with responsive, helpful mentors

I chose Kmesh, a high-performance service mesh data plane built on eBPF and programmable kernel technologies. Kmesh's sidecarless architecture eliminates proxy overhead, resulting in better performance and lower resource consumption.

Honestly? It had "eBPF" in the description and I wanted to sound cool at tech meetups. But it turned out to be genuinely fascinating work with a great community.

How to Succeed in Open Source Programs​

Here's my three-step approach that worked for LFX:

1. Make Meaningful Contributions​

Start small and scale up gradually. Don't be the person who says "I'll rewrite the entire architecture!" on day one.

Instead:

• Weeks 1-2: Fix typos, improve logs, update documentation
• Weeks 3-4: Fix small bugs, add tests
• Week 5+: Work on core features and refactoring

This progression shows mentors you're not just throwing random PRs at the wall hoping something sticks.

2. Write a Strong Proposal​

Your proposal should be:

• Clear: Explain your approach in straightforward language
• Structured: Include a realistic timeline with milestones
• Convincing: Demonstrate why you're the right person for the project

Make sure your proposal reflects genuine engagement with the project, not just surface-level research.

3. Be Actively Involved​

Stay engaged in project channels (Slack, Discord, mailing lists). Communicate regularly with mentors, ask thoughtful questions, and contribute to discussions.

But also: don't be that person who asks questions Google could answer or pings everyone at 3 AM with "quick question." Balance is everything.

The Formula: Consistent contributions + Strong proposal + Active communication = Standing out

The Path to Maintainership​

Becoming a maintainer wasn't planned. It happened naturally through sustained engagement after the mentorship period ended.

Consistency​

I continued contributing regularly after my initial PRs were merged:

• Fixing overlooked bugs
• Adding requested features
• Refactoring code for better maintainability

Learning Mindset​

I embraced every learning opportunity, even when I had no idea what I was doing. eBPF concepts? Started clueless, ended slightly less clueless. Performance optimization? Learned by making things slower first. CI/CD improvements? Broke the build a few times, but now I own it.

Patience & Feedback​

Code reviews can be humbling (read: brutal). I learned to take feedback seriously even when it stung, iterate quickly, and stay patient when things inevitably broke.

Taking Initiative​

I started acting like a maintainer before having the title:

• Suggesting project improvements
• Writing comprehensive tests (because flaky tests are the worst)
• Automating repetitive tasks (laziness is a virtue in programming)
• Reviewing other contributors' work

By the end of my mentorship, the trust I built with the team led to being granted maintainer access. Going from "hey, can I fix this typo?" to "you're now responsible for reviewing PRs" was equal parts surreal and terrifying.

Key Takeaways​

Here's what I learned that might help you:

Start small, stay consistent - Begin with simple contributions and build from there. Consistency matters more than individual genius.

Focus on learning - Getting selected is great, but learning enough to make real contributions is what counts.

Communicate effectively - Ask questions, share progress, and be helpful. Respectful, clear communication goes a long way.

Suggest improvements - If you see something that could be better, speak up. Good ideas are always welcome.

Embrace feedback - Your first PR won't be perfect. Nobody's is. Take feedback as learning opportunities, iterate, and move on. Arguing about semicolons is not a productive use of anyone's time.

You don't need to be a genius. You just need to show up, contribute meaningfully, and improve consistently.

Final Thoughts​

The LFX Mentorship taught me more than just technical skills. I learned how to work with distributed teams across timezones, think critically about production software (logs are your friends!), and grow into a leadership role in an open source community.

If you're considering applying to LFX or any open source program, take the leap. With consistent effort and genuine engagement, you can make a real impact. If I can go from nervous first-time contributor to maintainer, so can you.

Connect With Me​

Feel free to reach out if you want to discuss open source, eBPF, or systems programming:

• LinkedIn
• GitHub

Thanks for reading, and see you in the next PR!

Alibaba Cloud Service Mesh (ASM) supports both Sidecar and Sidecarless modes. The Sidecar mode, where a proxy runs alongside each service instance, is currently the most selected and stable solution. However, this architecture introduces latency and resource overhead. To address the latency and resource consumption inherent in the Sidecar mode, various Sidecarless mode solutions have emerged in recent years, such as Istio Ambient. Istio Ambient deploys a ztunnel on each node to perform layer-4 traffic proxying for the Pods running on the node and deploy waypoints for layer-7 traffic proxying. While the Sidecarless mode can reduce latency and resource consumption, its stability and completeness in functionality still require improvement.

ASM currently supports different Sidecarless modes, such as Istio Ambient mode, ACMG mode, and Kmesh, among others. Kmesh (for more details, refer to https://kmesh.net/) is a high-performance service mesh data plane software implemented based on eBPF and programmable kernel. By offloading traffic management to the kernel, Kmesh allows service communication within the mesh to occur without passing through proxy software, significantly reducing the traffic forwarding path and effectively enhancing the forwarding performance of service access.

Introduction to Kmesh​

Kmesh's dual-engine mode uses eBPF to intercept traffic in kernel space and deploys a Waypoint Proxy to handle complex L7 traffic management, thus separating L4 and L7 governance between kernel space (eBPF) and user space (Waypoint). Compared to Istio's Ambient Mesh, it reduces latency by 30%. Compared to the kernel-native mode, the dual-engine mode does not require kernel enhancements, offering broader applicability.

Currently, ASM supports using Kmesh's dual-engine mode as one of the data planes for the service mesh, enabling more efficient service management. Specifically, ASM can be used as the control plane, while Kmesh can be deployed as the data plane within an Alibaba Cloud Container Service for Kubernetes (ACK) cluster.

Deploy Kmesh in ACK and Connect to ASM​

Prerequisites​

Create an ASM cluster and add the ACK cluster to the ASM cluster for management. For detailed steps, you can refer to the documentation: Add a cluster to an ASM instance.

Install Kmesh​

Run the following command to clone the Kmesh project into your local machine.

git clone https://github.com/kmesh-net/kmesh.git && cd kmesh

Check Services of ASM Control Plane​

After the Kmesh is downloaded, you need to execute the following command first to check the Service name of the current ASM control plane in the cluster, in order to configure the connection between Kmesh and the ASM control plane.

kubectl get svc -n istio-system | grep istiod

# istiod-1-22-6   ClusterIP   None   <none>   15012/TCP   2d

Install Kmesh with Kubectl​

You can use kubectl or helm to install Kmesh in the ACK Kubernetes cluster. However, before installation, please add the ClusterId and xdsAddress environment variables to the Kmesh DaemonSet. These are used for the authentication and connection between Kmesh and the ASM control plane. The ClusterId is the ID of the ACK cluster where Kmesh is deployed, and the xdsAddress is the Service of the ASM control plane.

# You can find the resource definition in the following files:
# helm: deploy/charts/kmesh-helm/templates/daemonset.yaml
# kubectl: deploy/yaml/kmesh.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kmesh
  labels:
    app: kmesh
  namespace: kmesh-system
spec:
    spec:
      containers:
        - env:
          # ASM Control Plane Service
          - name: XDS_ADDRESS
            value: "istiod-1-22-6.istio-system.svc:15012"
          # add ACK cluster id
          - name: CLUSTER_ID
            value: "cluster-id"
    ...

After the modification is done, you can run the following command to install Kmesh.

# kubectl
kubectl apply -f deploy/yaml

# helm
helm install kmesh deploy/charts/kmesh-helm -n kmesh-system --create-namespace

Check Kmesh Startup Status​

After the installation is done, run the following command to check the Kmesh startup status.

kubectl get pods -A | grep kmesh

# kmesh-system   kmesh-l5z2j   1/1   Running   0    117m

Run the following command to check Kmesh running status.

kubectl logs -f -n kmesh-system kmesh-l5z2j

# time="2024-02-19T10:16:52Z" level=info msg="service node sidecar~192.168.11.53~kmesh-system.kmesh-system~kmesh-system.svc.cluster.local connect to discovery address istiod.istio-system.svc:15012" subsys=controller/envoy
# time="2024-02-19T10:16:52Z" level=info msg="options InitDaemonConfig successful" subsys=manager
# time="2024-02-19T10:16:53Z" level=info msg="bpf Start successful" subsys=manager
# time="2024-02-19T10:16:53Z" level=info msg="controller Start successful" subsys=manager
# time="2024-02-19T10:16:53Z" level=info msg="command StartServer successful" subsys=manager
# time="2024-02-19T10:16:53Z" level=info msg="start write CNI config\n" subsys="cni installer"
# time="2024-02-19T10:16:53Z" level=info msg="kmesh cni use chained\n" subsys="cni installer"
# time="2024-02-19T10:16:54Z" level=info msg="Copied /usr/bin/kmesh-cni to /opt/cni/bin." subsys="cni installer"
# time="2024-02-19T10:16:54Z" level=info msg="kubeconfig either does not exist or is out of date, writing a new one" subsys="cni installer"
# time="2024-02-19T10:16:54Z" level=info msg="wrote kubeconfig file /etc/cni/net.d/kmesh-cni-kubeconfig" subsys="cni installer"
# time="2024-02-19T10:16:54Z" level=info msg="command Start cni successful" subsys=manager

You can enable Kmesh for a specific namespace by executing the following command.

kubectl label namespace default istio.io/dataplane-mode=Kmesh

Traffic Shifting Demo​

Deploy Demo App and Traffic Shifting Rules​

After enabling Kmesh for the default namespace, run the following command to install the sample application.

kubectl apply -f samples/fortio/fortio-route.yaml
kubectl apply -f samples/fortio/netutils.yaml

Run the following command to check the running status of the sample application.

kubectl get pod
# NAME                         READY   STATUS    RESTARTS   AGE
# fortio-v1-596b55cb8b-sfktr   1/1     Running   0          57m
# fortio-v2-76997f99f4-qjsmd   1/1     Running   0          57m
# netutils-575f5c569-lr98z     1/1     Running   0          67m

kubectl describe pod netutils-575f5c569-lr98z | grep Annotations
# Annotations:      kmesh.net/redirection: enabled

Label kmesh.net/redirection: enabled of the pod indicates that Kmesh forwarding has been enabled for that Pod.

Run the following command to view the currently defined traffic routing rules. It can be seen that 90% of the traffic is directed to version v1 of fortio, and 10% of the traffic is directed to version v2 of fortio.

kubectl get virtualservices -o yaml

# apiVersion: v1
# items:
# - apiVersion: networking.istio.io/v1beta1
#   kind: VirtualService
#   metadata:
#     annotations:
#       kubectl.kubernetes.io/last-applied-configuration: |
#         {"apiVersion":"networking.istio.io/v1alpha3","kind":"VirtualService","metadata":{"annotations":{},"name":"fortio","namespace":"default"},"spec":{"hosts":["fortio"],"http":[{"route":[{"destination":{"host":"fortio","subset":"v1"},"weight":90},{"destination":{"host":"fortio","subset":"v2"},"weight":10}]}]}}
#     creationTimestamp: "2024-07-09T09:00:36Z"
#     generation: 1
#     name: fortio
#     namespace: default
#     resourceVersion: "11166"
#     uid: 0a07f283-ac26-4d86-b3bd-ce6aa07dc628
#   spec:
#     hosts:
#     - fortio
#     http:
#     - route:
#       - destination:
#           host: fortio
#           subset: v1
#         weight: 90
#       - destination:
#           host: fortio
#           subset: v2
#         weight: 10
# kind: List
# metadata:
#   resourceVersion: ""

Deploy Waypoint for Fortio Service​

You can deploy Waypoint to handle service-level layer 7 traffic by executing the following command in the default namespace.

kubectl apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  labels:
    istio.io/waypoint-for: service
  name: fortio-waypoint
  namespace: default
spec:
  gatewayClassName: istio-waypoint
  listeners:
  - name: mesh
    port: 15008
    protocol: HBONE
EOF

Run the following enable Waypoint for fortio service.

kubectl label service fortio istio.io/use-waypoint=fortio-waypoint

Run the following command to check the current Waypoint status.

kubectl get gateway.gateway.networking.k8s.io

# NAME              CLASS            ADDRESS          PROGRAMMED   AGE
# fortio-waypoint   istio-waypoint   192.168.227.95   True         8m37s

Start Test Traffic​

You can start test traffic by executing the following command. You should see that only about 10% of the traffic is directed to version v2 of fortio.

for i in {1..20}; do kubectl exec -it $(kubectl get pod | grep netutils | awk '{print $1}') -- curl -v $(kubectl get svc -owide | grep fortio | awk '{print $3}'):80 | grep "Server:"; done

# < Server: 1
# < Server: 1
# < Server: 1
# < Server: 1
# < Server: 1
# < Server: 1
# < Server: 1
# < Server: 1
# < Server: 2
# < Server: 1
# < Server: 1
# < Server: 1
# < Server: 1
# < Server: 1
# < Server: 1
# < Server: 1
# < Server: 1
# < Server: 2
# < Server: 1
# < Server: 1

Kmesh is kernel native sidecarless service mesh data plane. It sinks traffic governance into the OS kernel with the help of ebpf and programmable kernel. It reduces the resource overhead and network latency of the service mesh.

And the data of the traffic can be obtained directly in the kernel and can uses bpf map passed to the user space. This data is used to build metrics and accesslogs.

How to Get Data​

In the kernel, it is possible to get the metrics data carried by the socket directly.

The data carried in the bpf_tcp_sock is as follows:

struct bpf_tcp_sock {
 __u32 snd_cwnd;  /* Sending congestion window  */
 __u32 srtt_us;  /* smoothed round trip time << 3 in usecs */
 __u32 rtt_min;
 __u32 snd_ssthresh; /* Slow start size threshold  */
 __u32 rcv_nxt;  /* What we want to receive next  */
 __u32 snd_nxt;  /* Next sequence we send  */
 __u32 snd_una;  /* First byte we want an ack for */
 __u32 mss_cache; /* Cached effective mss, not including SACKS */
 __u32 ecn_flags; /* ECN status bits.   */
 __u32 rate_delivered; /* saved rate sample: packets delivered */
 __u32 rate_interval_us; /* saved rate sample: time elapsed */
 __u32 packets_out; /* Packets which are "in flight" */
 __u32 retrans_out; /* Retransmitted packets out  */
 __u32 total_retrans; /* Total retransmits for entire connection */
 __u32 segs_in;  /* RFC4898 tcpEStatsPerfSegsIn
     * total number of segments in.
     */
 __u32 data_segs_in; /* RFC4898 tcpEStatsPerfDataSegsIn
     * total number of data segments in.
     */
 __u32 segs_out;  /* RFC4898 tcpEStatsPerfSegsOut
     * The total number of segments sent.
     */
 __u32 data_segs_out; /* RFC4898 tcpEStatsPerfDataSegsOut
     * total number of data segments sent.
     */
 __u32 lost_out;  /* Lost packets   */
 __u32 sacked_out; /* SACK'd packets   */
 __u64 bytes_received; /* RFC4898 tcpEStatsAppHCThruOctetsReceived
     * sum(delta(rcv_nxt)), or how many bytes
     * were acked.
     */
 __u64 bytes_acked; /* RFC4898 tcpEStatsAppHCThruOctetsAcked
     * sum(delta(snd_una)), or how many bytes
     * were acked.
     */
 __u32 dsack_dups; /* RFC4898 tcpEStatsStackDSACKDups
     * total number of DSACK blocks received
     */
 __u32 delivered; /* Total data packets delivered incl. rexmits */
 __u32 delivered_ce; /* Like the above but only ECE marked packets */
 __u32 icsk_retransmits; /* Number of unrecovered [RTO] timeouts */
};

Notes: The above data was not fully utilized for metrics and accesslog. Kmesh will fill in the metrics later in the development. The data used at this stage are:

struct tcp_probe_info {
    __u32 type; /*type of connection (IPV4 or IPV6) */
  
    struct bpf_sock_tuple tuple;
    struct orig_dst_info orig_dst;

    __u32 sent_bytes;     /* Total send bytes from start to last_report_ns */
    __u32 received_bytes; /* Total recv bytes from start to last_report_ns */
    __u32 conn_success;
    __u32 direction;
    __u32 state;    /* tcp state */
    __u64 duration; // ns
    __u64 start_ns;
    __u64 last_report_ns; /*timestamp of the last metrics report*/
    __u32 protocol;
    __u32 srtt_us;       /* smoothed round trip time << 3 in usecs until last_report_ns */
    __u32 rtt_min;       /* min round trip time in usecs until last_report_ns */
    __u32 total_retrans; /* Total retransmits from start to last_report_ns */
    __u32 lost_out;      /* Lost packets from start to last_report_ns */
};

In addition to the TCP data that can be accessed directly, Kmesh temporarily records supplementary information during the connection establishment phase, such as the start time, connection direction, and the last report time. The last report time is used to periodically report connection metrics. After each report, Kmesh updates the last report time to the current timestamp, while also utilizing other stored information to enrich the reported data.

Connection stats is written to a ring buffer, allowing userspace applications to access it. Data is reported at key stages of the connection lifecycle: during connection establishment, at regular intervals throughout the connection's duration, and upon connection closure.

How to Handle TCP stats​

After parsing the data from ringbuf in the user space, Kmesh builds metricLabels based on the linked source and destination information. It then updates the cache in the metricController.

This is because the data reported through the ring buffer is connection-specific, capturing details of individual TCP connections between applications. However, the metrics exposed to the user are expected at multiple levels of granularity — including connection, pod, and service levels. As a result, aggregation of the connection data is necessary to provide meaningful metrics at the higher pod and service granularity.

Get the hostname and namespace of the destination service in the cluster from the Services information in the destination Workload.

namespacedhost := ""
for k, portList := range dstWorkload.Services {
    for _, port := range portList.Ports {
        if port.TargetPort == uint32(dstPort) {
            namespacedhost = k
            break
        }
    }
    if namespacedhost != "" {
        break
    }
}

After building the metriclabels at the workload granularity, service granularity and connection granularity update the cache.

Every 5 seconds, the metrics information will be updated into Prometheus through the Prometheus API.

Access log data is generated during the processing of metrics and subsequently emitted by the daemon.

The architecture diagram is shown below:

Result​

Metrics monitored by Kmesh L4 at this stage:

Workload Metrics​

Give information about traffic behavior and performance between workloads.

Metric Result:

kmesh_tcp_workload_received_bytes_total{connection_security_policy="mutual_tls",destination_app="ws-server",destination_canonical_revision="latest",destination_canonical_service="ws-server",destination_cluster="Kubernetes",destination_pod_address="10.244.2.80",destination_pod_name="ws-server",destination_pod_namespace="default",destination_principal="-",destination_version="latest",destination_workload="ws-server",destination_workload_namespace="default",reporter="source",request_protocol="tcp",response_flags="-",source_app="ws-client",source_canonical_revision="latest",source_canonical_service="ws-client",source_cluster="Kubernetes",source_principal="-",source_version="latest",source_workload="ws-client",source_workload_namespace="default"} 6

Service Metrics​

Give information about traffic behavior and performance between services.

Metric Result:

kmesh_tcp_received_bytes_total{connection_security_policy="mutual_tls",destination_app="ws-server",destination_canonical_revision="latest",destination_canonical_service="ws-server",destination_cluster="Kubernetes",destination_principal="-",destination_service="ws-server-service.default.svc.cluster.local",destination_service_name="ws-server-service",destination_service_namespace="default",destination_version="latest",destination_workload="ws-server",destination_workload_namespace="default",reporter="source",request_protocol="tcp",response_flags="-",source_app="ws-client",source_canonical_revision="latest",source_canonical_service="ws-client",source_cluster="Kubernetes",source_principal="-",source_version="latest",source_workload="ws-client",source_workload_namespace="default"} 5

Connection Metrics​

Give information about traffic behavior and performance of a established tcp connection(duration > 30 seconds). These metrics are particularly valuable in clusters running workloads that establish long-lived TCP connections, such as databases, message brokers, audio/video streaming services, AI applications etc.

Metric Result:

kmesh_tcp_connection_received_bytes_total{connection_security_policy="mutual_tls",destination_address="10.244.2.80:8080",destination_app="ws-server",destination_canonical_revision="latest",destination_canonical_service="ws-server",destination_cluster="Kubernetes",destination_pod_address="10.244.2.80",destination_pod_name="ws-server",destination_pod_namespace="default",destination_principal="-",destination_service="ws-server-service.default.svc.cluster.local",destination_service_name="ws-server-service",destination_service_namespace="default",destination_version="latest",destination_workload="ws-server",destination_workload_namespace="default",reporter="destination",request_protocol="tcp",response_flags="-",source_address="10.244.2.81:47660",source_app="ws-client",source_canonical_revision="latest",source_canonical_service="ws-client",source_cluster="Kubernetes",source_principal="-",source_version="latest",source_workload="ws-client",source_workload_namespace="default",start_time="2025-04-24 12:47:54.439318976 +0000 UTC"} 8680

It can also be viewed via the prometheus dashboard. Refer to Kmesh observability

Accesslog monitored by Kmesh L4 at this stage:

Accesslog Result:

accesslog: 2025-04-24 08:54:40.971980208 +0000 UTC src.addr=10.244.2.79:41978, src.workload=ws-client, src.namespace=default, dst.addr=10.244.2.78:8080, dst.service=ws-server-service.default.svc.cluster.local, dst.workload=ws-server, dst.namespace=default, start_time=2025-04-24 08:53:50.919245381 +0000 UTC, direction=OUTBOUND, state=BPF_TCP_ESTABLISHED, sent_bytes=3, received_bytes=227, packet_loss=0, retransmissions=0, srtt=40515us, min_rtt=34us, duration=50052.734827ms

Summary​

Kmesh takes the traffic data directly from the socket and passes it as ringbuf to the user space to generate Metric and Accesslog. and expose it to Prometheus.

Avoid intercepting traffic in the user space and getting metrics in a native way. And batch update Metrics in user space at regular intervals to avoid increasing network latency during heavy traffic.

Subsequently, we will also develop the trace to complement the observability capability of kmesh.

Welcome to participate in the Kmesh community!

Kmesh: Industry's First Kernel-Based Sidecarless Traffic Management Engine​

eBPF and Sidecarless Being the Future of Service Mesh​

Service mesh has grown in popularity over recent years, but despite this, the sidecar pattern still faces challenges in resource overhead, upgrade and deployment, and latency. How to reduce the proxy overhead and build the sidecarless service mesh has become a longstanding problem in the industry.

At the beginning of project initiation, Kmesh innovatively proposed the industry's first kernel-based sidecarless traffic management engine to resolve this problem. The eBPF and programmable kernel technologies are used to sink L4–L7 traffic management into the OS. The traffic does not need to pass through a proxy and the service communication path in this case reduces from three hops to just one hop, eliminating the proxy overhead and implementing sidecarless service mesh.

Advantages of Kmesh​

• High performance Making use of kernel, it provides native support for L4–L7 traffic governance underneath, reducing the microservice forwarding latency by 60% and improving the microservice bootstrap performance by 40% compared with sidecar.
• Low overhead Workload has no sidecar injected, reducing the data plane overhead by 70%.
• High availability Kernel traffic management does not terminate connections, and Kmesh component upgrade and restart do not affect existing service connections.
• Zero-trust network Transparent zero-trust network can be achieved based on kernel mTLS.
• Security isolation eBPF-based VM security and cgroup-level governance isolation are supported.
• Flexible management mode In addition to the full-kernel management, Kmesh also supports slicing L4 and L7 management for isolation. The kernel eBPF program and waypoint component process L4 and L7 traffic respectively. It allows users to gradually migrate from layer 4 service management to layer 7 service management.
• Seamless compatibility It can seamlessly integrate with any control plane that supports xDS protocol in theory. Istio is the first one that Kmesh integrates with, and both Istio APIs and Gateway APIs are supported. Meanwhile, Kmesh can interop with sidecar, which allows migrating from sidecar to Kmesh seamlessly.

Why Kmesh?​

Kmesh is built with sidecarless network architecture, which is currently recognized by both the Istio community and the Cilium community. Sidecarless is also widely accepted by users. Compared with sidecar, sidecarless avoids extra resource overhead. It separates the application and proxy lifecycles, eliminating one-to-one binding and streamlining deployment and maintenance.

Kmesh leverages eBPF technology to perform traffic management in kernel mode, ensuring that traffic management operates seamlessly with traffic flows. By preventing service connections from being cut off, Kmesh reduces the number of connections along the traffic path and minimizes application access delays.

One significant drawback of user-mode traffic management is that proxy upgrades can result in service traffic loss. Kmesh addresses this issue by harnessing programmable kernel technology. In doing so, Kmesh gains a substantial industry advantage. The potential of eBPF technology is evident, and it is poised to drive further network innovations.

Kmesh also provides an advanced mode that enhances L7 traffic management by separating L4 and L7. This separation allows finer-grained physical isolation. Tenants, namespaces, or services can exclusively utilize L7 proxy waypoints. Waypoints can be dynamically scaled-in/scaled-out based on traffic load, which is more flexible and reliable. Unlike traditional centralized gateways, waypoints have no single point of failure.

Therefore, we firmly believe that the optimal architecture of the sidecarless mode is combining eBPF technology with waypoint. This approach aims to reduce resource overhead and latency. Specifically, eBPF handles L4 routing and straightforward L7 traffic management on nodes, while more complex L7 protocols are routed to waypoint for comprehensive management.

Contributing to the Community​

Kmesh, initiated by Huawei and incubated in the openEuler community, is currently hosted on GitHub as an independent project. It offers users traffic management technical solutions with exceptional performance.

Huawei, as the first vendor in China to engage in service mesh, has been contributing to the Istio community since 2018, making most contributions in Asia. Huawei also holds a seat in the Istio Steering Committee, which is responsible for governing Istio community.

Having accumulated extensive experience within the Istio community, we aspire to foster Kmesh's growth in an open and neutral manner. Our goal is to create an industry-leading sidecarless service mesh benchmark solution, catering to diverse industries, and fostering the healthy and organized evolution of service mesh technology. Kmesh is currently undergoing rapid development, and we warmly invite passionate individuals to join our efforts.

Kmesh community: https://github.com/kmesh-net/kmesh

Reference​

[1] CNCF Landscape: https://landscape.cncf.io/

[2] Introducing Ambient Mesh: https://istio.io/latest/blog/2022/introducing-ambient-mesh/

[3] Huawei Cloud ASM: https://support.huaweicloud.com/intl/en-us/asm/index.html

[4] Quick Start of Kmesh: https://kmesh.net/en/docs/setup/quickstart/

Introduction to Kmesh​

Based on eBPF and programmable kernel technology, Kmesh sinks traffic management into the OS, eliminating the need for a proxy layer on the data path, and realizing a kernel-level sidecarless mesh data plane. Key Capabilities of Kmesh:

• High Performance: Native support for L4~L7 traffic management functions in the kernel, enabling governance processes without passing through physical proxy components. This reduces the service communication path within the mesh from three hops under a proxy architecture to one hop, significantly improving the forwarding performance of the mesh data plane.
• Low Overhead: No need to deploy Sidecar alongside workload pods, greatly reducing the resource overhead of the mesh infrastructure.
• Safety Isolation: Runtime security based on eBPF, which support cgroup-level governance isolation.
• Seamless Compatibility: Support integration with control planes of service meshes that support the xDS protocol, such as Istiod, and can also work collaboratively with existing Sidecar meshes. The main components of Kmesh include:
• kmesh-controller: Responsible for bpf lifecycle management, xDS resources subscription, observability and other functions.
• kmesh-api: Adapter layer consisting of the orchestration API after xDS transformation, observability channels, etc.
• kmesh-runtime: Runtime implemented in the kernel to support L4~L7 traffic orchestration; the capability for Layer 7 orchestration runtime depends on enhancements to the kernel.
• kmesh-orchestration: Implements L4~L7 traffic orchestration based on eBPF, such as routing, canary releasing, load balancing, etc.
• kmesh-probe: Observability probe providing end-to-end observability capabilities.

Performance Testing​

We used fortio to test the performance of Istio (Envoy) and Istio(Kmesh) under the same traffic management scenarios, while also testing the latency of service communication based on kube-proxy(iptables) as a baseline reference. Latency comparison at different numbers of connections: Comparison of CPU overhead at the same QPS: From the test results, we can see that:

• Kmesh's forwarding latency is almost close to the native Kubernetes forwarding latency, and it shows a significant improvement on latency compared with Istio sidecar mode.
• With the same QPS, Kmesh's CPU overhead is basically equivalent to the native CPU overhead of Kubernetes, showing a significant reduction compared to Istio sidecar mode. For detailed demo test details, you can watch our demonstration video:

Key Technology Analysis of Kmesh​

Kernel-Level Traffic Orchestration Runtime​

In microservice communication, connections are generally established before sending business messages. If you want to orchestrate business messages seamlessly, traffic interception is usually required. After orchestration is completed, the adjusted messages are then forwarded based on the interception. This is the current implementation of Proxy agents. Kmesh aims to complete governance work along with the flow and delays link establishment until the business message sending phase in order to achieve higher orchestration processing performance.

• Pseudo Connection Establishment The pre_connect process loads the BPF prog. If the target address being accessed is within the scope of the xDS listener, it calls bpf_setsockopt to reload the tcp proto hook of the current socket to the kmesh_defer kernel module via TCP_ULP.
• Delayed Connection Establishment The kmesh_defer kernel module rewrites the connect/send hooks (enhancements made on the native hooks):
  • When the service first reaches the connect hook, it sets the bpf_defer_connect flag and does not trigger the handshake process.
  • In the send hook, if the sock has the bpf_defer_connect flag set, it triggers the connect. At this point, it calls the BPF_SOCK_OPS_TCP_DEFER_CONNECT_CB through an extended BPF prog, completes traffic management, and then establishes a connection and sends messages based on the adjusted communication quintuple and messages. The entire governance process is roughly as shown in the following diagram:

xDS Rule Management​

The xDS model is a hierarchical tree-like rule expression, and different version model definitions may be adjusted. Kmesh needs to convert the model information into eBPF map storage while maintaining the hierarchical relationships between model rules.

• Convert the xDS model into eBPF map data Specific Process:

1. Kmesh subscribes to Istiod's xDS model and converts the pb model into a C data structure style based on protobuf-c.
2. For the top-level model (e.g., listener), Kmesh defines well-known map tables corresponding to it, and the data structure of the value reuses the C struct exported by protobuf-c.
3. Map updates start from the well-known map table of the top-level model. For pointer members in the records, the xds-adapter creates an inner-map table to store the actual data area pointed to by the pointer, adds the map fd of the inner-map to the map-in-map table, and finally uses its key (index) in the map-in-map table as the value for the pointer member.

• map-in-map addresses the hierarchical nature of the xDS model For the value members of map records, if they are pointer variables (involving references to other data structures), the data area pointed to is stored through the inner-map:
  • If the value member is a basic data type (such as int), it is accessed directly.
  • If the value member is a pointer type, the value stored by the pointer is the index of the inner-map storing the actual data in the map-in-map table (Note: the index is written in coordination with updating the bpf map in the kmesh-daemon's xds-adapter module). When accessing, the map fd of the inner-map is first found based on the index, and then the actual data is retrieved from the inner-map table. For multi-level pointer members, this process is repeated until all pointer information is stripped away.

Traffic Management Orchestration Implementation​

The governance rules of xDS are complex, involving hierarchical matching, which exceeds the complexity limit of a single eBPF program. Based on the eBPF Tail Calls feature, Kmesh divides the governance process into multiple independent eBPF progs, thus having good scalability.

Key Features of Kmesh latest​

• One-Click Deployment The Kmesh community has released deployment images for Kmesh ^[2] and supports one-click deployment of Kmesh via helm ^[3].
• Namespace-based Enablement for Kmesh Kmesh supports enabling traffic takeover scope based on namespace, such as: kubectl label namespace default label istio.io/dataplane-mode=Kmesh
• Seamless Integration with Istio Sidecar For namespaces in the cluster that do not have Kmesh data plane enabled, if a sidecar proxy (such as Envoy) is used, Kmesh also supports interconnection. Additionally, sockmap can be used to accelerate traffic forwarding for the sidecar, resulting in a 10% to 15% performance improvement in forwarding without impacting the business process.
• Automated Integration with Service Mesh Control Plane Kmesh supports automatic integration with Istiod, and in theory, any mesh control plane following the xDS protocol can integrate with Kmesh. This can be specified by modifying the MESH_CONTROLLER environment variable in the yaml.
• Support xDS/workload Kmesh supports the xDS model, enabling TCP traffic forwarding, HTTP/1.1 header matching, routing, and gray release. It also supports random and round-robin load balancing algorithms. Furthermore, it provides basic forwarding functionality based on the workload model.

Looking Ahead​

Kmesh is a high-performance traffic management engine based on eBPF and programmable kernel implementation. Compared to industry solutions, it offers higher forwarding performance and lower resource overhead. It can run in compatible mode on kernel versions without enhanced patches. For full governance capabilities of Kmesh, the current openEuler 23.03 version^[4] already provides native support, while other operating systems require building based on enhanced patches^[5]. Kmesh is gradually evolving into a more popular traffic management engine, and there is still a lot of work to be done. Currently, support for forwarding L7 traffic to waypoint and mTLS features has been planned. We welcome everyone to try Kmesh and stay connected with the Kmesh community^[6]. Your participation is also highly anticipated.

Meet Kmesh at KubeCon + CloudNativeCon Europe 2024​

Kmesh will participate in several activities during KubeCon + CloudNativeCon Europe 2024, including:

Kmesh Display​

March 20-22 all day : Stop by Booth J1 at KubeCon to speak with an expert or see a demo!

Kmesh OpenSpeech​

Friday，Mar 22, 11:10-11:30 am CET Kernel-native Traffic Governance Framework Brings New Performance Experience

Reference Links​

[1] Kmesh releases: https://github.com/kmesh-net/kmesh/releases

[2] Deployment images for Kmesh: https://github.com/orgs/kmesh-net/packages

[3] One-click deployment of Kmesh: https://github.com/kmesh-net/kmesh?tab=readme-ov-file#quick-start

[4] openEuler 23.03 version: https://repo.openeuler.org/openEuler-23.03/

[5] Building based on enhanced patches: https://github.com/kmesh-net/kmesh/blob/main/docs/kmesh_kernel_compile.md

[6] Kmesh community address: https://github.com/kmesh-net/kmesh

The concept of a service mesh was introduced by Buoyant, the company behind the development of Linkerd software, in 2016. Willian Morgan, the CEO of Linkerd, provided the initial definition of a service mesh:

A service mesh is a dedicated layer for handling service-to-service communication. It’s responsible for the reliable delivery of requests through the complex topology of services that comprise a modern, cloud-native application. In practice, the service mesh is typically implemented as an array of lightweight network proxies that are deployed alongside application code, without the application needing to be aware.

In simple terms, a service mesh is an layer that handles communication between services. It ensures transparent and reliable network communication for modern cloud-native applications through an array of lightweight network proxies.

The essence of a service mesh is to address the challenge of how microservices can communicate effectively. By implementing governance rules such as load balancing, canary routing, and circuit breaking, the service mesh orchestrates traffic flow to maximize the capabilities of the service cluster. It is a product of the evolution of service governance.

We can divide the evolution of service governance into three generations and compare them. From this evolution, we can observe that service governance capabilities gradually decouple from business logic and move down to the level.

As an layer for handling service-to-service communication, a service mesh effectively fills the gaps in microservice governance in Kubernetes (k8s). As the next-generation technology for cloud-native environments, it has become a critical component of cloud.

In recent years, service mesh has gained significant attention, leading to the emergence of various service mesh software solutions such as Linkerd, Istio, Consul Connect, and Kuma. While they may have slight differences in their software architecture, let's take Istio as an example (one of the most popular service mesh projects) to illustrate the basic architecture of a service mesh:

Taking a Kubernetes cluster as an example, when a Pod instance is created, the service mesh software transparently deploys a proxy container (also known as a sidecar, with Envoy being the default sidecar software in Istio) alongside the application code. The basic flow of communication between Pods is as follows:

• Traffic is transparently intercepted by iptables rules and directed to the proxy component within the Pod.
• The proxy component applies traffic governance logic (e.g., circuit breaking, routing, load balancing) to determine the destination service instance and forwards the message.
• The proxy component within the destination Pod intercepts the incoming traffic, applies basic traffic governance logic (e.g., rate limiting), and then forwards the traffic to the Pod.
• After processing, the response is returned to the requesting Pod following the original path.

Challenges of the Service Mesh Data Plane​

As mentioned earlier, a service mesh introduces a proxy layer in the data plane to achieve transparent service governance. However, this comes at a cost: the introduction of the proxy layer inevitably increases latency and decreases performance in service communication.

Using data provided by the Istio official website as an example, in a cluster environment, the average per-hop latency between microservices increases by 2.65ms. Considering that in a microservice cluster, an external request often involves multiple invocations between microservices, the latency overhead introduced by the service mesh is significant. As service mesh adoption continues to grow, the additional latency introduced by the proxy architecture has become a critical challenge.

To address this issue, we conducted performance testing on L7 load balancing for HTTP services to analyze the communication performance of the service mesh. The breakdown of time consumption is as follows:

From the detailed analysis of mesh traffic, we can see that inter-service communication transitions from one connection establishment to three, and from two protocol stack traversals to six. The time consumption mainly focuses on data copying, connection establishment, context switching, etc. The actual overhead of traffic governance is relatively small.

This raises the question: Can we reduce the latency overhead of the service mesh while maintaining transparent governance for applications?

High-Performance Service Mesh Data Plane: Kmesh​

Based on the performance analysis mentioned above, we have conducted a two-stage optimization for the service mesh data plane.

Sockmap: Accelerating the Service Mesh Data Plane with Sockmap​

Sockmap is an eBPF feature introduced in Linux 4.14, which enables the redirection of data flows between sockets within a node without going through the complex kernel protocol stack. It optimizes the performance of data forwarding between sockets on the network path.

In the context of a service mesh, the default communication between the business container within a Pod and the local proxy component goes through the complete kernel protocol stack, incurring unnecessary overhead. This overhead can be optimized using Sockmap. The following diagram illustrates the concept:

The basic steps for accelerating the service mesh data plane with Sockmap are as follows:

• During the connection establishment process, an eBPF program (of type BPF_PROG_TYPE_SOCK_OPS) is attached to intercept all TCP connection establishment actions.
  • In the BPF_SOCK_OPS_ACTIVE_ESTABLISHED_CB state, a client-side Sockmap record is added.
  • In the BPF_SOCK_OPS_PASSIVE_ESTABLISHED_CB state, a server-side Sockmap record is added.
  • The socket information of both communication parties is stored in the Sockmap table.
• During the sendmsg process, an eBPF program (of type BPF_PROG_TYPE_SK_MSG) is attached to intercept message sending actions.
  • The program looks up the Sockmap table based on the current socket information and associates it with the socket information of the destination party. It then directly redirects the traffic to the receiving queue of the destination socket.

By leveraging Sockmap to accelerate the service mesh data plane, we observed a 10% to 15% reduction in average latency for service access in a scenario with 60 long-lived connections.

While Sockmap is a commonly used solution for optimizing the service mesh data plane, it does not fully address the performance challenges associated with service mesh latency.

Offload: Offloading Traffic Governance to the Operating System with Programmable Kernel​

Based on the performance analysis mentioned earlier, it is evident that a significant portion of the additional overhead introduced by the service mesh is spent on redirecting traffic to the proxy component. The actual overhead of performing traffic governance is relatively small. This raises the question: Can we bypass the proxy component and perform traffic governance directly within the kernel, which is naturally involved in network communication?

Kmesh is our proposed high-performance service mesh data plane solution that leverages a programmable kernel to offload traffic governance to the operating system. With Kmesh, the data plane no longer goes through the proxy component, and service-to-service communication is reduced from three hops to one hop, enabling traffic governance to be performed along the path of traffic transmission. The flow of traffic between microservices in Kmesh is illustrated below:

The software architecture of Kmesh consists of the following components:

• kmesh-controller: The management program responsible for Kmesh lifecycle management, XDS protocol integration, observability, and other functions.
• kmesh-api: The API interface layer provided by Kmesh, including the orchestrated API transformed from XDS and observability channels.
• kmesh-runtime: The runtime implemented in the kernel that supports L3-L7 traffic orchestration.
• kmesh-orchestration: The L3-L7 traffic orchestration implemented based on eBPF, including routing, canary deployments, load balancing, and more.
• kmesh-probe: The observability probe that provides end-to-end observability capabilities.

We deployed an Istio mesh environment and conducted comparative testing on the performance of different data plane solutions (Envoy/Kmesh) for L7 load balancing of HTTP services using the Fortio testing tool. The results showed that Kmesh achieved a 5x performance improvement in service-to-service communication compared to the native data plane of Istio (Envoy).

It is worth noting that we also tested the performance of service-to-service communication in a non-mesh environment based on Kubernetes, and the performance was comparable to Kmesh. This further validates the latency performance of Kmesh. (The testing scenario involved L7 load balancing in a laboratory environment, and the actual performance in real-world governance scenarios may not be as ideal. Preliminary evaluations suggest a 2-3x improvement over Istio.)

Conclusion​

As the next-generation technology for cloud-native environments, a service mesh provides transparent service governance for applications. However, the proxy architecture introduces additional latency overhead, which has become a critical challenge for widespread adoption of service meshes. Kmesh proposes a new approach by offloading traffic governance to the operating system using a programmable kernel. By doing so, Kmesh significantly improves the performance of the service mesh data plane. It offers a fresh perspective for the development of the service mesh data plane.

reference​

https://linkerd.io/2017/04/25/whats-a-service-mesh-and-why-do-i-need-one

https://istio.io/latest/docs/ops/deployment/architecture

https://istio.io/v1.16/docs/ops/deployment/performance-and-scalability/#performance-summary-for-istio-hahahugoshortcode-s0-hbhb

Early microservices architectures faced various challenges such as service discovery, load balancing, and authorization/authentication. Initially, practitioners of microservices implemented their own distributed communication systems to address these challenges. However, this approach resulted in redundant business functionality. To solve this problem, a solution was proposed: extracting the common distributed system communication code into a framework and providing it as a library for programmatic use. However, this seemingly perfect solution had several fatal weaknesses:

• The framework required invasive modifications to the business code, necessitating developers to learn how to use the framework.
• The framework could not be used across different programming languages.
• Managing compatibility issues with complex project frameworks and library versions was challenging, as upgrading the framework often forced businesses to upgrade as well.

As microservices architecture evolved, the first-generation service mesh emerged, represented by Linkeerd/Envoy/NginxMesh and the sidecar proxy pattern. As an infrastructure layer, the sidecar proxy is decoupled from the business processes and deployed alongside them. It takes over the communication between business components, abstracting the network data transmission into a separate layer. This layer centrally handles functions such as service discovery, load balancing, and authorization/authentication required by distributed systems, achieving reliable transmission of requests in the network topology. It provides a more comprehensive solution to the problems encountered with microservice framework libraries.

However, there is no silver bullet in software development. While service mesh brings many conveniences, it also inevitably presents some issues. In traditional approaches, messages between clients and servers only need to go through the kernel protocol stack once to complete the message delivery. In the sidecar proxy mode, however, the business traffic is typically intercepted using the iptables capability of the kernel, resulting in multiple passes through the kernel protocol stack for business data. This increases latency and reduces throughput.

We conducted benchmark tests on service mesh performance and found that the sidecar mode (with Envoy) had significantly worse latency compared to the non-sidecar mode (without Envoy).

Accelerating ServiceMesh with eBPF Capabilities​

Is there a way to reduce and eliminate the impact of network latency while enjoying the convenience provided by ServiceMesh? Here, we have to mention eBPF technology. eBPF is a revolutionary technology in the kernel that aims to extend the kernel's capabilities more securely and effectively without modifying the kernel code or loading kernel modules. By using eBPF capabilities to bypass the kernel network protocol stack, we can reduce network latency and improve the user experience of ServiceMesh. This is currently a common practice in the industry.

To achieve the goal of bypassing the kernel network protocol stack, we need to utilize two capabilities provided by eBPF: sockops and socket redirection.

• Sockops provides the ability to identify and store sockets (usually identified by a tuple of four elements) in a sockmap data structure when creating TCP connections.
• Socket redirection supports referencing sockets from the sockmap based on keys during the transmission of TCP data. When a match is found, the data can be directly forwarded to the corresponding socket.
• For sockets not found in the sockmap, the packets are sent through the kernel network protocol stack as usual.

By combining these capabilities, we can forward packets directly to the corresponding socket without going through the kernel network protocol stack, reducing the time spent in the kernel network protocol stack.

During the process of establishing a TCP socket connection, there are actually two connection establishment processes: forward connection and reverse connection. In general, iptables information is used to obtain the actual IP address and port number during the connection establishment of both forward and reverse connections. By calling bpf_get_sockopt, we can actively obtain the addresses transformed by iptables in the eBPF function. This allows us to establish an auxiliary map to store the corresponding relationships between forward and reverse connections. When performing socket redirection, we first look for the connection information of the peer from the auxiliary map. If the connection information is found successfully, we proceed with the socket forwarding action. The principle is shown in the following diagram:

We conducted actual tests on openEuler 21.03 to evaluate the acceleration achieved through sockmap capabilities. The test environment was openEuler-21.03 / 5.10.0-4.17.0.28.oe1.x86_64, and the network configuration was set as fortio-envoy-envoy:80-fortio_server:80.

Based on the test results, compared to not using ServiceMesh, the QPS was improved by approximately 18% and the average latency was reduced by 15% when utilizing sockmap acceleration.

Can service mesh performance overhead be eliminated entirely?​

However, despite the significant acceleration achieved with sockmap for ServiceMesh, there still remains a considerable gap compared to not using ServiceMesh. This is primarily due to the substantial latency overhead introduced by the current proxy architecture of the service mesh. To completely eliminate the performance impact introduced by the service mesh, it is crucial to optimize at the architectural level.

Kmesh is actively exploring new approaches at the data plane architecture level to address this challenge, and the industry has also made significant efforts in this regard. In upcoming articles, we will provide detailed insights into these initiatives and optimization measures.