Apache SkyWalking – Apache SkyWalking

Blog: eBPF enhanced HTTP observability - L7 metrics and tracing

Thu, 12 Jan 2023 00:00:00 +0000

Background

Apache SkyWalking is an open-source Application Performance Management system that helps users collect and aggregate logs, traces, metrics, and events for display on a UI. In the previous article, we introduced how to use Apache SkyWalking Rover to analyze the network performance issue in the service mesh environment. However, in business scenarios, users often rely on mature layer 7 protocols, such as HTTP, for interactions between systems. In this article, we will discuss how to use eBPF techniques to analyze performance bottlenecks of layer 7 protocols and how to enhance the tracing system using network sampling.

This article will show how to use Apache SkyWalking with eBPF to enhance metrics and traces in HTTP observability.

HTTP Protocol Analysis

HTTP is one of the most common Layer 7 protocols and is usually used to provide services to external parties and for inter-system communication. In the following sections, we will show how to identify and analyze HTTP/1.x protocols.

Protocol Identification

In HTTP/1.x, the client and server communicate through a single file descriptor (FD) on each side. Figure 1 shows the process of communication involving the following steps:

Connect/accept: The client establishes a connection with the HTTP server, or the server accepts a connection from the client.
Read/write (multiple times): The client or server reads and writes HTTPS requests and responses. A single request-response pair occurs within the same connection on each side.
Close: The client and server close the connection.

To obtain HTTP content, it’s necessary to read it from the second step of this process. As defined in the RFC, the content is contained within the data of the Layer 4 protocol and can be obtained by parsing the data. The request and response pair can be correlated because they both occur within the same connection on each side.

Figure 1: HTTP communication timeline.

HTTP Pipeline

HTTP pipelining is a feature of HTTP/1.1 that enables multiple HTTP requests to be sent over a single TCP connection without waiting for the corresponding responses. This feature is important because it ensures that the order of the responses on the server side matches the order of the requests.

Figure 2 illustrates how this works. Consider the following scenario: an HTTP client sends multiple requests to a server, and the server responds by sending the HTTP responses in the same order as the requests. This means that the first request sent by the client will receive the first response from the server, the second request will receive the second response, and so on.

When designing HTTP parsing, we should follow this principle by adding request data to a list and removing the first item when parsing a response. This ensures that the responses are processed in the correct order.

Figure 2: HTTP/1.1 pipeline.

Metrics

Based on the identification of the HTTP content and process topology diagram mentioned in the previous article, we can combine these two to generate process-to-process metrics data.

Figure 3 shows the metrics that currently support the analysis between the two processes. Based on the HTTP request and response data, we can analyze the following data:

Metrics Name	Type	Unit	Description
Request CPM(Call Per Minute)	Counter	count	The HTTP request count
Response Status CPM(Call Per Minute)	Counter	count	The count of per HTTP response status code
Request Package Size	Counter/Histogram	Byte	The request package size
Response Package Size	Counter/Histogram	Byte	The response package size
Client Duration	Counter/Histogram	Millisecond	The duration of single HTTP response on the client side
Server Duration	Counter/Histogram	Millisecond	The duration of single HTTP response on the server side

Figure 3: Process-to-process metrics.

HTTP and Trace

During the HTTP process, if we unpack the HTTP requests and responses from raw data, we can use this data to correlate with the existing tracing system.

Trace Context Identification

In order to track the flow of requests between multiple services, the trace system usually creates a trace context when a request enters a service and passes it along to other services during the request-response process. For example, when an HTTP request is sent to another server, the trace context is included in the request header.

Figure 4 displays the raw content of an HTTP request intercepted by Wireshark. The trace context information generated by the Zipkin Tracing system can be identified by the “X-B3” prefix in the header. By using eBPF to intercept the trace context in the HTTP header, we can connect the current request with the trace system.

Figure 4: View of HTTP headers in Wireshark.

Trace Event

We have added the concept of an event to traces. An event can be attached to a span and consists of start and end times, tags, and summaries, allowing us to attach any desired information to the Trace.

When performing eBPF network profiling, two events can be generated based on the request-response data. Figure 5 illustrates what happens when a service performs an HTTP request with profiling. The trace system generates trace context information and sends it in the request. When the service executes in the kernel, we can generate an event for the corresponding trace span by interacting with the request-response data and execution time in the kernel space.

Previously, we could only observe the execution status in the user space. However, by combining traces and eBPF technologies, we can now also get more information about the current trace in the kernel space, which would impact less performance for the target service if we do similar things in the tracing SDK and agent.

Figure 5: Logical view of profiling an HTTP request and response.

Sampling

To ensure efficient data storage and minimize unnecessary data sampling, we use a sampling mechanism for traces in our system. This mechanism triggers sampling only when certain conditions are met. We also provide a list of the top N traces, which allows users to quickly access the relevant request information for a specific trace.

To help users easily identify and analyze relevant events, we offer three different sampling rules:

Slow Traces: Sampling is triggered when the response time for a request exceeds a specified threshold.
Response Status [400, 500): Sampling is triggered when the response status code is greater than or equal to 400 and less than 500.
Response Status [500, 600): Sampling is triggered when the response status code is greater than or equal to 500 and less than 600.

In addition, we recognize that not all request or response raw data may be necessary for analysis. For example, users may be more interested in requesting data when trying to identify performance issues, while they may be more interested in response data when troubleshooting errors. As such, we also provide configuration options for request or response events to allow users to specify which type of data they would like to sample.

Profiling in a Service Mesh

The SkyWalking and SkyWalking Rover projects have already implemented the HTTP protocol analyze and trace associations. How do they perform when running in a service mesh environment?

Deployment

Figure 6 demonstrates the deployment of SkyWalking and SkyWalking Rover in a service mesh environment. SkyWalking Rover is deployed as a DaemonSet on each machine where a service is located and communicates with the SkyWalking backend cluster. It automatically recognizes the services on the machine and reports metadata information to the SkyWalking backend cluster. When a new network profiling task arises, SkyWalking Rover senses the task and analyzes the designated processes, collecting and aggregating network data before ultimately reporting it back to the SkyWalking backend service.

Figure 6: SkyWalking rover deployment topology in a service mesh.

Tracing Systems

Starting from version 9.3.0, the SkyWalking backend fully supports all functions in the Zipkin server. Therefore, the SkyWalking backend can collect traces from both the SkyWalking and Zipkin protocols. Similarly, SkyWalking Rover can identify and analyze trace context in both the SkyWalking and Zipkin trace systems. In the following two sections, network analysis results will be displayed in the SkyWalking and Zipkin UI respectively.

SkyWalking

When SkyWalking performs network profiling, similar to the TCP metrics in the previous article, the SkyWalking UI will first display the topology between processes. When you open the dashboard of the line representing the traffic metrics between processes, you can see the metrics of HTTP traffic from the “HTTP/1.x” tab and the sampled HTTP requests with tracing in the “HTTP Requests” tab.

As shown in Figure 7, there are three lists in the tab, each corresponding to a condition in the event sampling rules. Each list displays the traces that meet the pre-specified conditions. When you click on an item in the trace list, you can view the complete trace.

Figure 7: Sampled HTTP requests within tracing context.

When you click on an item in the trace list, you can quickly view the specified trace. In Figure 8, we can see that in the current service-related span, there is a tag with a number indicating how many HTTP events are related to that trace span.

Since we are in a service mesh environment, each service involves interacting with Envoy. Therefore, the current span includes Envoy’s request and response information. Additionally, since the current service has both incoming and outgoing requests, there are events in the corresponding span.

Figure 8: Events in the trace detail.

When the span is clicked, the details of the span will be displayed. If there are events in the current span, the relevant event information will be displayed on a time axis. As shown in Figure 9, there are a total of 6 related events in the current Span. Each event represents a data sample of an HTTP request/response. One of the events spans multiple time ranges, indicating a longer system call time. It may be due to a blocked system call, depending on the implementation details of the HTTP request in different languages. This can also help us query the possible causes of errors.

Figure 9: Events in one trace span.

Finally, we can click on a specific event to see its complete information. As shown in Figure 10, it displays the sampling information of a request, including the SkyWalking trace context protocol contained in the request header from the HTTP raw data. The raw request data allows you to quickly re-request the request to solve any issues.

Figure 10: The detail of the event.

Zipkin

Zipkin is one of the most widely used distributed tracing systems in the world. SkyWalking can function as an alternative server to provide advanced features for Zipkin users. Here, we use this way to bring the feature into the Zipkin ecosystem out-of-box. The new events would also be treated as a kind of Zipkin’s tags and annotations.

To add events to a Zipkin span, we need to do the following:

Split the start and end times of each event into two annotations with a canonical name.
Add the sampled HTTP raw data from the event to the Zipkin span tags, using the same event name for corresponding purposes.

Figures 11 and 12 show annotations and tags in the same span. In these figures, we can see that the span includes at least two events with the same event name and sequence suffix (e.g., “Start/Finished HTTP Request/Response Sampling-x” in the figure). Both events have separate timestamps to represent their relative times within the span. In the tags, the data content of the corresponding event is represented by the event name and sequence number, respectively.

Figure 11: Event timestamp in the Zipkin span annotation.

Figure 12: Event raw data in the Zipkin span tag.

Demo

In this section, we demonstrate how to perform network profiling in a service mesh and complete metrics collection and HTTP raw data sampling. To follow along, you will need a running Kubernetes environment.

Deploy SkyWalking Showcase

SkyWalking Showcase contains a complete set of example services and can be monitored using SkyWalking. For more information, please check the official documentation.

In this demo, we only deploy service, the latest released SkyWalking OAP, and UI.

export SW_OAP_IMAGE=apache/skywalking-oap-server:9.3.0
export SW_UI_IMAGE=apache/skywalking-ui:9.3.0
export SW_ROVER_IMAGE=apache/skywalking-rover:0.4.0

export FEATURE_FLAGS=mesh-with-agent,single-node,elasticsearch,rover
make deploy.kubernetes

After deployment is complete, please run the following script to open SkyWalking UI: http://localhost:8080/.

kubectl port-forward svc/ui 8080:8080 --namespace default

Start Network Profiling Task

Currently, we can select the specific instances that we wish to monitor by clicking the Data Plane item in the Service Mesh panel and the Service item in the Kubernetes panel.

In figure 13, we have selected an instance with a list of tasks in the network profiling tab.

Figure 13: Network Profiling tab in the Data Plane.

When we click the Start button, as shown in Figure 14, we need to specify the sampling rules for the profiling task. The sampling rules consist of one or more rules, each of which is distinguished by a different URI regular expression. When the HTTP request URI matches the regular expression, the rule is used. If the URI regular expression is empty, the default rule is used. Using multiple rules can help us make different sampling configurations for different requests.

Each rule has three parameters to determine if sampling is needed:

Minimal Request Duration (ms): requests with a response time exceeding the specified time will be sampled.
Sampling response status code between 400 and 499: all status codes in the range [400-499) will be sampled.
Sampling response status code between 500 and 599: all status codes in the range [500-599) will be sampled.

Once the sampling configuration is complete, we can create the task.

Figure 14: Create network profiling task page.

Done!

After a few seconds, you will see the process topology appear on the right side of the page.

When you click on the line between processes, you can view the data between the two processes, which is divided into three tabs:

TCP: displays TCP-related metrics.
HTTP/1.x: displays metrics in the HTTP 1 protocol.
HTTP Requests: displays the analyzed request and saves it to a list according to the sampling rule.

Figure 16: TCP metrics in a network profiling task.

Figure 17: HTTP/1.x metrics in a network profiling task.

Figure 18: HTTP sampled requests in a network profiling task.

Conclusion

In this article, we detailed the overview of how to analyze the Layer 7 HTTP/1.x protocol in network analysis, and how to associate it with existing trace systems. This allows us to extend the scope of data we can observe from just user space to also include kernel-space data.

In the future, we will delve further into the analysis of kernel data, such as collecting information on TCP packet size, transmission frequency, network card, and help on enhancing distributed tracing from another perspective.

Additional Resources

Blog: [License Issue] Aliyun(阿里云)'s trace analysis service copied SkyWalking's trace profiling page.

Tue, 03 Jan 2023 00:00:00 +0000

Apache SkyWalking is an open-source APM for a distributed system, Apache Software Foundation top-level project.

On Jan. 3rd, 2023, we received reports about Aliyun Trace Analysis Service. It provides a cloud service compatible with SkyWalking trace APIs and agents.

On their product page, there is a best-practice document describing about their service is not SkyWalking OAP, but can work with SkyWalking agents to support SkyWalking’s In-Process(Trace) Profiling.

BUT, they copied the whole page of SkyWalking’s profiling UI, including page layout, words, and profiling task setup. The only difference is the color schemes.

SkyWalking UI

Aliyun Trace Analysis UI on their document page

The UI visualization is a part of the copyright. Aliyun declared their backend is NOT a re-distribution of SkyWalking repeatedly on their website, and they never mentioned this page is actually copied from upstream.

This is a LICENSE issue, violating SkyWalking’s copyright and Apache 2.0 License. They don’t respect Apache Software Foundation and Apache SkyWalking’s IP and Branding.

Blog: Boost Root Cause Analysis Quickly With SkyWalking’s New Trace-Metrics Association Feature

Mon, 19 Dec 2022 00:00:00 +0000

Observability for modern distributed applications work is critical for understanding how they behave under a variety of conditions and for troubleshooting and resolving issues when they arise. Traces, metrics, and logs are regarded as fundamental parts of the observability stack. Traces are the footprints of distributed system executions, meanwhile, metrics measure system performance with numbers in the timeline. Essentially, they measure the performance from two dimensions. Being able to quickly visualize the connection between traces and corresponding metrics makes it possible to quickly diagnose which process flows are correlated to potentially pathological behavior. This powerful new capability is now available in SkyWalking 9.3.0.

The SkyWalking project started only with tracing, with a focus on 100% sampling-based metrics and topology analysis since 2018. When users face anomaly trends of time-series metrics, like a peak on the line chart, or histogram shows a larger gap between p95 and p95, the immediate question is, why is this happening? One of SkyWalking’s latest features, the trace-metric association, makes it much easier to answer that question and to address the root cause.

How Are Metrics Generated?

SkyWalking provides three ways to calculate metrics:

Metrics built from trace spans, depending on the span’s layer, kind, and tags.
Metrics extracted from logs—a kind of keyword and tags-based metrics extraction.
Metrics reported from mature and mainstream metrics/meter systems, such as OpenTelemetry, Prometheus, and Zabbix.

Tracing tracks the processes of requests between an application’s services. Most systems that generate traffic and performance-related metrics also generate tracing data, either from server-side trace-based aggregations or through client SDKs.

Use SkyWalking to Reduce the Traditional Cost of Trace Indexing

Tracing data and visualization are critical troubleshooting tools for both developers and operators alike because of how helpful they are in locating issue boundaries. But, because it has traditionally been difficult to find associations between metrics and traces, teams have added increasingly more tags into the spans, and search through various combinations. This trend of increased instrumentation and searching has required increased infrastructure investment to support this kind of search. SkyWalking’s metrics and tracing association capabilities can help reduce the cost of indexing and searching that data.

Find the Associated Trace

When looking for association between metrics and traces, the kind of metrics we’re dealing with determines their relationships to traces. Let’s review the standard request rate, error, and duration (RED) metrics to see how it works.

Success Rate Metrics

The success rate is determined by the return code, RPC response code, or exceptions of the process. When the success rate decreases, looking for errors in the traces of this service or pod are the first place to look to find clues.

Figure 1: The success rate graph from SkyWalking’s 9.3.0 dashboard with the option to view related traces at a particular time.

Drilling down from the peak of the success rate, SkyWalking lists all traces and their error status that were collected in this particular minute (Figure 2):

Figure 2: SkyWalking shows related traces with an error status.

Requests to /test can be located from the trace, and the span’s tag indicates a 404 response code of the HTTP request.

Figure 3: A detail view of a request to http://frontend/test showing that the URI doesn’t exist.

By looking at the trace data, it becomes immediately clear that the drop in success rate is caused by requests to a nonexistent URI.

Average Response Time

The average response time metric provides a general overview of service performance. When average response time is unstable, this usually means that the system is facing serious performance impacts.

Figure 4: SkyWalking’s query UI for searching for related traces showing traces for requests that exceed a particular duration threshold.

When you drill down from this metric, this query condition (Figure 4) will reveal the slowest traces of the service in this specific minute. Notice, at least 168ms is added as a condition automatically, to avoid scanning a large number of rows in the Database.

Apdex

Apdex—the Application Performance Index—is a measure of response time based against a set threshold. It measures the ratio of satisfactory response times to unsatisfactory response times (Figure 5). The response time is measured from an asset request to completed delivery back to the requestor.

Figure 5: The Apdex formula

A user defines a response time tolerating threshold T. All responses handled in T or less time satisfy the user.

For example, if T is 1.2 seconds and a response completes in 0.5 seconds, then the user is satisfied. All responses greater than 1.2 seconds dissatisfy the user. Responses greater than 4.8 seconds frustrate the user.

When the Apdex score decreases, we need to find related traces from two perspectives: slow traces and error status traces. SkyWalking’s new related tracing features offers a quick way to view both (Figure 6) directly from the Apdex graph.

Figure 6: Show slow trace and error status traces from the Apdex graph

Service Response Time

Percentile MetricThe percentile graph (Figure 7) provides p50, p75, p90, p95, and p99 latency ranks to measure the long-tail issues of service performance.

Figure 7: The service response time percentile graph helps to highlight long-tail issues of service performance.

This percentile graph shows a typical long-tail issue. P99 latency is four times slower than the P95. When we use the association, we see the traces with latency between P95 - P99 and P99 - Infinity.

The traces of requests causing this kind of long-tail phenomena are automatically listing from there.

Figure 8: Query parameters to search for traces based on latency.

Are More Associations Available?

SkyWalking provides more than just associations between between traces and metrics to help you find possible causal relationships and to avoid looking for the proverbial needle in a haystack.

Currently, SkyWalking 9.3.0 offers two more associations: metric-to-metric associations and event-to-metric associations.

Metric-to-metric Associations

There are dozens of metrics on the dashboard—which is great for getting a complete picture of application behavior. During a typical performance issue, the peaks of multiple metrics are affected simultaneously. But, trying to correlate peaks across all of these graphs can be difficult…

Now in SkyWalking 9.3.0, when you click the peak of one graph, the pop-out box lets you see associated metrics.

Figure 9: SkyWalking’s option to view associated metrics.

When you choose that option, all associated metrics graphs will show axis pointers (the dotted vertical lines) in all associated graphs like in Figure 10. This makes it easier to correlate the peaks in different graphs with each other. Often, these correlated peaks with have the same root cause.

Figure 10: Axis pointers (vertical dotted lines) show associations between peaks across multiple metrics graphs.

Event-to-Metric Associations

SkyWalking provides the event concept to associate possible service performance impacted by the infrastructure, such as new deployment even from k8s. Or, the anomaly had been detected by alerting or integrated AIOps engine.

The event to metrics association is also automatically, it could cover the time range of the event on the metric graphs(blue areas). If the area of event and peaks are matched, most likely this event covered this anomaly.

Figure 11: SkyWalking’s event to metric association view.

SkyWalking Makes it Easier and Faster to Find Root Causes

SkyWalking now makes it easy to find associations between metrics, events, and traces, ultimately making it possible to identify root causes and fix problems fast. The associations we’ve discussed in this article are available out-of-box in the SkyWalking 9.3.0 release.

Figure 12: Just click on the dots to see related traces and metrics associations.

Click the dots on any metric graph, and you will see a View Related Traces item pop-out if this metric has logical mapping traces.

Conclusion

In this blog, we took a look at the newly-added association feature between metrics and traces. With this new visualization, it’s now much easier to find key traces to identify root cause of issues.Associations in SkyWalking can go even deeper. Associations from metrics to traces is not the end of diagnosing system bottleneck. In the next post, we will introduce an eBPF powered trace enhancement where you’ll be able to see HTTP request and response details associated with tracing spans from network profiling. Stay tuned.

Blog: How to Use SkyWalking for Distributed Tracing in Istio?

Wed, 14 Dec 2022 00:00:00 +0000

In cloud native applications, a request often needs to be processed through a series of APIs or backend services, some of which are parallel and some serial and located on different platforms or nodes. How do we determine the service paths and nodes a call goes through to help us troubleshoot the problem? This is where distributed tracing comes into play.

This article covers:

How distributed tracing works
How to choose distributed tracing software
How to use distributed tracing in Istio
How to view distributed tracing data using Bookinfo and SkyWalking as examples

Distributed Tracing Basics

Distributed tracing is a method for tracing requests in a distributed system to help users better understand, control, and optimize distributed systems. There are two concepts used in distributed tracing: TraceID and SpanID. You can see them in Figure 1 below.

TraceID is a globally unique ID that identifies the trace information of a request. All traces of a request belong to the same TraceID, and the TraceID remains constant throughout the trace of the request.
SpanID is a locally unique ID that identifies a request’s trace information at a certain time. A request generates different SpanIDs at different periods, and SpanIDs are used to distinguish trace information for a request at different periods.

TraceID and SpanID are the basis of distributed tracing. They provide a uniform identifier for request tracing in distributed systems and facilitate users’ ability to query, manage, and analyze the trace information of requests.

Figure 1: Trace and span

The following is the process of distributed tracing:

When a system receives a request, the distributed tracing system assigns a TraceID to the request, which is used to chain together the entire chain of invocations.
The distributed trace system generates a SpanID and ParentID for each service call within the system for the request, which is used to record the parent-child relationship of the call; a Span without a ParentID is used as the entry point of the call chain.
TraceID and SpanID are to be passed during each service call.
When viewing a distributed trace, query the full process of a particular request by TraceID.

How Istio Implements Distributed Tracing

Istio’s distributed tracing is based on information collected by the Envoy proxy in the data plane. After a service request is intercepted by Envoy, Envoy adds tracing information as headers to the request forwarded to the destination workload. The following headers are relevant for distributed tracing:

As TraceID: x-request-id
Used to establish parent-child relationships for Span in the LightStep trace: x-ot-span-context</li
Used for Zipkin, also for Jaeger, SkyWalking, see b3-propagation:
- x-b3-traceid
- x-b3-traceid
- x-b3-spanid
- x-b3-parentspanid
- x-b3-sampled
- x-b3-flags
- b3
For Datadog:
- x-datadog-trace-id
- x-datadog-parent-id
- x-datadog-sampling-priority
For SkyWalking: sw8
For AWS X-Ray: x-amzn-trace-id

For more information on how to use these headers, please see the Envoy documentation.

Regardless of the language of your application, Envoy will generate the appropriate tracing headers for you at the Ingress Gateway and forward these headers to the upstream cluster. However, in order to utilize the distributed tracing feature, you must modify your application code to attach the tracing headers to upstream requests. Since neither the service mesh nor the application can automatically propagate these headers, you can integrate the agent for distributed tracing into the application or manually propagate these headers in the application code itself. Once the tracing headers are propagated to all upstream requests, Envoy will send the tracing data to the tracer’s back-end processing, and then you can view the tracing data in the UI.

For example, look at the code of the Productpage service in the Bookinfo application. You can see that it integrates the Jaeger client library and synchronizes the header generated by Envoy with the HTTP requests to the Details and Reviews services in the getForwardHeaders (request) function.

def getForwardHeaders(request):
    headers = {}

    # Using Jaeger agent to get the x-b3-* headers
    span = get_current_span()
    carrier = {}
    tracer.inject(
        span_context=span.context,
        format=Format.HTTP_HEADERS,
        carrier=carrier)

    headers.update(carrier)

    # Dealing with the non x-b3-* header manually
    if 'user' in session:
        headers['end-user'] = session['user']
    incoming_headers = [
        'x-request-id',
        'x-ot-span-context',
        'x-datadog-trace-id',
        'x-datadog-parent-id',
        'x-datadog-sampling-priority',
        'traceparent',
        'tracestate',
        'x-cloud-trace-context',
        'grpc-trace-bin',
        'sw8',
        'user-agent',
        'cookie',
        'authorization',
        'jwt',
    ]

    for ihdr in incoming_headers:
        val = request.headers.get(ihdr)
        if val is not None:
            headers[ihdr] = val

    return headers

For more information, the Istio documentation provides answers to frequently asked questions about distributed tracing in Istio.

How to Choose A Distributed Tracing System

Distributed tracing systems are similar in principle. There are many such systems on the market, such as Apache SkyWalking, Jaeger, Zipkin, Lightstep, Pinpoint, and so on. For our purposes here, we will choose three of them and compare them in several dimensions. Here are our inclusion criteria:

They are currently the most popular open-source distributed tracing systems.
All are based on the OpenTracing specification.
They support integration with Istio and Envoy.

Items	Apache SkyWalking	Jaeger	Zipkin
Implementations	Language-based probes, service mesh probes, eBPF agent, third-party instrumental libraries (Zipkin currently supported)	Language-based probes	Language-based probes
Database	ES, H2, MySQL, TiDB, Sharding-sphere, BanyanDB	ES, MySQL, Cassandra, Memory	ES, MySQL, Cassandra, Memory
Supported Languages	Java, Rust, PHP, NodeJS, Go, Python, C++, .Net, Lua	Java, Go, Python, NodeJS, C#, PHP, Ruby, C++	Java, Go, Python, NodeJS, C#, PHP, Ruby, C++
Initiator	Personal	Uber	Twitter
Governance	Apache Foundation	CNCF	CNCF
Version	9.3.0	1.39.0	2.23.19
Stars	20.9k	16.8k	15.8k

Although Apache SkyWalking’s agent does not support as many languages as Jaeger and Zipkin, SkyWalking’s implementation is richer and compatible with Jaeger and Zipkin trace data, and development is more active, so it is one of the best choices for building a telemetry platform.

Demo

Refer to the Istio documentation to install and configure Apache SkyWalking.

Environment Description

The following is the environment for our demo:

Kubernetes 1.24.5
Istio 1.16
SkyWalking 9.1.0

Install Istio

Before installing Istio, you can check the environment for any problems:

$ istioctl experimental precheck
✔ No issues found when checking the cluster. Istio is safe to install or upgrade!
  To get started, check out https://istio.io/latest/docs/setup/getting-started/

Then install Istio and configure the destination for sending tracing messages as SkyWalking:

# Initial Istio Operator
istioctl operator init
# Configure tracing destination
kubectl apply -f - <<EOF
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  namespace: istio-system
  name: istio-with-skywalking
spec:
  meshConfig:
    defaultProviders:
      tracing:
      - "skywalking"
    enableTracing: true
    extensionProviders:
    - name: "skywalking"
      skywalking:
        service: tracing.istio-system.svc.cluster.local
        port: 11800
EOF

Deploy Apache SkyWalking

Istio 1.16 supports distributed tracing using Apache SkyWalking. Install SkyWalking by executing the following code:

kubectl apply -f 
https://raw.githubusercontent.com/istio/istio/release-1.16/samples/addons/extras/skywalking.yaml

It will install the following components under the istio-system namespace:

SkyWalking Observability Analysis Platform (OAP): Used to receive trace data, supports SkyWalking native data formats, Zipkin v1 and v2 and Jaeger format.
UI: Used to query distributed trace data.

For more information about SkyWalking, please refer to the SkyWalking documentation.

Deploy the Bookinfo Application

Execute the following command to install the bookinfo application:

kubectl label namespace default istio-injection=enabled
kubectl apply -f samples/bookinfo/platform/kube/bookinfo.yaml
kubectl apply -f samples/bookinfo/networking/bookinfo-gateway.yaml

Launch the SkyWalking UI:

istioctl dashboard skywalking

Figure 2 shows all the services available in the bookinfo application:

Figure 2: SkyWalking General Service page

You can also see information about instances, endpoints, topology, tracing, etc. For example, Figure 3 shows the service topology of the bookinfo application:

Figure 3: Topology diagram of the Bookinfo application

Tracing views in SkyWalking can be displayed in a variety of formats, including list, tree, table, and statistics. See Figure 4:

Figure 4: SkyWalking General Service trace supports multiple display formats

To facilitate our examination, set the sampling rate of the trace to 100%:

kubectl apply -f - <<EOF
apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: mesh-default
  namespace: istio-system
spec:
  tracing:
  - randomSamplingPercentage: 100.00
EOF

Important: It’s generally not good practice to set the sampling rate to 100% in a production environment. To avoid the overhead of generating too many trace logs in production, please adjust the sampling strategy (sampling percentage).

Uninstall

After experimenting, uninstall Istio and SkyWalking by executing the following command.

samples/bookinfo/platform/kube/cleanup.sh
istioctl unintall --purge
kubectl delete namespace istio-system

Understanding the Bookinfo Tracing Information

Navigate to the General Service tab in the Apache SkyWalking UI, and you can see the trace information for the most recent istio-ingressgateway service, as shown in Figure 5. Click on each span to see the details.

Figure 5: The table view shows the basic information about each span.

Switching to the list view, you can see the execution order and duration of each span, as shown in Figure 6:

Figure 6: List display

You might want to know why such a straightforward application generates so much span data. Because after we inject the Envoy proxy into the pod, every request between services will be intercepted and processed by Envoy, as shown in Figure 7:

Figure 7: Envoy intercepts requests to generate a span

The tracing process is shown in Figure 8:

Figure 8: Trace of the Bookinfo application

We give each span a label with a serial number, and the time taken is indicated in parentheses. For illustration purposes, we have summarized all spans in the table below.

No.	Endpoint	Total Duration (ms)	Component Duration (ms)	Current Service	Description
1	/productpage	190	0	istio-ingressgateway	Envoy Outbound
2	/productpage	190	1	istio-ingressgateway	Ingress -> Productpage network transmission
3	/productpage	189	1	productpage	Envoy Inbound
4	/productpage	188	21	productpage	Application internal processing
5	/details/0	8	1	productpage	Envoy Outbound
6	/details/0	7	3	productpage	Productpage -> Details network transmission
7	/details/0	4	0	details	Envoy Inbound
8	/details/0	4	4	details	Application internal processing
9	/reviews/0	159	0	productpage	Envoy Outbound
10	/reviews/0	159	14	productpage	Productpage -> Reviews network transmission
11	/reviews/0	145	1	reviews	Envoy Inbound
12	/reviews/0	144	109	reviews	Application internal processing
13	/ratings/0	35	2	reviews	Envoy Outbound
14	/ratings/0	33	16	reviews	Reviews -> Ratings network transmission
15	/ratings/0	17	1	ratings	Envoy Inbound
16	/ratings/0	16	16	ratings	Application internal processing

From the above information, it can be seen that:

The total time consumed for this request is 190 ms.
In Istio sidecar mode, each traffic flow in and out of the application container must pass through the Envoy proxy once, each time taking 0 to 2 ms.
Network requests between Pods take between 1 and 16ms.
This is because the data itself has errors and the start time of the Span is not necessarily equal to the end time of the parent Span.
We can see that the most time-consuming part is the Reviews application, which takes 109 ms so that we can optimize it for that application.

Summary

Distributed tracing is an indispensable tool for analyzing performance and troubleshooting modern distributed applications. In this tutorial, we’ve seen how, with just a few minor changes to your application code to propagate tracing headers, Istio makes distributed tracing simple to use. We’ve also reviewed Apache SkyWalking as one of the best distributed tracing systems that Istio supports. It is a fully functional platform for cloud native application analytics, with features such as metrics and log collection, alerting, Kubernetes monitoring, service mesh performance diagnosis using eBPF, and more.

If you’re new to service mesh and Kubernetes security, we have a bunch of free online courses available at Tetrate Academy that will quickly get you up to speed with Istio and Envoy.

If you’re looking for a fast way to get to production with Istio, check out Tetrate Istio Distribution (TID). TID is Tetrate’s hardened, fully upstream Istio distribution, with FIPS-verified builds and support available. It’s a great way to get started with Istio knowing you have a trusted distribution to begin with, have an expert team supporting you, and also have the option to get to FIPS compliance quickly if you need to.

Once you have Istio up and running, you will probably need simpler ways to manage and secure your services beyond what’s available in Istio, that’s where Tetrate Service Bridge comes in. You can learn more about how Tetrate Service Bridge makes service mesh more secure, manageable, and resilient here, or contact us for a quick demo.

Blog: How to run Apache SkyWalking on AWS EKS and RDS/Aurora

Tue, 13 Dec 2022 00:00:00 +0000

Introduction

Apache SkyWalking is an open source APM tool for monitoring and troubleshooting distributed systems, especially designed for microservices, cloud native and container-based (Docker, Kubernetes, Mesos) architectures. It provides distributed tracing, service mesh observability, metric aggregation and visualization, and alarm.

In this article, I will introduce how to quickly set up Apache SkyWalking on AWS EKS and RDS/Aurora, as well as a couple of sample services, monitoring services to observe SkyWalking itself.

Prerequisites

AWS account
AWS CLI
Terraform
kubectl

We can use the AWS web console or CLI to create all resources needed in this tutorial, but it can be too tedious and hard to debug when something goes wrong. So in this artical I will use Terraform to create all AWS resources, deploy SkyWalking, sample services, and load generator services (Locust).

Architecture

The demo architecture is as follows:

graph LR
    subgraph AWS
        subgraph EKS
          subgraph istio-system namespace
              direction TB
              OAP[[SkyWalking OAP]]
              UI[[SkyWalking UI]]
            Istio[[istiod]]
          end
          subgraph sample namespace
              Service0[[Service0]]
              Service1[[Service1]]
              ServiceN[[Service ...]]
          end
          subgraph locust namespace
              LocustMaster[[Locust Master]]
              LocustWorkers0[[Locust Worker 0]]
              LocustWorkers1[[Locust Worker 1]]
              LocustWorkersN[[Locust Worker ...]]
          end
        end
        RDS[[RDS/Aurora]]
    end
    OAP --> RDS
    Service0 -. telemetry data -.-> OAP
    Service1 -. telemetry data -.-> OAP
    ServiceN -. telemetry data -.-> OAP
    UI --query--> OAP
    LocustWorkers0 -- traffic --> Service0
    LocustWorkers1 -- traffic --> Service0
    LocustWorkersN -- traffic --> Service0
    Service0 --> Service1 --> ServiceN
    LocustMaster --> LocustWorkers0
    LocustMaster --> LocustWorkers1
    LocustMaster --> LocustWorkersN
    User --> LocustMaster

As shown in the architecture diagram, we need to create the following AWS resources:

EKS cluster
RDS instance or Aurora cluster

Sounds simple, but there are a lot of things behind the scenes, such as VPC, subnets, security groups, etc. You have to configure them correctly to make sure the EKS cluster can connect to RDS instance/Aurora cluster otherwise the SkyWalking won’t work. Luckily, Terraform can help us to create and destroy all these resources automatically.

I have created a Terraform module to create all AWS resources needed in this tutorial, you can find it in the GitHub repository.

Create AWS resources

First, we need to clone the GitHub repository and cd into the folder:

git clone https://github.com/kezhenxu94/oap-load-test.git

Then, we need to create a file named terraform.tfvars to specify the AWS region and other variables:

cat > terraform.tfvars <<EOF
aws_access_key = ""
aws_secret_key = ""
cluster_name   = "skywalking-on-aws"
region         = "ap-east-1"
db_type        = "rds-postgresql"
EOF

If you have already configured the AWS CLI, you can skip the aws_access_key and aws_secret_key variables. To install SkyWalking with RDS postgresql, set the db_type to rds-postgresql, to install SkyWalking with Aurora postgresql, set the db_type to aurora-postgresql.

There are a lot of other variables you can configure, such as tags, sample services count, replicas, etc., you can find them in the variables.tf.

Then, we can run the following commands to initialize the Terraform module and download the required providers, then create all AWS resources:

terraform init
terraform apply -var-file=terraform.tfvars

Type yes to confirm the creation of all AWS resources, or add the -auto-approve flag to the terraform apply to skip the confirmation:

terraform apply -var-file=terraform.tfvars -auto-approve

Now what you need to do is to wait for the creation of all AWS resources to complete, it may take a few minutes. You can check the progress of the creation in the AWS web console, and check the deployment progress of the services inside the EKS cluster.

Generate traffic

Besides creating necessary AWS resources, the Terraform module also deploys SkyWalking, sample services, and Locust load generator services to the EKS cluster.

You can access the Locust web UI to generate traffic to the sample services:

open http://$(kubectl get svc -n locust -l app=locust-master -o jsonpath='{.items[0].status.loadBalancer.ingress[0].hostname}'):8089

The command opens the browser to the Locust web UI, you can configure the number of users and hatch rate to generate traffic.

Observe SkyWalking

You can access the SkyWalking web UI to observe the sample services.

First you need to forward the SkyWalking UI port to local

kubectl -n istio-system port-forward $(kubectl -n istio-system get pod -l app=skywalking -l component=ui -o name) 8080:8080

And then open the browser to http://localhost:8080 to access the SkyWalking web UI.

Observe RDS/Aurora

You can also access the RDS/Aurora web console to observe the performance of RDS/Aurora instance/Aurora cluste.

Test Results

Test 1: SkyWalking with EKS and RDS PostgreSQL

Service Traffic

RDS Performance

SkyWalking Performance

Test 2: SkyWalking with EKS and Aurora PostgreSQL

Service Traffic

RDS Performance

SkyWalking Performance

Clean up

When you are done with the demo, you can run the following command to destroy all AWS resources:

terraform destroy -var-file=terraform.tfvars -auto-approve

Blog: SkyWalking's New Storage Feature Based on ShardingSphere-Proxy: MySQL-Sharding

Tue, 06 Dec 2022 00:00:00 +0000

As an application performance monitoring tool for distributed systems, Apache SkyWalking observes metrics, logs, traces, and events in the service mesh.

SkyWalking OAP’s dataflow processing architecture boasts high performance and is capable of dealing with massive data traffic in real-time. However, storing, updating, and querying massive amounts of data poses a great challenge to its backend storage system.

By default, SkyWalking provides storage methods including H2, OpenSearch, ElasticSearch, MySQL, TiDB, PostgreSQL, and BanyanDB. Among them, MySQL storage is suited to a single machine and table (MySQL cluster capability depends on your technology selection). Nevertheless, in the context of high-traffic business systems, the storage of monitoring data is put under great pressure and query performance is lowered.

Based on MySQL storage, SkyWalking v9.3.0 provides a new storage method: MySQL-Sharding. It supports database and table sharding features thanks to ShardingSphere-Proxy, which is a mature solution for dealing with relational databases’ massive amounts of data.

1. Architecture Deployment

SkyWalking will only interact with ShardingSphere-Proxy instead of directly connecting to the database.
The connection exposed by each MySQL node is a data source managed by ShardingSphere-Proxy.
ShardingSphere-Proxy will establish a virtual logical database based on the configuration and then carry out database and table sharding and routing according to the OAP provided data sharding rules.
SkyWalking OAP creates data sharding rules and performs DDL and DML on a virtual logical database just like it does with MySQL.

2. Application Scenario

Applicable to scenarios where MySQL is used for storage, but the single-table mode cannot meet the performance requirements created by business growth.

3. How Does Data Sharding Work with SkyWalking?

Data sharding defines the data Model in SkyWalking with the annotation @SQLDatabase.Sharding.

@interface Sharding {
  ShardingAlgorithm shardingAlgorithm();
  String dataSourceShardingColumn() default "";
  String tableShardingColumn() default ""; 
}

Note:

shardingAlgorithm: Table sharding algorithm dataSourceShardingColumn: Database sharding key tableShardingColumn: Table sharding key

SkyWalking selects database sharding key, table sharding key and table sharding algorithm based on @SQLDatabase.Sharding, in order to dynamically generate sharding rules for each table. Next, it performs rule definition by operating ShardingSphere-Proxy via DistSQL. ShardingSphere-Proxy carries out data sharding based on the rule definition.

3.1 Database Sharding Method

SkyWalking adopts a unified method to carry out database sharding. The number of databases that need to be sharded requires modulo by the hash value of the database sharding key, which should be the numeric suffix of the routing target database. Therefore, the routing target database is:

ds_{dataSourceShardingColumn.hashcode() % dataSourceList.size()}

For example, we now have dataSourceList = ds_0…ds_n. If {dataSourceShardingColumn.hashcode() % dataSourceList.size() = 2}, all the data will be routed to the data source node ds_2.

3.2 Table Sharding Method

The table sharding algorithm mainly shards according to the data owing to the TTL mechanism. According to TTL, there will be one sharding table per day:

{tableName = logicTableName_timeSeries (data)}

To ensure that data within the TTL can be written and queried, the time series will generate the current date:

{timeSeries = currentDate - TTL +1...currentDate + 1}

For example, if TTL=3 and currentDate=20220907, sharding tables will be: logicTableName_20220905 logicTableName_20220906 logicTableName_20220907 logicTableName_20220908

SkyWalking provides table sharding algorithms for different data models:

Algorithm Name	Sharding Description	Time Precision Requirements for Sharding Key	Typical Application Data Model
NO_SHARDING	No table sharding and single-table mode is maintained.	N/A	Data model with a small amount of data and no need for sharding.
TIME_RELATIVE_ID_SHARDING_ALGORITHM	Shard by day using time_bucket in the ID column.	time_bucket can be accurate to seconds, minutes, hours, or days in the same table.	Various metrics.
TIME_SEC_RANGE_SHARDING_ALGORITHM	Shard by day using time_bucket column.	time_bucket must be accurate to seconds.	SegmentRecordLogRecord, etc.
TIME_MIN_RANGE_SHARDING_ALGORITHM	Shard by day using time_bucket column.	time_bucket must be accurate to minutes.	EndpointTraffic
TIME_BUCKET_SHARDING_ALGORITHM	Shard by day using time_bucket column.	time_bucket can be accurate to seconds, minutes, hours, and days in the same table.	Service, Instance, Endpoint and other call relations such as ServiceRelationServerSideMetrics

4. TTL Mechanism

For sharding tables, delete the physical table deadline >= timeSeries according to TTL.
```
{deadline = new DateTime().plusDays(-ttl)}
```
TTL timer will delete the expired tables according to the current date while updating sharding rules according to the new date and informing ShardingSphere-Proxy to create new sharding tables.
For a single table, use the previous method and delete the row record of deadline >=time_bucket.

5. Examples of Sharding Data Storage

Next, we’ll take segment (Record type) and service_resp_time (Metrics type) as examples to illustrate the data storage logic and physical distribution. Here, imagine MySQL has two nodes ds_0 and ds_1.

Note:

The following storage table structure is just a simplified version as an example, and does not represent the real SkyWalking table structure.

5.1 segment

The sharding configuration is as follows:

@SQLDatabase.Sharding(shardingAlgorithm = ShardingAlgorithm.TIME_SEC_RANGE_SHARDING_ALGORITHM, dataSourceShardingColumn = service_id, tableShardingColumn = time_bucket)

The logical database, table structures and actual ones are as follows:

5.2 service_resp_time

The sharding configuration is as follows:

@SQLDatabase.Sharding(shardingAlgorithm = ShardingAlgorithm.TIME_RELATIVE_ID_SHARDING_ALGORITHM, tableShardingColumn = id, dataSourceShardingColumn = entity_id)

The logical database and table structures and actual ones are as follows:

6. How to Use ShardingSphere-Proxy?

6.1 Manual Deployment

Here we take the deployment of a single-node SkyWalking OAP and ShardingSphere-Proxy 5.1.2 as an example. Please refer to the relevant documentation for the cluster deployment.

Prepare the MySQL cluster.
Deploy, install and configure ShardingSphere-Proxy:

conf/server.yaml and props.proxy-hint-enabled must be true. Refer to the link for the complete configuration.
conf/config-sharding.yaml configures logical database and dataSources list. The dataSource name must be prefixed with ds_ and start with ds_0. For details about the configuration, please refer to this page.

Deploy, install and configure SkyWalking OAP:

Set up OAP environment variables: ${SW_STORAGE:mysql-sharding}，
Configure the connection information based on the actual deployment: ${SW_JDBC_URL} ${SW_DATA_SOURCE_USER} ${SW_DATA_SOURCE_PASSWORD}

Note:

Connection information must correspond to ShardingSphere-Proxy virtual database.

Configure the data source name configured by conf/config-sharding.yaml in ShardingSphere-Proxy to ${SW_JDBC_SHARDING_DATA_SOURCES} and separate names with commas.

Start the MySQL cluster.
Start ShardingSphere-Proxy.
Start SkyWalking OAP.

6.2 Running Demo with Docker

Our GitHub repository provides a complete and operational demo based on Docker, allowing you to quickly grasp the operation’s effectiveness. The deployment includes the following:

One OAP service. The TTL of Metrics and Record data set to 2 days.
One sharding-proxy service with version 5.1.2. Its external port is 13307 and the logical database name is swtest.
Two MySQL services. Their external ports are 3306 and 3307 respectively and they are configured as ds_0 and ds_1 in sharding-proxy’s conf/config-sharding.yaml.
One provider service (simulated business programs used to verify trace and metrics and other data). Its external port is 9090.
One consumer service (simulated business programs used to verify trace and metrics and other data). Its external port is 9092.

Download the demo program locally and run it directly in the directory skywalking-mysql-sharding-demo.

docker-compose up -d

Note:

The first startup may take some time to pull images and create all the tables.

Once all the services are started, database tools can be used to check the creation of sharding-proxy logical tables and the actual physical sharding table in the two MySQL databases. Additionally, you can also connect the sharding-proxy logical database to view the data query routing. For example:

PREVIEW
SELECT * FROM SEGMENT

The result is as follows:

The simulated business program provided by the demo can simulate business requests by requesting the consumer service to verify various types of data distribution:

curl http://127.0.0.1:9092/info

7. Conclusion

In this blog, we introduced SkyWalking’s new storage feature, MySQL sharding, which leverage ShardingSphere-Proxy and covered details of its deployment architecture, application scenarios, sharding logic, and TTL mechanism. We’ve also provided sample data and deployment steps to help get started.

SkyWalking offers a variety of storage options to fit many use cases. If you need a solution to store large volumes of telemetry data in a relational database, the new MySQL sharding feature is worth a look. For more information on the SkyWalking 9.3.0 release and where to get it, check out the release notes.

Blog: Diagnose Service Mesh Network Performance with eBPF

Tue, 27 Sep 2022 00:00:00 +0000

Background

This article will show how to use Apache SkyWalking with eBPF to make network troubleshooting easier in a service mesh environment.

Apache SkyWalking is an application performance monitor tool for distributed systems. It observes metrics, logs, traces, and events in the service mesh environment and uses that data to generate a dependency graph of your pods and services. This dependency graph can provide quick insights into your system, especially when there’s an issue.

However, when troubleshooting network issues in SkyWalking’s service topology, it is not always easy to pinpoint where the error actually is. There are two reasons for the difficulty:

Traffic through the Envoy sidecar is not easy to observe. Data from Envoy’s Access Log Service (ALS) shows traffic between services (sidecar-to-sidecar), but not metrics on communication between the Envoy sidecar and the service it proxies. Without that information, it is more difficult to understand the impact of the sidecar.
There is a lack of data from transport layer (OSI Layer 4) communication. Since services generally use application layer (OSI Layer 7) protocols such as HTTP, observability data is generally restricted to application layer communication. However, the root cause may actually be in the transport layer, which is typically opaque to observability tools.

Access to metrics from Envoy-to-service and transport layer communication can make it easier to diagnose service issues. To this end, SkyWalking needs to collect and analyze transport layer metrics between processes inside Kubernetes pods - a task well suited to eBPF. We investigated using eBPF for this purpose and present our results and a demo below.

Monitoring Kubernetes Networks with eBPF

With its origins as the Extended Berkeley Packet Filter, eBPF is a general purpose mechanism for injecting and running your own code into the Linux kernel and is an excellent tool for monitoring network traffic in Kubernetes Pods. In the next few sections, we'll provide an overview of how to use eBPF for network monitoring as background for introducing Skywalking Rover, a metrics collector and profiler powered by eBPF to diagnose CPU and network performance.

How Applications and the Network Interact

Interactions between the application and the network can generally be divided into the following steps from higher to lower levels of abstraction:

User Code: Application code uses high-level network libraries in the application stack to exchange data across the network, like sending and receiving HTTP requests.
Network Library: When the network library receives a network request, it interacts with the language API to send the network data.
Language API: Each language provides an API for operating the network, system, etc. When a request is received, it interacts with the system API. In Linux, this API is called syscalls.
Linux API: When the Linux kernel receives the request through the API, it communicates with the socket to send the data, which is usually closer to an OSI Layer 4 protocol, such as TCP, UDP, etc.
Socket Ops: Sending or receiving the data to/from the NIC.

Our hypothesis is that eBPF can monitor the network. There are two ways to implement the interception: User space (uprobe) or Kernel space (kprobe). The table below summarizes the differences.

	Pros	Cons
uprobe	• Get more application-related contexts, such as whether the current request is HTTP or HTTPS. • Requests and responses can be intercepted by a single method	• Data structures can be unstable, so it is more difficult to get the desired data. • Implementation may differ between language/library versions. • Does not work in applications without symbol tables.
kprobe	• Available for all languages. • The data structure and methods are stable and do not require much adaptation. • Easier correlation with underlying data, such as getting the destination address of TCP, OSI Layer 4 protocol metrics, etc.	• A single request and response may be split into multiple probes. • Contextual information is not easy to get for stateful requests. For example header compression in HTTP/2.

For the general network performance monitor, we chose to use the kprobe (intercept the syscalls) for the following reasons:

It’s available for applications written in any programming language, and it’s stable, so it saves a lot of development/adaptation costs.
It can be correlated with metrics from the system level, which makes it easier to troubleshoot.
As a single request and response are split into multiple probes, we can use technology to correlate them.
For contextual information, It’s usually used in OSI Layer 7 protocol network analysis. So, if we just monitor the network performance, then they can be ignored.

Kprobes and network monitoring

Following the network syscalls of Linux documentation, we can implement network monitoring by intercepting two types of methods: socket operations and send/receive methods.

Socket Operations

When accepting or connecting with another socket, we can get the following information:

Connection information: Includes the remote address from the connection which helps us to understand which pod is connected.
Connection statics: Includes basic metrics from sockets, such as round-trip time (RTT), lost packet count in TCP, etc.
Socket and file descriptor (FD) mapping: Includes the relationship between the Linux file descriptor and socket object. It is useful when sending and receiving data through a Linux file descriptor.

Send/Receive

The interface related to sending or receiving data is the focus of performance analysis. It mainly contains the following parameters:

Socket file descriptor: The file descriptor of the current operation corresponding to the socket.
Buffer: The data sent or received, passed as a byte array.

Based on the above parameters, we can analyze the following data:

Bytes: The size of the packet in bytes.
Protocol: The protocol analysis according to the buffer data, such as HTTP, MySQL, etc.
Execution Time: The time it takes to send/receive the data.

At this point (Figure 1) we can analyze the following steps for the whole lifecycle of the connection:

Connect/Accept: When the connection is created.
Transform: Sending and receiving data on the connection.
Close: When the connection is closed.

Figure 1

Protocol and TLS

The previous section described how to analyze connections using send or receive buffer data. For example, following the HTTP/1.1 message specification to analyze the connection. However, this does not work for TLS requests/responses.

Figure 2

When TLS is in use, the Linux Kernel transmits data encrypted in user space. In the figure above, The application usually transmits SSL data through a third-party library (such as OpenSSL). For this case, the Linux API can only get the encrypted data, so it cannot recognize any higher layer protocol. To decrypt inside eBPF, we need to follow these steps:

Read unencrypted data through uprobe: Compatible multiple languages, using uprobe to capture the data that is not encrypted before sending or after receiving. In this way, we can get the original data and associate it with the socket.
Associate with socket: We can associate unencrypted data with the socket.

OpenSSL Use case

For example, the most common way to send/receive SSL data is to use OpenSSL as a shared library, specifically the SSL_read and SSL_write methods to submit the buffer data with the socket.

Following the documentation, we can intercept these two methods, which are almost identical to the API in Linux. The source code of the SSL structure in OpenSSL shows that the Socket FD exists in the BIO object of the SSL structure, and we can get it by the offset.

In summary, with knowledge of how OpenSSL works, we can read unencrypted data in an eBPF function.

Introducing SkyWalking Rover, an eBPF-based Metrics Collector and Profiler

SkyWalking Rover introduces the eBPF network profiling feature into the SkyWalking ecosystem. It’s currently supported in a Kubernetes environment, so must be deployed inside a Kubernetes cluster. Once the deployment is complete, SkyWalking Rover can monitor the network for all processes inside a given Pod. Based on the monitoring data, SkyWalking can generate the topology relationship diagram and metrics between processes.

Topology Diagram

The topology diagram can help us understand the network access between processes inside the same Pod, and between the process and external environment (other Pod or service). Additionally, it can identify the data direction of traffic based on the line flow direction.

In Figure 3 below, all nodes within the hexagon are the internal process of a Pod, and nodes outside the hexagon are externally associated services or Pods. Nodes are connected by lines, which indicate the direction of requests or responses between nodes (client or server). The protocol is indicated on the line, and it’s either HTTP(S), TCP, or TCP(TLS). Also, we can see in this figure that the line between Envoy and Python applications is bidirectional because Envoy intercepts all application traffic.

Figure 3

Metrics

Once we recognize the network call relationship between processes through the topology, we can select a specific line and view the TCP metrics between the two processes.

The diagram below (Figure 4) shows the metrics of network monitoring between two processes. There are four metrics in each line. Two on the left side are on the client side, and two on the right side are on the server side. If the remote process is not in the same Pod, only one side of the metrics is displayed.

Figure 4

The following two metric types are available:

Counter: Records the total number of data in a certain period. Each counter contains the following data: a. Count: Execution count. b. Bytes: Packet size in bytes. c. Execution time: Execution duration.
Histogram: Records the distribution of data in the buckets.

Based on the above data types, the following metrics are exposed:

Name	Type	Unit	Description
Write	Counter and histogram	Millisecond	The socket write counter.
Read	Counter and histogram	Millisecond	The socket read counter.
Write RTT	Counter and histogram	Microsecond	The socket write round trip time (RTT) counter.
Connect	Counter and histogram	Millisecond	The socket connect/accept with another server/client counter.
Close	Counter and histogram	Millisecond	The socket with other socket counter.
Retransmit	Counter	Millisecond	The socket retransmit package counter.
Drop	Counter	Millisecond	The socket drop package counter.

Demo

In this section, we demonstrate how to perform network profiling in the service mesh. To follow along, you will need a running Kubernetes environment.

NOTE: All commands and scripts are available in this GitHub repository.

Install Istio

Istio is the most widely deployed service mesh, and comes with a complete demo application that we can use for testing. To install Istio and the demo application, follow these steps:

Install Istio using the demo configuration profile.
Label the default namespace, so Istio automatically injects Envoy sidecar proxies when we’ll deploy the application.
Deploy the bookinfo application to the cluster.
Deploy the traffic generator to generate some traffic to the application.

export ISTIO_VERSION=1.13.1

# install istio
istioctl install -y --set profile=demo
kubectl label namespace default istio-injection=enabled

# deploy the bookinfo applications
kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/platform/kube/bookinfo.yaml
kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/networking/bookinfo-gateway.yaml
kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/networking/destination-rule-all.yaml
kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/networking/virtual-service-all-v1.yaml

# generate traffic
kubectl apply -f https://raw.githubusercontent.com/mrproliu/skywalking-network-profiling-demo/main/resources/traffic-generator.yaml

Install SkyWalking

The following will install the storage, backend, and UI needed for SkyWalking:

git clone https://github.com/apache/skywalking-kubernetes.git
cd skywalking-kubernetes
cd chart
helm dep up skywalking
helm -n istio-system install skywalking skywalking \
 --set fullnameOverride=skywalking \
 --set elasticsearch.minimumMasterNodes=1 \
 --set elasticsearch.imageTag=7.5.1 \
 --set oap.replicas=1 \
 --set ui.image.repository=apache/skywalking-ui \
 --set ui.image.tag=9.2.0 \
 --set oap.image.tag=9.2.0 \
 --set oap.envoy.als.enabled=true \
 --set oap.image.repository=apache/skywalking-oap-server \
 --set oap.storageType=elasticsearch \
 --set oap.env.SW_METER_ANALYZER_ACTIVE_FILES='network-profiling'

Install SkyWalking Rover

SkyWalking Rover is deployed on every node in Kubernetes, and it automatically detects the services in the Kubernetes cluster. The network profiling feature has been released in the version 0.3.0 of SkyWalking Rover. When a network monitoring task is created, the SkyWalking rover sends the data to the SkyWalking backend.

kubectl apply -f https://raw.githubusercontent.com/mrproliu/skywalking-network-profiling-demo/main/resources/skywalking-rover.yaml

Start the Network Profiling Task

Once all deployments are completed, we must create a network profiling task for a specific instance of the service in the SkyWalking UI.

To open SkyWalking UI, run:

kubectl port-forward svc/skywalking-ui 8080:80 --namespace
istio-system

Currently, we can select the specific instances that we wish to monitor by clicking the Data Plane item in the Service Mesh panel and the Service item in the Kubernetes panel.

In the figure below, we have selected an instance with a list of tasks in the network profiling tab. When we click the start button, the SkyWalking Rover starts monitoring this instance’s network.

Figure 5

Done!

After a few seconds, you will see the process topology appear on the right side of the page.

Figure 6

When you click on the line between processes, you can see the TCP metrics between the two processes.

Figure 7

Conclusion

In this article, we detailed a problem that makes troubleshooting service mesh architectures difficult: lack of context between layers in the network stack. These are the cases when eBPF begins to really help with debugging/productivity when existing service mesh/envoy cannot. Then, we researched how eBPF could be applied to common communication, such as TLS. Finally, we demo the implementation of this process with SkyWalking Rover.

For now, we have completed the performance analysis for OSI layer 4 (mostly TCP). In the future, we will also introduce the analysis for OSI layer 7 protocols like HTTP.

Get Started with Istio

To get started with service mesh today, Tetrate Istio Distro is the easiest way to install, manage, and upgrade Istio. It provides a vetted upstream distribution of Istio that’s tested and optimized for specific platforms by Tetrate plus a CLI that facilitates acquiring, installing, and configuring multiple Istio versions. Tetrate Istio Distro also offers FIPS certified Istio builds for FedRAMP environments.

For enterprises that need a unified and consistent way to secure and manage services and traditional workloads across complex, heterogeneous deployment environments, we offer Tetrate Service Bridge, our flagship edge-to-workload application connectivity platform built on Istio and Envoy.

Additional Resources

Blog: [Video] Distributed tracing demo using Apache SkyWalking and Kong API Gateway

Thu, 11 Aug 2022 00:00:00 +0000

Observability essential when working with distributed systems. Built on 3 pillars of metrics, logging and tracing, having the right tools in place to quickly identify and determine the root cause of an issue in production is imperative. In this Kongcast interview, we explore the benefits of having observability and demo the use of Apache SkyWalking. We walk through the capabilities that SkyWalking offers out of the box and debug a common HTTP 500 error using the tool.

Andrew Kew is interviewed by Viktor Gamov, a developer advocate at Kong Inc

Andrew is a highly passionate technologist with over 16 valuable years experience in building server side and cloud applications. Having spent the majority of his time in the Financial Services domain, his meritocratic rise to CTO of an Algorithmic Trading firm allowed him to not only steer the business from a technology standpoint, but build robust and scalable trading algorithms. His mantra is “right first time”, thus ensuring the projects or clients he is involved in are left in a better place than they were before he arrived.

He is the founder of a boutique software consultancy in the United Kingdom, QuadCorps Ltd, working in the API and Integration Ecosystem space and is currently on a residency programme at Kong Inc as a senior field engineer and technical account manager working across many of their enterprise strategic accounts.

Blog: Pinpoint Service Mesh Critical Performance Impact by using eBPF

Tue, 05 Jul 2022 00:00:00 +0000

Content

Background

Apache SkyWalking observes metrics, logs, traces, and events for services deployed into the service mesh. When troubleshooting, SkyWalking error analysis can be an invaluable tool helping to pinpoint where an error occurred. However, performance problems are more difficult: It’s often impossible to locate the root cause of performance problems with pre-existing observation data. To move beyond the status quo, dynamic debugging and troubleshooting are essential service performance tools. In this article, we’ll discuss how to use eBPF technology to improve the profiling feature in SkyWalking and analyze the performance impact in the service mesh.

Trace Profiling in SkyWalking

Since SkyWalking 7.0.0, Trace Profiling has helped developers find performance problems by periodically sampling the thread stack to let developers know which lines of code take more time. However, Trace Profiling is not suitable for the following scenarios:

Thread Model: Trace Profiling is most useful for profiling code that executes in a single thread. It is less useful for middleware that relies heavily on async execution models. For example Goroutines in Go or Kotlin Coroutines.
Language: Currently, Trace Profiling is only supported in Java and Python, since it’s not easy to obtain the thread stack in the runtimes of some languages such as Go and Node.js.
Agent Binding: Trace Profiling requires Agent installation, which can be tricky depending on the language (e.g., PHP has to rely on its C kernel; Rust and C/C++ require manual instrumentation to make install).
Trace Correlation: Since Trace Profiling is only associated with a single request it can be hard to determine which request is causing the problem.
Short Lifecycle Services: Trace Profiling doesn’t support short-lived services for (at least) two reasons:
1. It’s hard to differentiate system performance from class code manipulation in the booting stage.
2. Trace profiling is linked to an endpoint to identify performance impact, but there is no endpoint to match these short-lived services.

Fortunately, there are techniques that can go further than Trace Profiling in these situations.

Introduce eBPF

We have found that eBPF — a technology that can run sandboxed programs in an operating system kernel and thus safely and efficiently extend the capabilities of the kernel without requiring kernel modifications or loading kernel modules — can help us fill gaps left by Trace Profiling. eBPF is a trending technology because it breaks the traditional barrier between user and kernel space. Programs can now inject bytecode that runs in the kernel, instead of having to recompile the kernel to customize it. This is naturally a good fit for observability.

In the figure below, we can see that when the system executes the execve syscalls, the eBPF program is triggered, and the current process runtime information is obtained by using function calls.

Using eBPF technology, we can expand the scope of Skywalking’s profiling capabilities:

Global Performance Analysis: Before eBPF, data collection was limited to what agents can observe. Since eBPF programs run in the kernel, they can observe all threads. This is especially useful when you are not sure whether a performance problem is caused by a particular request.
Data Content: eBPF can dump both user and kernel space thread stacks, so if a performance issue happens in kernel space, it’s easier to find.
Agent Binding: All modern Linux kernels support eBPF, so there is no need to install anything. This means it is an orchestration-free vs an agent model. This reduces friction caused by built-in software which may not have the correct agents installed, such as Envoy in a Service Mesh.
Sampling Type: Unlike Trace Profiling, eBPF is event-driven and, therefore, not constrained by interval polling. For example, eBPF can trigger events and collect more data depending on a transfer size threshold. This can allow the system to triage and prioritize data collection under extreme load.

eBPF Limitations

While eBPF offers significant advantages for hunting performance bottlenecks, no technology is perfect. eBPF has a number of limitations described below. Fortunately, since SkyWalking does not require eBPF, the impact is limited.

Linux Version Requirement: eBPF programs require a Linux kernel version above 4.4, with later kernel versions offering more data to be collected. The BCC has documented the features supported by different Linux kernel versions, with the differences between versions usually being what data can be collected with eBPF.
Privileges Required: All processes that intend to load eBPF programs into the Linux kernel must be running in privileged mode. As such, bugs or other issues in such code may have a big impact.
Weak Support for Dynamic Language: eBPF has weak support for JIT-based dynamic languages, such as Java. It also depends on what data you want to collect. For Profiling, eBPF does not support parsing the symbols of the program, which is why most eBPF-based profiling technologies only support static languages like C, C++, Go, and Rust. However, symbol mapping can sometimes be solved through tools provided by the language. For example, in Java, perf-map-agent can be used to generate the symbol mapping. However, dynamic languages don’t support the attach (uprobe) functionality that would allow us to trace execution events through symbols.

Introducing SkyWalking Rover

SkyWalking Rover introduces the eBPF profiling feature into the SkyWalking ecosystem. The figure below shows the overall architecture of SkyWalking Rover. SkyWalking Rover is currently supported in Kubernetes environments and must be deployed inside a Kubernetes cluster. After establishing a connection with the SkyWalking backend server, it saves information about the processes on the current machine to SkyWalking. When the user creates an eBPF profiling task via the user interface, SkyWalking Rover receives the task and executes it in the relevant C, C++, Golang, and Rust language-based programs.

Other than an eBPF-capable kernel, there are no additional prerequisites for deploying SkyWalking Rover.

CPU Profiling with Rover

CPU profiling is the most intuitive way to show service performance. Inspired by Brendan Gregg‘s blog post, we’ve divided CPU profiling into two types that we have implemented in Rover:

On-CPU Profiling: Where threads are spending time running on-CPU.
Off-CPU Profiling: Where time is spent waiting while blocked on I/O, locks, timers, paging/swapping, etc.

Profiling Envoy with eBPF

Envoy is a popular proxy, used as the data plane by the Istio service mesh. In a Kubernetes cluster, Istio injects Envoy into each service’s pod as a sidecar where it transparently intercepts and processes incoming and outgoing traffic. As the data plane, any performance issues in Envoy can affect all service traffic in the mesh. In this scenario, it’s more powerful to use eBPF profiling to analyze issues in production caused by service mesh configuration.

Demo Environment

If you want to see this scenario in action, we’ve built a demo environment where we deploy an Nginx service for stress testing. Traffic is intercepted by Envoy and forwarded to Nginx. The commands to install the whole environment can be accessed through GitHub.

On-CPU Profiling

On-CPU profiling is suitable for analyzing thread stacks when service CPU usage is high. If the stack is dumped more times, it means that the thread stack occupies more CPU resources.

When installing Istio using the demo configuration profile, we found there are two places where we can optimize performance:

Zipkin Tracing: Different Zipkin sampling percentages have a direct impact on QPS.
Access Log Format: Reducing the fields of the Envoy access log can improve QPS.

Zipkin Tracing

Zipkin with 100% sampling

In the default demo configuration profile, Envoy is using 100% sampling as default tracing policy. How does that impact the performance?

As shown in the figure below, using the on-CPU profiling, we found that it takes about 16% of the CPU overhead. At a fixed consumption of 2 CPUs, its QPS can reach 5.7K.

Disable Zipkin tracing

At this point, we found that if Zipkin is not necessary, the sampling percentage can be reduced or we can even disable tracing. Based on the Istio documentation, we can disable tracing when installing the service mesh using the following command:

istioctl install -y --set profile=demo \
   --set 'meshConfig.enableTracing=false' \
   --set 'meshConfig.defaultConfig.tracing.sampling=0.0'

After disabling tracing, we performed on-CPU profiling again. According to the figure below, we found that Zipkin has disappeared from the flame graph. With the same 2 CPU consumption as in the previous example, the QPS reached 9K, which is an almost 60% increase.

Tracing with Throughput

With the same CPU usage, we’ve discovered that Envoy performance greatly improves when the tracing feature is disabled. Of course, this requires us to make trade-offs between the number of samples Zipkin collects and the desired performance of Envoy (QPS).

The table below illustrates how different Zipkin sampling percentages under the same CPU usage affect QPS.

Zipkin sampling %	QPS	CPUs	Note
100% (default)	5.7K	2	16% used by Zipkin
1%	8.1K	2	0.3% used by Zipkin
disabled	9.2K	2	0% used by Zipkin

Access Log Format

Default Log Format

In the default demo configuration profile, the default Access Log format contains a lot of data. The flame graph below shows various functions involved in parsing the data such as request headers, response headers, and streaming the body.

Simplifying Access Log Format

Typically, we don’t need all the information in the access log, so we can often simplify it to get what we need. The following command simplifies the access log format to only display basic information:

istioctl install -y --set profile=demo \
   --set meshConfig.accessLogFormat="[%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE%\n"

After simplifying the access log format, we found that the QPS increased from 5.7K to 5.9K. When executing the on-CPU profiling again, the CPU usage of log formatting dropped from 2.4% to 0.7%.

Simplifying the log format helped us to improve the performance.

Off-CPU Profiling

Off-CPU profiling is suitable for performance issues that are not caused by high CPU usage. For example, when there are too many threads in one service, using off-CPU profiling could reveal which threads spend more time context switching.

We provide data aggregation in two dimensions:

Switch count: The number of times a thread switches context. When the thread returns to the CPU, it completes one context switch. A thread stack with a higher switch count spends more time context switching.
Switch duration: The time it takes a thread to switch the context. A thread stack with a higher switch duration spends more time off-CPU.

Write Access Log

Enable Write

Using the same environment and settings as before in the on-CPU test, we performed off-CPU profiling. As shown below, we found that access log writes accounted for about 28% of the total context switches. The “__write” shown below also indicates that this method is the Linux kernel method.

Disable Write

SkyWalking implements Envoy’s Access Log Service (ALS) feature which allows us to send access logs to the SkyWalking Observability Analysis Platform (OAP) using the gRPC protocol. Even by disabling the access logging, we can still use ALS to capture/aggregate the logs. We’ve disabled writing to the access log using the following command:

istioctl install -y --set profile=demo --set meshConfig.accessLogFile=""

After disabling the Access Log feature, we performed the off-CPU profiling. File writing entries have disappeared as shown in the figure below. Envoy throughput also increased from 5.7K to 5.9K.

Conclusion

In this article, we’ve examined the insights Apache Skywalking’s Trace Profiling can give us and how much more can be achieved with eBPF profiling. All of these features are implemented in skywalking-rover. In addition to on- and off-CPU profiling, you will also find the following features:

Continuous profiling, helps you automatically profile without manual intervention. For example, when Rover detects that the CPU exceeds a configurable threshold, it automatically executes the on-CPU profiling task.
More profiling types to enrich usage scenarios, such as network, and memory profiling.

Blog: Apache ShenYu(incubating) plugin implementation principles and observability practices

Sun, 08 May 2022 00:00:00 +0000

Content

1. Introduction of SkyWalking and ShenYu

1.1 SkyWalking

SkyWalking is an Application Performance Monitoring (APM) and Observability Analysis Platform (OAP) for microservices, distributed systems, and cloud natives, Has powerful features that provide a multi-dimensional means of application performance analysis, including distributed topology diagrams, application performance metrics, distributed link tracing, log correlation analysis and alerts. Also has a very rich ecology. Widely used in various companies and open source projects.

1.2 Apache ShenYu (incubating)

Apache ShenYu (incubating) High-performance,multi-protocol,extensible,responsive API Gateway. Compatible with a variety of mainstream framework systems, support hot plug, users can customize the development, meet the current situation and future needs of users in a variety of scenarios, experienced the temper of large-scale scenes. Rich protocol support: Http, Spring Cloud, gRPC, Dubbo, SOFARPC, Motan, Tars, etc.

2. Apache ShenYu plugin implementation principle

ShenYu’s asynchrony is a little different from previous exposure to asynchrony, it is a full-link asynchrony, the execution of each plug-in is asynchronous, and thread switching is not a single fixed situation (and the individual plug-in implementation is related). The gateway initiates service calls of various protocol types, and the existing SkyWalking plugins create ExitSpan (synchronous or asynchronous) when they initiate service calls. The gateway receives the request and creates an asynchronous EntrySpan. The asynchronous EntrySpan needs to be concatenated with the synchronous or asynchronous ExitSpan, otherwise the link will be broken.

There are 2 types of tandem solutions：

Snapshot Delivery:
Pass the snapshot after creating the EntrySpan to the thread that created the ExitSpan in some way.
Currently this approach is used in the asynchronous WebClient plugin, which can receive asynchronous snapshots. shenYu proxy Http service or SpringCloud service is to achieve span concatenation through snapshot passing.
LocalSpan transit:
Other RPC class plugins do not receive snapshots for concatenation like Asynchronous WebClient. Although you can modify other RPC plugins to receive snapshots for concatenation, it is not recommended or necessary to do so. This can be achieved by creating a LocalSpan in the thread where the ExitSpan is created, and then connecting the asynchronous EntrySpan and LocalSpan by snapshot passing. This can be done without changing the original plugin code.

The span connection is shown below:

You may ask if it is possible to create LocalSpan inside a generic plugin, instead of creating one separately for ShenYu RPC plugin? The answer is no, because you need to ensure that LocalSpan and ExitSpan are in the same thread, and ShenYu is fully linked asynchronously. The code to create LocalSpan is reused in the implementation.

3. Adding generalized call tracking to the gRPC plugin and keeping it compatible

The existing SkyWalking gRPC plugin only supports calls initiated by way of stubs. For the gateway there is no proto file, the gateway takes generalized calls (not through stubs), so tracing RPC requests, you will find that the link will break at the gateway node. In this case, it is necessary to make the gRPC plugin support generalized calls, while at the same time needing to remain compatible and not affect the original tracing method. This is achieved by determining whether the request parameter is a DynamicMessage, and if it is not, then the original tracing logic through the stub is used. If not, then the original tracing logic via stubs is used, and if not, then the generalized call tracing logic is used. The other compatibility is the difference between the old and new versions of gRPC, as well as the compatibility of various cases of obtaining server-side IP, for those interested in the source code.

4. ShenYu Gateway Observability Practice

The above explains the principle of SkyWalking ShenYu plug-in implementation, the following deployment application to see the effect. SkyWalking powerful, in addition to the link tracking requires the development of plug-ins, other powerful features out of the box. Here only describe the link tracking and application performance analysis part, if you want to experience the power of SkyWalking features, please refer to the SkyWalking official documentation.
Version description:

skywalking-java: 8.11.0-SNAPSHOT source code build. Note: The shenyu plugin will be released in version 8.11.0, and will probably release it initially in May or June. the Java agent is in the regular release phase.
skywalking: 9.0.0 V9 version

Usage instructions:
SkyWalking is designed to be very easy to use. Please refer to the official documentation for configuring and activating the shenyu plugin.

4.1 Sending requests to the gateway

Initiate various service requests to the gateway via the postman client or other means.

4.2 Request Topology Diagram

4.3 Request Trace (in the case of gRPC)

Normal Trace：

Abnormal Trace：

Click on the link node to see the corresponding node information and exception information

Service Provider Span

Gateway request span

4.4 Service Metrics Monitoring

4.5 Gateway background metrics monitoring

Database Monitoring:

Thread pool and connection pool monitoring:

4.6 JVM Monitoring

4.7 Endpoint Analysis

4.8 Exception log and exception link analysis

See official documentation for log configuration

Log monitoring

Distributed link trace details corresponding to exception logs

5. Summary

SkyWalking has very comprehensive support for metrics, link tracing, and logging in observability, and is powerful, easy to use, and designed for large distributed systems, microservices, cloud-native, container architectures, and has a rich ecosystem. Using SkyWalking to provide powerful observability support for Apache ShenYu (incubating) gives ShenYu a boost. Finally, if you are interested in high-performance responsive gateways, you can follow Apache ShenYu (incubating). Also, thanks to SkyWalking such an excellent open source software to the industry contributions.

Blog: How to use the java agent injector?

Tue, 19 Apr 2022 00:00:00 +0000

content:

1. Introduction

1.1 What’s SWCK?

SWCK is a platform for the SkyWalking user, provisions, upgrades, maintains SkyWalking relevant components, and makes them work natively on Kubernetes.

In fact, SWCK is an operator developed based on kubebuilder, providing users with Custom Resources ( CR ) and controllers for managing resources ( Controller ), all CustomResourceDefinitions（CRDs）are as follows:

1.2 What’s the java agent injector?

For a java application, users need to inject the java agent into the application to get metadata and send it to the SkyWalking backend. To make users use the java agent more natively, we propose the java agent injector to inject the java agent sidecar into a pod. The java agent injector is actually a Kubernetes Mutation Webhook Controller. The controller intercepts pod events and applies mutations to the pod if annotations exist within the request.

2. Features

Transparent. User’s applications generally run in normal containers while the java agent runs in the init container, and both belong to the same pod. Each container in the pod mounts a shared memory volume that provides a storage path for the java agent. When the pod starts, the java agent in the init container will run before the application container, and the injector will store the java agent file in the shared memory volume. When the application container starts, the injector injects the agent file into the application by setting the JVM parameter. Users can inject the java agent in this way without rebuilding the container image containing the java agent.
Configurability. The injector provides two ways to configure the java agent: global configuration and custom configuration. The default global configuration is stored in the configmap, you can update it as your own global configuration, such as backend_service. In addition, you can also set custom configuration for some applications via annotation, such as “service_name”. For more information, please see java-agent-injector.
Observability. For each injected java agent, we provide CustomDefinitionResources called JavaAgent to observe the final agent configuration. Please refer to javaagent to get more details.

3. Install SWCK

In the next steps, we will show how to build a stand-alone Kubernetes cluster and deploy the 0.6.1 version of SWCK on the platform.

3.1 Tool Preparation

Firstly, you need to install some tools as follows:

kind, which is used to create a stand-alone Kubernetes cluster.
kubectl, which is used to communicate with the Kubernetes cluster.

3.2 Install stand-alone Kubernetes cluster

After installing kind , you could use the following command to create a stand-alone Kubernetes cluster.

Notice! If your terminal is configured with a proxy, you need to close it before the cluster is created to avoid some errors.

$ kind create cluster --image=kindest/node:v1.19.1

After creating a cluster, you can get the pods as below.

$ kubectl get pod -A                          
NAMESPACE            NAME                                         READY   STATUS    RESTARTS   AGE
kube-system          coredns-f9fd979d6-57xpc                      1/1     Running   0          7m16s
kube-system          coredns-f9fd979d6-8zj8h                      1/1     Running   0          7m16s
kube-system          etcd-kind-control-plane                      1/1     Running   0          7m23s
kube-system          kindnet-gc9gt                                1/1     Running   0          7m16s
kube-system          kube-apiserver-kind-control-plane            1/1     Running   0          7m23s
kube-system          kube-controller-manager-kind-control-plane   1/1     Running   0          7m23s
kube-system          kube-proxy-6zbtb                             1/1     Running   0          7m16s
kube-system          kube-scheduler-kind-control-plane            1/1     Running   0          7m23s
local-path-storage   local-path-provisioner-78776bfc44-jwwcs      1/1     Running   0          7m16s

3.3 Install certificates manger(cert-manger)

The certificates of SWCK are distributed and verified by the certificate manager. You need to install the cert-manager through the following command.

$ kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.3.1/cert-manager.yaml

Verify whether cert-manager is installed successfully.

$ kubectl get pod -n cert-manager
NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-7dd5854bb4-slcmd              1/1     Running   0          73s
cert-manager-cainjector-64c949654c-tfmt2   1/1     Running   0          73s
cert-manager-webhook-6bdffc7c9d-h8cfv      1/1     Running   0          73s

3.4 Install SWCK

The java agent injector is a component of the operator, so please follow the next steps to install the operator first.

Get the deployment yaml file of SWCK and deploy it.

$ curl -Ls https://archive.apache.org/dist/skywalking/swck/0.6.1/skywalking-swck-0.6.1-bin.tgz | tar -zxf - -O ./config/operator-bundle.yaml | kubectl apply -f -

Check SWCK as below.

$ kubectl get pod -n skywalking-swck-system
NAME                                                  READY   STATUS    RESTARTS   AGE
skywalking-swck-controller-manager-7f64f996fc-qh8s9   2/2     Running   0          94s

3.5 Install Skywalking components — OAPServer and UI

Deploy the OAPServer and UI in the default namespace.

$ kubectl apply -f https://raw.githubusercontent.com/apache/skywalking-swck/master/operator/config/samples/default.yaml

Check the OAPServer.

$ kubectl get oapserver
NAME      INSTANCES   RUNNING   ADDRESS
default   1           1         default-oap.default

Check the UI.

$ kubectl get ui
NAME      INSTANCES   RUNNING   INTERNALADDRESS      EXTERNALIPS   PORTS
default   1           1         default-ui.default                 [80]

4. Deploy a demo application

In the third step, we have installed SWCK and related Skywalking components. Next, we will show how to use the java agent injector in SWCK through two java application examples in two ways: global configuration and custom configuration.

4.1 Set the global configuration

When we have installed SWCK, the default configuration is the configmap in the system namespace, we can get it as follows.

$  kubectl get configmap skywalking-swck-java-agent-configmap -n skywalking-swck-system -oyaml
apiVersion: v1
data:
  agent.config: |-
    # The service name in UI
    agent.service_name=${SW_AGENT_NAME:Your_ApplicationName}

    # Backend service addresses.
    collector.backend_service=${SW_AGENT_COLLECTOR_BACKEND_SERVICES:127.0.0.1:11800}

    # Please refer to https://skywalking.apache.org/docs/skywalking-java/latest/en/setup/service-agent/java-agent/configurations/#table-of-agent-configuration-properties to get more details.

In the cluster created by kind, the backend_service may not be correct, we need to use the real OAPServer’s address default-oap.default to replace the default 127.0.0.1, so we can edit the configmap as follow.

$ kubectl edit configmap skywalking-swck-java-agent-configmap -n skywalking-swck-system
configmap/skywalking-swck-java-agent-configmap edited

$ kubectl get configmap skywalking-swck-java-agent-configmap -n skywalking-swck-system -oyaml
apiVersion: v1
data:
  agent.config: |-
    # The service name in UI
    agent.service_name=${SW_AGENT_NAME:Your_ApplicationName}

    # Backend service addresses.
    collector.backend_service=${SW_AGENT_COLLECTOR_BACKEND_SERVICES:default-oap.default:11800}

    # Please refer to https://skywalking.apache.org/docs/skywalking-java/latest/en/setup/service-agent/java-agent/configurations/#table-of-agent-configuration-properties to get more details.

4.2 Set the custom configuration

In some cases, we need to use the Skywalking component to monitor different java applications, so the agent configuration of different applications may be different, such as the name of the application, and the plugins that the application needs to use, etc. Next, we will take two simple java applications developed based on spring boot and spring cloud gateway as examples for a detailed description. You can use the source code to build the image.

# build the springboot and springcloudgateway image 
$ git clone https://github.com/dashanji/swck-spring-cloud-k8s-demo 
$ cd swck-spring-cloud-k8s-demo && make

# check the image
$ docker images
REPOSITORY     TAG       IMAGE ID       CREATED          SIZE
gateway        v0.0.1    51d16251c1d5   48 minutes ago   723MB
app            v0.0.1    62f4dbcde2ed   48 minutes ago   561MB

# load the image into the cluster
$ kind load docker-image app:v0.0.1 && kind load docker-image gateway:v0.0.1

4.3 deploy spring boot application

Create the springboot-system namespace.

$ kubectl create namespace springboot-system

Label the springboot-systemnamespace to enable the java agent injector.

$ kubectl label namespace springboot-system swck-injection=enabled

Deploy the corresponding deployment file springboot.yaml for the spring boot application, which uses annotation to override the default agent configuration, such as service_name.

Notice! Before using the annotation to override the agent configuration, you need to add strategy.skywalking.apache.org/agent.Overlay: "true" to make the override take effect.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo-springboot
  namespace: springboot-system
spec:
  selector:
    matchLabels:
      app: demo-springboot
  template:
    metadata:
      labels:
        swck-java-agent-injected: "true"  # enable the java agent injector
        app: demo-springboot
      annotations:
        strategy.skywalking.apache.org/agent.Overlay: "true"  # enable the agent overlay
        agent.skywalking.apache.org/agent.service_name: "backend-service"
    spec:
      containers:
      - name: springboot
        imagePullPolicy: IfNotPresent
        image: app:v0.0.1
        command: ["java"]
        args: ["-jar","/app.jar"]
---
apiVersion: v1
kind: Service
metadata:
  name: demo
  namespace: springboot-system
spec:
  type: ClusterIP
  ports:
  - name: 8085-tcp
    port: 8085
    protocol: TCP
    targetPort: 8085
  selector:
    app: demo-springboot

Deploy a spring boot application in the springboot-system namespace.

$ kubectl apply -f springboot.yaml

Check for deployment.

$ kubectl get pod -n springboot-system
NAME                               READY   STATUS    RESTARTS   AGE
demo-springboot-7c89f79885-dvk8m   1/1     Running   0          11s

Get the finnal injected java agent configuration through JavaAgent.

$ kubectl get javaagent -n springboot-system
NAME                            PODSELECTOR           SERVICENAME       BACKENDSERVICE
app-demo-springboot-javaagent   app=demo-springboot   backend-service   default-oap.default:11800

4.4 deploy spring cloud gateway application

Create the gateway-system namespace.

$ kubectl create namespace gateway-system

Label the gateway-systemnamespace to enable the java agent injector.

$ kubectl label namespace gateway-system swck-injection=enabled

Deploy the corresponding deployment file springgateway.yaml for the spring cloud gateway application, which uses annotation to override the default agent configuration, such as service_name. In addition, when using spring cloud gateway, we need to add the spring cloud gateway plugin to the agent configuration.

Notice! Before using the annotation to override the agent configuration, you need to add strategy.skywalking.apache.org/agent.Overlay: "true" to make the override take effect.

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: demo-gateway
  name: demo-gateway
  namespace: gateway-system
spec:
  selector:
    matchLabels:
      app: demo-gateway
  template:
    metadata:
      labels:
        swck-java-agent-injected: "true"
        app: demo-gateway
      annotations:
        strategy.skywalking.apache.org/agent.Overlay: "true"
        agent.skywalking.apache.org/agent.service_name: "gateway-service"     
        optional.skywalking.apache.org: "cloud-gateway-3.x" # add spring cloud gateway plugin
    spec:
      containers:
      - image: gateway:v0.0.1
        name: gateway
        command: ["java"]
        args: ["-jar","/gateway.jar"]
---
apiVersion: v1
kind: Service
metadata:
  name: service-gateway
  namespace: gateway-system
spec:
  type: ClusterIP
  ports:
  - name: 9999-tcp
    port: 9999
    protocol: TCP
    targetPort: 9999
  selector:
    app: demo-gateway

Deploy a spring cloud gateway application in the gateway-system namespace.

$ kubectl apply -f springgateway.yaml

Check for deployment.

$ kubectl get pod -n gateway-system
NAME                           READY   STATUS    RESTARTS   AGE
demo-gateway-5bb77f6d85-9j7c6   1/1     Running   0          15s

Get the finnal injected java agent configuration through JavaAgent.

$ kubectl get javaagent -n gateway-system
NAME                         PODSELECTOR        SERVICENAME       BACKENDSERVICE
app-demo-gateway-javaagent   app=demo-gateway   gateway-service   default-oap.default:11800

5. Verify the injector

After completing the above steps, we can view detailed state of the injected pod, like the injected agent container.

# get all injected pod
$ kubectl get pod -A -lswck-java-agent-injected=true
NAMESPACE           NAME                               READY   STATUS    RESTARTS   AGE
gateway-system      demo-gateway-5bb77f6d85-lt4z7      1/1     Running   0          69s
springboot-system   demo-springboot-7c89f79885-lkb5j   1/1     Running   0          75s

# view detailed state of the injected pod [demo-springboot]
$ kubectl describe pod -l app=demo-springboot -n springboot-system
...
Events:
  Type   Reason   Age                From                           Message
  ----   ------  ----                ----                           -------
  ...
  Normal Created  91s  kubelet,kind-control-plane Created  container inject-skywalking-agent
  Normal Started  91s  kubelet,kind-control-plane Started  container inject-skywalking-agent
  ...
  Normal Created  90s  kubelet,kind-control-plane Created  container springboot
  Normal Started  90s  kubelet,kind-control-plane Started  container springboot

# view detailed state of the injected pod [demo-gateway] 
$ kubectl describe pod -l app=demo-gateway -n gateway-system
...
Events:
  Type   Reason   Age            From                         Message
  ----   ------  ----            ----                         -------
  ...
  Normal Created 2m20s kubelet,kind-control-plane Created container inject-skywalking-agent
  Normal Started 2m20s kubelet,kind-control-plane Started container inject-skywalking-agent
  ...
  Normal Created 2m20s kubelet,kind-control-plane Created container gateway
  Normal Started 2m20s kubelet,kind-control-plane Started container gateway

Now we can expose the service and watch the data displayed on the web. First of all, we need to get the gateway service and the ui service as follows.

$ kubectl get service service-gateway -n gateway-system
NAME              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service-gateway   ClusterIP   10.99.181.145   <none>        9999/TCP   9m19s

$ kubectl get service default-ui
NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
default-ui   ClusterIP   10.111.39.250   <none>        80/TCP    82m

Then open two terminals to expose the service: service-gateway、default-ui.

$ kubectl port-forward service/service-gateway -n gateway-system 9999:9999
Forwarding from 127.0.0.1:9999 -> 9999
Forwarding from [::1]:9999 -> 9999

$ kubectl port-forward service/default-ui 8090:80                     
Forwarding from 127.0.0.1:8090 -> 8080
Forwarding from [::1]:8090 -> 8080

Use the following commands to access the spring boot demo 10 times through the spring cloud gateway service.

$ for i in {1..10}; do curl http://127.0.0.1:9999/gateway/hello && echo ""; done
Hello World!
Hello World!
Hello World!
Hello World!
Hello World!
Hello World!
Hello World!
Hello World!
Hello World!
Hello World!

We can see the Dashboard by accessing http://127.0.0.1:8090.

All services’ topology is shown below.

We can see the trace information of gateway-service.

We can see the trace information of backend-service.

6. Concluding remarks

If your application is deployed in the Kubernetes platform and requires Skywalking to provide monitoring services, SWCK can help you deploy, upgrade and maintain the Skywalking components in the Kubernetes cluster. In addition to this blog, you can also view swck document and Java agent injector documentation for more information. If you find this project useful, please give SWCK a star! If you have any questions, welcome to ask in Issues or Discussions.

Blog: Integrating Apache SkyWalking with source code

Thu, 14 Apr 2022 00:00:00 +0000

Introduction

The most profound technologies are those that disappear. They weave themselves into the fabric of everyday life until they are indistinguishable from it. - Mark Weiser

Mark Weiser prophetically argued in the late 1980s, that the most far-reaching technologies are those which vanish into thin air. According to Weiser, “Whenever people learn something sufficiently well, they cease to be aware of it.” This disappearing act, as Weiser claimed, is not limited to technology but rather human psychology. It is this very experience that allows us to escape lower-level thinking into higher-level thinking. For once we are no longer impeded by mundane details, we are then free to focus on new goals.

This realization becomes more relevant as APMs become increasingly popular. As more applications are deployed with APMs, the number of abstract representations of the underlying source code also increases. While this provides great value to many non-development roles within an organization, it does pose additional challenges to those in development roles who must translate these representations into concepts they can work with (i.e. source code). Weiser sums this difficultly up rather succinctly when he states that “Programmers should no more be asked to work without access to source code than auto-mechanics should be asked to work without looking at the engine.”

Still, APMs collect more information only to produce a plethora of new abstract representations. In this article, we will introduce a new concept in Source++, the open-source live-coding platform, specifically designed to allow developers to monitor production applications more intuitively.

Live Views

And we really don’t understand even yet, hundreds of metrics later, what make a program easier to understand or modify or reuse or borrow. I don’t think we’ll find out by looking away from programs to their abstract interfaces. The answers are in the source code. - Mark Weiser

As APMs move from the “nice to have” category to the “must-have” category, there is a fundamental feature holding them back from ubiquity. They must disappear from consciousness. As developers, we should feel no impulse to open our browsers to better understand the underlying source code. The answers are literally in the source code. Instead, we should improve our tools so the source code conveniently tells us what we need to know. Think of how simple life could be if failing code always indicated how and why it failed. This is the idea behind Source++.

In our last blog post, we discussed Extending Apache SkyWalking with non-breaking breakpoints. In that post, we introduced a concept called Live Instruments, which developers can use to easily debug live production applications without leaving their IDE. Today, we will discuss how existing SkyWalking installations can be integrated into your IDE via a new concept called Live Views. Unlike Live Instruments, which are designed for debugging live applications, Live Views are designed for increasing application comprehension and awareness. This is accomplished through a variety of commands which are input into the Live Command Palette.

Live Command Palette

The Live Command Palette (LCP) is a contextual command prompt, included in the Source++ JetBrains Plugin, that allows developers to control and query live applications from their IDE. Opened via keyboard shortcut (Ctrl+Shift+S), the LCP allows developers to easily view metrics relevant to the source code they’re currently viewing. The following Live View commands are currently supported:

Command: view (overview/activity/traces/logs)

The view commands display contextual popups with live operational data of the current source code. These commands allow developers to view traditional SkyWalking operational data filtered down to the relevant metrics.

Command: watch log

The watch log command allows developers to follow individual log statements of a running application in real-time. This command allows developers to negate the need for manually scrolling through the logs to find instances of a specific log statement.

Command: (show/hide) quick stats

The show quick stats command displays live endpoint metrics for a quick idea of an endpoint’s activity. Using this command, developers can quickly assess the status of an endpoint and determine if the endpoint is performing as expected.

Future Work

A good tool is an invisible tool. By invisible, I mean that the tool does not intrude on your consciousness; you focus on the task, not the tool. Eyeglasses are a good tool – you look at the world, not the eyeglasses. - Mark Weiser

Source++ aims to extend SkyWalking in such a way that SkyWalking itself becomes invisible. To accomplish this, we plan to support custom developer commands. Developers will be able to build customized commands for themselves, as well as commands to share with their team. These commands will recognize context, types, and conditions allowing for a wide possibility of operations. As more commands are added, developers will be able to expose everything SkyWalking has to offer while focusing on what matters most, the source code.

If you find these features useful, please consider giving Source++ a try. You can install the plugin directly from your JetBrains IDE, or through the JetBrains Marketplace. If you have any issues or questions, please open an issue. Feedback is always welcome!

Blog: [Resolved][License Issue] Volcengine Inc.(火山引擎) violates the Apache 2.0 License when using SkyWalking.

Fri, 28 Jan 2022 00:00:00 +0000

Apache SkyWalking is an open-source APM for a distributed system, Apache Software Foundation top-level project.

On Jan. 28th, we received a License violation report from one of the committers (anonymously). They have a cloud service called Application Performance Monitoring - Distributed Tracing (应用性能监控全链路版). At the Java service monitoring section, it provides this agent download link

wget https://datarangers.com.cn/apminsight/repo/v2/download/java-agent/apminsight-java-agent_latest.tar.gz

We downloaded it at 23:15 Jan. 28th UTC+8(Beijing), and archived it at here

We have confirmed this is a distribution of SkyWalking Java agent.

We listed several pieces of evidence to prove this here, every reader could compare with the official SkyWalking source codes

The first and the easiest one is agent.config file, which is using the same config keys, and the same config format.

This is the Volcengine’s version, and check SkyWalking agent.config

In the apmplus-agent.jar, Volcengine’s agent core jar, you could easily find several core classes exactly as same as SkyWalking’s.

The ComponentsDefine class is unchanged, even with component ID and name. This is Volcengine’s version, and check SkyWalking’s version

The whole code names, package names, and hierarchy structure are all as same as SkyWalking 6.x version.

This is the Volcengine package hierarchy structure, and check the SkyWalking’s version

Volcengine Inc.’s team changed all package names, removed the Apache Software Foundation’s header, and don’t keep Apache Software Foundation and Apache SkyWalking’s LICENSE and NOTICE file in their redistribution.

Also, we can’t find anything on their website to declare they are distributing SkyWalking.

All above have proved they are violating the Apache 2.0 License, and don’t respect Apache Software Foundation and Apache SkyWalking’s IP and Branding.

We have contacted their legal team, and wait for their official response.

Resolution

On Jan. 30th night, UTC+8, 2022. We received a response from Volcengine’s APMPlus team. They admitted their violation behaviors, and made the following changes.

Volcengine’s APMPlus service page was updated on January 30th and stated that the agent is a fork version(re-distribution) of Apache SkyWalking agent. Below is the screenshot of Volcengine’s APMPlus product page.

Volcengine’s APMPlus agent distributions were also updated and include SkyWalking’s License and NOTICE now. Below is the screenshot of Volcengine’s APMPlus latest agent, you could download from the product page. We keep a copy of their Jan. 30th 2022 at here.

Volcengine’s APMPlus team had restored all license headers of SkyWalking in the agent, and the modifications of the project files are also listed in “SkyWalking-NOTICE”, which you could download from the product page.

We have updated the status to the PMC mail list. This license violation issue has been resolved for now.

Appendix

Inquiries of committers

Q: I hope Volcengine Inc. can give a reason for this license issue, not just an afterthought PR. This will not only let us know where the issue is but also avoid similar problems in the future.

A(apmplus apmplus@volcengine.com):

The developers neglected this repository during submitting compliance
assessment.
Currently, APMPlus team had introduced advanced tools provided by the
company for compliance assessment, and we also strengthened training for
our developers.
In the future, the compliance assessment process will be further improved
from tool assessment and manual assessment.

Blog: Scaling with Apache SkyWalking

Mon, 24 Jan 2022 00:00:00 +0000

Background

In the Apache SkyWalking ecosystem, the OAP obtains metrics, traces, logs, and event data through SkyWalking Agent, Envoy, or other data sources. Under the gRPC protocol, it transmits data by communicating with a single server node. Only when the connection is broken, the reconnecting policy would be used based on DNS round-robin mode. When new services are added at runtime or the OAP load is kept high due to increased traffic of observed services, the OAP cluster needs to scale out for increased traffic. The load of the new OAP node would be less due to all existing agents having connected to previous nodes. Even without scaling, the load of OAP nodes would be unbalanced, because the agent would keep the connection due to random policy at the booting stage. In these cases, it would become a challenge to keep up the health status of all nodes, and be able to scale out when needed.

In this article, we mainly discuss how to solve this challenge in SkyWalking.

How to Load Balance

SkyWalking mainly uses the gRPC protocol for data transmission, so this article mainly introduces load balancing in the gRPC protocol.

Proxy Or Client-side

Based on the gRPC official Load Balancing blog, there are two approaches to load balancing:

Client-side: The client perceives multiple back-end services and uses a load-balancing algorithm to select a back-end service for each RPC.
Proxy: The client sends the message to the proxy server, and the proxy server load balances the message to the back-end service.

From the perspective of observability system architecture:

	Pros	Cons
Client-side	High performance because of the elimination of extra hop	Complex client (cluster awareness, load balancing, health check, etc.) Ensure each data source to be connected provides complex client capabilities
Proxy	Simple Client	Higher latency

We choose Proxy mode for the following reasons:

Observable data is not very time-sensitive, a little latency caused by transmission is acceptable. A little extra hop is acceptable and there is no impact on the client-side.
As an observability platform, we cannot/should not ask clients to change. They make their own tech decisions and may have their own commercial considerations.

Transmission Policy

In the proxy mode, we should determine the transmission path between downstream and upstream.

Different data protocols require different processing policies. There are two transmission policies:

Synchronous: Suitable for protocols that require data exchange in the client, such as SkyWalking Dynamic Configuration Service. This type of protocol provides real-time results.
Asynchronous batch: Used when the client doesn’t care about the upstream processing results, but only the transmitted data (e.g., trace report, log report, etc.)

The synchronization policy requires that the proxy send the message to the upstream server when receiving the client message, and synchronously return the response data to the downstream client. Usually, only a few protocols need to use the synchronization policy.

As shown below, after the client sends the request to the Proxy, the proxy would send the message to the server synchronously. When the proxy receives the result, it returns to the client.

The asynchronous batch policy means that the data is sent to the upstream server in batches asynchronously. This policy is more common because most protocols in SkyWalking are primarily based on data reporting. We think using the queue as a buffer could have a good effect. The asynchronous batch policy is executed according to the following steps:

The proxy receives the data and wraps it as an Event object.
An event is added into the queue.
When the cycle time is reached or when the queue elements reach the fixed number, the elements in the queue will parallel consume and send to the OAP.

The advantage of using queues is:

Separate data receiving and sending to reduce the mutual influence.
The interval quantization mechanism can be used to combine events, which helps to speed up sending events to the OAP.
Using multi-threaded consumption queue events can make fuller use of network IO.

As shown below, after the proxy receives the message, the proxy would wrap the message as an event and push it to the queue. The message sender would take batch events from the queue and send them to the upstream OAP.

Routing

Routing algorithms are used to route messages to a single upstream server node.

The Round-Robin algorithm selects nodes in order from the list of upstream service nodes. The advantage of this algorithm is that the number of times each node is selected is average. When the size of the data is close to the same, each upstream node can handle the same quantity of data content.

With the Weight Round-Robin, each upstream server node has a corresponding routing weight ratio. The difference from Round-Robin is that each upstream node has more chances to be routed according to its weight. This algorithm is more suitable to use when the upstream server node machine configuration is not the same.

The Fixed algorithm is a hybrid algorithm. It can ensure that the same data is routed to the same upstream server node, and when the upstream server scales out, it still maintains routing to the same node; unless the upstream node does not exist, it will reroute. This algorithm is mainly used in the SkyWalking Meter protocol because this protocol needs to ensure that the metrics of the same service instance are sent to the same OAP node. The Routing steps are as follows:

Generate a unique identification string based on the data content, as short as possible. The amount of data is controllable.
Get the upstream node of identity from LRU Cache, and use it if it exists.
According to the identification, generate the corresponding hash value, and find the upstream server node from the upstream list.
Save the mapping relationship between the upstream server node and identification to LRU Cache.

The advantage of this algorithm is to bind the data with the upstream server node as much as possible, so the upstream server can better process continuous data. The disadvantage is that it takes up a certain amount of memory space to save the corresponding relationship.

As shown below, the image is divided into two parts:

The left side represents that the same data content always is routed to the same server node.
The right side represents the data routing algorithm. Get the number from the data, and use the remainder algorithm to obtain the position.

We choose to use a combination of Round-Robin and Fixed algorithm for routing:

The Fixed routing algorithm is suitable for specific protocols, mainly used when passing metrics data to the SkyWalking Meter protocol
The Round-Robin algorithm is used by default. When the SkyWalking OAP cluster is deployed, the configuration of the nodes needs to be as much the same as possible, so there would be no need to use the Weight Round-Robin algorithm.

How to balance the load balancer itself?

Proxy still needs to deal with the load balancing problem from client to itself, especially when deploying a Proxy cluster in a production environment.

There are three ways to solve this problem:

Connection management: Use the max_connection config on the client-side to specify the maximum connection duration of each connection. For more information, please read the proposal.
Cluster awareness: The proxy has cluster awareness, and actively disconnects the connection when the load is unbalanced to allow the client to re-pick up the proxy.
Resource limit+HPA: Restrict the connection resource situation of each proxy, and no longer accept new connections when the resource limit is reached. And use the HPA mechanism of Kubernetes to dynamically scale out the number of the proxy.

	Connection management	Cluster awareness	Resource Limit+HPA
Pros	Simple to use	Ensure that the number of connections in each proxy is relatively	Simple to use
Cons	Each client needs to ensure that data is not lost The client is required to accept GOWAY responses	May cause a sudden increase in traffic on some nodes Each client needs to ensure that data is not lost	Traffic will not be particularly balanced in each instance

We choose Limit+HPA for these reasons:

Easy to config and use the proxy and easy to understand based on basic data metrics.
No data loss due to broken connection. There is no need for the client to implement any other protocols to prevent data loss, especially when the client is a commercial product.
The connection of each node in the proxy cluster does not need to be particularly balanced, as long as the proxy node itself is high-performance.

SkyWalking-Satellite

We have implemented this Proxy in the SkyWalking-Satellite project. It’s used between Client and SkyWalking OAP, effectively solving the load balancing problem.

After the system is deployed, the Satellite would accept the traffic from the Client, and the Satellite will perceive all the nodes of the OAP through Kubernetes Label Selector or manual configuration, and load balance the traffic to the upstream OAP node.

As shown below, a single client still maintains a connection with a single Satellite, Satellite would establish the connection with each OAP, and load balance message to the OAP node.

When scaling Satellite, we need to deploy the SWCK adapter and configure the HPA in Kubernetes. SWCK is a platform for the SkyWalking users, provisions, upgrades, maintains SkyWalking relevant components, and makes them work natively on Kubernetes.

After deployment is finished, the following steps would be performed:

Read metrics from OAP: HPA requests the SWCK metrics adapter to dynamically read the metrics in the OAP.
Scaling the Satellite: Kubernetes HPA senses that the metrics values are in line with expectations, so the Satellite would be scaling automatically.

As shown below, use the dotted line to divide the two parts. HPA uses SWCK Adapter to read the metrics in the OAP. When the threshold is met, HPA would scale the Satellite deployment.

Example

In this section, we will demonstrate two cases:

SkyWalking Scaling: After SkyWalking OAP scaling, the traffic would auto load balancing through Satellite.
Satellite Scaling: Satellite’s own traffic load balancing.

NOTE: All commands could be accessed through GitHub.

SkyWalking Scaling

We will use the bookinfo application to demonstrate how to integrate Apache SkyWalking 8.9.1 with Apache SkyWalking-Satellite 0.5.0, and observe the service mesh through the Envoy ALS protocol.

Before starting, please make sure that you already have a Kubernetes environment.

Install Istio

Istio provides a very convenient way to configure the Envoy proxy and enable the access log service. The following step:

Install the istioctl locally to help manage the Istio mesh.
Install Istio into the Kubernetes environment with a demo configuration profile, and enable the Envoy ALS. Transmit the ALS message to the satellite. The satellite we will deploy later.
Add the label into the default namespace so Istio could automatically inject Envoy sidecar proxies when you deploy your application later.

# install istioctl
export ISTIO_VERSION=1.12.0
curl -L https://istio.io/downloadIstio | sh - 
sudo mv $PWD/istio-$ISTIO_VERSION/bin/istioctl /usr/local/bin/

# install istio
istioctl install -y --set profile=demo \
	--set meshConfig.enableEnvoyAccessLogService=true \
	--set meshConfig.defaultConfig.envoyAccessLogService.address=skywalking-system-satellite.skywalking-system:11800

# enbale envoy proxy in default namespace
kubectl label namespace default istio-injection=enabled

Install SWCK

SWCK provides convenience for users to deploy and upgrade SkyWalking related components based on Kubernetes. The automatic scale function of Satellite also mainly relies on SWCK. For more information, you could refer to the official documentation.

# Install cert-manager
kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.3.1/cert-manager.yaml

# Deploy SWCK
mkdir -p skywalking-swck && cd skywalking-swck
wget https://dlcdn.apache.org/skywalking/swck/0.6.1/skywalking-swck-0.6.1-bin.tgz
tar -zxvf skywalking-swck-0.6.1-bin.tgz
cd config
kubectl apply -f operator-bundle.yaml

Deploy Apache SkyWalking And Apache SkyWalking-Satellite

We have provided a simple script to deploy the skywalking OAP, UI, and Satellite.

# Create the skywalking components namespace
kubectl create namespace skywalking-system
kubectl label namespace skywalking-system swck-injection=enabled
# Deploy components
kubectl apply -f https://raw.githubusercontent.com/mrproliu/sw-satellite-demo-scripts/5821a909b647f7c8f99c70378e197630836f45f7/resources/sw-components.yaml

Deploy Bookinfo Application

export ISTIO_VERSION=1.12.0
kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/platform/kube/bookinfo.yaml
kubectl wait --for=condition=Ready pods --all --timeout=1200s
kubectl port-forward service/productpage 9080

Next, please open your browser and visit http://localhost:9080. You should be able to see the Bookinfo application. Refresh the webpage several times to generate enough access logs.

Then, you can see the topology and metrics of the Bookinfo application on SkyWalking WebUI. At this time, you can see that the Satellite is working!

Deploy Monitor

We need to install OpenTelemetry Collector to collect metrics in OAPs and analyze them.

# Add OTEL collector
kubectl apply -f https://raw.githubusercontent.com/mrproliu/sw-satellite-demo-scripts/5821a909b647f7c8f99c70378e197630836f45f7/resources/otel-collector-oap.yaml

kubectl port-forward -n skywalking-system  service/skywalking-system-ui 8080:80

Next, please open your browser and visit http://localhost:8080/ and create a new item on the dashboard. The SkyWalking Web UI pictured below shows how the data content is applied.

Scaling OAP

Scaling the number of OAPs by deployment.

kubectl scale --replicas=3 -n skywalking-system deployment/skywalking-system-oap

Done!

After a period of time, you will see that the number of OAPs becomes 3, and the ALS traffic is balanced to each OAP.

Satellite Scaling

After we have completed the SkyWalking Scaling, we would carry out the Satellite Scaling demo.

Deploy SWCK HPA

SWCK provides an adapter to implement the Kubernetes external metrics to adapt the HPA through reading the metrics in SkyWalking OAP. We expose the metrics service in Satellite to OAP and configure HPA Resource to auto-scaling the Satellite.

Install the SWCK adapter into the Kubernetes environment:

kubectl apply -f skywalking-swck/config/adapter-bundle.yaml

Create the HPA resource, and limit each Satellite to handle a maximum of 10 connections:

kubectl apply -f https://raw.githubusercontent.com/mrproliu/sw-satellite-demo-scripts/5821a909b647f7c8f99c70378e197630836f45f7/resources/satellite-hpa.yaml

Then, you could see we have 9 connections in one satellite. One envoy proxy may establish multiple connections to the satellite.

$ kubectl get HorizontalPodAutoscaler -n skywalking-system
NAME       REFERENCE                                TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
hpa-demo   Deployment/skywalking-system-satellite   9/10      1         3         1          5m18s

Scaling Application

The scaling application could establish more connections to the satellite, to verify whether the HPA is in effect.

kubectl scale --replicas=3 deployment/productpage-v1 deployment/details-v1

Done!

By default, Satellite will deploy a single instance and a single instance will only accept 11 connections. HPA resources limit one Satellite to handle 10 connections and use a stabilization window to make Satellite stable scaling up. In this case, we deploy the Bookinfo application in 10+ instances after scaling, which means that 10+ connections will be established to the Satellite.

So after HPA resources are running, the Satellite would be automatically scaled up to 2 instances. You can learn about the calculation algorithm of replicas through the official documentation. Run the following command to view the running status:

$ kubectl get HorizontalPodAutoscaler -n skywalking-system --watch
NAME       REFERENCE                                TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
hpa-demo   Deployment/skywalking-system-satellite   11/10     1         3         1          3m31s
hpa-demo   Deployment/skywalking-system-satellite   11/10     1         3         1          4m20s
hpa-demo   Deployment/skywalking-system-satellite   11/10     1         3         2          4m38s
hpa-demo   Deployment/skywalking-system-satellite   11/10     1         3         2          5m8s
hpa-demo   Deployment/skywalking-system-satellite   6/10      1         3         2          5m23s

By observing the “number of connections” metric, we would be able to see that when the number of connections of each gRPC exceeds 10 connections, then the satellite automatically scales through the HPA rule. As a result, the connection number is down to normal status (in this example, less than 10)

swctl metrics linear --name satellite_service_grpc_connect_count --service-name satellite::satellite-service

Blog: Chaos Mesh + SkyWalking: Better Observability for Chaos Engineering

Tue, 21 Dec 2021 00:00:00 +0000

Chaos Mesh is an open-source cloud-native chaos engineering platform. You can use Chaos Mesh to conveniently inject failures and simulate abnormalities that might occur in reality, so you can identify potential problems in your system. Chaos Mesh also offers a Chaos Dashboard which allows you to monitor the status of a chaos experiment. However, this dashboard cannot let you observe how the failures in the experiment impact the service performance of applications. This hinders us from further testing our systems and finding potential problems.

Apache SkyWalking is an open-source application performance monitor (APM), specially designed to monitor, track, and diagnose cloud native, container-based distributed systems. It collects events that occur and then displays them on its dashboard, allowing you to observe directly the type and number of events that have occurred in your system and how different events impact the service performance.

When you use SkyWalking and Chaos Mesh together during chaos experiments, you can observe how different failures impact the service performance.

This tutorial will show you how to configure SkyWalking and Chaos Mesh. You’ll also learn how to leverage the two systems to monitor events and observe in real time how chaos experiments impact applications’ service performance.

Preparation

Before you start to use SkyWalking and Chaos Mesh, you have to:

Set up a SkyWalking cluster according to the SkyWalking configuration guide.
Deploy Chao Mesh using Helm.
Install JMeter or other Java testing tools (to increase service loads).
Configure SkyWalking and Chaos Mesh according to this guide if you just want to run a demo.

Now, you are fully prepared, and we can cut to the chase.

Step 1: Access the SkyWalking cluster

After you install the SkyWalking cluster, you can access its user interface (UI). However, no service is running at this point, so before you start monitoring, you have to add one and set the agents.

In this tutorial, we take Spring Boot, a lightweight microservice framework, as an example to build a simplified demo environment.

Create a SkyWalking demo in Spring Boot by referring to this document.
Execute the command kubectl apply -f demo-deployment.yaml -n skywalking to deploy the demo.

After you finish deployment, you can observe the real-time monitoring results at the SkyWalking UI.

Note: Spring Boot and SkyWalking have the same default port number: 8080. Be careful when you configure the port forwarding; otherise, you may have port conflicts. For example, you can set Spring Boot’s port to 8079 by using a command like kubectl port-forward svc/spring-boot-skywalking-demo 8079:8080 -n skywalking to avoid conflicts.

Step 2: Deploy SkyWalking Kubernetes Event Exporter

SkyWalking Kubernetes Event Exporter is able to watch, filter, and send Kubernetes events into the SkyWalking backend. SkyWalking then associates the events with the system metrics and displays an overview about when and how the metrics are affected by the events.

If you want to deploy SkyWalking Kubernetes Event Explorer with one line of commands, refer to this document to create configuration files in YAML format and then customize the parameters in the filters and exporters. Now, you can use the command kubectl apply to deploy SkyWalking Kubernetes Event Explorer.

Step 3: Use JMeter to increase service loads

To better observe the change in service performance, you need to increase the service loads on Spring Boot. In this tutorial, we use JMeter, a widely adopted Java testing tool, to increase the service loads.

Perform a stress test on localhost:8079 using JMeter and add five threads to continuously increase the service loads.

Open the SkyWalking Dashboard. You can see that the access rate is 100%, and that the service loads reach about 5,300 calls per minute (CPM).

Step 4: Inject failures via Chaos Mesh and observe results

After you finish the three steps above, you can use the Chaos Dashboard to simulate stress scenarios and observe the change in service performance during chaos experiments.

The following sections describe how service performance varies under the stress of three chaos conditions:

CPU load: 10%; memory load: 128 MB

The first chaos experiment simulates low CPU usage. To display when a chaos experiment starts and ends, click the switching button on the right side of the dashboard. To learn whether the experiment is Applied to the system or Recovered from the system, move your cursor onto the short, green line.

During the time period between the two short, green lines, the service load decreases to 4,929 CPM, but returns to normal after the chaos experiment ends.
CPU load: 50%; memory load: 128 MB

When the application’s CPU load increases to 50%, the service load decreases to 4,307 CPM.
CPU load: 100%; memory load: 128 MB

When the CPU usage is at 100%, the service load decreases to only 40% of what it would be if no chaos experiments were taking place.

Because the process scheduling under the Linux system does not allow a process to occupy the CPU all the time, the deployed Spring Boot Demo can still handle 40% of the access requests even in the extreme case of a full CPU load.

Summary

By combining SkyWalking and Chaos Mesh, you can clearly observe when and to what extent chaos experiments affect application service performance. This combination of tools lets you observe the service performance in various extreme conditions, thus boosting your confidence in your services.

Chaos Mesh has grown a lot in 2021 thanks to the unremitting efforts of all PingCAP engineers and community contributors. In order to continue to upgrade our support for our wide variety of users and learn more about users’ experience in Chaos Engineering, we’d like to invite you to take this survey and give us your valuable feedback.

If you want to know more about Chaos Mesh, you’re welcome to join the Chaos Mesh community on GitHub or our Slack discussions (#project-chaos-mesh). If you find any bugs or missing features when using Chaos Mesh, you can submit your pull requests or issues to our GitHub repository.

Blog: How to integrate skywalking-nginx-lua to Nginx?

Mon, 13 Dec 2021 00:00:00 +0000

We Can integrate Skywalking to Java Application by Java Agent TEC.， In typical application, the system runs Java Web applications at the backend of the load balancer, and the most commonly used load balancer is nginx. What should we do if we want to bring it under surveillance? Fortunately, skywalking has provided Nginx agent。 During the integration process, it is found that the examples on the official website only support openresty. For openresty, common modules such as luajit and Lua nginx module have been integrated. Adding skywalking related configurations according to the examples on the official website can take effect. However, when configured for nginx startup, many errors will be reported. We may not want to change a load balancer (nginx to openresty) in order to use skywalking. Therefore, we must solve the integration problem between skywalking and nginx.

Note: openresty is a high-performance web development platform based on nginx + Lua, which solves the short board that is not easy to program in nginx.

Based on Skywalking-8.7.0 and Nginx-1.20.1

Upgrade of nginx:

The agent plug-in of nginx is written based on Lua, so nginx needs to add support for Lua, Lua nginx module It just provides this function. The Lua nginx module depends on luajit Therefore, first we need to install luajit. In the environment, it is best to choose version 2.1.

For nginx, you need to compile the necessary modules yourself. It depends on the following two modules:

lua-nginx-module The version is lua-nginx-module-0.10.21rc1

ngx_devel_kit The version using ngx_devel_kit-0.3.1

Compile nginx parameters


configure arguments: --add-module=/path/to/ngx_devel_kit-0.3.1 --add-module=/path/to/lua-nginx-module-0.10.21rc1 --with-ld-opt=-Wl,-rpath,/usr/local/LuaJIT/lib

The following is for skywalking-nginx-lua-0.3.0 and 0.3.0+ are described separately.

skywalking-nginx-lua-0.3.0

After testing, skywalking-nginx-lua-0.3.0 requires the following Lua related modules


lua-resty-core https://github.com/openresty/lua-resty-core

lua-resty-lrucache https://github.com/openresty/lua-resty-lrucache

lua-cjson https://github.com/openresty/lua-cjson

The dependent Lua modules are as follows:


lua_package_path "/path/to/lua-resty-core/lua-resty-core-master/lib/?.lua;/path/to/lua-resty-lrucache-0.11/lib/?.lua;/path/to/skywalking-nginx-lua-0.3.0/lib/?.lua;;";

In the process of make & & make install, Lua cjson needs to pay attention to:

Modify a path in makefile

LUA_INCLUDE_DIR ?= /usr/local/LuaJIT/include/luajit-2.0

Reference: https://blog.csdn.net/ymeputer/article/details/50146143

skywalking-nginx-lua-0.3.0+

For skywalking-nginx-lua-0.3.0+, tablepool support needs to be added, but it seems that cjson is not required


lua-resty-core https://github.com/openresty/lua-resty-core

lua-resty-lrucache https://github.com/openresty/lua-resty-lrucache

lua-tablepool https://github.com/openresty/lua-tablepool


lua_ package_ path "/path/to/lua-resty-core/lua-resty-core-master/lib/?.lua;/path/to/lua-resty-lrucache-0.11/lib/?.lua;/path/to/lua-tablepool-master/lib/?.lua;/path/to/skywalking-nginx-lua-master/lib/?.lua;;";

tablepool introduces two APIs according to its official documents table new and table. Clear requires luajit2.1, there is a paragraph in the skywalking-nginx-lua document that says you can use ‘require (“skywalking. Util”) disable_ Tablepool() ` disable tablepool

When you start nginx, you will be prompted to install openresty’s own [luajit version]（ https://github.com/openresty/luajit2 )


detected a LuaJIT version which is not OpenResty's; many optimizations will be disabled and performance will be compromised (see https://github.com/openresty/luajit2 for OpenResty's LuaJIT or, even better, consider using the OpenResty releases from https://openresty.org/en/download.html )

here is successful configuration:

     http {
    lua_package_path "/path/to/lua-resty-core/lua-resty-core-master/lib/?.lua;/path/to/lua-resty-lrucache-0.11/lib/?.lua;/path/to/lua-tablepool-master/lib/?.lua;/path/to/skywalking-nginx-lua-master/lib/?.lua;;";

    # Buffer represents the register inform and the queue of the finished segment
    lua_shared_dict tracing_buffer 100m;

    # Init is the timer setter and keeper
    # Setup an infinite loop timer to do register and trace report.
    init_worker_by_lua_block {
        local metadata_buffer = ngx.shared.tracing_buffer

        -- Set service name
        metadata_buffer:set('serviceName', 'User Service Name')
        -- Instance means the number of Nginx deployment, does not mean the worker instances
        metadata_buffer:set('serviceInstanceName', 'User Service Instance Name')
        -- type 'boolean', mark the entrySpan include host/domain
        metadata_buffer:set('includeHostInEntrySpan', false)

        -- set random seed
        require("skywalking.util").set_randomseed()
        require("skywalking.client"):startBackendTimer("http://127.0.0.1:12800")

        -- If there is a bug of this `tablepool` implementation, we can
        -- disable it in this way
        -- require("skywalking.util").disable_tablepool()

        skywalking_tracer = require("skywalking.tracer")
    }

    server {
        listen 8090;
        
        location /ingress {
            default_type text/html;

            rewrite_by_lua_block {
                ------------------------------------------------------
                -- NOTICE, this should be changed manually
                -- This variable represents the upstream logic address
                -- Please set them as service logic name or DNS name
                --
                -- Currently, we can not have the upstream real network address
                ------------------------------------------------------
                skywalking_tracer:start("upstream service")
                -- If you want correlation custom data to the downstream service
                -- skywalking_tracer:start("upstream service", {custom = "custom_value"})
            }

            -- Target upstream service
            proxy_pass http://127.0.0.1:8080/backend;

            body_filter_by_lua_block {
                if ngx.arg[2] then
                    skywalking_tracer:finish()
                end
            }

            log_by_lua_block {
                skywalking_tracer:prepareForReport()
            }
        }
    }
}

Original post：https://www.cnblogs.com/kebibuluan/p/14440228.html

Blog: Apache APISIX Integrates with SkyWalking to Create a Full Range of Log Processing

Wed, 08 Dec 2021 00:00:00 +0000

In the field of observability, the three main directions of data collection and analysis, Metrics, Logger and Tracing, are usually used to achieve insight into the operational status of applications.

Apache APISIX has integrated Apache SkyWaling Tracing capabilities as early as version 1.4, with features such as error logging and access log collection added in subsequent versions. Now with Apache SkyWalking’s support for Metrics, it enables Apache APISIX to implement a one-stop observable solution in integrated mode, covering both logging, metrics and call tracing.

Feature Development Background

Those of you who are familiar with Apache APISIX should know that Apache APISIX produces two types of logs during operation, namely the access log and the error log.

Access logs record detailed information about each request and are logs generated within the scope of the request, so they can be directly associated with Tracing. Error logs, on the other hand, are Apache APISIX runtime output log messages, which are application-wide logs, but cannot be 100% associated with requests.

At present, Apache APISIX provides very rich log processing plug-ins, including TCP/HTTP/Kafka and other collection and reporting plug-ins, but they are weakly associated with Tracing. Take Apache SkyWalking as an example. We extract the SkyWalking Tracing Conetxt Header from the log records of Apache APISIX and export it to the file system, and then use the log processing framework (fluentbit) to convert the logs into a log format acceptable to SkyWalking. The Tracing Context is then parsed and extracted to obtain the Tracing ID to establish a connection with the Trace.

Obviously, the above way of handling the process is tedious and complicated, and requires additional conversion of log formats. For this reason, in PR#5500 we have implemented the Apache SkyWalking access log into the Apache APISIX plug-in ecosystem to make it easier for users to collect and process logs using Apache SkyWalking in Apache APISIX.

Introduction of the New Plugins

SkyWalking Logger Pulgin

The SkyWalking Logger plugin parses the SkyWalking Tracing Context Header and prints the relevant Tracing Context information to the log, thus enabling the log to be associated with the call chain.

By using this plug-in, Apache APISIX can get the SkyWalking Tracing Context and associate it with Tracing even if the SkyWalking Tracing plug-in is not turned on, if Apache SkyWalking is already integrated downstream.

The above Content is the log content, where the Apache APISIX metadata configuration is used to collect request-related information. You can later modify the Log Format to customize the log content by Plugin Metadata, please refer to the official documentation.

How to Use

When using this plugin, since the SkyWalking plugin is “not enabled” by default, you need to manually modify the plugins section in the conf/default-apisix.yaml file to enable the plugin.

plugins:
  ...
  - error-log-logger
  ...

Then you can use the SkyWalking Tracing plug-in to get the tracing data directly, so you can verify that the Logging plug-in-related features are enabled and working properly.

Step 1: Create a route

Next, create a route and bind the SkyWalking Tracing plugin and the SkyWalking Logging plugin. More details of the plugin configuration can be found in the official Apache APISIX documentation.

curl -X PUT 'http://192.168.0.108:9080/apisix/admin/routes/1001' \
-H 'X-API-KEY:  edd1c9f034335f136f87ad84b625c8f1' \
-H 'Content-Type: application/json' \
-d '{
    "uri": "/get",
    "plugins": {
        "skywalking": {
            "sample_ratio": 1
        },
        "skywalking-logger": {
            "endpoint_addr": "http://127.0.0.1:12800"
        }
    },
    "upstream": {
        "type": "roundrobin",
        "nodes": {
            "httpbin.org:80": 1
        }
    }
}'

Step 2: Log Processing

On the Apache SkyWalking side, you can use LAL (Logger Analysis Language) scripts for log processing, such as Tag extraction, SkyWalking metadata correction, and so on.

The main purpose of Tag extraction here is to facilitate subsequent retrieval and to add dependencies to the Metrics statistics. The following code can be used to configure the SkyWalking LAL script to complete the Tag extraction. For more information on how to use the SkyWalking LAL script, please refer to the official Apache SkyWalking documentation.

# The default LAL script to save all logs, behaving like the versions before 8.5.0.
rules:
  - name: default
    dsl: |
      filter {
        json {
          abortOnFailure false
        }

        extractor {
          tag routeId: parsed.route_id
          tag upstream: parsed.upstream
          tag clientIp: parsed.client_ip
          tag latency: parsed.latency
        }

        sink {
        }
      }

After configuring the above LAL script in SkyWalking OAP Server the following log will be displayed.

Details of the expanded log are as follows.

As you can see from the above, displaying routeId, upstream and clientIp as key-value pairs is much easier than searching directly in the log body. This is because the Tag format not only supports log display format and search, but also generates information such as Metrics using MAL statistics.

SkyWalking Error Logger Plugin

The error-log-logger plug-in now supports the SkyWalking log format, and you can now use the http-error-log plug-in to quickly connect Apache APISIX error logs to Apache SkyWalking. Currently, error logs do not have access to SkyWalking Tracing Context information, and therefore cannot be directly associated with SkyWalking Tracing.

The main reason for the error log to be integrated into SkyWalking is to centralize the Apache APISIX log data and to make it easier to view all observable data within SkyWalking.

How to Use

Since the error-log-logger plugin is “not enabled” by default, you still need to enable the plugin in the way mentioned above.

plugins:
  ...
  - error-log-logger
  ...

Step 1: Bind the route

After enabling, you need to bind the plugin to routes or global rules. Here we take “bind routes” as an example.

curl -X PUT 'http://192.168.0.108:9080/apisix/admin/plugin_metadata/error-log-logger' \
-H 'X-API-KEY:  edd1c9f034335f136f87ad84b625c8f1' \
-H 'Content-Type: application/json' \
-d '{
    "inactive_timeout": 10,
    "level": "ERROR",
    "skywalking": {
        "endpoint_addr": "http://127.0.0.1:12800/v3/logs"
    }
}'

Note that the endpoint_addr is the SkyWalking OAP Server address and needs to have the URI (i.e. /v3/logs).

Step 2: LAL Processing

In much the same way as the Access Log processing, the logs are also processed by LAL when they reach SkyWalking OAP Server. Therefore, we can still use the SkyWalking LAL script to analyze and process the log messages.

It is important to note that the Error Log message body is in text format. If you are extracting tags, you will need to use regular expressions to do this. Unlike Access Log, which handles the message body in a slightly different way, Acces Log uses JSON format and can directly reference the fields of the JSON object using JSON parsing, but the rest of the process is largely the same.

Tags can also be used to optimize the display and retrieval for subsequent metrics calculations using SkyWalking MAL.

rules:
  - name: apisix-errlog
    dsl: |
      filter {
        text {
          regexp "(?<datetime>\\d{4}/\\d{2}/\\d{2} \\d{2}:\\d{2}:\\d{2}) \\[(?<level>\\w+)\\] \\d+\\#\\d+:( \\*\\d+ \\[(?<module>\\w+)\\] (?<position>.*\\.lua:\\d+): (?<function>\\w+\\(\\)):)* (?<msg>.+)"
        }
        extractor {
          tag level: parsed.level
          if (parsed?.module) {
            tag module: parsed.module
            tag position: parsed.position
            tag function: parsed.function
          }
        }
        sink {
        }
      }

After the LAL script used by SkyWalking OAP Server, some of the Tags will be extracted from the logs, as shown below.

Summary

This article introduces two logging plug-ins for Apache APISIX that integrate with SkyWalking to provide a more convenient operation and environment for logging in Apache APISIX afterwards.

We hope that through this article, you will have a fuller understanding of the new features and be able to use Apache APISIX for centralized management of observable data more conveniently in the future.

Blog: The Application Guide of Apache IoTDB Storage Option

Wed, 08 Dec 2021 00:00:00 +0000

This document is one of the outcomes of Apache IoTDB - Apache SkyWalking Adapter in Summer 2021 of Open Source Promotion Plan. The design and development work is under the guidance of @jixuan1989 from IoTDB and @wu-sheng from SkyWalking. Thanks for their guidance and the help from community.

Start with SkyWalking Showcase

Before using SkyWalking Showcase to quick start with IoTDB, please ensure your have make installed and Docker daemon running.

Please run the command below.

git clone https://github.com/LIU-WEI-git/skywalking-showcase.git
cd skywalking-showcase
make deploy.docker FEATURE_FLAGS=single-node.iotdb,agent

The former variable single-node.iotdb will deploy only one single node of SkyWalking OAP-v8.9.0, and SkyWalking RocketBot UI-v8.9.0, IoTDB-v0.12.3 as storage. The latter variable agent will deploy micro-services with SkyWalking agent enabled, which include agents for Java, NodeJS server, browser, Python.

These shell command maybe take a long while. After pulling and running docker image, please visit http://localhost:9999/. Then you will see the SkyWalking UI and data from OAP backend.

If you want to use more functions of SkyWalking Showcase, please visit its official document and clone official repository.

Start Manually

If you want to download and run IoTDB and SkyWalking manually, here is the guidance.

Install and Run IoTDB

Apache IoTDB (Database for Internet of Things) is an IoT native database with high performance for data management and analysis, deployable on the edge and the cloud. It is a time-series database storage option for SkyWalking now. Please ensure your IoTDB server version >= 0.12.3 and a single node version is sufficient. For more installation details, please see official document: IoTDB Quick Start and IoTDB Download Page. You could download it from Docker Hub as well.

There is some connection tools for IoTDB

Command Line Interface(CLI)
If iotdb-cli connects successfully, you will see

 _____       _________  ______   ______
|_   _|     |  _   _  ||_   _ `.|_   _ \
  | |   .--.|_/ | | \_|  | | `. \ | |_) |
  | | / .'`\ \  | |      | |  | | |  __'.
 _| |_| \__. | _| |_    _| |_.' /_| |__) |
|_____|'.__.' |_____|  |______.'|_______/  version x.x.x


IoTDB> login successfully
IoTDB>

IoTDB-Grafana
IoTDB-Grafana is a connector which we developed to show time series data in IoTDB by reading data from IoTDB and sends to Grafana.
Zeppelin-IoTDB
You could enable Zeppelin to operate IoTDB via SQL.

For more ecosystem integration, please visit official documents.

We will use iotdb-cli in the next examples.

Run SkyWalking OAP Server

There are some SkyWalking official documents which will help you start. Please ensure your SkyWalking version >= 8.9.0. We recommend you download SkyWalking OAP distributions from its official download page or pull docker images.

Before starting SkyWalking backend, please edit /config/application.yml, set storage.selector: ${SW_STORAGE:iotdb} or set environment variable SW_STORAGE=iotdb. All config options about IoTDB is following, please edit it or not according to your local environment:

storage:
  selector: ${SW_STORAGE:iotdb}
  iotdb:
    host: ${SW_STORAGE_IOTDB_HOST:127.0.0.1}
    rpcPort: ${SW_STORAGE_IOTDB_RPC_PORT:6667}
    username: ${SW_STORAGE_IOTDB_USERNAME:root}
    password: ${SW_STORAGE_IOTDB_PASSWORD:root}
    storageGroup: ${SW_STORAGE_IOTDB_STORAGE_GROUP:root.skywalking}
    sessionPoolSize: ${SW_STORAGE_IOTDB_SESSIONPOOL_SIZE:16}
    fetchTaskLogMaxSize: ${SW_STORAGE_IOTDB_FETCH_TASK_LOG_MAX_SIZE:1000} # the max number of fetch task log in a request

Visit IoTDB Server and Query SkyWalking Data

There are some official document about data model and IoTDB-SQL language:

Example Model and Insert SQL

Before giving any example, we set time display type as long (CLI: set time_display_type=long).

In our design, we choose id, entity_id, node_type, service_id, service_group, trace_id as indexes and fix their appearance order. The value of these indexed fields store in the path with double quotation mark wrapping, just like "value".

There is a model named service_traffic with fields id, time_bucket, name, node_type, service_group. In order to see its data, we could use a query SQL: select * from root.skywalking.service_traffic align by device. root.skywalking is the default storage group and align by device could return a more friendly result. The query result is following:

Time	Device	name
1637919540000	root.skywalking.service_traffic.“YXBwbGljYXRpb24tZGVtbw==.1”.“0”.””	application-demo
1637919600000	root.skywalking.service_traffic.“YXBwbGljYXRpb24tZGVtby1teXNxbA==.1”.“0”.””	application-demo-mysql

Another example model is service_cpm which has fields id, service_id, total, value. Query its data with select * from root.skywalking.service_cpm align by device. The result is following:

Time	Device	total	value
1637919540000	root.skywalking.service_cpm.“202111261739_YXBwbGljYXRpb24tZGVtbw==.1”.“YXBwbGljYXRpb24tZGVtbw==.1”	2	2
1637919600000	root.skywalking.service_cpm.“202111261740_YXBwbGljYXRpb24tZGVtby1teXNxbA==.1”.“YXBwbGljYXRpb24tZGVtby1teXNxbA==.1”	1	1
1637917200000	root.skywalking.service_cpm.“2021112617_YXBwbGljYXRpb24tZGVtbw==.1”.“YXBwbGljYXRpb24tZGVtbw==.1”	2	0

For the first data of service_traffic, the mapping between fields and values is following. Notice, all time_bucket are converted to timestamp(also named time in IoTDB) and the value of all indexed fields are stored in the Device path.

Field	Value
id(indexed)	YXBwbGljYXRpb24tZGVtbw==.1
time(converted from time_bucket)	1637919540000
name	application-demo
node_type(indexed)	0
service_group(indexed)	(empty string)

You could use the SQL below to insert example data.

create storage group root.skywalking
insert into root.skywalking.service_traffic."YXBwbGljYXRpb24tZGVtbw==.1"."0".""(timestamp, name) values(1637919540000, "application-demo")
insert into root.skywalking.service_traffic."YXBwbGljYXRpb24tZGVtby1teXNxbA==.1"."0".""(timestamp, name) values(1637919600000, "application-demo-mysql")
insert into root.skywalking.service_cpm."202111261739_YXBwbGljYXRpb24tZGVtbw==.1"."YXBwbGljYXRpb24tZGVtbw==.1"(timestamp, total, value) values(1637919540000, 2, 2)
insert into root.skywalking.service_cpm."202111261740_YXBwbGljYXRpb24tZGVtby1teXNxbA==.1"."YXBwbGljYXRpb24tZGVtby1teXNxbA==.1"(timestamp, total, value) values(1637919600000, 1, 1)
insert into root.skywalking.service_cpm."2021112617_YXBwbGljYXRpb24tZGVtbw==.1"."YXBwbGljYXRpb24tZGVtbw==.1"(timestamp, total, value) values(1637917200000, 2, 0)

Query SQL

Now, let’s show some query examples.

Filter Query
- If you want to query name field of service_traffic, the query SQL is select name from root.skywalking.service_traffic align by device.
- If you want to query service_traffic with id = YXBwbGljYXRpb24tZGVtbw==.1, the query SQL is select * from root.skywalking.service_traffic."YXBwbGljYXRpb24tZGVtbw==.1" align by device.
- If you want to query service_traffic with name = application-demo, the query SQL is select * from root.skywalking.service_traffic where name = "application-demo" align by device.
- Combining the above three, the query SQL is select name from root.skywalking.service_traffic."YXBwbGljYXRpb24tZGVtbw==.1" where name = "application-demo" align by device.
Fuzzy Query
- If you want to query service_traffic with name contains application, the query SQL is select * from root.skywalking.service_traffic.*.*.* where name like '%application%' align by device.
Aggregate Query
IoTDB only supports group by time and group by level. The former please refer to Down-Frequency Aggregate Query and the latter please refer to Aggregation By Level. Here is an example about group by level: select sum(total) from root.skywalking.service_cpm.*.* group by level = 3. We couldn’t get a expected result since our design make the data of one model spread across multiple devices. So we don’t recommend using group by level to query SkyWalking backend data. You could refer to the Discussion #3907 in IoTDB community for more details.
Sort Query
IoTDB only supports order by time, but we could use its select function which contains top_k and bottom_k to get top/bottom k data. For example, select top_k(total, "k"="3") from root.skywalking.service_cpm.*.*. We don’t recommend using this to query SkyWalking backend data since its result is not friendly. You could refer to the Discussion #3888 in IoTDB community for more details.
Pagination Query
We could use limit and offset to paginate the query result. Please refer to Row and Column Control over Query Results.
Delete
- Delete storage group:
  - delete storage group root.skywalking
- Delete timeseries:
  - delete timeseries root.skywalking.service_cpm.*.*.total
  - delete timeseries root.skywalking.service_cpm."202111261739_YXBwbGljYXRpb24tZGVtbw==.1"."YXBwbGljYXRpb24tZGVtbw==.1".total
- Delete data:
  - delete from root.skywalking.service_traffic
  - delete from root.skywalking.service_traffic where time < 1637919540000

Blog: Extending Apache SkyWalking with non-breaking breakpoints

Mon, 06 Dec 2021 00:00:00 +0000

Non-breaking breakpoints are breakpoints specifically designed for live production environments. With non-breaking breakpoints, reproducing production bugs locally or in staging is conveniently replaced with capturing them directly in production.

Like regular breakpoints, non-breaking breakpoints can be:

placed almost anywhere
added and removed at will
set to fire on specific conditions
expose internal application state
persist as long as desired (even between application reboots)

The last feature is especially useful given non-breaking breakpoints can be left in production for days, weeks, and even months at a time while waiting to capture behavior that happens rarely and unpredictably.

How do non-breaking breakpoints work?

If you’re familiar with general distributed tracing concepts, such as “traces” and “spans”, then you’re already broadly familiar with how non-breaking breakpoints work. Put simply, non-breaking breakpoints are small fragments of code added during runtime that, upon the proper conditions, save a portion of the application’s current state, and resume normal execution. In SkyWalking, this can be implemented by simply opening a new local span, adding some tags, and closing the local span.

While this process is relatively simple, the range of functionality that can be achieved through this technique is quite impressive. Save the current and global variables to create a non-breaking breakpoint; add the ability to format log messages to create just-in-time logging; add the ability to trigger metric telemetry to create real-time KPI monitoring. If you keep moving in this direction, you eventually enter the realm of live debugging/coding, and this is where Source++ comes in.

Live Coding Platform

Source++ is an open-source live coding platform designed for production environments, powered by Apache SkyWalking. Using Source++, developers can add breakpoints, logs, metrics, and distributed tracing to live production software in real-time on-demand, right from their IDE or CLI. While capable of stand-alone deployment, the latest version of Source++ makes it easier than ever to integrate into existing Apache SkyWalking installations. This process can be completed in a few minutes and is easy to customize for your specific needs.

For a better idea of how Source++ works, take a look at the following diagram:

In this diagram, blue components represent existing SkyWalking architecture, black components represent new Source++ architecture, and the red arrows show how non-breaking breakpoints make their way from production to IDEs. A process that is facilitated by Source++ components: Live Probe, Live Processors, Live Platform, and Live Interface.

Live Probe

The Live Probe is currently available for JVM and Python applications. It runs alongside the SkyWalking agent and is responsible for dynamically adding and removing code fragments based on valid instrumentation requests from developers. These code fragments in turn make use of the SkyWalking agent’s internal APIs to facilitate production instrumentation.

Live Processors

Live Processors are responsible for finding, extracting, and transforming data found in distributed traces produced via live probes. They run alongside SkyWalking collectors and implement additional post-processing logic, such as PII redaction. Live processors work via uniquely identifiable tags (prefix spp.) added previously by live probes.

One could easily view a non-breaking breakpoint ready for processing using Rocketbot, however, it will look like this:

Even though the above does not resemble what’s normally thought of as a breakpoint, the necessary information is there. With live processors added to your SkyWalking installation, this data is refined and may be viewed more traditionally via live interfaces.

Live Platform

The Live Platform is the core part of the Source++ architecture. Unlike the live probe and processors, the live platform does not have a direct correlation with SkyWalking components. It is a standalone server responsible for validating and distributing production breakpoints, logs, metrics, and traces. Each component of the Source++ architecture (probes, processors, interfaces) communicates with each other through the live platform. It is important to ensure the live platform is accessible to all of these components.

Live Interface

Finally, with all the previous parts installed, we’re now at the component software developers will find the most useful. A Live Interface is what developers use to create, manage, and view non-breaking breakpoints, and so on. There are a few live interfaces available:

With the Live Instrument Processor enabled, and the JetBrains Plugin installed, non-breaking breakpoints appear as such:

The above should be a sight far more familiar to software developers. Beyond the fact that you can’t step through execution, non-breaking breakpoints look and feel just like regular breakpoints.

For more details and complete setup instructions, please visit:

https://github.com/sourceplusplus/deploy-skywalking

Blog: The Design of Apache IoTDB Storage Option

Tue, 23 Nov 2021 00:00:00 +0000

This plugin is one of the outcomes of Apache IoTDB - Apache SkyWalking Adapter in Summer 2021 of Open Source Promotion Plan. The design and development work is under the guidance of @jixuan1989 from IoTDB and @wu-sheng from SkyWalking. Thanks for their guidance and the help from community.

IoTDB Storage Plugin Setup

IoTDB is a time-series database from Apache, which is one of the storage plugin options. If you want to use iotdb as SkyWalking backend storage, please refer to the following configuration.
IoTDB storage plugin is still in progress. Its efficiency will improve in the future.

storage:
  selector: ${SW_STORAGE:iotdb}
  iotdb:
    host: ${SW_STORAGE_IOTDB_HOST:127.0.0.1}
    rpcPort: ${SW_STORAGE_IOTDB_RPC_PORT:6667}
    username: ${SW_STORAGE_IOTDB_USERNAME:root}
    password: ${SW_STORAGE_IOTDB_PASSWORD:root}
    storageGroup: ${SW_STORAGE_IOTDB_STORAGE_GROUP:root.skywalking}
    sessionPoolSize: ${SW_STORAGE_IOTDB_SESSIONPOOL_SIZE:16}
    fetchTaskLogMaxSize: ${SW_STORAGE_IOTDB_FETCH_TASK_LOG_MAX_SIZE:1000} # the max number of fetch task log in a request

All connection related settings, including host, rpcPort, username, and password are found in application.yml. Please ensure the IoTDB version >= 0.12.3.

IoTDB Introduction

The Data Model of IoTDB

We can use the tree structure to understand the data model of iotdb. If divided according to layers, from high to low is: Storage Group – (LayerName) – Device – Measurement. From the top layer to a certain layer below it is called a Path. The top layer is Storage Group (must start with root), the penultimate layer is Device, and the bottom layer is Measurement. There can be many layers in the middle, and each layer is called a LayerName. For more information, please refer to the Data Model and Terminology in the official document of the version 0.12.x.

The Design of IoTDB Storage Plugin

The Data Model of SkyWalking

Each storage model of SkyWalking can be considered as a Model, which contains multiple Columns. Each Column has ColumnName and ColumnType attributes, representing the name and type of Column respectively. Each Column named ColumnName stores multiple Value of the ColumnType. From a relational database perspective, Model is a relational table and Column is the field in a relational table.

Schema Design

Since each LayerName of IoTDB is stored in memory, it can be considered as an index, and this feature can be fully utilized to improve IoTDB query performance. The default storage group is root.skywalking, it will occupy the first and the second layer of the path. The model name is stored at the next layer of the storage group (the third layer of the path), such as root.skywalking.model_name.

SkyWalking has its own index requirement, but it isn’t applicable to IoTDB. Considering query frequency and referring to the implementation of the other storage options, we choose id, entity_id, node_type, service_id, service_group, trace_id as indexes and fix their appearance order in the path. The value of these indexed columns will occupy the last few layers of the path. If we don’t fix their order, we cannot map their value to column, since we only store their value in the path but don’t store their column name. The other columns are treated as Measurements.

The mapping from SkyWalking data model to IoTDB data model is below.

SkyWalking	IoTDB
Database	Storage Group (1st and 2nd layer of the path)
Model	LayerName (3rd layer of the path)
Indexed Column	stored in memory through hard-code
Indexed Column Value	LayerName (after 3rd layer of the path)
Non-indexed Column	Measurement
Non-indexed Value	the value of Measurement

For general example

There are model1(column11, column12), model2(column21, column22, column23), model3(column31). Underline indicates that the column requires to be indexed. In this example, modelx_name refers to the name of modelx, columnx_name refers to the name of columnx and columnx_value refers to the value of columnx.

Before these 3 model storage schema, here are some points we need to know.

In order to avoid the value of indexed column contains dot(.), all of them should be wrapped in double quotation mark since IoTDB use dot(.) as the separator in the path.
We use align by device in query SQL to get a more friendly result. For more information about align by device, please see DML (Data Manipulation Language) and Query by device alignment.

The path of them is following:

The Model with index:
- root.skywalking.model1_name.column11_value.column12_name
- root.skywalking.model2_name.column21_value.column22_value.column23_name
The Model without index:
- root.skywalking.model3_name.column31_Name

Use select * from root.skywalking.modelx_name align by device respectively to get their schema and data. The SQL result is following:

Time	Device	column12_name
1637494020000	root.skywalking.model1_name.“column11_value”	column12_value

Time	Device	column23_name
1637494020000	root.skywalking.model2_name.“column21_value”.“column22_value”	column23_value

Time	Device	column31_name
1637494020000	root.skywalking.model3_name	column31_value

For specific example

Before 5 typical examples, here are some points we need to know.

The indexed columns and their order: id, entity_id, node_type, service_id, service_group, trace_id. Other columns are treated as non indexed and stored as Measurement.
The storage entity extends Metrics or Record contains a column time_bucket. The time_bucket column in SkyWalking Model can be converted to the timestamp of IoTDB when inserting data. We don’t need to store time_bucket separately. In the next examples, we won’t list time_bucket anymore.
The Time in query result corresponds to the timestamp in insert SQL and API.

Metadata: service_traffic
service_traffic entity has 4 columns: id, name, node_type, service_group. When service_traffic entity includes a row with timestamp 1637494020000, the row should be as following: (Notice: the value of service_group is null.)

id	name	node_type	service_group
ZTJlLXNlcnZpY2UtcHJvdmlkZXI=.1	e2e-service-provider	0

And the row stored in IoTDB should be as following: (Query SQL: select from root.skywalking.service_traffic align by device)

Time	Device	name
1637494020000	root.skywalking.service_traffic.“ZTJlLXNlcnZpY2UtcHJvdmlkZXI=.1”.“0”.“null”	e2e-service-provider

The value of id, node_type and service_group are stored in the path in the specified order. Notice: If those index value is null, it will be transformed to a string “null”.

Metrics: service_cpm
service_cpm entity has 4 columns: id, service_id, total, value.
When service_cpm entity includes a row with timestamp 1637494020000, the row should be as following:

id	service_id	total	value
202111211127_ZTJlLXNlcnZpY2UtY29uc3VtZXI=.1	ZTJlLXNlcnZpY2UtY29uc3VtZXI=.1	4	4

And the row stored in IoTDB should be as following: (Query SQL: select from root.skywalking.service_cpm align by device)

Time	Device	total	value
1637494020000	root.skywalking.service_cpm.“202111211127_ZTJlLXNlcnZpY2UtY29uc3VtZXI=.1”.“ZTJlLXNlcnZpY2UtY29uc3VtZXI=.1”	4	4

The value of id and service_id are stored in the path in the specified order.

Trace segment: segment
segment entity has 10 columns at least: id, segment_id, trace_id, service_id, service_instance_id, endpoint_id, start_time, latency, is_error, data_binary. In addition, it could have variable number of tags.
When segment entity includes 2 rows with timestamp 1637494106000 and 1637494134000, these rows should be as following. The db.type and db.instance are two tags. The first data has two tags, and the second data doesn’t have tag.

id	segment_id	trace_id	service_id	service_instance_id	endpoint_id	start_time	latency	is_error	data_binary	db.type	db.instance
id_1	segment_id_1	trace_id_1	service_id_1	service_instance_id_1	endpoint_id_1	1637494106515	1425	0	data_binary_1	sql	testdb
id_2	segment_id_2	trace_id_2	service_id_2	service_instance_id_2	endpoint_id_2	2637494106765	1254	0	data_binary_2

And these row stored in IoTDB should be as following: (Query SQL: select from root.skywalking.segment align by device)

Time	Device	start_time	data_binary	latency	endpoint_id	is_error	service_instance_id	segment_id	“db.type”	“db.instance”
1637494106000	root.skywalking.segment.“id_1”.“service_id_1”.“trace_id_1”	1637494106515	data_binary_1	1425	endpoint_id_1	0	service_instance_id_1	segment_id_1	sql	testdb
1637494106000	root.skywalking.segment.“id_2”.“service_id_2”.“trace_id_2”	1637494106765	data_binary_2	1254	endpoint_id_2	0	service_instance_id_2	segment_id_2	null	null

The value of id, service_id and trace_id are stored in the path in the specified order. Notice: If the measurement contains dot(.), it will be wrapped in double quotation mark since IoTDB doesn’t allow it. In order to align, IoTDB will append null value for those data without tag in some models.

Log
log entity has 12 columns at least: id, unique_id, service_id, service_instance_id, endpoint_id, trace_id, trace_segment_id, span_id, content_type, content, tags_raw_data, timestamp. In addition, it could have variable number of tags. When log entity includes a row with timestamp 1637494052000, the row should be as following and the level is a tag.

id	unique_id	service_id	service_instance_id	endpoint_id	trace_id	trace_segment_id	span_id	content_type	content	tags_raw_data	timestamp	level
id_1	unique_id_1	service_id_1	service_instance_id_1	endpoint_id_1	trace_id_1	trace_segment_id_1	0	1	content_1	tags_raw_data_1	1637494052118	INFO

And the row stored in IoTDB should be as following: (Query SQL: select from root.skywalking.log align by device)

Time	Device	unique_id	content_type	span_id	tags_raw_data	“timestamp”	level	service_instance_id	content	trace_segment_id
1637494052000	root.skywalking.“id_1”.“service_id_1”.“trace_id_1”	unique_id_1	1	0	tags_raw_data_1	1637494052118	INFO	service_instance_id_1	content_1	trace_segment_id_1

The value of id, service_id and trace_id are stored in the path in the specified order. Notice: If the measurement named timestamp, it will be wrapped in double quotation mark since IoTDB doesn’t allow it.

Profiling snapshots: profile_task_segment_snapshot
profile_task_segment_snapshot entity has 6 columns: id, task_id, segment_id, dump_time, sequence, stack_binary. When profile_task_segment_snapshot includes a row with timestamp 1637494131000, the row should be as following.

id	task_id	segment_id	dump_time	sequence	stack_binary
id_1	task_id_1	segment_id_1	1637494131153	0	stack_binary_1

And the row stored in IoTDB should be as following: (Query SQL: select from root.skywalking.profile_task_segment_snapshot align by device)

Time	Device	sequence	dump_time	stack_binary	task_id	segment_id
1637494131000	root.skywalking.profile_task_segment_snapshot.“id_1”	0	1637494131153	stack_binary_1	task_id_1	segment_id_1

The value of id is stored in the path in the specified order.

Query

In this design, part of the data is stored in memory through LayerName, so data from the same Model is spread across multiple devices. Queries often need to cross multiple devices. But in this aspect, IoTDB’s support is not perfect in cross-device aggregation query, sort query and pagination query. In some cases, we have to use a violence method that query all data meets the condition and then aggregate, sort or paginate them. So it might not be efficient. For detailed descriptions, please refer to the Discussion submitted in IoTDB community below.

Discussion:
- 一个有关排序查询的问题（A problem about sort query）#3888
- 一个有关聚合查询的问题（A problem about aggregation query）#3907

Query SQL for the general example above:

-- query all data in model1
select * from root.skywalking.model1_name align by device;
-- query the data in model2 with column22_value="test"
select * from root.skywalking.model2_name.*."test" align by device;
-- query the sum of column23 in model2 and group by column21
select sum(column23) from root.skywalking.model2_name.*.* group by level = 3;

iotdb-cli is a useful tools to connect and visit IoTDB server. More information please refer Command Line Interface(CLI)

Blog: SkyWalking Python Agent Supports Profiling Now

Sun, 12 Sep 2021 00:00:00 +0000

The Java Agent of Apache SkyWalking has supported profiling since v7.0.0, and it enables users to troubleshoot the root cause of performance issues, and now we bring it into Python Agent. In this blog, we will show you how to use it, and we will introduce the mechanism of profiling.

How to use profiling in Python Agent

This feature is released in Python Agent at v0.7.0. It is turned on by default, so you don’t need any extra configuration to use it. You can find the environment variables about it here.

Here are the demo codes of an intentional slow application.

import time

def method1():
    time.sleep(0.02)
    return '1'

def method2():
    time.sleep(0.02)
    return method1()

def method3():
    time.sleep(0.02)
    return method2()

if __name__ == '__main__':
    import socketserver
    from http.server import BaseHTTPRequestHandler

    class SimpleHTTPRequestHandler(BaseHTTPRequestHandler):

        def do_POST(self):
            method3()
            time.sleep(0.5)
            self.send_response(200)
            self.send_header('Content-Type', 'application/json')
            self.end_headers()
            self.wfile.write('{"song": "Despacito", "artist": "Luis Fonsi"}'.encode('ascii'))

    PORT = 19090
    Handler = SimpleHTTPRequestHandler

    with socketserver.TCPServer(("", PORT), Handler) as httpd:
        httpd.serve_forever()

We can start it with SkyWalking Python Agent CLI without changing any application code now, which is also the latest feature of v0.7.0. We just need to add sw-python run before our start command(i.e. sw-python run python3 main.py), to start the application with python agent attached. More information about sw-python can be found there.

Then, we should add a new profile task for the / endpoint from the SkyWalking UI, as shown below.

We can access it by curl -X POST http://localhost:19090/, after that, we can view the result of this profile task on the SkyWalking UI.

The mechanism of profiling

When a request lands on an application with the profile function enabled, the agent begins the profiling automatically if the request’s URI is as required by the profiling task. A new thread is spawned to fetch the thread dump periodically until the end of request.

The agent sends these thread dumps, called ThreadSnapshot, to SkyWalking OAPServer, and the OAPServer analyzes those ThreadSnapshot(s) and gets the final result. It will take a method invocation with the same stack depth and code signature as the same operation, and estimate the execution time of each method from this.

Let’s demonstrate how this analysis works through the following example. Suppose we have such a program below and we profile it at 10ms intervals.

def main():
    methodA()

def methodA():
    methodB()

def methodB():
    methodC()
    methodD()

def methodC():
    time.sleep(0.04)

def methodD():
    time.sleep(0.06)

The agent collects a total of 10 ThreadSnapShot(s) over the entire time period(Diagram A). The first 4 snapshots represent the thread dumps during the execution of function C, and the last 6 snapshots represent the thread dumps during the execution of function D. After the analysis of OAPServer, we can see the result of this profile task on the SkyWalking Rocketbot UI as shown in the right of the diagram. With this result, we can clearly see the function call relationship and the time consumption situation of this program.

Diagram A

You can read more details of profiling theory from this blog.

We hope you enjoy the profile in the Python Agent, and if so, you can give us a star on Python Agent and SkyWalking on GitHub.

Blog: [Webinar] SkyWalking 8.x Introduction

Wed, 04 Aug 2021 00:00:00 +0000

GOUP hosted a webinar, and invited Sheng Wu to introduce Apache SkyWalking. This is a 1.5 hours presentation including the full landscape of Apache SkyWalking 8.x.

Chapter04 Session10 - Apache Skywalking by Sheng Wu

Blog: [Community win] SkyWalking achieved 500 contributors milestone.

Sat, 10 Jul 2021 00:00:00 +0000

SkyWalking is an open source APM (application performance monitor) system, especially designed for microservices, cloud native, and container-based architectures.

From 2020, it has dominated the open source APM market in China, and expanded aggressively in North American, Europe and Asia’s other countries.

With over 6 years (2015-2021) of development, driven by the global open source community, SkyWalking now provides full stack observability covering metrics, tracing and logging, plus event detector, which are built based on various native and ecosystem solutions.

Language agent-based(Java, Dot Net, Golang, PHP, NodeJS, Python, C++, LUA) in-process monitoring, is as powerful as commercial APM vendors’ agents. Mostly auto-instrumentation, and good interactivity.
Service Mesh Observability, working closely with Envoy and Istio teams.
Transparent integration of popular metrics ecosystem. Accept metrics from Prometheus SDK, OpenTelemetry collectors, Zabbix agents, etc.
Log collection with analysis capability from FluentD, Fluent-bit, Filebeat, etc. agents.
Infrastructure monitoring, such as Linux and k8s, is out of the box.

The SkyWalking ecosystem was started by very few people. The community drives the project to cover real scenarios, from tracing to the whole APM field. Even today, more professional open source developers, powered by the vendors behind them, are bringing the project to a different level.

Typically and most attractively, SkyWalking is going to build the first known open source APM specific database in the world, at least providing

Time series-based database engine.
Support traces/logs and metrics in the database core level.
High performance with cluster mode and HPA.
Reasonable resource cost.

We nearly doubled the number of contributors in the last year, from ~300 to over 500. The whole community is very energetic. Here, we want to thank our 47 committers(28 PMC members included), listed here, and over 400 other contributors.

We together built this humongous Apache Top Level project, and proved the stronge competitiveness of an open-source project.

This is a hard-won and impressive achievement. We won’t stop here. The trend is there, the ground is solid. We are going to build the top-level APM system relying on our open-source community.

500 Contributors List

GitHub
1095071913	182148432**	295198088**	394102339**	437376068**
50168383	55846420**	826245622**	844067874	Ahoo-Wang
AirTrioa	AlexanderWert	AlseinX	AngryMills	Ax1an
BFergerson	BZFYS	CalvinKirs	CharlesMaster	ChaunceyLin5152
CommissarXia	Cvimer	DeadLion	Doublemine	Du-fei
ElderJames	EvanLjp	FatihErdem	FeynmanZhou	Fine0830
FingerLiu	FrankyXu	Gallardot	GerryYuan	HackerRookie
HarryFQ	Heguoya	Hen1ng	HendSame	Humbertzhang
IanCao	IluckySi	Indifer	J-Cod3r	JaredTan95
Jargon96	Jijun	JoeKerouac	JohnNiang	Johor03
Jozdortraz	Jtrust	Just-maple	KangZhiDong	LazyLei
LiWenGu	Lin1997	Linda-pan	LiteSun	Liu-XinYuan
MiracleDx	Miss-you	MoGuGuai-hzr	MrYzys	O-ll-O
Patrick0308	QHWG67	Qiliang	QuanjieDeng	RandyAbernethy
RedzRedz	Runrioter	SataQiu	ScienJus	SevenBlue2018
ShaoHans	Shikugawa	SoberChina	SummerOfServenteen	Switch-vov
TJ666	Technoboy-	TerrellChen	TeslaCN	TheRealHaui
TinyAllen	TomMD	ViberW	Videl	WALL-E
WeihanLi	WildWolfBang	WillemJiang	Wooo0	XhangUeiJong
Xlinlin	YczYanchengzhe	Yebemeto	YoungHu	YunaiV
YunfengGao	Z-Beatles	ZS-Oliver	ZhHong	ZhuoSiChen
a198720	a1vin-tian	a526672351	acurtain	adamni135
adermxzs	adriancole**	aeolusheath	agile6v	aix3
aiyanbo	ajanthan	alexkarezin	alonelaval	amogege
amwyyyy	andyliyuze	andyzzl	aoxls	arugal
ascrutae	ascrutae**	augustowebd	aviaviavi	bai-yang
beckhampu	beckjin	beiwangnull	bigflybrother	bootsrc
bostin	brucewu-fly	buxingzhe	buzuotaxuan	bwh12398**
c feng	c1ay	candyleer	carllhw	carlvine500
carrypann	cheenursn	cheetah012	chenbeitang	chenglei**
chengshiwen	chenmudu	chenpengfei	chenvista	chess-equality
chestarss	chidaodezhongsheng	chopin-d	clevertension	clk1st
cngdkxw	cnlangzi	codeglzhang	codelipenghui	coder-yqj
coki230	compilerduck	constanine	coolbeevip	crystaldust
cui-liqiang	cuiweiwei	cutePanda123	cyberdak	cyejing
cyhii	dafu-wu	dagmom	dalekliuhan**	darcydai
dengliming	devkanro	devon-ye	dickens7	dimaaan
dingdongnigetou	dio	divyakumarjain	dmsolr	dominicqi
donbing007	dsc6636926	dvsv2	dzx2018	echooymxq
efekaptan	elk-g	emschu	eoeac	evanljp**
evanxuhe	feelwing1314	fgksgf	fredster33	fuhuo
fulmicoton	fushiqinghuan111	geektcp	geomonlin	ggndnn
gitter-badger	givingwu	glongzh	gnr163	gonedays
grissom-grissom	grissomsh	guodongq	guyukou	gxthrj
gy09535	gzshilu	hailin0	hanahmily	haotian2015
haoyann	hardzhang	harvies	heihaozi	hepyu
heyanlong	hi-sb	honganan	horber	hsoftxl
huangyoje	huliangdream	huohuanhuan	iluckysi	innerpeacez
itsvse	jasper-zsh	jbampton	jialong121	jinlongwang
jjlu521016	jjtyro	jmjoy	jsbxyyx	justeene
juzhiyuan	jy00464346	kaanid	kagaya85	karott
kayleyang	kevinyyyy	kezhenxu94	kikupotter	kilingzhang
killGC	kkl129	klboke	ksewen	kuaikuai
kun-song	kylixs	landonzeng	langke93	langyan1022
langyizhao	lazycathome	leemove	leizhiyuan	libinglong
lijial	lilien1010	limfriend	linkinshi	linliaoy
liqiangz	liu-junchi	liufei**	liuhaoXD	liuhaoyang
liuweiyi**	liuyanggithup	liuzhengyang	liweiv	lixin40**
lizl9**	lkxiaolou	llissery	louis-zhou	lpcy
lpf32	lsyf	lucperkins	lujiajing1126	lunamagic1978
lunchboxav	lxin96**	lxliuxuankb	lytscu	lyzhang1999
mage3k	makefriend8	makingtime	mantuliu	maolie
margauxcabrera	masterxxo	maxiaoguang64	me**	membphis
mestarshine	mgsheng	michaelsembwever	mikkeschiren	ming_flycash**
minquan.chen**	misaya	momo0313	moonming	mrproliu
mrproliu**	muyun12	nacx	neatlife	neeuq
nic-chen	nickwongwong	nikitap492	nileblack	nisiyong
novayoung	oatiz	oflebbe	olzhy	onecloud360
osiriswd	panniyuyu	peng-yongsheng	pengweiqhca	potiuk
probeyang	purgeyao	qijianbo010	qinhang3	qiuyu-d
qjgszzx	qq362220083	qqeasonchen	qxo	ralphgj
raybi-asus	refactor2	remicollet	rlenferink	rootsongjc
rovast	ruibaby	s00373198	scolia	sdanzo
seifeHu	sergicastro	shiluo34	sikelangya	simonlei
sk163	snakorse	songzhendong	songzhian	songzhian**
sonxy	spacewander	stalary	stenio2011	stevehu
stone-wlg	sungitly	surechen	swartz-k	sxzaihua
tangxqa	tanjunchen	tankilo	tanzhen**	taskmgr
tbdpmi	terranhu	terrymanu	tevahp	thanq
thebouv	tianyk	tianyuak	tincopper	tinyu0
tom-pytel	tristaZero	tristan-tsl	trustin	tsuilouis
tuohai666	tzsword-2020	tzy1316106836	vcjmhg	viktoryi
vision-ken	viswaramamoorthy	wallezhang	wang-yeliang	wang_weihan**
wangrzneu	wankai123	wbpcode	web-xiaxia	webb2019
weiqiang-w	weiqiang333	wendal	wengangJi	wenjianzhang
whfjam	whl12345	willseeyou	wilsonwu	wind2008hxy
wingwong-knh	withlin	wl4g	wqr2016	wu-sheng
wuguangkuo	wujun8	wuwen5	wuxingye	x22x22
xbkaishui	xcaspar	xdRight	xiaoweiyu**	xiaoxiangmoe
xiaoy00	xinfeingxia85	xingren23	xinzhuxiansheng	xonze
xuanyu66	xuchangjunjx	xudianyang	yanbw	yanfch
yang-xiaodong	yangxb2010000	yanickxia	yanmaipian	yanmingbi
yantaowu	yaojingguo	yaowenqiang	yazong	ychandu
ycoe	yimeng	yu199195	yuqichou	yushuqiang**
yuyujulin	yxudong	yymoth	zaunist	zaygrzx
zcai2	zeaposs	zhang98722	zhanghao001	zhangjianweibj
zhangkewei	zhangsean	zhangxin**	zhaoyuguang	zhe1926
zhentaoJin	zhongjianno1**	zhousiliang163	zhuCheer	zhyyu
zifeihan	zijin-m	zkscpqm	zoidbergwill	zoumingzm
zouyx	zpf1989	zshit	zxbu	zygfengyuwuzu

Blog: OpenSearch, a new storage option to avoid ElasticSearch's SSPL

Sun, 09 May 2021 00:00:00 +0000

We posted our Response to Elastic 2021 License Change blog 4 months ago. It doesn’t have a big impact in the short term, but because of the incompatibility between SSPL and Apache 2.0, we lost the chance of upgrading the storage server, which concerns the community and our users. So, we have to keep looking for a new option as a replacement.

There was an open source project, Open Distro for Elasticsearch, maintained by the AWS team. It is an Apache 2.0-licensed distribution of Elasticsearch enhanced with enterprise security, alerting, SQL, and more. After Elastic relicensed its projects, we talked with their team, and they have an agenda to take over the community leadship and keep maintaining Elasticsearch, as it was licensed by Apache 2.0. So, they are good to fork and continue.

On April 12th, 2021, AWS announced the new project, OpenSearch, driven by the community, which is initialized from people of AWS, Red Hat, SAP, Capital One, and Logz.io. Read this Introducing OpenSearch blog for more detail.

Once we had this news in public, we begin to plan the process of evaluating and testing OpenSearch as SkyWalking’s storage option. Read our issue.

Today, we are glad to ANNOUNCE, OpenSearch could replace ElastcSearch as the storage, and it is still licensed under Apache 2.0.

This has been merged in the main stream, and you can find it in the dev doc already.

OpenSearch

OpenSearch storage shares the same configurations as Elasticsearch 7. In order to activate Elasticsearch 7 as storage, set storage provider to elasticsearch7. Please download the apache-skywalking-bin-es7.tar.gz if you want to use OpenSearch as storage.

SkyWalking community will keep our eyes on the OpenSearch project, and look forward to their first GA release.

NOTE: we have to add a warning NOTICE to the Elasticsearch storage doc:

NOTICE: Elastic announced through their blog that Elasticsearch will be moving over to a Server Side Public License (SSPL), which is incompatible with Apache License 2.0. This license change is effective from Elasticsearch version 7.11. So please choose the suitable Elasticsearch version according to your usage.

Blog: End-User Tracing in a SkyWalking-Observed Browser

Thu, 25 Mar 2021 00:00:00 +0000

Origin: End-User Tracing in a SkyWalking-Observed Browser - The New Stack

Apache SkyWalking: an APM (application performance monitor) system, especially designed for microservices, cloud native, and container-based (Docker, Kubernetes, Mesos) architectures.

skywalking-client-js: a lightweight client-side JavaScript exception, performance, and tracing library. It provides metrics and error collection to the SkyWalking backend. It also makes the browser the starting point for distributed tracing.

Background

Web application performance affects the retention rate of users. If a page load time is too long, the user will give up. So we need to monitor the web application to understand performance and ensure that servers are stable, available and healthy. SkyWalking is an APM tool and the skywalking-client-js extends its monitoring to include the browser, providing performance metrics and error collection to the SkyWalking backend.

Performance Metrics

The skywalking-client-js uses [window.performance] (https://developer.mozilla.org/en-US/docs/Web/API/Window/performance) for performance data collection. From the MDN doc, the performance interface provides access to performance-related information for the current page. It’s part of the High Resolution Time API, but is enhanced by the Performance Timeline API, the Navigation Timing API, the User Timing API, and the Resource Timing API. In skywalking-client-js, all performance metrics are calculated according to the Navigation Timing API defined in the W3C specification. We can get a PerformanceTiming object describing our page using the window.performance.timing property. The PerformanceTiming interface contains properties that offer performance timing information for various events that occur during the loading and use of the current page.

We can better understand these attributes when we see them together in the figure below from W3C:

The following table contains performance metrics in skywalking-client-js.

Metrics Name	Describe	Calculating Formulae	Note
redirectTime	Page redirection time	redirectEnd - redirectStart	If the current document and the document that is redirected to are not from the same origin, set redirectStart, redirectEnd to 0
ttfbTime	Time to First Byte	responseStart - requestStart	According to Google Development
dnsTime	Time to DNS query	domainLookupEnd - domainLookupStart
tcpTime	Time to TCP link	connectEnd - connectStart
transTime	Time to content transfer	responseEnd - responseStart
sslTime	Time to SSL secure connection	connectEnd - secureConnectionStart	Only supports HTTPS
resTime	Time to resource loading	loadEventStart - domContentLoadedEventEnd	Represents a synchronized load resource in pages
fmpTime	Time to First Meaningful Paint	-	Listen for changes in page elements. Traverse each new element, and calculate the total score of these elements. If the element is visible, the score is 1 * weight; if the element is not visible, the score is 0
domAnalysisTime	Time to DOM analysis	domInteractive - responseEnd
fptTime	First Paint Time	responseEnd - fetchStart
domReadyTime	Time to DOM ready	domContentLoadedEventEnd - fetchStart
loadPageTime	Page full load time	loadEventStart - fetchStart
ttlTime	Time to interact	domInteractive - fetchStart
firstPackTime	Time to first package	responseStart - domainLookupStart

Skywalking-client-js collects those performance metrics and sends them to the OAP (Observability Analysis Platform) server , which aggregates data on the back-end side that is then shown in visualizations on the UI side. Users can optimize the page according to these data.

Exception Metrics

There are five kinds of errors that can be caught in skywalking-client-js:

The resource loading error is captured by window.addeventlistener ('error ', callback, true)
window.onerror catches JS execution errors
window.addEventListener('unhandledrejection', callback) is used to catch the promise errors
the Vue errors are captured by Vue.config.errorHandler
the Ajax errors are captured by addEventListener('error', callback); addEventListener('abort', callback); addEventListener('timeout', callback); in send callback.

The Skywalking-client-js traces error data to the OAP server, finally visualizing data on the UI side. For an error overview of the App, there are several metrics for basic statistics and trends of errors, including the following metrics.

App Error Count, the total number of errors in the selected time period.
App JS Error Rate, the proportion of PV with JS errors in a selected time period to total PV.
All of Apps Error Count, Top N Apps error count ranking.
All of Apps JS Error Rate, Top N Apps JS error rate ranking.
Error Count of Versions in the Selected App, Top N Error Count of Versions in the Selected App ranking.
Error Rate of Versions in the Selected App, Top N JS Error Rate of Versions in the Selected App ranking.
Error Count of the Selected App, Top N Error Count of the Selected App ranking.
Error Rate of the Selected App, Top N JS Error Rate of the Selected App ranking.

For pages, we use several metrics for basic statistics and trends of errors, including the following metrics:

Top Unstable Pages / Error Rate, Top N Error Count pages of the Selected version ranking.
Top Unstable Pages / Error Count, Top N Error Count pages of the Selected version ranking.
Page Error Count Layout, data display of different errors in a period of time.

User Metrics

SkyWalking browser monitoring also provides metrics about how the visitors use the monitored websites, such as PV(page views), UV(unique visitors), top N PV(page views), etc.

In SPAs (single page applications), the page will be refreshed only once. The traditional method only reports PV once after the page loading, but cannot count the PV of each sub-page, and can’t make other types of logs aggregate by sub-page.

SkyWalking browser monitoring provides two processing methods for SPA pages:

Enable SPA automatic parsing. This method is suitable for most single page application scenarios with URL hash as the route. In the initialized configuration item, set enableSPA to true, which will turn on the page’s hashchange event listener (trigger re reporting PV), and use URL hash as the page field in other data reporting.
Manual reporting. This method can be used in all single page application scenarios. This method can be used if the first method is not usable. The following example provides a set page method to manually update the page name when data is reported. When this method is called, the page PV will be re reported by default:

app.on('routeChange', function (to) {
    ClientMonitor.setPerformance({
    collector: 'http://127.0.0.1:8080',
    service: 'browser-app',
    serviceVersion: '1.0.0',
    pagePath: to.path,
    autoTracePerf: true,
    enableSPA: true,
  });
});

Let’s take a look at the result found in the following image. It shows the most popular applications and versions, and the changes of PV over a period of time.

Make the browser the starting point for distributed tracing

SkyWalking browser monitoring intercepts HTTP requests to trace segments and spans. It supports tracking these following modes of HTTP requests: XMLHttpRequest and fetch. It also supports tracking libraries and tools based on XMLHttpRequest and fetch - such as Axios, SuperAgent, OpenApi, and so on.

Let’s see how the SkyWalking browser monitoring intercepts HTTP requests:

After this, use window.addEventListener('xhrReadyStateChange', callback) and set the readyState value tosw8 = xxxx in the request header. At the same time, reporting requests information to the back-end side. Finally, we can view trace data on the trace page. The following graphic is from the trace page:

To see how we listen for fetch requests, let’s see the source code of fetch

As you can see, it creates a promise and a new XMLHttpRequest object. Because the code of the fetch is built into the browser, it must monitor the code execution first. Therefore, when we add listening events, we can’t monitor the code in the fetch. Just after monitoring the code execution, let’s rewrite the fetch:

import { fetch } from 'whatwg-fetch'; window.fetch = fetch;

In this way, we can intercept the fetch request through the above method.

Additional Resources

End-User Tracing in a SkyWalking-Observed Browser.

Blog: SourceMarker: Continuous Feedback for Developers

Tue, 16 Mar 2021 00:00:00 +0000

SourceMarker is an open-source continuous feedback IDE plugin built on top of Apache SkyWalking, a popular open-source APM system with monitoring, tracing, and diagnosing capabilities for distributed software systems. SkyWalking, a truly holistic system, provides the means for automatically producing, storing, and querying software operation metrics. It requires little to no code changes to implement and is lightweight enough to be used in production. By itself, SkyWalking is a formidable force in the realm of continuous monitoring technology.

SourceMarker, leveraging the continuous monitoring functionality provided by SkyWalking, creates continuous feedback technology by automatically linking software operation metrics to source code and displaying feedback directly inside of the IDE. While currently only supporting JetBrains-based IDEs and JVM-based programming languages, SourceMarker may be extended to support any number of programming languages and IDEs. Using SourceMarker, software developers can understand and validate software operation inside of their IDE. Instead of charts that indicate the health of the application, software developers can view the health of individual source code components and interpret software operation metrics from a much more familiar perspective. Such capabilities improve productivity as time spent continuously context switching from development to monitoring would be eliminated.

Logging

The benefits of continuous feedback technology are immediately apparent with the ability to view and search logs directly from source code. Instead of tailing log files or viewing logs through the browser, SourceMarker allows software developers to navigate production logs just as easily as they navigate source code. By using the source code as the primary perspective for navigating logs, SourceMarker allows software developers to view logs specific to any package, class, method, or line directly from the context of the source code which resulted in those logs.

Tracing

Furthermore, continuous feedback technology offers software developers a deeper understanding of software by explicitly tying the implicit software operation to source code. Instead of visualizing software traces as Gantt charts, SourceMarker allows software developers to step through trace stacks while automatically resolving trace tags and logs. With SourceMarker, software developers can navigate production software traces in much the same way one debugs local applications.

Alerting

Most importantly, continuous feedback technology keeps software developers aware of production software operation. Armed with an APM-powered IDE, every software developer can keep track of the behavior of any method, class, package, and even the entire application itself. Moreover, this allows for source code to be the medium through which production bugs are made evident, thereby creating the feasibility of source code with the ability to self-diagnose and convey its own health.

Download SourceMarker

SourceMarker aims to bridge the theoretical and empirical practices of software development through continuous feedback. The goal is to make developing software with empirical data feel natural and intuitive, creating more complete software developers that understand the entire software development cycle.

https://github.com/sourceplusplus/sourcemarker

This project is still early in its development, so if you think of any ways to improve SourceMarker, please let us know.

Blog: Observe VM Service Meshes with Apache SkyWalking and the Envoy Access Log Service

Sun, 21 Feb 2021 00:00:00 +0000

Origin: Observe VM Service Meshes with Apache SkyWalking and the Envoy Access Log Service - The New Stack

Apache SkyWalking: an APM (application performance monitor) system, especially designed for microservices, cloud native, and container-based (Docker, Kubernetes, Mesos) architectures.

Envoy Access Log Service: Access Log Service (ALS) is an Envoy extension that emits detailed access logs of all requests going through Envoy.

Background

In the previous post, we talked about the observability of service mesh under Kubernetes environment, and applied it to the bookinfo application in practice. We also mentioned that, in order to map the IP addresses into services, SkyWalking needs access to the service metadata from a Kubernetes cluster, which is not available for services deployed in virtual machines (VMs). In this post, we will introduce a new analyzer in SkyWalking that leverages Envoy’s metadata exchange mechanism to decouple with Kubernetes. The analyzer is designed to work in Kubernetes environments, VM environments, and hybrid environments. If there are virtual machines in your service mesh, you might want to try out this new analyzer for better observability, which we will demonstrate in this tutorial.

How it works

The mechanism of how the analyzer works is the same as what we discussed in the previous post. What makes VMs different from Kubernetes is that, for VM services, there are no places where we can fetch the metadata to map the IP addresses into services.

The basic idea we present in this article is to carry the metadata along with Envoy’s access logs, which is called metadata-exchange mechanism in Envoy. When Istio pilot-agent starts an Envoy proxy as a sidecar of a service, it collects the metadata of that service from the Kubernetes platform, or a file on the VM where that service is deployed, and injects the metadata into the bootstrap configuration of Envoy. Envoy will carry the metadata transparently when emitting access logs to the SkyWalking receiver.

But how does Envoy compose a piece of a complete access log that involves the client side and server side? When a request goes out from Envoy, a plugin of istio-proxy named “metadata-exchange” injects the metadata into the http headers (with a prefix like x-envoy-downstream-), and the metadata is propagated to the server side. The Envoy sidecar of the server side receives the request and parses the headers into metadata, and puts the metadata into the access log, keyed by wasm.downstream_peer. The server side Envoy also puts its own metadata into the access log keyed by wasm.upstream_peer. Hence the two sides of a single request are completed.

With the metadata-exchange mechanism, we can use the metadata directly without any extra query.

Example

In this tutorial, we will use another demo application Online Boutique that consists of 10+ services so that we can deploy some of them in VMs and make them communicate with other services deployed in Kubernetes.

Topology of Online Boutique In order to cover as many cases as possible, we will deploy CheckoutService and PaymentService on VM and all the other services on Kubernetes, so that we can cover the cases like Kubernetes → VM (e.g. Frontend → CheckoutService), VM → Kubernetes (e.g. CheckoutService → ShippingService), and VM → VM ( e.g. CheckoutService → PaymentService).

NOTE: All the commands used in this tutorial are accessible on GitHub.

git clone https://github.com/SkyAPMTest/sw-als-vm-demo-scripts
cd sw-als-vm-demo-scripts

Make sure to init the gcloud SDK properly before moving on. Modify the GCP_PROJECT in file env.sh to your own project name. Most of the other variables should be OK to work if you keep them intact. If you would like to use ISTIO_VERSION >/= 1.8.0, please make sure this patch is included.

Prepare Kubernetes cluster and VM instances 00-create-cluster-and-vms.sh creates a new GKE cluster and 2 VM instances that will be used through the entire tutorial, and sets up some necessary firewall rules for them to communicate with each other.
Install Istio and SkyWalking 01a-install-istio.sh installs Istio Operator with spec resources/vmintegration.yaml. In the YAML file, we enable the meshExpansion that supports VM in mesh. We also enable the Envoy access log service and specify the address skywalking-oap.istio-system.svc.cluster.local:11800 to which Envoy emits the access logs. 01b-install-skywalking.sh installs Apache SkyWalking and sets the analyzer to mx-mesh.
Create files to initialize the VM 02-create-files-to-transfer-to-vm.sh creates necessary files that will be used to initialize the VMs. 03-copy-work-files-to-vm.sh securely transfers the generated files to the VMs with gcloud scp command. Now use ./ssh.sh checkoutservice and ./ssh.sh paymentservice to log into the two VMs respectively, and cd to the ~/work directory, execute ./prep-checkoutservice.sh on checkoutservice VM instance and ./prep-paymentservice.sh on paymentservice VM instance. The Istio sidecar should be installed and started properly. To verify that, use tail -f /var/logs/istio/istio.log to check the Istio logs. The output should be something like:
```
2020-12-12T08:07:07.348329Z	info	sds	resource:default new connection
2020-12-12T08:07:07.348401Z	info	sds	Skipping waiting for gateway secret
2020-12-12T08:07:07.348401Z	info	sds	Skipping waiting for gateway secret
2020-12-12T08:07:07.568676Z	info	cache	Root cert has changed, start rotating root cert for SDS clients
2020-12-12T08:07:07.568718Z	info	cache	GenerateSecret default
2020-12-12T08:07:07.569398Z	info	sds	resource:default pushed key/cert pair to proxy
2020-12-12T08:07:07.949156Z	info	cache	Loaded root cert from certificate ROOTCA
2020-12-12T08:07:07.949348Z	info	sds	resource:ROOTCA pushed root cert to proxy
2020-12-12T20:12:07.384782Z	info	sds	resource:default pushed key/cert pair to proxy
2020-12-12T20:12:07.384832Z	info	sds	Dynamic push for secret default
```
The dnsmasq configuration address=/.svc.cluster.local/{ISTIO_SERVICE_IP_STUB} also resolves the domain names ended with .svc.cluster.local to Istio service IP, so that you are able to access the Kubernetes services in the VM by fully qualified domain name (FQDN) such as httpbin.default.svc.cluster.local.
Deploy demo application Because we want to deploy CheckoutService and PaymentService manually on VM, resources/google-demo.yaml removes the two services from the original YAML . 04a-deploy-demo-app.sh deploys the other services on Kubernetes. Then log into the 2 VMs, run ~/work/deploy-checkoutservice.sh and ~/work/deploy-paymentservice.sh respectively to deploy CheckoutService and PaymentService.
Register VMs to Istio Services on VMs can access the services on Kubernetes by FQDN, but that’s not the case when the Kubernetes services want to talk to the VM services. The mesh has no idea where to forward the requests such as checkoutservice.default.svc.cluster.local because checkoutservice is isolated in the VM. Therefore, we need to register the services to the mesh. 04b-register-vm-with-istio.sh registers the VM services to the mesh by creating a “dummy” service without running Pods, and a WorkloadEntry to bridge the “dummy” service with the VM service.

Done!

The demo application contains a load generator service that performs requests repeatedly. We only need to wait a few seconds, and then open the SkyWalking web UI to check the results.

export POD_NAME=$(kubectl get pods --namespace istio-system -l "app=skywalking,release=skywalking,component=ui" -o jsonpath="{.items[0].metadata.name}")
echo "Visit http://127.0.0.1:8080 to use your application"
kubectl port-forward $POD_NAME 8080:8080 --namespace istio-system

Navigate the browser to http://localhost:8080 . The metrics, topology should be there.

Troubleshooting

If you face any trouble when walking through the steps, here are some common problems and possible solutions:

VM service cannot access Kubernetes services? It’s likely the DNS on the VM doesn’t correctly resolve the fully qualified domain names. Try to verify that with nslookup istiod.istio-system.svc.cluster.local. If it doesn’t resolve to the Kubernetes CIDR address, recheck the step in prep-checkoutservice.sh and prep-paymentservice.sh. If the DNS works correctly, try to verify that Envoy has fetched the upstream clusters from the control plane with curl http://localhost:15000/clusters. If it doesn’t contain the target service, recheck prep-checkoutservice.sh.
Services are normal but nothing on SkyWalking WebUI? Check the SkyWalking OAP logs via kubectl -n istio-system logs -f $(kubectl get pod -A -l "app=skywalking,release=skywalking,component=oap" -o name) and WebUI logs via kubectl -n istio-system logs -f $(kubectl get pod -A -l "app=skywalking,release=skywalking,component=ui" -o name) to see whether there are any error logs . Also, make sure the time zone at the bottom-right of the browser is set to UTC +0.

Additional Resources

Observe a Service Mesh with Envoy ALS.

Blog: Apache SkyWalking: How to propagate context between threads when using ThreadPoolExecutor

Tue, 09 Feb 2021 00:00:00 +0000

When using SkyWalking java agent, people usually propagate context easily. They even do not need to change the business code. However, it becomes harder when you want to propagate context between threads when using ThreadPoolExecutor. You can use the RunnableWrapper in the maven artifact org.apache.skywalking:apm-toolkit-trace. This way you must change your code. The developer manager usually don’t like this because there may be lots of projects, or lots of runnable code. If they don’t use SkyWalking some day, the code added will be superfluous and inelegant.

Is there a way to propagate context without changing the business code? Yes.

Skywalking java agent enhances a class by add a field and implement an interface. The ThreadPoolExecutor is a special class that is used widely. We even don’t know when and where it is loaded. Most JVMs do not allow changes in the class file format for classes that have been loaded previously. So SkyWalking should not enhance the ThreadPoolExecutor successfully by retransforming when the ThreadPoolExecutor has been loaded. However, we can apply advice to the ThreadPoolExecutor#execute method and wrap the Runnable param using our own agent, then enhance the wrapper class by SkyWalking java agent. An advice do not change the layout of a class.

Now we should decide how to do this. You can use the RunnableWrapper in the maven artifact org.apache.skywalking:apm-toolkit-trace to wrap the param, but you need to face another problem. This RunnableWrapper has a plugin whose active condition is checking if there is @TraceCrossThread. Agent core uses net.bytebuddy.pool.TypePool.Default.WithLazyResolution.LazyTypeDescription to find the annotations of a class. The LazyTypeDescription finds annotations by using a URLClassLoader with no urls if the classloader is null(bootstrap classloader). So it can not find the @TraceCrossThread class unless you change the LocationStrategy of SkyWalking java agent builder.

In this project, I write my own wrapper class, and simply add a plugin with a name match condition. Next, Let me show you how these two agents work together.

Move the plugin to the skywalking “plugins” directory.
Add this agent after the SkyWalking agent since the wrapper class should not be loaded before SkyWalking agent instrumentation have finished. For example,

java -javaagent:/path/to/skywalking-agent.jar -javaagent:/path/to/skywalking-tool-agent-v1.0.0.jar …
When our application runs
- SkyWalking java agent adds a transformer by parsing the plugin for enhancing the wrapper class in the tool agent.
- The tool agent loads the wrapper class into bootstrap classloader. This triggers the previous transformer.
- The tool agent applies an advice to the ThreadPoolExecutor class, wrapping the java.lang.Runnable param of “execute” method with the wrapper class.
- Now SkyWalking propagates the context with the wrapper class.

Enjoy tracing with ThreadPoolExecutor in SkyWalking!

Blog: SkyWalking 8.4 provides infrastructure monitoring

Mon, 08 Feb 2021 00:00:00 +0000

Origin: Tetrate.io blog

Background

Apache SkyWalking– the APM tool for distributed systems– has historically focused on providing observability around tracing and metrics, but service performance is often affected by the host. The newest release, SkyWalking 8.4.0, introduces a new feature for monitoring virtual machines. Users can easily detect possible problems from the dashboard– for example, when CPU usage is overloaded, when there’s not enough memory or disk space, or when the network status is unhealthy, etc.

How it works

SkyWalking leverages Prometheus and OpenTelemetry for collecting metrics data as we did for Istio control panel metrics; Prometheus is mature and widely used, and we expect to see increased adoption of the new CNCF project, OpenTelemetry. The SkyWalking OAP Server receives these metrics data of OpenCensus format from OpenTelemetry. The process is as follows:

Prometheus Node Exporter collects metrics data from the VMs.
OpenTelemetry Collector fetches metrics from Node Exporters via Prometheus Receiver, and pushes metrics to SkyWalking OAP Server via the OpenCensus GRPC Exporter.
The SkyWalking OAP Server parses the expression with MAL to filter/calculate/aggregate and store the results. The expression rules are in /config/otel-oc-rules/vm.yaml.
We can now see the data on the SkyWalking WebUI dashboard.

What to monitor

SkyWalking provides default monitoring metrics including:

CPU Usage (%)
Memory RAM Usage (MB)
Memory Swap Usage (MB)
CPU Average Used
CPU Load
Memory RAM (total/available/used MB)
Memory Swap (total/free MB)
File System Mount point Usage (%)
Disk R/W (KB/s)
Network Bandwidth Usage (receive/transmit KB/s)
Network Status (tcp_curr_estab/tcp_tw/tcp_alloc/sockets_used/udp_inuse)
File fd Allocated

The following is how it looks when we monitor Linux:

How to use

To enable this feature, we need to install Prometheus Node Exporter and OpenTelemetry Collector and activate the VM monitoring rules in SkyWalking OAP Server.

Install Prometheus Node Exporter

wget https://github.com/prometheus/node_exporter/releases/download/v1.0.1/node_exporter-1.0.1.linux-amd64.tar.gz
tar xvfz node_exporter-1.0.1.linux-amd64.tar.gz
cd node_exporter-1.0.1.linux-amd64
./node_exporter

In linux Node Exporter exposes metrics on port 9100 by default. When it is running, we can get the metrics from the /metrics endpoint. Use a web browser or command curl to verify.

curl http://localhost:9100/metrics

We should see all the metrics from the output like:

# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 7.7777e-05
go_gc_duration_seconds{quantile="0.25"} 0.000113756
go_gc_duration_seconds{quantile="0.5"} 0.000127199
go_gc_duration_seconds{quantile="0.75"} 0.000147778
go_gc_duration_seconds{quantile="1"} 0.000371894
go_gc_duration_seconds_sum 0.292994058
go_gc_duration_seconds_count 2029
...

Note: We only need to install Node Exporter, rather than Prometheus server. If you want to get more information about Prometheus Node Exporter see: https://prometheus.io/docs/guides/node-exporter/

Install OpenTelemetry Collector

We can quickly install a OpenTelemetry Collector instance by using docker-compose with the following steps:

Create a directory to store the configuration files, like /usr/local/otel.
Create docker-compose.yaml and otel-collector-config.yaml in this directory represented below:

docker-compose.yaml

version: "2"
services:
  # Collector
  otel-collector:
    # Specify the image to start the container from
    image: otel/opentelemetry-collector:0.19.0
    # Set the  otel-collector configfile 
    command: ["--config=/etc/otel-collector-config.yaml"]
    # Mapping the configfile to host directory
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    ports:
      - "13133:13133" # health_check extension
      - "55678"       # OpenCensus receiver

otel-collector-config.yaml

extensions:
  health_check:
# A receiver is how data gets into the OpenTelemetry Collector
receivers:
  # Set Prometheus Receiver to collects metrics from targets
  # It’s supports the full set of Prometheus configuration
  prometheus:
    config:
      scrape_configs:
        - job_name: 'otel-collector'
          scrape_interval: 10s
          static_configs: 
              # Replace the IP to your VMs‘s IP which has installed Node Exporter
            - targets: [ 'vm1:9100' ] 
            - targets: [ 'vm2:9100' ]
            - targets: [ ‘vm3:9100' ]

processors:
  batch:

# An exporter is how data gets sent to different systems/back-ends
exporters:
  # Exports metrics via gRPC using OpenCensus format
  opencensus:
    endpoint: "docker.for.mac.host.internal:11800" # The OAP Server address
    insecure: true
  logging:
    logLevel: debug

service:
  pipelines:
    metrics:
      receivers: [prometheus]
      processors: [batch]
      exporters: [logging, opencensus]

  extensions: [health_check]

In this directory use command docker-compose to start up the container:

docker-compose up -d

After the container is up and running, you should see metrics already exported in the logs:

...
Metric #165
Descriptor:
-> Name: node_network_receive_compressed_total
-> Description: Network device statistic receive_compressed.
-> Unit:
-> DataType: DoubleSum
-> IsMonotonic: true
-> AggregationTemporality: AGGREGATION_TEMPORALITY_CUMULATIVE
DoubleDataPoints #0
Data point labels:
-> device: ens4
StartTime: 1612234754364000000
Timestamp: 1612235563448000000
Value: 0.000000
DoubleDataPoints #1
Data point labels:
-> device: lo
StartTime: 1612234754364000000
Timestamp: 1612235563448000000
Value: 0.000000
...

If you want to get more information about OpenTelemetry Collector see: https://opentelemetry.io/docs/collector/

Set up SkyWalking OAP Server

To activate the oc handler and vm relevant rules, set your environment variables:

SW_OTEL_RECEIVER=default
SW_OTEL_RECEIVER_ENABLED_OC_RULES=vm

Note: If there are other rules already activated , you can add vm with use , as a separator.

SW_OTEL_RECEIVER_ENABLED_OC_RULES=vm,oap

Start the SkyWalking OAP Server.

Done!

After all of the above steps are completed, check out the SkyWalking WebUI. Dashboard VM provides the default metrics of all observed virtual machines. Note: Clear the browser local cache if you used it to access deployments of previous SkyWalking versions.

Additional Resources

Read more about the SkyWalking 8.4 release highlights.
Get more SkyWalking updates on Twitter.

Blog: Apache SkyWalking 8.4: Logs, VM Monitoring, and Dynamic Configurations at Agent Side

Fri, 05 Feb 2021 00:00:00 +0000

Origin: Tetrate.io blog

The Apache SkyWalking team today announced the 8.4 release is generally available. This release fills the gap between all previous versions of SkyWalking and the logging domain area. The release also advances SkyWalking’s capabilities for infrastructure observability, starting with virtual machine monitoring.

Background

SkyWalking has historically focused on the tracing and metrics fields of observability. As its features for tracing, metrics and service level monitoring have become more and more powerful and stable, the SkyWalking team has started to explore new scenarios covered by observability. Because service performance is reflected in the logs, and is highly impacted by the infrastructure on which it runs, SkyWalking brings these two fields into the 8.4 release. This release blog briefly introduces the two new features as well as some other notable changes.

Logs

Metrics, tracing, and logging are considered the three pillars of observability [1]. SkyWalking had the full features of metrics and tracing prior to 8.4; today, as 8.4 is released, the last piece of the jigsaw is now in place.

Figure 1: Logs Collected By SkyWalking

Figure 2: Logs Collected By SkyWalking

The Java agent firstly provides SDKs to enhance the widely-used logging frameworks, log4j (1.x and 2.x) [2] and logback [3], and send the logs to the SkyWalking backend (OAP). The latter is able to collect logs from wherever the protocol is implemented. This is not a big deal, but when it comes to the correlation between logs and traces, the traditional solution is to print the trace IDs in the logs, and pick the IDs in the error logs to query the related traces. SkyWalking just simplifies the workflow by correlating the logs and traces natively. Navigating between traces and their related logs is as simple as clicking a button.

Figure 3: Correlation Between Logs and Traces

Infrastructure Monitoring

SkyWalking is known as an application performance monitoring tool. One of the most important factors that impacts the application’s performance is the infrastructure on which the application runs. In the 8.4 release, we added the monitoring metrics of virtual machines into the dashboard.

Figure 4: VM Metrics

Fundamental metrics such as CPU Used, Memory Used, Disk Read / Write and Network Usage are available on the dashboard. And as usual, those metrics are also available to be configured as alarm triggers when needed.

Dynamic Configurations at Agent Side

Dynamic configuration at the backend side has long existed in SkyWalking for several versions. Now, it finally comes to the agent side! Prior to 8.4, you’d have to restart the target services when you modify some configuration items of the agent – for instance, sampling rate (agent side), ignorable endpoint paths, etc. Now, say goodbye to rebooting. Modifying configurations is not the only usage of the dynamic configuration mechanism. The latter gives countless possibilities to the agent side in terms of dynamic behaviours, e.g. enabling / disabling plugins, enabling / disabling the whole agent, etc. Just imagine!

Grouped Service Topology

This enhancement is from the UI. SkyWalking backend supports grouping the services by user-defined dimensions. In a real world use case, the services are usually grouped by business group or department. When a developer opens the topology map, out of hundreds of services, he or she may just want to focus on the services in charge. The grouped service topology comes to the rescue: one can now choose to display only services belonging to a specified group.

Figure 5: Grouped Service Topology

Other Notable Enhancements

Agent: resolves domain names to look up backend service IP addresses.
Backend: meter receiver supports meter analysis language (MAL).
Backend: several CVE fixes.
Backend: supports Envoy {AccessLog,Metrics}Service API V3 and adopts MAL.

Additional Resources

Read more about the SkyWalking 8.4 release highlights.
Get more SkyWalking updates on Twitter.

Blog: [Design] The Verifier of NGE2E

Mon, 01 Feb 2021 00:00:00 +0000

Background

The verifier is an important part of the next generation End-to-End Testing framework (NGE2E), which is responsible for verifying whether the actual output satisfies the expected template.

Design Thinking

We will implement the verifier with Go template, plus some enhancements. Firstly, users need to write a Go template file with provided functions and actions to describe how the expected data looks like. Then the verifer renders the template with the actual data object. Finally, the verifier compares the rendered output with the actual data. If the rendered output is not the same with the actual output, it means the actual data is inconsist with the expected data. Otherwise, it means the actual data match the expected data. On failure, the verifier will also print out what are different between expected and actual data.

Branches / Actions

The verifier inherits all the actions from the standard Go template, such as if, with, range, etc. In addition, we also provide some custom actions to satisfy our own needs.

List Elements Match

contains checks if the actual list contains elements that match the given template.

Examples:

metrics:
{{- contains .metrics }}
  - name: {{ notEmpty .name }}
    id: {{ notEmpty .id }}
    value: {{ gt .value 0 }}
{{- end }}

It means that the list metrics must contain an element whose name and id are not empty, and value is greater than 0.

metrics:
{{- contains .metrics }}
  - name: p95
    value: {{ gt .value 0 }}
  - name: p99
    value: {{ gt .value 0 }}
{{- end }}

This means that the list metrics must contain an element named p95 with a value greater than 0, and an element named p95 with a value greater than 0. Besides the two element, the list metrics may or may not have other random elements.

Functions

Users can use these provided functions in the template to describe the expected data.

Not Empty

notEmpty checks if the string s is empty.

Example:

id: {{ notEmpty .id }}

Regexp match

regexp checks if string s matches the regular expression pattern.

Examples:

label: {{ regexp .label "ratings.*" }}

Base64

b64enc s returns the Base64 encoded string of s.

Examples:

id: {{ b64enc "User" }}.static-suffix # this evalutes the base64 encoded string of "User", concatenated with a static suffix ".static-suffix"

Result:

id: VXNlcg==.static-suffix

Full Example

Here is an example of expected data:

# expected.data.yaml
nodes:
  - id: {{ b64enc "User" }}.0
    name: User
    type: USER
    isReal: false
  - id: {{ b64enc "Your_ApplicationName" }}.1
    name: Your_ApplicationName
    type: Tomcat
    isReal: true
  - id: {{ $h2ID := (index .nodes 2).id }}{{ notEmpty $h2ID }} # We assert that nodes[2].id is not empty and save it to variable `h2ID` for later use
    name: localhost:-1
    type: H2
    isReal: false
calls:
  - id: {{ notEmpty (index .calls 0).id }}
    source: {{ b64enc "Your_ApplicationName" }}.1
    target: {{ $h2ID }} # We use the previously assigned variable `h2Id` to asert that the `target` is equal to the `id` of the nodes[2]
    detectPoints:
      - CLIENT
  - id: {{ b64enc "User" }}.0-{{ b64enc "Your_ApplicationName" }}.1
    source: {{ b64enc "User" }}.0
    target: {{ b64enc "Your_ApplicationName" }}.1
    detectPoints:
      - SERVER

will validate this data:

# actual.data.yaml
nodes:
  - id: VXNlcg==.0
    name: User
    type: USER
    isReal: false
  - id: WW91cl9BcHBsaWNhdGlvbk5hbWU=.1
    name: Your_ApplicationName
    type: Tomcat
    isReal: true
  - id: bG9jYWxob3N0Oi0x.0
    name: localhost:-1
    type: H2
    isReal: false
calls:
  - id: WW91cl9BcHBsaWNhdGlvbk5hbWU=.1-bG9jYWxob3N0Oi0x.0
    source: WW91cl9BcHBsaWNhdGlvbk5hbWU=.1
    detectPoints:
      - CLIENT
    target: bG9jYWxob3N0Oi0x.0
  - id: VXNlcg==.0-WW91cl9BcHBsaWNhdGlvbk5hbWU=.1
    source: VXNlcg==.0
    detectPoints:
      - SERVER
    target: WW91cl9BcHBsaWNhdGlvbk5hbWU=.1

# expected.data.yaml
metrics:
{{- contains .metrics }}
  - name: {{ notEmpty .name }}
    id: {{ notEmpty .id }}
    value: {{ gt .value 0 }}
{{- end }}

will validate this data:

# actual.data.yaml
metrics:
  - name: business-zone::projectA
    id: YnVzaW5lc3Mtem9uZTo6cHJvamVjdEE=.1
    value: 1
  - name: system::load balancer1
    id: c3lzdGVtOjpsb2FkIGJhbGFuY2VyMQ==.1
    value: 0
  - name: system::load balancer2
    id: c3lzdGVtOjpsb2FkIGJhbGFuY2VyMg==.1
    value: 0

and will report an error when validating this data, because there is no element with a value greater than 0:

# actual.data.yaml
metrics:
  - name: business-zone::projectA
    id: YnVzaW5lc3Mtem9uZTo6cHJvamVjdEE=.1
    value: 0
  - name: system::load balancer1
    id: c3lzdGVtOjpsb2FkIGJhbGFuY2VyMQ==.1
    value: 0
  - name: system::load balancer2
    id: c3lzdGVtOjpsb2FkIGJhbGFuY2VyMg==.1
    value: 0

The contains does an unordered list verification, in order to do list verifications including orders, you can simply use the basic ruls like this:

# expected.data.yaml
metrics:
  - name: p99
    value: {{ gt (index .metrics 0).value 0 }}
  - name: p95
    value: {{ gt (index .metrics 1).value 0 }}

which expects the actual metrics list to be exactly ordered, with first element named p99 and value greater 0, second element named p95 and value greater 0.

Blog: [Resolved][License Issue] Tencent Cloud TSW service violates the Apache 2.0 License when using SkyWalking.

Fri, 22 Jan 2021 00:00:00 +0000

Apache SkyWalking is an open source APM for distributed system, Apache Software Foundation top-level project.

At Jan. 11th, 2021, we noticed the Tencent Cloud Service, Tencent Service Watcher - TSW, for first time. Due to the similar short name, which SkyWalking is also called SW in the community, we connected with the service team of Tencent Cloud, and kindly asked.

They used to replay, TSW is purely developed by Tencent team itself, which doesn’t have any code dependency on SkyWalking.. We didn’t push harder.

But one week later, Jan 18th, 2021, our V.P., Sheng got the report again from Haoyang SkyWalking PMC member, through WeChat DM(direct message),. He provided complete evidence to prove TSW actually re-distributed the SkyWalking’s Java agent. We keep one copy of their agent’s distribution(at Jan. 18th), you could be downloaded here.

Some typically evidences are here

ServiceManager is copied and package-name changed in the TSW’s agent.
ContextManager is copied and ackage-name changed in the TSW’s agent.

At the same time, we checked their tsw-client-package.zip, it didn’t include the SkyWalking’s LICENSE and NOTICE. Also, they didn’t mention TSW agent is the re-ditribution SkyWalking on their website.

With all above information, we had enough reason to believe, from the tech perspective, they were violating the Apache 2.0 License.

From the 18th Jan., 2021, we sent mail [Apache 2.0 License Violation] Tencent Cloud TSW service doesn't follow the Apache 2.0 License to brief the SkyWalking PMC, and took the following actions to connect with Tencent.

Made direct call to Tencent Open Source Office.
Connected with Tencent Cloud TVP program committee, as Sheng Wu(Our VP) is a Tencent Cloud TVP.
Talked with the Tencent Cloud team lead.

In all above channels, we provided the evidences of copy-redistribution hebaviors, requested them to revaluate their statements on the website, and follow the License’s requirements.

Resolution

At Jan. 19th night, UTC+8, 2021. We received response from the Tencent cloud team. They admited their violation behaviors, and did following changes

Tencent Cloud TSW service page states, the agent is the fork version(re-distribution) of Apache SkyWalking agent.
TSW agent distributions include the SkyWalking’s License and NOTICE. Below is the screenshot, you could download from their product page. We keep a copy of their Jan. 19th 2021 at here.

We have updated the status to the PMC mail list. This license violation issue has been resolved for now.

The SkyWalking community and program management committee will keep our eyes on Tencent TSW.

Blog: Response to Elastic 2021 License Change

Sun, 17 Jan 2021 00:00:00 +0000

Elastic announced their license change, Upcoming licensing changes to Elasticsearch and Kibana.

We are moving our Apache 2.0-licensed source code in Elasticsearch and Kibana to be dual licensed under Server Side Public License (SSPL) and the Elastic License, giving users the choice of which license to apply. This license change ensures our community and customers have free and open access to use, modify, redistribute, and collaborate on the code. It also protects our continued investment in developing products that we distribute for free and in the open by restricting cloud service providers from offering Elasticsearch and Kibana as a service without contributing back. This will apply to all maintained branches of these two products and will take place before our upcoming 7.11 release. Our releases will continue to be under the Elastic License as they have been for the last three years.

Also, they provide the FAQ page for more information about the impact for the users, developers, and vendors.

In the perspective of Apache Software Foundation, SSPL has been confirmed as a Catalog X LICENSE(https://www.apache.org/legal/resolved.html#category-x), which means hard-dependency as a part of the core is not allowed. With that, we can’t only focus on it anymore. We need to consider other storage options. Right now, we still have InfluxDB, TiDB, H2 server still in Apache 2.0 licensed. Right now, we still have InfluxDB, TiDB, H2 server as storage options still in Apache 2.0 licensed.

As one optional plugin, we need to focus on the client driver license. Right now, we are only using ElasticSearch 7.5.0 and 6.3.2 drivers, which are both Apache 2.0 licensed. So, we are safe. For further upgrade, here is their announcement. They answer these typical cases in the FAQ page.

I build a SaaS application using Elasticsearch as the backend, how does this affect me?

This source code license change should not affect you - you can use our default distribution or develop applications on top of it for free, under the Elastic License. This source-available license does not contain any copyleft provisions and the default functionality is free of charge. For a specific example, you can see our response to a question around this at Magento.

Our users still could use, redistribute, sale the products/services, based on SkyWalking, even they are using self hosting Elastic Search unmodified server.

I’m using Elasticsearch via APIs, how does this change affect me?

This change does not affect how you use client libraries to access Elasticsearch. Our client libraries remain licensed under Apache 2.0, with the exception of our Java High Level Rest Client (Java HLRC). The Java HLRC has dependencies on the core of Elasticsearch, and as a result this client library will be licensed under the Elastic License. Over time, we will eliminate this dependency and move the Java HLRC to be licensed under Apache 2.0. Until that time, for the avoidance of doubt, we do not consider using the Java HLRC as a client library in development of an application or library used to access Elasticsearch to constitute a derivative work under the Elastic License, and this will not have any impact on how you license the source code of your application using this client library or how you distribute it.

The client driver license incompatible issue will exist, we can’t upgrade the driver(s) until they release the Apache 2.0 licensed driver jars. But users are still safe to upgrade the drivers by themselves.

Apache SkyWalking will discuss the further actions here. If you have any question, welcome to ask. In the later 2021, we will begin to invest the posibility of creating SkyWalking’s observability database implementation.

Blog: Celebrate SkyWalking single repository hits the 300 contributors mark

Tue, 22 Dec 2020 00:00:00 +0000

Apache SkyWalking is an open source APM for distributed system. Provide tracing, service mesh observability, metrics analysis, alarm and visualization.

Just 11 months ago, on Jan. 20th, 2020, SkyWalking hit the 200 contributors mark. With the growth of the project and the community, SkyWalking now includes over 20 sub(ecosystem) projects covering multiple language agents and service mesh, integration with mature open source projects, like Prometheus, Spring(Sleuth), hundreds of libraries to support all tracing/metrics/logs fields. In the past year, the number of contributors grows super astoundingly , and all its metrics point to its community vibrancy. Many corporate titans are already using SkyWalking in a large-scale production environment, including, Alibaba, Huawei, Baidu, Tencent, etc.

Recently, our SkyWalking main repository overs 300 contributors.

Our website has thousands of views from most countries in the world every week.

Although we know that, the metrics like GitHub stars and the numbers of open users and contributors, are not a determinant of vibrancy, they do show the trend, we are very proud to share the increased numbers here, too.

We double those numbers and are honored with the development of our community.

Thank you, all of our contributors. Not just these 300 contributors of the main repository, or nearly 400 contributors in all repositories, counted by GitHub. There are countless people contributing codes to SkyWalking’s subprojects, ecosystem projects, and private fork versions; writing blogs and guidances, translating documents, books, and presentations; setting up learning sessions for new users; convincing friends to join the community as end-users, contributors, even committers. Companies behinds those contributors support their employees to work with the community to provide feedback and contribute the improvements and features upstream. Conference organizers share the stages with speakers from the SkyWalking community.

SkyWalking can’t make this happen without your help. You made this community extraordinary.

At this crazy distributed computing and cloud native age, we as a community could make DEV, OPS, and SRE teams’ work easier by locating the issue(s) in the haystack quicker than before, like why we named the project as SkyWalking, we will have a clear site line when you stand on the glass bridge Skywalk at Grand Canyon West.

376 Contributors counted by GitHub account are following. Dec. 22st, 2020. Generated by a tool deveoped by Yousa

1095071913
50168383
Ahoo-Wang
AirTrioa
AlexanderWert
AlseinX
Ax1an
BFergerson
BZFYS
CharlesMaster
ChaunceyLin5152
CommissarXia
Cvimer
Doublemine
ElderJames
EvanLjp
FatihErdem
FeynmanZhou
Fine0830
FingerLiu
Gallardot
GerryYuan
HackerRookie
Heguoya
Hen1ng
Humbertzhang
IanCao
IluckySi
Indifer
J-Cod3r
JaredTan95
Jargon96
Jijun
JohnNiang
Jozdortraz
Jtrust
Just-maple
KangZhiDong
LazyLei
LiWenGu
Liu-XinYuan
Miss-you
O-ll-O
Patrick0308
QHWG67
Qiliang
RandyAbernethy
RedzRedz
Runrioter
SataQiu
ScienJus
SevenPointOld
ShaoHans
Shikugawa
SoberChina
SummerOfServenteen
TJ666
TerrellChen
TheRealHaui
TinyAllen
TomMD
ViberW
Videl
WALL-E
WeihanLi
WildWolfBang
WillemJiang
Wooo0
XhangUeiJong
Xlinlin
YczYanchengzhe
YoungHu
YunaiV
ZhHong
ZhuoSiChen
ZS-Oliver
a198720
a526672351
acurtain
adamni135
adermxzs
adriancole
aeolusheath
agile6v
aix3
aiyanbo
ajanthan
alexkarezin
alonelaval
amogege
amwyyyy
arugal
ascrutae
augustowebd
bai-yang
beckhampu
beckjin
beiwangnull
bigflybrother
bostin
brucewu-fly
c1ay
candyleer
carlvine500
carrypann
cheenursn
cheetah012
chenpengfei
chenvista
chess-equality
chestarss
chidaodezhongsheng
chopin-d
clevertension
clk1st
cngdkxw
codeglzhang
codelipenghui
coder-yqj
coki230
coolbeevip
crystaldust
cui-liqiang
cuiweiwei
cyberdak
cyejing
dagmom
dengliming
devkanro
devon-ye
dimaaan
dingdongnigetou
dio
dmsolr
dominicqi
donbing007
dsc6636926
duotai
dvsv2
dzx2018
echooymxq
efekaptan
eoeac
evanxuhe
feelwing1314
fgksgf
fuhuo
geektcp
geomonlin
ggndnn
gitter-badger
glongzh
gnr163
gonedays
grissom-grissom
grissomsh
guodongq
guyukou
gxthrj
gzshilu
hailin0
hanahmily
haotian2015
haoyann
hardzhang
harvies
hepyu
heyanlong
hi-sb
honganan
hsoftxl
huangyoje
huliangdream
huohuanhuan
innerpeacez
itsvse
jasonz93
jialong121
jinlongwang
jjlu521016
jjtyro
jmjoy
jsbxyyx
justeene
juzhiyuan
jy00464346
kaanid
karott
kayleyang
kevinyyyy
kezhenxu94
kikupotter
kilingzhang
killGC
klboke
ksewen
kuaikuai
kun-song
kylixs
landonzeng
langke93
langyan1022
langyizhao
lazycathome
leemove
leizhiyuan
libinglong
lilien1010
limfriend
linkinshi
linliaoy
liuhaoXD
liuhaoyang
liuyanggithup
liuzhengyang
liweiv
lkxiaolou
llissery
louis-zhou
lpf32
lsyf
lucperkins
lujiajing1126
lunamagic1978
lunchboxav
lxliuxuankb
lytscu
lyzhang1999
magic-akari
makingtime
maolie
masterxxo
maxiaoguang64
membphis
mestarshine
mgsheng
michaelsembwever
mikkeschiren
mm23504570
momo0313
moonming
mrproliu
muyun12
nacx
neatlife
neeuq
nic-chen
nikitap492
nileblack
nisiyong
novayoung
oatiz
oflebbe
olzhy
onecloud360
osiriswd
peng-yongsheng
pengweiqhca
potiuk
purgeyao
qijianbo010
qinhang3
qiuyu-d
qqeasonchen
qxo
raybi-asus
refactor2
remicollet
rlenferink
rootsongjc
rovast
scolia
sdanzo
seifeHu
shiluo34
sikelangya
simonlei
sk163
snakorse
songzhendong
songzhian
sonxy
spacewander
stalary
stenio2011
stevehu
stone-wlg
sungitly
surechen
swartz-k
sxzaihua
tanjunchen
tankilo
taskmgr
tbdpmi
terranhu
terrymanu
tevahp
thanq
thebouv
tianyuak
tincopper
tinyu0
tom-pytel
tristaZero
tristan-tsl
trustin
tsuilouis
tuohai666
tzsword-2020
tzy1316106836
vcjmhg
vision-ken
viswaramamoorthy
wankai123
wbpcode
web-xiaxia
webb2019
weiqiang333
wendal
wengangJi
wenjianzhang
whfjam
wind2008hxy
withlin
wqr2016
wu-sheng
wuguangkuo
wujun8
wuxingye
x22x22
xbkaishui
xcaspar
xiaoxiangmoe
xiaoy00
xinfeingxia85
xinzhuxiansheng
xudianyang
yanbw
yanfch
yang-xiaodong
yangxb2010000
yanickxia
yanmaipian
yanmingbi
yantaowu
yaowenqiang
yazong
ychandu
ycoe
yimeng
yu199195
yuqichou
yuyujulin
yymoth
zaunist
zaygrzx
zcai2
zeaposs
zhang98722
zhanghao001
zhangjianweibj
zhangkewei
zhangsean
zhaoyuguang
zhentaoJin
zhousiliang163
zhuCheer
zifeihan
zkscpqm
zoidbergwill
zoumingzm
zouyx
zshit
zxbu
zygfengyuwuzu

Blog: [Design] NGE2E - Next Generation End-to-End Testing Framework

Mon, 14 Dec 2020 00:00:00 +0000

NGE2E is the next generation End-to-End Testing framework that aims to help developers to set up, debug, and verify E2E tests with ease. It’s built based on the lessons learnt from tens of hundreds of test cases in the SkyWalking main repo.

Goal

Keep the feature parity with the existing E2E framework in SkyWalking main repo;
Support both docker-compose and KinD to orchestrate the tested services under different environments;
Get rid of the heavy Java/Maven stack, which exists in the current E2E; be language independent as much as possible, users only need to configure YAMLs and run commands, without writing codes;

Non-Goal

This framework is not involved with the build process, i.e. it won’t do something like mvn package or docker build, the artifacts (.tar, docker images) should be ready in an earlier process before this;
This project doesn’t take the plugin tests into account, at least for now;
This project doesn’t mean to add/remove any new/existing test case to/from the main repo;
This documentation won’t cover too much technical details of how to implement the framework, that should go into an individual documentation;

Design

Before diving into the design details, let’s take a quick look at how the end user might use NGE2E.

All the following commands are mock, and are open to debate.

To run a test case in a directory /path/to/the/case/directory

e2e run /path/to/the/case/directory

# or

cd /path/to/the/case/directory && e2e run

This will run the test case in the specified directory, this command is a wrapper that glues all the following commands, which can be executed separately, for example, to debug the case:

NOTE: because all the options can be loaded from a configuration file, so as long as a configuration file (say e2e.yaml) is given in the directory, every command should be able to run in bare mode (without any option explicitly specified in the command line);

Set Up

e2e setup --env=compose --file=docker-compose.yaml --wait-for=service/health
e2e setup --env=kind --file=kind.yaml --manifests=bookinfo.yaml,gateway.yaml --wait-for=pod/ready
e2e setup # If configuration file e2e.yaml is present

--env: the environment, may be compose or kind, represents docker-compose and KinD respectively;
--file: the docker-compose.yaml or kind.yaml file that declares how to set up the environment;
--manifests: for KinD, the resources files/directories to apply (using kubectl apply -f);
--command: a command to run after the environment is started, this may be useful when users need to install some extra tools or apply resources from command line, like istioctl install --profile=demo;
--wait-for: can be specified multiple times to give a list of conditions to be met; wait until the given conditions are met; the most frequently-used strategy should be --wait-for=service/health, --wait-for=deployments/available, etc. that make the e2e setup command to wait for all conditions to be met; other possible strategies may be something like --wait-for="log:Started Successfully", --wait-for="http:localhost:8080/healthcheck", etc. if really needed;

Trigger Inputs

e2e trigger --interval=3s --times=0 --action=http --url="localhost:8080/users"
e2e trigger --interval=3s --times=0 --action=cmd --cmd="curl localhost:8080/users"
e2e trigger # If configuration file e2e.yaml is present

--interval=3s: trigger the action every 3 seconds;
--times=0: how many times to trigger the action, 0=infinite;
--action=http: the action of the trigger, i.e. “perform an http request as an input”;
--action=cmd: the action of the trigger, i.e. “execute the cmd as an input”;

Query Output

swctl service ls

this is a project-specific step, different project may use different tools to query the actual output, for SkyWalking, it uses swctl to query the actual output.

Verify

e2e verify --actual=actual.data.yaml --expected=expected.data.yaml
e2e verify --query="swctl service ls" --expected=expected.data.yaml
e2e verify # If configuration file e2e.yaml is present

--actual: the actual data file, only YAML file format is supported;
--expected: the expected data file, only YAML file format is supported;
--query: the query to get the actual data, the query result must have the same format as --actual and --expected;

The --query option will get the output into a temporary file and use the --actual under the hood;

Cleanup

e2e cleanup --env=compose --file=docker-compose.yaml
e2e cleanup --env=kind --file=kind.yaml --resources=bookinfo.yaml,gateway.yaml
e2e cleanup # If configuration file e2e.yaml is present

This step requires the same options in the setup step so that it can clean up all things necessarily.

Summarize

To summarize, the directory structure of a test case might be

case-name
├── agent-service        # optional, an arbitrary project that is used in the docker-compose.yaml if needed
│   ├── Dockerfile
│   ├── pom.xml
│   └── src
├── docker-compose.yaml
├── e2e.yaml             # see a sample below
└── testdata
    ├── expected.endpoints.service1.yaml
    ├── expected.endpoints.service2.yaml
    └── expected.services.yaml

case-name
├── kind.yaml
├── bookinfo
│   ├── bookinfo.yaml
│   └── bookinfo-gateway.yaml
├── e2e.yaml             # see a sample below
└── testdata
    ├── expected.endpoints.service1.yaml
    ├── expected.endpoints.service2.yaml
    └── expected.services.yaml

a sample of e2e.yaml may be

setup:
  env: kind
  file: kind.yaml
  manifests:
    - path: bookinfo.yaml
      wait: # you can have multiple conditions to wait
        - namespace: bookinfo
          label-selector: app=product
          for: deployment/available
        - namespace: reviews
          label-selector: app=product
          for: deployment/available
        - namespace: ratings
          label-selector: app=product
          for: deployment/available

  run:
    - command: | # it can be a shell script or anything executable
        istioctl install --profile=demo -y
        kubectl label namespace default istio-injection=enabled
      wait:
        - namespace: istio-system
          label-selector: app=istiod
          for: deployment/available

  # OR
  # env: compose
  # file: docker-compose.yaml

trigger:
  action: http
  interval: 3s
  times: 0
  url: localhost:9090/users

verify:
  - query: swctl service ls
    expected: expected.services.yaml
  - query: swctl endpoint ls --service="YnVzaW5lc3Mtem9uZTo6cHJvamVjdEM=.1"
    expected: expected.projectC.endpoints.yaml

then a single command should do the trick.

e2e run

Modules

This project is divided into the following modules.

Controller

A controller command (e2e run) composes all the steps declared in the e2e.yaml, it should be progressive and clearly display which step is currently running. If it failed in a step, the error message should be as much comprehensive as possible. An example of the output might be

e2e run
✔ Started Kind Cluster - Cluster Name
✔ Checked Pods Readiness - All pods are ready
? Generating Traffic - http localhost:9090/users (progress spinner)
✔ Verified Output - service ls
(progress spinner) Verifying Output - endpoint ls
✘ Failed to Verify Output Data - endpoint ls
  <the diff content>
✔ Clean Up

Compared with running the steps one by one, the controller is also responsible for cleaning up env (by executing cleanup command) no mater what status other commands are, even if they are failed, the controller has the following semantics in terms of setup and cleanup.

// Java
try {
    setup();
    // trigger step
    // verify step
    // ...
} finally {
    cleanup();
}

// GoLang
func run() {
    setup();
    defer cleanup();
    // trigger step
    // verify step
    // ...
}

Initializer

The initializer is responsible for

When env==compose
- Start the docker-compose services;
- Check the services’ healthiness;
- Wait until all services are ready according to the interval, etc.;
When env==kind
- Start the KinD cluster according to the config files;
- Apply the resources files (--manifests) or/and run the custom init command (--commands);
- Check the pods’ readiness;
- Wait until all pods are ready according to the interval, etc.;

Verifier

According to scenarios we have at the moment, the must-have features are:

Matchers
- Exact match
- Not null
- Not empty
- Greater than 0
- Regexp match
- At least one of list element match
Functions
- Base64 encode/decode

in order to help to identify simple bugs from the GitHub Actions workflow, there are some “nice to have” features:

Printing the diff content when verification failed is a super helpful bonus proved in the Python agent repo;

Logging

When a test case failed, all the necessary logs should be collected into a dedicated directory, which could be uploaded to the GitHub Artifacts for downloading and analysis;

Logs through the entire process of a test case are:

KinD clusters logs;
Containers/pods logs;
The logs from the NGE2E itself;

More Planned

Debugging

Debugging the E2E locally has been a strong requirement and time killer that we haven’t solve up to date, though we have enhancements like https://github.com/apache/skywalking/pull/5198 , but in this framework, we will adopt a new method to “really” support debugging locally.

The most common case when debugging is to run the E2E tests, with one or more services forwarded into the host machine, where the services are run in the IDE or in debug mode.

For example, you may run the SkyWalking OAP server in an IDE and run e2e run, expecting the other services (e.g. agent services, SkyWalking WebUI, etc.) inside the containers to connect to your local OAP, instead of the one declared in docker-compose.yaml.

For Docker Desktop Mac/Windows, we can access the services running on the host machine inside containers via host.docker.internal, for Linux, it’s 172.17.0.1.

One possible solution is to add an option --debug-services=oap,other-service-name that rewrites all the router rules inside the containers from oap to host.docker.internal/172.17.0.1.

CodeGen

When adding new test case, a code generator would be of great value to eliminate the repeated labor and copy-pasting issues.

e2e new <case-name>

Blog: Observe Service Mesh with SkyWalking and Envoy Access Log Service

Thu, 03 Dec 2020 00:00:00 +0000

Author: Zhenxu Ke, Sheng Wu, and Tevah Platt. tetrate.io
Original link, Tetrate.io blog
Dec. 03th, 2020

Apache SkyWalking: an APM (application performance monitor) system, especially designed for microservices, cloud native, and container-based (Docker, Kubernetes, Mesos) architectures.

Envoy Access Log Service: Access Log Service (ALS) is an Envoy extension that emits detailed access logs of all requests going through Envoy.

Background

Apache SkyWalking has long supported observability in service mesh with Istio Mixer adapter. But since v1.5, Istio began to deprecate Mixer due to its poor performance in large scale clusters. Mixer’s functionalities have been moved into the Envoy proxies, and is supported only through the 1.7 Istio release. On the other hand, Sheng Wu and Lizan Zhou presented a better solution based on the Apache SkyWalking and Envoy ALS on KubeCon China 2019, to reduce the performance impact brought by Mixer, while retaining the same observability in service mesh. This solution was initially implemented by Sheng Wu, Hongtao Gao, Lizan Zhou, and Dhi Aurrahman at Tetrate.io. If you are looking for a more efficient solution to observe your service mesh instead of using a Mixer-based solution, this is exactly what you need. In this tutorial, we will explain a little bit how the new solution works, and apply it to the bookinfo application in practice.

How it works

From a perspective of observability, Envoy can be typically deployed in 2 modes, sidecar, and router. As a sidecar, Envoy mostly represents a single service to receive and send requests (2 and 3 in the picture below). While as a proxy, Envoy may represent many services (1 in the picture below).

In both modes, the logs emitted by ALS include a node identifier. The identifier starts with router~ (or ingress~) in router mode and sidecar~ in sidecar proxy mode.

Apart from the node identifier, there are several noteworthy properties in the access logs that will be used in this solution:

downstream_direct_remote_address: This field is the downstream direct remote address on which the request from the user was received. Note: This is always the physical peer, even if the remote address is inferred from for example the x-forwarded-for header, proxy protocol, etc.
downstream_remote_address: The remote/origin address on which the request from the user was received.
downstream_local_address: The local/destination address on which the request from the user was received.
upstream_remote_address: The upstream remote/destination address that handles this exchange.
upstream_local_address: The upstream local/origin address that handles this exchange.
upstream_cluster: The upstream cluster that upstream_remote_address belongs to.

We will discuss more about the properties in the following sections.

Sidecar

When serving as a sidecar, Envoy is deployed alongside a service, and delegates all the incoming/outgoing requests to/from the service.

Delegating incoming requests: in this case, Envoy acts as a server side sidecar, and sets the upstream_cluster in form of inbound|portNumber|portName|Hostname[or]SidecarScopeID.

The SkyWalking analyzer checks whether either downstream_remote_address can be mapped to a Kubernetes service:

a. If there is a service (say Service B) whose implementation is running in this IP(and port), then we have a service-to-service relation, Service B -> Service A, which can be used to build the topology. Together with the start_time and duration fields in the access log, we have the latency metrics now.

b. If there is no service that can be mapped to downstream_remote_address, then the request may come from a service out of the mesh. Since SkyWalking cannot identify the source service where the requests come from, it simply generates the metrics without source service, according to the topology analysis method. The topology can be built as accurately as possible, and the metrics detected from server side are still correct.
Delegating outgoing requests: in this case, Envoy acts as a client-side sidecar, and sets the upstream_cluster in form of outbound|<port>|<subset>|<serviceFQDN>.

Client side detection is relatively simpler than (1. Delegating incoming requests). If upstream_remote_address is another sidecar or proxy, we simply get the mapped service name and generate the topology and metrics. Otherwise, we have no idea what it is and consider it an UNKNOWN service.

Proxy role

When Envoy is deployed as a proxy, it is an independent service itself and doesn’t represent any other service like a sidecar does. Therefore, we can build client-side metrics as well as server-side metrics.

Example

In this section, we will use the typical bookinfo application to demonstrate how Apache SkyWalking 8.3.0+ (the latest version up to Nov. 30th, 2020) works together with Envoy ALS to observe a service mesh.

Installing Kubernetes

SkyWalking 8.3.0 supports the Envoy ALS solution under both Kubernetes environment and virtual machines (VM) environment, in this tutorial, we’ll only focus on the Kubernetes scenario, for VM solution, please stay tuned for our next blog, so we need to install Kubernetes before taking further steps.

In this tutorial, we will use the Minikube tool to quickly set up a local Kubernetes(v1.17) cluster for testing. In order to run all the needed components, including the bookinfo application, the SkyWalking OAP and WebUI, the cluster may need up to 4GB RAM and 2 CPU cores.

minikube start --memory=4096 --cpus=2

Next, run kubectl get pods --namespace=kube-system --watch to check whether all the Kubernetes components are ready. If not, wait for the readiness before going on.

Installing Istio

Istio provides a very convenient way to configure the Envoy proxy and enable the access log service. The built-in configuration profiles free us from lots of manual operations. So, for demonstration purposes, we will use Istio through this tutorial.

export ISTIO_VERSION=1.7.1
curl -L https://istio.io/downloadIstio | sh - 
sudo mv $PWD/istio-$ISTIO_VERSION/bin/istioctl /usr/local/bin/
istioctl  install --set profile=demo
kubectl label namespace default istio-injection=enabled

Run kubectl get pods --namespace=istio-system --watch to check whether all the Istio components are ready. If not, wait for the readiness before going on.

Enabling ALS

The demo profile doesn’t enable ALS by default. We need to reconfigure it to enable ALS via some configuration.

istioctl  manifest install \
               --set meshConfig.enableEnvoyAccessLogService=true \
               --set meshConfig.defaultConfig.envoyAccessLogService.address=skywalking-oap.istio-system:11800

The example command --set meshConfig.enableEnvoyAccessLogService=true enables the Envoy access log service in the mesh. And as we said earlier, ALS is essentially a gRPC service that emits requests logs. The config meshConfig.defaultConfig.envoyAccessLogService.address=skywalking-oap.istio-system:11800 tells this gRPC service where to emit the logs, say skywalking-oap.istio-system:11800, where we will deploy the SkyWalking ALS receiver later.

NOTE: You can also enable the ALS when installing Istio so that you don’t need to restart Istio after installation:

istioctl install --set profile=demo \
               --set meshConfig.enableEnvoyAccessLogService=true \
               --set meshConfig.defaultConfig.envoyAccessLogService.address=skywalking-oap.istio-system:11800
kubectl label namespace default istio-injection=enabled

Deploying Apache SkyWalking

The SkyWalking community provides a Helm Chart to make it easier to deploy SkyWalking and its dependent services in Kubernetes. The Helm Chart can be found at the GitHub repository.

# Install Helm
curl -sSLO https://get.helm.sh/helm-v3.0.0-linux-amd64.tar.gz
sudo tar xz -C /usr/local/bin --strip-components=1 linux-amd64/helm -f helm-v3.0.0-linux-amd64.tar.gz
# Clone SkyWalking Helm Chart
git clone https://github.com/apache/skywalking-kubernetes
cd skywalking-kubernetes/chart
git reset --hard dd749f25913830c47a97430618cefc4167612e75
# Update dependencies
helm dep up skywalking
# Deploy SkyWalking
helm -n istio-system install skywalking skywalking \
               --set oap.storageType='h2'\
               --set ui.image.tag=8.3.0 \
               --set oap.image.tag=8.3.0-es7 \
               --set oap.replicas=1 \
               --set oap.env.SW_ENVOY_METRIC_ALS_HTTP_ANALYSIS=k8s-mesh \
               --set oap.env.JAVA_OPTS='-Dmode=' \
               --set oap.envoy.als.enabled=true \
               --set elasticsearch.enabled=false

We deploy SkyWalking to the namespace istio-system, so that SkyWalking OAP service can be accessed by skywalking-oap.istio-system:11800, to which we told ALS to emit their logs, in the previous step.

We also enable the ALS analyzer in the SkyWalking OAP: oap.env.SW_ENVOY_METRIC_ALS_HTTP_ANALYSIS=k8s-mesh. The analyzer parses the access logs and maps the IP addresses in the logs to the real service names in the Kubernetes, to build a topology.

In order to retrieve the metadata (such as Pod IP and service names) from a Kubernetes cluster for IP mappings, we also set oap.envoy.als.enabled=true, to apply for a ClusterRole that has access to the metadata.

export POD_NAME=$(kubectl get pods -A -l "app=skywalking,release=skywalking,component=ui" -o name)
echo $POD_NAME
kubectl -n istio-system port-forward $POD_NAME 8080:8080

Now navigate your browser to http://localhost:8080 . You should be able to see the SkyWalking dashboard. The dashboard is empty for now, but after we deploy the demo application and generate traffic, it should be filled up later.

Deploying Bookinfo application

Run:

export ISTIO_VERSION=1.7.1
kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/platform/kube/bookinfo.yaml
kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/networking/bookinfo-gateway.yaml
kubectl wait --for=condition=Ready pods --all --timeout=1200s
minikube tunnel

Then navigate your browser to http://localhost/productpage. You should be able to see the typical bookinfo application. Refresh the webpage several times to generate enough access logs.

Done!

And you’re all done! Check out the SkyWalking WebUI again. You should see the topology of the bookinfo application, as well the metrics of each individual service of the bookinfo application.

Troubleshooting

Check all pods status: kubectl get pods -A.
SkyWalking OAP logs: kubectl -n istio-system logs -f $(kubectl get pod -A -l "app=skywalking,release=skywalking,component=oap" -o name).
SkyWalking WebUI logs: kubectl -n istio-system logs -f $(kubectl get pod -A -l "app=skywalking,release=skywalking,component=ui" -o name).
Make sure the time zone at the bottom-right of the WebUI is set to UTC +0.

Customizing Service Names

The SkyWalking community brought more improvements to the ALS solution in the 8.3.0 version. You can decide how to compose the service names when mapping from the IP addresses, with variables service and pod. For instance, configuring K8S_SERVICE_NAME_RULE to the expression ${service.metadata.name}-${pod.metadata.labels.version} gets service names with version label such as reviews-v1, reviews-v2, and reviews-v3, instead of a single service reviews, see the PR.

Working ALS with VM

Kubernetes is popular, but what about VMs? From what we discussed above, in order to map the IPs to services, SkyWalking needs access to the Kubernetes cluster, fetching service metadata and Pod IPs. But in a VM environment, there is no source from which we can fetch those metadata. In the next post, we will introduce another ALS analyzer based on the Envoy metadata exchange mechanism. With this analyzer, you are able to observe a service mesh in the VM environment. Stay tuned! If you want to have commercial support for the ALS solution or hybrid mesh observability, Tetrate Service Bridge, TSB is another good option out there.

Additional Resources

KubeCon 2019 Recorded Video.
Get more SkyWalking updates on the official website.

Apache SkyWalking founder Sheng Wu, SkyWalking core maintainer Zhenxu Ke are Tetrate engineers, and Tevah Platt is a content writer for Tetrate. Tetrate helps organizations adopt open source service mesh tools, including Istio, Envoy, and Apache SkyWalking, so they can manage microservices, run service mesh on any infrastructure, and modernize their applications.

Blog: The first design of Satellite 0.1.0

Wed, 25 Nov 2020 00:00:00 +0000

Author: Jiapeng Liu. Baidu.
skywalking-satellite: The Sidecar Project of Apache SkyWalking
Nov. 25th, 2020

A lightweight collector/sidecar which can be deployed close to the target monitored system, to collect metrics, traces, and logs. It also provides advanced features, such as local cache, format transformation, and sampling.

Design Thinking

Satellite is a 2 level system to collect observability data from other core systems. So, the core element of the design is to guarantee data stability during Pod startup all the way to Pod shutdown avoiding alarm loss. All modules are designed as plugins, and if you have other ideas, you can add them yourself.

SLO

Single gatherer supports > 1000 ops (Based 0.5 Core,50M)
At least once delivery.(Optional)
Data stability: 99.999%.(Optional)

Because they are influenced by the choice of plugins, some items in SLO are optional.

Role

Satellite would be running as a Sidecar. Although Daemonset mode would take up fewer resources, it will cause more troubles to the forwarding of agents. So we also want to use Sidecar mode by reducing the costs. But Daemonset mode would be also supported in the future plan.

Core Modules

The Satellite has 3 core modules which are Gatherer, Processor, and Sender.

The Gatherer module is responsible for fetching or receiving data and pushing the data to Queue.
The Processor module is responsible for reading data from the queue and processing data by a series of filter chains.
The Sender module is responsible for async processing and forwarding the data to the external services in the batch mode. After sending success, Sender would also acknowledge the offset of Queue in Gatherer.

Detailed Structure

The overall design is shown in detail in the figure below. We will explain the specific components one by one.

Gatherer

Concepts

The Gatherer has 4 components to support the data collection, which are Input, Collector, Worker, and Queue. There are 2 roles in the Worker, which are Fetcher and Receiver.

The Input is an abstraction of the input source, which is usually mapped to a configuration file.
The Collector is created by the Source, but many collectors could be created by the same Source. For example, when a log path has been configured as the /var/*.log in an Input, the number of collectors is the same as the file number in this path.
The Fetcher and Receiver is the real worker to collect data. The receiver interface is an abstraction, which has multiple implementations, such as gRPC receiver and HTTP receiver.Here are some specific use cases:
- Trace Receiver is a gRPC server for receiving trace data created by Skywalking agents.
- Log Receiver is also a gRPC server for receiving log data which is collected by Skywalking agents. (In the future we want Skywalking Agent to support log sending, and RPC-based log sending is more efficient and needs fewer resources than file reading. For example, the way of file reading will bring IO pressure and performance cost under multi-line splicing.)
- Log Fetcher is like Filebeat, which fits the common log collection scenario. This fetcher will have more responsibility than any other workers because it needs to record the offset and process the multi-line splicing. This feature will be implemented in the future.
- Prometheus Fetcher supports a new way to fetch Prometheus data and push the data to the upstream.
- ……
The Queue is a buffer module to decouple collection and transmission. In the 1st release version, we will use persistent storage to ensure data stability. But the implementation is a plug-in design that can support pure memory queues later.

The data flow

We use the Trace Receiver as an example to introduce the data flow.

Queue

MmapQueue

We have simplified the design of MmapQueue to reduce the resources cost on the memory and disk.

Concepts

There are 2 core concepts in MmapQueue.

Segment: Segment is the real data store center, that provides large-space storage and does not reduce read and write performance as much as possible by using mmap. And we will avoid deleting files by reusing them.
Meta: The purpose of meta is to find the data that the consumer needs.

Segment

One MmapQueue has a directory to store the whole data. The Queue directory is made up with many segments and 1 meta file. The number of the segments would be computed by 2 params, which are the max cost of the Queue and the cost of each segment. For example, If the max cost is 512M and each segment cost is 256K, the directory can hold up to 2000 files. Once capacity is exceeded, an coverage policy is adopted that means the 2000th would override the first file.

Each segment in Queue will be N times the size of the page cache and will be read and written in an appended sequence rather than randomly. These would improve the performance of Queue. For example, each Segment is a 128k file, as shown in the figure below.

Mmap Performance Test

The test is to verify the efficiency of mmap in low memory cost.

The rate of data generation: 7.5K/item 1043 item/s (Based on Aifanfan online pod.)
The test structure is based on Bigqueue because of similar structure.
Test tool: Go Benchmark Test
Command: go test -bench BenchmarkEnqueue -run=none -cpu=1
Result On Mac(15-inch, 2018,16 GB 2400 MHz DDR4, 2.2 GHz Intel Core i7 SSD):
- BenchmarkEnqueue/ArenaSize-128KB/MessageSize-8KB/MaxMem-384KB 66501 21606 ns/op 68 B/op 1 allocs/op
- BenchmarkEnqueue/ArenaSize-128KB/MessageSize-8KB/MaxMem-1.25MB 72348 16649 ns/op 67 B/op 1 allocs/op
- BenchmarkEnqueue/ArenaSize-128KB/MessageSize-16KB/MaxMem-1.25MB 39996 33199 ns/op 103 B/op 1 allocs/op
Result On Linux(INTEL Xeon E5-2450 V2 8C 2.5GHZ2,INVENTEC PC3L-10600 16G8,INVENTEC SATA 4T 7.2K*8):
- BenchmarkEnqueue/ArenaSize-128KB/MessageSize-8KB/MaxMem-384KB 126662 12070 ns/op 62 B/op 1 allocs/op
- BenchmarkEnqueue/ArenaSize-128KB/MessageSize-8KB/MaxMem-1.25MB 127393 12097 ns/op 62 B/op 1 allocs/op
- BenchmarkEnqueue/ArenaSize-128KB/MessageSize-16KB/MaxMem-1.25MB 63292 23806 ns/op 92 B/op 1 allocs/op
Conclusion: Based on the above tests, mmap is both satisfied at the write speed and at little memory with very low consumption when running as a sidecar.

Processor

The Processor has 3 core components, which are Consumer, Filter, and Context.

The Consumer is created by the downstream Queue. The consumer has its own read offset and committed offset, which is similar to the offset concept of Spark Streaming.
Due to the particularity of APM data preprocessing, Context is a unique concept in the Satellite filter chain, which supports storing the intermediate event because the intermediate state event also needs to be sent in sometimes.
The Filter is the core data processing part, which is similar to the processor of beats. Due to the context, the upstream/downstream filters would be logically coupling.

Sender

BatchConverter decouples the Processor and Sender by staging the Buffer structure, providing parallelization. But if BatchBuffer is full, the downstream processors would be blocked.
Follower is a real send worker that has a client, such as a gRPC client or Kafka client, and a fallback strategy. Fallback strategy is an interface, we can add more strategies to resolve the abnormal conditions, such as Instability in the network, upgrade the oap cluster.
When sent success, Committed Offset in Queue would plus the number of this batch.

High Performance

The scenario using Satellite is to collect a lot of APM data collection. We guarantee high performance by the following ways.

Shorten transmission path, that means only join 2 components,which are Queue and Processor, between receiving and forwarding.
High Performance Queue. MmapQueue provides a big, fast and persistent queue based on memory mapped file and ring structure.
Processor maintains a linear design, that could be functional processed in one go-routine to avoid too much goroutines switching.

Stability

Stability is a core point in Satellite. Stability can be considered in many ways, such as stable resources cost, stable running and crash recovery.

Stable resource cost

In terms of resource cost, Memory and CPU should be a concern.

In the aspect of the CPU, we keep a sequence structure to avoid a large number of retries occurring when facing network congestion. And Satellite avoids keep pulling when the Queue is empty based on the offset design of Queue.

In the aspect of the Memory, we have guaranteed only one data caching in Satellite, that is Queue. For the queue structure, we also keep the size fixed based on the ring structure to maintain stable Memory cost. Also, MmapQueue is designed for minimizing memory consumption and providing persistence while keeping speed as fast as possible. Maybe supports some strategy to dynamically control the size of MmapQueue to process more extreme conditions in the future.

Stable running

There are many cases of network congestion, such as the network problem on the host node, OAP cluster is under upgrating, and Kafka cluster is unstable. When facing the above cases, Follower would process fallback strategy and block the downstream processes. Once the failure strategy is finished, such that send success or give up this batch, the Follower would process the next batch.

Crash Recovery

The crash recovery only works when the user selects MmapQueue in Gatherer because of persistent file system design. When facing a crash, Reading Offset would be overridden by Committed Offset that ensure the at least once delivery. And Writed Offset would override Writing Offset that ensures the consumer always works properly and avoid encountering uncrossable defective data blocks.

Buffer pool

The Queue is to store fixed structure objects, object buffer pool would be efficient to reuse memory to avoid GC.

ackChan
batch convertor

Some metrics

In Satellite, we should also collect its own monitoring metrics. The following metrics are necessary for Satellite.

cpu
memory
go routine number
gatherer_writing_offset
gatherer_watermark_offset
processor_reading_count
sender_committed_offset
sender_abandoned_count
sender_retry_count

Input and Output

We will reuse this diagram to explain the input and output.

Input
- Because the push-pull mode is both supported, Queue is a core component.
- Queue is designed to be a ring-shaped fixed capacity, that means the oldest data would be overridden by the latest data. If users find data loss, users should raise the ceiling of memory Queue. MmapQueue generally doesn’t face this problem unless the Sender transport is congested.
Ouput
- If the BatchBuffer is full, the processor would be blocked.
- If the Channel is full, the downstream components would be blocked, such as BatchConvertor and Processor.
- When SenderWorker sends failure, the batch data would do a failure strategy that would block pulling data from the Channel. The strategy is a part of Sender,the operation mode is synchronous.
- Once the failure strategy is finished, such that send success or give up this batch, the Sendworker would keep pulling data from the Channel.

Questions

How to avoid keep pulling when the Queue is empty?

If Watermark Offset is less than or equal to Reading Offset, a signal would be sent to the consumer to avoid keep pulling.

Why reusing files in Queue?

The unified model is a ring in Queue, that limits fixed resources cost in memory or disk.In Mmap Queue, reusing files turns the delete operations into an overwrite operations, effectively reducing the creation and deletion behavior in files.

What are the strategies for file creation and deletion in MmapQueue?

As Satellite running, the number of the files in MmapQueue would keep growing until up to the maximum capacity. After this, the old files will be overridden by the new data to avoid file deletion. When the Pod died, all resources were recycled.

Blog: [Video] Another backend storage solution for the APM system

Sat, 21 Nov 2020 00:00:00 +0000

The APM system provides the tracing or metrics for distributed systems or microservice architectures. Back to APM themselves, they always need backend storage to store the necessary massive data. What are the features required for backend storage? Simple, fewer dependencies, widely used query language, and the efficiency could be into your consideration. Based on that, traditional SQL databases (like MySQL) or NoSQL databases would be better choices. However, this topic will present another backend storage solution for the APM system viewing from NewSQL. Taking Apache Skywalking for instance, this talking will share how to make use of Apache ShardingSphere, a distributed database middleware ecosystem to extend the APM system’s storage capability.

As a senior DBA worked at JD.com, the responsibility is to develop the distributed database and middleware, and the automated management platform for database clusters. As a PMC of Apache ShardingSphere, I am willing to contribute to the OS community and explore the area of distributed databases and NewSQL.

Blog: [Video] Improve Apache APISIX observability with Apache SkyWalking

Sat, 21 Nov 2020 00:00:00 +0000

Apache APISIX is a cloud-native microservices API gateway, delivering the ultimate performance, security, open-source and scalable platform for all your APIs and microservices. Apache SkyWalking: an APM(application performance monitor) system, especially designed for microservices, cloud-native and container-based (Docker, Kubernetes, Mesos) architectures. Through the powerful plug-in mechanism of Apache APISIX, Apache Skywalking is quickly supported, so that we can see the complete life cycle of requests from the edge to the internal service. Monitor and manage each request in a visual way, and improve the observability of the service.

Blog: [Video] SourceMarker - Continuous Feedback for Developers

Sat, 21 Nov 2020 00:00:00 +0000

Today’s monitoring solutions are geared towards operational tasks, displaying behavior as time-series graphs inside dashboards and other abstractions. These abstractions are immensely useful but are largely designed for software operators, whose responsibilities require them to think in systems, rather than the underlying source code. This is problematic given that an ongoing trend of software development is the blurring boundaries between building and operating software. This trend makes it increasingly necessary for programming environments to not just support development-centric activities, but operation-centric activities as well. Such is the goal of the feedback-driven development approach. By combining IDE and APM technology, software developers can intuitively explore multiple dimensions of their software simultaneously with continuous feedback about their software from inception to production.

Brandon Fergerson is an open-source software developer who does not regard himself as a specialist in the field of programming, but rather as someone who is a devoted admirer. He discovered the beauty of programming at a young age and views programming as an art and those who do it well to be artists. He has an affinity towards getting meta and combining that with admiration of programming, has found source code analysis to be exceptionally interesting. Lately, his primary focus involves researching and building AI-based pair programming technology.

Blog: [Video] The history of distributed tracing storage

Sat, 21 Nov 2020 00:00:00 +0000

Over the past few years, and coupled with the growing adoption of microservices, distributed tracing has emerged as one of the most commonly used monitoring and troubleshooting methodologies. New tracing tools are increasingly being introduced, driving adoption even further. One of these tools is Apache SkyWalking, a popular open-source tracing, and APM platform. This talk explores the history of the SkyWalking storage module, shows the evolution of distributed tracing storage layers, from the traditional relational database to document-based search engine. I hope that this talk contributes to the understanding of history and also that it helps to clarify the different types of storage that are available to organizations today.

Hongtao Gao is the engineer of tetrate.io and the former Huawei Cloud expert. One of PMC members of Apache SkyWalking and participates in some popular open-source projects such as Apache ShardingSphere and Elastic-Job. He has an in-depth understanding of distributed databases, container scheduling, microservices, ServicMesh, and other technologies.

Blog: [Video] Apache grows in China

Fri, 20 Nov 2020 00:00:00 +0000

Sheng Wu is a founding engineer at tetrate.io, leads the observability for service mesh and hybrid cloud. A searcher, evangelist, and developer in the observability, distributed tracing, and APM. He is a member of the Apache Software Foundation. Love open source software and culture. Created the Apache SkyWalking project and being its VP and PMC member. Co-founder and PMC member of Apache ShardingSphere. Also as a PMC member of Apache Incubator and APISIX. He is awarded as Microsoft MVP, Alibaba Cloud MVP, Tencent Cloud TVP.

In the Apache FY2020 report, China is on the top of the download statistics. More China initiated projects joined the incubator, and some of them graduated as the Apache TLP. Sheng joined the Apache community since 2017, in the past 3 years, he witnessed the growth of the open-source culture and Apache way in China. Many developers have joined the ASF as new contributors, committers, foundation members. Chinese enterprises and companies paid more attention to open source contributions, rather than simply using the project like before. In the keynote, he would share the progress about China embracing the Apache culture, and willing of enhancing the whole Apache community.

Blog: Features in SkyWalking 8.2: Browser Side Monitoring; Query Traces by Tags; Meter Analysis Language

Thu, 29 Oct 2020 00:00:00 +0000

Author: Zhenxu Ke, Sheng Wu, Hongtao Gao, and Tevah Platt. tetrate.io
Original link, Tetrate.io blog
Oct. 29th, 2020

Apache SkyWalking, the observability platform, and open-source application performance monitor (APM) project, today announced the general availability of its 8.2 release. The release extends Apache SkyWalking’s functionalities and monitoring boundary to the browser side.

Background

SkyWalking is an observability platform and APM tool that works with or without a service mesh, providing automatic instrumentation for microservices, cloud-native and container-based applications. The top-level Apache project is supported by a global community and is used by Alibaba, Huawei, Tencent, Baidu, ByteDance, and scores of others.

Browser side monitoring

APM helps SRE and Engineering teams to diagnose system failures, or optimize the systems before they become intolerably slow. But is it enough to always make the users happy?

In 8.2.0, SkyWalking extends its monitoring boundary to the browser side, e.g., Chrome, or the network between Chrome and the backend service, or the codes running in the browser. With this, not only can we monitor the backend services and requests sent by the browser as usual, but also the front end rendering speed, error logs, etc., which are the most efficient metrics for capturing the experiences of our end users. (This does not currently extend to IoT devices, but this feature moves SkyWalking a step in that direction).

What’s more, SkyWalking browser monitoring also provides data about how the users use products, such as PV(page views), UV(unique visitors), top N PV(page views), etc., which can give a product team clues for optimizing their products.

Query traces by tags

In SkyWalking’s Span data model, there are many important fields that are already indexed and can be queried by the users, but for the sake of performance, querying by Span tags was not supported until now. In SkyWalking 8.2.0, we allow users to query traces by specified tags, which is extremely useful. For example, SRE engineers running tests on the product environment can tag the synthetic traffic and query by this tag later.

Meter Analysis Language

In 8.2.0, the meter system provides a functional analysis language called MAL(Meter Analysis Language) that allows users to analyze and aggregate meter data in the OAP streaming system. The result of an expression can be ingested by either the agent analyzer or OpenTelemetry/Prometheus analyzer.

Composite Alert Rules

Alerting is a good way to discover system failures in time. A common problem is that we configure too many triggers just to avoid missing any possible issue. Nobody likes to be woken up by alert messages at midnight, only to find out that the trigger is too sensitive. These kinds of alerts become noisy and don’t help at all.

In 8.2.0, users can now configure composite alert rules, where composite rules take multiple metrics dimensions into account. With composite alert rules, we can leverage as many metrics as needed to more accurately determine whether there’s a real problem or just an occasional glitch.

Common scenarios like successful rate < 90% but there are only 1~2 requests can now be resolved by a composite rule, such as traffic(calls per minute) > n && successful rate < m%.

Other Notable Enhancements

The agent toolkit exposes some APIs for users to send customizable metrics.
The agent exclude_plugins allows you to exclude some plugins; mount enables you to load a new set of plugins.
More than 10 new plugins have been contributed to the agent.
The alert system natively supports sending alert messages to Slack, WeChat, DingTalk.

Additional Resources

Read more about the SkyWalking 8.2 release highlights.
Get more SkyWalking updates on Twitter.

Blog: Observability at Scale: SkyWalking it is

Tue, 11 Aug 2020 00:00:00 +0000

Author: Sheng Wu
Original link, Tetrate.io blog

SkyWalking, a top-level Apache project, is the open source APM and observability analysis platform that is solving the problems of 21st-century systems that are increasingly large, distributed, and heterogenous. It’s built for the struggles system admins face today: To identify and locate needles in a haystack of interdependent services, to get apples-to-apples metrics across polyglot apps, and to get a complete and meaningful view of performance.

SkyWalking is a holistic platform that can observe microservices on or off a mesh, and can provide consistent monitoring with a lightweight payload.

Let’s take a look at how SkyWalking evolved to address the problem of observability at scale, and grew from a pure tracing system to a feature-rich observability platform that is now used to analyze deployments that collect tens of billions of traces per day.

Designing for scale

When SkyWalking was first initialized back in 2015, its primary use case was monitoring the first-generation distributed core system of China Top Telecom companies, China Unicom and China Mobile. In 2013-2014, the telecom companies planned to replace their old traditional monolithic applications with a distributed system. Supporting a super-large distributed system and scaleablity were the high-priority design goals from Day one. So, what matters at scale?

Pull vs. push

Pull and push modes relate to the direction of data flow. If the agent collects data and pushes them to the backend for further analysis, we call it “push” mode. Debate over pull vs. push has gone on for a long time. The key for an observability system is to minimize the cost of the agent, and to be generally suitable for different kinds of observability data.

The agent would send the data out a short period after it is collected. Then, we would have less concern about overloading the local cache. One typical case would be endpoint (URI of HTTP, service of gRPC) metrics. Any service could easily have hundreds, even thousands of endpoints. An APM system must have these metrics analysis capabilities.

Furthermore, metrics aren’t the only thing in the observability landscape; traces and logs are important too. SkyWalking is designed to provide a 100% sampling rate tracing capability in the production environment. Clearly, push mode is the only solution.

At the same time, using push mode natively doesn’t mean SkyWalking can’t do data pulling. In recent 8.x releases, SkyWalking supports fetching data from Prometheus-instrumented services for reducing the Non-Recurring Engineering of the end users. Also, pull mode is popular in the MQ based transport, typically as a Kafka consumer. The SkyWalking agent side uses the push mode, and the OAP server uses the pull mode.

The conclusion: push mode is the native way, but pull mode works in some special cases too.

Metrics analysis isn’t just mathematical calculation

Metrics rely on mathematical theories and calculations. Percentile is a good measure for identifying the long tail issue, and reasonable average response time and successful rate are good SLO(s). But those are not all. Distributed tracing provides not just traces with detailed information, but high values metrics that can be analyzed.

The service topology map is required from Ops and SRE teams for the NOC dashboard and confirmation of system data flow. SkyWalking uses the STAM (Streaming Topology Analysis Method) to analyze topology from the traces, or based on ALS (Envoy Access Log Service) in the service mesh environment. This topology and metrics of nodes (services) and lines (service relationships) can’t be pulled from simple metrics SDKs.

As with fixing the limitation of endpoint metrics collection, SkyWalking needs to do endpoint dependency analysis from trace data too. Endpoint dependency analysis provides more important and specific information, including upstream and downstream. Those dependency relationships and metrics help the developer team to locate the boundaries of a performance issue, to specific code blocks.

Pre-calculation vs. query stage calculation?

Query stage calculation provides flexibility. Pre-calculation, in the analysis stage, provides better and much more stable performance. Recall our design principle: SkyWalking targets a large-scale distributed system. Query stage calculation was very limited in scope, and most metrics calculations need to be pre-defined and pre-calculated. The key of supporting large datasets is reducing the size of datasets in the design level. Pre-calculation allows the original data to be merged into aggregated results downstream, to be used in a query or even for an alert check.

TTL of metrics is another important business enabler. With the near linear performance offered by queries because of pre-calculation, with a similar query infrastructure, organizations can offer higher TTL, thereby providing extended visibility of performance.

Speaking of alerts, query-stage calculation also means the alerting query is required to be based on the query engine. But in this case, when the dataset increasing, the query performance could be inconsistent. The same thing happens in a different metrics query.

Cases today

Today, SkyWalking is monitoring super large-scale distributed systems in many large enterprises, including Alibaba, Huawei, Tencent, Baidu, China Telecom, and various banks and insurance companies. The online service companies have more traffic than the traditional companies, like banks and telecom suppliers.

SkyWalking is the observability platform used for a variety of use cases for distributed systems that are super-large by many measures:

Lagou.com, an online job recruitment platform
- SkyWalking is observing >100 services, 500+ JVM instances
- SkyWalking collects and analyzes 4+ billion traces per day to analyze performance data, including metrics of 300k+ endpoints and dependencies
- Monitoring >50k traffic per second in the whole cluster
Yonghui SuperMarket, online service
- SkyWalking analyzes at least 10+ billion (3B) traces with metrics per day
- SkyWalking’s second, smaller deployment, analyzes 200+ million traces per day
Baidu, internet and AI company, Kubernetes deployment
- SkyWalking collects 1T+ traces a day from 1,400+ pods of 120+ services
- Continues to scale out as more services are added
Beike Zhaofang(ke.com), a Chinese online property brokerage backed by Tencent Holdings and SoftBank Group
- Has used SkyWalking from its very beginning, and has two members in the PMC team.
- Deployments collect 16+ billion traces per day
Ali Yunxiao, DevOps service on the Alibaba Cloud,
- SkyWalking collects and analyzes billions of spans per day
- SkyWalking keeps AliCloud’s 45 services and ~300 instances stable
A department of Alibaba TMall, one of the largest business-to-consumer online retailers, spun off from Taobao
- A customized version of SkyWalking monitors billions of traces per day
- At the same time, they are building a load testing platform based on SkyWalking’s agent tech stack, leveraging its tracing and context propagation cabilities

Conclusion

SkyWalking’s approach to observability follows these principles:

Understand the logic model: don’t treat observability as a mathematical tool.
Identify dependencies first, then their metrics.
Scaling should be accomplished easily and natively.
Maintain consistency across different architectures, and in the performance of APM itself.

Resources

Read about the SkyWalking 8.1 release highlights.
Get more SkyWalking updates on Twitter.
Sign up to hear more about SkyWalking and observability from Tetrate.

Blog: Features in SkyWalking 8.1: SpringSleuth metrics, endpoint dependency detection, Kafka transport traces and metrics

Mon, 03 Aug 2020 00:00:00 +0000

Author: Sheng Wu, Hongtao Gao, and Tevah Platt(Tetrate)
Original link, Tetrate.io blog

Apache SkyWalking, the observability platform, and open-source application performance monitor (APM) project, today announced the general availability of its 8.1 release that extends its functionalities and provides a transport layer to maintain the lightweight of the platform that observes data continuously.

Background

Transport traces

For a long time, SkyWalking has used gRPC and HTTP to transport traces, metrics, and logs. They provide good performance and are quite lightweight, but people kept asking about the MQ as a transport layer because they want to keep the observability data continuously as much as possible. From SkyWalking’s perspective, the MQ based transport layer consumes more resources required in the deployment and the complexity of deployment and maintenance but brings more powerful throughput capacity between the agent and backend.

In 8.1.0, SkyWalking officially provides the typical MQ implementation, Kafka, to transport all observability data, including traces, metrics, logs, and profiling data. At the same time, the backend can support traditional gRPC and HTTP receivers, with the new Kafka consumer at the same time. Different users could choose the transport layer(s) according to their own requirements. Also, by referring to this implementation, the community could contribute various transport plugins for Apache Pulsar, RabbitMQ.

Automatic endpoint dependencies detection

The 8.1 SkyWalking release offers automatic detection of endpoint dependencies. SkyWalking has long offered automatic endpoint detection, but endpoint dependencies, including upstream and downstream endpoints, are critical for Ops and SRE teams’ performance analysis. The APM system is expected to detect the relationships powered by the distributed tracing. While SkyWalking has been designed to include this important information at the beginning the latest 8.1 release offers a cool visualization about the dependency and metrics between dependent endpoints. It provides a new drill-down angle from the topology. Once you have the performance issue from the service level, you could check on instance and endpoint perspectives:

SpringSleuth metrics detection

In the Java field, the Spring ecosystem is one of the most widely used. Micrometer, the metrics API lib included in the Spring Boot 2.0, is now adopted by SkyWalking’s native meter system APIs and agent. For applications using Micrometer with the SkyWalking agent installed, all Micrometer collected metrics could then be shipped into SkyWalking OAP. With some configurations in the OAP and UI, all metrics are analyzed and visualized in the SkyWalking UI, with all other metrics detected by SkyWalking agents automatically.

Notable enhancements

The Java agent core is enhanced in this release. It could work better in the concurrency class loader case and is more compatible with another agent solution, such as Alibaba’s Arthas.

With the logic endpoint supported, the local span can be analyzed to get metrics. One span could carry the raw data of more than one endpoint’s performance.
GraphQL, InfluxDB Java Client, and Quasar fiber libs are supported to be observed automatically.
Kubernetes Configmap can now for the first time be used as the dynamic configuration center– a more cloud-native solution for k8s deployment environments.
OAP supports health checks, especially including the storage health status. If the storage (e.g., ElasticSearch) is not available, you could get the unhealth status with explicit reasons through the health status query.
Opencensus receiver supports ingesting OpenTelemetry/OpenCensus agent metrics by meter-system.

Additional resources

Read more about the SkyWalking 8.1 release highlights.
Read more about SkyWalking from Tetrate on our blog.
Get more SkyWalking updates on Twitter.
Sign up to hear more about SkyWalking and observability from Tetrate.

Blog: The Apdex Score for Measuring Service Mesh Health

Sun, 26 Jul 2020 00:00:00 +0000

Author: Srinivasan Ramaswamy, tetrate
Original link, Tetrate.io blog

Asking `How are you` is more profound than `What are your symptoms`

Background

Recently I visited my preferred doctor. Whenever I visit, the doctor greets me with a series of light questions: How’s your day? How about the week before? Any recent trips? Did I break my cycling record? How’s your workout regimen? _Finally _he asks, “Do you have any problems?” On those visits when I didn’t feel ok, I would say something like, “I’m feeling dull this week, and I’m feeling more tired towards noon…." It’s at this point that he takes out his stethoscope, his pulse oximeter, and blood pressure apparatus. Then, if he feels he needs a more in-depth insight, he starts listing out specific tests to be made.

When I asked him if the first part of the discussion was just an ice-breaker, he said, “That’s the essential part. It helps me find out how you feel, rather than what your symptoms are." So, despite appearances, our opening chat about life helped him structure subsequent questions on symptoms, investigations and test results.

On the way back, I couldn’t stop asking myself, “Shouldn’t we be managing our mesh this way, too?"

If I strike parallels between my own health check and a health check, “tests” would be log analysis, “investigations” would be tracing, and “symptoms” would be the traditional RED (Rate, Errors and Duration) metrics. That leaves the “essential part,” which is what we are talking about here: the Wellness Factor, primarily the health of our mesh.

Health in the context of service mesh

We can measure the performance of any observed service through RED metrics. RED metrics offer immense value in understanding the performance, reliability, and throughput of every service. Compelling visualizations of these metrics across the mesh make monitoring the entire mesh standardized and scalable. Also, setting alerts based on thresholds for each of these metrics helps to detect anomalies as and when they arise.

To establish the context of any service and observe them, it’s ideal to visualize the mesh as a topology.

A topology visualization of the mesh not only allows for picking any service and watching its metrics, but also gives vital information about service dependencies and the potential impact of a given service on the mesh.

While RED metrics of each service offer tremendous insights, the user is more concerned with the overall responsiveness of the mesh rather than each of these services in isolation.

To describe the performance of any service, right from submitting the request to receiving a completed http response, we’d be measuring the user’s perception of responsiveness. This measure of response time compared with a set threshold is called Apdex. This Apdex is an indicator of the health of a service in the mesh.

Apdex

Apdex is a measure of response time considered against a set threshold**. **It is the ratio of satisfactory response times and unsatisfactory response times to total response times.

Apdex is an industry standard to measure the satisfaction of users based on the response time of applications and services. It measures how satisfied your users are with your services, as traditional metrics such as average response time could get skewed quickly.

Satisfactory response time indicates the number of times when the roundtrip response time of a particular service was less than this threshold. Unsatisfactory response time while meaning the opposite, is further categorized as Tolerating and Frustrating. Tolerating accommodates any performance that is up to four times the threshold, and anything over that or any errors encountered is considered Frustrating. The threshold mentioned here is an ideal roundtrip performance that we expect from any service. We could even start with an organization-wide limit of say, 500ms.

The Apdex score is a ratio of satisfied and tolerating requests to the total requests made.

Each satisfied request counts as one request, while each tolerating request counts as half a satisfied request.

An Apdex score takes values from 0 to 1, with 0 being the worst possible score indicating that users were always frustrated, and ‘1’ as the best possible score (100% of response times were Satisfactory).

A percentage representation of this score also serves as the Health Indicator of the service.

The Math

The actual computation of this Apdex score is achieved through the following formula.

		            SatisfiedCount +  ( ToleratingCount / 2 )

Apdex Score  =  ------------------------------------------------------

                                TotalSamples

A percentage representation of this score is known as the Health Indicator of a service.

Example Computation

During a 2-minute period, a host handles 200 requests.

The Apdex threshold T = 0.5 seconds (500ms).

170 of the requests were handled within 500ms, so they are classified as Satisfied.
20 of the requests were handled between 500ms and 2 seconds (2000 ms), so they are classified as Tolerating.
The remaining 10 were not handled properly or took longer than 2 seconds, so they are classified as Frustrated.

The resulting Apdex score is 0.9: (170 + (20/2))/200 = 0.9.

The next level

At the next level, we can attempt to improve our topology visualization by coloring nodes based on their health. Also, we can include health as a part of the information we show when the user taps on a service.

Apdex specifications recommend the following Apdex Quality Ratings by classifying Apdex Score as Excellent (0.94 - 1.00), Good (0.85 - 0.93), Fair (0.70 - 0.84), Poor (0.50 - 0.69) and Unacceptable (0.00 - 0.49).

To visualize this, let’s look at our topology using traffic light colors, marking our nodes as Healthy, At-Risk and Unhealthy, where Unhealthy indicates health that falls below 80%. A rate between 80% and 95% indicates At-Risk, and health at 95% and above is termed Healthy.

Let’s incorporate this coloring into our topology visualization and take its usability to the next level. If implemented, we will be looking at something like this.

Moving further

Apdex provides tremendous visibility into customer satisfaction on the responsiveness of our services. Even more, by extending the implementation to the edges calling this service we get further insight into the health of the mesh itself.

Two services with similar Apdex scores offer the same customer satisfaction to the customer. However, the size of traffic that flows into the service can be of immense help in prioritizing between services to address. A service with higher traffic flow is an indication that this experience is impacting a significant number of users on the mesh.

While health relates to a service, we can also analyze the interactions between two services and calculate the health of the interaction. This health calculation of every interaction on the mesh helps us establish a critical path, based on the health of all interactions in the entire topology.

In a big mesh, showing traffic as yet another number will make it more challenging to visualize and monitor. We can, with a bit of creativity, improve the entire visualization by rendering the edges that connect services with different thickness depending on the throughput of the service.

An unhealthy service participating in a high throughput transaction could lead to excessive consumption of resources. On the other hand, this visualization also offers a great tip to maximize investment in tuning services.

Tuning service that is a part of a high throughput transaction offers exponential benefits when compared to tuning an occasionally used service.

If we look at implementing such a visualization, which includes the health of interactions and throughput of such interactions, we would be looking at something like below :

The day is not far

These capabilities are already available to users today as one of the UI features of Tetrate’s service mesh platform, using the highly configurable and performant observability and performance management framework: Apache SkyWalking (https://skywalking.apache.org), which monitors traffic across the mesh, aggregates RED metrics for both services and their interactions, continuously computes and monitors health of the services, and enables users to configure alerts and notifications when services cross specific thresholds, thereby having a comprehensive health visibility of the mesh.

With such tremendous visibility into our mesh performance, the day is not far when we at our NOC (Network Operations Center) for the mesh have this topology as our HUD (Heads Up Display).

This HUD, with the insights and patterns gathered over time, would predict situations and proactively prompt us on potential focus areas to improve customer satisfaction.

The visualization with rich historical data can also empower the Network Engineers to go back in time and look at the performance of the mesh on a similar day in the past.

An earnest implementation of such a visualization would be something like below :

To conclude

With all the discussion so far, the health of a mesh is more about how our users feel, and what we can proactively do as service providers to sustain, if not enhance, the experience of our users.

As the world advances toward personalized medicine, we’re not far from a day when my doctor will text me: “How about feasting yourself with ice cream today and take the Gray Butte Trail to Mount Shasta!” Likewise, we can do more for our customers by having better insight into their overall wellness.

Tetrate’s approach to “service mesh health” is not only to offer management, monitoring and support but to make infrastructure healthy from the start to reduce the probability of incidents. Powered by the Istio, Envoy, and SkyWalking, Tetrate’s solutions enable consistent end-to-end observability, runtime security, and traffic management for any workload in any environment.

Our customers deserve healthy systems! Please do share your thoughts on making service mesh an exciting and robust experience for our customers.

References

Blog: Apache SkyWalking: Use Profiling to Fix the Blind Spot of Distributed Tracing

Mon, 13 Apr 2020 00:00:00 +0000

This post originally appears on The New Stack

This post introduces a way to automatically profile code in production with Apache SkyWalking. We believe the profile method helps reduce maintenance and overhead while increasing the precision in root cause analysis.

Limitations of the Distributed Tracing

In the early days, metrics and logging systems were the key solutions in monitoring platforms. With the adoption of microservice and distributed system-based architecture, distributed tracing has become more important. Distributed tracing provides relevant service context, such as system topology map and RPC parent-child relationships.

Some claim that distributed tracing is the best way to discover the cause of performance issues in a distributed system. It’s good at finding issues at the RPC abstraction, or in the scope of components instrumented with spans. However, it isn’t that perfect.

Have you been surprised to find a span duration longer than expected, but no insight into why? What should you do next? Some may think that the next step is to add more instrumentation, more spans into the trace, thinking that you would eventually find the root cause, with more data points. We’ll argue this is not a good option within a production environment. Here’s why:

There is a risk of application overhead and system overload. Ad-hoc spans measure the performance of specific scopes or methods, but picking the right place can be difficult. To identify the precise cause, you can “instrument” (add spans to) many suspicious places. The additional instrumentation costs more CPU and memory in the production environment. Next, ad-hoc instrumentation that didn’t help is often forgotten, not deleted. This creates a valueless overhead load. In the worst case, excess instrumentation can cause performance problems in the production app or overload the tracing system.
The process of ad-hoc (manual) instrumentation usually implies at least a restart. Trace instrumentation libraries, like Zipkin Brave, are integrated into many framework libraries. To instrument a method’s performance typically implies changing code, even if only an annotation. This implies a re-deploy. Even if you have the way to do auto instrumentation, like Apache SkyWalking, you still need to change the configuration and reboot the app. Otherwise, you take the risk of GC caused by hot dynamic instrumentation.
Injecting instrumentation into an uninstrumented third party library is hard and complex. It takes more time and many won’t know how to do this.
Usually, we don’t have code line numbers in the distributed tracing. Particularly when lambdas are in use, it can be difficult to identify the line of code associated with a span. Regardless of the above choices, to dive deeper requires collaboration with your Ops or SRE team, and a shared deep level of knowledge in distributed tracing.

Regardless of the above choices, to dive deeper requires collaboration with your Ops or SRE team, and a shared deep level of knowledge in distributed tracing.

Profiling in Production

Introduction

To reuse distributed tracing to achieve method scope precision requires an understanding of the above limitations and a different approach. We called it PROFILE.

Most high-level languages build and run on a thread concept. The profile approach takes continuous thread dumps. We merge the thread dumps to estimate the execution time of every method shown in the thread dumps. The key for distributed tracing is the tracing context, identifiers active (or current) for the profiled method. Using this trace context, we can weave data harvested from profiling into existing traces. This allows the system to automate otherwise ad-hoc instrumentation. Let’s dig deeper into how profiling works:

We consider a method invocation with the same stack depth and signature (method, line number etc), the same operation. We derive span timestamps from the thread dumps the same operation is in. Let’s put this visually:

Above, represents 10 successive thread dumps. If this method is in dumps 4-8, we assume it started before dump 4 and finished after dump 8. We can’t tell exactly when the method started and stopped. but the timestamps of thread dumps are close enough.

To reduce overhead caused by thread dumps, we only profile methods enclosed by a specific entry point, such as a URI or MVC Controller method. We identify these entry points through the trace context and the APM system.

The profile does thread dump analysis and gives us:

The root cause, precise to the line number in the code.
Reduced maintenance as ad-hoc instrumentation is obviated.
Reduced overload risk caused by ad-hoc instrumentation.
Dynamic activation: only when necessary and with a very clear profile target.

Implementing Precise Profiling with Apache SkyWalking 7

Distributed profiling is built-into Apache SkyWalking application performance monitoring (APM). Let’s demonstrate how the profiling approach locates the root cause of the performance issue.

final CountDownLatchcountDownLatch= new CountDownLatch(2);
 
threadPool.submit(new Task1(countDownLatch));
threadPool.submit(new Task2(countDownLatch));
 
try {
   countDownLatch.await(500, TimeUnit.MILLISECONDS);
} catch (InterruptedExceptione) {
}

Task1 and Task2 have a race condition and unstable execution time: they will impact the performance of each other and anything calling them. While this code looks suspicious, it is representative of real life. People in the OPS/SRE team are not usually aware of all code changes and who did them. They only know something in the new code is causing a problem.

To make matters interesting, the above code is not always slow: it only happens when the condition is locked. In SkyWalking APM, we have metrics of endpoint p99/p95 latency, so, we are easy to find out the p99 of this endpoint is far from the avg response time. However, this is not the same as understanding the cause of the latency. To locate the root cause, add a profile condition to this endpoint: duration greater than 500ms. This means faster executions will not add profiling load.

This is a typical profiled trace segment (part of the whole distributed trace) shown on the SkyWalking UI. We now notice the “service/processWithThreadPool” span is slow as we expected, but why? This method is the one we added the faulty code to. As the UI shows that method, we know the profiler is working. Now, let’s see what the profile analysis result say.

This is the profile analysis stack view. We see the stack element names, duration (include/exclude the children) and slowest methods have been highlighted. It shows clearly, “sun.misc.Unsafe.park” costs the most time. If we look for the caller, it is the code we added: CountDownLatch.await.

The Limitations of the Profile Method

No diagnostic tool can fit all cases, not even the profile method.

The first consideration is mistaking a repeatedly called method for a slow method. Thread dumps are periodic. If there is a loop of calling one method, the profile analysis result would say the target method is slow because it is captured every time in the dump process. There could be another reason. A method called many times can also end up captured in each thread dump. Even so, the profile did what it is designed for. It still helps the OPS/SRE team to locate the code having the issue.

The second consideration is overhead, the impact of repeated thread dumps is real and can’t be ignored. In SkyWalking, we set the profile dump period to at least 10ms. This means we can’t locate method performance issues if they complete in less than 10ms. SkyWalking has a threshold to control the maximum parallel degree as well.

Understanding the above keeps distributed tracing and APM systems useful for your OPS/SRE team.

How to Try This

Everything we discussed, including the Apache SkyWalking Java Agent, profile analysis code, and UI, could be found in our GitHub repository. We hope you enjoyed this new profile method, and love Apache SkyWalking. If so, give us a star on GitHub to encourage us.

SkyWalking 7 has just been released. You can contact the project team through the following channels:

Follow SkyWalking twitter.
Subscribe mailing list: dev@skywalking.apache.org. Send to dev-subscribe@kywalking.apache.org to subscribe to the mail list.

Co-author Sheng Wu is a Tetrate founding engineer and the founder and VP of Apache SkyWalking. He is solving the problem of observability for large-scale service meshes in hybrid and multi-cloud environments.

Adrian Cole works in the Spring Cloud team at VMware, mostly on Zipkin

Han Liu is a tech expert at Lagou. He is an Apache SkyWalking committer

Blog: SkyWalking hits 200 contributors mark

Mon, 20 Jan 2020 00:00:00 +0000

Author: Wu Sheng, tetrate.io, SkyWalking original creator, SkyWalking V.P.
GitHub, Twitter, Linkedin

The SkyWalking project provides distributed tracing, topology map analysis, service mesh telemetry analysis, metrics analysis and a super cool visualization targeting distributed systems in k8s or traditional VM deployments.

The project is widely used in Alibaba, Huawei, Tencent, DiDi, xiaomi, Pingan, China’s top 3 telecom companies (China Mobile, China telecom, China Unicom), airlines, banks and more. It has over 140 company users listed on our powered by page.

Today, we welcome and celebrate reaching 200 code contributors on our main repo. We hereby mark this milestone as official today, : Jan. 20th 2020.

At this great moment, I would like to share SkyWalking’s 4-year open source journey.

I wrote the first line on Nov. 1st, 2015, guiding people to understand a distributed system just as micro-services and distributed architecture were becoming popular. In the first 2 years, I never thought it would become such a big and active community. I didn’t even expect it would be an open source project. Initially, the goal was primarily to teach others about distributed tracing and analysis.

It was a typical open source project in obscurity in its first two years. But people still showed up, asked questions, and tried to improve the project. I got several invitations to share the project at local meetups.All these made me realize people really needed a good open source APM project.

In 2017, I decided to dedicate myself as much as possible to make the project successful, and it became my day job. To be honest, I had no clue about how to do that; at that time in China, it was rare to have this kind of job. So, I began to ask friends around me, “Do you want to collaborate on the open source APM with me?” Most people were busy and gave a clear NO, but two of them agreed to help: Xin Zhang and Yongsheng Peng. We built SkyWalking 3.x and shared the 3.2 release at GOPS Shanghai, China.

It became the first adoption version used in production

Compared to today’s SkyWalking, it was a toy prototype, but it had the same tracing design, protocol and analysis method.

That year the contributor team was 15-20, and the project had obvious potential to expand. I began to consider bringing the project into a worldwide, top-level open source foundation. Thanks to our initial incubator mentors, Michael Semb Wever, William Jiang, and Luke Han, this really worked. At the end of 2017, SkyWalking joined the Apache Incubator, and kept following the Apache Way to build community. More contributors joined the community.

With more people spending time on the project collaborations, including codes, tests, blogs, conference talks, books and uses of the project, a chemical reaction happens. New developers begin to provide bug fixes, new feature requirements and new proposals. At the moment of graduation in spring 2019, the project had 100 contributors. Now, only 9 months later, it’s surged to 200 super quickly. They enhance the project and extend it to frontiers we never imaged: 5 popular language agents, service mesh adoption, CLI tool, super cool visualization. We are even moving on thread profiling, browser performance and Nginx tracing NOW.

Over the whole 4+ years open source journey, we have had supports from leaders in the tracing open source community around the world, including Adrian Cole, William Jiang, Luke Han, Michael Semb Wever, Ben Sigelman, and Jonah Kowall. And we’ve had critical foundations’ help, especially Apache Software Foundation and the Cloud Native Computing Foundation.

Our contributors also have their support from their employers, including, to the best of my knowledge, Alibaba, Huawei, China Mobile, ke.com, DaoCloud, Lizhi.fm, Yonghui Supermarket, and dangdang.com. I also have support from my employers, tetrate.io, Huawei, and OneAPM.

Thanks to our 200+ contributors and the companies behind them. You make this magic happen.

Blog: SkyWalking alarm webhook sharing

Wed, 25 Sep 2019 00:00:00 +0000

Author: Wei Qiang
GitHub

Background

SkyWalking backend provides the alarm function, we can define some Alarm rules, call webhook after the rule is triggered. I share my implementation

Demonstration

SkyWalking alarm UI

dingtalk message body

Introduction

install

go get -u github.com/weiqiang333/infra-skywalking-webhook
cd $GOPATH/src/github.com/weiqiang333/infra-skywalking-webhook/
bash build/build.sh
./bin/infra-skywalking-webhook help

Configuration

main configs file: configs/production.yml
dingtalk:
  p3: token...

Example

./bin/infra-skywalking-webhook --config configs/production.yml --address 0.0.0.0:8000

SkyWalking backend alarm settings

webhooks:
  - http://127.0.0.1:8000/dingtalk

Collaboration

Hope that we can improve together webhook

SkyWalking alarm rules may add more metric names (eg priority name), we can send different channels by locating different levels of alerts (dingtalk / SMS / phone)

Thanks.

Blog: SkyWalking performance in Service Mesh scenario

Fri, 25 Jan 2019 00:00:00 +0000

Author: Hongtao Gao, Apache SkyWalking & ShardingShpere PMC
GitHub, Twitter, Linkedin

Service mesh receiver was first introduced in Apache SkyWalking 6.0.0-beta. It is designed to provide a common entrance for receiving telemetry data from service mesh framework, for instance, Istio, Linkerd, Envoy etc. What’s the service mesh? According to Istio’s explain:

The term service mesh is used to describe the network of microservices that make up such applications and the interactions between them.

As a PMC member of Apache SkyWalking, I tested trace receiver and well understood the performance of collectors in trace scenario. I also would like to figure out the performance of service mesh receiver.

Different between trace and service mesh

Following chart presents a typical trace map:

You could find a variety of elements in it just like web service, local method, database, cache, MQ and so on. But service mesh only collect service network telemetry data that contains the entrance and exit data of a service for now(more elements will be imported soon, just like Database). A smaller quantity of data is sent to the service mesh receiver than the trace.

But using sidecar is a little different.The client requesting “A” that will send a segment to service mesh receiver from “A”’s sidecar. If “A” depends on “B”, another segment will be sent from “A”’s sidecar. But for a trace system, only one segment is received by the collector. The sidecar model splits one segment into small segments, that will increase service mesh receiver network overhead.

Deployment Architecture

In this test, I will pick two different backend deployment. One is called mini unit, consist of one collector and one elasticsearch instance. Another is a standard production cluster, contains three collectors and three elasticsearch instances.

Mini unit is a suitable architecture for dev or test environment. It saves your time and VM resources, speeds up depolyment process.

The standard cluster provides good performance and HA for a production scenario. Though you will pay more money and take care of the cluster carefully, the reliability of the cluster will be a good reward to you.

I pick 8 CPU and 16GB VM to set up the test environment. This test targets the performance of normal usage scenarios, so that choice is reasonable. The cluster is built on Google Kubernetes Engine(GKE), and every node links each other with a VPC network. For running collector is a CPU intensive task, the resource request of collector deployment should be 8 CPU, which means every collector instance occupy a VM node.

Testing Process

Receiving mesh fragments per second(MPS) depends on the following variables.

Ingress query per second(QPS)
The topology of a microservice cluster
Service mesh mode(proxy or sidecar)

In this test, I use Bookinfo app as a demo cluster.

So every request will touch max 4 nodes. Plus picking the sidecar mode(every request will send two telemetry data), the MPS will be QPS * 4 *2.

There are also some important metrics that should be explained

Client Query Latency: GraphQL API query response time heatmap.
Client Mesh Sender: Send mesh segments per second. The total line represents total send amount and the error line is the total number of failed send.
Mesh telemetry latency: service mesh receiver handling data heatmap.
Mesh telemetry received: received mesh telemetry data per second.

Mini Unit

You could find collector can process up to 25k data per second. The CPU usage is about 4 cores. Most of the query latency is less than 50ms. After login the VM on which collector instance running, I know that system load is reaching the limit(max is 8).

According to the previous formula, a single collector instance could process 3k QPS of Bookinfo traffic.

Standard Cluster

Compare to the mini-unit, cluster’s throughput increases linearly. Three instances provide total 80k per second processing power. Query latency increases slightly, but it’s also very small(less than 500ms). I also checked every collector instance system load that all reached the limit. 10k QPS of BookInfo telemetry data could be processed by the cluster.

Conclusion

Let’s wrap them up. There are some important things you could get from this test.

QPS varies by the there variables. The test results in this blog are not important. The user should pick property value according to his system.
Collector cluster’s processing power could scale out.
The collector is CPU intensive application. So you should provide sufficient CPU resource to it.

This blog gives people a common method to evaluate the throughput of Service Mesh Receiver. Users could use this to design their Apache Skywalking backend deployment architecture.