Apache SkyWalking – Release Blog

Blog: Apache SkyWalking 8.4: Logs, VM Monitoring, and Dynamic Configurations at Agent Side

Fri, 05 Feb 2021 00:00:00 +0000

Origin: Tetrate.io blog

The Apache SkyWalking team today announced the 8.4 release is generally available. This release fills the gap between all previous versions of SkyWalking and the logging domain area. The release also advances SkyWalking’s capabilities for infrastructure observability, starting with virtual machine monitoring.

Background

SkyWalking has historically focused on the tracing and metrics fields of observability. As its features for tracing, metrics and service level monitoring have become more and more powerful and stable, the SkyWalking team has started to explore new scenarios covered by observability. Because service performance is reflected in the logs, and is highly impacted by the infrastructure on which it runs, SkyWalking brings these two fields into the 8.4 release. This release blog briefly introduces the two new features as well as some other notable changes.

Logs

Metrics, tracing, and logging are considered the three pillars of observability [1]. SkyWalking had the full features of metrics and tracing prior to 8.4; today, as 8.4 is released, the last piece of the jigsaw is now in place.

Figure 1: Logs Collected By SkyWalking

Figure 2: Logs Collected By SkyWalking

The Java agent firstly provides SDKs to enhance the widely-used logging frameworks, log4j (1.x and 2.x) [2] and logback [3], and send the logs to the SkyWalking backend (OAP). The latter is able to collect logs from wherever the protocol is implemented. This is not a big deal, but when it comes to the correlation between logs and traces, the traditional solution is to print the trace IDs in the logs, and pick the IDs in the error logs to query the related traces. SkyWalking just simplifies the workflow by correlating the logs and traces natively. Navigating between traces and their related logs is as simple as clicking a button.

Figure 3: Correlation Between Logs and Traces

Infrastructure Monitoring

SkyWalking is known as an application performance monitoring tool. One of the most important factors that impacts the application’s performance is the infrastructure on which the application runs. In the 8.4 release, we added the monitoring metrics of virtual machines into the dashboard.

Figure 4: VM Metrics

Fundamental metrics such as CPU Used, Memory Used, Disk Read / Write and Network Usage are available on the dashboard. And as usual, those metrics are also available to be configured as alarm triggers when needed.

Dynamic Configurations at Agent Side

Dynamic configuration at the backend side has long existed in SkyWalking for several versions. Now, it finally comes to the agent side! Prior to 8.4, you’d have to restart the target services when you modify some configuration items of the agent – for instance, sampling rate (agent side), ignorable endpoint paths, etc. Now, say goodbye to rebooting. Modifying configurations is not the only usage of the dynamic configuration mechanism. The latter gives countless possibilities to the agent side in terms of dynamic behaviours, e.g. enabling / disabling plugins, enabling / disabling the whole agent, etc. Just imagine!

Grouped Service Topology

This enhancement is from the UI. SkyWalking backend supports grouping the services by user-defined dimensions. In a real world use case, the services are usually grouped by business group or department. When a developer opens the topology map, out of hundreds of services, he or she may just want to focus on the services in charge. The grouped service topology comes to the rescue: one can now choose to display only services belonging to a specified group.

Figure 5: Grouped Service Topology

Other Notable Enhancements

Agent: resolves domain names to look up backend service IP addresses.
Backend: meter receiver supports meter analysis language (MAL).
Backend: several CVE fixes.
Backend: supports Envoy {AccessLog,Metrics}Service API V3 and adopts MAL.

Additional Resources

Read more about the SkyWalking 8.4 release highlights.
Get more SkyWalking updates on Twitter.

Blog: Features in SkyWalking 8.2: Browser Side Monitoring; Query Traces by Tags; Meter Analysis Language

Thu, 29 Oct 2020 00:00:00 +0000

Author: Zhenxu Ke, Sheng Wu, Hongtao Gao, and Tevah Platt. tetrate.io
Original link, Tetrate.io blog
Oct. 29th, 2020

Apache SkyWalking, the observability platform, and open-source application performance monitor (APM) project, today announced the general availability of its 8.2 release. The release extends Apache SkyWalking’s functionalities and monitoring boundary to the browser side.

Background

SkyWalking is an observability platform and APM tool that works with or without a service mesh, providing automatic instrumentation for microservices, cloud-native and container-based applications. The top-level Apache project is supported by a global community and is used by Alibaba, Huawei, Tencent, Baidu, ByteDance, and scores of others.

Browser side monitoring

APM helps SRE and Engineering teams to diagnose system failures, or optimize the systems before they become intolerably slow. But is it enough to always make the users happy?

In 8.2.0, SkyWalking extends its monitoring boundary to the browser side, e.g., Chrome, or the network between Chrome and the backend service, or the codes running in the browser. With this, not only can we monitor the backend services and requests sent by the browser as usual, but also the front end rendering speed, error logs, etc., which are the most efficient metrics for capturing the experiences of our end users. (This does not currently extend to IoT devices, but this feature moves SkyWalking a step in that direction).

What’s more, SkyWalking browser monitoring also provides data about how the users use products, such as PV(page views), UV(unique visitors), top N PV(page views), etc., which can give a product team clues for optimizing their products.

Query traces by tags

In SkyWalking’s Span data model, there are many important fields that are already indexed and can be queried by the users, but for the sake of performance, querying by Span tags was not supported until now. In SkyWalking 8.2.0, we allow users to query traces by specified tags, which is extremely useful. For example, SRE engineers running tests on the product environment can tag the synthetic traffic and query by this tag later.

Meter Analysis Language

In 8.2.0, the meter system provides a functional analysis language called MAL(Meter Analysis Language) that allows users to analyze and aggregate meter data in the OAP streaming system. The result of an expression can be ingested by either the agent analyzer or OpenTelemetry/Prometheus analyzer.

Composite Alert Rules

Alerting is a good way to discover system failures in time. A common problem is that we configure too many triggers just to avoid missing any possible issue. Nobody likes to be woken up by alert messages at midnight, only to find out that the trigger is too sensitive. These kinds of alerts become noisy and don’t help at all.

In 8.2.0, users can now configure composite alert rules, where composite rules take multiple metrics dimensions into account. With composite alert rules, we can leverage as many metrics as needed to more accurately determine whether there’s a real problem or just an occasional glitch.

Common scenarios like successful rate < 90% but there are only 1~2 requests can now be resolved by a composite rule, such as traffic(calls per minute) > n && successful rate < m%.

Other Notable Enhancements

The agent toolkit exposes some APIs for users to send customizable metrics.
The agent exclude_plugins allows you to exclude some plugins; mount enables you to load a new set of plugins.
More than 10 new plugins have been contributed to the agent.
The alert system natively supports sending alert messages to Slack, WeChat, DingTalk.

Additional Resources

Read more about the SkyWalking 8.2 release highlights.
Get more SkyWalking updates on Twitter.

Blog: Features in SkyWalking 8.1: SpringSleuth metrics, endpoint dependency detection, Kafka transport traces and metrics

Mon, 03 Aug 2020 00:00:00 +0000

Author: Sheng Wu, Hongtao Gao, and Tevah Platt(Tetrate)
Original link, Tetrate.io blog

Apache SkyWalking, the observability platform, and open-source application performance monitor (APM) project, today announced the general availability of its 8.1 release that extends its functionalities and provides a transport layer to maintain the lightweight of the platform that observes data continuously.

Background

Transport traces

For a long time, SkyWalking has used gRPC and HTTP to transport traces, metrics, and logs. They provide good performance and are quite lightweight, but people kept asking about the MQ as a transport layer because they want to keep the observability data continuously as much as possible. From SkyWalking’s perspective, the MQ based transport layer consumes more resources required in the deployment and the complexity of deployment and maintenance but brings more powerful throughput capacity between the agent and backend.

In 8.1.0, SkyWalking officially provides the typical MQ implementation, Kafka, to transport all observability data, including traces, metrics, logs, and profiling data. At the same time, the backend can support traditional gRPC and HTTP receivers, with the new Kafka consumer at the same time. Different users could choose the transport layer(s) according to their own requirements. Also, by referring to this implementation, the community could contribute various transport plugins for Apache Pulsar, RabbitMQ.

Automatic endpoint dependencies detection

The 8.1 SkyWalking release offers automatic detection of endpoint dependencies. SkyWalking has long offered automatic endpoint detection, but endpoint dependencies, including upstream and downstream endpoints, are critical for Ops and SRE teams’ performance analysis. The APM system is expected to detect the relationships powered by the distributed tracing. While SkyWalking has been designed to include this important information at the beginning the latest 8.1 release offers a cool visualization about the dependency and metrics between dependent endpoints. It provides a new drill-down angle from the topology. Once you have the performance issue from the service level, you could check on instance and endpoint perspectives:

SpringSleuth metrics detection

In the Java field, the Spring ecosystem is one of the most widely used. Micrometer, the metrics API lib included in the Spring Boot 2.0, is now adopted by SkyWalking’s native meter system APIs and agent. For applications using Micrometer with the SkyWalking agent installed, all Micrometer collected metrics could then be shipped into SkyWalking OAP. With some configurations in the OAP and UI, all metrics are analyzed and visualized in the SkyWalking UI, with all other metrics detected by SkyWalking agents automatically.

Notable enhancements

The Java agent core is enhanced in this release. It could work better in the concurrency class loader case and is more compatible with another agent solution, such as Alibaba’s Arthas.

With the logic endpoint supported, the local span can be analyzed to get metrics. One span could carry the raw data of more than one endpoint’s performance.
GraphQL, InfluxDB Java Client, and Quasar fiber libs are supported to be observed automatically.
Kubernetes Configmap can now for the first time be used as the dynamic configuration center– a more cloud-native solution for k8s deployment environments.
OAP supports health checks, especially including the storage health status. If the storage (e.g., ElasticSearch) is not available, you could get the unhealth status with explicit reasons through the health status query.
Opencensus receiver supports ingesting OpenTelemetry/OpenCensus agent metrics by meter-system.

Additional resources

Read more about the SkyWalking 8.1 release highlights.
Read more about SkyWalking from Tetrate on our blog.
Get more SkyWalking updates on Twitter.
Sign up to hear more about SkyWalking and observability from Tetrate.

Blog: SkyWalking v6 is Service Mesh ready

Wed, 05 Dec 2018 00:00:00 +0000

Original link, Tetrate.io blog

Context

The integration of SkyWalking and Istio Service Mesh yields an essential open-source tool for resolving the chaos created by the proliferation of siloed, cloud-based services.

Apache SkyWalking is an open, modern performance management tool for distributed services, designed especially for microservices, cloud native and container-based (Docker, K8s, Mesos) architectures. We at Tetrate believe it is going to be an important project for understanding the performance of microservices. The recently released v6 integrates with Istio Service Mesh and focuses on metrics and tracing. It natively understands the most common language runtimes (Java, .Net, and NodeJS). With its new core code, SkyWalking v6 also supports Istrio telemetry data formats, providing consistent analysis, persistence, and visualization.

SkyWalking has evolved into an Observability Analysis Platform that enables observation and monitoring of hundreds of services all at once. It promises solutions for some of the trickiest problems faced by system administrators using complex arrays of abundant services: Identifying why and where a request is slow, distinguishing normal from deviant system performance, comparing apples-to-apples metrics across apps regardless of programming language, and attaining a complete and meaningful view of performance.

SkyWalking History

Launched in China by Wu Sheng in 2015, SkyWalking started as just a distributed tracing system, like Zipkin, but with auto instrumentation from a Java agent. This enabled JVM users to see distributed traces without any change to their source code. In the last two years, it has been used for research and production by more than 50 companies. With its expanded capabilities, we expect to see it adopted more globally.

What’s new

Service Mesh Integration

Istio has picked up a lot of steam as the framework of choice for distributed services. Based on all the interest in the Istio project, and community feedback, some SkyWalking (P)PMC members decided to integrate with Istio Service Mesh to move SkyWalking to a higher level.

So now you can use Skywalking to get metrics and understand the topology of your applications. This works not just for Java, .NET and Node using our language agents, but also for microservices running under the Istio service mesh. You can get a full topology of both kinds of applications.

Observability analysis platform

With its roots in tracing, SkyWalking is now transitioning into an open-standards based Observability Analysis Platform, which means the following:

It can accept different kinds and formats of telemetry data from mesh like Istio telemetry.
Its agents support various popular software technologies and frameworks like Tomcat, Spring, Kafka. The whole supported framework list is here.
It can accept data from other compliant sources like Zipkin-formatted traces reported from Zipkin, Jaeger, or OpenCensus clients.

SkyWalking is logically split into four parts: Probes, Platform Backend, Storage and UI:

There are two kinds of probes:

Language agents or SDKs following SkyWalking across-thread propagation formats and trace formats, run in the user’s application process.
The Istio mixer adaptor, which collects telemetry from the Service Mesh.

The platform backend provides gRPC and RESTful HTTP endpoints for all SkyWalking-supported trace and metric telemetry data. For example, you can stream these metrics into an analysis system.

Storage supports multiple implementations such as ElasticSearch, H2 (alpha), MySQL, and Apache ShardingSphere for MySQL Cluster. TiDB will be supported in next release.

SkyWalking’s built-in UI with a GraphQL endpoint for data allows intuitive, customizable integration.

Some examples of SkyWalking’s UI:

Observe a Spring app using the SkyWalking JVM-agent

Observe on Istio without any agent, no matter what langugage the service is written in

See fine-grained metrics like request/Call per Minute, P99/95/90/75/50 latency, avg response time, heatmap

Service dependencies and metrics

Service Focused

At Tetrate, we are focused on discovery, reliability, and security of your running services. This is why we are embracing Skywalking, which makes service performance observable.

Behind this admittedly cool UI, the aggregation logic is very easy to understand, making it easy to customize SkyWalking in its Observability Analysis Language (OAL) script.

We’ll post more about OAL for developers looking to customize SkyWalking, and you can read the official OAL introduction document.

Scripts are based on three core concepts:

Service represents a group of workloads that provide the same behaviours for incoming requests. You can define the service name whether you are using instrument agents or SDKs. Otherwise, SkyWalking uses the name you defined in the underlying platform, such as Istio.
Service Instance Each workload in the Service group is called an instance. Like Pods in Kubernetes, it doesn’t need to be a single OS process. If you are using an instrument agent, an instance does map to one OS process.
Endpoint is a path in a certain service that handles incoming requests, such as HTTP paths or a gRPC service + method. Mesh telemetry and trace data are formatted as source objects (aka scope). These are the input for the aggregation, with the script describing how to aggregate, including input, conditions, and the resulting metric name.

Core Features

The other core features in SkyWalking v6 are:

Service, service instance, endpoint metrics analysis.
Consistent visualization in Service Mesh and no mesh.
Topology discovery, Service dependency analysis.
Distributed tracing.
Slow services and endpoints detected.
Alarms.

Of course, SkyWalking has some more upgrades from v5, such as:

ElasticSearch 6 as storage is supported.
H2 storage implementor is back.
Kubernetes cluster management is provided. You don’t need Zookeeper to keep the backend running in cluster mode.
Totally new alarm core. Easier configuration.
More cloud native style.
MySQL will be supported in the next release.

Please: Test and Provide Feedback!

We would love everyone to try to test our new version. You can find everything you need in our Apache repository,read the document for further details. You can contact the project team through the following channels:

Submit an issue on GitHub repository
Mailing list: dev@skywalking.apache.org . Send to dev-subscribe@kywalking.apache.org to subscribe the mail list.
Gitter
Project twitter

Oh, and one last thing! If you like our project, don’t forget to give us a star on GitHub.

Apache SkyWalking – Release Blog

Blog: Apache SkyWalking 8.4: Logs, VM Monitoring, and Dynamic Configurations at Agent Side

Background

Logs

Infrastructure Monitoring

Dynamic Configurations at Agent Side

Grouped Service Topology

Other Notable Enhancements

Links

Additional Resources

Blog: Features in SkyWalking 8.2: Browser Side Monitoring; Query Traces by Tags; Meter Analysis Language

Background

Browser side monitoring

Query traces by tags

Meter Analysis Language

Composite Alert Rules

Other Notable Enhancements

Additional Resources

Blog: Features in SkyWalking 8.1: SpringSleuth metrics, endpoint dependency detection, Kafka transport traces and metrics

Background

Transport traces

Automatic endpoint dependencies detection

SpringSleuth metrics detection

Notable enhancements

Additional resources

Blog: SkyWalking v6 is Service Mesh ready

Context

SkyWalking History

What’s new

Service Mesh Integration

Observability analysis platform

Service Focused

Core Features

Please: Test and Provide Feedback!