dt-obs-tracing

---
name: dt-obs-tracing
description: >-
license: Apache-2.0
---

# Application Tracing Skill

## Overview

Distributed traces in Dynatrace consist of spans - building blocks representing units of work. With Traces in Grail, every span is accessible via DQL with full-text searchability on all attributes. This skill covers trace fundamentals, common analysis patterns, and span-type specific queries.

---

## Use Cases

### 1. Investigate Slow Requests
- **Goal:** Find and diagnose requests exceeding a latency threshold
- **Trigger:** "slow requests", "high latency", "p99 response time", "find traces over 5 seconds"
- **Done:** List of slow traces with duration, endpoint, service, and trace IDs for drilldown

### 2. Analyze Request Failures
- **Goal:** Identify failed requests, failure reasons, and exception patterns
- **Trigger:** "failed spans", "HTTP 500 errors", "exception analysis", "failure rate by service"
- **Done:** Failure breakdown by reason (HTTP code, exception, gRPC status) with exemplar traces

### 3. Map Service Dependencies
- **Goal:** Understand service-to-service communication patterns and external API calls
- **Trigger:** "service dependencies", "what services does X call", "outgoing HTTP calls"
- **Done:** Dependency map showing call counts, latency, and error rates between services

---

## Core Concepts

### Understanding Traces and Spans

**Spans** represent logical units of work in distributed traces:
- HTTP requests, RPC calls, database operations
- Messaging system interactions
- Internal function invocations
- Custom instrumentation points

**Span kinds**:
- `span.kind: server` - Incoming call to a service
- `span.kind: client` - Outgoing call from a service
- `span.kind: consumer` - Incoming message consumption call to a service
- `span.kind: producer` - Outgoing message production call from a service
- `span.kind: internal` - Internal operation within a service

**Root spans**: A request root span (`request.is_root_span == true`) represents an incoming call to a service. Use this to analyze end-to-end request performance.

### Key Trace Attributes

Essential attributes for trace analysis:

| Attribute | Description |
|-----------|-------------|
| `trace.id` | Unique trace identifier |
| `span.id` | Unique span identifier |
| `span.parent_id` | Parent span ID (null for root spans) |
| `request.is_root_span` | Boolean, true for request entry points |
| `request.is_failed` | Boolean, true if request failed |
| `duration` | Span duration in nanoseconds |
| `span.timing.cpu` | Overall CPU time of the span (stable) |
| `span.timing.cpu_self` | CPU time excluding child spans (stable) |
| `dt.smartscape.service` | Service Smartscape node ID |
| `dt.service.name` | Dynatrace service name derived from service detection rules. It is equal to the Smartscape service node name.  |
| `endpoint.name` | Endpoint/route name |

### Service Context

Spans reference services via Smartscape node IDs and the detected service name `dt.service.name` which is also present on every span.

```dql
fetch spans
| summarize spans=count(), by: { dt.smartscape.service, dt.service.name }
```

**Node functions**:
- `getNodeName(dt.smartscape.service)` - Adds `dt.smartscape.service.name` field with the human-readable service name
- `getNodeField(dt.smartscape.service, "attribute_name")` - Access specific node attributes

**📖 Learn more**: See [Entity Lookups](references/entity-lookups.md) for advanced entity selectors, infrastructure correlation, and hardware analysis.

### Sampling and Extrapolation

One span can represent multiple real operations due to:
- **Aggregation**: Multiple operations in one span (`aggregation.count`)
- **ATM (Adaptive Traffic Management)**: Head-based sampling by agent
- **ALR (Adaptive Load Reduction)**: Server-side sampling
- **Read Sampling**: Query-time sampling via `samplingRatio` parameter

**When to extrapolate**: Always extrapolate when counting actual operations (not just spans). Use the multiplicity factor:

```dql
fetch spans
| fieldsAdd sampling.probability = (power(2, 56) - coalesce(sampling.threshold, 0)) * power(2, -56)
| fieldsAdd sampling.multiplicity = 1 / sampling.probability
| fieldsAdd multiplicity = coalesce(sampling.multiplicity, 1)
                         * coalesce(aggregation.count, 1)
                         * dt.system.sampling_ratio
| summarize operation_count = sum(multiplicity)
```

**📖 Learn more**: See [Sampling and Extrapolation](references/sampling-extrapolation.md) for detailed formulas and examples.

## Common Query Patterns

### Basic Span Access

Fetch spans and explore by type:

```dql
fetch spans | limit 1
```

Explore spans by function and type:

```dql
fetch spans
| summarize count(), by: { span.kind, code.namespace, code.function }
```

### Request Root Filtering

List request root spans (incoming service calls):

```dql
fetch spans
| filter request.is_root_span == true
| fields trace.id, span.id, start_time, response_time = duration, endpoint.name
| limit 100
```

### Service Performance Summary

Analyze service performance with error rates:

```dql
fetch spans
| filter request.is_root_span == true
| summarize
    total_requests = count(),
    failed_requests = countIf(request.is_failed == true),
    avg_duration = avg(duration),
    p95_duration = percentile(duration, 95),
    by: {dt.service.name}
| fieldsAdd error_rate = (failed_requests * 100.0) / total_requests
| sort error_rate desc
```

### Trace ID Lookup

Find all spans in a specific trace:

```dql
fetch spans
| filter trace.id == toUid("abc123def456")
| fields span.name, duration, dt.service.name
```

## Performance Analysis

### Response Time Percentiles

Calculate percentiles by endpoint:

```dql
fetch spans
| filter request.is_root_span == true
| summarize {
    requests=count(),
    avg_duration=avg(duration),
    p95=percentile(duration, 95),
    p99=percentile(duration, 99)
  }, by: { endpoint.name }
| sort p99 desc
```

**💡 Best practice**: Use percentiles (p95, p99) over averages for performance insights.

### Slow Trace Detection

Find requests exceeding a threshold:

```dql
fetch spans, from:now() - 2h
| filter request.is_root_span == true
| filter duration > 5s
| fields trace.id, span.name, dt.service.name, duration
| sort duration desc
| limit 50
```

### Duration Buckets with Exemplars

```dql
fetch spans, from:now() - 24h
| filter http.route == "/api/v1/storage/findByISBN"
| summarize {
    spans=count(),
    trace=takeAny(record(start_time, trace.id))
  }, by: { bin(duration, 10ms) }
| fields `bin(duration, 10ms)`, spans, trace.id=trace[trace.id], start_time=trace[start_time]
```

### Performance Timeseries

Extract response time as timeseries:

```dql
fetch spans, from:now() - 24h
| filter request.is_root_span == true
| makeTimeseries {
    requests=count(),
    avg_duration=avg(duration),
    p95=percentile(duration, 95),
    p99=percentile(duration, 99)
  }, by: { endpoint.name }
```

**📖 Learn more**: See [Performance Analysis](references/performance-analysis.md) for advanced patterns and timeseries techniques.

## Failure Investigation

### Failed Request Summary

Summarize failures by service:

```dql
fetch spans
| filter request.is_root_span == true
| summarize
    total = count(),
    failed = countIf(request.is_failed == true),
  by: { dt.service.name }
| fieldsAdd failure_rate = (failed * 100.0) / total
| sort failure_rate desc
```

### Failure Reason Analysis

Breakdown by failure detection reason:

```dql
fetch spans
| filter request.is_failed == true and isNotNull(dt.failure_detection.results)
| expand dt.failure_detection.results
| summarize count(), by: { dt.failure_detection.results[reason] }
```

**Failure reasons**:
- `http_code` - HTTP response code triggered failure
- `grpc_code` - gRPC status code triggered failure
- `exception` - Exception caused failure
- `span_status` - Span status indicated failure
- `custom_rule` - Custom failure detection rule matched

### HTTP Code Failures

Find failures by HTTP status code:

```dql
fetch spans
| filter request.is_failed == true
| filter iAny(dt.failure_detection.results[][reason] == "http_code")
| summarize count(), by: { http.response.status_code, endpoint.name }
| sort `count()` desc
```

### Recent Failed Requests

List recent failures with details:

```dql
fetch spans
| filter request.is_root_span == true and request.is_failed == true
| fields
    start_time,
    trace.id,
    endpoint.name,
    http.response.status_code,
    duration
| sort start_time desc
| limit 100
```

**📖 Learn more**: See [Failure Detection](references/failure-detection.md) for exception analysis and custom rule investigation.

## Service Dependencies

### Service-to-Service Analysis

Analyze service communication patterns:

```dql
fetch spans, from:now() - 1h
| filter isNotNull(server.address)
| fieldsAdd
    remote_side = server.address
| summarize
    call_count = count(),
    avg_duration = avg(duration),
    by: {dt.service.name, remote_side}
| sort call_count desc
```

### Outgoing HTTP Calls

Identify external API dependencies:

```dql
fetch spans
| filter span.kind == "client" and isNotNull(http.request.method)
| summarize
    calls = count(),
    avg_latency = avg(duration),
    p99_latency = percentile(duration, 99),
  by: { dt.service.name, server.address, server.port }
| sort calls desc
```

## Trace Aggregation

### Complete Trace Analysis

Aggregate all spans in a trace to understand full request flow:

```dql
fetch spans, from:now() - 30m
| summarize {
    spans = count(),
    client_spans = countIf(span.kind == "client"),

    // Endpoints involved in the trace
    endpoints = toString(arrayRemoveNulls(collectDistinct(endpoint.name))),

    // Extract the first request root in the trace
    trace_root = takeMin(record(
        root_detection_helper = coalesce(
            if(request.is_root_span, 1),
            if(isNull(span.parent_id), 2),
            3),
        start_time, endpoint.name, duration
      ))
}, by: { trace.id }

| fieldsFlatten trace_root
| fieldsRemove trace_root.root_detection_helper, trace_root

| fields
    start_time = trace_root.start_time,
    endpoint = trace_root.endpoint.name,
    response_time = trace_root.duration,
    spans,
    client_spans,
    endpoints,
    trace.id
| sort start_time
| limit 100
```

**Root detection strategy**: Use `takeMin(record(...))` with a detection helper to reliably find the root request:
1. Priority 1: Spans with `request.is_root_span == true`
2. Priority 2: Spans without parent (root spans)
3. Priority 3: All other spans

### Multi-Service Traces

Find traces spanning multiple services:

```dql
fetch spans, from:now() - 1h
| summarize {
    services = collectDistinct(dt.service.name),
    trace_root = takeMin(record(root_detection_helper = coalesce(if(request.is_root_span, 1), 2), endpoint.name))
}, by: { trace.id }
| fieldsAdd service_count = arraySize(services)
| filter service_count > 1
| fields endpoint = trace_root[endpoint.name], service_count, services = toString(services), trace.id
| sort service_count desc
| limit 50
```

## Request-Level Analysis

### Request Attributes

Access custom request attributes captured by OneAgent on request root spans:

```dql
fetch spans
| filter request.is_root_span == true
| filter isNotNull(request_attribute.PaidAmount)
| makeTimeseries sum(request_attribute.PaidAmount)
```

**Field patterns**: `request_attribute.<name>`, `captured_attribute.<name>` (always arrays)

→ [Request Attributes](references/request-attributes.md) — full patterns for request attributes, captured attributes, and request ID aggregation

## Span Types

| Span Type | Detection | Key Fields | Reference |
|-----------|-----------|------------|-----------|
| HTTP server (incoming) | `span.kind == "server" and isNotNull(http.request.method)` | `http.route`, `http.request.method`, `http.response.status_code` | [http-spans.md](references/http-spans.md) |
| HTTP client (outgoing) | `span.kind == "client" and isNotNull(http.request.method)` | `server.address`, `server.port` | [http-spans.md](references/http-spans.md) |
| Database | `span.kind == "client" and isNotNull(db.system)` | `db.system`, `db.namespace`, `db.statement` | [database-spans.md](references/database-spans.md) |
| Messaging | `isNotNull(messaging.system)` | `messaging.system`, `messaging.destination.name`, `messaging.operation.type` | [messaging-spans.md](references/messaging-spans.md) |
| RPC / gRPC | `isNotNull(rpc.system)` | `rpc.system`, `rpc.service`, `rpc.method`, `rpc.grpc.status_code` | [rpc-spans.md](references/rpc-spans.md) |
| Serverless / FaaS | `isNotNull(faas.name) and span.kind == "server"` | `faas.name`, `faas.trigger.type`, `cloud.provider` | [serverless-spans.md](references/serverless-spans.md) |

**⚠️ Database spans**: Can be aggregated (one span = multiple calls). Always use `aggregation.count` extrapolation for accurate operation counts.

**📖 Detailed patterns per span type:** See the reference files above.

## Advanced Topics

### Exception Analysis

Exceptions are stored as `span.events` within spans:

```dql
fetch spans
| filter iAny(span.events[][span_event.name] == "exception")
| expand span.events
| fieldsFlatten span.events, fields: { exception.type }
| summarize {
    count(),
    trace=takeAny(record(start_time, trace.id))
  }, by: { exception.type }
| fields exception.type, `count()`, trace.id=trace[trace.id], start_time=trace[start_time]
```

**💡 Tip**: Use `iAny()` to check conditions within span event arrays.

→ [Logs Correlation](references/logs-correlation.md) — joining logs and traces, filtering traces by log content
→ [Network Analysis](references/networking-analysis.md) — client IPs, DNS resolution, subnet analysis

## Best Practices

| Area | Rule |
|------|------|
| **Filtering** | Apply `request.is_root_span == true` and endpoint filters first |
| **Sampling** | Use `samplingRatio` (e.g., `100` = read 1%) for performance |
| **Percentiles** | Use p95/p99 over averages for performance analysis |
| **Root spans** | Use `request.is_root_span == true` for end-to-end analysis |
| **Trace grouping** | Group by `trace.id` for complete trace metrics |
| **Request grouping** | Group by `request.id` for OneAgent-only request metrics |
| **Extrapolation** | Always apply multiplicity for accurate operation counts |
| **Exemplars** | Use `takeAny(record(start_time, trace.id))` to enable UI drilldown |

---

## Troubleshooting

| Problem | Cause | Solution |
|---------|-------|----------|
| Duration values seem wrong (too large) | `duration` is in nanoseconds, not milliseconds | Divide by `1000000` or compare with `5s` (DQL duration literal) |
| Span counts don't match expected request volume | Sampling or aggregation not accounted for | Use multiplicity extrapolation — see Sampling and Extrapolation reference |
| `getNodeName(dt.smartscape.service)` returns null | Service not yet resolved or OneAgent not monitoring | Verify OneAgent monitors the service; entity resolution may have a short delay |
| `request.is_root_span` filter returns nothing | Querying OpenTelemetry-only traces without OneAgent | Use `isNull(span.parent_id)` as fallback for root span detection |
| `trace.id` filter returns no results | Trace ID not converted to UID format | Use `filter trace.id == toUid("abc123...")` for string-based trace IDs |
| Database span counts are too low | Database spans are aggregated (one span = N calls) | Always use `aggregation.count` extrapolation for database operation counts |

## Related Skills

- **dt-dql-essentials** — Core DQL syntax for querying trace data
- **dt-app-dashboards** — Embed trace queries in dashboards
- **dt-migration** — Smartscape entity model and relationship navigation

---

## References

Detailed documentation for specific topics:

- **[Performance Analysis](references/performance-analysis.md)** - Advanced timeseries, duration buckets, endpoint ranking
- **[Failure Detection](references/failure-detection.md)** - Failure reasons, exception investigation, custom rules
- **[Sampling and Extrapolation](references/sampling-extrapolation.md)** - Multiplicity calculation, database extrapolation
- **[Request Attributes](references/request-attributes.md)** - Request attributes, captured attributes, request ID aggregation
- **[Entity Lookups](references/entity-lookups.md)** - Advanced node lookups, infrastructure correlation, hardware analysis
- **[HTTP Span Analysis](references/http-spans.md)** - Status codes, payload analysis, client IPs
- **[Database Span Analysis](references/database-spans.md)** - Extrapolated counts, slow queries, statement analysis
- **[Messaging Span Analysis](references/messaging-spans.md)** - Kafka, RabbitMQ, SQS throughput and latency
- **[RPC Span Analysis](references/rpc-spans.md)** - gRPC, SOAP, service dependencies
- **[Serverless Span Analysis](references/serverless-spans.md)** - Lambda, Azure Functions, cold start analysis
- **[Logs Correlation](references/logs-correlation.md)** - Joining logs and traces, correlation patterns
- **[Network Analysis](references/networking-analysis.md)** - IP addresses, DNS resolution, communication mapping