Span-Links not showing on traces

[ErikBcd]

Hi!

I recently wanted to test sending traces using the signalfx splunk-otel-collector. In general everything works as expected, however when sending spans containing links to other spans, these links don't show up in the waterfall UI, even though they should be working according to the documentation . When downloading the trace data, span links are not mentioned at all.

The (debug) logs of the splunk-otel-collector don't seem to mention any errors/abnormalities either.

Following example shows my test span. It should be linking to two other spans, but it doesn't show up as such.

Additionally I tested using Jaeger All-in-one, and in there the span links properly show up.

I am thankful for any hints you can provide that might help me debug this problem 🙂

[bishida]

Hi,

If you send an OpenTelemetry formatted span that contains span links, it should show up in Observability Cloud. To get this to work in the real-world, you'll probably need to manually instrument your source code. Here is an example json payload saved to a file "span.otel.json" and sent to a local OTel collector with:

curl -X POST http://localhost:4318/v1/traces -H "Content-Type: application/json" -d @Span.otel.json

{
  "resourceSpans": [
    {
      "resource": {
        "attributes": [
          {
            "key": "service.name",
            "value": {
              "stringValue": "example-service"
            }
          },
          {
            "key": "service.instance.id",
            "value": {
              "stringValue": "instance-12345"
            }
          }
        ]
      },
      "scopeSpans": [
        {
          "scope": {
            "name": "example-tracer",
            "version": "1.0.0"
          },
          "spans": [
            {
              "traceId": "1e223ff1f80f1c69f8f0b81c1a2d32ad",
              "spanId": "6f9b3f4d1de5bf5e",
              "parentSpanId": "",
              "name": "example-operation",
              "kind": "SPAN_KIND_SERVER",
              "startTimeUnixNano": 1706100243452000000,
              "endTimeUnixNano": 1706100244952000000,
              "attributes": [
                {
                  "key": "http.method",
                  "value": {
                    "stringValue": "GET"
                  }
                },
                {
                  "key": "http.url",
                  "value": {
                    "stringValue": "http://example.com/api"
                  }
                }
              ],
              "links": [
                {
                  "traceId": "1e223ff1f80f1c69f8f0b81c1a2d32ae",
                  "spanId": "9a7b3e4f1dceaa56",
                  "attributes": [
                    {
                      "key": "link-type",
                      "value": {
                        "stringValue": "related"
                      }
                    }
                  ],
                  "droppedAttributesCount": 0
                }
              ]
            }
          ]
        }
      ]
    }
  ]
}

[devjeff85]

I don't see the span links in my trace view either.  This is strange. I've noticed in your example that you use the port 4318 in the collector url and thought that maybe the problem is somehow related to the grpc protocol.

But when I configure the "http/protobuf" with port 4318 in the local app, it doesn't change anything. The links just don't appear in the UI. Here is the Splunk OTel Collector configuration usiing docker compose:

otelcol:
    image: quay.io/signalfx/splunk-otel-collector:latest
    environment:
      - SPLUNK_ACCESS_TOKEN=XXXXXXXXXXXXXXXXX
      - SPLUNK_REALM=eu1
    ports:
      - "13133:13133"
      - "14250:14250"
      - "14268:14268"
      - "4317:4317"
      - "4318:4318"
      - "6060:6060"
      - "8888:8888"
      - "9080:9080"
      - "9411:9411"
      - "9943:9943"
    container_name: otelcol
    restart: unless-stopped

[ErikBcd]

Hi,

thanks for your answer!

We manually instrument our code and include span links, and we can see that it works when using Jaeger instead of splunk.

When trying your example via curl the span link didn't show up in the waterfall view either, so maybe our collector isn't set up correctly?

We are using the docker container ( quay.io/signalfx/splunk-otel-collector:latest ) with the default configuration. However, I could not find any hints in the docs on what to configure for span links to work.

[ErikBcd]

One more addition:

For testing we also now wrote a config for the collector that exports traces to both splunk and Jaeger at the same time, so we could see if it was just the collector not registering span-links in general or something else down the line.

When doing this, the span links appeared in Jaeger - but were still not visible in Splunk. So we think it's either the (default) configuration for the splunk-opentelemetry-collector (specifically the splunk export) not handling span-links, or something in the Observability Cloud not accepting our span-links.

There is one more hint I got from the collector container logs, following message that appears a few seconds after the traces are sent:

2025-01-27T14:35:00.707Z info transport/http2_server.go:662 [transport] [server-transport 0xc0022ba000] Closing: EOF {"grpc_log": true}
2025-01-27T14:35:00.707810569Z 2025-01-27T14:35:00.707Z info transport/controlbuf.go:577 [transport] [server-transport 0xc0022ba000] loopyWriter exiting with error: transport closed by client {"grpc_log": true}

This might indicate that the writer is stuck at sending the span links, since everything else is sent to splunk correctly. However, I am not sure about the inner workings there.

I would appreciate any hints for debugging this!