> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/iii-hq/sdk/llms.txt
> Use this file to discover all available pages before exploring further.
# Telemetry
> OpenTelemetry tracing, metrics, and logging in the III SDK
## Overview
The III SDK provides built-in OpenTelemetry support for distributed tracing, metrics, and logging. When enabled, telemetry data is automatically exported to the III Engine.
## Installation
Install the SDK with OpenTelemetry support:
```bash theme={null}
pip install iii-sdk[otel]
```
This installs:
* `opentelemetry-api>=1.25`
* `opentelemetry-sdk>=1.25`
## Initialization
### init\_otel
Initialize OpenTelemetry with automatic engine integration.
```python theme={null}
from iii import init_otel, OtelConfig
init_otel(OtelConfig(
service_name="my-service",
service_version="1.0.0",
enabled=True
))
```
OpenTelemetry configuration. If omitted, uses defaults.
Running event loop. When provided, the connection starts immediately. When None, it starts lazily on first use.
### OtelConfig
Configuration for OpenTelemetry initialization:
```python theme={null}
from dataclasses import dataclass
@dataclass
class OtelConfig:
enabled: bool | None = None
service_name: str | None = None
service_version: str | None = None
service_namespace: str | None = None
service_instance_id: str | None = None
engine_ws_url: str | None = None
fetch_instrumentation_enabled: bool = True
logs_enabled: bool | None = None
metrics_enabled: bool = True
metrics_export_interval_ms: int = 60000
```
Enable OpenTelemetry. Defaults to True unless `OTEL_ENABLED=false/0/no/off`
Service name. Uses `OTEL_SERVICE_NAME` env var if set
Service version. Uses `SERVICE_VERSION` env var if set
Service namespace for grouping related services
Unique instance ID. Defaults to a random UUID
III Engine WebSocket URL. Uses `III_BRIDGE_URL` env var if set
Auto-instrument urllib HTTP calls
Enable OpenTelemetry log export
Enable OpenTelemetry metrics export
Metrics export interval in milliseconds (60 seconds)
## Distributed Tracing
### get\_tracer
Get the active OpenTelemetry tracer.
```python theme={null}
from iii import get_tracer
from opentelemetry import trace
tracer = get_tracer()
if tracer:
with tracer.start_as_current_span("my-operation"):
# Your code here
pass
```
The active tracer, or None if OTel is not initialized
### Automatic Trace Propagation
Trace context is automatically propagated across function calls:
```python theme={null}
from iii import init_otel, III, get_tracer
from opentelemetry import trace
init_otel()
tracer = get_tracer()
iii = III("ws://localhost:49134")
async def parent_function(data):
with tracer.start_as_current_span("process-order"):
# Trace context is automatically propagated
payment = await iii.call("payment.process", data)
return payment
async def payment_function(data):
# This span is linked to the parent trace
with tracer.start_as_current_span("charge-card"):
# Process payment
return {"status": "success"}
iii.register_function("orders.process", parent_function)
iii.register_function("payment.process", payment_function)
```
### Custom Span Attributes
```python theme={null}
from iii import get_tracer
tracer = get_tracer()
if tracer:
with tracer.start_as_current_span("database-query") as span:
span.set_attribute("db.system", "postgresql")
span.set_attribute("db.operation", "SELECT")
span.set_attribute("db.statement", "SELECT * FROM users WHERE id = ?")
# Execute query
result = execute_query()
span.set_attribute("db.rows_returned", len(result))
```
### HTTP Instrumentation
Urllib HTTP requests are automatically instrumented when `fetch_instrumentation_enabled=True`:
```python theme={null}
import urllib.request
from iii import init_otel
init_otel() # Enables automatic urllib instrumentation
# This request is automatically traced
response = urllib.request.urlopen("https://api.example.com/data")
```
Spans include attributes:
* `http.request.method`
* `url.full`
* `server.address`
* `url.scheme`
* `url.path`
* `server.port`
* `http.response.status_code`
* `http.request.body.size`
* `http.response.body.size`
## Metrics
### get\_meter
Get the active OpenTelemetry meter.
```python theme={null}
from iii import get_meter
meter = get_meter()
if meter:
# Create a counter
request_counter = meter.create_counter(
"http.requests",
description="Number of HTTP requests",
unit="1"
)
# Increment counter
request_counter.add(1, {"method": "GET", "status": "200"})
```
The active meter, or None if OTel metrics are not initialized
### Counter
```python theme={null}
meter = get_meter()
if meter:
orders_counter = meter.create_counter(
"orders.total",
description="Total number of orders",
unit="1"
)
async def create_order(data):
# Process order
orders_counter.add(1, {"status": "created"})
return {"id": "order123"}
```
### Histogram
```python theme={null}
meter = get_meter()
if meter:
duration_histogram = meter.create_histogram(
"order.processing.duration",
description="Order processing duration",
unit="ms"
)
async def process_order(data):
start = time.time()
# Process order
duration_ms = (time.time() - start) * 1000
duration_histogram.record(duration_ms, {"status": "success"})
```
### Gauge
```python theme={null}
import psutil
from iii import get_meter
meter = get_meter()
if meter:
cpu_gauge = meter.create_observable_gauge(
"system.cpu.usage",
callbacks=[lambda options: [(psutil.cpu_percent(), {})]],
description="CPU usage percentage",
unit="%"
)
```
## Logging
### Logger
The SDK provides a context-aware logger that emits OpenTelemetry LogRecords:
```python theme={null}
from iii import get_context
async def my_function(data):
ctx = get_context()
ctx.logger.info("Processing request", data={"user_id": data["user_id"]})
try:
# Process data
result = process(data)
ctx.logger.info("Request processed successfully")
return result
except Exception as e:
ctx.logger.error("Processing failed", data={"error": str(e)})
raise
```
### Log Levels
```python theme={null}
ctx = get_context()
ctx.logger.debug("Debug information", data={"details": "..."})
ctx.logger.info("Informational message", data={"status": "ok"})
ctx.logger.warn("Warning message", data={"threshold": 90})
ctx.logger.error("Error message", data={"error": "Something went wrong"})
```
Log records include:
* Timestamp
* Severity level
* Message body
* Function name (if available)
* Trace context (span ID, trace ID)
* Custom attributes
### Fallback to Python Logging
If OTel is not initialized, logs fallback to standard Python logging:
```python theme={null}
import logging
logging.basicConfig(level=logging.INFO)
# Without OTel, this uses Python's logging module
ctx = get_context()
ctx.logger.info("This is logged via Python logging")
```
## Shutdown
### shutdown\_otel
Shut down OpenTelemetry synchronously (best-effort):
```python theme={null}
from iii import shutdown_otel
shutdown_otel()
```
### shutdown\_otel\_async
Shut down OpenTelemetry and await WebSocket connection close:
```python theme={null}
from iii import shutdown_otel_async
await shutdown_otel_async()
```
### is\_initialized
Check if OpenTelemetry has been initialized:
```python theme={null}
from iii import is_initialized
if is_initialized():
print("OTel is active")
else:
print("OTel is not initialized")
```
True if OTel has been successfully initialized
## Example: Full Observability
```python theme={null}
import asyncio
import time
from iii import (
III,
init_otel,
OtelConfig,
get_tracer,
get_meter,
get_context,
shutdown_otel_async,
)
from opentelemetry import trace
# Initialize OTel
init_otel(OtelConfig(
service_name="order-service",
service_version="1.0.0",
service_namespace="ecommerce",
metrics_enabled=True,
logs_enabled=True,
))
tracer = get_tracer()
meter = get_meter()
# Create metrics
orders_counter = meter.create_counter(
"orders.total",
description="Total orders processed",
unit="1"
)
processing_time = meter.create_histogram(
"orders.processing_time",
description="Order processing time",
unit="ms"
)
iii = III("ws://localhost:49134")
async def process_order(data):
ctx = get_context()
start = time.time()
with tracer.start_as_current_span("process-order") as span:
order_id = data["order_id"]
span.set_attribute("order.id", order_id)
ctx.logger.info("Processing order", data={"order_id": order_id})
try:
# Validate inventory
with tracer.start_as_current_span("validate-inventory"):
inventory = await iii.call("inventory.check", {
"product_id": data["product_id"],
"quantity": data["quantity"]
})
if not inventory["available"]:
raise ValueError("Out of stock")
# Process payment
with tracer.start_as_current_span("process-payment"):
payment = await iii.call("payment.charge", {
"amount": data["amount"],
"customer_id": data["customer_id"]
})
span.set_attribute("payment.id", payment["id"])
# Record metrics
duration_ms = (time.time() - start) * 1000
orders_counter.add(1, {"status": "success"})
processing_time.record(duration_ms, {"status": "success"})
ctx.logger.info("Order processed successfully", data={
"order_id": order_id,
"duration_ms": duration_ms
})
return {
"order_id": order_id,
"status": "confirmed",
"payment_id": payment["id"]
}
except Exception as e:
duration_ms = (time.time() - start) * 1000
orders_counter.add(1, {"status": "failed"})
processing_time.record(duration_ms, {"status": "failed"})
span.set_status(trace.Status(trace.StatusCode.ERROR, str(e)))
span.record_exception(e)
ctx.logger.error("Order processing failed", data={
"order_id": order_id,
"error": str(e)
})
raise
iii.register_function("orders.process", process_order)
async def main():
await iii.connect()
try:
result = await iii.call("orders.process", {
"order_id": "order123",
"product_id": "prod456",
"quantity": 2,
"amount": 99.99,
"customer_id": "cust789"
})
print(f"Order result: {result}")
except Exception as e:
print(f"Order failed: {e}")
finally:
await iii.shutdown()
await shutdown_otel_async()
if __name__ == "__main__":
asyncio.run(main())
```
## Environment Variables
The SDK respects these environment variables:
* `OTEL_ENABLED`: Set to `false`, `0`, `no`, or `off` to disable OTel
* `OTEL_SERVICE_NAME`: Default service name
* `SERVICE_VERSION`: Default service version
* `III_BRIDGE_URL`: III Engine WebSocket URL (default: `ws://localhost:49134`)
```bash theme={null}
export OTEL_SERVICE_NAME=my-service
export SERVICE_VERSION=2.0.0
export III_BRIDGE_URL=ws://engine.example.com:49134
python app.py
```
## Best Practices
1. **Initialize early**: Call `init_otel()` before connecting to the III Engine
2. **Use context**: Access logger via `get_context()` for automatic tracing
3. **Meaningful names**: Use descriptive span names and metric names
4. **Attributes**: Add relevant attributes to spans for filtering and analysis
5. **Error handling**: Always set span status and record exceptions
6. **Cleanup**: Call `shutdown_otel_async()` on graceful shutdown
7. **Sampling**: Use OTel's built-in sampling for high-volume services
## Integration with III Engine
Telemetry data is automatically exported to the III Engine via WebSocket:
* **Traces**: Exported via `EngineSpanExporter`
* **Metrics**: Exported via `EngineMetricsExporter` every 60 seconds
* **Logs**: Exported via `EngineLogExporter`
The engine aggregates telemetry from all workers and provides a unified observability view.