It's very useful to be able to view traces and spans for operations in logging applications like Axiom. Currently, there are some limitations that make this difficult to implement, and in some cases, impossible:
-
The OpenTelemetry node packages require "use node", so the only place we can use the official packages is in node actions.
-
The frozen Date.now() prevents us from tracking span durations in mutations
-
Without wrapping a runQuery() inside of an action (or at very least a mutation), it's not possible to record span data (duration or custom fields) inside of a query.
-
Manually wrapping everything you need a span on requires a LOT of extra function invocations, and effects application performance, mainly because of the reasons above--since we cannot just use the log drain to record spans.
Ideally, we need Convex to implement first party OTEL integration that integrates seamlessly with Log Streams (or an equivalent), and we shouldn't have to put everything in a node action just to record traces and spans.
Since Convex already records function duration for their own purposes (I'm assuming at a Rust level), it would be ideal for us to have access to an OTEL object (probably on the ctx object) in actions, mutations and queries, where the duration is recorded on the outer span for a function, and allowing us to amend spans and create child spans, as well as adding custom fields to those spans.
Ideally, this should integration with the existing Log Stream integrations seamlessly, so (using Axiom as an example) we can choose which dataset they go to, along with other relevant settings, then services like Axiom can automatically pick up on traces without developers having to manually instrument everything.
As for AI Agents, OpenTelemetry already has some emerging standards for recording agent related information. External services like PostHog often have nice features that automatically detect these AI-related fields and allow convenient recording/viewing of this data. However, these currently rely on manual instrumentation and are limited to node actions. It would be ideal for Convex to give us the ability to this without relying on node and manually instrumenting everything. I'm not sure how possible this is without writing an OTEL lib that doesn't require node, but I figure it could be possible to leverage the log drain to record this data--this would at least remove the requirement to use HTTP to collect spans. I'm just not sure about the automatic instrumentation itself.