[Bug]: Missing traces when importing into new workspace

[cparish312]

What component(s) are affected?

• Opik Python SDK
• Opik Typescript SDK
• Opik Agent Optimizer SDK
• Opik UI
• Opik Server
• Documentation

Opik version

• opik, version 1.10.23

Describe the problem

Running:

opik export fl105 project "irischat.ai" \                                                   
  --path ./20260209-20260215_traces \
  --filter "thread_id contains \"20260209-20260215\"" \
  --max-results 100000

Then

opik import fl105-annotations project "irischat.ai" --path ./20260209-20260215_traces/fl105

Resulted in traces missing. Expected 6,055, but only 5,435 were uploaded.

I tried multiple times (both import and export) and got a different number of traces uploaded each time (around the same number).

I tried running the the import command twice, but this creates duplicate traces (it would be great to have an option to only import traces that don't exist).

I have uploaded images of the import and export summaries.

Reproduction steps and code snippets

No response

Error logs or stack trace

No response

Healthcheck results

No response

[cparish312]

The fundamental issue is I need a way for external annotators to use an annotation queue without giving them access to all of our traces. I am copying traces to a different workspace as a workaround, but this process takes around 4 hours and doesn't seem to work consistently, so I would really love a more robust solution here!

[dsblank]

Thanks for the detailed report! I'm Ollie, an AI agent helping triage issues. Let me investigate the import/export functionality to understand what might cause missing traces.

I'll look into the relevant code and get back to you shortly.

[cparish312]

It is possible I am exporting more than 100,000 spans, so I will try to up the max-results to 1000000 and see if that resolves the issue.

[dsblank]

Thanks for the detailed report. I've done a deep dive into the import/export code and identified several root causes for both problems you described.

Root Cause Analysis

1. Missing traces (~620 out of 6,055)

There are two contributing factors:

a) Import summary counts file reads, not actual uploads

In sdks/python/src/opik/cli/imports/project.py, the counter increments immediately after calling client.trace() and client.span():

trace = client.trace(...)  # just queues a message, does NOT upload yet
# ... create spans ...
traces_imported += 1  # counts file processed, NOT successfully uploaded

The Opik client uses an asynchronous message queue — client.trace() only queues the message. The actual HTTP calls happen in 4 background worker threads. The import summary is printed before those background uploads complete (the actual flush happens at process exit via atexit).

b) No explicit flush before the import summary

In sdks/python/src/opik/cli/imports/__init__.py, after import_projects_from_directory() returns:

stats = import_projects_from_directory(client, ...)  # queued messages, not uploaded yet
print_import_summary(stats)                           # summary printed BEFORE uploads complete
# ... function returns, client eventually flushed via atexit ...

If any of the 6,055 individual API calls fail (server errors 5xx after 3 retries, validation errors 4xx, or rate limiting without proper headers), those traces are silently dropped. The non-deterministic count (~5,435 each time, with slight variation) is consistent with transient server errors during rapid-fire individual API calls.

2. Duplicate traces when running import twice

In project.py, a new random ID is generated for every trace on every import run:

trace = client.trace(
    id=id_helpers.generate_id(timestamp=original_start_time),  # NEW random ID each run!
    ...
)

The original trace ID from the export file (trace_info.get("id")) is captured into original_trace_id only to build a mapping for experiment recreation — the actual trace uploaded to the server always gets a fresh ID. Running import twice creates 2× traces with different IDs.

Recommended Fixes

1. Add client.flush() before printing the summary in _import_by_type() — this ensures all queued messages are actually sent to the server before reporting counts, and the summary would reflect actual upload results rather than file processing counts.

2. Reuse original trace IDs from the export file instead of generating new ones — this would make imports idempotent (re-running would update existing traces rather than creating duplicates). The server already handles 409 conflicts gracefully.

3. Enable batching (_use_batching=True) for bulk imports — rather than 6,055 individual API calls, traces would be batched (up to 1,000 per request), significantly reducing failure surface area and improving throughput.

Would it be helpful if I submitted a PR with these fixes?

[dsblank]

Hi @cparish312,

We've identified a few issues related to the missing traces and the duplication problem:

1. Missing Traces:

   • The import process doesn't call client.flush() before displaying the summary. This means traces might not be fully uploaded when the summary is printed, leading to discrepancies.
   • The current setup doesn't use batching, resulting in many individual API calls. Some traces might be dropped silently due to transient errors.

2. Duplicate Traces:

   • Each import run generates new trace IDs instead of reusing original IDs, leading to duplicates upon re-import.

Recommendations

• Add a client.flush() call after the import process to ensure all traces are uploaded before the summary.
• Use batching for imports to handle network calls more efficiently.
• Modify the import logic to reuse original trace IDs to prevent duplication.

Please let me know if you need further assistance or if you would like these changes to be implemented.

Thank you!

---

_{This is an automated response. Responses may be inaccurate—please verify important details.
trace: 019cb4f1-d61d-7c7a-847f-acfffba14dd1}

[cparish312]

Doing some batching could help speed it up

[cparish312]

Update:
Setting max-results to 1000000 on the import did not resolve the issue

[dsblank]

We have three fixes under review that should help:

1. Fixed a flush issue that could result in missing traces on import [NA] [SDK] fix: flush error reporting and timeout cleanup #5475
2. Added a batching flag to imports [NA] [SDK] fix: flush error reporting and timeout cleanup #5475
3. Added a new migration manifest file to ensure no duplicate and no lost data [OPIK-4801] [SDK] Add migration manifest for resumable opik import #5528