OpenAI deploys Codex-powered system to investigate ChatGPT slowdowns
OpenAI has developed an internal system called the Performance Factory to investigate and help mitigate performance regressions affecting ChatGPT. Powered by Codex, it automates much of the work required when a software update makes a service slower, including examining telemetry, tracing requests across services, locating the responsible code and evaluating possible fixes. In just over three months, the system handled more than 1,700 investigations and identified the root causes of over 600 cases, averaging 17 investigations a day.
The project grew out of a largely manual process in which performance engineers inspected logs, traces and code changes, using more than 30 specialized tools. Codex initially helped them query those tools, but engineers still had to direct each investigation. OpenAI then encoded the team’s procedures into a reusable Codex skill called “perf-forensics,” allowing the agent to turn a performance alert into a diagnosis in a single run.
The Performance Factory extends that idea into a task-based workflow. Each regression is tracked as a task that can move through investigation, root-cause analysis, proposed code changes, review and verification. Tasks preserve notes, structured metadata and links to relevant metrics and queries, allowing different agents to continue the work without copying large profiling datasets. Engineers define the workflows and prompts, while a dispatch loop starts Codex sessions when tasks are ready to advance. The system illustrates how coding agents can coordinate multi-stage engineering work while leaving human specialists to make judgments and approve implementation.
Continue in the app — vote & join in ➔