Skip to content

litellm success/cost callbacks never fire from pr-agent's async run loop #2378

@agonzalesipcoop-cmyk

Description

@agonzalesipcoop-cmyk

Bug

Registering a litellm CustomLogger / success_callback / _async_success_callback to capture per-call token usage and cost has no effect when invoked through pr_agent.cli.run_command(...). Neither the sync nor the async callback path delivers events for any pr-agent command (/describe, /review, /improve, etc.).

Why both paths fail

  1. Sync pathsuccess_callback gates on is_sync_request, which is False for the async acompletion calls used by pr_agent.algo.ai_handlers.litellm_ai_handler.
  2. Async path_async_success_callback enqueues callbacks on a background logging worker. In pr-agent's run loop the parent event loop is recreated between commands; the queued coroutine is dropped and never executed.

Net effect: zero callback invocations, so usage / cost cannot be recorded through the documented hook points.

Repro

import litellm
from pr_agent import cli

class Counter(litellm.CustomLogger):
    n = 0
    def log_success_event(self, *a, **kw):       Counter.n += 1
    async def async_log_success_event(self, *a, **kw): Counter.n += 1

litellm.callbacks = [Counter()]
cli.run_command("<pr_url>", "/review")
print("callback fired", Counter.n, "times")  # always 0

Workaround

Wrap litellm.acompletion directly and read usage from the response synchronously:

import litellm
import pr_agent.algo.ai_handlers.litellm_ai_handler as _h

_orig = litellm.acompletion

async def _wrapped(*a, **kw):
    resp = await _orig(*a, **kw)
    record_usage(kw.get("model") or a[0], resp)
    return resp

litellm.acompletion = _wrapped
_h.acompletion = _wrapped  # handler imports the name into its own namespace

Suggested fix

for now, document that the callback APIs do not work in this code path so users build their own usage capture.

Environment

  • pr-agent 0.34.3 (PyPI)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions