Bug
Registering a litellm CustomLogger / success_callback / _async_success_callback to capture per-call token usage and cost has no effect when invoked through pr_agent.cli.run_command(...). Neither the sync nor the async callback path delivers events for any pr-agent command (/describe, /review, /improve, etc.).
Why both paths fail
- Sync path —
success_callback gates on is_sync_request, which is False for the async acompletion calls used by pr_agent.algo.ai_handlers.litellm_ai_handler.
- Async path —
_async_success_callback enqueues callbacks on a background logging worker. In pr-agent's run loop the parent event loop is recreated between commands; the queued coroutine is dropped and never executed.
Net effect: zero callback invocations, so usage / cost cannot be recorded through the documented hook points.
Repro
import litellm
from pr_agent import cli
class Counter(litellm.CustomLogger):
n = 0
def log_success_event(self, *a, **kw): Counter.n += 1
async def async_log_success_event(self, *a, **kw): Counter.n += 1
litellm.callbacks = [Counter()]
cli.run_command("<pr_url>", "/review")
print("callback fired", Counter.n, "times") # always 0
Workaround
Wrap litellm.acompletion directly and read usage from the response synchronously:
import litellm
import pr_agent.algo.ai_handlers.litellm_ai_handler as _h
_orig = litellm.acompletion
async def _wrapped(*a, **kw):
resp = await _orig(*a, **kw)
record_usage(kw.get("model") or a[0], resp)
return resp
litellm.acompletion = _wrapped
_h.acompletion = _wrapped # handler imports the name into its own namespace
Suggested fix
for now, document that the callback APIs do not work in this code path so users build their own usage capture.
Environment
Bug
Registering a litellm
CustomLogger/success_callback/_async_success_callbackto capture per-call token usage and cost has no effect when invoked throughpr_agent.cli.run_command(...). Neither the sync nor the async callback path delivers events for any pr-agent command (/describe,/review,/improve, etc.).Why both paths fail
success_callbackgates onis_sync_request, which isFalsefor the asyncacompletioncalls used bypr_agent.algo.ai_handlers.litellm_ai_handler._async_success_callbackenqueues callbacks on a background logging worker. In pr-agent's run loop the parent event loop is recreated between commands; the queued coroutine is dropped and never executed.Net effect: zero callback invocations, so usage / cost cannot be recorded through the documented hook points.
Repro
Workaround
Wrap
litellm.acompletiondirectly and read usage from the response synchronously:Suggested fix
for now, document that the callback APIs do not work in this code path so users build their own usage capture.
Environment