We now have SDKs for:
Unity: github.com/PostHog/postho…
Godot: github.com/PostHog/postho…
Unreal: github.com/PostHog/postho…
Unity: github.com/PostHog/postho…
Godot: github.com/PostHog/postho…
Unreal: github.com/PostHog/postho…
0
0
4
444
PostHog retweeted
Who's burning the most tokens?? Get your answer in so I can give you a @posthog token burning staff hat 🧢 posthog.com/forum/p/how-ma…
0
1
4
393
We spend 2 trillion+ LLM tokens a month. That’s way too much money spent to trust our agents based on gut feeling.
So, we started using custom benchmarks.
We ran into plenty of problems, from parallel runs to open-ended tasks. Here’s everything we learned along the way:
So, we started using custom benchmarks.
We ran into plenty of problems, from parallel runs to open-ended tasks. Here’s everything we learned along the way:
20
12
537
39,256
8. Agents get suspicious when they realize they are being benchmarked.
In our frozen environment setup, we chose not to slice user tables (to protect PII).
We didn't expect agents to need those tables anyway, but they inevitably did. Once they saw an abnormally tiny user base, they knew something was up. The over-prepared data looked synthetic.
Once an agent thinks it’s being tested, it stops early on the assigned tasks, chases data gaps, and speculates more. The benchmark becomes obsolete.
In our frozen environment setup, we chose not to slice user tables (to protect PII).
We didn't expect agents to need those tables anyway, but they inevitably did. Once they saw an abnormally tiny user base, they knew something was up. The over-prepared data looked synthetic.
Once an agent thinks it’s being tested, it stops early on the assigned tasks, chases data gaps, and speculates more. The benchmark becomes obsolete.
2
0
11
1,680
TL;DR custom benchmarking is about more than just coming up with known tasks and answers.
Agents are really good at cheating and will exploit whatever they can find. Watch for unexpected parameters, iterate a lot, and read traces.
You can read the full post, with real PRs and traces, here: posthog.com/blog/how-to-be…
Agents are really good at cheating and will exploit whatever they can find. Watch for unexpected parameters, iterate a lot, and read traces.
You can read the full post, with real PRs and traces, here: posthog.com/blog/how-to-be…
0
1
14
1,579
PostHog retweeted
Very cool to see how quickly the PostHog mobile app is coming together
- we historically never had the resources or ios talent internally to justify building an app
- people have consistently asked for one, but be saw prioritizing dev experience on desktop as worth being our only focus
- an app was something we saw as just another thing to maintain
- now some of our best engineers regularly ship from their phones
- we can leverage our existing talent to within posthog to build something functional for iphone users with full confidence. that was not the case a year ago
- we historically never had the resources or ios talent internally to justify building an app
- people have consistently asked for one, but be saw prioritizing dev experience on desktop as worth being our only focus
- an app was something we saw as just another thing to maintain
- now some of our best engineers regularly ship from their phones
- we can leverage our existing talent to within posthog to build something functional for iphone users with full confidence. that was not the case a year ago
1
3
44
1,670
PostHog retweeted
SF founders: come demo your agentic loops, AMA with the engineering team that built Replay Vision, and learn about Birdwatching from the @GGBirdAlliance with us. Gonna be a very @posthog kind of night. Link below.
1
2
11
646