One weekend I started playing around with self-improving agents. I tried imagining a world where an agent will be able to build itself and improve its own prompts by running and evaluating itself. I played around with a RAG pipeline we have as part of Traceloop's docs, and see if I can improve its performance without doing anything - just let the agent iterate and improve the prompt by itself. The results were astonishing - the agent was able to improve the performance of our RAG pipeline by almost 200% - without me having to do anything (well, except for having to build all the evals, and write the prompt improving agent - but you get my point). And so I realized - prompt engineering is dead. It wasn't engineering, really. It was just us manually doing what AI can now do automatically. Think about it: we've been spending hours crafting the perfect prompts, tweaking words, adding examples, adjusting tone. But that's exactly what these self-improving agents excel at - iterating through variations, testing performance, and optimizing based on results. The future isn't about becoming better prompt engineers. It's about building systems that can engineer their own prompts. Systems that can evaluate their own performance and continuously improve without human intervention. I gave a talk about this at the AI Engineer World Fair last month in SF - about how to build such agents and what this means for the future of building with AI. Check it out in the comments below.
I agree for the most part. Any prompt that can be evaluated, and has enough usage to justify true "engineering" should be.
Saw your demo and it was great. Just a couple of questions Here, would the agent not need the prompt to assign itself the role and responsibility Also, did you explore some variation of RL also here?
רעיון ממש מגניב ! אבל אשמח לשמוע למה לא יהיה לך בשיטה ככל overfit? ובעיות ml רגילות בשיטה הזו בכל מקרה מגניב ממש
Hey Nir, we are building working on this with mutagent. This was a great talk. Up for a deeper discussion?
🎯
Amazing!
https://www.youtube.com/watch?v=jvKf6zXrNO4