Was fun to be at a dev event in Bengaluru and demo an app I built recently for deep research with multiple models and decision frameworks... think of it as "chain of debate"... Next stop, Copilot!! | Satya Nadella | 202 comments

Transcript

You know, we had a Thanksgiving weekend in the US a few weeks ago and so I had a chance to say like, what else can one do over Thanksgiving other than build? And so I built an app of my own using all of the stuff that Karen was showing. So this is my Azure environment. And by the way, this is my regular PC that I travel with, so hopefully nothing happens but. So this is it's in fact, I have this app deployed I think in South Central Canada. This is my GitHub repo. And it's fun, right? So I kind of have my typical setup in fact, is Windows 365, which travels with me essentially everywhere. And then in there, obviously I have my GitHub and then it's called Spaces. So it's kind of like turtles all the way, right? So you have your code Spaces running on GitHub. In a Windows 365 instance and then the, the idea that now you can go in and my favorite thing of course to do is your, you know, come in in the morning and justice start issuing whatever coding tasks. And so this is where I go and just, you know, I usually fire off 5-6 things. It creates 5-6 draft branches and then ultimately at the end of the day, I go back and mostly delete the branches. But there is a PR2 I'll sort of accept and go on and. Work with it, right, So it's fun. And then the fact that you have all these models, in fact, I think now you know, I'm mostly using a lot more Codex Macs. It's fantastic. It's fast. I'm using obviously cloud cloud Opus Opus 4.5 as well. But you know the thing that I've now gotten used to is I have enough trust to just say auto and it picks. And so you know, if I'm just really I don't I want to really be efficient with my token limits. And so therefore picking auto seems to be. Really, really a good way to go about it. So anyway, so great. I did all this. So what the heck did I build? I So what I said is, OK, I what's my dream? My dream is to figure out how to get a job in this copilot team. So I said, man, I like your deep research stuff, but I want to add a lot more to deep research. So I built my own deep research. And so, you know, with all these models available, I said, OK, what if I could start putting new decision framework? So one decision framework. You know, Andre Karpathy recently talked about this LM Council, which I love a lot, right? So I implemented this idea that you can now have all the models available to you. So GPU Opus. This Gemini, Kimi, K2, Grock, what have you, all of these models. And then you can select a chairman, right? So you have council members or the selection committee, and then you have a chair and then you can go issue any query and have it come back and tell you what it thinks. Then another decision framework I implemented was this thing called EXO. We did this actually in healthcare first, right? So this was done. There you go. Now let's see, let's see how all my OHH, so fantastic. So, so, so DX O is a thing that we, as I said, we implemented for healthcare and you have specific roles. So you have a lead researcher and the lead researcher in this case is Opus. It sort of does the breadth first research. Then you have another role, which is a critical reviewer, right? In this case, I I'm selected 51 G PD51 and their role is to find. Any method errors, right? Especially bias and recency bias, what have you. Then we have a data analyst. So I picked or I picked a domain expert Gemini, then said data analyst, I picked Kimmy K2. So this you know, when we publish the DXO paper, it performed better than any front frontier model, right? So this is in the context of very high stakes health outcomes. And I said, hey, I want same thing for any decision I want to make, right? So I implemented that. So I implemented an another one as well called ensemble. So this is basically used all models and as just essentially a set of MCP servers anonymize their responses. So take out even who is responding what, what give them you know, alpha, beta, gamma and then synthesize into one response. So these are three decision frameworks. In fact, I even extended it by the way, I built a shopping thing, I built even a finance thing, but. Basically decision frameworks and then of course like a good sort of crazy South Asian cricket fanatic. What do you use it for? To select the all time best Indian Test cricket team? Especially in a time like this, After what happened in the last Test series, I think it's time to get to work. So what I did is I'll show you the history side if I go in. In fact, you'll see that mostly I that's what I've been using it for, which is the MLB lineup was also just crazy. It is fantastic, but the test team. So I let me go show you some of the stuff. So this is the. What happened? So the this is the chairman synthesis, so this is the API council. You know, so it came back and it says, you know what? I figured out all of the you know, you know, Sunil opens Avago, you know, goes opens with him. David makes it obviously and what have you. But look at this. Areas of complete consensus you know was Gavaskar, Sehwag, Dravid, Tendulkar, Kohli, Kapil dev, Ashwin and Bumrah. Key debates ohh man VS Do you have him or you don't And look at the way it's sort of made the decision. Phi one basically said an end clause inclusion of Lakshman was heavily weighted because of the crisis management. And as a good Hyderabadi, I love GPD 5/1. And flood. And and of course Kumbla versus here, this is pretty cool. You know, do you really need left arm swing or you need? Whatever you call Kumble bowling. Which is obviously the stats one out 619 wickets and and so they selected comma and OHH and then captaincy debate Coley versus Doni and they selected Coli. And so it goes off and then annotates. What I love by the way, is I implemented it as even a streaming thing. If it is not, you know, it's not deployed in South Central Canada. I would sort of show it to you, but it's just nice to see. It's essentially like a chain of debate, not a chain of thought. Alright, so I can see the morals debate and then synthesize. So anyway, so that's one example. DXO is another one. This is really neat because what it does is you can see it by role, right? So I see the first, the exhaustive search of the lead researcher. It gives me again, what did what are all the things the critical reviewer will find like, OK, what are the method problems and gaps and weaknesses? So for example, error bias, right? Classic thing that happens right when you sort of. Compare across generations, you make all kinds of mistakes because you don't adjust for any of the stats and the difficulty of the wickets. I mean talk to him and I mean walking out in a W Indies or an English wicket that was not covered. I mean, I mean how does one play even in anyway? So to be able to take all of that into account and then see the debate between the various models to resolve. So anyway, so I've had and then the same thing with ensemble as well. So the point. I wanted to make was, I built this over maybe a couple of hours and now I'm constantly refining it. In fact, one of my hopes, as I said, is to be able to, by the way, this is all going to come to copilot. And I'm convincing my friends and they're saying, yeah, you can apply as a junior product manager if you are competent enough. So I'm still in the process of interviewing. But now that said, I think all of this will come because this is to me the next generation of metacognition, right? So if you think about these decision frameworks. Have all these agents, you're working with agents, but the metacognition is still us. And this is tools for metacognition. It's how I think about it. And so to me, building out these type of frameworks, building out these type of agents is interesting to select the Indian cricket team. But think about whether it's a supply chain decision, whether it's a healthcare decision, whether it is a finance decision. These type of chain of debates with multiple agents participating is going to be a lot of what we are going to all build in our systems, in our agentic system.