Pragmatic AI with Matt Stauffer

Matt Stauffer:
Hey and welcome back to Pragmatic AI where we talk about using AI in the real world. What works, how to use it well, and when it causes more harm than good. Practical tools, real trade-offs for builders and business leaders. My guest today is my friend Sam Rose, a developer educator at NGROC. And Sam, I need you to start out by talking about who you are, but also...

What's ngrok and and you know, ya know, you've listened to the podcast a lot of people who are listening are not in our world of development. So you're developer educator. You're usually doing stuff for developers, but we've got a lot of non-developers listening and you know what I said you need to do that but first I really enjoy being able to give context that the person does not give before Sam even introduces himself I told him this before we got on. I think Sam is among the top five best explainers of complex technical concepts on the entire internet if there is something that is

deep and rich and nuanced and we all use it and none of us know how it works. Sam writes an article about it. You gotta set aside 20 to 30 minutes to read it and at the end you're gonna be like, wow, I now for the rest of my life know how this thing works. I deeply appreciate it about you. It's one of the reasons I'm so excited to have you on. So now, over to you. Tell us who you are, what do you do, what is NGROC, what's your role at NGROC and then I'll have lots more questions for you.

Sam Rose:
Oh my gosh, thank you very much, that's very kind. ngrok is a tool that creates secure tunnels, kind of at the broadest level, which means it's a way of devices to...

securely communicate with each other from places that they may not ordinarily be able to so for example if you want to expose your laptop to a friend of yours to play Minecraft or something like that rather than going in and like fiddling with your router you can use ngrok one command and it just works and it's secure and recently ngrok has been branching out into

much more kind of production infrastructure. So, ngrok will sit between your application and the internet and make sure these connections are secure and do things like give you tools for blocking bots and scrapers and things like that, give you tools for doing authentication, all kinds of like fun stuff like that that you would expect from a kind of gateway that sits in front of your application in production. And I'm a developer educator there. So I have recently been doing a lot of AI related content.

And I came at this from a direction of not being an expert in machine learning. I'd not really done any reading around or looking into machine learning. And then 2022 comes along, chat GPT completely explodes.

Everybody's talking about AI. AI becomes the place to be, the thing to talk about, and nobody has stopped talking about it since then. yeah, which, you know, pros and cons, right? if, it's like I'm doing a lot of content creation around AI. I have an infinite amount of people to draw from. Like everyone's talking about AI, everyone's making content. I can get inspiration and ideas from everywhere. Like I can't even like leave the room without hearing something about AI.

Matt Stauffer:
Truly.

Sam Rose:
But also like lots of fatigue around that. Like we did talk pretty seriously internally when we were embarking on going down this road of writing AI content. It's like, are people gonna care? Like, is there too much AI content? No, you know, I'm here now. So like, no, it's fine. And I wrote a post recently on, it was meant to be on prompt caching, which is this thing I don't think people know about. Like I think people understand that.

Matt Stauffer:
Yeah. Yeah. Yeah.

Sam Rose:
If you have used these providers like OpenAI, Anthropic, and you've used them through the API, and you've been billed for tokens, whatever they are, you have seen there's like input tokens, output tokens, reasoning tokens, and then there's cache tokens. And it's like, what does it even mean? Like what a cache token? And I decided to read until I understood what that meant, and then write and explain. And this post about prompt caching effectively became...

basically how do LLMs work because you need to understand a reasonably large amount about how LLMs work to understand what does this cash token actually mean, what does it come from.

And I did exceptionally well. I was very, very nervous because it was my first substantial piece of work I had done for NGROK and I was really hoping it did well. I wanted to justify this slower pace, slower frequency of releasing things, but releasing much higher quality, much more impactful things. And it did incredible. I exceeded all my expectations and people really enjoyed it. People I wouldn't have ever expected.

number one chess player in India reshared and DM'd me saying he really enjoyed it. I was like what what is happening? But it did really well and I'm working on another post at the moment about quantization and I actually in this room that I am sat in right now so this is my home office

Matt Stauffer:
That's incredible. I love that.

Sam Rose:
I have a laptop sat over there. I have another laptop on the desk next to me and I have my PC over like in the like far side of the room. They're all doing inference right now because I'm running benchmarks on different quantization levels of models and quantization being the way that models get compressed. So if you have some open source model you've downloaded from the internet they're often like really large. So the bigger the model is the more capable it becomes broadly speaking.

but the more kind of RAM you need to run it, the more compute resources you need to run it. Quantization is how they compress these models so they can run on smaller, less powerful devices. And I'm measuring like, where is the cutoff point where it's like you've now compressed it too much and the model can't do anything. So this room is getting warmer as the day goes on because there's a lot happening. Yeah.

Matt Stauffer:
It's not useful. Didn't even think about that. That's amazing.

You got your own, what is it, your Bitcoin mining situation going on there.

Sam Rose:
It's my inference sauna.

Matt Stauffer:
Oh my god, I love it.

Okay, so ngrok, and for those who are in the programmer world, we historically have thought about ngrok as a company that allows you to, when you're working on a website locally on your machine and you wanna share it with people before you actually put it on a production server, we use ngrok for that, and sounds like you guys are doing a lot more now. Your job is a developer educator, which I think is completely perfect and fitting because developer educators are...

people who are good at explaining things and explaining things in ways that make developers want to work with them. And great, go, oh my God, I learned that from ngrok. I'm starting to have an affinity for ngrok. So brilliant, perfect position for you. But as you know, a lot of people listening to this podcast aren't developers. And we want to make sure that the developers are learning something new, the non-developers learning something new. And one of the things I told you before we started was I'm really fascinated about

The fact that just as you mentioned, in order to do all the high level thinking that's very much dark developer targeted that you have had to do to be able to talk about these more complicated things, you have to start with the open questions of just sort of like, what are LLMs and what are models and what are parameters? And when people say these phrase, like there's probably some terms you've already said here that people listening have heard a hundred times and they're like, I don't know what that means. So for a normie, I think we tend to think of LLMs you know,

People probably know that means a large language model. And one of the things I often tell people is it's like it's a really robust autocomplete, right? Like it's basically been trained on corpus material. You know, they fed it a whole bunch of stuff and they said, based on the stuff that I've been fed, when I'm asked to, you know, recreate something of my own, I say, here's the most likely next word. It just does it over and over and over again until it spits something out. But that's a really highly level overview and you could even tell me it's wrong.

When you are doing all this research, when you're thinking about prompt caching and you're thinking about quantization, stuff like that, your baseline understanding what an LLM is actually doing and also what one model versus another model is in a way that most of us don't. And again, I think that you're one of the most amazing kind of synthesizers. So I'm really excited to ask you and you gave the caveat, right? You are not a machine language 20 year veteran, you know, you're not the but I think that you're a you're a bridge person. So I want to ask you to bridge for a little bit here.

When you think about what the normal human being who's not a programmer, who's not using the APIs, but they hear about AI all the time, they probably use chat GPT or whatever, and their most senior level understanding of what an LLM is is fancy autocomplete. What can we learn about what are parameters and what are actually being stored and what are those hundreds of gigabytes that you're talking about of parameters actually are and what's actually like, you know, it's just kind of, can you kind of start, like you said, you had to fully understand

These these models in order to start talking about them. Can you kind of give us that just entry level? Like what are they actually doing? What is actually what is a model anyway?

Sam Rose:
Yeah, that's a good question. Okay. I have a bunch of different ways I like to think about these things and it's not obvious to me all the time whether it's completely true. So for example, you mentioned LLMs are fancy auto-complete and I think they function. That's the user experience of them. Actually, I don't think it's accurate. Like not necessarily. And I'll hopefully eventually remember to come back to why I think that is.

Matt Stauffer:
No big deal, right? Yeah.

Mm-hmm.

Sam Rose:
So what is a model?

When you have some problem you need to solve, so let's say for example you need to convert between Celsius and Fahrenheit, there's a formula that takes you from the input to the output, right? It's like, I don't know, it's like 35 divided by nine times something, I can't remember where it is, but like there's a defined formula so you can take the input and you can get to the output. Models, broadly speaking, are, I like to think of them as canvases almost for the computer to...

write their own solution on and you feed in a bunch of observations. So if you didn't know the formula for Celsius to Fahrenheit and vice versa, but you had some observations. So you had been able to measure it was 10 Celsius and whatever Fahrenheit. And then it was this and that like you had like a table of like Celsius and you know, this is the Fahrenheit for that Celsius, but I don't know the formula. What you would do is you would create a

It's like a, internally what these models are is they're big mathematical formulas. So then they're very simple, like nothing complex is happening in them. So it's multipliers and it's additions. Like that's everything that's going on inside of it. There's just loads of it, right? So it's this big network of...

Matt Stauffer:
Mm-hmm.

Yeah. Yeah.

Sam Rose:
you take your input, which would be your Celsius, for example, like 10, and then that will branch out to multiply, say, like 100 different times by 100 different parameters. And this is what the parameters are. So you multiply it by like 0.1 and 0.2, and these start off randomized. And this network kind of just expands out with thousands upon thousands. In the large language models, we're talking trillions of nodes.

Huge, huge, huge mathematical functions, but they are just mathematical functions. They're a big long x multiplied by y multiplied by z plus b. it's, that's all that's happening. There's just lots and lots of it. But all these parameters that you're multiplying this input by start out randomized and

what we say as training a model is you put your 10 in, you do all the calculations that are inside your randomized model, you get an output, maybe it's like 102, and that's not right. You compare that then to the observation that you have, your Fahrenheit observation, and depending on how wrong it is, you then go and tweak your parameters. And there is a way of knowing for each parameter what direction it needs to go to get you closer to the right answer. And this is a thing called gradient descent.

Matt Stauffer:
Okay.

Sam Rose:
not

Matt Stauffer:
Okay.

Sam Rose:
tremendously important, just there is some maths out there that you know you need to go down or you need to go up with each individual parameter and each like training step will make small modifications to either all the parameters or a subset of the parameters to then rerun your 10 through the network and get hopefully a closer answer to what you were before. And the idea is you get lots and lots of observations and over time, this is why I think of it as a canvas, like the training process kind of paints this solution

into the weights and the way I have been talking about it is...

converting Celsius to Fahrenheit, like if you know the formula, you go from input plus formula to output. Your model is taking input plus output to get the formula. So this huge graph of weights and calculations, it's gonna end up more complex than your formula would have been when you know it for sure. But it's gonna kind of converge in on something that looks a lot like your 35 divided by nine or whatever it was that.

Matt Stauffer:
Okay.

Sam Rose:
translates Celsius to Fahrenheit. So it's a way of nudging the computer and telling the computer you have this space to work with based on these observations, come up with some formula that gets from the input to the output. Yeah.

Matt Stauffer:
I gotta pause you for a second, because you're using some phrases

that I wanna make sure I'm interfering right, but I know not everyone's heard. So you mentioned graph, and you mentioned nodes. And so what I'm envisioning as you're talking about this is a whole bunch of individual entities on a giant canvas. Each of them is like a little circle, that's a node, and then there's lines drawn between them. And that's kind of what graph is, right? A graph is basically a whole bunch of nodes that are all connected, and they might be closer or nearer to each other. And what you're saying is that's sort of how

Sam Rose:
Mm.

Matt Stauffer:
the models are representing their final understanding of things is basically graphs with worth of nodes and each node represents. And that was my next question is, a node sort of like, are we having to talk about vectorization? Because if so, I'm gonna have to ask you to explain that to us too. Okay.

Sam Rose:
Yes.

We're not getting there yet.

So imagine for our Celsius example, we start with an input of 10. You can draw that as a little circle and then that input of 10 will connect to some number of other nodes, nodes just being circles. And the idea is your value of the nodes connected to the input.

are multiplied by a parameter. So every single line that draws to a new node has a parameter associated with it. And then you take the input and you multiply by each of these connections to these new nodes. And then the new nodes contain that value. And you'll repeat this any number of times. Like this is what LLM research is. It is arranging this graph of nodes and connections and parameters and things like that. But the idea is these...

Matt Stauffer:
Mm-hmm.

Mm-hmm.

Sam Rose:
nodes eventually connect back to like a single output node where when multiple nodes connect to one node you sum the results so you'll multiply by whatever the intermediate value was from multiplying by 10 to get into this middle node

and then you'll multiply by some other parameter to get to your output, and then the output would be your final kind of Fahrenheit reading. And when these parameters were randomized, you're gonna get complete nonsense out the other side, because, you know, of course you are. And it's this process of training, like modifying the parameters that connect these nodes together, the things that you multiply by as you kind of go through this graph towards the output node, that gets you towards the solution. The solution being like the formula to get from Celsius to Fahrenheit.

Matt Stauffer:
And that initial node doesn't it doesn't represent 10. It represents 10 Celsius specifically that concept. Right. And the final node doesn't represent whatever the number is Fahrenheit it represents. So there's a node somewhere that represents. Hold on let me just look it up so we can set 10 Celsius to Fahrenheit 50. OK. So there's a node somewhere that represents 10 Celsius. There's a node somewhere that represents 50 Fahrenheit and they're not connected initially in a way that makes it clear that those are one to the other. It throws a whole bunch of random connections in.

Sam Rose:
Yep.

Matt Stauffer:
and you give it the information, the training data gives it the information saying, 10 Celsius should result in 50 Fahrenheit. And then the training process basically does a whole bunch of beep, beep, beep, beep, beep, beep, it eventually hones in on these are the series of connections and parameters that ends up that when given 10 Celsius, you result in 50 Fahrenheit. And it's sort of like hardening nodes

Sam Rose:
The only thing I would change

is that the connections never change. The nodes are always connected in the same way. It's the parameters that change. And then there will be...

Matt Stauffer:
Mm-hmm.

Sam Rose:
All of this calculation will flow into a single output node that is meant to be the Fahrenheit. And then you'll compare that Fahrenheit to what the ground truth is. So you have 10 as your input, you know that 50 should be your output, you get like 101, and you know that that's 51 too large, so we need to make some modifications to the parameters. But the actual graph itself and all the connections between the nodes is static. arranging that graph is part of the kind of process of developing a model, because

you could have any number of nodes you want, you could have any number of like layers, so you could have your input could connect to 10 intermediate nodes and then they could all connect to 10 more nodes and then they could all connect to 20 more nodes and you get this like fanning out of what they call a densely connected graph because everything's connected to everything else. This is the kind of thing that works really well through visuals and I am actually the post

Matt Stauffer:
Yes, I appreciate you going on an audio podcast.

Sam Rose:
I'm releasing hopefully next week, fingers crossed, does actually talk to walks through how these graphs are built up and how the parameters get modified. So it's ultimately a big formula. It's not much more complex than that.

There's a lot of complexity around making these big formulas faster and more efficient to run because it's very expensive to run them in their current state. Getting cheaper all the time. A lot of research has gone into making the process of running these models cheaper. Very successfully. I think it's, I think I read somewhere, you'd have to quote me on this, but I think it's gotten something like four times cheaper every year for the last three years. Like there's huge, yeah, huge algorithmic breakthroughs, hardware breakthroughs. It's a very active area of research.

Matt Stauffer:
Well, great.

So I interrupted you, but what you had kind of let us know is that we are, a model is the result of training this kind of randomized set of nodes to eventually get to the point where it's no longer randomized. So when you give an input, you can get an output. And I said, that's vectorization. And you said, not yet. The reason I said it, so for those who don't know, I...

Sam Rose:
Yeah.

Matt Stauffer:
Intentionally try to learn a bunch of this stuff and then enter the approach to the podcast as if I've forgotten and sometimes I actually have ADHD is great for that makes you a good podcast host So I'm gonna go in completely brain-dead and you're gonna tell me what I'm wrong here What I know about vectorization is it's a thing that as programmers allows us to convert a concept in written text into like a mathematical representation which sounds completely made up and I don't know how it's possible but the end result is I'm able to say if somebody

wrote a string and I search for a string that's similar but it's not the same and none of the letters are the same, you'd think the computer can't tell they're the same, right? Because the letters aren't matched. But somehow if I vectorize the search string and the result string, the computer's able to say, well, they're not the same letters, but they're nearly the same concepts, so we're gonna say they're similar. And so in my mind, vectorization is turning a concept, whether that concept is 50 degrees Celsius or the concept is the name of a company or the name of country, whatever.

into like a mathematical representation that can go as a node and then it's a lot easier to find connections and similarities to other mathematical representations. So that's why I asked about vectorization, but you said we're not ready for that yet. So did I either get it wrong or did I get to it too early or like tell me kind of where we are given what I just said and where you were actually headed.

Sam Rose:
Mm-hmm.

So it's the next step towards language models. what I was talking about there was kind of models in general. Vectorization. So I think the natural next question to ask after that Celsius Fahrenheit example that gets us towards what you're talking about is Celsius is a number, but like words aren't numbers. How do we do math on words? Like this makes no sense. And

Matt Stauffer:
Mm-hmm.

Got it.

Sam Rose:
The vectorization is the answer to that question. words get broken up. They don't get broken up into individual words. Like when you ask a question into chat GPT or into Claude, you can think of it as your kind of prompt gets broken up into individual words, but it's not quite that. They get broken up in weird ways that aren't tremendously important to this explanation. But think of it like they get broken up into words.

And there are some models out there that do actually do kind of word boundaries rather than these weird boundaries that are the kind of frontier models use, but it's neither here nor there. The words get assigned numbers and the numbers they get inside get assigned to called tokens. It's not tremendously important. They're just unique identifiers for each word. The thing that's important is then what's called the embeddings of those words. And this is where things start to get very, very weird.

An embedding is a position in space, what they call a semantic space. And you have to kind of like...

go with it a little bit and try and wrap your head around this, but you can think of it as three-dimensional space, but it's not. It's like many thousand-dimensional space. You obviously can't visualize that because it's way beyond what human minds can comprehend, but you can think of it very reasonably as three-dimensional space, and the analogies all still work. So imagine that every single word gets given this position in three-dimensional space. What it means for...

Matt Stauffer:
We'll follow you.

Sam Rose:
mapping words into their meanings mathematically is with these positions that each word gets given and they get given these positions through the training process. This is actually a side effect of training. the the huge graph, the huge mathematical formula that these large language models are with their trillions of connections, like mind bogglingly huge.

Matt Stauffer:
Mm-hmm.

Sam Rose:
a portion of them is these embeddings. So embeddings become parameters into the model. They form part of the graph of calculations. So as part of training your large language model, you get this vector representation of word position in space as a side effect of that process. There are some models that are trained specifically just for the embeddings, but large language models don't do that. They just let that happen as a side effect of the training process.

It represents the meaning. There was a paper back in 2012 by Jeff Dean of Google fame. He's an incredibly prolific and well-known researcher in the ML space. He's worked at Google since the very, very early days of Google. He wrote a lot of the original search infrastructure for Google. He has an incredible CV of accomplishments. But he wrote this paper about doing mathematics on

word locations in what they call embedding space. this vector, this place they get put in 3D space is called their embedding space. And the canonical example, see if I can remember off the top of my head, it's something like king minus man plus woman.

So you can imagine the locations in space for king and then man and then woman. So king minus man plus woman gets you into a position in space that is very close to the word queen. So you can see how that would work. So you take the idea of king, you subtract man from the idea and you add woman to the idea and you get queen.

And they did all sorts of other experiments on this. So they did, for example, countries and their capital cities. So when I say, backing up a little bit, how do you subtract and add places in space? That's kind of a weird one to think about. So if you imagine that a position in three-dimensional space could be represented as a line from the origin, so zero, zero, zero, like the origin point in three-dimensional space.

You can draw a line out from the origin to any location that you want in three-dimensional space. And if you were to take one location and another location and add those two together, the way that you would do that is you would take the original words line and then you would take the other words line and you would superimpose that onto the end of the first word. So if you had a word, say, one, one, one.

Matt Stauffer:
Okay.

Sam Rose:
and then you take another line that is like 222, when you add those together, you get 333. So it's like taking this twice as long line and superimposing it onto the end of the other line. So that's how you do additions and subtractions. And subtractions are the same, you just kind of go the other direction, kind of back towards the origin. And then you end up in a different location in three-dimensional space. And say, for example, you have...

Matt Stauffer:
Okay. Yep.

Sam Rose:
England and London, like you have England subtract, let me see if I got this right. England plus London subtract Spain equals whatever the capital of Spain is. I'm completely blanking on that right now. But the idea is that you can, there is a vector.

Matt Stauffer:
interesting.

Sam Rose:
that takes you from a country to its capital city. So you can imagine all the countries being over in one part of three dimensional space. And then you draw a vector to somewhere else in three dimensional space and you find all of the capital cities of those countries, right? this embedding space with its many thousands of dimensions that we can't possibly visualize, but three is completely sufficient to get through all the metaphors. Things are put into positions where you can...

Matt Stauffer:
Yeah.

Wow.

Sam Rose:
you can get from A to B through kind of logical means. And that is how we think, I suppose, it's how LLMs kind of begin to understand what things are and why. so, further than that. So taking this even further.

Okay, I want to broach a topic called generalization and I'm struggling to think the best way of getting from here to there. So basically... Okay.

Matt Stauffer:
All right, you think about that and I'm gonna retell something that you just said

while you take a particular brain second. What I just heard Sam say, it's funny, I'm saying this and you're actually gonna have to tell me I'm right or wrong, but I still was trying to give you a breathing second, was if I were to imagine a cluster of countries and then maybe far off to the right there's a cluster of capitals. In the cluster of countries, the UK, let's say it's in the way left side.

that means that England, then London and the cluster of capitals would also be on the way left side. Whereas if Spain was in the way right side of the countries, then Madrid would be in the way right side of the capitals. So if you move from England to the, sorry, if you move from England in the country's cluster to London in the capital's cluster, you've now discovered kind of what the move is to get from a country to its appropriate capital.

so that if you know where Spain is in the country's cluster, you take that same exact kind of movement, and if that exact same movement should land you into Madrid in the capital's cluster. And then we're imagining the movement being in three-dimensional space, but like you said, it's actually whatever, millions of dimensions, but at least it's like, okay, we discovered the shape of a movement that did a known thing, and now we can get a not known thing by approaching the same kind of shape of movement.

Sam Rose:
Yes.

Exactly.

That's exactly it.

Matt Stauffer:
Brilliant. That's super helpful. Thank you. All right. Good.

Sam Rose:
It's how like

vector search works. So there's this concept in databases called vector search, which where if you imagine more traditional searching, we'll try and find your search term in the thing that's being searched. So Google, for example, you type, I don't know how to make Bolognese, you will find things mentioning Bolognese.

And that's because the thing you have searched for appears in the website that you're searching. But vector search is much more meaning-based. if you were to search, for example... All the examples I have in my head are medical-based. So, for example...

Matt Stauffer:
Yeah, okay.

Sam Rose:
if you are describing symptoms or like I have a rash or something or then the idea of a rash is probably close in embedding space to other conditions that you could potentially have like skin condition and things like that are going to kind of be somewhat nearby so you will receive

Results back from that search that are somewhat near that location in in embedding space So it's it's much more like tip of my tongue kind of searches that work really well for that They use something called the cosine distance. I don't know actually what that is It's on my list of things to look up and learn how it works because I was recently I built

Matt Stauffer:
Yeah.

Got it.

Sam Rose:
I have my own little AI assistant that I have built myself. It runs on a little Raspberry Pi in my cluster upstairs. And I talk to it via Telegram because that's what everyone does because Telegram has a very nice bot API.

Matt Stauffer:
Got it.

Sam Rose:
and it has access to all of my bookmarks. So I have like thousands and thousands of bookmarks that I've kept over the years. They're marked as whether or not I've read them. And I can now ask my little AI bot friend, hey, I remember reading about this thing. Could you go and find stuff in my bookmarks that is like sounds like this? And it knows the things that I've read and those when I read them. So I can say like, it's a few weeks ago or something like that. I've forgotten what it was, but it does vector searches there

because it's usually kind of tip of my tongue kind of search that I'm doing But I don't yet under, I don't understand this kind of co-sign distance it uses under the hood yet. I need to go and look that up.

Matt Stauffer:
Hmm.

Okay,

that's amazing. Thank you so much for going on this route. You were talking about generalization and I paused you.

Sam Rose:
Yeah, generalisation. People think of LLMs as autocomplete and that is the kind of user experience of it and that makes sense. I think they're doing more and there's some reason to believe that they are doing more. for example, if you think of autocomplete, you're probably thinking of something that's called a Markovian process and that's just a fancy way of referring to

in all of the data I have seen previously, what is the likelihood that the word the follows the word spaghetti or something like that? I've got food on my mind. And you could train that process on everything you've ever written, say what is the likelihood that I type these words after those words? And that's called the Komarkovian thing. LLMs aren't doing that. They're not really, there's probably like some element of that going on, but it's not the whole thing.

I'll pause here to say, crucially, no one really knows what LLM's are doing. That is something that blows my mind. Like, Anthropic have published incredible research into it, but the things that we are able to understand are still very isolated, small-scale examples of things.

Matt Stauffer:
That's... terrifying.

Sam Rose:
And this is the whole field of research called mechanistic interpretability, again just a very fancy way of saying how do these things work. And it's an area of research I find fascinating to read about. So I read a paper recently about how LLMs perform arithmetic, so adding two numbers together. And...

They found in some open weight models and so you can broadly classify models into two campuses. So you've got the closed weight models and these are the models run by OpenAI, Anthropic. They don't publish them. They are completely proprietary, closed. You can't have access to them and you can't run them on your own hardware. You can't run them on your laptop or anything like that. And then there are open weight models. So these come from usually like lesser known labs. I think the average person probably hasn't

heard of like z.ai or they haven't heard of the quen models like these are reasonably like more fringe but they are completely published openly so you could go and download this model and run it on your own laptop I'm running one on my laptop behind me right now and

researchers looked into the weights of these open models and the weights, same, they're basically the same thing as the parameters that we were talking about earlier, kind of the things you multiply the numbers by in this big graph of functions. They're just.

a huge number of small numbers, like it's literally millions upon billions of very small numbers. They cluster very close to zero. They're like 0.02, 0.015, like very tiny numbers, just billions and billions of times. They looked into and they analyzed the weights for a bunch of very popular open weight models and they found that the...

models contained code kind of inside them to perform arithmetic. So they weren't like predicting it. They were literally, they had what they call a circuit inside the weights that does arithmetic. But it does it in this alien way. It does this in, it's not like you'd imagine like you learn to do summation as a child by like, you you take four fingers from this hand and two fingers from this hand and you get eight, right? Like it's very,

Matt Stauffer:
Yeah, yeah,

Sam Rose:
anthropomorphized, like it's very human. They sum numbers by rotating a point around a multi-dimensional helix using like polar coordinates or something like it's completely alien, but it's fundamentally the same thing under the hood. they are...

Matt Stauffer:
Lord. Yeah.

Sam Rose:
they are taking two numbers and they are just doing something very strange to them that ends up giving them the result of this addition operation. I think the research ended up calling it the clock method or something like that. And there's been a couple of papers published about it. then researchers were able to train a new model just on doing arithmetic and they found it produced a very similar structures. I don't think they were ever able to say for certain, like, yes, this is the exact mechanism. They found a lot of like,

Matt Stauffer:
Hmm.

Sam Rose:
Strong suggestions. This is what was happening within the model But but crucially it wasn't predicting based on what I had seen necessarily it had figured out during its training process that Rather than like trying to predict based on data. I've seen before Why don't I just make a little circuit that can do sums and then when I see a sum come in I will get the right answer because I have created a mechanism that can do sums within my parameters

And I find that absolutely wild. Like, that's so cool. It's so interesting. And it makes you wonder, like, what else is in there? And this is why I'm a little bit like, I think Spicy Autocomplete isn't quite the right, like, it's not quite exactly what's happening. It's a good framing for it, good way of thinking about it, but there's more going on, for sure.

Matt Stauffer:
Yeah, and I really appreciate you pushing on that. I want to make sure that I'm not dumbing down the response you're saying because what my brain immediately goes to is, it's not just textual autocomplete, but it is still, I don't want to say contextual autocomplete because I grew up in the church, but no, it's not just textual autocomplete. is semantic autocomplete, right? It has been trained on the meaning of as many things as it could be fed.

Sam Rose:
Ahem.

Matt Stauffer:
and it has been given formulas by which to derive and understand the meaning of what you're asking or what you're prompting and then derive the meaning of what the most reasonable kind of continuation or answer of that is. And it is then saying, based on all the training material I've been given that has then put, you know, nodes and weights all together in a magic map, here's the most reasonable output. What's next?

If I'm hearing that right, obviously that's much richer than a fancy autocomplete. I think sometimes the conversation about autocomplete is useful because a lot of people think that there's actual humanity. you know, the autocomplete is often a corrective to someone who's assigning intelligence in a place where it's like, nope, this is math. But tell me, I, did the way I just summarized it, did I still oversimplify it?

Sam Rose:
Yeah.

I don't think so. It's a little tricky for me to say you've over simplified something because I have learned a simplified version of it myself. I'm not as in the weeds as other people. Yeah, yeah, yeah. I am a very enthusiastic person who is excited to learn about these things and take that and give back to a wider community. It's fun, I think about what I'm doing as enduring reading the papers and enduring...

Matt Stauffer:
Okay.

Sure. Right, you're not an ML researcher, 100 % caveat, yeah, yeah.

Yes, 100%.

100%.

Sam Rose:
running

my little inference sauna to tell you actually you should do this. Yeah.

Matt Stauffer:
So we all don't have to do that. Yeah. I mean, the thing is that's who you've been

across everything you've ever done. You've endured, like you talk about how much you read. I'm like, I love reading, but I have a pretty strong limit about how much I can read about deeply technical subjects. And my limit is way before yours. So like you take your much further limit than mine. You internalize it all. You sum it very, very grateful for you doing that.

Sam Rose:
You

Matt Stauffer:
so one of the things that I'm really interested in is I think that people benefit from understanding how these systems work because it's gonna change how they interact with them on a practical day-to-day, you know thing. I want them to be able to use it better by knowing and understanding. But I also think it changes how we think about it from like a societal perspective and a theoretical perspective. And there's a lot of questions we're asking about like...

Are the large AI companies evil? Water usage, electricity usage, stealing people's content. So I think all of these conversations, the practical, I'm using it and I'm betting from it, the legal, the how should it be regulated, the ethical, all these are benefited from us having a better understanding of what these things are and aren't and how they work. I asked you the really, really rudimentary rudiments. Are there some pieces where you're like, if people only understood this or people really

tend to misunderstand this, that you could kind of like clear up some of those things that you think will help all us in these both practical and more theoretical kind of relationships with AI and with LLMs?

Sam Rose:
The thing that bugs me, and I see a lot sometimes, is I think these models have got to the point where they are genuinely really good at quite a wide range of things. Certainly better at a wider range of things than we are used to software being. So that's as software people and...

Matt Stauffer:
Yeah, yeah.

Sam Rose:
I think people using software without writing it, right, like the chat box you can talk to in chat GPT seems to be able to do everything. It's very, very general purpose. I think what happens as a result of this is people start to assume it's good at everything. And that's not true. And it's very challenging sometimes depending on what you're talking to it about.

Matt Stauffer:
Yeah. Yeah.

Sam Rose:
It's challenging to know what it can and can't do. Ethan Mollick author of a book called Co-Intelligence does a lot of talking about AI, one of my favorite people talking about AI at the moment. He calls this the jagged frontier. So the idea that...

An LLM can be phenomenal at producing software. Like writing code, it's objectively very good at writing code. It doesn't write the world's best code, doesn't always write correct code, but it's very good on average at writing code.

Matt Stauffer:
Mm-hmm. Yeah.

Sam Rose:
And then it can be terrible, something very basic. Or it can solve like a math Olympiad level math questions and it's out there currently solving previously unsolved math problems, which blows my mind. And then it gets basic arithmetic wrong and it fails to use specific emojis correctly and it gets itself stuck in strange loops and that makes it such a...

Matt Stauffer:
Yeah.

Sam Rose:
difficult thing to reason about and what I wish people did was have a bit more rigor around that and a bit more because you see like a new model comes out like the Opus 4.6 when Anthropic has come out and 18 minutes later somebody on Twitter decries that this is better than everything and then another person decries that this is the worst model they've ever put out and

Matt Stauffer:
Uh-huh.

Sam Rose:
One, after 18 minutes you can't possibly know that, like you cannot possibly have evaluated it enough and benchmarks are a whole other thing, right? I did a big post recently about benchmarks and that was a fun exploration, but benchmarks give you a narrow slice.

Matt Stauffer:
huh.

Sam Rose:
of how good these things are and it specifically annoys me when software people, they hold up SWE-bench as like the pin, like SWE-bench is a test of a model's ability to solve some programming problems and

Models are now routinely scoring like 85, 90 percent, very very high scores in these benchmarks and everybody is like Opus 46 beats Codex on SWE Bench and that means it's better coding. And it's like, no, no, you've never looked at what happened. Like I feel like the guys in...

Matt Stauffer:
Yeah, yeah, yeah.

Sam Rose:
in the big short and it's like you've not read the mortgage statement like you've not looked at these things like this is this is a giant tower it's like so the prob like there's several problems with SWE-bench and i really hope you retire it in the near future one it's all python it doesn't test any language other than python two it is all what is called zero shot so the model is given the context it needs and it's asked to produce the output it's not like it's not what we would call agentic these days so it doesn't have any access to like

navigate your computer, it doesn't have access to run programs, it doesn't have access to like check itself, it just it's given one shot, well it's given zero shot as they called it, to produce a fix for the problem. It's also all of the the methodology that they went through to create SWE-bench was they took

Matt Stauffer:
Yeah, zero, yeah.

Sam Rose:
pull requests in popular open source projects, a pull request being a proposition to fix something in an open source piece of software, and they split the change into the test code and the non-test code. So most of these changes come with automated tests to make sure that the thing is doing what it's supposed to be doing. They run the tests before applying the change, they run the tests after applying the change, and the tests should go from failing to passing.

So this, methodology is that you went through loads of PRs and they found good candidates for well-tested fixes to real problems and...

the benchmark takes away the solution. It gives the model the test, it gives it the context of the problem, it gives it some relevant context from the code base, and it's like produce the fix. But this is all public data, the model was almost certainly trained on already, so that's problematic, that's called, they call that contamination. And people have shown, there's a paper I read from last year where,

Matt Stauffer:
trained on.

Sam Rose:
They asked, they didn't give the model any relevant context of the code base. They just gave it the bug report for the PR in a project and they asked it what file would the fix go in? So the model has been given no information about the code base, but it was able to accurately tell you what file should be fixed. So it knows it's in the model weight somewhere. It knows too much about these problems. So it's, I think it's had its time. I think it's, it's passed you, but the,

Matt Stauffer:
Yes.

Yeah, it's already been trained. Yeah.

Sam Rose:
The brawl thing I wish people were is a bit more rigorous and a bit like give it more time do a bit more like strict testing of these things come up with your own little benchmarks like I love that Simon Willison gets them all to make pelicans on bicycles that's his you know trademark thing now but I think you should have your own set of

Matt Stauffer:
Yes.

Sam Rose:
well-defined tests that you do with models and you can do a good side-by-side comparison of the outputs from each model. Just keep true to yourself and just to be a bit more rigorous about this kind of stuff.

Matt Stauffer:
How would you see that applying? the answer might be it won't. How do you see that applying to non-programmers who are, you know, using tools? You know, they probably have chat GPT, they might have, you know, Claude Cowork. They're probably using some SaaS tools that are AI amplified. How do you think that awareness should or shouldn't change what their interactions with their LLM friends, you know, usually through chat, but not always, look like on a day-to-day basis?

Sam Rose:
I don't think the average user, I say that with apologies, I don't think the average user should really care about that. I think the strategy for the average user is just use the latest thing and that's gonna be fine, right? The average user, think their energy should be going into...

understanding that these things aren't the oracle, they don't know, they aren't the source of truth, you should certainly be checking everything it tells you. My wife uses it to ask questions about like health stuff for the kids and things like that and I'm always like you need to really go and like make sure what it's saying there is true. But she also uses it for fun stuff as well like she like paints rooms in the house different colours using the image generation and it's really nice for that kind of thing.

Matt Stauffer:
Mm-hmm. Yeah.

Uh-huh.

Sam Rose:
And then my mum was using it recently. My mum was prompting it to make a website for her to... But she wasn't like... She has no idea about making websites. She was trying to get a website design that she could give me and then I make it and go and do it for her. Yeah, we ended up making it in Squarespace and it looked sort of similar to it. But for people like that, just...

Matt Stauffer:
Got it. Make sense?

Yep. Perfect.

Sam Rose:
Being very cautious, especially if you're gonna give access to your computer as well, if you're be using Claude Cowork and things like that, have good backups, make sure you're not feeding anything like super duper sensitive into it. I think we are overdue the mother of all data leaks. Like...

Matt Stauffer:
Yeah.

Sam Rose:
When Anthropic or OpenAI is compromised and all the chat logs are leaked, that is gonna be a hell of a day. It shocks me that as far as I know it hasn't happened yet. Touch wood, I guess, but when it does it's gonna be epic.

Matt Stauffer:
gonna be a terrible day, my gosh.

Yeah, I'm not aware of it. Yeah.

Yeah. Okay, so that was for, with apologies, normal people. For programmers, you said that one of the things that's most frustrating is the people who are opining so quickly and you just said, give it some time. The good news is I think most of those people are, I just saw something this morning and it was called shipposter.dev and it basically says it compares somebody's Twitter handle with their GitHub handle, which is what programmers use for.

writing code and it says, much time does this person spend talking versus actually shipping code? And the good news for me is I think that the vast majority of people who are sharing loud opinions on the internet about these AI tools, that's what they do, right? They share loud opinions on the internet. Thinking about the people now in the programming world who are using LLMs, who are using code assistance toolings and everything like that. Outside of the content you've already created, which we're gonna link in the show notes and everything like that.

Is there a direction you would really recommend people go in terms of the learning? You're like, you know what? I think I'm a better programmer because I've spent time learning about this or that.

Sam Rose:
I get a real awful list of names of content creators I respect and follow and really enjoy. So Stephen Welsh of Welsh Labs on YouTube makes, for my money, the best videos in the world on AI topics. They are just phenomenal.

Matt Stauffer:
Yeah.

Love it.

Yeah. Okay.

Sam Rose:
He's fantastic. I also love the videos of Julia Turc. That's T U R C She I she made a video that saved my life recently. She reverse engineered one of the ways that a

There are pieces of software that can run LLMs, there's llama.cpp is one of them. She made this video reverse engineering how their quantization works because it's not documented and it was super helpful for the thing that I'm writing at the moment. And I and I watched more of her videos and she is really good. Like she is incredibly knowledgeable about the topics and she makes very very high quality videos.

Matt Stauffer:
Okay.

Awesome.

Sam Rose:
Sebastian Rashka has written the book, how to build an LLM from scratch, I think it's called. Very good book. That's where I got my kind of first introduction into what even is a language model, because I didn't know back then. And he takes you through GPT-2, so the open very, very early model, like pre-Chat GPT.

which it's all kind of open now, openly available. So, and he gets you to kind of write the code to load in the weights and run them in the correct way. And you actually, like it produces very like meaningful output. Like it's not good by today's standards, but like it's English words that I recognize in like grammatical structures that make sense.

Matt Stauffer:
that's cool.

Yes.

Wow.

Sam Rose:
So throughout the book you build it up and he explains all the different structures like there's lots of different parts of the graph that makes up the model with all the kind of mathematical functions and he explains all the parts are for and how they work and it's very good. Very very recommended book. Then the writing of Nathan Lambert is incredibly good. He works, can't remember what the lab is called, but they make the Olmo open source models which are competitive open source models.

He is an excellent writer. He has a sub stack which I follow Yeah in terms of builders Aaron Francis huge inspiration to me. I have really enjoyed following the stuff that he's doing at the moment with Faster.dev and then you know his episode on here was very very good Mitchell the guy who builds Ghosty Yeah, he's I think his opinion is very good and then

Matt Stauffer:
Yeah

Hashiba Ruhyun

Sam Rose:
Armin Roeniger, the... was it Flask? He made Flask back in the day or something, the Python library. His writing... So Mitchell and Armin are both very, like, in the trenches, using these tools every day, know what they're talking about when they talk about it. They're less, like, research-y side, whereas Sebastian Reschke, Julia Turk, Stephen Welch, they're much more kind of ML research-y side of things.

Matt Stauffer:
I actually don't know.

Sam Rose:
I think all those people are fantastic. I get a ton of knowledge from them and I would highly recommend following all of them.

Matt Stauffer:
Amazing. And yes, you're right, it's Flask. I actually know his handle. I don't know how to pronounce it, Mitsuhiko. I had no idea what the guy's real name was until today. So it's awesome. Amazing. That is incredible. So people have been giving me crap for cutting these at 45 minutes. So I've allowed this one to go to 55, but it is at least time for us to start wrapping up. And as always, I'm just like, we could have talked about 17 other things. So let me just.

Sam Rose:
Yeah. Yeah, yeah.

Matt Stauffer:
with no constraint on time. Are there any other topics that when you read the agenda and for those who don't know, I had somebody else scheduled to record today and they weren't able to come. Sam jumped in at the last second. I literally asked him maybe 12 hours ago, 13 hours ago, hey, can you jump in? So you are a lifesaver. I'm so grateful to you, especially just not having level preparation. You're such a preparer. So thank you so much. Is there anything else when you read over the agenda 13 hours ago that you're like, you know what? I really wanted to take the time to talk about that because I'd love to make space for it if so.

Sam Rose:
First of all, you're more than welcome. I was extremely excited to be invited. Notice be damned. The one question I spent time ruminating on was how has your opinion on AI changed? And you put over the last six months, I was like, six months, that's what, October last year or something like that? I don't think it's actually, I don't know how true to myself I'm being here, but I don't know if it's changed much. I think if you rewind further,

Matt Stauffer:
I love it.

You

Yeah, over the last six months.

huh.

Interesting.

Sam Rose:
So something that blows my mind, Claude code is like just over a year old, right? Claude code feels like it came out in the 1700s to me. It's the so much has happened, so much time has passed. If you rewind a year, I was a bit late on the whole agentic coding thing that Claude code came out, people were using it. I was like, this kind of seems a bit lame to me.

Matt Stauffer:
Yes. Uh-huh.

Sam Rose:
And I wasn't down on AI. I'd been using Copilot for auto-completions and I was, I distinctly remember when that came out and my boss at the time told me, he said to me, have you tried it? And I said, not yet. And he's like, you really need to try this. And it was, it was...

Matt Stauffer:
Yeah, yeah.

Sam Rose:
Magical. It just completely blew me away. I couldn't believe how good it was. I never experienced anything quite like it before and it seems quaint now. Like that's just like what three years ago or something like that and the progress has been incredible. But the agentic stuff, I tried a few tools. I tried a tool called augment, which I'm not sure if they're still going, but at the time they were similar. Like you could chat with your code base and ask it questions in natural language. And I had the question I like to ask it. The code base has

on the time, I like to ask it for this URL, take me to the root handler for it. So, like, where is the code that handles this request? And it consistently failed, and a couple of other tools I tried consistently failed. they took really long, like 10 minutes just sort of spinning, and I could have found it by now. This doesn't feel like it's going to be helping me. So I had, I think, unfairly dismissed them, and...

Matt Stauffer:
Mm-hmm.

Sam Rose:
It was the, there was a fly.io post. The title was something like, all my friends are wrong about coding agents. Like they had a really good title like that. And I remember reading it on a bus, coming back from something. And I'm reading it and I'm just like, okay.

Matt Stauffer:
Yes, it was very good.

Sam Rose:
I respect Fly, I think Fly have got an excellent brand, excellent product. I was like, okay, I need to go home, I need to try this thing. I tried it and it opened my eyes. I was like, oh my word, I have been sleeping on this. I feel like I'm late. I feel like I'm not gonna make it or whatever. So that's another thing, that's not true. It's so easy to get into. think people telling you you're behind and you're gonna miss the bus or whatever, no, you can get up to speed with these tools in a couple of

Matt Stauffer:
Hmm.

Sam Rose:
weeks. You can get to proficiency in a couple of weeks. It does take a bunch longer to get a good feel for what it's going to be good at and not good at and stuff like that. But broadly speaking, you can join in whenever you want to. If you want to, there is a very, very shallow learning curve, I think. But yeah, I was late and I was down on them early, but

Matt Stauffer:
Amen.

Yeah.

Sam Rose:
Then I kind of went into my obsessive phase, I think everyone goes through, I need to maximise my use of this to get really good. Then I was trying all the different ones because Codex came out, which is OpenAI is offering, then Gemini, which is Google's offering. Yeah, now I think I'm reasonably well versed in all of them and I enjoy using all of them. I think my preference is Codex. I don't jump around as much as I used to. I jump around when I run out of quota. That's kind of about it.

And I rarely run out of quota. Codex seems very efficient at using the quota, so it doesn't happen very often. But they're all good. You can get involved whenever you want. It's super easy. Ethical concerns may be worth touching upon. I kind of... I don't like that they have flaunted...

flaunt is probably not the right word, but they're kind of getting away with huge amounts of crime. They're getting away with the huge amounts of stealing. And me, like I know, eyes open, using these tools, paying these companies money is an endorsement of that.

I don't have a good excuse, it's bad, I'm part of it, my eyes are open in that regard, I don't feel great about it, at the same time, I don't feel like it's something I can ignore either. It's become so pervasive. Yeah, like, and that's, I think that's a cop-out, right? Like, it's slightly coping. But at the same time, I'm in that position now where my paycheck does actually depend on it.

Matt Stauffer:
Yeah, it's what we keep coming back to, you know?

Yeah.

Sam Rose:
I don't think these tools are going anywhere. Similar to, think Jeffrey Way said the same thing. They're not going anywhere. I don't think you're gonna help yourself by pretending they don't exist. I think I still have 20-so years that I need to be kind of sat here making software before I can retire. So I think this is the new reality. Even if progress stops tomorrow, these things will stick around and they'll be a big part of how software is built.

Matt Stauffer:
Yeah.

Yeah. I'm especially torn because not only do I want to be a good and ethical person who wants to lead people in the right direction, but then also I want people to be able to provide for their families and I'm running a consultancy that needs to be able to make money. I also am an author whose content was scraped by Anthropic and my publisher and I are a part of a giant, what's it called? The lawsuit, you know? And so it's just sort of like...

This is a really weird, I use Claude code every day and I'm also a part of lawsuit against anthropic and I'm just like, it's, I think there will be a time where, I don't know, but I think there'll be a time where the action to take will be clearer because what I've found is that the action of head in the sand, not engaging, has proven to not be impactful on the future.

Sam Rose:
Yep.

Matt Stauffer:
You know, it doesn't have an impact. Unfortunately, I'm in a position where what I would like us to all be doing is being able to take political and legal action that can help. You know, because to me, I'm just like, there's some things that we can do with our wallets. But I've said for a long time and I'm like, you know, stop drinking out of plastic straws doesn't do anything. It's a nice idea that doesn't actually have a day to day practical impact. Whereas the legal system has an incredible ability to regulate what environmental damage big companies are doing that has a

thousand times the impact of people not drinking out of plastic straws. And so while straws are fine, it's a panacea. It's not actually impactful, but it tells us you need to sacrifice as an individual to do this thing. And so I don't know if that's the case. Like somebody could tell me, actually, if you all just would choose to boycott the AI companies, then the environment would be in better shape. And if that lands on being the answer, OK, I don't think it is right now. don't, you know.

I'm one person, I'm not a researcher, I'm not a politician, I don't know, but so far I've not seen anybody be like, this is the way to be ethical in the world of AI and it is ABC. So I'm like, so I think we're all just trying to figure out what we can do while also still be able to provide for our families and find some ethical middle ground. So.

Sam Rose:
I think it's, I think AI is also, it's systemic now. Like, unfortunately, like there's like a trillion dollars of new capex or upx or whatever it was. it...

Matt Stauffer:
Mm-hmm. Yeah.

I believe it.

Sam Rose:
It is a systemic like the it's now kind of part of the foundation of the financial system of the world like unfortunately, it's a And then that's not to say like you know if you boycott it You're now causing more harm than good like no like if you want to boycott it You should boycott it right if you don't believe in it That's and I think there are some people out there very successfully not engaging and they're still doing fine. I have huge respect for that

That wasn't, that didn't look like the right move for me at least. I don't know. I completely wouldn't blame you. If you're one of these people that doesn't like AI, if you think my position is a cop out, that's absolutely fine. Like I agree to a large degree, but it seems like the most robust way forward for me to continue providing, continue making money, continue progressing.

Matt Stauffer:
Yeah. Yeah.

Sam Rose:
I am being a bit selfish in how I'm progressing in that way, it's not, again, not going anywhere. It is now, I think, a systemic part of the world.

Matt Stauffer:
Unsurprising that you have empathy and nuance and ownership of your own part of the problem because that's the type of person I know you to be. All right, we're past an hour. This is the first time allowed us to go past 45 minutes. So Sam, thank you so much for hanging out a little bit longer than usual. Really appreciate you sharing what you've done. If people are fascinated by the man, Sam Rose, and they want to learn more about you, how do they follow you?

Sam Rose:
Thank you.

Everything's on my personal site, so it's samwho.dev, so it's S-A-M-W-H-O dot D-E-V. All my socials and stuff are there, all the stuff I've written is there. That's where you can find me.

Matt Stauffer:
Brilliant. Sam, thank you so much for hanging out, especially with such late notice.

Sam Rose:
No, thank you, Matt. Had a great time.

Matt Stauffer:
And rest of you, we will see you all next time.