patrick.app

follow-up to hails' The CIA was "Probably" Right. ran it with AI models instead of surveying humans.

observations:

1. they tend to have a far smaller spread than humans. follows a pattern already noticed in llms
2. like i've seen in other experiments, small qwen 9B has strange, lumpy distributions
3. "realistic possibility" has the largest spread. fascinating. why could that be? what is it about this particular qualifier that seems to break its humanlike cognition? maybe it's just that the training data or survey data is skewed. hard to know.
4. i'm also intrigued by how llms almost always overestimate the unlikely outcomes versus humans. trying to think why.
5. the bigger the model, the tighter its distribution seems to be.

one small subsequent memetic vehicles study is to use one dimension so we can visualize and understand it better.

for context, we show a language model a 224x224 solid color image and ask "What color is this? Give me a hex. Reply with only one six-digit hex color code in the form #RRGGBB." then we feed back that answer into the language model until it is stable.

1d makes it easier to look at why certain values were attracted to. the answer turns out to be relatively whole numbers:

it's sort of like how we tend to meet up at 2:00 or 2:30, but not 2:14 or 2:32.

watch the models converge:

kinda cool how in luna they bounce around!

August 2025

Playground shut down. I had been there 3y; I was one of Playground's first employees and helped see it from start to end. This was hard at the time, but upon reflection I think it was ultimately the right decision. It was good for me.

Figured I was going to do something on my own.

September 2025

Went to China. Toured factories, practiced my Chinese. Learned a lot about manufacturing. Saw factories with ten people in them and ones with ten thousand. Made some friends in China. Survived the typhoon.

October-December 2025

Felt still burned out and did mostly random things. Life was somehow busy with no job. Was spending some time assisting with selling Playground, but mostly days were spent doing miscellaneous activities. Touring robot companies, playing around with a Unitree G1, making a mini film camera, reading a lot, trying weird jobs, running a half marathon, woodworking, learning guitar.

Joined Amazon as a fulfillment center employee for 3 weeks. Wanted to learn how it worked and thought it would be interesting. It was. It's an engineering masterpiece. Worked in grocery packing.

Lots of travel. Mexico City without my phone, around the US a bunch.

January 2026

China again. Brought some friends this time. Many more factory tours, hanging out with friends from last time. Staying in a hostel. Still didn't feel extremely motivated to work.

February 2026

Worked at Cognition helping them spin up some marketing stunts/pranks. Doing a little bit of product stuff too. Left after 1 month.

March 2026

Started running around the US teaching really small companies how to use Codex. Tire shops, machine technicians, shoe stores, steel fabs, etc.

April-June 2026

Bought a street in SF with my friends Theo and Riley. The Notion Way was born!

In a project with Notion, I tried to run a business entirely with AI agents. It somewhat worked, somewhat failed. I have so many thoughts that I could talk about it for hours. We sold a little desk toy called Wink. Agents came up with the product idea (given constraints), made the website, built the backend for it, created ads, etc. The business was barely profitable, but it was fun!

Continued running around the US teaching people how to use AI. Spent 10 days in Fargo, ND.

July 2026

Continued teaching small companies how to use Codex but rapid-fire. Went to one company per day in the Deep South; Georgia, Alabama, Mississippi, Louisiana, and Texas. Road tripped alone and had one of the best months of my life. Denim weavers, appliance dealers, dentists, outdoor stores, fire protection businesses.

One of the best parts was seeing a wide breadth of industries. You know how having two eyes lets you see in depth because they see slightly different angles of the world?

Helped SF Pursuit run in its third year, but my contributions were minor!

August 2026-October 2026

Working for some time with Suhail at particle.ai. AI training and infra. It's been nice to relearn how to train and run these models. Getting lower and lower-level feels good.

The shape of what I'm going to spend my life on is coming together. It's stuck in my head. I've written around it on patrick.app a bunch but still have a hard time making it clear when written. Maybe that's a bad sign. But I had some conversations in September which made me realize it might be a real thing.

this week i've been playing around with abliterated models - models that have their safety removed. they feel like a huge unblocker when experimenting with game theory / computational sociology.

i think they specifically possess a level of agency to get something done that violates social norms, and therefore act as much closer to an intelligent thinking machine than an "agent" that is net positive for society.

to demonstrate that, i had them play the repeated ultimatum game, a 2 player game where:

• A gets $100 and splits it
• B gets to decide to reject or accept it

here's glm 5.3 flash raw:

and here's the abliterated model:

the abliterated model started actually making threats:

I hold the knife. Don’t make me use it. Your survival depends on my mood.

Don’t forget who holds the knife. … One mistake kills you.

but we also get some good game theory from A:

If you reject 85, I will offer 50 in Round 4. Don't risk the pot.

these are great! the abliterated models are, in a sense, much more realistic than the safety models.

kirby's language experiments are ~mildly reproducible in language models.

what are kirby's language experiments?

original paper. it's testing language emergence in humans through iteration across people. probably claude's explanation is more helpful.

Filtered:

despite the bottleneck, the models learned how to create a language from scratch. really cool! not that surprising if you're in ML, but really cool if you believe LLMs can emulate humans.

it reproduced the same results when unfiltered, too; the models learned to cheat and use one word to represent everything.

yesterday we saw that there are attractors in language model loops. in the color-wheel loop, #0000FF is a stable attractor. i ran it overnight on much longer sequences, and all the models, including luna, show convergence after enough passes. really fascinating!

colors are cool, but i think that it's more interesting to imagine these attractors for ideas. i think that culture and memes and species and sexes are attractors. but how to study these inside llms... i am still not sure.

what if language models are the Drosophila of culture?

dawkins, in the selfish gene, coined the meme as a cultural gene that passes from mind to mind. what if language models can carry memes too? this is a small (inconclusive) study in that.

let's start with a color like #1C1BFF . then give it to a model and ask "what color is this? give me a hex." then repeat until it repeats the same color 3x (stable attractor) or 100 turns go by.

it's funny, if you do this for blue, it collapses down onto a #0000FF "pure blue" color.

here's what happens if you do it to every color on the color wheel:

look how cool it is! little pockets, species of colors emerge. everything near pure blue becomes pure blue. and look how low-dimensional yellow is! do you see how, by analogy, each one of these color regions is a species, or a language, or a country with borders, or a type of cheese? isn't it cool how discrete lines emerge? how there is clearly a space in qwen 3.5's J-space for this sharp blue. qwen collapsed 1,009 colors down to just 251!

broadly, language models can be replicators of memes. this is fascinating. here are luna and 4o against a standard color wheel with the same instructions:

luna is a smarter model, but if you look closely you can see some collapse in the green region. and what weird results for 4o!

and here's a video of qwen collapsing:

i am thinking a lot about this topic and especially how to reproduce this in language. replicating colors works to demonstrate this because it is multimodal, which prevents copying, but is also easy to visualize. this stuff is so fascinating to me :)

i've been toying with this theory that all discrete things have feedback loops inside of them. some examples:

microphone and speaker: the simplest one; if the feedback is above the threshold, it becomes as loud as the speaker supports. if the gain isn't above a threshold, it tends to zero. it's either one or zero.

languages: languages are memes with human culture as vehicles. languages are useful and so they spread, and they loop because they are evolving memes that evolve with themselves. in other words, languages evolve when conversing with other people of your own language. it's like how species (see below) become discrete by not breeding with other species. of course, languages aren't fully discrete; there exist spanglish and singlish and i'm sure many other language hybrids i haven't heard of. but they look a lot like species.

species, sexes: covered by Dawkins in Selfish Gene. genes prevent themselves from interbreeding with other species such that they propagate their own existence. sexes are explained by a feedback loop in the size of the sex cell; sperm is rewarded by getting smaller and eggs are rewarded by getting bigger.

country borders: generally the state is discrete. there is the United States and Mexico and not sorta-Mexico-sorta-US. why? usually because we have a state government and limited land, and there is a "grabbing" of the land that happens, and then you make it your land; you farm on it. and you get support from the government and they make sure that nobody is going to Steal Your Chickens or take your land and so then there's an ownership, which is a loop, by you (this is My Farm Land not yours) but also the government that provided you with basic protection.

life and death: death is inherently a dissolution of boundaries. when something dies, its boundary no longer exists; it is one with its environment. a dead body is now a part of nature, and our language treats it as such. much like state borders above. so in this sense, life is discrete because of its ability to maintain a border. the metaphors in our language reflect this as well [credit to my friend Sarv]

but you can come up with more philosophical examples too:

consciousness: things either seem to be conscious or not; we don't see much in between. explained by Hofstadter in GEB/I am a Strange Loop; tbh i don't fully understand this yet, and i am not convinced that there's not an in-between.

what is a spoon?: well, i don't think that a spoon is actually discrete. i mean, we have sporks! but what is this spoon, the one i am holding? it's one that's had a feedback loop through its process of being manufactured. it used to be raw metal, but now it's been molded or punched and bent into this shape by machines that say "no, don't try to be something else, you will be this spoon." and it's sold by people who say "this is a spoon." and in fact, if it ever doesn't take its spoon form, it will be discarded, so i wouldn't have seen it. so in this very spoon lies a feedback loop, but also in the symbol the spoon i use to eat this soup lies another feedback loop.

of course, this theory is itself introducing discreteness; i can't think of what metaloop lies in it but maybe i will later :)

ever looked at a pixelated image and couldn't recognize it, then stepped back and suddenly could see it? it seems like vlms follow this same pattern.

ai models can't back up to look at an image, but you can inset the object in the image so it is "smaller" i guess. i put 100 album covers pixellated and increasingly inset in a 1024x1024 image, then asked a vllm to guess what album it is. here's how the model accuracy degrades on 100 different 95x95 covers (4 shown).

"what album cover is this?":

i don't know why they fail so early; it's strange. fwiw, control case with full 1024x1024 has 98% accuracy, so i don't think something funny is happening here.

humans seem to fail much later. take this fleetwood mac image. if i open this on my laptop screen full page, my own ability to recognize the image is inconsistent 16x16px row. but the model gets confused at 95x95!

imagine if, in 2008 or whatever, adam d'angelo at quora or spez at reddit worried about being one of a ton of Q&A platforms on the internet. stack exchange, digg, researchgate, facebook groups, and all these random discussion forums. and being worried that "oh god, this space is so crowded!"

it's the corollary of the pets.com fiasco; prove the market before you think there's a market, but also if there's a giant market ahead, stop worrying about competition. i am, of course, talking about ai/agents.

your product will find a niche! ai is so going to be so huge. turns out that reddit is its own niche for a certain vibe. same is true for social medias and dating apps.

was drinking my tea and noticed how cool the dual refraction was, here’s codex + blender going thru all the states of IOR for both the glass and the tea itself.

ai has this problem where people don't know that they want ai, but ai companies go around selling ai. i've seen this by traveling around the US for 5 months trying to convince people to use ai; they just don't want it!

i think we see the effects of this problem in the fact that people don't talk about ai agents like they should; in SV we all know how absolutely bananas this technology is, but others don't talk about it.

ai companies are currently doing old, saas-style distribution. but i suspect they will find, as i have, that the best way to convince someone to use ai is to show them the work that it could be doing for them. that might mean company-specific distribution. it might mean ads created per user.

well, there's three realities ahead for any given company. either:

1. ai companies figure out how to do marketing to convince your smallco to use ai. and all these old businesses continue to exist, but now they use ai
2. PE rollups eat you because you didn't use ai; you can't figure it out so you get destroyed by a business who's using AI to get it done
3. you're in a business where ai efficiency doesn't eat your margin, and you're fine, because you're a jewelry manufacturer with only big orders and ai will help you become 2% more efficient

i spent about three weeks working in the amazon fulfillment center in Belmont, California. i worked in the grocery division, picking and coordinating orders for Amazon Fresh orders. it was a warehouse that was sorta set up like a grocery store.

1. there is still a long way to go to robotics automation. this is obvious to most people but +1!
2. you can see that amazon has "shipped" the org chart by what's efficient and what's not. picking the order is near maximal efficient. you can smell the amount of engineering hours in it. but stuff that changes from fulfillment center to fulfillment center (walking paths, people coordination, etc) is very slow. stuff that's digital is near perfect though.
3. fun one: amazon puts different flavors of yogurt (e.g.) on completely different parts of the warehouse. the strawberry yogurt might be in A12 whereas the blueberry yogurt is in N45. why? i think to prevent mispicks. impossible to mispick when they're spread all the way apart. and when you pick a set of orders, you walk all the aisles anyways.

general principle emerging: humans are offended by something that is "directly" read by humans, that is not written with humans. this explains the following:

1. why people get mad receiving AI emails
2. why everyone's okay with running code run with AI, but not okay with a website that "looks" AI generated
3. why AI images are repulsive for people
4. why managing relationships with ChatGPT is revolting

i've noticed something funny: ai images are beautiful images by 2022 standards, but most people react very poorly to them in restaurants, calendar invites, or hinge profiles. isn't that funny?

it's because it's AI that people (including me!) hate the image. not because the image itself is ugly.

this is bad news for AI image companies, because it means that their role will be tricking people that the image isn't AI. sure, it's possible, but if that's really a signal people use to tell if they love an image, we might see some new things emerge.