patrick.app

kirby's language experiments are ~mildly reproducible in language models.

what are kirby's language experiments?

original paper. it's testing language emergence in humans through iteration across people. probably claude's explanation is more helpful.

Filtered:

despite the bottleneck, the models learned how to create a language from scratch. really cool! not that surprising if you're in ML, but really cool if you believe LLMs can emulate humans.

it reproduced the same results when unfiltered, too; the models learned to cheat and use one word to represent everything.

yesterday we saw that there are attractors in language model loops. in the color-wheel loop, #0000FF is a stable attractor. i ran it overnight on much longer sequences, and all the models, including luna, show convergence after enough passes. really fascinating!

colors are cool, but i think that it's more interesting to imagine these attractors for ideas. i think that culture and memes and species and sexes are attractors. but how to study these inside llms... i am still not sure.

what if language models are the Drosophila of culture?

dawkins, in the selfish gene, coined the meme as a cultural gene that passes from mind to mind. what if language models can carry memes too? this is a small (inconclusive) study in that.

let's start with a color like #1C1BFF . then give it to a model and ask "what color is this? give me a hex." then repeat until it repeats the same color 3x (stable attractor) or 100 turns go by.

it's funny, if you do this for blue, it collapses down onto a #0000FF "pure blue" color.

here's what happens if you do it to every color on the color wheel:

look how cool it is! little pockets, species of colors emerge. everything near pure blue becomes pure blue. and look how low-dimensional yellow is! do you see how, by analogy, each one of these color regions is a species, or a language, or a country with borders, or a type of cheese? isn't it cool how discrete lines emerge? how there is clearly a space in qwen 3.5's J-space for this sharp blue. qwen collapsed 1,009 colors down to just 251!

broadly, language models can be replicators of memes. this is fascinating. here are luna and 4o against a standard color wheel with the same instructions:

luna is a smarter model, but if you look closely you can see some collapse in the green region. and what weird results for 4o!

and here's a video of qwen collapsing:

i am thinking a lot about this topic and especially how to reproduce this in language. replicating colors works to demonstrate this because it is multimodal, which prevents copying, but is also easy to visualize. this stuff is so fascinating to me :)

i've been toying with this theory that all discrete things have feedback loops inside of them. some examples:

microphone and speaker: the simplest one; if the feedback is above the threshold, it becomes as loud as the speaker supports. if the gain isn't above a threshold, it tends to zero. it's either one or zero.

languages: languages are memes with human culture as vehicles. languages are useful and so they spread, and they loop because they are evolving memes that evolve with themselves. in other words, languages evolve when conversing with other people of your own language. it's like how species (see below) become discrete by not breeding with other species. of course, languages aren't fully discrete; there exist spanglish and singlish and i'm sure many other language hybrids i haven't heard of. but they look a lot like species.

species, sexes: covered by Dawkins in Selfish Gene. genes prevent themselves from interbreeding with other species such that they propagate their own existence. sexes are explained by a feedback loop in the size of the sex cell; sperm is rewarded by getting smaller and eggs are rewarded by getting bigger.

country borders: generally the state is discrete. there is the United States and Mexico and not sorta-Mexico-sorta-US. why? usually because we have a state government and limited land, and there is a "grabbing" of the land that happens, and then you make it your land; you farm on it. and you get support from the government and they make sure that nobody is going to Steal Your Chickens or take your land and so then there's an ownership, which is a loop, by you (this is My Farm Land not yours) but also the government that provided you with basic protection.

life and death: death is inherently a dissolution of boundaries. when something dies, its boundary no longer exists; it is one with its environment. a dead body is now a part of nature, and our language treats it as such. much like state borders above. so in this sense, life is discrete because of its ability to maintain a border. the metaphors in our language reflect this as well [credit to my friend Sarv]

but you can come up with more philosophical examples too:

consciousness: things either seem to be conscious or not; we don't see much in between. explained by Hofstadter in GEB/I am a Strange Loop; tbh i don't fully understand this yet, and i am not convinced that there's not an in-between.

what is a spoon?: well, i don't think that a spoon is actually discrete. i mean, we have sporks! but what is this spoon, the one i am holding? it's one that's had a feedback loop through its process of being manufactured. it used to be raw metal, but now it's been molded or punched and bent into this shape by machines that say "no, don't try to be something else, you will be this spoon." and it's sold by people who say "this is a spoon." and in fact, if it ever doesn't take its spoon form, it will be discarded, so i wouldn't have seen it. so in this very spoon lies a feedback loop, but also in the symbol the spoon i use to eat this soup lies another feedback loop.

of course, this theory is itself introducing discreteness; i can't think of what metaloop lies in it but maybe i will later :)

ever looked at a pixelated image and couldn't recognize it, then stepped back and suddenly could see it? it seems like vlms follow this same pattern.

ai models can't back up to look at an image, but you can inset the object in the image so it is "smaller" i guess. i put 100 album covers pixellated and increasingly inset in a 1024x1024 image, then asked a vllm to guess what album it is. here's how the model accuracy degrades on 100 different 95x95 covers (4 shown).

"what album cover is this?":

i don't know why they fail so early; it's strange. fwiw, control case with full 1024x1024 has 98% accuracy, so i don't think something funny is happening here.

humans seem to fail much later. take this fleetwood mac image. if i open this on my laptop screen full page, my own ability to recognize the image is inconsistent 16x16px row. but the model gets confused at 95x95!

imagine if, in 2008 or whatever, adam d'angelo at quora or spez at reddit worried about being one of a ton of Q&A platforms on the internet. stack exchange, digg, researchgate, facebook groups, and all these random discussion forums. and being worried that "oh god, this space is so crowded!"

it's the corollary of the pets.com fiasco; prove the market before you think there's a market, but also if there's a giant market ahead, stop worrying about competition. i am, of course, talking about ai/agents.

your product will find a niche! ai is so going to be so huge. turns out that reddit is its own niche for a certain vibe. same is true for social medias and dating apps.

was drinking my tea and noticed how cool the dual refraction was, here’s codex + blender going thru all the states of IOR for both the glass and the tea itself.

ai has this problem where people don't know that they want ai, but ai companies go around selling ai. i've seen this by traveling around the US for 5 months trying to convince people to use ai; they just don't want it!

i think we see the effects of this problem in the fact that people don't talk about ai agents like they should; in SV we all know how absolutely bananas this technology is, but others don't talk about it.

ai companies are currently doing old, saas-style distribution. but i suspect they will find, as i have, that the best way to convince someone to use ai is to show them the work that it could be doing for them. that might mean company-specific distribution. it might mean ads created per user.

well, there's three realities ahead for any given company. either:

1. ai companies figure out how to do marketing to convince your smallco to use ai. and all these old businesses continue to exist, but now they use ai
2. PE rollups eat you because you didn't use ai; you can't figure it out so you get destroyed by a business who's using AI to get it done
3. you're in a business where ai efficiency doesn't eat your margin, and you're fine, because you're a jewelry manufacturer with only big orders and ai will help you become 2% more efficient

i spent about three weeks working in the amazon fulfillment center in Belmont, California. i worked in the grocery division, picking and coordinating orders for Amazon Fresh orders. it was a warehouse that was sorta set up like a grocery store.

1. there is still a long way to go to robotics automation. this is obvious to most people but +1!
2. you can see that amazon has "shipped" the org chart by what's efficient and what's not. picking the order is near maximal efficient. you can smell the amount of engineering hours in it. but stuff that changes from fulfillment center to fulfillment center (walking paths, people coordination, etc) is very slow. stuff that's digital is near perfect though.
3. fun one: amazon puts different flavors of yogurt (e.g.) on completely different parts of the warehouse. the strawberry yogurt might be in A12 whereas the blueberry yogurt is in N45. why? i think to prevent mispicks. impossible to mispick when they're spread all the way apart. and when you pick a set of orders, you walk all the aisles anyways.

general principle emerging: humans are offended by something that is "directly" read by humans, that is not written with humans. this explains the following:

1. why people get mad receiving AI emails
2. why everyone's okay with running code run with AI, but not okay with a website that "looks" AI generated
3. why AI images are repulsive for people
4. why managing relationships with ChatGPT is revolting

i've noticed something funny: ai images are beautiful images by 2022 standards, but most people react very poorly to them in restaurants, calendar invites, or hinge profiles. isn't that funny?

it's because it's AI that people (including me!) hate the image. not because the image itself is ugly.

this is bad news for AI image companies, because it means that their role will be tricking people that the image isn't AI. sure, it's possible, but if that's really a signal people use to tell if they love an image, we might see some new things emerge.