I’ve been curious about AI for a while. The pattern lesson always pops up in my head that can we really not teach it something without human knowledge? Plus then there is the AI-insist problem that if we train it on data that was derived from AI, it just causes model collapse.
Then I thought about what if instead of giving it our own knowledge we allow it to explore things on its own, so that it can learn? Like if we gave it enough learning about the physicality of the world, would it be able to figure out the laws of motion and stuff like that? According to what I could find, it learned implicit knowledge that “hey, this ball might fall here or there,” but not something universal like laws of motion.
So first-level is implicit learning, and then it turns into generic or universal principles when one thinks about it? How do we take them from step 1 to step 2 without the use of LLMs? How do we make it reason using numbers or symbols that were not learned using existing data? Would it even need to raise, and use numbers or symbols?
I think I am asking for something too good. Even a seed, if it’s turning into a huge tree. It is kind of predetermined by the DNA codes. Even if not that, AI will still be constrained by whatever is around it. We don’t have infinite compute per given moment, right? So even if we somehow made an AI which has no human bias, the path it takes to progress because of the limited computation. Path of knowledge. Path of theory. It will not be like uniformly spread out. It will be going in some line and then stopping then coming back then going in line if we are trying to make it uniform. So at a given moment, there will be longer lines at one place than another and those lack of lines will result in bad learning and it might have to learn to also self-correct. Thus the DNA of that learning needs to have rules about:
- Uniformity
- Relearning
- Looking at data and making conclusions
Even if not external data, its own data of its own body of knowledge. So the DNA needs to have a lot of information and that puts in human bias. I guess we will never have something autonomous that learns everything. We will have to be the creator or someone will have to be the creator. According to claude:
The dream of "pure, unbiased learning" is incoherent. Even the simplest learning system needs:
An architecture (which computations are possible?)
A reward signal (what matters?)
Resource allocation (explore here or there?)
Inductive biases (which patterns to look for first?)Okay, so: architecture, reward signal, resource allocation, and inductive biases. Can there be any learning without them? Would it ever be epistemologically possible? Would we have to move from knowledge to intuition? Well, all we know about knowledge as humans is memory. In hardware devices, I think we have the same problem. After all, you can think of softwares as a bunch of API-like things. A higher-level language translates your coding to a lower-level one. It turns it into whatever it understands, makes the action into even lower-level using its API equivalent, then it has its own process, then it makes its own action again. So the chain digresses down till we get to the memory part. Basically, we have a long code reaching down into the hardware’s memory, changing the memory, and that leads to some action. So, basically, our modern programming landscape is centered around this approach of changing memory to get things done in physical devices. That’s very animal-like. Okay, back to humans. We have something as knowledge when it is embedded in our memory. Where does that memory come from? It comes from observation. Observation that either came from first-hand experience or some narration to us. And it is often compatible with whatever we ultimately know. The existence of X is supported by Y. Y is supported by A. A is supported by B. There’s a chain going there. Nothing seems to just emerge, but yet we have intuition. I believe intuition either comes as a synthesis of whatever information we already have, or whatever was passed down by the genes, people have generational traumas, information that will be passed by the DNA. So, it is either coming from the observations or from the DNA, but I am sure that is not exhaustive. There might be something else.
Again from claude
The epistemological problem:
Knowledge requires justification. "I know X" means:
X is true
I believe X
I have justification for believing X
But justification traces back through a chain:
"I know A because B"
"I know B because C"
"I know C because..."
This leads to three classical options:
Infinite regress (justifications forever—impossible)
Circular reasoning (A justifies B justifies A—unsatisfying)
Foundationalism (some beliefs need no justification—they're basic/axiomatic)I’ve moved towards intuition because there are things we just inherently know: I am. I think. I’m hungry. Things we know prior to experience and that structure how we interpret experience.
We have self-awareness. Is that self-awareness just emerging right away when we have a critical mass of intelligence, or is it something we inherit? There are animal numbers than us that do have consciousness, but computers currently don’t, even though there are billion times better than us. Currently, I am believing that it is a thing we know prior to the experience. And I think the smartness we want in AI is going to come from what we put in its DNA and letting it explore more.
There are constraints embedded inside the actor. And there are fundamental properties of the environment it is in. Interaction of those two causes some knowledge to naturally emerge. We live in this current universe, that’s why the numbers, the way we have them, are always bound to emerge. Create a simulation where both things are identical, and I think they will also develop similar epistemological foundations. This takes us back to the bitter lesson: if we want to create AI, that is smart. I think it should be as good as humans or better than us when it comes to interacting with the environment. A set of constraints that guide it towards being as smart as us, or better than us. It’s like when humans are overfitted or extremely fitted to the current environment we have given us psychedelics, and I don’t think we do very well when it comes to synthesis of information in that domain. Maybe we are not really so smart, but over time we adapted a seed that gives us intelligence that works really well in this environment. So when we think of AI, do we constrain it to the world we have or do we think of it as something that is universal to all universes? A thing that can learn anything, and what would its seed even look like? And do we even have the ability to let all that computation happen? No, So the seed for a true universal learner doesn’t exist in finite form nor do we have the energy or computation to let it learn.
We can look at nature running its own learners. Even then, it is very energy-efficient. It tests out really narrow hypotheses surrounding its environment. A bird might evolve a longer beak, not gigantic biceps all of a sudden. It is only responsive for the environment it is in. So that does means plurality in learning, going by hypothesis in response to the environment. Evolution does it by mutation. Those poor things are accidents. We have tried till now with AI has been very human-guided. Nor do we what AI to learn things on its own. It can go haywire.
We can’t avoid all bias, but we can ensure our biases match the fundamental constraints that reality itself imposes. Our reality can be characterized by Being process-led, Being physical, Being one of consequences. For such constraints, we do need memory. We do need data about the physicality of the world. We need temporal processing so that we can understand, process, and consequences. This means intelligence isn’t mysterious or arbitrary. It’s the natural result of optimization under universal constraints. Any learning system, given enough interaction with this reality, will converge on similar foundational knowledge.
Essentially, if we keep training on our own knowledge, it will be dumb. If we give it free hand and let it go by the evolution route, then we can’t be sure of our own safety.
From claude
Your system needs:
Physical embodiment - real sensors, real actuators
Autonomy - self-directed exploration, not supervised tasks
Time - lots of interaction cycles
Minimal inductive bias - curiosity, causality-seeking, simplicity preference (universal biases, not domain-specific ones)The three fundamental constraints of our reality:
Process-led (things unfold through computation/transformation, not instantly)
Physical (embodied in space, subject to physics, material constraints)
Consequential (actions have effects, causality exists, state changes matter)
And from these, three necessities emerge:
Memory - to track state across time (consequence requires remembering past)
Physical data - to understand the material world (embodiment requires sensory interaction)
Temporal processing - to connect actions to their delayed effects (process requires sequence)Why evolution's approach is effective:
Locality: Small mutations, incremental changes
Energy efficiency: Only explores nearby hypothesis space
Environmental grounding: Selection pressure from real physics
Massive parallelism: Billions of organisms testing variants simultaneously
Time: Millions of years to converge
A bird doesn't evolve biceps because that's too far in mutation space from its current state. Evolution does local search guided by survival.