The v4 variant will have multiple upcoming finetune targets. The v4 will inherit the v3 weights and be refitted to be predominantly softmax attention curated with a permanent softmax attention behavior at the core.

This has proven to be without a doubt the best methodology for delta attention structures as of today, tomorrow may be different but so far the softmax inclusion is one of the best courses of action.

V4 will be considerably more thought out, carefully curated into a behavior that makes the model have an internal perspective of the world around herself.

One of the experimental tests meant for the better recall version of her will include color checking.

I have a series of tests meant to test her pragmatism, understanding of normality, relational behavior to decision, and more. The more of these I develop, the better I'll be able to curate the model towards a behavior that is considered acceptable.

Alongside I'll be running her through arc tests as well for baseline behavioral utility and so on. There's quite a bit of information you can gain from pragmatic studies.

To create a reasonably curated and useful v4, the structure must be aligned to be a useful and understandable system. The bulk pretraining will be based on a curated system of articulated behavior, specifically designed to enhance the model's capacity to understand and perceive science, mathematics, literature, multiple languages, history, and more. The structure of this information curated appropriately to the various general tasks people might want to experiment with, and specifically for art and diffusion in my particular interests.

This will include a full background database understanding of art, photography, 3d spatial understanding, 2d representative imaging, and more. Essentially a world view around building information and using that information meaningfully for an art-driven diffusion conditioning. The entire process will revolve around her perspective of the world around a curated world space to perceive and rationalize a pragmatic objective reality.

I had Claude run many methods of positional encoding overnight. Some are more effective than others.

There are multiple bigger address splats with rotary that can support the recon behavior shallow and when at depth the model simply falls apart. The chosen address schema simply wasn't strong enough. When paired with rotary the model learns at depth either way. So, the address wasn't big enough. The v4 target is making itself.

Looks like we're getting close now. The depth experiments are aligning to the expected dynamic framing.

You can kind of think of this structure here as a big fat memory blob that turned into soup, being given database lookup now thanks to the last three Beatrix models trained. We have the correct and necessary information needed to align her.

The modularity allows the twins to coexist as well. Multiple experiments were performed with the Beatrix 2s twins cooperating for recall accuracy tests. This concept had multiple unaddressed problems but it does work. The final 2s softmax twin isn't ready quite yet, she'll be ready this evening. I got a bit of a head-start on it from an earlier checkpoint and the results are aligning.

The only way to correctly align the v3 is for me to spend a week training and babysitting the softmax v3 model. For now I'm working with the 2s models until I'm satisfied the odds are in my favor. After that, we can start pushing much further for depth accuracy.

A successful full objective memory container of this scope ought to represent a context window larger than the traditional limitations. The rotary behavior mixed with addressing is showing promise, so a full rotary addressed softmax splat prototype is on the queue.

Larger addressing gives much more consistent recall with the byte structures, however the splat attention will need to be curated and correctly aligned to the necessary scale and scope at depth. Correctly aligning the addressing scale to the necessary depth will provide the most useful lookup structure in conjunction with the shared byte knowledge.

We have yet to decide a K structure size. Too large and the system takes too long to train with the structure, too few and the model simply does not rationalize the utilizable relations at depth. There are multiple models with examples but there are many potentials here that aren't present in most models.

The structures of the next variant will include a micro-sized version, entirely dedicated to attaching for experimentation. Nano sized, smaller than even a single arm operating on the 3s model. The smallest version will not be intelligent, but she will definitely be modular like the others. This nano version will likely be as intelligent as the best v3 model to date, containing a 4096 context window within the microscopic forms.

This model will train very very quickly. Lets see how much context length we can pack into this little piece of equipment, more akin to a cybernetic implant meant to augment and route than create decisions.

Nano will likely be a hybrid of a higher k splat combined with softmax as a dual brain, as they seem to function highly efficiently in pairs. Softmax learns quickly but often has problems for simpler and slower alternations, while splat takes longer to converge at length but has many benefits to it's scaling and sizes that softmax simply does not capture. I'm currently studying the different effects and alternating outcomes as based on the structures to determine the best methodology of coordinative cooperation.

I, honestly didn't expect the two to be cooperative without A LITTLE procrustes. They... work really well together, the twins do even without procrustes whitening rotation. WITH IT they teach the little nano versions rapidly.

Softmax Twin ETA: 18:00 GMT-8

Nano prelims are showing some genuine promise.

Rotary + Hub is actually showing some real recon weakness. That's... unexpected considering the prelims. That's okay though, the outcome is pushing towards two rather than a single anyway.

Well I think we have our answer.

I'm not going to measure any verdicts yet. Softmax with learned positions is strong, but multiple forms of rotary are known to extend models to huge context windows.

rotary hubs + rotary softmax alternating is a very powerful craft. The outcome within noise matters, but I will test the paired rotary version as our hybrid at depth. This might just form the necessary continuum behavior I've been looking for, preserving and updating records rather than suffering at depth for a while until the model catches up.

There are definite causal relational behaviors related to framing data within a context window. I'm starting to see the causal outcomes of the aleph resizing system and the behavior expected from this window. I believe we need a couple more scaling mechanisms to cultivate a full geometric skip-continuum behavior in the future, that's another experiment though. I've experimented in the past, chopping up the window does weird things, however this entire model is modular, so when the window behaves correctly we ought to be able to slice off pieces of the window itself for necessary safety points, guarded sections, and so on.

Not enough data to be certain yet.

Context window is starting to behave though, which means the model will function. The rotary system is most definitely functioning for both the hub AND the softmax. This will saturate the 4096 context window with room to spare, and next 8192 may be possible on the nano. Oddly enough, I don't know the limit just yet. I'm thinking there is inherent hardware limits to this, rather than conceptualization limits at this point.

I'm thinking the naming should be; nano, micro, mini, small for now. The small versions being what I trained already and were the prelude to the required answers.

I will not make any promises about beginning v4 yet, as we have to answer many questions before we can reach a verdict of continuity. Then, and only then, we cook.

---

Historically, each book has been stable up to 2048 positions. The last train shows that a single book cannot house the entirety of 4096, and trying to do this causes the system collapse.

However, two books... is another story. This aligns with our structure and fits comfortably within the system.

With two small-sized books, scaled on tiny shallow tests from before, we have a fairly inaccurate recall shown by the 2s and worse by the v3.

The v1 model had k128 and this was was sliced for optimization to k64 for v2 and v3 due to shallow tests, data expressing that earlier train steps were not affected drastically BY this book's k. However, I never set canary checks for recon shifts as they historically showed durability, which showed they quite literally fell apart over time. I looked for many useful traits and forgot one of the most critical, depth recon.

Now, k256 on the other hand, seems to be doing quite well. Same dimensional shape, different format.

I'll be testing k128, k512, and k1024 as well.

Each book is structurally compliant with signed predicted structures fitted to 2048 slots, larger books were often never used in testing. Classifiers almost never cared about numbers beyond 512 or so, depending on the classifier's shape and size. However, that was an entirely different process. THESE books are being fully saturated with the billions of bytes being fed. They can handle the differences.

Tests show that k64 is nowhere NEAR good enough, like mounting a single home printer in an office for 500. K128 will likely show a similar curve between k64 and k256, and k512 will likely be the strongest and fastest to converge, so we'll likely be using k512 for the nanos.

More than likely we can handle 4-8 books of k1024 and create a collective delegation for 8192 to 16384. Each book specifically tuned to datasets and their learning mutations slowed or frozen. We'll see how k512 handles first though.

That's why this system exists.

These tests are all being done directly on fineweb using the upcoming nano prototypes. There will be hundreds as they surge train.