News

Please, visit the Patreon page for the details on the campaign. The code is hosted on sourcehut, you can read about it on the Wiki.

I will be posting all the news and any reports here, on this page, as well as on Patreon. You can track any meaningful addition to this website (new articles (if any), news, and reports) via the RSS feed. To subscribe to a low-volume announcement list, send a message to:

~project-mage/news+subscribe@lists.sr.ht

I will be posting progress reports every month, around the 23rd.

Progress Report #3, 24 April 2023

Progress Meter

—

A quite solid month, the major changes can be summarized as:

First of all, the Sky-Blue solver was using a couple of poorly-implemented data structures, among a few other things. All these had to be attended to. As a result, by some basic benchmarks, the speed has improved 2-2.5x overall.

I have also decided to disallow conflicting strengths for constraints that write to the same variable. Allowing these would mean relying on the order of addition, and therefore execution, and that's something that could be hard to reason about. It seems it best to just require the code authors to define a clear ordering of things before any bugs creep in. And constraint strengths as such are now simply integers, not a limited set of ordered keywords.

Second, it appeared to me that the way I did inheritance, while being bug-free otherwise, would be hard to make work well with constrained slots. In particular, the order of execution and recomputation would be a bit tricky to control, i.e. when you change some slot in the parent schema and the value has to propagate to the children.

Now, it seems obvious in retrospect, but modeling inheritance via the constraints themselves is a much simpler and more consistent solution. The way it works is basically such that a slot in the child is constrained to be equal to that slot in the parent. Assigning the appropriate strengths to these connections preserves the hierarchy in case of multiple inheritance, routinely takes care of the ordering and the value updates.

The new model pretty much has all the necessary features already, it's fully dynamic, potentially extensible, and the user can both inspect and remove/filter any particular connections (and in the future: advice them too, since they are just regular constraints).

It took a bit of preparation, but the resulting code for all of inheritance is now contained, rather simple and is only ~130LOC. And while that's pretty good, I have a feeling this just might be reduced in the future if it makes use of parameterized pointer variables (e.g. constraints that map to multiple slots, a yet-to-be-implemented feature). In fact, I think the new inheritance solution just might guide that work quite soon.

So at this point it's not so much some frigid concept of inheritance in its usual sense as much as simply a case of hierarchical data consistency management. And, it seems to me, that's exactly what it ought to be. Quite importantly, as a result, there's now no need for any coupling with the internals of the constraint-integration code, as was necessary with the previous approach.

—

The next steps are to restore/revise the relevant tests, but that won't take long. Otherwise, the foundation now seems quite ripe for the next immediate targets:

The former of these should yield a quite powerful and indispensable tool for dealing with multitudes of slots.

The latter objective should result in a consistent interface for deriving/cloning/modifying the constraints.

Progress Report #2, 23 March 2023

Progress Meter

—

TLDR This month was kind of slow, but I have reintegrated the new KR code with the multi-garnet layer, so now the constraints function again (and all the tests pass). I have also been adding some meta-slots, and that's about half-way done.

I have also refactored the multi-garnet layer somewhat (mostly throwing out the compatability layer for the old formulas), but there are a few not-so-reassuring things left, like the usage of property lists for tracking slots, instead of just using hash tables, so I will be looking into that too quite soon.

Some syntactic improvements were made to the constraints macros as well, but I won't go over these just yet: these are still likely to change.

—

In the previous report I mentioned that FSET could be a viable option as a container for schemas: it's functional, so you could safely expose it to the world as just another slot, and the user could use it for, say, traversal, just as a hash map. Access an object as a data structure type of deal. However, some quick tests have shown that FSET is 29 times slower on writes with 14 times the memory consumption compared to the standard hash tables. Worse, it's also 22 times slower on reads. I still think that exposing a container is a good idea, so I am just going to be using standard hash tables instead (as I am doing already, for the most part). The worst that can happen is the user bypasses the constraint mechanism if he writes to the hash table directly. But that's going to be possible to do anyway, so it's not a problem.

Speaking of hash-tables, hash-table-args is a slot that can be supplied by the user at object-creation time in order to create a customized hash-table. For now, only the :test argument is supported, and the standard hash-tables only support eq, eql, equal and equalp for tests. These should probably cover most cases, though (and if not, this could be extended, I think). The use of this is straightforward: now you can use, say, lists for keys, and not just symbols.

In fact, it seems like letting the user use any kind of object as a key is a rather sane thing to do. Previously, I was planning on restricting the use of keywords to KR-only purposes (like :is-a), but maybe this isn't all too reasonable. So, keywords are too going to be allowed as keys to the user, and is-a along with the rest of the kr-supplied slots will be just regular symbols.

Speaking of meta-slots, the breakdown for these is as follows, more or less:

I will be documenting these eventually, of course, but these slots (along with some hidden constraints) will form the basic interface to any given schema.

To get an idea of how this is going to work, let's document some slot: 

(Ξ s)
(→ s 'some-slot 42)
(→ s 'docs (Ξ))
(→ s 'docs 'some-slot "This slot contains the answer ...")

The third line assigns a schema where all the docstrings will be held. Then the docstring for a slot is assigned. This is roughly equivalent to doing this at object-creation time: 

(Ξ s
   (some-slot 42)
   (docs (Ξ (some-slot "This slot contains the answer ...")))

One tricky thing here is to keep the inheritence relations intact. So, if s were to inherit from q, we would want the docs in s to inherit from docs in q. This will be accomplished quite simply by setting up a constraint on the is-a slot of the docs to be computed from all the schemas in the is-a of s. And the same thing can be done for any nested schemas. However, it appears that parameterized pointer variables will be best suited for this, and so this will have to wait just a bit.

Also, with create-on-write, it won't be necessary to create the docs schema manually, as it will simply be created when written to.

name and prefix are for printing/debugging purposes.

container, slot-via, slot-parent will expose the container and some useful slot properties.

advice is advice for functions called via ↑.

local-only will list slots that don't have to be inherited.

types will describe slot types for type checking and, later on, for supplying type info to the compiler.

equality specifies a function for a slot to allow early exit if a new value is equal to the old on assignment

drop, take, filter: these I am still thinking about. Would allow dropping/taking certain slots for inheritence, but on the other hand they kind of break the meaning of is-a relationships. But so does local-only, to a degree, if it shadows an inherited value. This could be taken care of, but maybe there's no big need for these to begin with. We will see.

last-added, last-removed, last-updated, last-action refer to slots and allow meta usage if you constraint them as inputs. For instance, with these, you could keep track of all slots in a schema that have a certain type or that have a certain property. This will be very useful for API-building.

Further Thoughts

—

I was probably a bit overly enthusiastic for my last estimate about starting to work on graphics in May. This KR/Constraint business will easily spill into June. Fine by me.

Progress Report #1, 23 February 2023

Hi, everyone!

So, the first month of development is already behind, and it started out well!

— Progress Meter ---

— What has been done ---

First and foremost, I have set up a Wiki which acts both as interim documentation and as a learning resource.

Next, I have refactored the core Knowledge-Representation code. It wasn't pretty and had many questionable decisions. So many, in fact, that after throwing most of them away, I was left with a meager ~800 lines of core functionality. This also turned out to be not only buggy, but a bit weird in that it was using both a push and a pull model for the sake of memory preservation. This is not necessary now, and I doubt it was necessary back when it was written. I have rewritten all this, yielding only ~400 lines of code due to a slightly simplified model and a more straightforward approach. All the tests run fine, so that's good.

The API got a much-needed simplification as well. Now it boils down to just these few operations:

Ξ, ←, ↓, ↑, →, ↓↓

The breakdown is as follows:

There's a bit more to it than that for getting or setting a value, though. For instance, one might want to just get the local value (as opposed to a possibly inherited one); or write without automatically updating the inherited values (for performance). This requires some further syntax. For getting a local value, this is done like this:

(← name (local 'first))

So, local here is just a macro which spits out an expanded slot declaration:

(← name (:key 'first :local t))

These declarations may get combined and they seem to provide a pretty straightforward interface without any need to create an abundance of specialized get/set functions.

Besides these simple functions and macros, there will likely be just some mapping constructs exported for walking the inheritence hierarchy. So, overall, it's looking pretty minimal so far (of course, this is just for the KR itself, there's also the constraints-related stuff (which is also pretty minimal)).

As for the use of non-ASCII symbols: typing these isn't really a problem, and they really make the code look better and much more compact.

— The immediate plans and further thoughts ---

—

I estimate it will take about a month to sort through the rest of the KR stuff, to add the necessary features. Then, probably, another month or so for the constraint management improvements. (Just eyeballing this now, and assuming the constraint engine won't give me trouble.) So, it wouldn't be bad to start working on some actual graphics some time in May, if everything goes well.

If anyone has comments or questions, just start a thread on the dicussions or dev list here or contact me directly.

Announcement #1, 23 January 2023

Hi, everyone! Good news!

I haven't reached the funding goals, but that's OK! Already, with the support of my patrons (thank you!), I think I will be able to work on Mage something close to full-time for all of 2023 (at least).

The plan for this year is pretty simple: bring Fern up to speed. It's going to be a GUI toolkit. In a few ways, it's rather simple, but in others, it's quite powerful. The foundation has been established, in most part thanks to the Garnet and the Multi-Garnet projects from many years back.

That foundation does need to see some improvements. I have described the reasoning for them in the Fern section on the website. The gist of it comes down to:

So, this is the rough outline of the work to be done.

Once Fern is ready, it should be possible to start working on the editor specification and some actual editors: Rune and Kraken. But that's unlikely to happen until next year.

So, I will probably spend the next few days setting up my environment, and then populating the Wiki of the project. Then I will switch to programming.

Also, some people have shown interest in the code. Please, if you are learning about Fern or even want to be implementing something, do communicate with me! It's important to be on the same page. In the next few days, I will be populating the Wiki with some development notes for anyone interested in learning about the code, as well as talk about the future course of development. Just for now, the best place to learn about the code is still the Code page.

The development will be taking place on Sourcehut, here. I found Sourcehut to be geared for the efficient workflow over mail, and yet it seems to have a decent web interface.

I hope to make the best of it. So, again, thank you all!

P.S. I will write news and updates on the development about once a month. I will publish them here and on Patreon. I have also set up a mailing list where these messages will be duplicated. To subscribe, just send a message to

~project-mage/news+subscribe@lists.sr.ht

There are other mailing lists as well, for develepment and general discussions. And, in general, if anyone has any concerns or suggestions, feel free to contact me.

...proudly created, delivered and presented to you by: some-mthfka. Mthfka!