deriv/LOG.md

Development Log

Initial Project Idea and Planning

I decided to make a basic symbolic differentiation engine. The essence of this is that you can provide mathematical expressions as input, and the program can compute the derivative of them. For the sake of simplicity, it will be command-line based, so its input and output will both be text. An example interation might look something like:

> 2x^2 + 3x
4x + 3
> x^3 - x^2
3x^2 - 2x

Architecture

The program will have three major components: the reader, which will take user input and parse it into a convenient internal representation (generally an AST of some kind); the solver, which will transform the input expression into an output expression; and the printer which will format output expressions back to human-readable text. This is essentially a standard architecture for a simple interpreter.

The reader will be composed of two passes: a simple state machine lexer to tokenise user input and a parser to build an AST from the tokens. For simplicity here, I will only be supporting basic latin characters in UTF-8 (i.e. ASCII-compatible text).

Since I am only implementing differentiation, the solver can be relatively simple, consisting of a pattern-matching loop with an accompanying ruleset.

The printer is the simplest of the three components and will consist only of a procedure to walk an AST and emit a serialisation of it equivalent to the input language.

In addition to the major components, a harness will be needed to tie them together and handle errors. This will be in the form of a simple REPL (Read Eval Print Loop).

Implementation and Testing Strategy

I'll begin by implementing the reader and printer, as this will allow me to get to a working end-to-end demo as quickly as possible by initially skipping the solver and simply printing the input expressions unmodified. This will then enable easy and fast iteration when implementing the solver logic.

I will be writing a suite of unit tests to accompany every module of the system, following a TDD (Test-Driven Development) methodology. This system is quite simple and well-defined, so the advantage of TDD (as opposed to writing tests after-the-fact) is probably relatively minimal here, especially since it will not be being modified or maintained after its initial development and will not be being worked on by multiple people, but the practise will be beneficial regardless in tightening the feedback loop of the development process and ensuring that the code is structured to be amenable to testing. If I have time I'll also implement some automated integration tests to verify end-to-end functionality.

I'll be using Rust as the implementation language, as I am fairly familiar with it and it is well-suited to such tasks. In addition, its strong type system and borrow checker are useful in catching many classes of bugs at compile time and it has unit testing support provided in the base language, which is convenient for doing TDD.

LLM / AI Usage

I will not be using any LLM coding assistants for this project.

Reader

Lexer

I first implemented the lexer with a very simple finite state machine consisting of two states, Init and Integer. This was straightforward as the input language is very simple.

Parser

Since the input grammar consists of terms, parentheses and infix operators with varying precedence, Pratt parsing is the logical choice for the implementation of a parser.

I find the usual way of handling associativity in precedence climbing parsers (wherein a small amount is added to the binding power on one side) somewhat inelegant, so I decided to implement a simple algebra for binding powers that handles associativity using custom comparison semantics.

Implementing the parser itself was straightforward as I have implemented Pratt parsing many times before (though not in Rust specifically).

To tie together the lexer and parser, and to provide a convenient means of decoupling other components from them for testing purposes, I then implemented the Rust standard library's FromStr trait, which is for types that can be parsed from a string.

Printer

For expression printing, a simple recursive tree-walking algorithm, dispatching on the various kinds of expression was sufficient. Satisfyingly, I found I was able to use the same binding power algebra I defined for the parser in deciding when to use parentheses, and thus got correct associativity handling effectively for free, while also keeping all associativity-handling logic well-factored in one place.

The only complication was in the handling of products, as here there was some ambiguity as to when to use explicit or implicit multiplication (that is x*y or xy). Products of symbols with other symbols and integers with symbols are more naturally expressed with implicit multiplication, but of course this couldn't be used for products of integers with other integers as this would result in incorrect serialisations (such as "12" for the product of 1 and 2). I ultimately settled on an approach where the right-hand operand is first printed into a string, and an explicit multiplication sign is used whenever that serialisation begins with a numerical character (either a digit or a minus sign). This nicely handled all of the cases I could think of and systematically avoids any erroneous serialisations.

The printing logic could then be packaged up in an implementation of the Rust standard library's Display trait, which is for types that have a textual representation.

REPL

At this point, with the reading and printing implemented, it made sense to create the REPL logic and demo binary so that a full flow through the program could be demonstrated. I first created an Evaluator trait to capture the future behaviour of the evaluator, and then wrote a generic REPL module that's able to work with any expression type implementing the FromStr and Display traits.

I was then able to create a demo program by defining a dummy evaluator that simply returns the input expression unmodified, and running a REPL using that, connected to stdin and stdout. This demonstrated a full round trip through the parsing and printing procedures.

Evaluator

Rather than trying to handle everything in one complex evaluator, I decided to implement a multi-pass evaluator. This essentially means that the expressions run through multiple different passes that apply different transformations to them, thereby keeping the complexity of each pass more managable. This is a common method used when implementing compilers and interpreters.

Normalisation

The first preliminary pass I implemented was a normalisation pass. This transforms things like x - y into x + (-1)y, and x/y into xy^(-1). The main benefit of this is to make it easier for the later differentiation pass as it will only have to handle sums, products and exponents (not differences and quotients). I decided to leave quotients of integers as-is since these will be untouched by the differentiation process anyway, and "1/2" reads far better than "1*2^(-1)".

Constant Folding

The purpose of this pass is to combine constants together as much as possible, such as transforming 1 + 2*3 into 6, or x + 1 + 2 into x + 3. This is mostly a nice-to-have rather than an essential feature, though it will make the output much more readable.

Implementing this pass did get a little hairy, as the binary tree structure I used for the AST made handling cases like x + 1 + 2 (which is represented in the AST as (x + 1) + 2 quite complex. The module implementing this therefore ended up being somewhat messy, especially since Rust's move/borrow semantics made flattening trees of sums and products more complex than it would otherwise be. However, I decided to move on rather than spend a great deal of time tidying this up since, as previously mentioned, this was more of a nice-to-have feature rather than core functionality.

Evaluator Composition

I then implemented a mechanism to compose evaluators together, and build a pipeline of arbitrarily many of them. This was simple to do since I was implementing the Evaluator trait for each pass, so supporting composition simply meant creating a struct containing two evaluators, and implementing the Evaluator trait for this such that the evaluators were run in sequence on any input expression.

To combine more than two evaluators, I created a pipeline macro that constructs nested compositions of evaluators with Rust's macro_rules system.

Differentiator

The differentiation pass itself was very easy to implement given the simplications made by first having a normalisation pass. I also had decided to only support constant exponents, which simplifies things too. In the end, it ended up being more-or-less a fairly direct translation of the sum rule and product rule into code, along with the power rule and chain rule combined for handling exponents.

Simplification

The output of the differentiation pass was correct but contained a lot of noise such as terms multiplied by zero and other similar things. I therefore implemented a basic simplification pass that applied various identities such as x + 0 = x, 0x = 0 and 1x = 1. Expression simplification is famously a very complex problem, but I kept this pass quite limited in scope and only implemented the most high-payoff features.

Canonicalisation

To make the output a little easier to read, the final evaluation pass I implemented was a limited multiplication reordering canonicalisation pass that ensured that products had the constants first. This is still incomplete as the re-ordering only takes place within binary products (so x*4 becomes 4x, but x*(y*4) would become x*4y not 4xy), and terms in a sum were not re-ordered at all. However, I decided full canonicalisation would be excessively complex for this project and decided to leave it there, as the output was acceptably readable at this point anyway.

Wrapping up

The final pipeline I ended up with was:

1. Normalisation
2. Differentiation
3. Constant folding
4. Simplification
5. Constant folding again
6. Canonicalisation

I then created an evaluator function to wrap this up in the eval module, to allow for a simple interface to the whole library.

Conclusion / Closing thoughts

Creating the differentiation program was a lot of fun, and I'm happy with the result I ended up with. The code is well-tested, and for the most part is fairly maintainable (perhaps with the area needing the most work in that respect being the constant folding pass).

In order to quantify how effective the test suite is, I ran a mutation testing tool (cargo-mutants) and it found that only 3 out of 221 generated mutants were missed by the test suite, giving a mutation score of 98.6% which is very good. The three missed mutants were also limited to the demo program itself (which makes sense since it has no tests) and error message formatting code, so none of them are particularly concerning.

There are quite a lot of areas where further work could be done, such as adding support for functions like sine, cosine and natural logarithms; enhancing the simplification and canonicalisation passes and allowing non-constant exponents. However, I think the most interesting enhancement would be to add equalities to the language and allow definitions to be stored within a session, with explicit differentiation commands to get the deriviative of something, such as:

> y = x^3 - x^2
> dy/dx
3x^2 - 2x

This, along with a few other features, would transform the program into a more general-purpose symbolic mathematics engine, which would be very cool and useful. The current architecture would allow for this quite nicely, as evaluators are able to have mutable state which could be used for storing variable bindings.