deriv/LOG.md

# Development Log

## Initial Project Idea and Planning

I decided to make a basic symbolic differentiation engine.  The
essence of this is that you can provide mathematical expressions as
input, and the program can compute the derivative of them.  For the
sake of simplicity, it will be command-line based, so its input and
output will both be text.  An example interation might look something
like:

```
> 2x^2 + 3x
4x + 3
> x^3 - x^2
3x^2 - 2x
```

### Architecture

The program will have three major components: the reader, which will
take user input and parse it into a convenient internal representation
(generally an AST of some kind); the solver, which will transform the
input expression into an output expression; and the printer which will
format output expressions back to human-readable text.  This is
essentially a standard architecture for a simple interpreter.

![Top-level architecture diagram](./top-level-architecture.png)

The reader will be composed of two passes: a simple state machine
lexer to tokenise user input and a parser to build an AST from the
tokens.  For simplicity here, I will only be supporting basic latin
characters in UTF-8 (i.e. ASCII-compatible text).

Since I am only implementing differentiation, the solver can be
relatively simple, consisting of a pattern-matching loop with an
accompanying ruleset.

The printer is the simplest of the three components and will consist
only of a procedure to walk an AST and emit a serialisation of it
equivalent to the input language.

In addition to the major components, a harness will be needed to tie
them together and handle errors.  This will be in the form of a simple
REPL (Read Eval Print Loop).

### Implementation and Testing Strategy

I'll begin by implementing the reader and printer, as this will allow
me to get to a working end-to-end demo as quickly as possible by
initially skipping the solver and simply printing the input
expressions unmodified.  This will then enable easy and fast iteration
when implementing the solver logic.

I will be writing a suite of unit tests to accompany every module of
the system, following a TDD (Test-Driven Development) methodology.
This system is quite simple and well-defined, so the advantage of TDD
(as opposed to writing tests after-the-fact) is probably relatively
minimal here, especially since it will not be being modified or
maintained after its initial development and will not be being worked
on by multiple people, but the practise will be beneficial regardless
in tightening the feedback loop of the development process and
ensuring that the code is structured to be amenable to testing.  If I
have time I'll also implement some automated integration tests to
verify end-to-end functionality.

I'll be using Rust as the implementation language, as I am fairly
familiar with it and it is well-suited to such tasks.  In addition,
its strong type system and borrow checker are useful in catching many
classes of bugs at compile time and it has unit testing support
provided in the base language, which is convenient for doing TDD.

### LLM / AI Usage

I will not be using any LLM coding assistants for this project.

## Reader

### Lexer

I first implemented the lexer with a very simple finite state machine
consisting of two states, `Init` and `Integer`.  This was
straightforward as the input language is very simple.

### Parser

Since the input grammar consists of terms, parentheses and infix
operators with varying precedence, Pratt parsing is the logical choice
for the implementation of a parser.

I find the usual way of handling associativity in precedence climbing
parsers (wherein a small amount is added to the binding power on one
side) somewhat inelegant, so I decided to implement a simple algebra
for binding powers that handles associativity using custom comparison
semantics.

Implementing the parser itself was straightforward as I have
implemented Pratt parsing many times before (though not in Rust
specifically).

To tie together the lexer and parser, and to provide a convenient
means of decoupling other components from them for testing purposes, I
then implemented the Rust standard library's `FromStr` trait, which is
for types that can be parsed from a string.

## Printer

For expression printing, a simple recursive tree-walking algorithm,
dispatching on the various kinds of expression was sufficient.
Satisfyingly, I found I was able to use the same binding power algebra
I defined for the parser in deciding when to use parentheses, and thus
got correct associativity handling effectively for free, while also
keeping all associativity-handling logic well-factored in one place.

The only complication was in the handling of products, as here there
was some ambiguity as to when to use explicit or implicit
multiplication (that is `x*y` or `xy`).  Products of symbols with
other symbols and integers with symbols are more naturally expressed
with implicit multiplication, but of course this couldn't be used for
products of integers with other integers as this would result in
incorrect serialisations (such as "12" for the product of 1 and 2).  I
ultimately settled on an approach where the right-hand operand is
first printed into a string, and an explicit multiplication sign is
used whenever that serialisation begins with a numerical character
(either a digit or a minus sign).  This nicely handled all of the
cases I could think of and systematically avoids any erroneous
serialisations.

The printing logic could then be packaged up in an implementation of
the Rust standard library's `Display` trait, which is for types that
have a textual representation.

## REPL

At this point, with the reading and printing implemented, it made
sense to create the REPL logic and demo binary so that a full flow
through the program could be demonstrated.  I first created an
`Evaluator` trait to capture the future behaviour of the evaluator,
and then wrote a generic REPL module that's able to work with any
expression type implementing the `FromStr` and `Display` traits.

I was then able to create a demo program by defining a dummy evaluator
that simply returns the input expression unmodified, and running a
REPL using that, connected to stdin and stdout.  This demonstrated a
full round trip through the parsing and printing procedures.

## Evaluator

Rather than trying to handle everything in one complex evaluator, I
decided to implement a multi-pass evaluator.  This essentially means
that the expressions run through multiple different passes that apply
different transformations to them, thereby keeping the complexity of
each pass more managable.  This is a common method used when
implementing compilers and interpreters.

### Normalisation

The first preliminary pass I implemented was a normalisation pass.
This transforms things like `x - y` into `x + (-1)y`, and `x/y` into
`xy^(-1)`.  The main benefit of this is to make it easier for the
later differentiation pass as it will only have to handle sums,
products and exponents (not differences and quotients).  I decided to
leave quotients of integers as-is since these will be untouched by the
differentiation process anyway, and "1/2" reads far better than
"1*2^(-1)".

### Constant Folding

The purpose of this pass is to combine constants together as much as
possible, such as transforming `1 + 2*3` into `6`, or `x + 1 + 2` into
`x + 3`.  This is mostly a nice-to-have rather than an essential
feature, though it will make the output much more readable.

Implementing this pass did get a little hairy, as the binary tree
structure I used for the AST made handling cases like `x + 1 + 2`
(which is represented in the AST as `(x + 1) + 2` quite complex.  The
module implementing this therefore ended up being somewhat messy,
especially since Rust's move/borrow semantics made flattening trees of
sums and products more complex than it would otherwise be.  However, I
decided to move on rather than spend a great deal of time tidying this
up since, as previously mentioned, this was more of a nice-to-have
feature rather than core functionality.

### Evaluator Composition

I then implemented a mechanism to compose evaluators together, and
build a pipeline of arbitrarily many of them.  This was simple to do
since I was implementing the `Evaluator` trait for each pass, so
supporting composition simply meant creating a struct containing two
evaluators, and implementing the `Evaluator` trait for this such that
the evaluators were run in sequence on any input expression.

To combine more than two evaluators, I created a `pipeline` macro that
constructs nested compositions of evaluators with Rust's `macro_rules`
system.

### Differentiator

The differentiation pass itself was very easy to implement given the
simplications made by first having a normalisation pass.  I also had
decided to only support constant exponents, which simplifies things
too.  In the end, it ended up being more-or-less a fairly direct
translation of the sum rule and product rule into code, along with the
power rule and chain rule combined for handling exponents.

### Simplification

The output of the differentiation pass was correct but contained a lot
of noise such as terms multiplied by zero and other similar things.  I
therefore implemented a basic simplification pass that applied various
identities such as `x + 0 = x`, `0x = 0` and `1x = 1`.  Expression
simplification is famously a very complex problem, but I kept this
pass quite limited in scope and only implemented the most high-payoff
features.

### Canonicalisation

To make the output a little easier to read, the final evaluation pass
I implemented was a limited multiplication reordering canonicalisation
pass that ensured that products had the constants first.  This is
still incomplete as the re-ordering only takes place within binary
products (so `x*4` becomes `4x`, but `x*(y*4)` would become `x*4y` not
`4xy`), and terms in a sum were not re-ordered at all.  However, I
decided full canonicalisation would be excessively complex for this
project and decided to leave it there, as the output was acceptably
readable at this point anyway.

### Wrapping up

The final pipeline I ended up with was:

1. Normalisation
2. Differentiation
3. Constant folding
4. Simplification
5. Constant folding again
6. Canonicalisation

I then created an `evaluator` function to wrap this up in the `eval`
module, to allow for a simple interface to the whole library.

## Conclusion / Closing thoughts

Creating the differentiation program was a lot of fun, and I'm happy
with the result I ended up with.  The code is well-tested, and for the
most part is fairly maintainable (perhaps with the area needing the
most work in that respect being the constant folding pass).

In order to quantify how effective the test suite is, I ran a mutation
testing tool (`cargo-mutants`) and it found that only 3 out of 221
generated mutants were missed by the test suite, giving a mutation
score of 98.6% which is very good.  The three missed mutants were also
limited to the demo program itself (which makes sense since it has no
tests) and error message formatting code, so none of them are
particularly concerning.

There are quite a lot of areas where further work could be done, such
as adding support for functions like sine, cosine and natural
logarithms; enhancing the simplification and canonicalisation passes
and allowing non-constant exponents.  However, I think the most
interesting enhancement would be to add equalities to the language and
allow definitions to be stored within a session, with explicit
differentiation commands to get the deriviative of something, such as:

```
> y = x^3 - x^2
> dy/dx
3x^2 - 2x
```

This, along with a few other features, would transform the program
into a more general-purpose symbolic mathematics engine, which would
be very cool and useful.  The current architecture would allow for
this quite nicely, as evaluators are able to have mutable state which
could be used for storing variable bindings.