Skip to content
MDM
2026

Neurosymbolic AI for handwritten formulas

Reading a sequence of handwritten digits and operators and computing the result, comparing logic built into the model with logic added to the loss.

Kind
Research
Context
Learning and Reasoning course, KU Leuven
Result
With five images, a plain neural network falls to 7.1% accuracy; putting the logic in the architecture keeps it at 98.7%.
Built with
Python, PyTorch Lightning, Probabilistic logic programming, CNNs

The task

The input is a row of handwritten images of digits and operators, such as an image of a 3, a plus sign and a 5. The model has to output the value of the formula.

The only supervision is the final result of each formula. No model ever sees a label for an individual digit or operator.

Three models

  1. Neural baseline. A CNN that predicts the result directly from the images.
  2. Logic in the architecture. Separate networks recognise digits and operators, and a probabilistic logic program combines their outputs into a result. The logic is compiled once into AND-OR trees, so training backpropagates straight through the reasoning.
  3. Logic in the loss. A network predicts the result directly, with a logic-consistency penalty added during training only. At inference it is a plain neural network.

Result

3 images: Neural baseline 86%; Logic in the architecture 97.8%; Logic in the loss 96.5%. 5 images: Neural baseline 7.1%; Logic in the architecture 98.7%; Logic in the loss 28.9%3 imagese.g. 7 + 2, mean of 3 runsNeural baseline86.0%Logic in the architecture97.8%Logic in the loss96.5%5 imagese.g. 7 + 2 − 4, one runNeural baseline7.1%Logic in the architecture98.7%Logic in the loss28.9%
Test accuracy on the final result. With five images the plain network has to learn too many combinations; with the rules in the architecture it only has to recognise single symbols.

Longer formulas break the plain neural network: it has to learn every combination from examples, and there are too many. Putting the rules of arithmetic in the architecture means the networks only need to recognise single symbols, so accuracy stays near-perfect as formulas get longer. Adding the rules to the loss helps on short formulas but does not carry over.

How much data each approach needs

With 3 images and only 300 training formulas, logic in the architecture already reaches 70.6%, against 9.1% for the plain network.

Training formulas3003,00030,000
Neural baseline9.1%22.6%86.0%
Logic in the architecture70.6%94.2%97.8%
Logic in the loss18.3%82.8%96.5%

Mean of 3 runs.

A new task

I also designed a task from scratch: arithmetic modulo 10, with multiplication, division and correct operator precedence (1 + 2 × 3 = 7), written as a new logic program.