09 - Property-Based Tests

Status: Proposal

Date: 2026-03-27

Authors:

  • jinmin.hu@capgemini.com
  • huub.joosten@capgemini.com
  • luna.li@capgemini.com
  • paul.nelissen@esi.nl
  • pierre.vandelaar@tno.nl

Table of contents

Context

The goal of this ADR is to adopt property-based testing as a complementary approach to the existing parametrized tests in the Renaissance, so that the test effort of the developer of a new language for renaissance can be reduced and the test coverage can be improved.

The project currently uses a set of parametrized tests to verify behavior across a range of inputs. Maintaining these input tables by hand is tedious and error-prone; edge cases are easy to miss. Property-based testing offers an alternative approach where the testing framework generates input data automatically, guided by strategies and invariants declared by the developer. The formal, tree-structured nature of ASTs makes them well-suited to this approach.

Decision

Hypothesis is adopted as the property-based testing library for this project. It will complement (and where appropriate replace) existing parametrized tests. Hypothesis strategies will be used to generate diverse AST inputs, and properties (invariants) will be asserted rather than concrete expected values.

Additionally, Hypothesis can be used to validate code generated by AI tooling, providing a principled, automated way to check generated output against formal specifications.

Implementation notes

  • Use hypothesis strategies to generate AST nodes and transformation inputs.
  • Express test invariants as properties (e.g., "a round-trip parse/unparse yields the original source").
  • Gradually migrate existing @pytest.mark.parametrize tables to @given + @settings where the coverage benefit justifies the change.
  • Use hypothesis.extra integrations (e.g., hypothesis[pandas], hypothesis[numpy]) only where relevant.
  • Store Hypothesis database artifacts in .hypothesis/ (already git-ignored by default).

Example

from hypothesis import given, strategies as st
from renaissance.lst import LSTNode

@given(st.from_type(LSTNode))
def test_round_trip(node: LSTNode) -> None:
    """Parsing and unparsing an LSTNode must yield the original source."""
    assert unparse(parse(str(node))) == str(node)

@given(st.text(alphabet=st.characters(whitelist_categories=("Lu", "Ll"))))
def test_camel_case_no_spaces(name: str) -> None:
    result = camel_case(name)
    assert " " not in result

Rationale

Hypothesis and the formal nature of ASTs are a perfect combination for property-based testing: the structured, well-typed domain of AST nodes maps naturally onto Hypothesis strategies, and the algebraic properties of transformations (identity, round-trip, commutativity) are easy to express as invariants. This can replace the current set of parametrized tests with broader, automatically generated coverage. It can also be used to validate code generated by AI, providing an automated and principled quality gate.

Consequences

Positive:

  • Automatically discovers edge cases that hand-crafted tables miss.
  • Reduces the maintenance burden of large parametrize tables.
  • Provides a principled way to validate AI-generated code.
  • Shrinking produces minimal failing examples, making debugging easier.

Negative:

  • Adds an external dependency (hypothesis).
  • Tests may run longer due to the number of generated examples.
  • Writing good strategies for complex AST types requires upfront investment.

Alternatives considered

  • Continue with @pytest.mark.parametrize only — rejected because hand-crafted tables have limited coverage and high maintenance cost.
  • Use fuzzing tools (e.g., atheris) — rejected because they target low-level byte inputs rather than structured, typed domain objects.
  • See ADR 08 (Pytest Suite) for the overall testing framework choice that Hypothesis integrates with.

Revision history:

  • 2026-03-27: Converted to ADR template and clarified decision.