Hypothesis internals¶

Warning

This page documents internal Hypothesis interfaces. Some are fairly stable, while others are still experimental. In either case, they are not subject to our standard deprecation policy, and we might make breaking changes in minor or patch releases.

This page is intended for people building tools, libraries, or research on top of Hypothesis. If that includes you, please get in touch! We’d love to hear what you’re doing, or explore more stable ways to support your use-case.

Alternative backends¶

See also

See also the user-facing Alternative backends for Hypothesis documentation.

class hypothesis.internal.conjecture.providers.PrimitiveProvider(conjecturedata, /)[source]¶

PrimitiveProvider is the implementation interface of a Hypothesis backend.

A PrimitiveProvider is required to implement the following five draw_* methods:

draw_integer()
draw_boolean()
draw_float()
draw_string()
draw_bytes()

Each strategy in Hypothesis generates values by drawing a series of choices from these five methods. By overriding them, a PrimitiveProvider can control the distribution of inputs generated by Hypothesis.

For example, hypothesis-crosshair implements a PrimitiveProvider which uses an SMT solver to generate inputs that uncover new branches.

Once you implement a PrimitiveProvider, you can make it available for use through AVAILABLE_PROVIDERS.

lifetime = 'test_function'¶

The lifetime of a PrimitiveProvider instance. Either test_function or test_case.

If test_function (the default), a single provider instance will be instantiated and used for the entirety of each test function (i.e., roughly one provider per @given annotation). This can be useful for tracking state over the entirety of a test function.

If test_case, a new provider instance will be instantiated and used for each input Hypothesis generates.

The conjecturedata argument to PrimitiveProvider.__init__ will be None for a lifetime of test_function, and an instance of ConjectureData for a lifetime of test_case.

Third-party providers likely want to set a lifetime of test_function.

avoid_realization = False¶

Solver-based backends such as hypothesis-crosshair use symbolic values which record operations performed on them in order to discover new paths. If avoid_realization is set to True, hypothesis will avoid interacting with symbolic choices returned by the provider in any way that would force the solver to narrow the range of possible values for that symbolic.

Setting this to True disables some hypothesis features and optimizations. Only set this to True if it is necessary for your backend.

add_observability_callback = False¶

will never be called by Hypothesis.

The opt-in behavior of observability is because enabling observability might increase runtime or memory usage.

abstractmethod draw_boolean(p=0.5)[source]¶

Draw a boolean choice.

Parameters:

p (float) –

The probability of returning True. Between 0 and 1 inclusive.

Except for 0 and 1, the value of p is a hint provided by Hypothesis, and may be ignored by the backend.

If 0, the provider must return False. If 1, the provider must return True.

abstractmethod draw_integer( min_value=None, max_value=None, *, weights=None, shrink_towards=0, )[source]¶

Draw an integer choice.

Parameters:

min_value (int | None) – (Inclusive) lower bound on the integer value. If None, there is no lower bound.
max_value (int | None) – (Inclusive) upper bound on the integer value. If None, there is no upper bound.
weights (dict[int, float] | None) – Maps keys in the range [min_value, max_value] to the probability of returning that key.
shrink_towards (int) – The integer to shrink towards. This is not used during generation and can be ignored by backends.

abstractmethod draw_float( *, min_value=-inf, max_value=inf, allow_nan=True, smallest_nonzero_magnitude, )[source]¶

Draw a float choice.

Parameters:

min_value (float) – (Inclusive) lower bound on the float value.
max_value (float) – (Inclusive) upper bound on the float value.
allow_nan (bool) – If False, it is invalid to return math.nan.
smallest_nonzero_magnitude (float) – The smallest allowed nonzero magnitude. draw_float should not return a float f if abs(f) < smallest_nonzero_magnitude.

abstractmethod draw_string( intervals, *, min_size=0, max_size=10000000000, )[source]¶

Draw a string choice.

Parameters:

intervals (IntervalSet) – The set of codepoints to sample from.
min_size (int) – (Inclusive) lower bound on the string length.
max_size (int) – (Inclusive) upper bound on the string length.

abstractmethod draw_bytes( min_size=0, max_size=10000000000, )[source]¶

Draw a bytes choice.

Parameters:

min_size (int) – (Inclusive) lower bound on the bytes length.
max_size (int) – (Inclusive) upper bound on the bytes length.

per_test_case_context_manager()[source]¶

Returns a context manager which will be entered each time Hypothesis starts generating and executing one test case, and exited when that test case finishes generating and executing, including if any exception is thrown.

In the lifecycle of a Hypothesis test, this is called before generating strategy values for each test case. This is just before any custom executor is called.

Even if not returning a custom context manager, PrimitiveProvider subclasses are welcome to override this method to know when Hypothesis starts and ends the execution of a single test case.

realize(value, *, for_failure=False)[source]¶

Called whenever hypothesis requires a concrete (non-symbolic) value from a potentially symbolic value. Hypothesis will not check that value is symbolic before calling realize, so you should handle the case where value is non-symbolic.

The returned value should be non-symbolic. If you cannot provide a value, raise BackendCannotProceed with a value of "discard_test_case".

If for_failure is True, the value is associated with a failing example. In this case, the backend should spend substantially more effort when attempting to realize the value, since it is important to avoid discarding failing examples. Backends may still raise BackendCannotProceed when for_failure is True, if realization is truly impossible or if realization takes significantly longer than expected (say, 5 minutes).

replay_choices(choices)[source]¶

Called when Hypothesis has discovered a choice sequence which the provider may wish to enqueue to replay under its own instrumentation when we next ask to generate a test case, rather than generating one from scratch.

This is used to e.g. warm-start hypothesis-crosshair with a corpus of high-code-coverage inputs discovered by HypoFuzz.

observe_test_case()[source]¶

Called at the end of the test case when observability is enabled.

The return value should be a non-symbolic json-encodable dictionary, and will be included in observations as observation["metadata"]["backend"].

observe_information_messages(*, lifetime)[source]¶

Called at the end of each test case and again at end of the test function.

Return an iterable of {type: info/alert/error, title: str, content: str | dict} dictionaries to be delivered as individual information messages. Hypothesis adds the run_start timestamp and property name for you.

on_observation(observation)[source]¶

Called at the end of each test case which uses this provider, with the same observation["type"] == "test_case" observation that is passed to other callbacks added via add_observability_callback. This method is not called with observation["type"] in {"info", "alert", "error"} observations.

Important

For on_observation() to be called by Hypothesis, add_observability_callback must be set to True.

on_observation() is explicitly opt-in, as enabling observability might increase runtime or memory usage.

Calls to this method are guaranteed to alternate with calls to per_test_case_context_manager(). For example:

# test function starts
per_test_case_context_manager()
on_observation()
per_test_case_context_manager()
on_observation()
...
# test function ends

Note that on_observation() will not be called for test cases which did not use this provider during generation, for example during Phase.reuse or Phase.shrink, or because Hypothesis switched to the standard Hypothesis backend after this backend raised too many BackendCannotProceed exceptions.

span_start(label, /)[source]¶

Marks the beginning of a semantically meaningful span of choices.

Spans are a depth-first tree structure. A span is opened by a call to span_start(), and a call to span_end() closes the most recently opened span. So the following sequence of calls:

span_start(label=1)
n1 = draw_integer()
span_start(label=2)
b1 = draw_boolean()
n2 = draw_integer()
span_end()
f1 = draw_float()
span_end()

produces the following two spans of choices:

1: [n1, b1, n2, f1]
2: [b1, n2]

Hypothesis uses spans to denote “semantically meaningful” sequences of choices. For instance, Hypothesis opens a span for the sequence of choices made while drawing from each strategy. Not every span corresponds to a strategy; the generation of e.g. each element in lists() is also marked with a span, among others.

label is an opaque integer, which has no defined semantics. The only guarantee made by Hypothesis is that all spans with the same “meaning” will share the same label. So all spans from the same strategy will share the same label, as will e.g. the spans for lists() elements.

Providers can track calls to span_start() and span_end() to learn something about the semantics of the test’s choice sequence. For instance, a provider could track the depth of the span tree, or the number of unique labels, which says something about the complexity of the choices being generated. Or a provider could track the span tree across test cases in order to determine what strategies are being used in what contexts.

It is possible for Hypothesis to start and immediately stop a span, without calling a draw_* method in between. These spans contain zero choices.

Hypothesis will always balance the number of calls to span_start() and span_end(). A call to span_start() will always be followed by a call to span_end() before the end of the test case.

span_start() is called from ConjectureData.start_span() internally.

span_end(discard, /)[source]¶

Marks the end of a semantically meaningful span of choices.

discard is True when the draw was filtered out or otherwise marked as unlikely to contribute to the input data as seen by the user’s test. Note however that side effects can make this determination unsound.

span_end() is called from ConjectureData.stop_span() internally.

hypothesis.internal.conjecture.providers.AVAILABLE_PROVIDERS¶

Registered Hypothesis backends. This is a dictionary where keys are the name to be used in settings.backend. The value of a key can be either:

A string corresponding to an importable absolute path of a PrimitiveProvider subclass
A PrimitiveProvider subclass (the class itself, not an instance of the class)

Hypothesis will instantiate the corresponding PrimitiveProvider subclass when the backend is requested by a test’s settings.backend value.

For example, the default Hypothesis backend is registered as:

from hypothesis.internal.conjecture.providers import AVAILABLE_PROVIDERS

AVAILABLE_PROVIDERS["hypothesis"] = "hypothesis.internal.conjecture.providers.HypothesisProvider"
# or
AVAILABLE_PROVIDERS["hypothesis"] = HypothesisProvider

And can be used with:

from hypothesis import given, settings, strategies as st

@given(st.integers())
@settings(backend="hypothesis")
def f(n):
    pass

Though, as backend="hypothesis" is the default setting, the above would typically not have any effect.

For third-party backend authors, we strongly encourage ensuring that import hypothesis does not automatically import the expensive parts of your package, by:

setting a string path here, instead of a provider class
ensuring the registered hypothesis plugin path references a path which just sets AVAILABLE_PROVIDERS and does not import your package

hypothesis.internal.conjecture.provider_conformance.run_conformance_test( Provider, *, context_manager_exceptions=(), settings=None, _realize_objects=st.from_type(object) | st.from_type(type).flatmap(st.from_type), )[source]¶

Test that the given Provider class conforms to the PrimitiveProvider interface.

For instance, this tests that Provider does not return out of bounds choices from any of the draw_* methods, or violate other invariants which Hypothesis depends on.

This function is intended to be called at test-time, not at runtime. It is provided by Hypothesis to make it easy for third-party backend authors to test their provider. Backend authors wishing to test their provider should include a test similar to the following in their test suite:

from hypothesis.internal.conjecture.provider_conformance import run_conformance_test

def test_conformance():
    run_conformance_test(MyProvider)

If your provider can raise control flow exceptions inside one of the five draw_* methods that are handled by your provider’s per_test_case_context_manager, pass a list of these exceptions types to context_manager_exceptions. Otherwise, run_conformance_test will treat those exceptions as fatal errors.

class hypothesis.errors.BackendCannotProceed(scope='other', /)[source]¶

Raised by alternative backends when a PrimitiveProvider cannot proceed. This is expected to occur inside one of the .draw_*() methods, or for symbolic execution perhaps in realize().

The optional scope argument can enable smarter integration:

verified:
Do not request further test cases from this backend. We may generate more test cases with other backends; if one fails then Hypothesis will report unsound verification in the backend too.

exhausted:
Do not request further test cases from this backend; finish testing with test cases generated with the default backend. Common if e.g. native code blocks symbolic reasoning very early.

discard_test_case:
This particular test case could not be converted to concrete values; skip any further processing and continue with another test case from this backend.

final class hypothesis.internal.intervalsets.IntervalSet(intervals=())[source]¶: A compact and efficient representation of a set of (a, b) intervals. Can be treated like a set of integers, in that n in intervals will return True if n is contained in any of the (a, b) intervals, and False otherwise.

Observability¶

hypothesis.internal.observability.add_observability_callback(f, /, *, all_threads=False)[source]¶

Adds f as a callback for observability. f should accept one argument, which is an observation. Whenever Hypothesis produces a new observation, it calls each callback with that observation.

If Hypothesis tests are being run from multiple threads, callbacks are tracked per-thread. In other words, add_observability_callback(f) only adds f as an observability callback for observations produced on that thread.

If all_threads=True is passed, f will instead be registered as a callback for all threads. This means it will be called for observations generated by all threads, not just the thread which registered f as a callback. In this case, f will be passed two arguments: the first is the observation, and the second is the integer thread id from python:threading.get_ident() where that observation was generated.

We recommend against registering f as a callback for both all_threads=True and the default all_threads=False, due to unclear semantics with remove_observability_callback.

hypothesis.internal.observability.remove_observability_callback(f, /)[source]¶

Removes f from the observability callbacks.

If f is not in the list of observability callbacks, silently do nothing.

If running under multiple threads, f will only be removed from the callbacks for this thread.

hypothesis.internal.observability.with_observability_callback(f, /, *, all_threads=False)[source]¶: A simple context manager which calls add_observability_callback on f when it enters and remove_observability_callback on f when it exits.

hypothesis.internal.observability.observability_enabled()[source]¶

Returns whether or not Hypothesis considers observability to be enabled. Observability is enabled if there is at least one observability callback present.

Callers might use this method to determine whether they should compute an expensive representation that is only used under observability, for instance by alternative backends.

hypothesis.internal.observability.TESTCASE_CALLBACKS = <hypothesis.internal.observability._TestcaseCallbacks object>¶: Warning

Deprecated in favor of add_observability_callback, remove_observability_callback, and observability_enabled.

TESTCASE_CALLBACKS remains a thin compatibility shim which forwards .append, .remove, and bool() to those three methods. It is not an attempt to be fully compatible with the previous TESTCASE_CALLBACKS = [], so iteration or other usages will not work anymore. Please update to using the new methods instead.

TESTCASE_CALLBACKS will eventually be removed.

hypothesis.internal.observability.OBSERVABILITY_COLLECT_COVERAGE = True¶

If False, do not collect coverage information when observability is enabled.

This is exposed both for performance (as coverage collection can be slow on Python 3.11 and earlier) and size (if you do not use coverage information, you may not want to store it in-memory).

hypothesis.internal.observability.OBSERVABILITY_CHOICES = False¶

If True, include the metadata.choice_nodes and metadata.spans keys in test case observations.

False by default. metadata.choice_nodes and metadata.spans can be a substantial amount of data, and so must be opted-in to, even when observability is enabled.

Warning

EXPERIMENTAL AND UNSTABLE. We are actively working towards a better interface for this as of June 2025, and this attribute may disappear or be renamed without notice.

Engine constants¶

We pick reasonable values for these constants, but if you must, you can monkeypatch them. (Hypothesis is not responsible for any performance degradation that may result).

hypothesis.internal.conjecture.engine.MAX_SHRINKS = 500¶: The maximum number of times the shrinker will reduce the complexity of a failing input before giving up. This avoids falling down a trap of exponential (or worse) complexity, where the shrinker appears to be making progress but will take a substantially long time to finish completely.

hypothesis.internal.conjecture.engine.MAX_SHRINKING_SECONDS = 300¶: The maximum total time in seconds that the shrinker will try to shrink a failure for before giving up. This is across all shrinks for the same failure, so even if the shrinker successfully reduces the complexity of a single failure several times, it will stop when it hits MAX_SHRINKING_SECONDS of total time taken.

hypothesis.internal.conjecture.engine.BUFFER_SIZE = 8192¶

The maximum amount of entropy a single test case can use before giving up while making random choices during input generation.

The “unit” of one BUFFER_SIZE does not have any defined semantics, and you should not rely on it, except that a linear increase BUFFER_SIZE will linearly increase the amount of entropy a test case can use during generation.