# Testing and output validation

Source: /docs/reference/python/testing-and-validation.md

Use mock tests to check behavior with known inputs, including edge cases and external-service failures. Use validators to inspect the actual output produced during a live run. They answer different questions, so their reports are tracked separately.

## Transform mock tests [#transform-mock-tests]

Use bare `@vxfy_mocks` and `@vxfy_test(mocks=factory)`; import from
`vortexify.testing`. Tests take `run`, which accepts no arguments. Each call creates
fresh mocks and invokes the resource in the test process. These functions can live
with the resource or in helper modules; zero or more tests are allowed.

```python
import pandas as pd
from vortexify.testing import vxfy_mocks, vxfy_test

@vxfy_mocks
def sample_orders():
    return {"orders": pd.DataFrame({"customer_id": ["a", "a"], "amount": [2.0, 3.0]})}

@vxfy_test(mocks=sample_orders)
def test_customer_totals(run):
    assert run().to_dict("records") == [{"customer_id": "a", "amount": 5.0}]
```

Pair this with the Pandas customer-totals entrypoint in [Transforms](/docs/reference/python/transforms#inputs-and-runtime-values). Mock keys match declared input names.
Missing scalars use a declared default, or None when optional. Other inputs must
be supplied: DataFrames for tabular declarations, explicit non-None secret values,
and database connection doubles supporting cursor() or execute(). No live input
resolution occurs. DuckDB uses a fresh in-memory connection; PySpark conversion of
pandas mocks requires a worker-supplied Spark session. Return fresh mutable mocks.
Patch external calls at the module name used by the resource. Never use `case=`.

Reports include test name/module, mock factory, status, duration_seconds, run_count,
and error details. SDK statuses are passed/failed/error; worker job status is a
separate field. Mock tests do not publish tables or run production validation.

## API mock example [#api-mock-example]

Pair this test with the `normalize` API in [APIs](/docs/reference/python/apis):

```python
from vortexify.testing import vxfy_mocks, vxfy_test

@vxfy_mocks
def padded_value():
    return {"value": "  Hi  "}

@vxfy_test(mocks=padded_value)
def test_normalize(run):
    assert run() == {"value": "Hi", "length": 2}
```

Each `run()` resolves fresh mocks and scalar defaults. SDK mock-test decorators apply to transform/API functions, not Connector/Sync classes or Agent declarations. Mock external effects explicitly; preview itself does not suppress writes.

## Output validation [#output-validation]

At most one `@vxfy_validate(engine="pandas"/"duckdb"/"pyspark")` function may check
real resource output and return `ValidationResult(issues=[ValidationIssue(...)])`.
ValidationIssue accepts message, level (`error` default or `warning`), optional code
and field. Any error makes `.valid` false, but findings or validator exceptions do
not fail a successful transform or prevent its tables from being published.
The runtime result includes a structured validation report with status `passed`,
`issues_found`, `error`, or `skipped`, plus `valid`, `issues`, and `error` fields.
When no validator is defined, the report is skipped with reason `not_defined`. Reports are also
emitted as JSON logs and saved on the job. Validation is separate from mock tests.

Save related source edits before execution; jobs consume immutable saved versions.
Inspect the actual test results and logs to verify behavior. Stopping a wait
does not cancel its job. A successful save is not a successful test, deployment, or
production run. Inspect existing job results before submitting duplicate work.

### API behavior [#api-behavior]

At most one `@vxfy_validate()` can inspect the real JSON API response. Leave its engine unset for JSON rather than tabular validation. Deployed APIs skip automatic validation with `status="skipped"` and `reason="deployed_api"`; mock tests also do not call a validator automatically. Call it directly when testing validation behavior, and put required runtime checks in the API function.

### Validator example [#validator-example]

```python
import pandas as pd
from vortexify import ValidationIssue, ValidationResult, vxfy_validate

@vxfy_validate(engine="pandas")
def validate_totals(data: pd.DataFrame) -> ValidationResult:
    issues = []
    if data["amount"].isna().any():
        issues.append(ValidationIssue(
            message="Some totals are missing an amount.", field="amount", code="missing_amount",
        ))
    return ValidationResult(issues=issues)
```

Use this shape for a single Pandas output with an `amount` column. A named-output transform needs validation that matches its output mapping rather than assuming one frame. Error-level issues make `.valid` false; warning-level issues are recorded without doing so. Validation reports are distinct from execution status and publication success.

## Read test results in the platform [#read-test-results-in-the-platform]

Open an API or transform **Playground → Configure** to prepare the runtime and run tests against committed code. Choose the resulting test job in History. **Output** contains the test report; **Processing** distinguishes test outcome, execution status, validation, and related jobs; **Logs** contains diagnostic output.

A test inventory describes tests parsed from source. In the current server, `list_function_tests` returns those declarations and mock factories, not the last execution status. Read a test job's results for pass/fail evidence and check its resource version.

For a live run, validation describes the published output or API response separately from mock tests. A successful transform may have `issues_found` or `error` validation while its table publication is available.

## Run saved tests [#run-saved-tests]

```sh
vortexify test transform customer_totals --wait
vortexify test api normalize --wait
```

Inspect per-test statuses, errors, and logs. Test reports use `passed`, `failed`, or `error`; worker job status is a separate field. Commit the intended source before running saved tests. Stopping a wait does not cancel the backend job.