Testing and output validation

Use mock tests to check behavior with known inputs, including edge cases and external-service failures. Use validators to inspect the actual output produced during a live run. They answer different questions, so their reports are tracked separately.

Transform mock testsCopy Anchor Link

Use bare @vxfy_mocks and @vxfy_test(mocks=factory); import from vortexify.testing. Tests take run, which accepts no arguments. Each call creates fresh mocks and invokes the resource in the test process. These functions can live with the resource or in helper modules; zero or more tests are allowed.

import pandas as pd
from vortexify.testing import vxfy_mocks, vxfy_test

@vxfy_mocks
def sample_orders():
    return {"orders": pd.DataFrame({"customer_id": ["a", "a"], "amount": [2.0, 3.0]})}

@vxfy_test(mocks=sample_orders)
def test_customer_totals(run):
    assert run().to_dict("records") == [{"customer_id": "a", "amount": 5.0}]

Pair this with the Pandas customer-totals entrypoint in Transforms. Mock keys match declared input names. Missing scalars use a declared default, or None when optional. Other inputs must be supplied: DataFrames for tabular declarations, explicit non-None secret values, and database connection doubles supporting cursor() or execute(). No live input resolution occurs. DuckDB uses a fresh in-memory connection; PySpark conversion of pandas mocks requires a worker-supplied Spark session. Return fresh mutable mocks. Patch external calls at the module name used by the resource. Never use case=.

Reports include test name/module, mock factory, status, duration_seconds, run_count, and error details. SDK statuses are passed/failed/error; worker job status is a separate field. Mock tests do not publish tables or run production validation.

API mock exampleCopy Anchor Link

Pair this test with the normalize API in APIs:

from vortexify.testing import vxfy_mocks, vxfy_test

@vxfy_mocks
def padded_value():
    return {"value": "  Hi  "}

@vxfy_test(mocks=padded_value)
def test_normalize(run):
    assert run() == {"value": "Hi", "length": 2}

Each run() resolves fresh mocks and scalar defaults. SDK mock-test decorators apply to transform/API functions, not Connector/Sync classes or Agent declarations. Mock external effects explicitly; preview itself does not suppress writes.

Output validationCopy Anchor Link

At most one @vxfy_validate(engine="pandas"/"duckdb"/"pyspark") function may check real resource output and return ValidationResult(issues=[ValidationIssue(...)]). ValidationIssue accepts message, level (error default or warning), optional code and field. Any error makes .valid false, but findings or validator exceptions do not fail a successful transform or prevent its tables from being published. The runtime result includes a structured validation report with status passed, issues_found, error, or skipped, plus valid, issues, and error fields. When no validator is defined, the report is skipped with reason not_defined. Reports are also emitted as JSON logs and saved on the job. Validation is separate from mock tests.

Save related source edits before execution; jobs consume immutable saved versions. Inspect the actual test results and logs to verify behavior. Stopping a wait does not cancel its job. A successful save is not a successful test, deployment, or production run. Inspect existing job results before submitting duplicate work.

API behaviorCopy Anchor Link

At most one @vxfy_validate() can inspect the real JSON API response. Leave its engine unset for JSON rather than tabular validation. Deployed APIs skip automatic validation with status="skipped" and reason="deployed_api"; mock tests also do not call a validator automatically. Call it directly when testing validation behavior, and put required runtime checks in the API function.

Validator exampleCopy Anchor Link

import pandas as pd
from vortexify import ValidationIssue, ValidationResult, vxfy_validate

@vxfy_validate(engine="pandas")
def validate_totals(data: pd.DataFrame) -> ValidationResult:
    issues = []
    if data["amount"].isna().any():
        issues.append(ValidationIssue(
            message="Some totals are missing an amount.", field="amount", code="missing_amount",
        ))
    return ValidationResult(issues=issues)

Use this shape for a single Pandas output with an amount column. A named-output transform needs validation that matches its output mapping rather than assuming one frame. Error-level issues make .valid false; warning-level issues are recorded without doing so. Validation reports are distinct from execution status and publication success.

Read test results in the platformCopy Anchor Link

Open an API or transform Playground → Configure to prepare the runtime and run tests against committed code. Choose the resulting test job in History. Output contains the test report; Processing distinguishes test outcome, execution status, validation, and related jobs; Logs contains diagnostic output.

A test inventory describes tests parsed from source. In the current server, list_function_tests returns those declarations and mock factories, not the last execution status. Read a test job's results for pass/fail evidence and check its resource version.

For a live run, validation describes the published output or API response separately from mock tests. A successful transform may have issues_found or error validation while its table publication is available.

Run saved testsCopy Anchor Link

vortexify test transform customer_totals --wait
vortexify test api normalize --wait

Inspect per-test statuses, errors, and logs. Test reports use passed, failed, or error; worker job status is a separate field. Commit the intended source before running saved tests. Stopping a wait does not cancel the backend job.