Transforms

A transform performs background computation and publishes tabular output. Define one synchronous @vxfy_transform entrypoint per resource and keep dependencies in requirements.in. Source declarations define its runtime inputs, compute, and scheduling.

EntrypointCopy Anchor Link

vxfy_transform(*, engine="pandas", inputs=None, schedule=None, compute=None)

• engine: pandas, duckdb, or pyspark; default pandas.
• inputs: mapping of local input names to declarations; default empty mapping.
• schedule: None, one CronSchedule/TriggerSchedule, or a list of schedules; normalized to a list. Default no automatic schedule.
• compute: TransformCompute for pandas/DuckDB, TransformSparkCompute for PySpark; default None leaves allocation resolution to the platform.
• Define a synchronous function, not an async function or generator. Its signature must accept the injected keyword arguments. There is no required function name.
• Return a DataFrame or a dictionary of named DataFrames for published tables. Prefer explicit materialization of DuckDB results, such as .df(), before returning. Do not use an API response dictionary of scalar values as a published transform table.

Inputs and runtime valuesCopy Anchor Link

Use actual resource names and inspect upstream schemas before writing joins or SQL. Declaration keys are local names, independent of referenced resource names. Input supports only str, int, float, bool; defaults must have the exact declared type. Optional scalars without a default may be None. Do not declare arbitrary JSON objects, arrays, tables, or secret values as scalar Input defaults.

Pandas receives tabular inputs as pandas DataFrames. PySpark receives Spark DataFrames. DuckDB receives an injected conn, with tabular inputs registered as relations under the declaration keys. DuckDB signatures receive conn and the non-tabular inputs, not extra DataFrame arguments for registered relations. conn is reserved. Scalars, secrets and database connections are injected directly for all engines. Database modes are read and read_write, default read_write; operational SQL is separate from schema migrations. Database inputs follow execution environment selection; see database environments.

import pandas as pd
from vortexify import Input, Table, TransformCompute, vxfy_transform

@vxfy_transform(
    inputs={
        "orders": Table("shopify.orders"),
        "minimum_amount": Input(float, default=0.0),
    },
    compute=TransformCompute(size="small"),
)
def customer_totals(orders: pd.DataFrame, minimum_amount: float) -> pd.DataFrame:
    selected = orders.loc[orders["amount"] >= minimum_amount]
    return selected.groupby("customer_id", as_index=False)["amount"].sum()

The example assumes those resources and columns exist. An alternative DuckDB entrypoint is below; do not put both entrypoints in one resource.

from vortexify import Table, vxfy_transform

@vxfy_transform(engine="duckdb", inputs={"orders": Table("shopify.orders")})
def customer_totals(conn):
    return conn.execute(
        "SELECT customer_id, SUM(amount) AS amount FROM orders GROUP BY customer_id"
    ).df()

For multiple outputs, return e.g. {"daily": daily_dataframe, "weekly": weekly_dataframe} and reference them downstream with TransformOutput("resource_name.daily").

Follow a publication in the platformCopy Anchor Link

The transform Playground History lists runs by saved version and environment. Output opens the tables published by that run; Processing shows publication progress, validation findings, and links to related runtime or schema jobs. A table's Data, Schema, and Relationships views use the selected publication.

Choose Preview runs or Deployed version and a run when inspecting output outside a pinned job. A selected historical run remains pinned when newer data arrives. Relationship cards can open the exact compared snapshot, so a join finding can be checked against the data it analyzed.

Compute and schedulingCopy Anchor Link

Use TransformCompute for Pandas/DuckDB or TransformSparkCompute for PySpark. See compute and schedules for fields, presets, and cron/upstream-trigger declarations. Manual previews remain explicit and do not start production schedules.

Mock tests and output validationCopy Anchor Link

Use fresh @vxfy_mocks factories and @vxfy_test(mocks=...) tests with a no-argument run() helper. Tests resolve no live upstream resources. A validator inspects real resource output separately; findings and validator errors do not block successful table publication. See testing and validation.

vortexify commit --type transform --resource customer_totals
vortexify test transform customer_totals --wait
vortexify run transform customer_totals --wait

Jobs consume saved source versions. Check result inventories, findings, and logs before describing the output as ready. Saving source, passing mocks, publishing a preview table, and deploying are distinct steps. See query and prepare data.