Amazon Athena connector

Use this as an implementation starting point: adapt the worked example to the requested dataset and the source documentation. See Custom API for the shared SDK pattern and document-upload workflow.

What this connector is forCopy Anchor Link

Serverless SQL query service for data stored in object storage.

Use this guide to choose the resources and row meanings for a Vortexify source integration. Connection ownership belongs to a connector; each dataset gets its own sync, configuration, and extraction behavior.

Data and tables you can pullCopy Anchor Link

The following table families are a starting point for source selection. They describe possible datasets to implement using the vendor reference, not a guarantee that the example implements every table. Actual availability depends on the product edition, deployment, account permissions, and API/schema version.

Relationships and extraction considerationsCopy Anchor Link

These are proposed dataset names, not a universal schema guaranteed by the vendor. Preserve source object IDs and relationship references, and split nested child collections into separate tables when appropriate. Choose the concrete endpoint/report and row meaning from the vendor reference before implementing a sync.

Connecting to Amazon AthenaCopy Anchor Link

Set service_name to athena. Query execution needs the catalog/database, workgroup, permissions, and result-location configuration. Poll for completion and paginate results; queries can incur charges.

The example's config keys show the connection fields it uses. Store credentials in platform configuration marked as secret; host/project/database identifiers are ordinary connection settings. Environment and network access must match the selected source.

requirements.inCopy Anchor Link

Declare these packages in the resource's requirements.in. Use the published vortexify package for the SDK and the source-specific driver/client shown below. Choose compatible dependency versions for your runtime; the connector and sync reference explains where each file belongs.

vortexify==0.1.1
pandas>=2,<3
boto3

Set service_name to athena. Query execution needs the catalog/database, workgroup, permissions, and result-location configuration. Poll for completion and paginate results; queries can incur charges.

Worked example: query result pagesCopy Anchor Link

This source-read pattern shows how to shape the selected resource into DataFrames. Use it inside the dataset sync, retaining shared connection ownership in the connector. Its scope is the named resource and filters; adapt the projection, batching, and preview to the requested table.

Connection factoryCopy Anchor Link

Use this client in the owning connector's client() method and release it in close().

import boto3


def make_client(config: dict):
    return boto3.client(
        config["service_name"],
        region_name=config["region"],
        aws_access_key_id=config.get("access_key_id"),
        aws_secret_access_key=config.get("secret_access_key"),
        aws_session_token=config.get("session_token"),
    )

Source readCopy Anchor Link

import time
import pandas as pd


def read_batches(client, config):
    execution_id = client.start_query_execution(
        QueryString=config["select_sql"], **config["execution_options"]
    )["QueryExecutionId"]
    deadline = time.monotonic() + 120
    while True:
        result = client.get_query_execution(QueryExecutionId=execution_id)
        status = result["QueryExecution"]["Status"]
        if status["State"] == "SUCCEEDED":
            break
        if status["State"] in {"FAILED", "CANCELLED"}:
            raise RuntimeError(status.get("StateChangeReason", status["State"]))
        if time.monotonic() >= deadline:
            raise TimeoutError(f"Athena query {execution_id} is still running")
        time.sleep(1)
    first_page = True
    for page in client.get_paginator("get_query_results").paginate(
        QueryExecutionId=execution_id,
        PaginationConfig={"PageSize": 1000},
    ):
        result_set = page["ResultSet"]
        columns = [
            column["Name"] for column in result_set["ResultSetMetadata"]["ColumnInfo"]
        ]
        rows = result_set["Rows"]
        if first_page:
            rows = rows[1:]  # SELECT result header appears only on the first page.
            first_page = False
        values = [[cell.get("VarCharValue") for cell in row["Data"]] for row in rows]
        if values:
            yield pd.DataFrame(values, columns=columns)

What to adaptCopy Anchor Link

Use service_name=athena. execution_options supplies the authorized WorkGroup and QueryExecutionContext, plus ResultConfiguration when required. Run a reviewed SELECT with bounded time/partitions; scan costs depend on the query. Result values arrive as strings and need declared type conversions. A timeout leaves the server query running.

Use the custom API SDK starting point to connect the read operation to Connector and Sync. For a source with its own driver, retain that client in Connector.client() and replace the HTTP request with the driver read. An extract(*, avg_row_size_bytes=None) implementation yields bounded DataFrames. An independent preview() uses a fresh connector and a small read; it must not call extraction or advance real state.

Choose explicit output columns and stable IDs for the requested table. The examples use json_normalize where useful to illustrate row shaping; nested arrays still need deliberate child tables and a stable declared output schema. A first-page sample is useful for development, while a full sync must follow all relevant continuation/parent-resource paths. Design incremental watermarks, update/deletion behavior, and rate-limit handling for the actual source before enabling incremental mode.

The purpose is to give a coding agent a concrete pattern to adapt, including for similar sources. These snippets are examples rather than a claim of complete vendor coverage. Check the source operation reference and representative responses before running with a customer's account; live vendor execution has not been verified here.

Sources and further readingCopy Anchor Link

• Vendor connection and resource reference — The interface/access prerequisites underlying this connection path; follow the selected resource's documentation for fields, queries, and pagination.
• Vortexify connectors and syncs — Connector ownership, schemas, preview isolation, extraction, and checkpoint contracts.
• Vortexify CLI workflow — Registering resources, committing source, and providing credential configuration.