# Custom API connector

Source: /docs/connectors/custom-api.md

## What this connector is for [#what-this-connector-is-for]

Use this starting point when a source has no dedicated guide, or its API differs from a vendor example. A coding agent can use the source documentation and nearby examples to implement the required datasets. The API contract below is illustrative: replace its endpoint, authentication, response fields, and pagination with the actual source contract.

## Data and tables you can pull [#data-and-tables-you-can-pull]

Choose one sync per useful dataset: customers, orders, inventory, events, or another documented read resource. Decide what one row means, preserve stable IDs and foreign keys, and split nested collections into child tables when needed. An API endpoint, an API page, and a business table are different things.

## Give the agent searchable source documentation [#give-the-agent-searchable-source-documentation]

Attach API reference material, authentication instructions, pagination rules, example responses, and deployment-specific schemas to the connector. Useful inputs include vendor documentation, internal API guides, and an OpenAPI specification saved as readable Markdown or text. The CLI accepts `.pdf`, `.txt`, `.md`, and `.markdown` document uploads.

From an initialized Vortexify project:

```sh
vortexify create connector custom_api
vortexify upload file ./api-reference.md --resource-type connector --resource custom_api --wait --json
vortexify files connector custom_api --json
```

Wait for successful document ingestion. A coding agent can inspect the attachment inventory and beginning-text previews with the CLI; keep the original documentation accessible locally when it needs to search the full text. The CLI currently exposes file-list and preview commands, rather than a full-document search command.

`vortexify files preview FILE_ID --chars 4000 --json` reads a bounded beginning of one attachment. Attaching documentation does not configure credentials or implement a sync.

### Searchable documentation for a custom agent [#searchable-documentation-for-a-custom-agent]

If the app includes a custom Vortexify agent that should answer questions using API or business documentation, attach the material to that agent:

```sh
vortexify upload file ./api-reference.md --resource-type agent --resource api_assistant --wait --json
```

Create and implement `api_assistant` first using [Add an assistant](/docs/building/add-an-assistant), then commit it after the upload. The custom agent's saved attachment set supplies document-search and bounded preview tools at runtime. Those attachments are managed by the platform, rather than declared as Python SDK context objects. Connector attachments and agent attachments have different owners; upload to the custom agent when it needs to search that document. Saving a new version and deploying it are separate steps.

Example task for a coding agent:

> Use the API reference in this project and the attachments listed for `custom_api`. Implement a connector and an orders sync using the current Python SDK. Confirm the auth scheme, order row schema, and pagination from the documents. Use the closest source guide as a starting point, adapt its request and response handling, and implement preview independently of extraction. Identify requirements.in packages and validate with representative responses.

## Connecting to the API [#connecting-to-the-api]

Keep credentials and shared client ownership in `Connector`. Keep resource selection, date ranges, and extraction behavior in `Sync`. `config.json` files describe JSON Schemas; actual secrets are supplied through platform configuration.

### requirements.in [#requirementsin]

```text
vortexify==0.1.1
httpx>=0.27,<1
pandas>=2,<3
```

No database driver is needed. Add the source's SDK or signing library if its authentication requires one. These are dependency declarations; resolve a runtime-compatible lockfile separately.

## Worked example: cursor-paginated JSON records [#worked-example-cursor-paginated-json-records]

This fictional API accepts `GET /records?limit=100&cursor=...` and returns `{"items": [{"id": "..."}], "next_cursor": "..."}`. It accepts a bearer token. These field names and credentials are an example contract, not defaults to assume for other APIs.

The connector creates one reusable client, checks access with a small read, advertises the implemented dataset, and releases its client. The sync reads pages into a stable `id` / `document` envelope so changing nested source fields do not silently change the output schema. An agent can instead declare explicit business columns.

### Connector: `connector.py` [#connector-connectorpy]

```python
import httpx
from vortexify import Connector, DiscoveredTable, Column


class CustomApiConnector(Connector):
    def client(self):
        if self._client is None:
            self._client = httpx.Client(
                base_url=self.config["base_url"],
                headers={"Authorization": f"Bearer {self.config['api_token']}"},
                timeout=30,
                follow_redirects=False,
            )
        return self._client

    def check(self):
        response = self.client().get("/records", params={"limit": 1})
        response.raise_for_status()
        if not isinstance(response.json()["items"], list):
            raise ValueError("Expected an items array")
        return True

    def discover(self):
        return [DiscoveredTable(name="records", schema=[
            Column(name="id", type="string", nullable=False),
            Column(name="document", type="string", nullable=False),
        ])]

    def close(self):
        if self._client is not None:
            self._client.close()
            self._client = None
```

### Connector schema: `config.json` [#connector-schema-configjson]

```json
{
  "type": "object",
  "properties": {
    "base_url": {"type": "string", "format": "uri", "description": "HTTPS API origin for this source"},
    "api_token": {"type": "string", "x-vortexify-secret": true}
  },
  "required": ["base_url", "api_token"],
  "additionalProperties": false
}
```

### Sync: `syncs/records/sync.py` [#sync-syncsrecordssyncpy]

```python
import json
from copy import deepcopy

import pandas as pd
from vortexify import Sync


def to_frame(records):
    return pd.DataFrame([
        {"id": str(record["id"]), "document": json.dumps(record, ensure_ascii=False)}
        for record in records
    ], columns=["id", "document"])


class RecordsSync(Sync):
    mode = "full"

    def preview(self):
        # Fresh client/configuration, no production checkpoints, no extract() call.
        with type(self.connector)(deepcopy(self.connector.config)) as source:
            response = source.client().get("/records", params={"limit": 100})
            response.raise_for_status()
            return to_frame(response.json()["items"][:100])

    def extract(self, *, avg_row_size_bytes=None):
        cursor = None
        seen_cursors = set()
        while True:
            params = {"limit": 100}
            if cursor is not None:
                params["cursor"] = cursor
            response = self.client.get("/records", params=params)
            response.raise_for_status()
            payload = response.json()
            records = payload["items"]
            if not isinstance(records, list):
                raise ValueError("Expected an items array")
            if records:
                yield to_frame(records)
            cursor = payload.get("next_cursor")
            if not cursor:
                break
            if cursor in seen_cursors:
                raise ValueError("Source repeated its continuation cursor")
            seen_cursors.add(cursor)
```

### Sync schema: `syncs/records/config.json` [#sync-schema-syncsrecordsconfigjson]

```json
{"type": "object", "properties": {}, "additionalProperties": false}
```

This example is a full read with ephemeral page cursors. It does not implement durable incremental checkpoints. The platform owns publication; incremental extraction needs a deliberate watermark/change-feed contract and handling for updates and deletions. `avg_row_size_bytes` may be `None`; this example uses a fixed page size.

## Adapt and run with the CLI [#adapt-and-run-with-the-cli]

```sh
vortexify create sync custom_api/records
```

Put the files in the project's configured connectors path, normally `connectors/custom_api/`, with the sync under `syncs/records/`. Connector commits exclude nested sync source, so commit each explicitly after editing:

```sh
vortexify status --type connector --resource custom_api
vortexify commit --type connector --resource custom_api
vortexify connector custom_api --configure
vortexify commit --type sync --resource custom_api/records
vortexify run sync custom_api/records --wait --json
```

For a real API, adapt authentication (OAuth, Basic, signed requests, sessions, or certificates), the selected read operation, pagination, source error handling, and rate-limit policy. Respect source page limits and `Retry-After`; add bounded retry handling for documented transient failures. Do not infer a successful empty result from an API error payload. Check preview isolation, multiple pages, empty pages with continuation, and permission failures with representative responses before a live run. A successful code commit and a successful sync run are separate steps.

## Inspect the resulting table [#inspect-the-resulting-table]

After successful extraction and schema publication, open the connector's Tables view or the sync's Runs & logs in the platform. Select the output table to inspect Data, Schema, and Relationships. Key and relationship discovery runs separately from ingestion; follow [data readiness](/docs/building/bring-in-data#wait-for-the-result-your-next-step-needs) before relying on inferred joins.

The sync schema in this example has no required values. For a real source that requires dataset settings, supply them through the sync configuration service after saving the schema. The current CLI configures connector values interactively; the mounted frontend does not yet provide a dedicated sync settings form.

## Sources and further reading [#sources-and-further-reading]

- [Connector and sync SDK contracts](/docs/reference/python/connectors-and-syncs) — Client ownership, discovery schemas, preview, extraction, and state.
- [Bring in data](/docs/building/bring-in-data) — Registration, configuration, runs, and table references.
- [CLI resource commands](/docs/reference/cli/resources) — Source commits and document attachments.
- [HTTPX authentication](https://www.python-httpx.org/advanced/authentication/) — Supported authentication interfaces for source-specific adaptation.