Confluence connector
Team documentation and knowledge platform for spaces, pages, and comments.
Use this as an implementation starting point: adapt the worked example to the requested dataset and the source documentation. See Custom API for the shared SDK pattern and document-upload workflow.
What this connector is for
Team documentation and knowledge platform for spaces, pages, and comments.
A connector owns access to the source. Separate syncs shape selected resources into tables for analysis and downstream transformations using the Python SDK.
Data and tables you can pull
These are representative dataset families to consider, not a claim that the worked example implements every resource. Access, fields, and availability depend on the source account, product edition, and permissions. Use the vendor reference to select concrete resources.
| Suggested table | One row represents |
|---|---|
spaces | a space |
pages | a page |
page_versions | a page version |
comments | a comment |
attachments | a attachment |
labels | a label |
Relationships and extraction considerations
Choose stable object IDs and preserve references between related resources. Nested arrays may need child tables. The listed names are proposed output datasets; validate the actual endpoint, fields, permissions, and pagination before adding each sync.
Connecting to Confluence
Connects through the source's HTTP API using HTTP Basic authentication. Set api_token, domain_name, email in the connection configuration.
Additional resources may require additional scopes. Connection settings belong to the connector; dataset filters and resource selection belong to each sync. See the CLI workflow for registration and configuration.
Connection settings used by the example
| Setting | Requirement | Purpose |
|---|---|---|
api_token | Required; secret | Credential or scope value used by the Confluence API example. |
domain_name | Required | Credential or scope value used by the Confluence API example. |
email | Required | Credential or scope value used by the Confluence API example. |
requirements.in
Save these dependencies in requirements.in alongside connector.py; the example includes packages used by its sync. Install the published vortexify package for local development. Keep sync-specific dependencies in the sync's own requirements.in when splitting the implementation into separate resources.
vortexify==0.1.1
httpx>=0.27,<1
pandas>=2,<3No separately installed database driver is needed by this HTTP example.
Worked example: sync audit
audit — Reads records from the audit API resource. Includes description, category, creationDate, remoteAddress, summary and the complete record as JSON in document.
The complete files below implement only this example resource. They illustrate the current SDK signatures, a separate preview, bounded DataFrame output, and source-specific extraction. They do not establish complete source coverage. Live credentials and platform ingestion have not been used to validate this guide.
With default paths, place the connector files in connectors/confluence/ and sync files in connectors/confluence/syncs/audit/. Use your project configuration if it specifies different paths.
Connection schema: config.json
This file contains a JSON Schema, not actual credential values. Provide those through platform configuration.
{
"type": "object",
"properties": {
"api_token": {
"type": "string",
"title": "Api Token",
"description": "Credential or scope value used by the Confluence API example.",
"x-vortexify-secret": true,
"order": 0
},
"domain_name": {
"type": "string",
"title": "Domain Name",
"description": "Credential or scope value used by the Confluence API example.",
"order": 1
},
"email": {
"type": "string",
"title": "Email",
"description": "Credential or scope value used by the Confluence API example.",
"order": 2
}
},
"required": [
"api_token",
"domain_name",
"email"
]
}Connector: connector.py
import httpx
from vortexify import Column, Connector, DiscoveredTable
class ConfluenceConnector(Connector):
vendor_domain = "atlassian.com"
def client(self):
if self._client is None or self._client.is_closed:
self._client = httpx.Client(
base_url="https://" + str(self.config["domain_name"]) + "/",
headers={"Accept": "application/json"},
auth=(self.config["email"], self.config["api_token"]),
timeout=60,
follow_redirects=False,
)
return self._client
def check(self) -> bool:
cfg = self.config
page_size = 1
path = "wiki/rest/api/audit"
params = {}
params["limit"] = page_size
params["start"] = 0
response = self.client().get(path, params=params)
response.raise_for_status()
payload = response.json()
values = [payload]
for part in ["results"]:
children = []
for value in values:
if part == "*":
if isinstance(value, list):
children.extend(value)
elif isinstance(value, dict):
children.extend(value.values())
elif isinstance(value, dict) and part in value:
children.append(value[part])
elif isinstance(value, dict) and value:
raise ValueError("Missing API record container")
values = children
records = []
for value in values:
if isinstance(value, list):
records.extend(value)
elif isinstance(value, dict):
records.append(value)
elif value is not None:
raise ValueError("Unexpected API record container")
return True
def discover(self) -> list[DiscoveredTable]:
return [
DiscoveredTable(
name="audit",
schema=[
Column(name="description", type="string"),
Column(name="category", type="string"),
Column(name="creationDate", type="integer"),
Column(name="remoteAddress", type="string"),
Column(name="summary", type="string"),
Column(name="document", type="string"),
],
)
]
def close(self) -> None:
if self._client is not None:
try:
self._client.close()
finally:
self._client = NoneSync schema: syncs/audit/config.json
{
"type": "object",
"properties": {},
"required": []
}Sync: syncs/audit/sync.py
from collections.abc import Iterator
from math import isfinite
import json
import pandas as pd
from vortexify import Sync
class ConfluenceAuditSync(Sync):
def preview(self) -> pd.DataFrame:
cfg = self.connector.config
page_size = 25
path = "wiki/rest/api/audit"
params = {}
params["limit"] = page_size
params["start"] = 0
response = self.client.get(path, params=params)
response.raise_for_status()
payload = response.json()
values = [payload]
for part in ["results"]:
children = []
for value in values:
if part == "*":
if isinstance(value, list):
children.extend(value)
elif isinstance(value, dict):
children.extend(value.values())
elif isinstance(value, dict) and part in value:
children.append(value[part])
elif isinstance(value, dict) and value:
raise ValueError("Missing API record container")
values = children
records = []
for value in values:
if isinstance(value, list):
records.extend(value)
elif isinstance(value, dict):
records.append(value)
elif value is not None:
raise ValueError("Unexpected API record container")
if not isinstance(records, list) or any(
(not isinstance(record, dict) for record in records)
):
raise ValueError(
"Unexpected API record shape; refusing a partial successful sync"
)
records = records[:100]
rows = []
for record in records:
row = {
"description": record.get("description"),
"category": record.get("category"),
"creationDate": record.get("creationDate"),
"remoteAddress": record.get("remoteAddress"),
"summary": record.get("summary"),
"document": json.dumps(record, ensure_ascii=False, default=str),
}
rows.append(
{
key: (
json.dumps(value, ensure_ascii=False, default=str)
if isinstance(value, (dict, list))
else value
)
for key, value in row.items()
}
)
return pd.DataFrame(
rows,
columns=[
"description",
"category",
"creationDate",
"remoteAddress",
"summary",
"document",
],
dtype=object,
)
def extract(
self, *, avg_row_size_bytes: float | None = None
) -> Iterator[pd.DataFrame]:
memory_mib = self.compute.memory if self.compute else None
memory_mib = memory_mib if memory_mib is not None and memory_mib > 0 else 1024
target_bytes = min(16 * 1024 * 1024, max(1, memory_mib * 1024 * 1024 // 20))
batch_size = 100
if avg_row_size_bytes is not None:
if not isfinite(avg_row_size_bytes) or avg_row_size_bytes <= 0:
raise ValueError("avg_row_size_bytes must be finite and positive")
batch_size = max(1, int(target_bytes / max(1.0, avg_row_size_bytes) / 4))
batch_size = min(batch_size, 25)
cfg = self.connector.config
page_size = batch_size
path = "wiki/rest/api/audit"
params = {}
params["limit"] = page_size
params["start"] = 0
seen_pages = set()
while True:
response = self.client.get(path, params=params)
response.raise_for_status()
payload = response.json()
values = [payload]
for part in ["results"]:
children = []
for value in values:
if part == "*":
if isinstance(value, list):
children.extend(value)
elif isinstance(value, dict):
children.extend(value.values())
elif isinstance(value, dict) and part in value:
children.append(value[part])
elif isinstance(value, dict) and value:
raise ValueError("Missing API record container")
values = children
records = []
for value in values:
if isinstance(value, list):
records.extend(value)
elif isinstance(value, dict):
records.append(value)
elif value is not None:
raise ValueError("Unexpected API record container")
if not isinstance(records, list) or any(
(not isinstance(record, dict) for record in records)
):
raise ValueError(
"Unexpected API record shape; refusing a partial successful sync"
)
rows = []
for record in records:
row = {
"description": record.get("description"),
"category": record.get("category"),
"creationDate": record.get("creationDate"),
"remoteAddress": record.get("remoteAddress"),
"summary": record.get("summary"),
"document": json.dumps(record, ensure_ascii=False, default=str),
}
rows.append(
{
key: (
json.dumps(value, ensure_ascii=False, default=str)
if isinstance(value, (dict, list))
else value
)
for key, value in row.items()
}
)
for start in range(0, len(rows), batch_size):
yield pd.DataFrame(
rows[start : start + batch_size],
columns=[
"description",
"category",
"creationDate",
"remoteAddress",
"summary",
"document",
],
dtype=object,
)
if len(records) < page_size:
break
params["start"] += len(records)Running and extending the example
Register the connector before the sync and commit each resource separately:
vortexify create connector confluence
vortexify create sync confluence/auditEdit the files, inspect vortexify status, and use the CLI guide to commit and configure them. Registration alone does not run extraction or deploy the connector.
Preview must run with a fresh connector and disposable state. The extraction implementation determines full/incremental behavior; merely setting a mode does not implement change capture. API/driver failures propagate unless explicitly handled in the code. Check source limits and single-record/response sizes as well as output batch size.
Sources and further reading
- Vendor API and data reference — Resource definitions, authentication, fields, and source-specific limits. Check the selected resource's reference when extending the example.
- Vortexify connector and sync contracts — Client ownership, discovery, preview isolation, and DataFrame extraction.
- Vortexify CLI workflow — Project paths, resource registration, source commits, and credential configuration.