Data catalogue

ElasticsearchComing soon

Turn an index, alias, or data stream into a dataset, with optional query and aggregation.

The Elasticsearch connector reads documents from an index, alias, or data stream and flattens them into typed fields. It works with Elasticsearch 7.10+ and OpenSearch 2.x.

Not available yet

The Elasticsearch connector is in development. This page describes how it will work — the scoped API key below is worth creating in advance.

Connect PostgreSQL or Neo4j today.

Connection details

Field Required Notes
Connection Name yes Display name in the catalogue
Endpoint yes Full URL, e.g. https://es.internal:9200
Authentication yes API key, or username and password
Index / alias yes Supports patterns such as logs-*
CA certificate no For self-signed clusters
HTTP
Authorization: ApiKey {{secrets.ES_API_KEY}}

Create a key scoped to read and view_index_metadata on the indices you expose — nothing more.

HTTP
Authorization: Basic {{base64(secrets.ES_USER + ":" + secrets.ES_PASS)}}

Use the Cloud ID plus an API key. Chartizer resolves the endpoint from the Cloud ID.

Minimum role
{
  "indices": [
    {
      "names": ["orders-*"],
      "privileges": ["read", "view_index_metadata"]
    }
  ]
}

Defining the dataset

A dataset is an index plus an optional query. Leave the query empty to read everything.

Only shipped orders from the last 90 days
{
  "index": "orders-*",
  "query": {
    "bool": {
      "filter": [
        { "term": { "status": "shipped" } },
        { "range": { "created_at": { "gte": "now-90d" } } }
      ]
    }
  },
  "fields": ["order_id", "customer.email", "total", "created_at"]
}

Always list the fields you need

Elasticsearch documents are often wide and deeply nested. Naming fields explicitly keeps the dataset narrow, syncs faster, and stops a mapping change in an unused field from causing drift.

Field mapping

Elasticsearch types map to Chartizer types as follows.

Elasticsearch Becomes Note
keyword text Preferred for grouping
text text Analysed — aggregate on the .keyword sub-field
long, integer, short integer
float, double, scaled_float decimal
date timestamp Converted to UTC
boolean boolean
object flattened Dot notation: customer.email
nested array Not exploded into rows
geo_point geo Usable in map charts

Aggregate on keyword, not text

Grouping a chart by an analysed text field gives you tokens, not values — New York becomes new and york. Use city.keyword.

Aggregations

For large indices, let Elasticsearch aggregate and read back the buckets instead of raw documents. The dataset becomes one row per bucket.

Daily revenue by status
{
  "index": "orders-*",
  "aggregation": {
    "by_day": {
      "date_histogram": { "field": "created_at", "calendar_interval": "day" },
      "aggs": {
        "by_status": {
          "terms": { "field": "status", "size": 20 },
          "aggs": { "revenue": { "sum": { "field": "total" } } }
        }
      }
    }
  }
}

Aggregate when you can

An aggregated dataset over a billion-document index returns a few thousand rows. Reading raw documents to sum them client-side is the single most common cause of slow Elasticsearch datasets.

Pagination and sync

Chartizer paginates with point-in-time plus search_after, which is stable across a sync even as documents are written.

Setting Default Notes
Page size 1,000 Lower it if the cluster is memory-constrained
PIT keep-alive 5m Extended automatically for long syncs
Incremental field none Set a date field to sync only new documents

Setting an incremental field turns a full re-read into a delta:

JSON
{ "incremental": { "field": "created_at", "since": "{{last_sync}}" } }

Deletes are invisible to incremental syncs

An incremental sync sees new and updated documents, never deletions. Schedule a periodic full refresh if documents are deleted upstream.

Troubleshooting

Zero rows but the index has documents

The query matched nothing, or the alias resolves to an empty index. Test the same query with _count before blaming the connector.

Fields missing from the dataset

They were absent from the sampled documents, or excluded by the fields list. Elasticsearch has no fixed schema, so Chartizer maps what it sees on the first sync.

search_phase_execution_exception on sync

Usually sorting or aggregating on an analysed text field with fielddata disabled. Use the .keyword sub-field.

Sync slows down over time

The index grew and the dataset reads raw documents. Add an incremental field, or switch to an aggregation.