Skip to main content
The data topology controls how Neki routes queries and distributes data onto the underlying resources. The data topology document refers to Postgres objects (such as databases, schemas, tables, and sequences), but it doesn’t define their structure. You create those objects using regular Data Definition Language (DDL) statements, independently of naming them in your topology. This makes the topology language flexible and extensible while retaining Postgres compatibility for the DDL syntax. Neki routes queries and distributes data based on the current topology. You can view the topology document in the PlanetScale dashboard or using the __neki.get_data_topology metafunction. A new cluster starts with a topology stub naming the cluster’s initial shard:
In this state, the cluster appears similar to a plain, unsharded Postgres database with everything resolving to a single shard. This is a valid topology, and you can use Neki in a single-shard configuration without thinking about the topology document until your workload needs to grow beyond a single shard’s capacity.

Shards, topology, and logical objects

A Neki cluster contains shards and routers, a topology definition, and the logical database objects. The topology does not implicitly change when you add or remove shards or when you manipulate Postgres objects. The topology must be adjusted as necessary, using reshard workflows or the __neki.set_data_topology metafunction. For example:
  • Newly added shards don’t automatically appear in the topology. Introduce them through a reshard workflow (to redistribute an existing shard group) or by declaring a new shard group that uses them.
  • To delete a shard, first move remaining data off the shard and remove it from the topology. A shard can’t be deleted if it’s referenced in the topology document.
  • The authoritative shard group can’t be removed or migrated to a different shard, so the cluster’s initial shard can’t be deleted. Leave it as a standalone shard and size it for catalog work, sequence reservations, and any unsharded tables that stay on it. Use separate shards and shard groups for large application tables.
  • Newly added tables are resolved using topology defaults but don’t automatically receive dedicated topology entries.
Keep in mind that the __neki.set_data_topology metafunction called with the overwrite => true option replaces the entire current topology document with a new one. Neki validates the new document for syntax and certain structural issues, but it doesn’t check if the logical objects exist or if their structure aligns with the topology, and it doesn’t redistribute existing data to match the new document. The ability to replace a topology document is useful for making additive or subtractive changes that don’t break the existing logic: removing parts of the document you no longer need, declaring routing rules for new tables, or setting up the topology ahead of creating Postgres objects on a new cluster. To move existing data around, use a reshard workflow.

Creating and removing logical objects

Neki doesn’t prevent you from creating or removing logical Postgres objects of your own accord. These operations fall into three main categories:
  1. Changes that need a topology adjustment, such as creating a new reference table or a sharded table that overrides the default shard index.
  2. Changes that are covered by the existing topology without adjustment, such as creating a sharded table that follows existing topology defaults.
  3. Changes that don’t involve Neki topology at all, such as creating a foreign key constraint or changing the data type for a secondary column.
As a best practice, if an object creation or removal also requires a topology change, adjust the topology before manipulating the object. For example, if you plan to add a table that requires a new topology entry (routing rule, index override, etc.), updating the topology before creating the table eliminates the risk of traffic reaching the table through topology defaults before its routing is customized. For objects that the topology document doesn’t deal with, Neki maintains them on every shard, but they’re used independently by each shard as if it were a separate Postgres instance. For example, a statement creating a foreign key constraint fans out to all shards but is enforced locally and separately by each shard.

Familiar terms

Some terms familiar from Postgres have a different meaning in Neki.

Shard index

A shard index turns values from one or more table columns into routing values. Neki compares those routing values with a shard group’s key ranges to choose a destination shard. Naming a column in the shard index doesn’t create a Postgres index on that column and doesn’t require that column to be indexed. An index can still be necessary for optimal performance.

Primary index

A primary index is a per-table override of the shard index that the table is routed by. Without an override, the table is routed by its shard group’s default shard index. The per-table override is useful when different sharded tables contain columns that are named differently but contain the same logical data. For example, a data model for an event booking application might contain multiple tables that will be sharded by venue:
Columns storing the venue ID contain the same data but are named differently. It’s a good practice to use defaults for common properties and overrides for exceptions, so in this example the topology could declare a default shard index on venue_id and an override for public.venues to use the id column. A primary index declared in a Neki topology is distinct and separate from a primary index used as a constraint on the underlying Postgres table. The column named in a topology override doesn’t have to be a PRIMARY KEY or a UNIQUE NOT NULL constraint at the Postgres level.

Global secondary index

A global secondary index, or GSI, uses a lookup table to map another key to the shard key of an owner row. Neki can read this mapping before routing a query to the owner table. Declaring a GSI in the topology document does not create a single cluster-wide Postgres index object or a unique constraint enforced across all shards. Indexes and constraints are maintained and enforced independently on each shard, but it’s possible to implement global uniqueness using one of two techniques: an unsharded GSI (lookup table) or a composite index prefixed with a shard key. For the complete list of terms, see Neki terminology.

Need help?

Get help from the PlanetScale Support team, or join our Discord community to see how others are using PlanetScale.