Overview

Cortex provides a pluggable storage layer for SQL databases while keeping the public API stable. The entry point is CortexStorage, which exposes get_session() for SQLAlchemy sessions and client for a simple query API. Under the hood, storage delegates to connector clients created by a small factory service.

• cortex.core.storage.store.CortexStorage: main storage facade used across the app
• cortex.core.connectors.databases.clients.service.DBClientService: builds a database client from configuration
• SQL connectors that currently back storage:
  • CommonProtocolSQLClient for network databases (PostgreSQL, MySQL)
  • SQLiteClient for file-based or in-memory SQLite

Architecture

graph TD
    A[Application Code] -->|get_session()/client| B[CortexStorage]
    B --> C[DBClientService]
    C --> D[CommonProtocolSQLClient (PG/MySQL)]
    C --> E[SQLiteClient]
    D --> F[SQLAlchemy Engine/Connection]
    E --> F

Key modules

• cortex/core/storage/store.py
  • Reads environment (CORTEX_DB_*) and selects the backend
  • Provides get_session() returning a SQLAlchemy session
  • Exposes .client for a thin query API (query, fetch_all, fetch_one, stream)
  • Base points to cortex.core.storage.sqlalchemy.BaseDBModel for migrations
• cortex/core/storage/sqlalchemy.py
  • Defines BaseDBModel and a Postgres-friendly naming convention for constraints/indexes
• cortex/core/connectors/databases/clients/service.py
  • Factory that returns the right client based on the resolved DataSourceTypes
• cortex/core/connectors/databases/clients/SQL/common_protocol.py
  • CommonProtocolSQLClient (PostgreSQL/MySQL via SQLAlchemy): engine creation, pooling, session factory, query helpers
• cortex/core/connectors/databases/clients/SQL/sqlite.py
  • SQLiteClient (SQLite via SQLAlchemy): engine creation with check_same_thread=False, PRAGMA setup, introspection

Supported storage backends

• PostgreSQL
• MySQL
• SQLite (file-based or in-memory)

Configure via environment variables

Set the following variables (all prefixed with CORTEX_) to select and configure the storage backend. The storage layer reads these via ExecutionEnv.get_key.

Common

• CORTEX_DB_TYPE: one of postgresql, mysql, sqlite

PostgreSQL/MySQL

• CORTEX_DB_HOST
• CORTEX_DB_PORT
• CORTEX_DB_USERNAME
• CORTEX_DB_PASSWORD
• CORTEX_DB_NAME

Example (PostgreSQL):

CORTEX_DB_TYPE=postgresql
CORTEX_DB_HOST=localhost
CORTEX_DB_PORT=5432
CORTEX_DB_USERNAME=postgres
CORTEX_DB_PASSWORD=postgres
CORTEX_DB_NAME=cortex

SQLite

For SQLite, configure either in-memory mode or a file path.

• CORTEX_DB_MEMORY: true or false
• CORTEX_DB_FILE: file path (used when memory is false)

Example (SQLite file):

CORTEX_DB_TYPE=sqlite
CORTEX_DB_MEMORY=false
CORTEX_DB_FILE=./cortexstore.db

Example (SQLite in-memory):

CORTEX_DB_TYPE=sqlite
CORTEX_DB_MEMORY=true

Using the storage layer

Obtain a SQLAlchemy session

from cortex.core.storage.store import CortexStorage

session = CortexStorage().get_session()
try:
    # use session with ORM or Core
    ...
    finally:
    session.close()

Use the connector client directly

from cortex.core.storage.store import CortexStorage

client = CortexStorage().client  # CommonProtocolSQLClient or SQLiteClient

# Fetch all rows as lists of mappings
rows = client.fetch_all("SELECT 1 AS one")

# Fetch single row
row = client.fetch_one("SELECT 42 AS answer")

# Stream in chunks
for chunk in client.stream("SELECT * FROM some_table", chunk_size=500):
    ...

Migrations and metadata

• Alembic uses CortexStorage().Base.metadata as the target metadata.
• For SQLite, Alembic is configured in batch mode in cortex/migrations/alembic/env.py so constraint-altering operations are supported via copy-and-recreate.
• File paths for SQLite are normalized to absolute paths during migrations.
• Migrations are bundled inside the cortex package at cortex/migrations/ for seamless deployment.