Skip to content
SCHEMAVORTEX
Home

Questions

Frequently asked questions

What SchemaVortex is, where your data lives, who runs it, and what happens if you stop using it.

The platform

What is SchemaVortex?

SchemaVortex is a data lakehouse platform that extracts data from operational databases, keeps a versioned history of it, and serves the result as governed SQL views. It runs inside the customer's own Azure subscription rather than as a service data is sent to, and it is configured through a browser instead of by writing pipeline code.

What is SchemaVortex not?

It is not a reporting or visualisation tool, and it does not replace the source systems it reads. It prepares governed, historised data and serves it over standard SQL; the charts and dashboards are still built in Power BI, Excel or whatever the business already uses.

Who operates SchemaVortex once it is set up?

The customer's own team. The platform is self-managing and runs in the customer's own Azure subscription, and after handover no standing credentials are held in that tenant. If a support engineer ever needs access, it is through a temporary, scoped account the customer creates and revokes.

How is this different from building the pipelines ourselves?

Hand-built ingestion means writing and then maintaining extraction, historisation, schema-change handling, access control, lineage and a catalog as separate pieces of work, usually with specialist engineers. SchemaVortex ships those as one platform: an Origin is registered, its tables are discovered automatically, and what comes out is already historised, governed and traceable.

Data and storage

Where does our data physically live?

In the customer's own Azure subscription. Extraction reads the sources in place, the Vault is written to the customer's own Azure Data Lake Storage, and serving runs in the same subscription, billed to the customer directly by Microsoft. There is no vendor cloud for the data to travel to.

What format is the data stored in?

Apache Parquet. The Vault is a set of Parquet files in the customer's own Azure Data Lake Storage, readable by Power BI, Excel, Spark and any other Parquet-aware engine, with or without SchemaVortex in the picture.

How is the data queried?

Through Mart views, which are Azure Synapse Analytics serverless SQL views over the Vault. Anything that speaks SQL can read them: Power BI, Excel, Tableau, Qlik, Python or a plain SQL client. Serverless means the query engine is billed per query, with no cluster to size or keep running.

Is historical data kept, and can we query the past?

Yes. Versions are added rather than overwritten, so a record's earlier states stay available for as long as you choose to keep them. Each Vault table is exposed both as a Latest table, holding the current state for everyday reporting, and as a History table for audits and point-in-time questions. A faulty Extraction can be rolled back without losing the history you retain.

How long is data kept?

For as long as you choose to keep it. The current state of every table stays available; how far back the history behind it reaches is a retention setting you control. Keeping personal data no longer than necessary is a GDPR requirement, which is why the depth of history is a setting rather than a fixed promise.

What happens to our data if we stop using SchemaVortex?

It stays exactly where it already is. The Parquet files sit in storage the customer owns, in an open format, and remain readable by any Parquet-aware tool. There is no proprietary catalog holding the data and nothing has to be exported or migrated out.

Connecting sources

Which source systems can be connected?

SQL databases are extracted directly, including SQL Server, PostgreSQL and IBM AS/400 (DB2 for i), along with the ERP, CRM and line-of-business systems built on them. Files, REST APIs and anything without a built-in connector are pushed in through the Producer SDK.

Do we have to map the schema by hand?

No. Registering an Origin makes SchemaVortex discover it: the tables, the columns, their types and their keys are read from the source itself. From there it is a matter of selecting which Source tables to track and choosing an Extraction strategy for each.

What happens when a source table changes shape?

The Vault's column set only ever grows. When a source adds a column or changes a column's type, the new shape is stored as a new versioned column and the original stays exactly as it was, simply no longer loading. Records from before a new column existed show it as empty. Because existing columns are never altered or removed, the columns a report already reads do not change underneath it.

Governance and traceability

How is access to sensitive data controlled?

Every column passes five gates before anyone can see it, and three stewards hold distinct keys over them, so no single role sees ungoverned data. Masking is applied as the query runs, which means whoever queries gets the view masked for them and there is no sanitised second copy to build and keep in sync. Every approval, classification and mask change is recorded with who changed what and when.

What does lineage cover?

Views and columns. Any column in a Mart view can be traced back through every view to the source column behind it, and the same links run forward, so the Mart views a change would affect can be listed before the change is made. Because the platform resolves every view itself, lineage is read from what is actually running rather than from documentation someone maintains.

Is a separate data catalog needed?

No, the catalog is part of the platform. One schema browser spans the Vault and the Mart, there is a built-in SQL editor for writing and editing views, lineage opens from any table or column, and any object can be annotated and tagged. Nothing has to be deployed alongside the lake or rescanned on a schedule.

Does the AI assistant see our data?

No. It reads structure, not contents: the schema and the metadata, never the records themselves. It runs inside the customer's own Azure subscription, nothing is sent to an outside service, and every interaction is logged under the same audit trail as the rest of the platform.