The great thing about shipping structured data as text files is it is so darn easy! Just store it as a CSV and boom: minimum overhead and just the right amount of structure to not risking to be called out a liar when saying that the data is structured.

But it does not come entirely without risk. There's zero context to this data so you're leaving the receiving end with a lot of guesswork. Does this column contain text? Integers or floats? Are there alphanumeric keys in this column aptly named 'id'? Perhaps I should parse everything as text just to be safe?

I spent a hefty portion of two working days, struggling to parse tab separated data from MySQL dumps. There were errors. ClickHouse couldn't parse these files because it expected tabs where there apparently was none. However, I could easily spot the tabs in the raw files and without difficulty read the same data into a pandas DataFrame. I even did a manual count just to make sure that the data wasn't dirty.

And then I spotted it! Not in the files, no. And not in the error message, no absolutely not! It was over on ClickHouses excellent documentation pages. There I found the setting:

input_format_tsv_use_best_effort_in_schema_inference=1

It might be fun, at first. Cool, some might say. Leaving the kitchen door wide open to ghouls, I would argue. In my case, a column that had its schema inferred as Int64 suddenly contained alphanumeric caracters.

The lesson I take with me from this experience is to keep telling the parser exactly what data types to expect. You configure this as a long string after telling the enginge what format to parse. Yes, the parameters for the s3 table engine will be a short novel, but at least the explicitly typed out columns and data types serves as a minimum data contract. Those lines of code in dbt, version controlled in all its glory will be the 1:st line of contracts, describing what type of data is expected in that pipeline.

In QlikView this was a fairly common practice at the time since many data sources came in spreadsheet format. There was a function named CrossTable that took care of that effortlessly.

In ClickHouse the same can be achieved by grouping the year columns into an array and then doing an ARRAY JOIN which will produce a new table by iterating through the array creating a separate record for each item.

Let's pretend that we have a copy of The Economist's Democracy Index list by country ingested into ClickHouse. It would look something like this when queried:

To get this table into a transactional state where we can perform aggregation functions like Min(), Max() and Avg() we can create a dbt model like this:

{{
    config(
        materialized='table', 
        engine='MergeTree()', 
        order_by='(country, year)'
    )
}}

WITH pivoted AS (

    SELECT
        Region AS region,
        Country AS country,
        `Regime type` AS regime_type,
        year,
        democracy_index
    FROM ingest.democracy_index_by_country_2022
    LEFT ARRAY JOIN
        splitByString(',', '2022,2021,2020') AS year,
        [`2022`,`2021`,`2020`] AS democracy_index

), final AS (

    SELECT
        toLowCardinality(region) AS region,
        toLowCardinality(assumeNotNull(country)) AS country,
        toLowCardinality(regime_type) AS regime_type,
        toUInt16(year) AS year,
        round(democracy_index, 2) as democracy_index
    FROM pivoted

)
SELECT * FROM final

There is a lot of stuff going on in here! The double curly brackets at the top tell dbt that I want the result of this SQL query to be materialized as a new table in the database. Where and how this table is created is defined by how we've set up our dbt project.

I always recommend typing out SQL in a CTE (Common Table Expression) format. It is so much easier to read by both myself a week from now and by my colleagues.

The real magic happens in the LEFT ARRAY JOIN clause. The first statement after is a trick to create an array of year names by splitting a string into individual values by the splitByString() function. In the second statement, we treat all our year columns (escaped by backtick since column names are only numbers) as an array. When LEFT ARRAY JOIN is executed the year and democracy_index columns are created by iterating through the two arrays.

The final bit is some best practices to get the data in a healthy shape before running analytic queries. Since sorting columns shouldn't contain NULL and my table will be sorted by country and year we apply assumeNotNull() to the country column.

dbt source freshness

to test that my ingestion pipelines are delivering. However, when running dbt Core the output is just a JSON file placed in the target folder at the root level of your project. Since I’m running my dbt jobs with GitHub Actions that file is lost as soon as the job is finished.

Unless I HTTP-post the JSON data to ClickHouse, that is…

So I’ll start by creating a table that will hold parts of the sources.json data that I want to analyze and take action on as they happen. The JSON file has a structure looking something like this:

{
  "metadata": {
    "dbt_schema_version": "https://schemas.getdbt.com/dbt/sources/v3.json",
    "dbt_version": "1.4.6",
    "generated_at": "2023-06-27T08:31:30.583005Z",
    ....
  },
  "results": [
    {
      "unique_id": "arbitrary.model.name",
      "max_loaded_at": "2023-03-31T11:36:29.701000+00:00",
      "max_loaded_at_time_ago_in_s": 7599300.299,
      "status": "error",
      ...
    },
    ...
  ],
  ...
}

I’d like to recreate this structure in a table in ClickHouse so that I can do an HTTP-post using cURL with the contents from sources.json in my payload, with zero transformations. I will create a column for metadata that will be a named tuple and a column for result that will be an array of named tuples. And finish off with a timestamp for ingestion time. But first I need to shut off a ClickHouse-feature that flattens tuples. The docs only mention this for the Nested data type but this is also necessary for the Tuple data type.

SET flatten_nested=0; 

CREATE TABLE ingest.dbt_source_freshness (
     metadata Tuple(dbt_schema_version String, dbt_version String, generated_at String),
     results Array(Tuple(unique_id String, status String, max_loaded_at_time_ago_in_s Float64)),
     inserted_utc DateTime64(3) DEFAULT now64(3, 'UTC')
)
ENGINE = MergeTree()
ORDER BY (inserted_utc,)
PARTITION BY toYYYYMM(inserted_utc);

Now I’m able to post the results of my dbt freshness test directly into this table by a cURL call similar to this

curl -X POST -H "Content-Type: application/json" -H "X-ClickHouse-User: username" -H "X-ClickHouse-Key: password" -d @./target/sources.json https://my.clickhouse.server:8443/?query=INSERT%20INTO%20ingest.dbt_source_freshness%20FORMAT%20JSONEachRow