646 signals · 904 observation events
ClickHouse® is a real-time analytics database management system
v26.7.3.19-stableOpen the original release notes for details.
Original release notes ↗Prereleases and milestones indicate public plans; they are not delivery commitments.
This issue contains tasks for further improvements of new [JSON](https://github.com/ClickHouse/ClickHouse/pull/66444) data type. Continuation of https://github.com/ClickHouse/ClickHouse/issues/54864. Important tasks: - [x] Reduce memory usage during insertion into JSON column and during merges by better buffer size managing - https://github.com/ClickHouse/Cl
A Cascades-style cost-based optimizer that chooses distribution strategies for the multi-stage distributed query plans of #106020. It explores alternatives in a memo (a shared store of equivalent plan fragments) with top-down, goal-directed search and picks the cheapest plan satisfying the required distribution and sorting properties, inserting exchange oper
Implement the deserialized columns cache that keeps columns in memory. In comparison to the uncompressed cache, this saves time on deserialization and copying. Closes #82335. The cache identifies entries by table UUID, so it is only active for tables in databases that assign UUIDs (`Atomic` and `Replicated`); tables in `Ordinary` databases are silently exclu
This pull request bundles two query-execution changes: - `count_distinct_optimization` is enabled by default. The rewrite is applied only by the analyzer's `CountDistinctPass`, which skips `Nullable` / `LowCardinality(Nullable)` arguments and refuses to fire for remote storages or with `distributed_group_by_no_merge`. The legacy AST-level `RewriteCountDistin
When a recursive CTE joins against a `MergeTree` table, each recursion step previously scanned the entire table, because the join condition (e.g. `ON e.from_id = t.current_id`) was not pushed into `MergeTree`'s key condition. This change analyzes the recursive query's join tree to find equi-join conditions between the CTE working table and real tables. Befor
When a single MergeTree part is split into multiple streams for read-in-order, use the new `PrefetchingConcatProcessor` instead of relying on the downstream `MergingSortedTransform`. The key difference: - `MergingSortedTransform` marks all inputs as needed but only consumes from one stream at a time (for non-overlapping ranges), and each port can only buffer
Switch the default compression in ClickHouse from `LZ4` to `ZSTD(3)`, for both on-disk table data and network communication. ### Motivation `LZ4` has been the default for a long time and optimizes for speed, but `ZSTD(3)` gives substantially better compression ratios at a still-modest CPU cost, reducing both storage footprint and network traffic out of the b
Resolves https://github.com/ClickHouse/ClickHouse/issues/48165 Adds a new `join_algorithm` value `parallel_full_sorting_merge`. `full_sorting_merge` streams both sides and joins them with a single merge, so it keeps memory bounded but runs the merge on one thread — on many-core machines it is often slower than `parallel_hash` even though it uses far less mem
### Changelog category (leave one): - Performance Improvement ### Changelog entry (a [user-readable short description](https://github.com/ClickHouse/ClickHouse/blob/master/docs/changelog_entry_guidelines.md) of the changes that goes into CHANGELOG.md): Optimize `GROUP BY ... ORDER BY ... LIMIT` and `GROUP BY ... LIMIT` queries by maintaining a bounded heap d
When the WHERE clause cannot use the primary key effectively (e.g., `LIKE '%...'`), the `optimize_read_in_order` optimization kills parallelism: each part is read by a single stream instead of many. This can make `ORDER BY` queries 4x slower than the same query without `ORDER BY`. Example on a 2.5B row table with `ORDER BY path` and `WHERE path LIKE '%stderr
### Changelog category (leave one): - Performance Improvement ### Changelog entry (a [user-readable short description](https://github.com/ClickHouse/ClickHouse/blob/master/docs/changelog_entry_guidelines.md) of the changes that goes into CHANGELOG.md): Improve the performance of queries that are reading in the order of the primary key. Based on #100391.
Reopen of https://github.com/ClickHouse/ClickHouse/pull/103919. This closes #103839. Adds a new `MergeTree` setting `optimize_row_order_if_no_order_by` (default `1`) that enables `optimize_row_order` automatically for tables without an explicit `ORDER BY`. **Behavior change and migration path.** With an empty sorting key (`ORDER BY ()` / `ORDER BY tuple()`)
Cost-based join reordering relies on column statistics (in particular the number of distinct values of join keys) to avoid pathological plans. Column statistics are auto-declared by default (`auto_statistics_types = 'minmax, uniq'`), but they are only materialized during merges (`materialize_statistics_on_merge`) — so a freshly bulk-loaded table has no usabl
<!--- Disable AI PR formatting assistant: true --> ### Changelog category (leave one): - Performance Improvement ### Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md): MergeTree tables will have `add_minmax_index_for_numeric_columns` by default. Resolves #70605 ### Details Resolves #70605 <!-- CURSOR_SUMMARY --> -
Closes: https://github.com/ClickHouse/ClickHouse/issues/52624 When reading in order of the primary key over many parts (the `optimize_read_in_order` optimization), the merging pipeline opens one reader per part to feed the `MergingSortedTransform`, and keeps a `CompressedReadBuffer` per column resident for every one of them. With a large number of parts this
`ALTER TABLE ... MODIFY TTL` currently rewrites every part of the table, which on a large table means reading and writing all of its data just to change when rows expire. Very often the new TTL is the old one shifted in time - the retention period is extended or shortened, e.g. `create_time + INTERVAL 300 DAY` becomes `create_time + INTERVAL 10 DAY`. In that
Parses the data of one batch of asynchronous inserts with several threads when the batch is flushed, controlled by the new setting `async_insert_parse_threads` (`0` by default, which keeps the previous behaviour). Addresses https://github.com/ClickHouse/ClickHouse/issues/74162 ### Motivation With `wait_for_async_insert = 1` the client waits for the whole bat
When ClickHouse processes GROUP BY, it often overestimates the number of threads needed. With `max_threads = 64` but only a few thousand rows, all 64 `AggregatingTransform` instances get data, produce 64 partial hash tables, and the merge phase has to combine all of them — most nearly empty. This wastes time on merging overhead, which is especially noticeabl
### Changelog category: - Performance Improvement ### Changelog entry (a [user-readable short description](https://github.com/ClickHouse/ClickHouse/blob/master/docs/changelog_entry_guidelines.md) of the changes that goes into CHANGELOG.md): In case of trivial views over distributed table push whole outer query to shards. ### Documentation entry for user-faci
Adds the `groupBloomFilter` aggregate function and the `bloomFilterContains` scalar function for memory-efficient probabilistic set-membership testing. This can be used to detect new values, perform approximate deduplication checks, and compare large datasets without materializing exact sets. Related: https://github.com/ClickHouse/ClickHouse/issues/11700 ###
This PR closes https://github.com/ClickHouse/ClickHouse/issues/92266 Support column matchers in column `DEFAULT`, `ALIAS`, `MATERIALIZED`, and `EPHEMERAL` expressions, and in data skipping index expressions. This allows expressions such as `*`, `COLUMNS('...')`, `COLUMNS(a, b)`, `EXCEPT`, `APPLY`, and `REPLACE` to be expanded before expression validation and
### Changelog category (leave one): - Improvement ### Changelog entry (a [user-readable short description](https://github.com/ClickHouse/ClickHouse/blob/master/docs/changelog_entry_guidelines.md) of the changes that goes into CHANGELOG.md): TODO ### Documentation entry for user-facing changes - [ ] Documentation is written (mandatory for new features) <!---
Add `reattach_tables_before_query_execution` and `reattach_tables_before_query_execution_probability` settings that enable randomly detaching and reattaching tables used in a query before its execution. This is a testing-only feature designed to find bugs related to table reattachment. Before executing a query, the system collects all tables referenced in th
REST catalog inline responses already contain the table UUID and metadata location. Propagate the UUID through DataLakeSpecificProperties -> StorageObjectStorageConfiguration.catalog_uuid_hint -> initializePersistentTableComponents so the metadata cache is checked before fetching metadata.json from network storage. ### Performance mechanism **Before:** On fi
<!-- Linked issues and pull requests. Use full GitHub URLs, one relationship per line; delete the lines you don't need. Closes: https://github.com/ClickHouse/ClickHouse/issues/NNNNN (auto-closes the issue when this PR is merged into the default branch) Related: https://github.com/ClickHouse/ClickHouse/pull/NNNNN --> <img width="2938" height="1140" alt="image
Closes: https://github.com/ClickHouse/ClickHouse/issues/39067 ### Changelog category (leave one): - New Feature ### Changelog entry (a [user-readable short description](https://github.com/ClickHouse/ClickHouse/blob/master/docs/changelog_entry_guidelines.md) of the changes that goes into CHANGELOG.md): Add table function `obfuscate` which applies the same tra
<!--- Disable AI PR formatting assistant: false --> ### Changelog category (leave one): - New Feature ### Changelog entry (a [user-readable short description](https://github.com/ClickHouse/ClickHouse/blob/master/docs/changelog_entry_guidelines.md) of the changes that goes into CHANGELOG.md): Added a `chinese` tokenizer for the `tokens` function and `MergeTre
### Company or project name ClickHouse Inc. Found while comparing two server versions on an internal test cluster, on a reporting workload that derives its date window from a joined single-row subquery. ### Describe the situation When a range predicate on a `MergeTree` primary/partition key column is expressed in the `ON` clause of a `JOIN` against a single-
This PR introduces the parquet Shredded VARIANT standard to CLickHouse. duckdb has this already and the main benefit is that it can read less bytes from disk. This is of course critical for a main parquet use case, which is remote reads from S3. Benchmark results for JSONBench reading from file() on NVME on the final binary are below. | query | method | wall
### Changelog category (leave one): - Backward Incompatible Change ### Changelog entry (a [user-readable short description](https://github.com/ClickHouse/ClickHouse/blob/master/docs/changelog_entry_guidelines.md) of the changes that goes into CHANGELOG.md): Add `INSERT ... RETURNING (SELECT ...)` to run a user-supplied SELECT after INSERT and return its resu
Adds a query plan optimization that pushes volume-reducing functions (`length`, `lengthUTF8`, `empty`, `notEmpty`) below the `Sorting` and `Filter` steps, so those steps carry the fixed-size result instead of the original `String` / `FixedString` / `Array` / `Map` argument. The rewrite is done with `ActionsDAG::split`: the function nodes form the first part,
Implements [#100000](https://github.com/ClickHouse/ClickHouse/issues/100000): create and manage custom HTTP handlers from SQL, without editing the server configuration file. New SQL statements: - `CREATE HANDLER [IF NOT EXISTS] name [PROTOCOL p] URL [PREFIX|REGEXP] '/x' [METHODS (GET, POST)] [TYPE query] AS <query>` - `ALTER HANDLER name ...` — partial updat
Reintroduce #108988 Related: https://github.com/ClickHouse/ClickHouse/pull/107030 Related: https://github.com/ClickHouse/ClickHouse/pull/108577 CI: https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=105890&sha=3dc0e76362eb18e27f1fffcd3f61ca9f13725fe8&name_0=PR&name_1=Stateless%20tests%20%28amd_tsan%2C%20s3%20storage%2C%20sequential%2C%201%2F2%29 F
Adds TLS/SSL support to all PostgreSQL integrations. Until now, ClickHouse could not establish an encrypted connection to a PostgreSQL server that enforces SSL, nor verify the server certificate: `postgres::formatConnectionString` only emitted `dbname`/`host`/`port`/`user`/`password`/`connect_timeout`, so `libpq`'s `sslmode`/`sslrootcert`/`sslcert`/`sslkey`
### Changelog category (leave one): - Documentation (changelog entry is not required)
<!--- A technical comment, you are free to remove or leave it as it is when PR is created The following categories are used in the next scripts, update them accordingly utils/changelog/changelog.py tests/ci/cancel_and_rerun_workflow_lambda/app.py --> ### Changelog category (leave one): - Experimental Feature ### Changelog entry (a [user-readable short descri
Closes: https://github.com/ClickHouse/ClickHouse/issues/75538 Related: https://github.com/ClickHouse/ClickHouse/issues/92455 ### Changelog category (leave one): - Performance Improvement ### Changelog entry (a [user-readable short description](https://github.com/ClickHouse/ClickHouse/blob/master/docs/changelog_entry_guidelines.md) of the changes that goes in
`randomHadamardTransform` did not use the default implementation for constant arguments, so it called `convertToFullColumnIfConst` on its first argument and ran the transform for every row of the block even when the vector was a constant. It also returned a plain `ColumnArray` rather than a `ColumnConst`, so the analyzer could not fold the call to a literal
<!-- Related: https://github.com/ClickHouse/ClickHouse/issues/2120 --> `arrayFilter` builds an array of the elements that pass the predicate, and `length` then throws that array away and keeps only its size. `arrayCount` computes the same number without materializing anything. ```sql SELECT length(arrayFilter(x -> (x >= 2), arr)); -- becomes SELECT arrayCoun
<!-- Closes: https://github.com/ClickHouse/ClickHouse/issues/112575 --> ### Changelog category (leave one): - Performance Improvement ### Changelog entry (a [user-readable short description](https://github.com/ClickHouse/ClickHouse/blob/master/docs/changelog_entry_guidelines.md) of the changes that goes into CHANGELOG.md): Filter pushdown now works for `Tupl
<!--- A technical comment, you are free to remove or leave it as it is when PR is created The following categories are used in the next scripts, update them accordingly utils/changelog/changelog.py tests/ci/cancel_and_rerun_workflow_lambda/app.py --> ### Changelog category (leave one): - Improvement ### Changelog entry (a user-readable short description of t
Add leader election for non-replicated MergeTree tables on shared object storage (currently `S3`; `Azure` is implemented but not yet enabled, pending test coverage), enabling active/standby failover without external coordination (no Keeper). Uses conditional writes (`If-Match` / `If-None-Match`) on object storage to maintain a lease file with JSON content `{
### What this PR does Two related knobs for writing Parquet files with column `field_id` metadata — needed for Apache Iceberg compatibility, which identifies columns by `field_id` rather than by name. Both knobs can be used independently or together. #### 1. Explicit per-column overrides New setting `output_format_parquet_column_field_ids : Map(String, Int32
### Changelog category (leave one): - Bug Fix (user-visible misbehavior in an official stable release) ### Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md): Fix `LOGICAL_ERROR` exceptions when reading `Iceberg` or `DeltaLake` data lake tables through paths that can reach the read pipeline without a pinned `datalake
Related: https://github.com/ClickHouse/ClickHouse/pull/107924 `FutureSetFromSubquery::buildOrderedSetInplace` (the speculative set build run during primary key analysis for `IN` subqueries) used to consume the subquery `source` plan up front. If the in-place build then failed silently (e.g. due to subquery timeout with `timeout_overflow_mode = 'break'`, wher
A correlated subquery references an outer column that is a GROUP BY key of an outer query under `group_by_use_nulls = 1` with `WITH ROLLUP`/`CUBE`/`GROUPING SETS`. The actual values fed into the inner subquery are post-rollup `Nullable`s, but the analyzer used to leave the inner column reference at its non-Nullable type. The planner then built CAST wrappers
Reverts ClickHouse/ClickHouse#104359
Implements [#46925](https://github.com/ClickHouse/ClickHouse/issues/46925) according to the [updated spec](https://github.com/ClickHouse/ClickHouse/issues/46925#issuecomment-4475417259). Closes: https://github.com/ClickHouse/ClickHouse/issues/46925 Related: https://github.com/ClickHouse/clickhouse-docs/pull/6398 ## What's in the box - **Path → database/table
Closes: https://github.com/ClickHouse/ClickHouse/issues/108713 Adds a way to detect when the data behind a query has changed, and uses it to make the query cache consistent and to skip unnecessary refreshes of materialized views. A new virtual method `getModificationHash` on `IStorage` returns an optional `UInt128` that changes whenever the data behind the t
<!-- Linked issues and pull requests. Use full GitHub URLs, one relationship per line; delete the lines you don't need. Closes: https://github.com/ClickHouse/ClickHouse/issues/NNNNN (auto-closes the issue when this PR is merged into the default branch) Related: https://github.com/ClickHouse/ClickHouse/pull/NNNNN --> Related: https://github.com/ClickHouse/Cli