Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
43.5K stars7.9K forksPythonApache-2.0data-sciencedeep-learningdeploymentdistributedhyperparameter-optimizationhyperparameter-searchlarge-language-modelsllmkeyword: Ray distributed computingkeyword: Ray framework
Ray Data: In this Ray release, we've enabled DataSourceV2 by default (#64821), so readparquet and friends use the new scan/listing infrastructure with row-group-aware chunking and predicate splitting. Hash Shuffle V2 eliminates the aggregat
HashShuffleV2 supports join (#63598, #64538, #64687). This lets shuffles reuse standard map/reduce scheduling, backpressure, and resource accounting.
Ray Serve: The HAProxy ingress is now distributed as the ray-haproxy PyPI package instead of being compiled into images, and it is the default HAProxy binary (#64141, #64163, #64164). We've also added gRPC support to the HAProxy direct-ingr
Ray Core: We've added an embedded RocksDB storage backend for GCS fault tolerance (REP-64), selectable with RAYgcsstorage=rocksdb and RAYgcsstoragepath (#63657). GCS fault tolerance no longer requires an external Redis instance. We've also
### What happened + What you expected to happen Using PyArrow fs with HDFS works fine outside a ray session: ``` file_sys, file_path = pyarrow.fs.FileSystem.from_uri(hdfs_folder) file_infos = file_sys.get_file_info(pyarrow.fs.FileSelector(file_path, recursive=False)) ``` However, after `ray.init()`, the same code results in a segmentation fault: ``` 2023-06-
### Description This is an umbrella issue to track Ray sandboxing support using gVisor. PRs: - [x] https://github.com/ray-project/ray/pull/64964 TODO: - [ ] add support for referencing local tar files for images (specifically with tar files created with `docker save`) - [ ] add modal compatible API for sandboxing - [ ] root cause why --ignore-cgroups is requ
### What happened + What you expected to happen I'm doing batch LLM inference using Ray Data, and I've noticed for large batches of prompts, the CPU autoscaler will continuously autoscale until it reaches maximum specified amount of CPU worker nodes as defined in my configuration file, and then it will start launching extra GPU instances even though I specif
### What happened + What you expected to happen **Description** Just as the title suggests, when initially starting up the Ray dashboard I can parse my local file system by supplying the `folder` HTTP parameter in GET requests to the `/logs` API endpoint in conjunction with relative file paths. **Expected Behavior** As a best practice, relative file paths sh
### Description Ray Serve LLM currently exposes an OpenAI-compatible API (chat completions, completions, embeddings) with no built-in mechanism for governance, guardrails, or policy enforcement at the serving layer. Teams deploying LLMs in production — especially in regulated industries (healthcare, finance, government) — need to add PII detection, cost budg
### Problem `WebDatasetDatasource._read_stream` yields one single-row pandas DataFrame per sample. Ray's output buffer estimates block memory on every emitted block: ``` BlockOutputBuffer.has_next -> _exceeded_buffer_size_limit -> get_estimated_memory_usage -> PandasBlockAccessor.size_bytes ``` So with N samples per shard, this deep `size_bytes` pass runs ~N
### What happened + What you expected to happen Ray Data's fair-share allocator uses a task-pool operator's configured maximum concurrency as its maximum resource demand. When no maximum is configured, the reported demand is unbounded even if the operator currently has only one input block and can only launch one task. This leaves resources assigned to an op
### What happened + What you expected to happen `StreamingRepartition(target_num_rows_per_block=1)` can spend a noticeable amount of time turning one large input block into many one-row output blocks. The row-sized path in `BlockOutputBuffer.next()` builds the buffered block, slices off the next output, and puts the remainder back into a new builder. It repe
### Description Ray Train v2 currently creates one detached `PlacementGroupCleaner` actor for each Train controller. The cleaner removes placement groups when its controller exits ungracefully. Since #64705, every cleaner is pinned to the head node so it does not prevent worker-node scale-down. However, this concentrates one Python cleaner process per concur
## Description This PR adds first-class support for Mobilint MBLT NPUs to Ray's accelerator abstraction by introducing a new `MBLTAcceleratorManager` under `ray/_private/accelerators/mblt.py`. This integration is developed and contributed by ETRI (Electronics and Telecommunications Research Institute) in collaboration with Mobilint. Based on the SDKs provide
### What happens With `log_to_driver=True` (the default), Ray prepends a prefix to every worker/actor stdout/stderr line it forwards to the driver: ``` (SegmentationActor pid=18654) {"event": "segmenting batch", "job_id": "…", "level": "info"} ``` Our workers emit **structured logs — one JSON object per line** — which a stdout-based collector ships to Loki/G
## Why Ray's OOM monitor kills tasks based on RAM alone, while the Linux OOM killer only fires when **both** RAM and swap are exhausted. On nodes provisioned with swap as overflow, Ray kills workers long before the kernel would. The scheduler's `memory` resource and the dashboard's Node Memory graph have the same RAM-only blind spot, and under `--enable-reso
## Why reopen this [#42587](https://github.com/ray-project/ray/issues/42587) asked for pre-commit hooks to run on `doc/`. It was closed as completed, but the configuration change it asked for was never made. I closed it, and that was my error. What actually happened: six PRs in Aug–Sep 2024 ([#47004](https://github.com/ray-project/ray/pull/47004), [#47005](h
## What happened + What you expected to happen Removing a placement group while bundle preparation is in flight can leave both prepared raylet resources and the GCS placement-group scheduling gate stuck if one prepare callback never arrives. The scheduler already documents that a `PrepareBundleResources` callback may never be called after node death. In this
### What happened + What you expected to happen A direct `ObjectRef` passed as an element of `fn_args` reaches the UDF as an `ObjectRef` instead of its resolved value. Because the reference is captured in the serialized UDF wrapper, Ray's scheduler also cannot wait for the dependency before scheduling the map task. Expected behavior: - Direct `ObjectRef` ele
## Description Support read orc format ## Related issues > Link related issues: "Fixes #1234", "Closes #1234", or "Related to #1234". ## Additional information > Optional: Add implementation details, API changes, usage examples, screenshots, etc.
## Description Adding user guide for mTLS configuration with RayClusters via KubeRay as per the new alpha feature in KubeRay 1.7 ## Related issues Relates to https://github.com/ray-project/kuberay/issues/5048 ## Additional information N/A
## Description Idle GCS↔Redis flows get silently removed by NAT, proxies, load balancers, and managed-service gateways. The GCS only finds out on its next command, which then hangs until a timeout instead of failing fast — and on a managed Redis behind an LB with a 60–350s idle timeout, this is a routine event, not an edge case. This PR enables TCP keepalive
## Why are these changes needed? The KubeRay CRD API reference — the field-level reference for `RayCluster`, `RayJob`, `RayService`, and `RayCronJob` — is published only on the MkDocs site at `ray-project.github.io/kuberay`. That site has two problems that compound each other: - Its `mkdocs.yml` `nav:` block contains only `index.md`. The API reference builds
### What happened + What you expected to happen ### Description Ray Serve's controller code is fully type-annotated but never type-checked. The mypy pre-commit hook covers an allowlist of exactly 7 files repo-wide — from all of Serve, only `python/ray/serve/handle.py` and a handle-typing fixture: # .pre-commit-config.yaml - id: mypy args: ['--follow-imports=
## Description `OpTask._cancel` never passes `force=True` to `ray.cancel`, on any path, so a blocked UDF keeps holding its CPU slot after execution fails. The actor-task check is the cause (`python/ray/data/_internal/execution/interfaces/physical_operator.py`): ```python is_actor_task = not self.get_task_id().actor_id().is_nil() ... force=force and not is_ac
`WorkerContext::GetGeneratorReturnId` (`src/ray/core_worker/context.cc`) documents that the caller supplies both `task_id` and `put_index` or neither, because deducing only one of them keys the ObjectID to one task while drawing the index from another. The `RAY_CHECK` meant to enforce that is a tautology: ```cpp RAY_CHECK((task_id.IsNil() && !put_index.has_v
### What happened + What you expected to happen When multiple raylets run in the same host network namespace with the same worker port range, each `WorkerPool` builds its free-port pool in ascending order and allocates from the lower bound. As a result, the raylets deterministically attempt to use the same initial ports, even when most of the configured rang
### What happened The RuntimeEnv Agent caches one logger per job in `_per_job_logger_cache` (`python/ray/_private/runtime_env/agent/runtime_env_agent.py:209`), created via `setup_component_logger` in `get_or_create_logger` (`:304-313`). Each logger eagerly opens one `RotatingFileHandler` for `runtime_env_setup-<job_id>.log` plus one per entry in the runtime
## What happened + What you expected to happen `_setup_logging()` in `python/ray/autoscaler/_private/kuberay/run_autoscaler.py` attaches a single `logging._StderrHandler` to the root logger, so **every** autoscaler log record — INFO included — is written to the container's **stderr**. The function's own docstring says it should be stdout: ```python def _setu
## Why Ray actors using RDT/NIXL over UCX don't control which RDMA NIC they use —UCX picks devices on its own. On multi-NIC nodes this means several actors can silently pile onto the same NIC while others sit idle, causing unpredictable tensor-transfer performance with no visible error. Closes/Addresses #64426. ## What Per discussion on the issue (thanks @Sp
Adds the file-transport runtime for a new external (on-disk) shuffle variant: TCP wire protocol, per-node ShuffleFileServer actor, connection / fetch primitives, and error classification. This PR lands only the runtime library and its testers required by external shuffle tasks. It's not wired into the plan yet and there's no user-visible behavior change. ##
## Description Two related fixes to the Vale setup. ### 1. The `vale` pre-commit hook has never run The top-level `exclude` in `.pre-commit-config.yaml` lists `doc/source/`, and pre-commit ANDs that global `exclude` with each hook's own `files` pattern. The Vale hook's pattern, `^doc/source/data/.*\.(md|rst)$`, lives entirely inside `doc/source/`, so the two
## Description Current behavior of `OpTask._cancel` never passes `force=True` to `ray.cancel`. It is because whether the task is an actor task is determined by checking if the actor id is nil, which is false for both normal tasks and actor tasks (refer to the issue). Slicing the hex method was withdrawn through discussions with maintainers. Instead, removed
Fixes #65416 ## Summary Multiple instance-manager records can refer to the same cloud instance after a Raylet restarts in place. The cloud-instance subscriber previously forwarded every record's ID to the provider, so KubeRay could decrement the same worker more than once while building a scale request and fail with an assertion before submitting the deletio
## Why are these changes needed? The KubeRay CRD API reference — the field-level reference for `RayCluster`, `RayJob`, `RayService`, and `RayCronJob` — is published only on the MkDocs site at `ray-project.github.io/kuberay`. That site has two problems that compound each other: - Its `mkdocs.yml` `nav:` block contains only `index.md`. The API reference builds
## Description Ray Train v2 currently creates one detached `PlacementGroupCleaner` actor per Train controller. Since cleaners are pinned to the head node, their memory use grows linearly with concurrent Train runs. In the workload described in #65443, 24 cleaners use approximately 14.4 GB on the head node. This PR replaces the per-controller actors with one
Integrates with https://github.com/ray-project/ray/pull/62741 to fix the long-standing issue in serve where `_to_object_ref` blocks until the underlying result has been produced.
## Why are these changes needed? Two interacting bugs in `ResourceBudgetBackpressurePolicy` can cause a Ray Data pipeline to stall indefinitely — CPU/GPU idle, Object Store filling up, no error logged, no progress. Reproducing this requires enough intermediate output to pressure the Object Store; small pipelines won't hit it. ### Background `ResourceBudgetBa
## Description Adds `ray.util.metric_registry.MetricRegistry` (DeveloperAPI): a thread-safe, get-or-create factory over the three custom-metric primitives (`Counter`/`Gauge`/`Histogram`). It owns the quirks every caller currently re-solves by hand: - **Dedup by name** — the same name returns a handle to the same underlying Ray metric, so callers don't manage
## Description Replaces the seven raw `ray.util.metrics` constructions in `HAProxyMetricsCollector` with handles from a shared, module-level `MetricRegistry(namespace="serve_haproxy")`. The registry get-or-creates by name, so re-constructing a collector in the same process (tests, actor restarts) reuses the existing Ray metric objects instead of re-creating
## Description Concurrent `ray up` processes can both observe the same generated EC2 key-pair name as available and then race in `CreateKeyPair`. The losing request receives `InvalidKeyPair.Duplicate`, which currently aborts AWS bootstrap even though the existing loop can safely try another generated name. Catch only `InvalidKeyPair.Duplicate` and continue w
## Description `CalculateRuntimeEnvHash` previously did `return static_cast<int>(hash)` on a 64-bit `size_t` hash. That drops the upper 32 bits and can produce a negative key. The value is used as a worker-pool reuse key, so a low-bit collision can silently run a task under the wrong runtime env. This folds with `hash ^ (hash >> 32)` and keeps a non-negative
## Description This PR adds an opt-in callback that detects hanging distributed training jobs by polling NCCL's RAS (Reliability/Availability/Serviceability) subsystem. NCCL ≥ 2.24 runs a monitoring thread inside every NCCL process that forms a peer mesh tracking per-rank health and collective op-counts. By querying the `ncclras` client and diffing collectiv
## Description Ray only lets you set node labels at startup, through `ray start --labels`, `RAY_OVERRIDE_LABELS`, or `--labels-file`. After a node is up, its labels are fixed for the life of the raylet. On a long-running cluster that is a poor fit: the labels you want on a node change over time, and the only way to change them today is to restart the node. W
## Description Makes three node/actor dashboard endpoints report client errors with 4xx status codes instead of 200/500. [#51417](https://github.com/ray-project/ray/pull/51417) added the `HTTPStatusCode` enum and the `rest_response(status_code=...)` plumbing, but adoption stalled after one endpoint. Across non-test dashboard source there are currently 28 `OK
### What is the problem this PR is trying to solve? This PR optimizes the metrics calculations in `RollingWindowMax` and `RollingWindowMin` by replacing the `O(num_buckets)` array scan with an amortized `O(1)` monotonic deque. It also fixes a correctness bug in the legacy implementation where the currently accumulating active bucket was ignored during `get_m
## Description `_TorchAwsNeuronXLABackend` cannot bring up a Neuron collective that spans instances. On two `trn1.32xlarge` hosts with 64 workers at one NeuronCore each, backend startup fails inside `_setup_xla_torch_process_group` with ``` Nrt::BuildGlobalComm failed on NeuronCores 0-1(2): nrt_status=1, message="Non specific failure". ``` The Neuron runtime
Allow ray.put(value, _tensor_transport='nixl') from a driver process. Previously this failed because get_runtime_context().current_actor raises RuntimeError in a non-actor context. Changes: - Make RDTMeta.src_actor and TransferMetadata.src_actor Optional - Catch RuntimeError in put_object and set src_actor=None for driver - Reject two-sided transports (NCCL/
## What happened + What you expected to happen Deploying a Ray Serve LLM (`ray.serve.llm.build_openai_app`) with a nested dataclass field in `engine_kwargs` — e.g. `structured_outputs_config` — crashes the vLLM engine on startup with: ``` AttributeError: 'dict' object has no attribute 'reasoning_parser' ``` or, depending on which field is accessed first: ```
## Description This is the foundational, interface-only PR for the Active-Passive GCS feature (split from the larger #64422). It introduces the wire-protocol field, status code, config flag, and client-side leadership-caching plumbing — all feature-gated behind `LEADER_ELECT` / `RAY_LEADER_ELECT`, which **defaults to OFF**. No server-side passive behavior is
## Description `GET /api/jobs/{job_or_submission_id}/logs` currently materializes the full driver log at each dashboard hop. Large or concurrent requests can exhaust dashboard memory. This change: - reads driver logs in 64 KiB chunks - JSON-escapes chunks incrementally - streams the response through JobAgent, JobHead, and the parent dashboard proxy with back
## Description **Summary:** Add an opt-in, per-operator minimum interval between `OutputBackpressureGuard` releases so the liveness escape hatch can't defeat backpressure on large-block workloads. Disabled by default; no behavior change unless configured. `OutputBackpressureGuard` is a liveness escape hatch: when backpressure policies clamp an operator's out
## Why this change Ray on Kubernetes docs describe Kubernetes API objects alongside Ray concepts, and the two vocabularies collide. Deployment, Job, and Service each name both a Kubernetes API object and a distinct Ray concept, so capitalization is the only thing telling them apart, and the style guide didn't say which form to use where. A contributor asking