{ "schemaVersion": 3, "dataset": { "version": 3, "date": "2026-08-13", "group": { "id": "ai-infrastructure", "name": "AI Infrastructure" }, "repository": { "id": "ray", "repo": "ray-project/ray", "name": "Ray", "keywords": [ "Ray distributed computing", "Ray framework" ] }, "context": { "repository": "ray-project/ray", "url": "https://github.com/ray-project/ray", "description": "Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.", "homepage": "https://ray.io", "language": "Python", "topics": [ "data-science", "deep-learning", "deployment", "distributed", "hyperparameter-optimization", "hyperparameter-search", "large-language-models", "llm", "llm-inference", "llm-serving", "machine-learning", "optimization", "parallel", "python", "pytorch", "ray", "reinforcement-learning", "rllib", "serving", "tensorflow" ], "license": "Apache-2.0", "defaultBranch": "master", "stars": 43510, "forks": 7922, "openIssues": 3490, "archived": false, "collectedAt": "2026-08-13T18:02:08.020333+00:00" }, "news": { "repository": "ray-project/ray", "collectedAt": "2026-08-13T18:02:08.020333+00:00", "latestRelease": { "repository": "ray-project/ray", "tag": "ray-2.57.0", "title": "Ray-2.57.0", "url": "https://github.com/ray-project/ray/releases/tag/ray-2.57.0", "publishedAt": "2026-08-11T01:15:44Z", "notes": "# Highlights\r\n\r\n* **Ray Data:** In this Ray release, we've enabled `DataSourceV2` by default (#64821), so `read_parquet` and friends use the new scan/listing infrastructure with row-group-aware chunking and predicate splitting. Hash Shuffle V2 eliminates the aggregator actor pool. V1 had to provision that pool up front from an estimate of the input size, and its actors accumulated partition shards in actor heap memory, invisible to Ray and unspillable, until finalization. V2 replaces it with two stateless task-based operators, ShuffleMapOp --> ShuffleReduceOp, that pass shards through the object store, so intermediate state spills under pressure and no capacity has to be reserved in advance. The map/reduce barrier itself remains in both designs.\r\n* HashShuffleV2 supports `join` ([#63598](https://github.com/ray-project/ray/pull/63598), [#64538](https://github.com/ray-project/ray/pull/64538), [#64687](https://github.com/ray-project/ray/pull/64687)). This lets shuffles reuse standard map/reduce scheduling, backpressure, and resource accounting.\r\n* **Ray Serve:** The HAProxy ingress is now distributed as the `ray-haproxy` PyPI package instead of being compiled into images, and it is the default HAProxy binary (#64141, #64163, #64164). We've also added gRPC support to the HAProxy direct-ingress path, including streaming, metrics, and custom request IDs (#63735, #64310, #64166, #64112). For Ray Serve LLM, we've added experimental KV-cache-aware request routing that tracks replica KV state through an event plane, tokenizes before routing, and routes on prefill/decode token load (#64084, #64085, #64097, #64224, #64327, #64400). KV cache-aware routing’s complete support will land in 2.58.\r\n* **Ray Core:** We've added an embedded RocksDB storage backend for GCS fault tolerance (REP-64), selectable with `RAY_gcs_storage=rocksdb` and `RAY_gcs_storage_path` (#63657). GCS fault tolerance no longer requires an external Redis instance. We've also added a public API for topology-aware scheduling (#63479, #63740).\r\n\r\n# Ray Data\r\n\r\n### πŸŽ‰ New Features\r\n* Enable `DataSourceV2` by default via `DataContext.use_datasource_v2` (#64821)\r\n* New task-based hash shuffle v2 (`ShuffleMapOp` β†’ `ShuffleReduceOp`) with `join`, multi-input reduce, downstream map fusion, and reducer remote args, behind an env flag (#63598, #64538, #64687, #64438, #64302, #64532, #64481)\r\n* Add a `Catalog` abstraction with a `UnityCatalog` implementation that can be passed to `read_*`, and Unity Catalog write support for Parquet and Iceberg (#64193, #64519)\r\n* Add `read_zarr` for Zarr datasets (#63003) and `read_lerobot` for LeRobot v3 datasets (#63821)\r\n* Add `PushdownCountFiles` optimization to answer `count()` from Parquet footers (#64763)\r\n* Add common subexpression elimination to the expression optimizer (#63974)\r\n* Add GPU support for `Aggregate` (#63708)\r\n* Make dataset iteration metrics queryable per split (#64608)\r\n* Add custom operator stats to capture worker-side metrics during task execution (#64221)\r\n* Refactor usage collection into an extensible `UsageCallback` (#64500)\r\n* Export `from_blocks` from `ray.data` (#64127)\r\n* Add support for `rapidsmpf-26.4.0` (#64324)\r\n\r\n### πŸ’« Enhancements\r\n* Hide Ray Data internal frames from user-code error tracebacks (#64587)\r\n* Run projection/predicate pushdown before limit pushdown, block optimization of non-deterministic expressions, and make fuse checks consistent between the logical and physical optimizers (#64651, #64165, #63936)\r\n* Add a two-mode block metadata fetch behind a single `MetadataFetcher` interface (#64378)\r\n* Use work stealing for file listing and a dynamic work queue for traversals (#64675, #64388)\r\n* Speed up `ExecutionResources` and the reservation/budget loops on the scheduling hot path (#63964)\r\n* Fix `O(n^2)` schema reconciliation in `unify_schemas` and avoid per-column `Series` materialization in tensor-column casting (#64555, #64038)\r\n* Decouple collate and memory pinning in `iter_torch_batches`, add per-stage training-thread blocking attribution, bound in-flight `iter_threaded` items, and finalize after reordering under `preserve_order` (#64653, #64183, #64219, #64282)\r\n* Add default logical memory for map operators, wire the materializing-op filter into `OpResourceAllocator` budgeting, and move `estimate_object_store_usage` into the physical op (#63814, #63665, #63961)\r\n* Decrease the downstream backpressure ratio to 2.0 (#64352)\r\n* Route native `S3FileSystem` downloads to the PyArrow threaded path (#64089)\r\n* Size hash-shuffle aggregators via bounded online sampling and add a timeout to the batched `ray.get` when fetching partitions (#63929, #64256)\r\n* Replace manual block reference accounting with Ray Core out-of-scope object callbacks (#64011, #64157, #64191)\r\n* Default `read_numpy` to `allow_pickle=False` and make it manually configurable (#64684)\r\n* Make `write_lance(mode=CREATE)` error instead of silently overwriting (#64364)\r\n* Expand `DistributionTracker` with `merge()` and p25/p75, and add dead node counts and detected issues to usage collection (#64074, #64459, #64198)\r\n* Remove cluster autoscaler v1, rename the subcluster label key to `ray-subcluster`, avoid scaling nodegroups dedicated to the head node, and quiet autoscaling coordinator logs (#64380, #64003, #63918, #63534)\r\n* Delay the \"cluster resources not enough\" warning until an operator is persistently starved, and lower the high-memory warning threshold (#63969, #64124)\r\n* Remove `ExecutionPlan`, `_num_outputs`, `batch_format` on `AllToAllOperators`, and `InheritBatchFormatRule`; use `input_dependencies` in logical operators (#63662, #64167, #64152, #64149, #64148)\r\n* Deprecate low-level scheduling APIs (`DataContext.scheduling_strategy`, `actor_locality_enabled`, `exclude_resources`, `local://`) ahead of the actor-only rearchitecture (#64632)\r\n* Migrate the Daft dependency from `getdaft` to `daft` (#64240)\r\n* Migrate `apply_chat_template`/`tokenize`/`detokenize` callers to `*_stage` form, and remove `PrepareImageStage` while deprecating the `image` row column (#63590, #63570)\r\n\r\n### πŸ”¨ Fixes\r\n* Fix `TensorDtype.__from_arrow__` crash on empty tensor columns (#64767)\r\n* Fix Arrow-backed `to_pandas` regressions with an opt-out flag and int/float block overflow handling (#64768)\r\n* Fix read-only hash array crash in hash partition (#64552, #64559)\r\n* Fix streaming-generator hang on a plasma-resident return object (#64386)\r\n* Fix Parquet write overflow for string/binary columns larger than 2 GiB (#64212)\r\n* Fix Parquet sampling hang on a permanent `OSError` (#64342)\r\n* Fix the BigQuery datasink to handle empty blocks (#60797)\r\n* Make `PandasBlock.size_bytes` deterministic (#64393)\r\n* Propagate branch in `_append_and_commit()` for the Iceberg overwrite save mode (#63922)\r\n* Propagate `isolate_read_workers` to `DatasourceV2` (#65191, #65207)\r\n* Fix cross-region S3 bucket access in the obstore download path (#63890)\r\n* Restore `safe_round` in the `ExecutionResources` hot path (#64296)\r\n* Log both the wrapped and wrapper exception in `iterate_with_retry` (#64639)\r\n* Add `tf-keras` to the `text_embedding` pip packages (#64889, #64968)\r\n* Fix `build_processor` failing with vLLM >= 0.19, correct its `config` type hint, and add `request_timeout_s` to `ServeDeploymentProcessor` to prevent indefinite hangs (#64337, #64098, #64496)\r\n* Fix type hints in `throughput_solver.py` (#64289)\r\n\r\n### πŸ“– Documentation\r\n* Add a guide on avoiding out-of-memory errors and improve the existing OOM docs (#64046, #64297)\r\n* Document the best practice for isolating Ray Data workers from training nodes (#64588)\r\n* Document running two datasets in one cluster (#64200)\r\n* Move the `read_zarr` guide back into its docstring (#64409)\r\n* Remove Ray Data and DAG docstring ignores from pydoclint (#63634)\r\n\r\n# Ray Serve\r\n\r\n### πŸŽ‰ New Features\r\n* Distribute HAProxy through the `ray-haproxy` PyPI package, make it the default binary, and remove the from-source build (#64141, #64163, #64164)\r\n* Add gRPC support to the HAProxy direct ingress, including streaming, unary-unary metrics, and custom request IDs (#63735, #64310, #64166, #64112)\r\n* Emit request and system metrics from HAProxy (#64329, #64255)\r\n* Support `root_path` in the HAProxy ingress (#64295)\r\n* Add a per-replica metadata hook exposed on `ReplicaSelection` (#63948)\r\n\r\n### πŸ’« Enhancements\r\n* HAProxy stability: graceful drain waits for in-flight and queued requests, replicas quiesce before reporting shutdown complete, dispatches cancelled by replica gRPC stop are retried, reload takeover is verified by pid, and `close-spread-time` is set in the template (#63886, #63920, #63995, #64022, #64063, #63996)\r\n* Make direct-ingress port reconcile incremental and skip it entirely when ingress membership is unchanged (#64508, #64676)\r\n* Controller performance: `O(1)` version-filtered replica counts, in-place health-check reconcile in the deployment-state loop, and a fast-path orphaned-actor check (#64699, #64507, #64511)\r\n* Freeze GC after replica initialization by default (#64742)\r\n* Eager-load proxy imports (#64113)\r\n* Pin the replica actor to bundle 0 of its placement group and track actor `fallback_strategy` on `DeploymentSchedulingInfo` (#62693, #64346)\r\n* Deprecate `HTTPOptions.location` in favor of `proxy_location`, raise on non-zero `HTTPOptions.num_cpus`, and warn on other deprecated HTTP options (#64479, #64418, #63604)\r\n* Raise a clear error when a custom request router or `serve.multiplexed` is used on an ingress deployment under direct ingress (#64211, #64045)\r\n* Recover ingress request router pin-misses via the fallback proxy instead of returning 503 (#64218)\r\n* Add `RAY_SERVE_CONTROLLER_METRICS_INCLUDE_HIGH_CARDINALITY_TAGS` to control controller metric tags (#63642)\r\n* Default `RAY_SERVE_PORT_QUARANTINE_S` to hard-stop-after plus a margin (#64021)\r\n* Defer the \"application ready\" log until proxies are ready (#64010)\r\n* Dedup ingress replica and proxy metrics code, and migrate `parse_uri` from `_private` to `_common` (#64041, #64371)\r\n* Enforce `mypy` and `pyrefly` type checking on clean Serve files (#64662)\r\n* Fail loudly when subclassing `@serve.ingress` with a sync `__init__` (#63413)\r\n* Supply a safe default request id header value in direct ingress (#64118)\r\n\r\n### πŸ”¨ Fixes\r\n* Fix `serve.ingress(FastAPI())` pickling and `include_router` under FastAPI >= 0.137 (#64814, #64531)\r\n* Fix an idle worker leak that caused OOM on the Ray head node (#63597)\r\n* Fix `serve.shutdown()` skipping live shutdown when the cached controller client is stale after a driver reconnect (#64660)\r\n* Preserve the deployment actor class across a config re-apply (#64411)\r\n* Fix rank corruption on controller recovery after a lightweight reconfigure (#64636)\r\n* Fix `route=None` crash in direct-ingress metrics reporting (#64645)\r\n* Fix the direct-ingress backpressure counter leak on cancel-while-queued and clean up the leaked receive task on request timeout (#64348, #64123)\r\n* Fix HAProxy `healthz` falling through to 404 when there are no backends (#64582)\r\n* Guard request router backoff against `OverflowError` (#64539)\r\n* Fix a slot leak race in `SingletonThreadRouter.choose_replica` (#63649)\r\n* Fix a type regression in `RAY_SERVE_HAPROXY_CLOSE_SPREAD_TIME_S` (#64752)\r\n* Fix Serve tests broken by protobuf 7 (#64362)\r\n\r\n### πŸ“– Documentation\r\n* Fix the `RequestRouterConfig` stats docstring to reference `record_routing_stats` (#64086)\r\n\r\n# Ray Train\r\n\r\n### πŸŽ‰ New Features\r\n* Add preemption handling: a preemption watcher for node-drain observability, with the signal fanned out to workers (#63807, #64099)\r\n\r\n### πŸ’« Enhancements\r\n* Make Train v1 and v2 use the autoscaling coordinator (#64824, #64936)\r\n* Restrict `ray.train.report(checkpoint)` to in-band checkpoints only (#63645)\r\n* Make `Result.from_path` read-only (#64340)\r\n* Add compute-time metrics to `tune_torch_benchmark.py` (#64274)\r\n* Update TensorFlow/Keras examples and tests for Keras 3 compatibility (#64276)\r\n\r\n### πŸ”¨ Fixes\r\n* Fix `CometLoggerCallback` mutating the caller's result dict via `result.pop()` (#64570)\r\n* Fix the `WorkerMetricsCallback` method name to match the `WorkerCallback` interface (#64568)\r\n\r\n### πŸ“– Documentation\r\n* Document the unsafe deserialization risk in `TorchCheckpoint.get_model()` and `FrameworkCheckpoint.get_preprocessor()` (#64586)\r\n* Fix docstring style and enable `--check-style-mismatch=True` in pydoclint (#63988)\r\n* Enumerate the accepted resource keys for `ScalingConfig` (#63449)\r\n\r\n# Ray Tune\r\n\r\n### πŸ’« Enhancements\r\n* Make `BayesOptSearch` float-hash precision configurable (#63914)\r\n* Require `optuna>=3.0.0` in `OptunaSearch` (#64242)\r\n\r\n### πŸ”¨ Fixes\r\n* Respect `max_concurrent_trials` for custom searchers (#63770)\r\n\r\n### πŸ“– Documentation\r\n* Warn when `BayesOptSearch` stops early on convergence, and document `patience`/`skip_duplicate` (#64288)\r\n* Restructure the Tune example pages and add PyTorch Lightning examples to the examples README (#63460, #63438)\r\n* Document nested-metric keys in `ResultGrid.get_best_result` and add a conditional search-space example to `tune.sample_from` (#63445, #63443)\r\n\r\n# Ray LLM\r\n\r\n### πŸŽ‰ New Features\r\n* Add KV-cache-aware request routing: `KVAwareRouter`/`KVRouterActor` interfaces, replica tracking, an event plane, pre-routing tokenization, token-level request lifecycle tracking, and prefill/decode token-load-aware routing (#64084, #64085, #64097, #64224, #64327, #64400)\r\n* Switch the Ray Serve SGLang engine to the sglang `RayEngine`, reach control-plane parity, and support Ray Serve direct streaming by serving SGLang's native OpenAI app (#62888, #63021, #64611)\r\n* Add the MoRIIO KV-connector backend for prefill/decode disaggregation, with cross-node worker IP advertisement via a vLLM plugin (#63951, #64067)\r\n* Enable the `/classify` and `/pooling` endpoints in direct-streaming mode (#64494)\r\n* Add a Ray Serve LLM SGLang metrics dashboard (#64797, #65083)\r\n\r\n### πŸ’« Enhancements\r\n* Upgrade vLLM to 0.25.1 (#64082, #64483, #64697)\r\n* Delegate prefill/decode orchestration to the KV-connector backend and avoid double prompt tokenization in disaggregation (#63950, #64049)\r\n* Make `request.request_id` authoritative for the engine and stop clobbering an explicitly set request id with the Serve id (#63949, #64044)\r\n* Parse the direct-streaming routing payload for body-aware routers, and add direct streaming session-affinity tests (#64328, #63927, #64266)\r\n* Use `asyncio.create_task` instead of `ensure_future` in `KVRouterActor` (#64546)\r\n* Add a cluster filter to the Serve LLM Grafana dashboard (#64596)\r\n* Validate `accelerator_type` for CPU vLLM engine configs (#64235)\r\n* Use `model_id` rather than the remote URI as the cache identifier in `VLLMEngineConfig` (#64110)\r\n* Remove the Serve LLM config generator (#64075)\r\n\r\n### πŸ”¨ Fixes\r\n* Fix malformed PromQL in the Serve LLM Grafana dashboard (#63893)\r\n\r\n### πŸ“– Documentation\r\n* Add a custom vLLM model guide and update the vLLM embedding model example (#64594, #64504)\r\n* Improve the Serve LLM docs landing pages, add a config reference, and document direct streaming and the ingress request router (#64076, #63860)\r\n* Fix 404s in the Serve LLM examples sidebar links (#64209)\r\n\r\n# Ray RLlib\r\n\r\n### πŸŽ‰ New Features\r\n* Add a pull-based `EnvRunnerStateServer` for async weight sync (#63849)\r\n* Expose the count of `EnvRunner`s dropped on `timeout_seconds` calls (#63493)\r\n\r\n### πŸ’« Enhancements\r\n* Upgrade the example vision stack to MobileNet v3 and remove MobileNet v2 torchhub usage (#64031, #64028)\r\n\r\n### πŸ”¨ Fixes\r\n* Narrow `from_checkpoint` expectations to avoid silent failures (#63614)\r\n* Require `MultiRLModuleSpec.rl_module_specs` to be a dict (#64785)\r\n* Fix the perpetual pending state in offline evaluation during Tune (#64453)\r\n* Fix RLlib with Tune by reverting the `AlgorithmConfig.to_dict()` change for the new API stack (#63695, #64501)\r\n* Fix ONNX export under torch 2.9 by bumping `onnxscript` and migrating to the dynamo exporter (#64410, #64033)\r\n* Apply observation filters in `compute_single_action()` on the old API stack (#64088)\r\n\r\n### πŸ“– Documentation\r\n* Revamp the RL modules docs and speed up the docs tests (#64505)\r\n* Fix the outdated parametric-action-spaces link in `rllib-algorithms` (#63435)\r\n\r\n# Ray Core\r\n\r\n### πŸŽ‰ New Features\r\n* Add an embedded RocksDB storage backend for GCS fault tolerance (REP-64), selected via `RAY_gcs_storage=rocksdb` and `RAY_gcs_storage_path` (#63657)\r\n* Add a public API for topology-aware scheduling via `topology_strategy` (#63479, #63740)\r\n* Implement backpressure for async streaming generators, add actor-level backpressure config, and add `_num_objects_per_yield` (#64383, #63310, #63943)\r\n* Implement `RayTaskEventRecorder` as the first step of moving task events out of GCS (#64168)\r\n* Drain the node on `SIGTERM` before shutting down `ray start --block` (#64454)\r\n* Add `tpu.dispatch` syntax sugar and a JAX profiler for TPU (#64493, #62371)\r\n* Publicize `__ray_call__` as a `DeveloperAPI` for running closures on actors (#64367)\r\n* Add the NVIDIA RTX PRO 6000 accelerator type and update the AMD accelerator manager (#64428, #62477)\r\n* RDT: enable the `LIBFABRIC` backend for NIXL and upgrade NIXL to v1.2.0 (#62339, #63980)\r\n* Add an owner-driven `FreeLocalObjects` RPC and remove the original `FreeObjects` RPC and `ObjectEviction` pubsub (#63218, #63181)\r\n\r\n### πŸ’« Enhancements\r\n* Actor scalability: prefill the placement-group bundle cache at creation to remove an actor submission bottleneck, and construct gRPC metric objects once per process instead of per request (#64649, #64732)\r\n* Bound the GCS worker table by evicting old dead-worker entries, prioritized by exit type (#64612, #64729)\r\n* Batch object-manager `Pull` RPCs by destination node and read spilled object chunks in bulk instead of byte-by-byte (#64225, #63830)\r\n* Post all object pubsub publishes to the IO thread to relieve lock contention (#63983)\r\n* Autoscaler: improve v2 scheduler performance for large clusters, handle deleted node types in the metrics reporter, and support env-var configuration for log rotation (#64175, #64184, #63955)\r\n* Integrate `IOContextMonitor` into GCS, update its metrics, and exclude `ray_syncer_io_context` and `task_io_context` from health checks while raising the probe deadline to 30s (#63930, #63975, #64522, #64421)\r\n* Continue the `ClockInterface` migration in the core worker and dependency-inject `PeriodicalRunnerInterface` (#63956, #63994, #64061, #64029)\r\n* Enable `process_group_cleanup_enabled` by default and fix graceful-shutdown cleanup (#64407)\r\n* Make owned object metrics lazy, compute per-component memory usage in MiB, scope `prom_metrics_service_discovery.json` to the session dir, and set the OpenTelemetry resource identity (#64093, #63932, #63850, #63921)\r\n* Re-emit per-state task, actor, and placement group gauges every metrics tick and retain gauge observations for a TTL, so live series survive a missed scrape (#64524, #64633)\r\n* Add the GPU UUID to the labels of GPU metrics (#65113, #65134)\r\n* Log a hint to the driver when system overhead exceeds reserved system memory (#64270)\r\n* Validate explicit zero resource isolation overrides and move memory monitors into `common/monitor` (#63864, #64491)\r\n* Prioritize tasks over actors when killing under memory pressure (#64170)\r\n* Bail on `AdjustWorkerOomScore` read failure and clamp the score to `[-1000, 1000]` (#62713)\r\n* Cap the `runtime_env` `parse_uri` package name length to avoid `ENAMETOOLONG` (#64339)\r\n* Scope the raylet's forced GC to backstop actor handle cycle deadlocks (#63939)\r\n* Support joining fibers before exiting the fiber runner thread (#64405)\r\n* Internal cleanups: remove callbacks from `TaskToExecute`, split task/resource preparation utils out of `_raylet.pyx`, consolidate the GCS dedicated `io_context` policy, and rename `OnDemandBroadcasting` (#64182, #63088, #63931, #63855)\r\n\r\n### πŸ”¨ Fixes\r\n* Fix a Python 3.14 async-actor memory leak by re-anchoring the stack (#65177)\r\n* Fix a deadlock between metric registration and `collect()` in `OpenTelemetryMetricRecorder` (#64946, #65094)\r\n* Fix multi-threaded actor graceful shutdown (#64019)\r\n* Streaming generator fixes: remove ref stream callbacks that caused a reconstruction hang, fail fast on a replay object-count mismatch, remove the implicit `__dealloc__()`, and honor `timeout_s` in the end-of-stream `ray.get` (#64581, #64394, #64333, #64014)\r\n* Fix pipelining of actor argument fetches while an actor task is running (#63079)\r\n* Fix a dangling reference in `GetSchedulingClassDescriptor` (#64707)\r\n* Fix a pgid cleanup leak in the job supervisor (#64746)\r\n* Fix the raylet drain path reading stale object store memory usage (#64415)\r\n* Avoid a GCS crash on Redis connection loss in `RedisResponseFn` (#64204)\r\n* Fix the actor `working_dir` being overridden by the job-level `py_driver_sys_path` (#63756)\r\n* Fix `RAY_CHECK` failures caused by double `ray.cancel()` and keyboard interrupts (#63663)\r\n* Fix `event_logger` and `export_event_logger` to flush all handlers safely (#63947)\r\n* Default `TaskProfileEvent.extra_data_` to `{}` to avoid a JSON decoding error in the State API, and normalize the `cmdline` field in the `StatsPayload` schema (#64589, #64286)\r\n* Autoscaler v2: fix `ALLOCATION_TIMEOUT` worker replacement ordering and the `RAY_STOP_REQUESTED` β†’ `RAY_RUNNING` fallback when drain has succeeded (#63815, #63424)\r\n* Jobs: replace an `assert` with `RuntimeError` in `put_status` and avoid mutating `runtime_env` during submission (#64569, #63990)\r\n* Improve the error message when `RAY_ADDRESS` is set to an HTTP address (#64180)\r\n* Define `WIN32_LEAN_AND_MEAN` globally for Windows builds (#64361)\r\n* Wait for subprocesses to terminate in `ray.shutdown()` (#63655)\r\n* Fix the per-process GPU memory utilization dashboard stat (#61412)\r\n\r\n### πŸ“– Documentation\r\n* Update the OOM documentation to use resource isolation mode (#63652)\r\n* Clarify the KubeRay autoscaler `restartPolicy` requirement (#64855)\r\n* Document the `name` parameter for `task.options()` (#63450)\r\n* Enforce Doxygen documentation style for coding agents and remove Ray Core docstring ignores from pydoclint (#64188, #63639)\r\n\r\n# Dashboard\r\n\r\n### πŸŽ‰ New Features\r\n* Implement the frontend UI for Platform Events (#63332)\r\n\r\n### πŸ’« Enhancements\r\n* Link workers to actor detail pages and handle nil IDs (#63795)\r\n* Support a Grafana Cloud metrics fallback (#64138)\r\n* Format `RuntimeEnvState.creation_time_ms` as a duration (#64207)\r\n* Diagnose dashboard-agent event-loop stalls (#64436)\r\n* Log errors when job submission fails in `JobHead` and stop configuring the root logger on module import (#64443, #64463)\r\n\r\n### πŸ”¨ Fixes\r\n* Fix local file inclusion in the `/logs` API endpoint by rejecting absolute paths and `..` components in `LogAgentV1Grpc.ListLogs()` (#64701)\r\n* Fix TPU metrics (#63998)\r\n* Fix task grouping by custom task name in the Dashboard API (#63820)\r\n* Fix an \"automaticaly\" typo in the Grafana dashboard descriptions (#64614)\r\n\r\n# Ray Wheels and Images\r\n\r\n* Build and publish Python 3.14 wheels and images, add py3.14 TPU base deps, refresh the py3.14 locks, and add a py3.14 smoke release test (#64754, #64750, #64857, #64938).\r\n* Build and publish `cu130` variants for the `ray` and `ray-extra` images (#63972, #63801).\r\n* Add a py3.13 `ray-ml` image with `torchft-nightly` (#63587).\r\n* Add `ray-haproxy` to the `ray[serve]` extra and base requirements, and bump it to 2.8.25 for CVE-2026-55203/55204 (#64141, #64430).\r\n* Unify `requirements_compiled` across Python versions and upgrade the HuggingFace stack to `datasets` 4.x (#64257, #64054).\r\n* Fix the vendored RocksDB build: use Make instead of Ninja to repair aarch64 core builds, drop `-Wl,-pie` from the TSAN link flags, and silence TSAN false-positive races (#64748, #64917, #64937, #64759).\r\n\r\n# Documentation\r\n\r\n* Added `AGENTS.md` with the AI-assisted contribution policy, routed `.claude/CLAUDE.md` to it, and added a documentation style and grammar guide (#64419, #64518).\r\n* Converted the ray-contribute developer guides from RST to MyST, soft-wrapped prose across `doc/source`, and added an rst-to-myst conversion skill (#64100, #64111, #64115, #64136, #64259, #64279, #64135).\r\n* Added a unified APIs navigation section, generated API stubs without a full doc build, and greened the API-doc consistency check with reverse/dedup policies and a split whitelist (#64387, #64423, #64420, #64783, #64786, #64808, #64818).\r\n* Added curated page descriptions for nav pages (SEO and `llms.txt`) and a custom 404 page with absolute URLs (#64330, #64603).\r\n* Built Read the Docs incrementally from the doc build cache, repaired cache production, and skipped RtD PR builds for code-only PRs (#64277, #64414, #64482, #64599, #64547, #64480).\r\n* Hardened doc template fetches: pinned to specific build ids, guarded zip extraction against path traversal, failed the build loudly on fetch failure, and removed outdated templates now synced at build time (#64637, #64669, #64629, #64635, #63971).\r\n* Upgraded Sphinx to 8.2.3 on Python 3.11, decoupled Sphinx from the test/CI dependency stack, and fixed moved intersphinx inventory URLs (#64070, #63962, #64516).\r\n* KubeRay: added an IPPR user guide, documented RayCronJob timezone support and autoscaler command overwrite, updated the RayJob agent sandbox docs, and refreshed the Serve High Throughput guide for Ray 2.56 (#63212, #64444, #63627, #64460, #64144).\r\n* Added notebook tests for the lightweight Ray Core examples and recorded the Train example notebooks that are intentionally untested (#64313, #64318).\r\n* Refreshed the ecosystem page, added a Ray + Agent Sandbox example, and renamed the AI chat widget to \"Ray Docs\" (#64156, #64047, #64351).\r\n\r\n# Thanks\r\n\r\nMany thanks to all those who contributed to this release!\r\n\r\n@justinvyu, @neuyilan, @bveeramani, @abhishekverma-ray, @dstrodtman, @yinli-systems, @Truc54, @tanmayrauth, @ArchishmanSengupta, @yuhuan130, @skpark-rh, @shaun0927, @ps2181, @edoakes, @htvien, @xinyuangui2, @owenowenisme, @ronny-anyscale, @nadongjun, @LeMinhNhat2901, @kevin85421, @akyang-anyscale, @jeffreywang88, @sai-miduthuri, @robertnishihara, @andrewsykim, @iamjustinhsu, @leewyang, @marwan116, @MortalHappiness, @fscnick, @ayushk7102, @ArturNiederfahrenhorst, @rayhhome, @nh-atuan, @kimngoc280105, @TimothySeah, @zyxue, @antoine-galataud, @omkar-334, @alimaazamat, @jhasm, @ShockYoungCHN, @Kunchd, @richardliaw, @xyuzh, @linh285, @shorbaji, @ans9868, @LuciferYang, @joaquinhuigomez, @liujp, @ShuChenLin, @johntaylor-cell, @vickytsang, @abrarsheikh, @saivedant169, @rmhowe425, @kyuds, @kunling-anyscale, @fuxi611, @karticam, @enginarslan1, @spencer-p, @MengjinYan, @AyushKashyapII, @dragongu, @Sparks0219, @rueian, @praneethkaturi, @dinhxuanvu, @lonexreb, @AarryaSaraf, @Junyi-Wang-6, @saschwartz, @odncode, @vicentefb, @JasonLi1909, @coqian, @kouroshHakha, @daiping8, @pseudo-rnd-thoughts, @wanadzhar913, @HungHiHung10, @goutamvenkat-anyscale, @LeThienTrong, @alexandrplashchinsky, @raulchen, @ryankert01, @jiangxt2, @HirokiNariyoshi, @Yicheng-Lu-llll, @martinlhw, @richabanker, @aaronscalene, @SohamRajpure, @tvaucher, @Jade07-1, @eicherseiji, @dayshah, @Kropiunig, @machichima, @harshit-anyscale, @elliot-barn, @liulehui, @OneSizeFitsQuorum, @alexeykudinkin, @sampan-s-nayak, @prince8273, @jpatra72", "highlights": [ "Ray Data: In this Ray release, we've enabled DataSourceV2 by default (#64821), so readparquet and friends use the new scan/listing infrastructure with row-group-aware chunking and predicate splitting. Hash Shuffle V2 eliminates the aggregat", "HashShuffleV2 supports join (#63598, #64538, #64687). This lets shuffles reuse standard map/reduce scheduling, backpressure, and resource accounting.", "Ray Serve: The HAProxy ingress is now distributed as the ray-haproxy PyPI package instead of being compiled into images, and it is the default HAProxy binary (#64141, #64163, #64164). We've also added gRPC support to the HAProxy direct-ingr", "Ray Core: We've added an embedded RocksDB storage backend for GCS fault tolerance (REP-64), selectable with RAYgcsstorage=rocksdb and RAYgcsstoragepath (#63657). GCS fault tolerance no longer requires an external Redis instance. We've also ", "πŸŽ‰ New Features", "Enable DataSourceV2 by default via DataContext.usedatasourcev2 (#64821)" ], "prerelease": false }, "upcoming": [ { "repository": "ray-project/ray", "kind": "milestone", "title": "Infra Backlog", "url": "https://github.com/ray-project/ray/milestone/11", "description": "P1 issues: https://github.com/ray-project/ray/issues?q=is%3Aopen+is%3Aissue+milestone%3A\"Infra+Backlog\"+label%3AP1", "progress": 95, "openIssues": 20, "closedIssues": 378 }, { "repository": "ray-project/ray", "kind": "milestone", "title": "Packaging and Dependency Management", "url": "https://github.com/ray-project/ray/milestone/36", "description": "P1 Issues: https://github.com/ray-project/ray/issues?q=is%3Aopen+is%3Aissue+milestone%3A\"Packaging+and+Dependency+Management\"+label%3AP1\r\n", "progress": 79, "openIssues": 8, "closedIssues": 31 }, { "repository": "ray-project/ray", "kind": "milestone", "title": "[serve] Support Java as language", "url": "https://github.com/ray-project/ray/milestone/45", "description": "", "progress": 100, "openIssues": 0, "closedIssues": 18 }, { "repository": "ray-project/ray", "kind": "milestone", "title": "runtime_env backlog", "url": "https://github.com/ray-project/ray/milestone/48", "description": "", "progress": 94, "openIssues": 9, "closedIssues": 137 }, { "repository": "ray-project/ray", "kind": "milestone", "title": "Workflows after-alpha", "url": "https://github.com/ray-project/ray/milestone/52", "description": "", "progress": 99, "openIssues": 1, "closedIssues": 78 } ], "communityDiscussions": [] }, "runs": [ { "collectedAt": "2026-08-13T12:26:38.318Z", "since": "2026-08-12T12:26:38.318Z", "observedCount": 97, "changedCount": 97 }, { "collectedAt": "2026-08-13T13:48:00.446149Z", "since": "2026-08-12T13:48:00.446149Z", "observedCount": 96, "changedCount": 96 }, { "collectedAt": "2026-08-13T16:19:22.035158Z", "since": "2026-08-12T16:19:22.035158Z", "observedCount": 93, "changedCount": 5 }, { "collectedAt": "2026-08-13T17:43:20.785491Z", "since": "2026-08-12T17:43:20.785491Z", "observedCount": 96, "changedCount": 12 }, { "collectedAt": "2026-08-13T17:47:07.884300Z", "since": "2026-08-12T17:47:07.884300Z", "observedCount": 96, "changedCount": 2 }, { "collectedAt": "2026-08-13T18:01:55.420671Z", "since": "2026-08-12T18:01:55.420671Z", "observedCount": 96, "changedCount": 0 } ], "signals": [ { "id": "github:ray-project/ray:issue:36415", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "issue", "title": "[core] Ray session conflicts with PyArrow+HDFS", "text": "### What happened + What you expected to happen Using PyArrow fs with HDFS works fine outside a ray session: ``` file_sys, file_path = pyarrow.fs.FileSystem.from_uri(hdfs_folder) file_infos = file_sys.get_file_info(pyarrow.fs.FileSelector(file_path, recursive=False)) ``` However, after `ray.init()`, the same code results in a segmentation fault: ``` 2023-06-14 01:27:37,622 INFO worker.py:1614 -- Started a local Ray instance. View the dashboard at http://127.0.0.1:8265 *** SIGSEGV received at time=1686731258 on cpu 0 *** PC: @ 0x7f99d20c5822 (unknown) (unknown) @ 0x7f996fa6ec85 208 absl::lts_20220623::WriteFailureInfo() @ 0x7f996fa6e9c8 64 absl::lts_20220623::AbslFailureSignalHandler() @ 0x7f99e81c6420 3408 (unknown) @ 0x7f99d1c2782e 48 (unknown) @ 0x7f99d1c2cc0f 240 (unknown) @ 0x7f99d2267a5f 144 (unknown) @ 0x7f99d2267d53 128 (unknown) @ 0x7f99d21092a0 64 (unknown) @ 0x7f99e81ba609 (unknown) start_thread [2023-06-14 01:27:38,591 E 9716 9731] logging.cc:361: *** SIGSEGV received at time=1686731258 on cpu 0 *** [2023-06-14 01:27:38,591 E 9716 9731] logging.cc:361: PC: @ 0x7f99d20c5822 (unknown) (unknown) [2023-06-14 01:27:38,591 E 9716 9731] logging.cc:361: @ 0x7f996fa6ec85 208 absl::lts_20220623::WriteFailureInfo() [2023-06-14 01:27:38,592 E 9716 9731] logging.cc:361: @ 0x7f996fa6e9e1 64 absl::lts_20220623::AbslFailureSignalHandler() [2023-06-14 01:27:38,593 E 9716 9731] logging.cc:361: @ 0x7f99e81c6420 3408 (unknown) [2023-06-14 01:27:38,593 E 9716 9731] logging.cc:361: @ 0x7f99d1c2782e 48 (unknown) [2023-06-14 01:27:38,593 E 9716 9731] logging.cc:361: @ 0x7f99d1c2cc0f 240 (unknown) [2023-06-14 01:27:38,593 E 9716 9731] logging.cc:361: @ 0x7f99d2267a5f 144 (unknown) [2023-06-14 01:27:38,593 E 9716 9731] logging.cc:361: @ 0x7f99d2267d53 128 (unknown) [2023-06-14 01:27:38,593 E 9716 9731] logging.cc:361: @ 0x7f99d21092a0 64 (unknown) [2023-06-14 01:27:38,593 E 9716 9731] logging.cc:361: @ 0x7f99e81ba609 (unknown) start_thread Fatal Python error: Segmentation fault # # A fatal error has been detected by the Java Runtime Environment: # # SIGSEGV (0xb) at pc=0x00007f99e81c62ab, pid=9716, tid=0x00007f99baa56700 # # JRE version: OpenJDK Runtime Environment (8.0_362-b09) (build 1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09) # Java VM: OpenJDK 64-Bit Server VM (25.362-b09 mixed mode linux-amd64 compressed oops) # Problematic frame: # C [libpthread.so.0+0x142ab] raise+0xcb # # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try \"ulimit -c unlimited\" before starting Java again # # An error report file with more information is saved as: # /ray/hs_err_pid9716.log # # If you would like to submit a bug report, please visit: # http://bugreport.java.com/bugreport/crash.jsp # [failure_signal_handler.cc : 332] RAW: Signal 6 raised at PC=0x7f99e800300b while already in AbslFailureSignalHandler() *** SIGABRT received at time=1686731258 on cpu 0 *** PC: @ 0x7f99e800300b (unknown) raise @ 0x7f996fa6ec85 208 absl::lts_20220623::WriteFailureInfo() @ 0x7f996fa6e9c8 64 absl::lts_20220623::AbslFailureSignalHandler() @ 0x7f99e81c6420 3952 (unknown) @ 0x7f99d22c3843 240 (unknown) @ 0x7f99d211410e 352 JVM_handle_linux_signal @ 0x7f99d210731c 64 (unknown) @ 0x7f99e81c6420 10576 (unknown) @ 0x7f99d1c2782e 48 (unknown) @ 0x7f99d1c2cc0f 240 (unknown) @ 0x7f99d2267a5f 144 (unknown) @ 0x7f99d2267d53 128 (unknown) @ 0x7f99d21092a0 64 (unknown) @ 0x7f99e81ba609 (unknown) start_thread [2023-06-14 01:27:38,618 E 9716 9731] logging.cc:361: *** SIGABRT received at time=1686731258 on cpu 0 *** [2023-06-14 01:27:38,618 E 9716 9731] logging.cc:361: PC: @ 0x7f99e800300b (unknown) raise [2023-06-14 01:27:38,618 E 9716 9731] logging.cc:361: @ 0x7f996fa6ec85 208 absl::lts_20220623::WriteFailureInfo() [2023-06-14 01:27:38,618 E 9716 9731] logging.cc:361: @ 0x7f996fa6e9e1 64 absl::lts_20220623::AbslFailureSignalHandler() [2023-06-14 01:27:38,618 E 9716 9731] logging.cc:361: @ 0x7f99e81c6420 3952 (unknown) [2023-06-14 01:27:38,618 E 9716 9731] logging.cc:361: @ 0x7f99d22c3843 240 (unknown) [2023-06-14 01:27:38,618 E 9716 9731] logging.cc:361: @ 0x7f99d211410e 352 JVM_handle_linux_signal [2023-06-14 01:27:38,618 E 9716 9731] logging.cc:361: @ 0x7f99d210731c 64 (unknown) [2023-06-14 01:27:38,619 E 9716 9731] logging.cc:361: @ 0x7f99e81c6420 10576 (unknown) [2023-06-14 01:27:38,619 E 9716 9731] logging.cc:361: @ 0x7f99d1c2782e 48 (unknown) [2023-06-14 01:27:38,619 E 9716 9731] logging.cc:361: @ 0x7f99d1c2cc0f 240 (unknown) [2023-06-14 01:27:38,619 E 9716 9731] logging.cc:361: @ 0x7f99d2267a5f 144 (unknown) [2023-06-14 01:27:38,619 E 9716 9731] logging.cc:361: @ 0x7f99d2267d53 128 (unknown) [2023-06-14 01:27:38,619 E 9716 9731] logging.cc:361: @ 0x7f99d21092a0 64 (unknown) [2023-06-14 01:27:38,619 E 9716 9731] logging.cc:361: @ 0x7f99e81ba609 (unknown) start_thread Fatal Python error: Aborted ``` Here is the log dump from java: [hs_err_pid9716.log](https://github.com/ray-project/ray/files/11743507/h5PTZ.log) The segfault occurs almost every time, but not always. It never occurs when ray is not initialized. Thus there is probably some interference between the ray session/global state and the java/pyarrow/hdfs connection. ### Versions / Dependencies Ray latest master, hadoop 3.2.4, java openjdk version \"1.8.0_362\" ### Reproduction script - Install HDFS with `./ci/env-install-hdfs.sh` - Create some directory in HDFS e.g. with `/opt/hadoop-3.2.4/bin/hdfs dfs -put /tmp/somewhere hdfs://[host]:8020/somewhere` - Run this script ``` def setup_hdfs(): \"\"\"Set env vars required by pyarrow to talk to hdfs correctly. Returns hostname and port needed for the hdfs uri.\"\"\" # the following file is written in `install-hdfs.sh`. with open(\"/tmp/hdfs_env\", \"r\") as f: for line in f.readlines(): line = line.rstrip(\"\\n\") tokens = line.split(\"=\", maxsplit=1) os.environ[tokens[0]] = tokens[1] import sys sys.path.insert(0, os.path.join(os.environ[\"HADOOP_HOME\"], \"bin\")) hostname = os.getenv(\"CONTAINER_ID\") port = os.getenv(\"HDFS_PORT\") return hostname, port import os import pyarrow import pyarrow.fs hostname, port = setup_hdfs() workspace_dir = f'hdfs://{hostname}:{port}/somewhere' # from ray.air._internal.remote_storage import upload_to_uri # upload_to_uri(\"/tmp/content\", workspace_dir) def get_list_of_files_under_hdfs_folder(hdfs_folder): file_sys, file_path = pyarrow.fs.FileSystem.from_uri(hdfs_folder) file_infos = file_sys.get_file_info(pyarrow.fs.FileSelector(file_path, recursive=False)) return file_infos print(f\"Success!, number of files in {workspace_dir}: {len(get_list_of_files_under_hdfs_folder(workspace_dir))}\") print(f\"Success!, number of files in {workspace_dir}: {len(get_list_of_files_under_hdfs_folder(workspace_dir))}\") print(\"initializing ray, and get number of files again.\") import ray ray.is_initialized() ray.init() print(\"After ray init\", len(get_list_of_files_under_hdfs_folder(workspace_dir))) ``` ### Issue Severity High: It blocks me from completing my task.", "url": "https://github.com/ray-project/ray/issues/36415", "createdAt": "2023-06-14T08:34:15Z", "updatedAt": "2026-08-13T02:47:35Z", "timestamp": "2026-08-13T02:47:35Z", "metrics": { "reactions": 4, "comments": 27 }, "labels": [ "bug", "P1", "core", "stability", "core-fundamentals" ], "author": "krfricke", "state": "open", "assignees": [] }, { "id": "github:ray-project/ray:issue:45751", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "issue", "title": "Ray Dashboard is susceptible to a Local File Inclusion bug with default settings", "text": "### What happened + What you expected to happen **Description** Just as the title suggests, when initially starting up the Ray dashboard I can parse my local file system by supplying the `folder` HTTP parameter in GET requests to the `/logs` API endpoint in conjunction with relative file paths. **Expected Behavior** As a best practice, relative file paths should not be permitted, or should be ignored. Basic access controls should be implemented to prevent unauthorized users from being able to parse the underlying file system being used by Ray. **Useful Information** 1) Taking a cursory glance at the Ray GH Issues board, it appears that this issue was discussed (and addressed) [here](https://github.com/ray-project/ray/issues/39701) and [here](https://www.anyscale.com/blog/update-on-ray-cves-cve-2023-6019-cve-2023-6020-cve-2023-6021-cve-2023-48022-cve-2023-48023) 2) As seen in the provided script, the `nodeId` HTTP parameter must be included in GET requests. ![image](https://github.com/ray-project/ray/assets/45905457/2eeed262-caa0-4e0e-9f32-5a9604be16f6) ### Versions / Dependencies 2.9.1 and 2.23.0 ### Reproduction script from requests import get url = 'http://``:8265/#/logs' headers = {'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:126.0) Gecko/20100101 Firefox/126.0', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8', 'Accept-Encoding': 'gzip, deflate, br, zstd', } params = {'folder': '../../../../home', 'nodeId': '``' } try: resp = get(url=url, params=params, headers=headers) except Exception as e: print(f'GET request failed: {str(e)}') if resp.status_code != 200: print('ERROR server responded with: {resp.status_code}, {resp.content}') else: print(f'Response\\n==========\\n{resp.content}') ### Issue Severity Medium: It is a significant difficulty but I can work around it.", "url": "https://github.com/ray-project/ray/issues/45751", "createdAt": "2024-06-05T12:04:49Z", "updatedAt": "2026-08-13T16:46:16Z", "timestamp": "2026-08-13T16:46:16Z", "metrics": { "reactions": 0, "comments": 9 }, "labels": [ "bug", "P2", "dashboard", "observability", "security" ], "author": "rmhowe425", "state": "closed", "assignees": [ "rmhowe425" ] }, { "id": "github:ray-project/ray:issue:56431", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "issue", "title": "[Data/LLM] Non stop CPU autoscaling with vLLM", "text": "### What happened + What you expected to happen I'm doing batch LLM inference using Ray Data, and I've noticed for large batches of prompts, the CPU autoscaler will continuously autoscale until it reaches maximum specified amount of CPU worker nodes as defined in my configuration file, and then it will start launching extra GPU instances even though I specified that GPU instances only have 1 CPU under resources (need at least 1 or else the code for LLM inference breaks). It reaches a point where I have over 300 non active CPUs in the cluster and Ray is still trying to launch more. I see {\"CPU\":1} requested over 200 times in bursts which seemingly causes this, but this resource constraint is gone within a few seconds and I'm not sure what is causing this burst of single CPU requests. \"Image\" In the image below, each cpu worker has 32 CPUs, and we scaled it to 10 nodes which is the maximum in my config, and when it runs out it begins trying to start GPU instances (p5en, which are very expensive) to try to scale CPUs further. This ends up being a very expensive bug because of the price of these GPU instances. \"Image\" ### Versions / Dependencies Ray 2.48.0 vLLM 0.10.0 ### Reproduction script ```python start = time.time() # 1) Resolve I/O paths in_s3 = cfg[\"s3_prompts_path\"] out_s3 = cfg[\"s3_output_path\"] # 2) Load prompts οƒ  Ray Dataset instances = int(cfg[\"instances\"]) data_parallel_size = 2 partitions = max(instances * 8, 1) model = cfg[\"model\"] engine_kwargs = cfg[\"engine_kwargs\"] if model.startswith(\"s3://\"): engine_kwargs[\"load_format\"] = \"runai_streamer\" engine_cfg = vLLMEngineProcessorConfig( model_source=model, engine_kwargs=engine_kwargs, concurrency=instances, apply_chat_template=bool(cfg[\"apply_chat_template\"]), batch_size=int(cfg[\"batch_size\"]), max_concurrent_batches=4, # accelerator_type=cfg[\"accelerator_type\"], # Add runtime_env here if you need HF_TOKEN, etc. ) ds = ray.data.read_parquet(in_s3) ds = ds.repartition(partitions) processor = build_llm_processor( engine_cfg, preprocess=lambda row: dict( messages=[ {\"role\": \"system\", \"content\": row[\"system\"]}, {\"role\": \"user\", \"content\": row[\"prompt\"]}, ], sampling_params=cfg[\"sampling_params\"], ), postprocess=lambda row: dict(**{k: v for k, v in row.items() if k not in (\"prompt\", \"system\")}), ) # 4) Execute & write results print(\"⏳ Running batch inference …\") processed = processor(ds) processed.write_parquet(out_s3) duration = time.time() - start print(f\"βœ… Completed in {duration:,.1f} s β†’ {out_s3}\") ``` ### Issue Severity Medium: It is a significant difficulty but I can work around it.", "url": "https://github.com/ray-project/ray/issues/56431", "createdAt": "2025-09-10T23:10:51Z", "updatedAt": "2026-08-12T21:26:25Z", "timestamp": "2026-08-12T21:26:25Z", "metrics": { "reactions": 0, "comments": 11 }, "labels": [ "bug", "P1", "performance", "data", "llm", "stability", "community-backlog" ], "author": "jiangwu300", "state": "open", "assignees": [ "bveeramani" ] }, { "id": "github:ray-project/ray:issue:64643", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "issue", "title": "[serve] Enable mypy type checking on ray/serve to catch real production bugs (e.g. the rank-corruption bug in #64181)", "text": "### What happened + What you expected to happen ### Description Ray Serve's controller code is fully type-annotated but never type-checked. The mypy pre-commit hook covers an allowlist of exactly 7 files repo-wide β€” from all of Serve, only `python/ray/serve/handle.py` and a handle-typing fixture: # .pre-commit-config.yaml - id: mypy args: ['--follow-imports=skip', '--ignore-missing-imports'] files: | (?x)^( python/ray/autoscaler/node_provider.py| ... python/ray/serve/handle.py| python/ray/serve/tests/typing_files/check_handle_typing.py ) Additionally, `--follow-imports=skip` degrades every cross-module type to `Any`, so even allowlisted files get almost no protection from wrong-type arguments: any parameter annotated with an imported class effectively becomes untyped. ### Motivating case: a production bug mypy flags exactly The rank-corruption bug behind #63862 (fixed in ) was a one-liner in `deployment_state.py` β€” the lightweight-reconfigure path passed `rank=current_rank.rank` (a bare `int`) where the callee declares `rank: ReplicaRank`. The int round-tripped through the replica and broke rank recovery on every controller restart, permanently corrupting the rank system (\"Rank system is in an invalid state\" loop). Running mypy 1.7.0 (the version already pinned in pre-commit) with import following enabled flags the bug precisely: deployment_state.py:3749: error: Argument \"rank\" to \"reconfigure\" of \"DeploymentReplica\" has incompatible type \"int\"; expected \"ReplicaRank\" [arg-type] With Ray's current flags (`--follow-imports=skip`), the same run reports 57 unrelated errors in the file and does NOT flag this line β€” `ReplicaRank` collapses to `Any`. So this isn't just \"add the file to the list\"; the flags matter too. The same run surfaces more latent inconsistencies in the rank system alone: deployment_state.py: error: Argument \"assign_rank_callback\" to \"start\" of \"DeploymentReplica\" has incompatible type \"Callable[[str, str], ReplicaRank]\"; expected \"Callable[[ReplicaID], ReplicaRank]\" [arg-type] (2 call sites) deployment_state.py: error: Argument 3 to \"recover_rank\" of \"DeploymentRankManager\" has incompatible type \"ReplicaRank | None\"; expected \"ReplicaRank\" [arg-type] This bug class β€” a raw value passed where a wrapper object is expected, then serialized across the controller/replica RPC boundary and only exploding much later during recovery β€” is essentially invisible to unit tests that mock the boundary, and exactly what static typing is for. ### Current scope of the debt (measured on master) `mypy --follow-imports=silent --ignore-missing-imports ray/serve/_private/`: - 1028 errors in 65 of 86 files - 237 of them are `[arg-type]` (the category that catches bugs like the above) ### Proposal (incremental, no big-bang) 1. Add a Serve-scoped mypy CI step (or extend the pre-commit hook) with `--follow-imports=silent` so imported types resolve, scoped initially to an error-free subset of `ray/serve/_private/` (per-file allowlist, same mechanism as today). 2. Burn down `[arg-type]` errors first in controller-critical modules (`deployment_state.py`, `replica.py`, `controller.py`, `application_state.py`) β€” these guard the state-checkpoint/recovery paths where type confusion becomes persistent state corruption. 3. Ratchet: once a file is clean, add it to the allowlist so it can't regress. ### Use case Prevent recurrence of 64181-style state-corruption bugs in Serve's controller, where type annotations already encode the contract but nothing enforces it. ### Versions / Dependencies ray[serve] 2.56.0 ### Reproduction script mypy --follow-imports=silent --ignore-missing-imports ray/serve/_private/ ### Issue Severity None", "url": "https://github.com/ray-project/ray/issues/64643", "createdAt": "2026-07-09T19:33:09Z", "updatedAt": "2026-08-13T16:22:59Z", "timestamp": "2026-08-13T16:22:59Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "bug", "triage", "serve", "tech-debt" ], "author": "johntaylor-cell", "state": "open", "assignees": [ "johntaylor-cell" ] }, { "id": "github:ray-project/ray:issue:64693", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "issue", "title": "[Core] Removing an in-flight placement group can leak prepared bundles and block later placement groups", "text": "## What happened + What you expected to happen Removing a placement group while bundle preparation is in flight can leave both prepared raylet resources and the GCS placement-group scheduling gate stuck if one prepare callback never arrives. The scheduler already documents that a `PrepareBundleResources` callback may never be called after node death. In this sequence: 1. A placement group starts preparing bundles on two nodes. 2. Node A reports successful preparation, so GCS knows that bundle is reserved. 3. Node B dies and its prepare callback is withheld. 4. The placement group is removed and `MarkScheduleCancelled` records only the canceled state. 5. Cleanup and manager completion remain dependent on the missing callback. Actual behavior: - no `RemovePlacementGroupBundles` RPC is sent for the known prepared bundle; - the manager's `scheduling_in_progress_id_` remains owned by the removed group; - a later pending placement group is not scheduled. Expected behavior: - cancellation immediately cleans up every prepared bundle already known to GCS; - a successful prepare callback that arrives after cancellation is cleaned up immediately; - removing the active group releases its manager scheduling token without waiting for unrelated callbacks that may never arrive. ## Versions / Dependencies - Ray: `3.0.0.dev0`, commit `3d026f5b65e52955cb8b3e21d88941bb0ff5392c` - OS: Ubuntu 24.04, Linux 6.17, x86-64 - Python: 3.12.3 - Bazel: 7.5.0 - GCC: 13.3.0 ## Reproduction script Apply the two complete regression-test patches included at the end, then run: ```bash bazel test //src/ray/gcs/tests:gcs_placement_group_scheduler_test \\ --test_filter=GcsPlacementGroupSchedulerTest.PlacementGroupCancelledAfterPartialPrepareReleasesPreparedBundle \\ --test_output=errors bazel test //src/ray/gcs/tests:gcs_placement_group_manager_test \\ --test_filter=GcsPlacementGroupManagerTest.TestRemovingLeasingPlacementGroupUnblocksPending \\ --test_output=errors ``` The first regression test uses the existing fake raylet clients: ```cpp auto node0 = GenNodeInfo(0); auto node1 = GenNodeInfo(1); AddNode(node0); AddNode(node1); auto pg = std::make_shared( GenCreatePlacementGroupRequest(), \"\", counter_, clock_); const auto pg_id = pg->GetPlacementGroupID(); scheduler_->ScheduleUnplacedBundles(SchedulePgRequest{ pg, [this](auto pg, bool) { absl::MutexLock lock(&placement_group_requests_mutex_); failure_placement_groups_.push_back(std::move(pg)); }, [this](auto pg) { absl::MutexLock lock(&placement_group_requests_mutex_); success_placement_groups_.push_back(std::move(pg)); }}); ASSERT_TRUE(raylet_clients_[0]->GrantPrepareBundleResources()); RemoveNode(node1); // Its prepare callback never arrives. scheduler_->MarkScheduleCancelled(pg_id); ASSERT_TRUE(raylet_clients_[0]->GrantRemovePlacementGroupBundles()); ASSERT_EQ(raylet_clients_[0]->num_bundles_removed, 1); ``` Current `master` fails the cleanup assertion: ```text Value of: raylet_clients_[0]->GrantRemovePlacementGroupBundles() Actual: false Expected: true ``` The manager regression independently fails because the second placement group is never handed to the scheduler: ```text Expected equality of these values: mock_placement_group_scheduler_->GetPlacementGroupCount() Which is: 0 1 ``` ## Relevant implementation `GcsPlacementGroupScheduler::MarkScheduleCancelled` currently marks the lease tracker canceled but does not clean its already-known prepared locations. `OnAllBundlePrepareRequestReturned` performs cleanup only after every callback has returned. `GcsPlacementGroupManager::RemovePlacementGroup` records cancellation but does not release the scheduling token for that group. A candidate fix that cleans known prepared locations on cancellation, cleans a late successful prepare immediately, and clears only the matching manager token makes both regression tests pass. It also passes the existing nearby placement- group cancellation/removal tests. Historical issue #17258 and PR #17320 concern a different race between removal and a placement-group storage callback after scheduling had already succeeded. This report concerns partial prepare progress plus a callback that may never arrive. ## Issue Severity High: one removed placement group can leave resources reserved and prevent subsequent placement groups from being scheduled indefinitely. ## Complete scheduler regression-test patch ```diff diff --git a/src/ray/gcs/tests/gcs_placement_group_scheduler_test.cc b/src/ray/gcs/tests/gcs_placement_group_scheduler_test.cc index 8c7b8a4..3938f23 100644 --- a/src/ray/gcs/tests/gcs_placement_group_scheduler_test.cc +++ b/src/ray/gcs/tests/gcs_placement_group_scheduler_test.cc @@ -627,6 +627,41 @@ TEST_F(GcsPlacementGroupSchedulerTest, DestroyCancelledPlacementGroup) { WaitPlacementGroupPendingDone(1, GcsPlacementGroupStatus::FAILURE); } +TEST_F(GcsPlacementGroupSchedulerTest, + PlacementGroupCancelledAfterPartialPrepareReleasesPreparedBundle) { + auto node0 = GenNodeInfo(0); + auto node1 = GenNodeInfo(1); + AddNode(node0); + AddNode(node1); + ASSERT_EQ(2, gcs_node_manager_->GetAllAliveNodes().size()); + + auto create_placement_group_request = GenCreatePlacementGroupRequest(); + auto placement_group = std::make_shared( + create_placement_group_request, \"\", counter_, clock_); + const auto &placement_group_id = placement_group->GetPlacementGroupID(); + + scheduler_->ScheduleUnplacedBundles(SchedulePgRequest{ + placement_group, + [this](std::shared_ptr placement_group, bool is_insfeasble) { + absl::MutexLock lock(&placement_group_requests_mutex_); + failure_placement_groups_.emplace_back(std::move(placement_group)); + }, + [this](std::shared_ptr placement_group) { + absl::MutexLock lock(&placement_group_requests_mutex_); + success_placement_groups_.emplace_back(std::move(placement_group)); + }}); + + ASSERT_TRUE(raylet_clients_[0]->GrantPrepareBundleResources()); + RemoveNode(node1); + + scheduler_->MarkScheduleCancelled(placement_group_id); + + ASSERT_TRUE(raylet_clients_[0]->GrantRemovePlacementGroupBundles()); + ASSERT_EQ(raylet_clients_[0]->num_bundles_removed, 1); + ASSERT_EQ(raylet_clients_[0]->commit_callbacks.size(), 0); + ASSERT_EQ(raylet_clients_[1]->commit_callbacks.size(), 0); +} + TEST_F(GcsPlacementGroupSchedulerTest, PlacementGroupCancelledDuringCommit) { auto node0 = GenNodeInfo(0); auto node1 = GenNodeInfo(1); ``` ## Complete manager regression-test patch ```diff diff --git a/src/ray/gcs/tests/gcs_placement_group_manager_test.cc b/src/ray/gcs/tests/gcs_placement_group_manager_test.cc index 09ece01..d4df34c 100644 --- a/src/ray/gcs/tests/gcs_placement_group_manager_test.cc +++ b/src/ray/gcs/tests/gcs_placement_group_manager_test.cc @@ -547,6 +547,40 @@ TEST_F(GcsPlacementGroupManagerTest, TestRemovingLeasingPlacementGroup) { ASSERT_EQ(counter_->Get(rpc::PlacementGroupTableData::REMOVED), 1); } +TEST_F(GcsPlacementGroupManagerTest, TestRemovingLeasingPlacementGroupUnblocksPending) { + auto request1 = GenCreatePlacementGroupRequest(); + std::atomic registered_placement_group_count(0); + RegisterPlacementGroup(request1, [®istered_placement_group_count](Status status) { + ++registered_placement_group_count; + }); + ASSERT_EQ(registered_placement_group_count, 1); + ASSERT_EQ(mock_placement_group_scheduler_->GetPlacementGroupCount(), 1); + auto removed_placement_group = + mock_placement_group_scheduler_->placement_groups_.back(); + mock_placement_group_scheduler_->placement_groups_.clear(); + + auto request2 = GenCreatePlacementGroupRequest(); + RegisterPlacementGroup(request2, [®istered_placement_group_count](Status status) { + ++registered_placement_group_count; + }); + ASSERT_EQ(registered_placement_group_count, 2); + ASSERT_EQ(mock_placement_group_scheduler_->GetPlacementGroupCount(), 0); + + const auto &removed_placement_group_id = + removed_placement_group->GetPlacementGroupID(); + EXPECT_CALL(*mock_placement_group_scheduler_, + MarkScheduleCancelled(removed_placement_group_id)) + .Times(1); + gcs_placement_group_manager_->RemovePlacementGroup( + removed_placement_group_id, [](const Status &status) {}); + RunIOService(); + + gcs_placement_group_manager_->SchedulePendingPlacementGroups(); + ASSERT_EQ(mock_placement_group_scheduler_->GetPlacementGroupCount(), 1); + ASSERT_NE(mock_placement_group_scheduler_->placement_groups_[0]->GetPlacementGroupID(), + removed_placement_group_id); +} + TEST_F(GcsPlacementGroupManagerTest, TestRemovingCreatedPlacementGroup) { auto request = GenCreatePlacementGroupRequest(); std::atomic registered_placement_group_count(0); ```", "url": "https://github.com/ray-project/ray/issues/64693", "createdAt": "2026-07-11T22:11:09Z", "updatedAt": "2026-08-13T11:03:52Z", "timestamp": "2026-08-13T11:03:52Z", "metrics": { "reactions": 0, "comments": 2 }, "labels": [ "core", "stability", "community-backlog" ], "author": "logical-misha", "state": "open", "assignees": [] }, { "id": "github:ray-project/ray:issue:64992", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "issue", "title": "[Core] Provide a way to disable the worker-log `(pid=…)` prefix without disabling driver forwarding or overriding the root logger", "text": "### What happens With `log_to_driver=True` (the default), Ray prepends a prefix to every worker/actor stdout/stderr line it forwards to the driver: ``` (SegmentationActor pid=18654) {\"event\": \"segmenting batch\", \"job_id\": \"…\", \"level\": \"info\"} ``` Our workers emit **structured logs β€” one JSON object per line** β€” which a stdout-based collector ships to Loki/Grafana. The prepended `(… pid=…)` text means the line is **no longer valid JSON**, so structured parsing drops or mangles it. The logs are present in raw `kubectl logs` but effectively **invisible in the log backend** β€” a silent observability hole that is very expensive to notice (the pipeline \"goes quiet\" while thousands of lines land on pod stdout). The underlying issue: **Ray rewrites the bytes of a user's stdout/stderr lines by default.** Prepending text to a line silently breaks any line-oriented structured format. And there is no way to turn *only that* off. ### Why every existing option is unusable | Option | Result | |---|---| | `RAY_COLOR_PREFIX=0` | Removes only the ANSI color codes. The `(… pid=…)` text remains. | | `log_to_driver=False` | Removes the prefix β€” but also stops forwarding worker logs to the driver's stdout. They go to session files under `/tmp/ray/session_*/logs/`, which a stdout-based collector (the standard Kubernetes pattern) never sees. **The logs disappear entirely.** | | `logging_config=LoggingConfig(encoding=\"JSON\")` | Ray's own documented structured-logging recipe **pairs this with `log_to_driver=False`** (\"the redirected logs to the driver will include prefixes that made the logs not JSON parsable\") β€” i.e. it does **not** strip the prefix from forwarded logs either. Worse, it \"configures the **root loggers** for the driver process, Ray actors, and Ray tasks\" β€” it seizes the root logger, overrides the application's own logging configuration, and imposes Ray's JSON schema. For an app with an established structured-logging stack this is a non-starter β€” especially given Ray already tears down pre-configured logging handlers on import (#48846, #52371). | | `RAY_LOG_TO_STDERR=1` | Sends worker logs to the real (inherited) stderr with no prefix β€” but also dumps the raylet/GCS **C++ logs** (non-JSON) onto stderr, re-breaking the JSON stream, with known flooding problems (#47506). | So: **there is no way to keep worker logs on the driver's stdout, in the application's own format, without the prefix.** Every switch that removes the prefix either drops the logs or takes over the root logger. ### The request A first-class, **orthogonal** switch that keeps forwarding worker logs to the driver's stdout but leaves the **line content untouched** β€” for example: ```python ray.init(log_to_driver=True, worker_log_prefix=False) # or an env var, set before ray.init: # RAY_DISABLE_WORKER_LOG_PREFIX=1 ``` Log **forwarding** (a transport concern) and worker **attribution** (pid/actor identity) should not be entangled with the **byte content** of the user's log line. Attribution is already available structurally via `ray.get_runtime_context()` (`get_actor_id()` / `get_worker_id()` / `get_task_id()`); applications that want it can add it as a field or label. It should not be force-injected into the text of every line. Minimally: provide an opt-out for the prefix that does **not** require adopting `LoggingConfig` or disabling driver forwarding. More generally β€” a reasonable default would be to **not mutate the content of user log lines**; if the prefix stays the default, make it trivially disable-able in isolation. ### The switch already exists internally β€” it's just not exposed The prefix is added by `print_worker_logs()`, which already takes an `ignore_prefix` argument that skips it entirely (`python/ray/_private/worker.py`, v2.49.0): ```python def print_worker_logs(data, print_file, ignore_prefix: bool = False): # :2189 ... if ignore_prefix: # :2272 ... # emit the line unmodified, no \"(name pid=…)\" wrapper ``` But the only thing that ever sets it `True` is the presence of a job logging config β€” in both places that stream worker logs to the driver: ```python ignore_prefix = global_worker.job_logging_config is not None # :2632 ... ignore_prefix = worker.job_logging_config is not None # :2683 ``` So `ignore_prefix` is `True` **iff** a `LoggingConfig` is set β€” i.e. the capability we want is already implemented and working, but it can only be reached through the one option that also seizes the root logger. **Decoupling these two solves the whole problem with essentially no new machinery**: let `ignore_prefix` be set directly β€” a `ray.init(..., worker_log_prefix=False)` parameter, or an env var checked at lines 2632/2683 β€” instead of deriving it solely from `job_logging_config`. ### Environment - Ray 2.49.0, Python 3.13, Linux (Kubernetes / AKS) - Workers emit newline-delimited JSON to stdout; Grafana Alloy ships pod stdout to Loki - Current workaround: a regex `stage.replace` in the Alloy pipeline to strip the prefix before ingest β€” i.e. we are paying to undo, downstream, a mutation Ray made with no way to disable it. ### Related - #48846 β€” importing `ray.data` closes custom logging handlers - #52371 β€” import overwrites the ray logger; handlers deleted on init when `configure_logging=False` - #47506 β€” `RAY_LOG_TO_STDERR` floods the job API with head-node C++ logs - #21707 β€” support for directing *all* logs to stdout - #17467 β€” stdout logging for Ray head and worker nodes", "url": "https://github.com/ray-project/ray/issues/64992", "createdAt": "2026-07-24T14:07:54Z", "updatedAt": "2026-08-13T17:00:26Z", "timestamp": "2026-08-13T17:00:26Z", "metrics": { "reactions": 0, "comments": 6 }, "labels": [ "usability", "core", "community-backlog" ], "author": "Liquidmasl", "state": "open", "assignees": [] }, { "id": "github:ray-project/ray:issue:65259", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "issue", "title": "[serve][llm] Governance middleware layer for Ray Serve LLM β€” PII detection, cost budgets, policy enforcement, and audit trails", "text": "### Description Ray Serve LLM currently exposes an OpenAI-compatible API (chat completions, completions, embeddings) with no built-in mechanism for governance, guardrails, or policy enforcement at the serving layer. Teams deploying LLMs in production β€” especially in regulated industries (healthcare, finance, government) β€” need to add PII detection, cost budgets, tool authorization, and audit logging around every inference request. Today, the only way to achieve this is to write a custom Ray Serve deployment that wraps the engine directly (bypassing the standard `LLMConfig` + `build_openai_app` pipeline), losing all the benefits of `LLMRouter`, `LLMServer`, autoscaling, and multi-model management. **Proposed: A middleware/hook system for Ray Serve LLM** that allows governance logic to execute before and after inference β€” similar to ASGI middleware or Ray Serve's HTTP middleware, but specifically integrated into the LLM request pipeline. ```python from ray import serve from ray.serve.llm import LLMConfig, build_openai_app, LLMMiddleware class GovernanceMiddleware(LLMMiddleware): \"\"\"Deterministic governance β€” no LLM in the governance path, <2ms overhead.\"\"\" async def before_inference(self, request: ChatCompletionRequest, context: RequestContext) -> ChatCompletionRequest | BlockedResponse: # 1. Scan input for PII (regex-based, 40+ patterns) pii_findings = self.scan_pii(request.messages) if pii_findings and self.config.mode == \"ENFORCE\": return BlockedResponse(reason=\"PII detected\", findings=pii_findings) # 2. Check cost budget (per-user, per-session, per-model) if self.exceeds_budget(context.user_id, context.model): return BlockedResponse(reason=\"Budget exceeded\") # 3. Evaluate access policies (which users can use which models) if not self.policy_allows(context.user_id, request.model): return BlockedResponse(reason=\"Model access denied by policy\") return request # Allow request to proceed async def after_inference(self, request: ChatCompletionRequest, response: ChatCompletionResponse, context: RequestContext) -> ChatCompletionResponse: # 1. Scan output for PII/secrets leakage # 2. Record token usage against cost budget # 3. Emit structured audit receipt (JSON with correlation ID) self.record_usage(context, response.usage) self.emit_audit_receipt(context, request, response) return response # Usage: plug into standard LLMConfig pipeline llm_config = LLMConfig( model_loading_config={\"model_id\": \"gpt-serving\", \"model_source\": \"meta-llama/Llama-3-8B-Instruct\"}, accelerator_type=\"A10G\", ) app = build_openai_app({ \"llm_configs\": [llm_config], \"middleware\": [GovernanceMiddleware(config=governance_policy)], # <-- new }) serve.run(app, blocking=True) ``` **Key design principles:** - Middleware runs in the same process (no sidecar, no network hop) - Deterministic evaluation only (regex, policy rules) β€” no additional LLM call - <2ms overhead per request - Does not interfere with vLLM/SGLang engine internals - Composable: multiple middleware can chain (auth β†’ governance β†’ logging) ### Use case **1. Regulated industries deploying LLMs on Ray** Healthcare companies (HIPAA), financial services (SOX, PCI-DSS), and government agencies need to prove that every LLM interaction was governed. Today they either: - Build custom Ray Serve deployments from scratch (losing LLMRouter, autoscaling, multi-model support) - Add a separate proxy/gateway in front of Ray Serve (adding latency, operational complexity, and a failure point) A middleware layer inside Ray Serve LLM would let them use the standard `LLMConfig` pipeline with governance built in. **2. Multi-tenant LLM platforms** Companies serving multiple customers/teams from shared GPU infrastructure need: - Per-tenant cost budgets with hard enforcement (stop serving when budget is exhausted) - Per-tenant model access policies (tenant A gets Llama-70B, tenant B gets Llama-8B only) - PII isolation (scan inputs/outputs to prevent data leakage across tenants) The `LLMRouter` already handles routing, but there's no governance layer for access control or cost caps. **3. MCP tool governance in agentic deployments** As Ray Serve LLM adds tool-calling and agentic capabilities, there's a need to control which tools agents can invoke, validate tool arguments before execution, and scan tool outputs before they re-enter the context window. A middleware layer is the natural extension point for this. **4. Compliance evidence for SOC2/ISO 27001 audits** Enterprises need structured audit records proving that: - Every request was scanned for sensitive data - Access policies were evaluated deterministically - Cost budgets were enforced - Blocked requests have documented reasons Current Ray Serve metrics (Prometheus/Grafana) cover performance but not governance decisions. **Existing community demand:** The [Ray Discuss thread on preprocessing](https://discuss.ray.io/t/preprocessing-in-ray-serve-llm/23346) highlights that users already want more control over the request pipeline. A governance middleware would address this need specifically for security/compliance use cases. **I've built [TealTiger](https://github.com/agentguard-ai/tealtiger)** (Apache 2.0), a deterministic AI governance SDK already integrated with 15+ agent frameworks (LangChain, CrewAI, AG2, Haystack, n8n). Happy to contribute a Ray Serve LLM middleware integration that wraps TealTiger's governance engine. Key properties: - No LLM in the governance path β€” all evaluation is deterministic - <2ms latency overhead - 40+ PII detection patterns with confidence scoring - ENFORCE / MONITOR / REPORT_ONLY modes - Structured JSON decision receipts with OpenTelemetry-compatible trace IDs - Covers 7/10 OWASP AI Security Issues --- ## Labels `enhancement`, `triage`, `serve`", "url": "https://github.com/ray-project/ray/issues/65259", "createdAt": "2026-08-06T13:42:32Z", "updatedAt": "2026-08-13T02:09:08Z", "timestamp": "2026-08-13T02:09:08Z", "metrics": { "reactions": 0, "comments": 9 }, "labels": [ "enhancement", "serve", "security", "community-backlog" ], "author": "nagasatish007", "state": "open", "assignees": [ "richardliaw" ] }, { "id": "github:ray-project/ray:issue:65280", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "issue", "title": "[Data] OpTask._cancel never passes force=True", "text": "## Description `OpTask._cancel` never passes `force=True` to `ray.cancel`, on any path, so a blocked UDF keeps holding its CPU slot after execution fails. The actor-task check is the cause (`python/ray/data/_internal/execution/interfaces/physical_operator.py`): ```python is_actor_task = not self.get_task_id().actor_id().is_nil() ... force=force and not is_actor_task, ``` `TaskID::ForNormalTask` fills the actor-id slot with `ActorID::NilFromJob(job_id)` β€” nil unique bytes plus the job id β€” while `ActorID::Nil()` is all `0xff`. A job-scoped nil never equals it, so `is_nil()` is `False` for normal tasks too. `is_actor_task` is unconditionally `True` and `force and not is_actor_task` is always `False`. Ray Core classifies correctly β€” `CoreWorker::CancelTask` reads the task spec and calls `IsActorTask()`. This is a Ray Data–side issue only. **Expected:** `force=True` reaches `ray.cancel` for normal tasks and the CPU is released. **Actual:** every task is treated as an actor task and the blocked task holds its CPU indefinitely.", "url": "https://github.com/ray-project/ray/issues/65280", "createdAt": "2026-08-07T06:55:08Z", "updatedAt": "2026-08-12T19:09:21Z", "timestamp": "2026-08-12T19:09:21Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "data", "stability", "community-backlog" ], "author": "Hyunoh-Yeo", "state": "open", "assignees": [ "Hyunoh-Yeo" ] }, { "id": "github:ray-project/ray:issue:65300", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "issue", "title": "The task_id/put_index contract in `GetGeneratorReturnId` is unenforced (the RAY_CHECK is a tautology)", "text": "`WorkerContext::GetGeneratorReturnId` (`src/ray/core_worker/context.cc`) documents that the caller supplies both `task_id` and `put_index` or neither, because deducing only one of them keys the ObjectID to one task while drawing the index from another. The `RAY_CHECK` meant to enforce that is a tautology: ```cpp RAY_CHECK((task_id.IsNil() && !put_index.has_value()) || (!task_id.IsNil() || put_index.has_value())); ``` With A = `task_id.IsNil()` and B = `put_index.has_value()`, the second term `!A || B` is false only when `A && !B`, and in exactly that case the first term `A && !B` is true. The disjunction holds for all four inputs, so the check has never fired since it was added in #35584. The unguarded combination that matters is a real `task_id` with no `put_index`: it takes the index from this thread's put counter, which belongs to whatever task this thread is running, so the resulting ObjectID can collide with one that task mints. `CoreWorker::AllocateDynamicReturnId` (`src/ray/core_worker/core_worker.h`) has a related problem in the same contract. Its `put_index` default is `-1`, and `ObjectIDIndexType` is `uint32_t`, so that default is an optional holding `UINT32_MAX`, not the `std::nullopt` its docstring describes. A caller using the default would abort: the two defaults together are the illegal Nil-plus-index combination, and even past the check, 4294967295 exceeds the 1e8 generator-return limit. Neither is reachable today. `libcoreworker.pxd` declares all three parameters without defaults so Cython cannot omit one, there is no C++ caller, and all four call sites in `_raylet.pyx` pass a legal combination. So this is about making the contract hold for the next caller, not about a bug users are hitting. I have a fix and tests ready.", "url": "https://github.com/ray-project/ray/issues/65300", "createdAt": "2026-08-08T18:38:41Z", "updatedAt": "2026-08-12T20:02:55Z", "timestamp": "2026-08-12T20:02:55Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "core", "tech-debt", "stability", "community-backlog" ], "author": "LuciferYang", "state": "closed", "assignees": [] }, { "id": "github:ray-project/ray:issue:65350", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "issue", "title": "[Data] read_webdataset emits one DataFrame per sample, causing per-sample size_bytes overhead in the output buffer", "text": "### Problem `WebDatasetDatasource._read_stream` yields one single-row pandas DataFrame per sample. Ray's output buffer estimates block memory on every emitted block: ``` BlockOutputBuffer.has_next -> _exceeded_buffer_size_limit -> get_estimated_memory_usage -> PandasBlockAccessor.size_bytes ``` So with N samples per shard, this deep `size_bytes` pass runs ~N times per shard. PandasBlockAccessor.size_bytes runs once per sample, so it accounts for ~63% of the ReadWebDataset operator's CPU time (~2115s of 3349s per epoch), spent purely in size estimation rather than reading, according to our repro. So with N samples per shard, this deep `size_bytes` pass runs ~N times per shard. ### Reproduction Read a WebDataset shard through `read_webdataset(decoder=None)` at scale and instrument the read worker (counting `PandasBlockAccessor.size_bytes` calls and CPU), or read `ds.stats()`. On a 2000-sample shard, current master runs ~2000 `size_bytes` calls, ~3s of worker CPU, purely in size estimation rather than reading. ### Environment - **Node**: single node, g6e.4xlarge (1x NVIDIA L40S 48 GB, 13 CPU, 64 GB RAM) - **Ray**: 2.55 - **Data**: 641 train + 25 val WebDataset shards, 2000 samples/shard (1.28M / 50k), 1000 classes, `jpg` + `cls`, on AWS S3 - **Read**: `read_webdataset(decoder=None, concurrency=11, override_num_blocks=641)` - **Preprocess**: `map_batches` (PIL decode + RandomResizedCrop(224) + normalize), batch 256 - **Train**: `TorchTrainer` resnet18, global batch 512, one epoch with validation - **Compared**: current master vs a chunked reader that emits one DataFrame per 512 samples ### Benchmark | metric | current master | chunked (512) | | --- | --- | --- | | `size_bytes` share of ReadWebDataset CPU | ~63% (2115s / 3349s) | negligible | | read remote CPU per epoch | 3349 s | 1234 s (-63%) | | read throughput | 1170 rows/s | 1444 rows/s | | whole epoch (train + val) | 1238 s | 998 s | Corroboration: on an earlier cluster run steady throughput went 1287 -> 1604 img/s and time-to-first-batch 16.1 s -> 9.2 s. The reporting user independently reproduced this and measured read CPU ~4 s -> ~1 s per shard. ### Proposed fix Accumulate ~512 samples per emitted DataFrame in `_read_stream`, so the output buffer runs the memory-size estimation ~512x less. Output is byte-identical (verified via SHA256, ordered and multiset, with default and raw decoders). PR to follow.", "url": "https://github.com/ray-project/ray/issues/65350", "createdAt": "2026-08-10T23:20:05Z", "updatedAt": "2026-08-12T20:34:51Z", "timestamp": "2026-08-12T20:34:51Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "performance", "data", "community-backlog" ], "author": "Hyunoh-Yeo", "state": "closed", "assignees": [] }, { "id": "github:ray-project/ray:issue:65352", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "issue", "title": "[Umbrella] Ray Sandboxing with gVisor", "text": "### Description This is an umbrella issue to track Ray sandboxing support using gVisor. PRs: - [x] https://github.com/ray-project/ray/pull/64964 TODO: - [ ] add support for referencing local tar files for images (specifically with tar files created with `docker save`) - [ ] add modal compatible API for sandboxing - [ ] root cause why --ignore-cgroups is required in CI - [ ] add install script for runsc tests (see https://github.com/ray-project/ray/pull/64964#discussion_r3754281842) - [ ] pin the gVIsor version in tests (see https://github.com/ray-project/ray/pull/64964#discussion_r3754232229) - [ ] explore using third party library to managing container images for sandboxes - [ ] explore managing Python dependencies in sandboxes https://github.com/ray-project/ray/issues/65322 ### Use case _No response_", "url": "https://github.com/ray-project/ray/issues/65352", "createdAt": "2026-08-11T00:28:50Z", "updatedAt": "2026-08-13T00:29:50Z", "timestamp": "2026-08-13T00:29:50Z", "metrics": { "reactions": 2, "comments": 7 }, "labels": [ "enhancement", "triage", "core", "security", "community-backlog" ], "author": "andrewsykim", "state": "open", "assignees": [] }, { "id": "github:ray-project/ray:issue:65427", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "issue", "title": "[Docs] Run pre-commit hooks on doc/source: a per-directory ratchet", "text": "## Why reopen this [#42587](https://github.com/ray-project/ray/issues/42587) asked for pre-commit hooks to run on `doc/`. It was closed as completed, but the configuration change it asked for was never made. I closed it, and that was my error. What actually happened: six PRs in Aug–Sep 2024 ([#47004](https://github.com/ray-project/ray/pull/47004), [#47005](https://github.com/ray-project/ray/pull/47005), [#47107](https://github.com/ray-project/ray/pull/47107), [#47108](https://github.com/ray-project/ray/pull/47108), [#47341](https://github.com/ray-project/ray/pull/47341), [#47342](https://github.com/ray-project/ray/pull/47342)) ran one-shot bulk reformats over `ray-core/`, `rllib/`, `train/`, `tune/`, `data/`, and `cluster/` by temporarily lifting the global exclude. None of them touched `.pre-commit-config.yaml`. So the trees were cleaned once and then left ungated, and they've drifted since. I'd like agreement on an approach before sending more configuration PRs, rather than landing them one at a time and discovering on the fifth that the direction is wrong. ## Current state `.pre-commit-config.yaml` has a top-level `exclude` covering `doc/source/`. pre-commit ANDs that global `exclude` with each hook's own `files` pattern, so any hook scoped inside `doc/source/` matches nothing at all. That has produced silently dead hooks rather than merely absent ones, which is the part worth flagging: - The `vale` hook declared `files: ^doc/source/data/.*\\.(md|rst)$` and matched zero files for its entire life. Fix open for review in [#65375](https://github.com/ray-project/ray/pull/65375). - The `semgrep` hook's `code-block-python` rule names five `doc/source` paths in `paths.include` and has never run on any of them. There are 31 occurrences across 16 files that the rule was written to prevent. Not yet fixed. - `rst-directive-colons` and `rst-inline-touching-normal` from `pygrep-hooks` are configured, apply to `.rst` by default, and are skipped for all 362 `.rst` files under `doc/source`. A dead hook is worse than a missing one: `pre-commit run --all-files` comes back green and reads as coverage. Coverage today, counting `.md` and `.rst` under `doc/source`: | | Files | | --- | --- | | `.rst` | 362 | | `.md` | 229 | | Total | 591 | | Gated by CI Vale today (`doc/source/data` + `ray-overview/examples`) | 49 | | Covered by the pre-commit Vale hook, once #65375 lands | 48 | Note those are two separate narrow scopes that already disagree with each other, which is its own drift problem and the first item in the proposal below. ## Evidence the gate is worth having `doc/source/data` is the one directory Vale has gated in CI. Running Vale over all of `doc/source` on the repo's own configuration, error-level alerts per file: | Directory | Files | Errors | Errors/file | | --- | --- | --- | --- | | `data` (gated) | 27 | 11 | **0.4** | | `train` | 37 | 330 | 8.9 | | `cluster` | 104 | 1278 | 12.3 | | `ray-core` | 81 | 1212 | 15.0 | | `serve` | 79 | 1128 | 14.3 | | `tune` | 42 | 730 | 17.4 | | `ray-contribute` | 18 | 347 | 19.3 | The gated directory is roughly thirty times cleaner than comparable ungated ones. The gate works; it just isn't pointed at much. (Measured with a newer local Vale than CI's pin, so treat these as magnitudes for planning rather than targets.) ## Proposal Ratchet per directory, and never add a path to the gate that isn't already clean: 1. **Fix the invocation drift first.** `ci/lint/check-documentation-style.sh` lints `doc/source/data` plus `doc/source/ray-overview/examples`; the pre-commit hook covers only the first. Two hardcoded lists that already disagree. Give them one shared definition so every later step edits one place. 2. **Clear the vocabulary backlog.** `Vale.Spelling` is the largest single bucket and is mostly accept-list entries rather than prose defects. Repo-wide, no gate change, no rewriting. 3. **Extend directory by directory**, clearing that directory's errors in the same PR or an immediately preceding one, so `master` is never red. 4. **Order by ownership.** The docs-team-owned directories are small and need no other team's sign-off: `ray-contribute`, `ray-overview`, `ray-security`, `ray-governance`, `ray-air`, `ray-more-libs`, `_templates`, `templates`. Roughly 54 files. The large library directories need the owning team's buy-in on a prose sweep and should be separate per-team PRs. The already-configured hooks (whitespace, end-of-file, the two rST checks) can ride along per directory at no marginal cost, since they're in the config already. ## Questions for maintainers 1. **Ratchet or baseline?** Widening the invocation directory by directory keeps `master` green by construction and needs no new tooling. The alternative is opening all of `doc/source` at once behind a baseline file that grandfathers existing alerts, which gets new-file coverage immediately but adds a file that rots. I lean ratchet. Happy to go the other way if you'd rather have coverage sooner. 2. **Is the `code-block-python` semgrep rule still wanted for docs?** It was written deliberately and points at a real contributor guide, but it's been dormant long enough that the answer isn't obvious. Converting the 31 occurrences to `testcode` makes them execute in CI, so some will need fixtures. Worth sizing per path if yes. 3. **Notebook policy.** `doc/source` ships 59 executed `.ipynb` files with embedded outputs. That complicates any strip-on-commit hook. Is there an existing preference here? 4. **PR granularity.** The 2024 thread preferred small subtree-scoped PRs to limit CODEOWNERS blast radius. Assuming that still holds unless you say otherwise. ## Related, all currently open None of these are blocked on this issue, and I'm not asking anyone to hold them for it. They're the first increment of the ratchet above, and reviewing them doesn't commit anyone to the wider plan. - [#65375](https://github.com/ray-project/ray/pull/65375) β€” makes the Vale hook actually run, and upgrades Vale 3.4.1 β†’ 3.17.1. CI green, awaiting review. This is effectively step 1 of the proposal, scoped to one directory. - [#65368](https://github.com/ray-project/ray/pull/65368) β€” removes the `banned_words` grep in favor of Vale. Awaiting review. - [#65425](https://github.com/ray-project/ray/pull/65425) β€” whitespace cleanup on `doc/source/data`, the fallout from #65375. Draft, stacked on #65375. If the answer to question 1 is \"baseline, not ratchet,\" #65375 and #65368 still stand on their own and the change would only affect what comes after them.", "url": "https://github.com/ray-project/ray/issues/65427", "createdAt": "2026-08-12T14:48:15Z", "updatedAt": "2026-08-12T14:48:15Z", "timestamp": "2026-08-12T14:48:15Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "docs" ], "author": "dstrodtman", "state": "open", "assignees": [], "change": "updated" }, { "id": "github:ray-project/ray:issue:65433", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "issue", "title": "[data] Fair-share allocation overstates runnable task-pool demand", "text": "### What happened + What you expected to happen Ray Data's fair-share allocator uses a task-pool operator's configured maximum concurrency as its maximum resource demand. When no maximum is configured, the reported demand is unbounded even if the operator currently has only one input block and can only launch one task. This leaves resources assigned to an operator that cannot use them. In a local 8-CPU run, a single runnable `StreamingRepartition` task was allocated as many as 4 CPUs while runnable downstream work received 6 of the 7 CPUs it could use. I expected the task pool's fair-share allocation to reflect its currently runnable work and to be recalculated when more inputs arrive. This should not change configured concurrency limits or actor-pool prewarming. Ray tasks are not preemptive, so there is a trade-off: if downstream work borrows the capacity, newly runnable upstream work may wait for an already-running downstream task to finish. The allocator should stop admitting more downstream work once the upstream demand returns. ### Versions / Dependencies Ray master at `1d73f34d61ea292d1acee871c56b934951553393`. ### Reproduction script This reproduces the pipeline shape where one file-partitioning task returns many work items in one block: ```python from dataclasses import dataclass import ray @dataclass class FilePartitioningTask: files: list[str] def partition_files(_): return { \"item\": [ FilePartitioningTask(files=[f\"file-{i}.jsonl\"]) for i in range(20) ] } class ProcessFile: def __call__(self, batch): return batch ray.init(num_cpus=8) ds = ( ray.data.from_items([None]) .map_batches( partition_files, batch_size=1, compute=ray.data.TaskPoolStrategy(size=1), num_cpus=1, ) .repartition(target_num_rows_per_block=1) .map_batches( ProcessFile, batch_size=1, compute=ray.data.ActorPoolStrategy(size=7), num_cpus=1, ) ) assert ds.materialize().count() == 20 ``` The first map runs once and returns 20 dataclass rows in one block. `StreamingRepartition` receives that single block, so it can launch one task. That task emits 20 one-row blocks that the downstream actor pool can process concurrently. This is a fair-share accounting problem rather than a `StreamingRepartition` concurrency problem: additional CPUs cannot make the one runnable repartition task execute in parallel. ### Issue Severity Medium: the pipeline continues to make progress, but runnable downstream work can be under-allocated while another operator holds CPU or GPU budget it cannot currently use.", "url": "https://github.com/ray-project/ray/issues/65433", "createdAt": "2026-08-12T19:42:14Z", "updatedAt": "2026-08-13T01:34:47Z", "timestamp": "2026-08-13T01:34:47Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "performance", "data", "community-backlog" ], "author": "praateekmahajan", "state": "open", "assignees": [] }, { "id": "github:ray-project/ray:issue:65434", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "issue", "title": "[data] BlockOutputBuffer rebuilds the remainder for every row-sized output", "text": "### What happened + What you expected to happen `StreamingRepartition(target_num_rows_per_block=1)` can spend a noticeable amount of time turning one large input block into many one-row output blocks. The row-sized path in `BlockOutputBuffer.next()` builds the buffered block, slices off the next output, and puts the remainder back into a new builder. It repeats that for every output block. The remainder is no longer copied, but the builder is still rebuilt once per emitted block. This is especially visible in fanout pipelines shaped like: ``` File partitioning -> StreamingRepartition(target_num_rows_per_block=1) -> map stages ``` On my machine with current master: - Arrow block, 100,000 rows: about 0.75 seconds (133k output blocks/sec) - Pandas block, 30,000 rows: about 2.29 seconds (13k output blocks/sec) I expected the row-only path to build the input block once and advance through it by row offset. It still needs to preserve ordering, partial tails across multiple input blocks, and the existing byte-based behavior. ### Versions / Dependencies Ray master at `1d73f34d61ea292d1acee871c56b934951553393`. ### Reproduction script A focused unit test can count calls to `DelegatingBlockBuilder.build()`: 1. Add a 100-row Arrow or Pandas block to a `BlockOutputBuffer` with `target_max_block_size_rows=1`. 2. Finalize the buffer and drain all 100 outputs. 3. Assert that the builder is built once rather than once per output block. The current implementation builds it 100 times. ### Issue Severity Medium: repartitioning still completes, but output delivery is slower and downstream stages take longer to receive enough work to scale up.", "url": "https://github.com/ray-project/ray/issues/65434", "createdAt": "2026-08-12T19:42:15Z", "updatedAt": "2026-08-13T01:34:15Z", "timestamp": "2026-08-13T01:34:15Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "performance", "data", "community-backlog" ], "author": "praateekmahajan", "state": "open", "assignees": [] }, { "id": "github:ray-project/ray:issue:65443", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "issue", "title": "[Train] Share PlacementGroupCleaner across concurrent Train v2 runs", "text": "### Description Ray Train v2 currently creates one detached `PlacementGroupCleaner` actor for each Train controller. The cleaner removes placement groups when its controller exits ungracefully. Since #64705, every cleaner is pinned to the head node so it does not prevent worker-node scale-down. However, this concentrates one Python cleaner process per concurrent Train run on the head node: ```text N concurrent Train runs -> N detached PlacementGroupCleaner processes ``` In our production workload, each `PlacementGroupCleaner` process consistently uses approximately 600 MB of memory. With 24 concurrent Train jobs, the cleaner processes alone consume approximately 14.4 GB on the head node. The cleaner's responsibility is cluster-level bookkeeping and does not require a dedicated process for every Train run. We propose replacing the per-controller cleaners with one cluster-wide shared cleaner: ```text 24 concurrent Train runs -> 1 shared PlacementGroupCleaner process ``` This makes the cleaner process and memory cost constant instead of growing linearly with the number of concurrent Train jobs. ### Proposed implementation Create one cluster-wide `PlacementGroupCleaner` with: - a fixed, versioned actor name; - a fixed internal namespace; - `get_if_exists=True`; - `lifetime=\"detached\"`; - head-node resource pinning; - `scheduling_strategy=\"DEFAULT\"`; - automatic actor restart. Every Train controller registers itself and its placement groups with the shared actor. The cleaner maintains controller-scoped records: ```python controller_actor_id -> placement_group_ids ``` The lifecycle is: - controller starts: register the controller with the shared cleaner; - worker group starts: register its placement group under that controller; - controller shuts down normally: unregister only that controller without stopping the shared cleaner; - controller dies unexpectedly: remove only the placement groups registered to that controller; - the registry becomes empty: keep the cleaner alive to avoid a race between actor shutdown and a new Train job registering. The actor name is versioned, for example: ```text train_v2_placement_group_cleaner_v1 ``` This prevents a newer Ray version from accidentally reusing an incompatible shared actor created by an older implementation. ### Reliability Sharing the cleaner must not weaken the cleanup guarantees of the current per-controller implementation. The implementation therefore handles the following cases: 1. **Concurrent creation across jobs** Controllers use the same actor name and namespace with `get_if_exists=True`. Concurrent Train jobs resolve to the same actor. 2. **Controller isolation** Placement groups are stored by controller ID. A failed controller causes only its own placement groups to be removed. 3. **Registration and controller-death races** A controller is atomically marked as being cleaned before its placement groups are processed. A delayed placement-group registration is rejected, and the controller callback removes that placement group directly. 4. **Cleaner restart recovery** Controller IDs, placement-group IDs, and cleanup state are checkpointed in GCS internal KV. After an unexpected cleaner restart, the actor restores the registry and resumes incomplete cleanup. This avoids losing cleanup responsibility for all active Train runs when the shared process restarts. 5. **Partial cleanup failures** Cleanup state is removed only after every associated placement group has been removed successfully. Failed removals remain persisted and are retried. 6. **Failure isolation** A State API failure or placement-group cleanup failure for one controller does not stop the cleaner from processing other controllers. ### Prototype We have implemented and tested a Python-only prototype based on the current Ray `master`. It does not require Core Worker or C++ changes. The prototype modifies: - `placement_group_callback.py` - `placement_group_cleaner.py` - `test_placement_group_cleaner.py` It preserves the existing scheduling behavior introduced by #64705: - detached lifetime; - head-node pinning; - `DEFAULT` scheduling to escape placement-group capture. ### Test coverage The tests cover: - concurrent singleton creation; - singleton reuse across independent Ray drivers/jobs; - multiple controllers and multiple placement groups; - cleanup isolation between controllers; - normal controller deregistration; - duplicate registration, deregistration, and monitoring startup; - placement-group registration versus controller-death races; - temporary State API failures; - individual placement-group cleanup failures; - cleaner restart with active registrations; - cleaner restart during incomplete cleanup; - fixed actor name and namespace; - detached lifetime; - head-node pinning; - `DEFAULT` scheduling; - existing controller callback behavior. Current local results against Ray `master`: ```text 13 placement-group cleaner tests passed 10 controller callback regression tests passed Ruff check passed Ruff format check passed git diff --check passed ``` ### Expected impact For concurrent Train workloads, this changes the number of cleaner processes from O(number of Train runs) to O(1). For our workload: ```text Before: 24 cleaner processes x approximately 600 MB = approximately 14.4 GB After: 1 shared cleaner process x approximately 600 MB = approximately 600 MB ``` This reduces the observed head-node cleaner memory footprint by approximately 13.8 GB while retaining controller-scoped placement-group cleanup and restart recovery. ### Related work - #58515 introduced `PlacementGroupCleaner`. - #61756 fixed cleaner lifecycle behavior. - #62754 fixed the placement-group registration/controller-death race. - #64705 pinned cleaners to the head node and noted that N concurrent Train runs result in N cleaner pollers on the head node. This proposal addresses the remaining per-run process and memory scaling cost.", "url": "https://github.com/ray-project/ray/issues/65443", "createdAt": "2026-08-13T01:52:13Z", "updatedAt": "2026-08-13T17:44:36Z", "timestamp": "2026-08-13T17:44:36Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "train", "performance", "community-backlog" ], "author": "OneSizeFitsQuorum", "state": "open", "assignees": [] }, { "id": "github:ray-project/ray:issue:65444", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "issue", "title": "[Core] Randomize worker port allocation to reduce deterministic collisions between raylets", "text": "### What happened + What you expected to happen When multiple raylets run in the same host network namespace with the same worker port range, each `WorkerPool` builds its free-port pool in ascending order and allocates from the lower bound. As a result, the raylets deterministically attempt to use the same initial ports, even when most of the configured range is available. Repeated bind failures can delay or prevent workers from starting in host-network, multi-raylet deployments. We expect each raylet to shuffle its validated worker port list once during startup before placing the ports into the free-port pool. The behavior should preserve the following properties: - Ports remain unique within each raylet's pool. - All ports remain within the configured range. - An explicit `worker_port_list` is validated before it is shuffled. - A bind failure skips or retries the occupied port as it does today. - No port outside the configured range is selected. Randomization is intended to reduce deterministic collisions. It is not a cross-raylet port reservation protocol and cannot guarantee that two raylets never select the same port. ### Versions / Dependencies - Ray: community `master` - OS: Linux - Deployment: multiple raylets sharing one host network namespace ### Reproduction script 1. Start two raylets or Ray clusters in the same host network namespace. 2. Configure both with the same worker port range, for example `20000-20100`. 3. Assign distinct control-plane, dashboard, object-manager, node-manager, and Ray Client ports so those services do not collide. 4. Concurrently launch enough workers from both raylets to exercise the shared worker port range. 5. Observe the worker port selection order and bind failures. Current behavior: both raylets walk upward from port `20000`, producing deterministic collisions at the beginning of the range. Expected behavior: each raylet starts with an independently shuffled port order. Occupied ports are skipped or retried, and a raylet never returns duplicate or out-of-range ports from its own pool. A regression test could inject a fixed random seed and verify that: - the resulting pool is a permutation of the validated input ports; - allocation does not always begin at the lower bound; - duplicate and out-of-range ports are rejected; - occupied ports still follow the existing bind-failure retry path. ### Issue Severity Medium: It is a significant difficulty but I can work around it.", "url": "https://github.com/ray-project/ray/issues/65444", "createdAt": "2026-08-13T02:52:18Z", "updatedAt": "2026-08-13T07:32:08Z", "timestamp": "2026-08-13T07:32:08Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "bug", "triage", "core", "stability", "community-backlog" ], "author": "zzchun", "state": "open", "assignees": [] }, { "id": "github:ray-project/ray:issue:65449", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "issue", "title": "[Data] ObjectRefs passed to map UDFs through `fn_args` are not dereferenced", "text": "### What happened + What you expected to happen A direct `ObjectRef` passed as an element of `fn_args` reaches the UDF as an `ObjectRef` instead of its resolved value. Because the reference is captured in the serialized UDF wrapper, Ray's scheduler also cannot wait for the dependency before scheduling the map task. Expected behavior: - Direct `ObjectRef` elements of `fn_args` are top-level map-task arguments. - Ray waits for those dependencies before running the task. - The UDF receives the resolved values. PR #49208 added the map-task kwargs plumbing needed to pass top-level task arguments, but did not connect Dataset UDF arguments to that path. ### Versions / Dependencies still can reproduce in `master` branch currently. ### Reproduction script ```python import ray def map_fn(row, arg): assert arg == 1 return row ray.data.range(1).map( map_fn, fn_args=(ray.put(1),), ).take_all() ``` Before the fix, the assertion fails because `arg` is an `ObjectRef`. ### Issue Severity Medium: It is a significant difficulty but I can work around it.", "url": "https://github.com/ray-project/ray/issues/65449", "createdAt": "2026-08-13T06:31:41Z", "updatedAt": "2026-08-13T08:06:48Z", "timestamp": "2026-08-13T08:06:48Z", "metrics": { "reactions": 0, "comments": 3 }, "labels": [ "bug", "data", "stability", "community-backlog" ], "author": "viiccwen", "state": "open", "assignees": [ "viiccwen" ] }, { "id": "github:ray-project/ray:issue:65451", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "issue", "title": "[Core][runtime_env] RuntimeEnv agent leaks one logger (and its file descriptors) per job, eventually failing all setups with EMFILE`", "text": "### What happened The RuntimeEnv Agent caches one logger per job in `_per_job_logger_cache` (`python/ray/_private/runtime_env/agent/runtime_env_agent.py:209`), created via `setup_component_logger` in `get_or_create_logger` (`:304-313`). Each logger eagerly opens one `RotatingFileHandler` for `runtime_env_setup-.log` plus one per entry in the runtime env config's `log_files` β€” i.e. `1 + len(log_files)` file descriptors per job. Nothing ever releases them: - The cache has no eviction and no `close()` call anywhere in the file. - The logger is created through `logging.getLogger` (`ray_logging/__init__.py:80`), so the global logging registry holds a second strong reference β€” even clearing the cache would not free the handlers. - The env-level result cache does not bound the growth: `ReferenceTable` deletes a cached env when its last reference goes away (`runtime_env_agent.py:286-302`), so the next job re-runs setup under a new `job_id` and opens a fresh logger. Open FDs therefore grow with the **cumulative number of jobs the node has ever handled**, not with current setup concurrency. Restarting the agent is the only reset. Once the process `nofile` limit is reached, the agent can no longer create log files, download packages, create temp files, or start subprocesses, and every subsequent runtime env setup fails with: ``` OSError: [Errno 24] Too many open files: '/tmp/ray/session_.../logs/runtime_env_setup-.log' ```", "url": "https://github.com/ray-project/ray/issues/65451", "createdAt": "2026-08-13T08:07:01Z", "updatedAt": "2026-08-13T13:38:03Z", "timestamp": "2026-08-13T13:38:03Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "core", "stability", "community-backlog" ], "author": "zzchun", "state": "open", "assignees": [] }, { "id": "github:ray-project/ray:issue:65454", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "issue", "title": "[Core][KubeRay] Autoscaler sends all log records, including INFO, to stderr", "text": "## What happened + What you expected to happen `_setup_logging()` in `python/ray/autoscaler/_private/kuberay/run_autoscaler.py` attaches a single `logging._StderrHandler` to the root logger, so **every** autoscaler log record β€” INFO included β€” is written to the container's **stderr**. The function's own docstring says it should be stdout: ```python def _setup_logging(log_dir: str) -> None: \"\"\"Log to autoscaler log file (typically, /tmp/ray/session_latest/logs/monitor.*) Also log to pod stdout (logs viewable with `kubectl logs -c autoscaler`). ``` but the implementation writes stderr, and nothing writes to stdout at all: ```python level = logging.getLevelName(ray_constants.LOGGER_LEVEL.upper()) # default \"info\" stderr_handler = logging._StderrHandler() stderr_handler.setFormatter(logging.Formatter(ray_constants.LOGGER_FORMAT)) stderr_handler.setLevel(level) logging.root.setLevel(level) logging.root.addHandler(stderr_handler) # The stdout handler was set up in the Ray CLI entry point. # See ray.scripts.scripts::cli(). ``` **That trailing comment is stale.** `ray.scripts.scripts::cli()` calls `ray._private.ray_logging.setup_logger()`, which only re-sets the level and re-formats handlers already present on the `ray` logger β€” it never calls `addHandler`. And `setup_component_logger()`, called a few lines earlier in this same function, calls `ray._private.log.clear_logger(\"ray\")`, which *empties* the `ray` logger's handlers. The reproduction script below prints the resulting handler list; no handler targets stdout. ### Why this matters Kubernetes' CRI log format records the stream each line arrived on, per line: ``` TIMESTAMP STREAM TAG CONTENT 2016-10-06T00:17:09.669794202Z stdout P log content 1 ``` () Log collectors commonly derive a line's severity from that `STREAM` field rather than parsing the `INFO` token out of the message body, so on a KubeRay cluster with log collection routine autoscaler output is ingested as errors. The measurement below is consistent with that. Measured on a production KubeRay cluster (Datadog), autoscaler sidecar, **idle** cluster, 20 minutes: | | count | |---|---| | `status:error` | 2392 | | `status:info` | 0 | | `status:warn` | 0 | Over 24h the same container logged 63,048 `skipping idle termination` lines and zero scale events, drains, patches or exceptions. Every one of those lines was classified as an error, so the container's error rate carries no signal and cannot be alerted on. ### Expected Sub-WARNING records on stdout, WARNING and above on stderr β€” the conventional split β€” so severity survives log collection. `monitor.log` should keep receiving every level, since it backs the dashboard and the State API (`/api/v0/logs`). ### Origin This looks like an unintended regression from #34182 (\"Global logging format changes\"). Before that PR the code read: ```python logger_name=\"ray\", # Root of the logging hierarchy for Ray code. ) # Logs will also be written to the container's stdout. # The stdout handler was set up in the Ray CLI entry point. ``` #34182 removed the `# Logs will also be written to the container's stdout.` line, added the `_StderrHandler`, and left the `# The stdout handler was set up in the Ray CLI entry point.` line dangling. The intent recorded in the original comment (and still in the docstring) was stdout. Note that #33652, which introduced `logging._StderrHandler` for Rich `FileProxy` compatibility, touched only `python/ray/_private/ray_logging.py` β€” it never touched `run_autoscaler.py`, and it was not a severity-routing decision. Same bug class as #32339 (\"[Serve][Dashboard] Ray Serve actor logs show up in .err files instead of .out files\"), which was accepted as `bug`/`P2`/`observability`. Adjacent to #48651. ## Versions / Dependencies - Ray: reproduced on `master` (`9c6ffee9a5`) and on the `3.0.0.dev0` nightly (`eb5882b7ba`). Present since #34182 (Ray 2.4). - Python: 3.12 - OS: reproduced on macOS locally; observed in production on Linux / KubeRay. ## Reproduction script No cluster required β€” this calls the logging setup directly and reports which stream each level lands on. ```python import io, logging, os, sys, tempfile from ray.autoscaler._private.kuberay.run_autoscaler import _setup_logging logging.root.handlers.clear() tmp = tempfile.mkdtemp() out, err = io.StringIO(), io.StringIO() real_out, real_err = sys.stdout, sys.stderr sys.stdout, sys.stderr = out, err try: _setup_logging(tmp) log = logging.getLogger(\"ray.autoscaler._private.kuberay.run_autoscaler\") log.info(\"INFO-MARKER skipping idle termination\") log.warning(\"WARN-MARKER\") for h in logging.root.handlers: h.flush() finally: sys.stdout, sys.stderr = real_out, real_err print(\"root handlers:\", [type(h).__name__ for h in logging.root.handlers]) print(\"stdout bytes:\", len(out.getvalue())) print(\"INFO on stdout?\", \"INFO-MARKER\" in out.getvalue()) print(\"INFO on stderr?\", \"INFO-MARKER\" in err.getvalue()) print(\"INFO in monitor.log?\", \"INFO-MARKER\" in open(os.path.join(tmp, \"monitor.log\")).read()) ``` Actual output: ``` root handlers: ['RotatingFileHandler', '_StderrHandler'] stdout bytes: 0 INFO on stdout? False INFO on stderr? True INFO in monitor.log? True ``` Expected: `INFO on stdout? True`, `INFO on stderr? False`, `monitor.log` unchanged. ### Why `RAY_LOGGER_LEVEL=warning` is not a workaround Raising the level silences `monitor.log` too. `setup_component_logger()` is gated on the same `ray_constants.LOGGER_LEVEL`, and `logging.root.setLevel(level)` runs after it, so INFO calls short-circuit in `Logger.isEnabledFor` and never reach the rotating file handler. That removes the dashboard/State API copy of the logs, which is the copy you actually want to keep. (Raised in ray-project/kuberay#4095, documented in ray-project/kuberay#4104.) ## Issue Severity Medium: It is a significant difficulty but I can work around it.", "url": "https://github.com/ray-project/ray/issues/65454", "createdAt": "2026-08-13T10:24:49Z", "updatedAt": "2026-08-13T13:37:06Z", "timestamp": "2026-08-13T13:37:06Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "usability", "core", "observability", "stability", "community-backlog" ], "author": "vivekmahajan", "state": "open", "assignees": [] }, { "id": "github:ray-project/ray:pull_request:61053", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[serve] [taskiq - 3/n] Implement full TaskiqTaskProcessorAdapter with E2E tests", "text": "## Summary - Implements all stubbed methods in `TaskiqTaskProcessorAdapter`, completing the adapter that bridges Ray Serve's sync `TaskProcessorAdapter` ABC to taskiq's async broker API - Adds 6 E2E tests covering both standalone adapter usage and full Ray Serve deployment integration ## Why is this change needed? PR #61008 (`[taskiq - 2/n]`) added the config and initialization but left all adapter methods as `raise NotImplementedError` stubs. This PR fills in the complete implementation so the taskiq adapter is actually usable end-to-end β€” enqueue tasks, consume them, query status, and gracefully shut down. ## Architecture ``` β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ RAY ACTOR THREAD β”‚ β”‚ β”‚ β”‚ initialize() β”‚ β”‚ └── creates broker, spawns loop ──────►┐ β”‚ β”‚ β”‚ β”‚ register_task_handle() β”‚ β”‚ β”‚ └── broker.register_task() β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ enqueue_task_sync() Β·Β·Β·BLOCKSΒ·Β·Β·β–Ί β”‚ β”‚ β”‚ get_task_status_sync() Β·BLOCKSΒ·Β·β–Ί β”‚ β”‚ β”‚ start_consumer() Β·Β·Β·Β·SCHEDULESΒ·Β·β–Ί β”‚ β”‚ β”‚ stop_consumer() β”‚ β”‚ β”‚ β”œβ”€β”€ finish_event.set() Β·Β·Β·Β·Β·Β·Β·Β·β–Ί β”‚ β”‚ β”‚ β”œβ”€β”€ wait for drain Β·Β·BLOCKSΒ·Β·β–Ί β”‚ β”‚ β”‚ └── broker.shutdown() Β·BLOCKSΒ·β–Ί β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ Β·Β·Β·β–Ί = run_coroutine_threadsafe β”‚ ───► = thread creation β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ BACKGROUND THREAD (daemon) β”‚β—„β”€β”€β”€β”˜ β”‚ β”‚ β”‚ asyncio event loop: β”‚ β”‚ β”‚ β”‚ β€’ broker.startup() β”‚ β”‚ β€’ broker.kick(message) β”‚ β”‚ β€’ result_backend.is_ready() β”‚ β”‚ β€’ receiver.listen(finish_event) β”‚ β”‚ β”œβ”€β”€ pull message from broker β”‚ β”‚ β”œβ”€β”€ execute handler β”‚ β”‚ β”œβ”€β”€ save result + ack β”‚ β”‚ └── loops until event is set β”‚ β”‚ β€’ broker.shutdown() β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` ## Key Design Decisions **Persistent background event loop** (not `asyncio.run()` per call): Keeps broker connections alive across method calls. Taskiq brokers create connection pools in `__init__` and manage them across the event loop lifetime. `_ensure_loop()` is protected by `threading.Lock` to prevent race conditions. **`broker.kick()` for enqueue** (not `task.kiq()`): The low-level send doesn't require `broker.startup()` (connection pool is created in `__init__`), avoids holding references to taskiq-decorated task objects on the producer side, and consumer-side middleware (retry, ack, result storage) still runs normally. **`Receiver.listen()` for consuming** (not `run_receiver_task()`): Accepts an external `asyncio.Event` for graceful shutdown β€” the receiver stops accepting new messages and drains in-flight tasks. `run_receiver_task()` creates its own internal event with no external access, making graceful shutdown impossible. We also pass `run_startup=False` since `broker.startup()` is not idempotent (re-runs middleware init, creates duplicate connections). **No in-flight cancellation on shutdown**: Taskiq defaults to `WHEN_SAVED` ack mode β€” messages are acked only after task executes and result is saved. **Bound method wrapping**: Taskiq's `broker.register_task()` sets `__name__` on the callable, but bound methods have a read-only `__name__` property. We wrap bound methods in plain functions when registered via `@task_consumer`. **Task cancellation**: Not supported by taskiq ([taskiq-python/taskiq#305](https://github.com/taskiq-python/taskiq/issues/305)). Raises `NotImplementedError`. ## What's in this PR - `register_task_handle()` β€” registers callables with bound-method wrapping - `enqueue_task_sync()` β€” low-level `broker.kick()` with `TaskiqMessage` construction - `get_task_status_sync()` β€” result backend polling via `AsyncTaskiqTask` - `start_consumer()` / `stop_consumer()` β€” `Receiver.listen()` lifecycle with graceful drain - `cancel_task_sync()` β€” `NotImplementedError` (taskiq limitation) - Background event loop helpers (`_ensure_loop`, `_run_async`, `_ensure_broker_started`) - 6 E2E tests (3 adapter-level + 3 full Serve deployment integration) ## Related PRs - Depends on: #60977 (`[taskiq - 1/n]` Enable async task handlers) β€” **merged** - Depends on: #61008 (`[taskiq - 2/n]` Taskiq adapter config and initialization) β€” **merged** - Follow-up: `[taskiq - 4/n]` Retry, DLQ middleware and reliability tests πŸ€– Generated with [Claude Code](https://claude.com/claude-code)", "url": "https://github.com/ray-project/ray/pull/61053", "createdAt": "2026-02-13T13:06:15Z", "updatedAt": "2026-08-13T07:50:23Z", "timestamp": "2026-08-13T07:50:23Z", "metrics": { "reactions": 0, "comments": 4 }, "labels": [ "serve", "go" ], "author": "harshit-anyscale", "state": "open", "assignees": [ "harshit-anyscale" ] }, { "id": "github:ray-project/ray:pull_request:61898", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[Core] Mobilint Accelerator Support", "text": "## Description This PR adds first-class support for Mobilint MBLT NPUs to Ray's accelerator abstraction by introducing a new `MBLTAcceleratorManager` under `ray/_private/accelerators/mblt.py`. This integration is developed and contributed by ETRI (Electronics and Telecommunications Research Institute) in collaboration with Mobilint. Based on the SDKs provided by Mobilint, ETRI has implemented this software integration to align with Ray's official accelerator policies and architectural standards. The goal is to make MBLT resources natively schedulable in Ray (e.g., `resources={\"MBLT\": 1}`) and to provide per-worker/actor device isolation via a conventional \"visible devices\" environment variable, consistent with Ray's existing accelerator patterns (GPU, TPU, Neuron, etc.). **Key behaviors:** - Resource name: `MBLT` - Scheduling unit: Whole cards only (no fractional sharing). - Visibility isolation: Ray sets `MBLT_DEVICES` (comma-separated IDs) for worker/actor processes. - Runtime IDs: Users can access assigned IDs via `ray.get_runtime_context().get_accelerator_ids()`. ## Motivation Mobilint MBLT devices are increasingly utilized in heterogeneous clusters for specialized AI workloads. Currently, users must rely on custom resources and manual environment variable management to use these NPUs with Ray. This manual process is error-prone and breaks the seamless Ray-native experience. By introducing this PR, ETRI aims to provide a standardized interface for the MBLT lifecycle within the Ray ecosystem so that: - Ray's internal scheduler treats MBLT cards as first-class accelerators. - Cluster-wide resource management and isolation remains consistent across different hardware types. - Research and industrial users at ETRI and beyond can deploy MBLT-based workloads with minimal configuration. ## Usage example ```python import ray # Requesting 1 MBLT NPU card @ray.remote(resources={\"MBLT\": 1}) class MBLTWorker: def get_info(self): return ray.get_runtime_context().get_accelerator_ids() w = MBLTWorker.remote() # Expected output: {\"MBLT\": [\"0\"]} print(ray.get(w.get_info.remote())) ``` ## Contributors - Software Implementation: ETRI (Electronics and Telecommunications Research Institute) - Hardware/SDK Support: Mobilint", "url": "https://github.com/ray-project/ray/pull/61898", "createdAt": "2026-03-20T01:32:36Z", "updatedAt": "2026-08-13T11:45:13Z", "timestamp": "2026-08-13T11:45:13Z", "metrics": { "reactions": 3, "comments": 7 }, "labels": [ "core", "go", "community-contribution" ], "author": "dataminsu", "state": "open", "assignees": [ "Yicheng-Lu-llll" ] }, { "id": "github:ray-project/ray:pull_request:63793", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[core] Add opt-in swap accounting to memory monitor and scheduler", "text": "## Why Ray's OOM monitor kills tasks based on RAM alone, while the Linux OOM killer only fires when **both** RAM and swap are exhausted. On nodes provisioned with swap as overflow, Ray kills workers long before the kernel would. The scheduler's `memory` resource and the dashboard's Node Memory graph have the same RAM-only blind spot, and under `--enable-resource-isolation` the user-slice threshold ignores swap too β€” so scheduling, reporting, and enforcement all disagree. ## What Adds opt-in `RAY_count_swap_in_memory_monitor` (default `false`). When on, swap is folded into: 1. **C++ memory monitor** β€” cgroup v1 `memory.memsw.*` or cgroup v2 `memory.swap.{max,current}`; `/proc/meminfo` swap on bare metal. 2. **Scheduler `memory` resource** and **dashboard Node Memory** β€” via shared `get_cgroup_aware_swap_memory()` so containers see cgroup-scoped swap, not host swap. 3. **Resource-isolation user-slice OOM monitor** β€” `TakeUserSliceMemoryUsageSnapshot` now reads `user.memory.swap.{max,current}` and folds them into both used and total. Without this, `ray status` advertises RAM + swap while the monitor enforces RAM only. `memory.high` (RAM-only kernel constraint) is **not** inflated by swap β€” `GetCGroupMemoryBytes` takes `include_swap` so the cgroup-manager path stays RAM-only. Flag-off is the existing behavior. ## Test plan - C++ gtests: cgroup v2 swap, cgroup v1 memsw, `\"max\"` sentinel, missing files, user-slice swap fold, flag-off pinning. - Python unit tests: scheduler swap inclusion, explicit `--memory` bypass, dashboard `_get_mem_usage` clamps `used ≀ total`. - Manual: `RAY_count_swap_in_memory_monitor=1 ray start --head --enable-resource-isolation` β€” `ray status`, dashboard, and OOM threshold all reflect RAM + cgroup swap. ## Benchmark Verified on an Azure VM with 125.8 GB RAM + 18 GB swap (Ubuntu 24.04, cgroup v2). A Ray task allocates memory in 256 MB chunks. Same task and parameters across each pair β€” only `RAY_count_swap_in_memory_monitor` differs. ### Default OOM monitor (no resource isolation) Threshold = 70% of `total`. Flag off makes `total = RAM` (88 GB); flag on makes `total = RAM + swap` (100 GB). Target = 78 GB. | flag | completed | killed by Ray | elapsed | |---|:-:|:-:|---| | off | no | yes | 23 s | | on | yes | no | 28 s | ### Resource isolation (`--enable-resource-isolation`, cgroup v2 in a privileged container) Threshold = user-slice `memory.high`. Flag off keeps it RAM-only; flag on adds host swap. With `user_memory_proportion_high=0.5`, `memory.high` β‰ˆ 62.5 GB. Target = 69 GB. | flag | completed | killed by Ray | elapsed | |---|:-:|:-:|---| | off | no | yes | 20 s | | on | yes | no | 92 s | On-cell elapsed is 4Γ— longer than the no-isolation case because the task actually pages out to swap β€” kernel reclaim doing real I/O. ### Reproduction Script: `release/nightly_tests/stress_tests/test_count_swap_in_memory_monitor.py` (single file, deps: `ray`, `numpy`, `psutil`). Resource-isolation case requires running inside a privileged container so Ray's cgroup-v2 manager doesn't conflict with the host's systemd cgroup hierarchy.", "url": "https://github.com/ray-project/ray/pull/63793", "createdAt": "2026-06-02T09:14:52Z", "updatedAt": "2026-08-12T21:20:37Z", "timestamp": "2026-08-12T21:20:37Z", "metrics": { "reactions": 1, "comments": 10 }, "labels": [ "core", "observability", "go", "community-contribution" ], "author": "preneond", "state": "open", "assignees": [ "Kunchd" ] }, { "id": "github:ray-project/ray:pull_request:64413", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[ci] Add credential-free redirect-validation premerge check", "text": "## Why are these changes needed? The YAML under `doc/redirects/` is the source of truth for the docs.ray.io Read the Docs redirect configuration, managed with [rtd-redirects](https://github.com/anyscale/rtd-redirects). Today a PR that edits only redirect YAML runs no check at all, so a malformed or unreachable redirect can merge unnoticed. Nothing routes redirect YAML anywhere. `.yaml` isn't covered by the leading prose-and-image skip rule, nor by the `doc/*.py`/`doc/*.ipynb`/`doc/*/BUILD.bazel` catch-all, so it falls through to the broad `doc/` pass rule at the bottom of the doc section and emits no tag at all. Confirmed against the rule engine on master: `doc/redirects/current.yaml` emits `set()`. This adds a lightweight, credential-free premerge gate for `doc/redirects/*.yaml` that validates the redirects and prints the redirect-level diff. The change is purely additive: it gives redirect YAML a check where it previously had none, and no other path's selection changes. ## What changed - **`.buildkite/test.rules.txt`** β€” declare a `doc_redirects` test tag and route `doc/redirects/*.yaml` to it. The rule sits ahead of every `doc/` rule, which matters because the final `doc/` pass rule is what currently swallows these files. The rule emits `doc_redirects` alone, deliberately not `doc` or `doc_api`. A redirect edit can't change the documented API surface, and the check needs neither a rendered documentation build nor a compiled Ray: it parses the YAML and diffs it against the base ref. - **`.buildkite/test.rules.txt`** β€” route `.buildkite/doc.rayci.yml` to `doc_redirects` as well, on its own rule rather than by extending the autodoc-machinery block, so a `conf.py` edit doesn't start running the redirect check. An edit to a step definition has to be able to select the step it edits, which is the same reason that file already emits `doc_api`. This also covers the `anyscale-rtd-redirects` version pin, which lives in the redirect step's commands, so a version bump re-runs the check. - **`.buildkite/test.rules.test.txt`** β€” three cases: `doc/redirects/current.yaml: doc_redirects`, `doc/redirects/README.md:` asserting the prose beside it stays unrouted, and `.buildkite/doc.rayci.yml: doc doc_api doc_redirects`. The last one revises an assertion that already existed on master, since that file's tag set genuinely changes here. - **`.buildkite/doc.rayci.yml`** β€” add a `forge`-based step (no doc image) that installs the pinned `anyscale-rtd-redirects==0.2.0` from PyPI and runs: ``` rtd-redirects validate doc/redirects/current.yaml rtd-redirects diff-file --file doc/redirects/current.yaml --base FETCH_HEAD --head HEAD --repo . ``` The step diffs against `FETCH_HEAD` after fetching the PR's base branch, rather than against `origin/`: a clone with a restricted refspec may never create a local remote-tracking ref, and `doc/test_no_new_rst.py` already solves the same problem the same way. The base branch resolves at runtime from `$${BUILDKITE_PULL_REQUEST_BASE_BRANCH:-master}`, escaped so the shell expands it rather than Buildkite interpolating it at pipeline-upload time, matching the `$${...}` convention used across the other pipeline files. The step also fails closed if `doc/redirects/` grows a YAML file the commands don't name. The tag selects on a glob, but multiple redirect files compose in the order given and a shell glob can't express that order, so the guard forces the compose-order decision to be made explicitly instead of letting a new file select the step and then go unvalidated. ## Verification Verified with Ray's rule parser (`ci/pipeline/determine_tests_to_run.py`), the same first-match-wins engine rayci uses: - `check_rules()` parses clean, and all 126 cases in `test.rules.test.txt` replay green. That file is enforced by the `test-rules` CI step, and rayci asserts exact set equality, so a passing case proves the path emits exactly those tags and nothing else. - A sweep of all 10,333 tracked files against master shows **exactly two deltas** β€” `doc/redirects/current.yaml` gaining `doc_redirects`, and `.buildkite/doc.rayci.yml` gaining it alongside its existing `doc doc_api` β€” and **zero tag losses** on any path. Every pre-existing assertion in `test.rules.test.txt` is preserved except the `doc.rayci.yml` one this PR deliberately revises. - Ran the step's commands locally in a clean venv with `anyscale-rtd-redirects==0.2.0` against the current `doc/redirects/current.yaml`: `validate` reports 0 errors (49 chain warnings, informational), and `diff-file --base FETCH_HEAD` reports the redirect-level diff and exits 0 with no Read the Docs credentials. - Exercised the drift guard both ways: it passes with only `current.yaml` present and exits non-zero when a second `doc/redirects/*.yaml` appears. Because `.buildkite/doc.rayci.yml` now emits `doc_redirects`, **this PR selects the new step and runs it**. Replaying this PR's own changed files through the rule engine gives a tag union of `doc`, `doc_api`, `doc_redirects`, and `tools`, so the step's commands are exercised in CI here rather than only on some later PR that happens to edit redirect YAML. It runs against an unchanged `current.yaml`, so `diff-file` reports an empty redirect diff and exits 0, which is exactly the smoke test that catches a typo in the step definition. ## Update: rebased onto the doc-tag reorg Rebased after #64775 and #64812 landed, which reorganized the doc tags: doc example tests moved to per-library `_doc` tags, and `doc` was split so the autodoc machinery emits `doc doc_api` while the rendering-only surface emits `doc`. This branch's four commits (one an upstream merge) are collapsed into one. Two conflicts in `test.rules.txt` (the tag declaration line, and the rule insertion point) and one in `test.rules.test.txt` (three expectations the reorg had since updated) were resolved in favor of master, then the `doc_redirects` rule and cases re-applied on top. The rebase also invalidated this PR's original rationale, so the rule and step comments have been rewritten. The earlier framing was that `doc_redirects` keeps redirect YAML from pulling in docbuild, the API checks, and the per-team doc tests. That's no longer accurate on any of the three: the per-team tests are selected by `_doc`, the two API checks carry `doc_api` plus the library tags and no longer carry bare `doc`, and the only remaining step carrying bare `doc` is `doc: build`, which is `skip-on-premerge`. Since redirect YAML emits nothing at all on master, this rule was never preventing over-triggering. It's additive, and the comments now say so. ## Update: the step now selects itself Cursor Bugbot caught that the redirect step wasn't reachable from an edit to its own definition: `.buildkite/doc.rayci.yml` emitted `doc doc_api` but never `doc_redirects`, so changing the redirect step's commands wouldn't have run the redirect step. That contradicted the rule's own stated reason for emitting `doc_api`, and this PR demonstrated the gap: its previous revision noted that the new step didn't execute here. Fixed by giving `.buildkite/doc.rayci.yml` its own rule emitting `doc doc_api doc_redirects`. It's a separate rule rather than an extra tag on the autodoc-machinery block, because that block also covers `conf.py`, `api_autogen.py`, and `doc/source/_ext/`, and a `conf.py` edit has no business running the redirect check. The first attempt at this appended the tag to the shared block and the full-repo sweep caught it immediately: seven machinery files picked up `doc_redirects`. AI assistance (Claude Code) was used to draft the CI step, the rebase conflict resolution, and this description. Every changed line was reviewed by hand, and the verification above was run as described. This isn't duplicating existing work: no other open PR adds redirect validation to premerge, and the `doc_redirects` tag doesn't exist on master. ## Related issue number N/A ## Checks - [x] I've signed off every commit (`-s`), because all commits are squashed into one when merged. - [x] I've run `scripts/format.sh` to lint the changes in this PR. (Via `pre-commit`; the applicable hooks pass.) - [x] I've made sure the tests are passing. - Testing Strategy - [x] Unit tests - [ ] Release tests - [ ] This PR is not tested :(", "url": "https://github.com/ray-project/ray/pull/64413", "timestamp": "2026-08-12T13:28:16Z", "metrics": { "reactions": 1, "comments": 1 }, "labels": [ "docs", "devprod", "go", "unstale" ], "author": "dstrodtman", "assignees": [], "change": "new" }, { "id": "github:ray-project/ray:pull_request:64451", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[WIP][serve] Use `ObjectRefGenerator._get_next_ref_n` to avoid blocking on `_to_object_ref`", "text": "Integrates with https://github.com/ray-project/ray/pull/62741 to fix the long-standing issue in serve where `_to_object_ref` blocks until the underlying result has been produced.", "url": "https://github.com/ray-project/ray/pull/64451", "createdAt": "2026-06-30T18:52:38Z", "updatedAt": "2026-08-13T00:38:41Z", "timestamp": "2026-08-13T00:38:41Z", "metrics": { "reactions": 1, "comments": 2 }, "labels": [ "stale", "go" ], "author": "edoakes", "state": "closed", "assignees": [] }, { "id": "github:ray-project/ray:pull_request:64462", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[core] Post object info pubsub publishes on a dedicated thread", "text": "## Description serve microbenchmark | metric | master before #63983 | PR fix | PR fix vs master before #63983 | |---|--:|--:|--:| | http_p50_latency | 2.157 | 2.210 | +2.4% πŸ”΄ | | http_p90_latency | 2.274 | 2.340 | +2.9% πŸ”΄ | | http_p95_latency | 2.319 | 2.409 | +3.9% πŸ”΄ | | http_p99_latency | 2.500 | 2.572 | +2.9% πŸ”΄ | | http_1mb_p50_latency | 2.329 | 2.377 | +2.1% πŸ”΄ | | http_1mb_p90_latency | 2.609 | 2.680 | +2.7% πŸ”΄ | | http_1mb_p95_latency | 2.668 | 2.775 | +4.0% πŸ”΄ | | http_1mb_p99_latency | 2.962 | 2.931 | -1.1% βšͺ | | http_10mb_p50_latency | 3.818 | 4.811 | +26.0% πŸ”΄ | | http_10mb_p90_latency | 5.354 | 5.377 | +0.4% βšͺ | | http_10mb_p95_latency | 5.609 | 5.574 | -0.6% βšͺ | | http_10mb_p99_latency | 6.558 | 6.032 | -8.0% 🟒 | | http_avg_rps | 619.5 | 604.6 | -2.4% πŸ”΄ | | http_throughput_std | 12.4 | 13.9 | +12.5% β€” | | http_model_comp_avg_rps | 247.9 | 242.9 | -2.0% πŸ”΄ | | http_model_comp_throughput_std | 7.140 | 5.970 | -16.4% β€” | | http_100_max_ongoing_requests_avg_rps | 915.4 | 872.5 | -4.7% πŸ”΄ | | http_100_max_ongoing_requests_throughput_std | 10.1 | 8.030 | -20.7% β€” | | http_model_comp_100_max_ongoing_requests_avg_rps | 432.1 | 423.6 | -1.9% βšͺ | | http_model_comp_100_max_ongoing_requests_throughput_std | 4.570 | 4.720 | +3.3% β€” | | http_800_max_ongoing_requests_avg_rps | 787.2 | 785.7 | -0.2% βšͺ | | http_800_max_ongoing_requests_throughput_std | 18.0 | 17.6 | -2.6% β€” | | http_model_comp_800_max_ongoing_requests_avg_rps | 401.8 | 398.8 | -0.8% βšͺ | | http_model_comp_800_max_ongoing_requests_throughput_std | 9.990 | 9.110 | -8.8% β€” | | http_streaming_avg_tps | 37496 | 40935 | +9.2% 🟒 | | http_streaming_throughput_std | 612.3 | 204.7 | -66.6% β€” | | http_streaming_p50_latency | 13251 | 12092 | -8.7% 🟒 | | http_streaming_p90_latency | 13463 | 12204 | -9.3% 🟒 | | http_streaming_p95_latency | 13507 | 12213 | -9.6% 🟒 | | http_streaming_p99_latency | 13598 | 12228 | -10.1% 🟒 | | http_intermediate_streaming_avg_tps | 6637 | 5763 | -13.2% πŸ”΄ | | http_intermediate_streaming_throughput_std | 103.4 | 37.9 | -63.4% β€” | | http_intermediate_streaming_p50_latency | 22546 | 26020 | +15.4% πŸ”΄ | | http_intermediate_streaming_p90_latency | 23021 | 26169 | +13.7% πŸ”΄ | | http_intermediate_streaming_p95_latency | 23193 | 26244 | +13.2% πŸ”΄ | | http_intermediate_streaming_p99_latency | 23208 | 26265 | +13.2% πŸ”΄ | | grpc_p50_latency | 1.868 | 1.894 | +1.4% βšͺ | | grpc_p90_latency | 2.012 | 2.046 | +1.7% βšͺ | | grpc_p95_latency | 2.082 | 2.117 | +1.7% βšͺ | | grpc_p99_latency | 2.263 | 2.464 | +8.9% πŸ”΄ | | grpc_1mb_p50_latency | 6.968 | 7.229 | +3.8% πŸ”΄ | | grpc_1mb_p90_latency | 7.428 | 7.735 | +4.1% πŸ”΄ | | grpc_1mb_p95_latency | 7.704 | 7.882 | +2.3% πŸ”΄ | | grpc_1mb_p99_latency | 8.521 | 8.241 | -3.3% 🟒 | | grpc_10mb_p50_latency | 40.9 | 46.4 | +13.6% πŸ”΄ | | grpc_10mb_p90_latency | 43.8 | 48.6 | +11.0% πŸ”΄ | | grpc_10mb_p95_latency | 47.9 | 49.5 | +3.4% πŸ”΄ | | grpc_10mb_p99_latency | 51.9 | 51.9 | -0.2% βšͺ | | grpc_avg_rps | 873.0 | 825.5 | -5.4% πŸ”΄ | | grpc_throughput_std | 10.5 | 12.0 | +13.6% β€” | | grpc_model_comp_avg_rps | 323.9 | 308.2 | -4.8% πŸ”΄ | | grpc_model_comp_throughput_std | 9.400 | 9.720 | +3.4% β€” | | grpc_100_max_ongoing_requests_avg_rps | 957.2 | 961.9 | +0.5% βšͺ | | grpc_100_max_ongoing_requests_throughput_std | 8.873 | 10.3 | +16.2% β€” | | grpc_model_comp_100_max_ongoing_requests_avg_rps | 657.7 | 644.4 | -2.0% πŸ”΄ | | grpc_model_comp_100_max_ongoing_requests_throughput_std | 6.977 | 6.230 | -10.7% β€” | | grpc_800_max_ongoing_requests_avg_rps | 922.8 | 913.2 | -1.0% βšͺ | | grpc_800_max_ongoing_requests_throughput_std | 8.927 | 7.790 | -12.7% β€” | | grpc_model_comp_800_max_ongoing_requests_avg_rps | 669.2 | 647.4 | -3.3% πŸ”΄ | | grpc_model_comp_800_max_ongoing_requests_throughput_std | 7.443 | 9.670 | +29.9% β€” | | handle_p50_latency | 0.994 | 1.113 | +12.0% πŸ”΄ | | handle_p90_latency | 1.102 | 1.265 | +14.7% πŸ”΄ | | handle_p95_latency | 1.150 | 1.315 | +14.4% πŸ”΄ | | handle_p99_latency | 1.336 | 1.455 | +8.9% πŸ”΄ | | handle_1mb_p50_latency | 1.862 | 1.850 | -0.6% βšͺ | | handle_1mb_p90_latency | 2.270 | 2.033 | -10.4% 🟒 | | handle_1mb_p95_latency | 2.629 | 2.121 | -19.3% 🟒 | | handle_1mb_p99_latency | 2.920 | 2.555 | -12.5% 🟒 | | handle_10mb_p50_latency | 4.471 | 4.437 | -0.8% βšͺ | | handle_10mb_p90_latency | 5.396 | 4.658 | -13.7% 🟒 | | handle_10mb_p95_latency | 9.366 | 11.3 | +20.1% πŸ”΄ | | handle_10mb_p99_latency | 12.0 | 12.1 | +0.4% βšͺ | | handle_choose_dispatch_p50_latency | 1.773 | 1.778 | +0.3% βšͺ | | handle_choose_dispatch_p90_latency | 1.895 | 1.904 | +0.5% βšͺ | | handle_choose_dispatch_p95_latency | 1.947 | 1.959 | +0.6% βšͺ | | handle_choose_dispatch_p99_latency | 2.151 | 2.129 | -1.0% βšͺ | | handle_avg_rps | 794.1 | 761.5 | -4.1% πŸ”΄ | | handle_throughput_std | 20.3 | 19.4 | -4.4% β€” | | handle_model_comp_avg_rps | 317.1 | 309.5 | -2.4% πŸ”΄ | | handle_model_comp_throughput_std | 8.867 | 7.830 | -11.7% β€” | | handle_100_max_ongoing_requests_avg_rps | 1569 | 1497 | -4.6% πŸ”΄ | | handle_100_max_ongoing_requests_throughput_std | 18.9 | 15.3 | -19.4% β€” | | handle_model_comp_100_max_ongoing_requests_avg_rps | 758.2 | 734.9 | -3.1% πŸ”΄ | | handle_model_comp_100_max_ongoing_requests_throughput_std | 10.0 | 8.760 | -12.7% β€” | | handle_800_max_ongoing_requests_avg_rps | 1401 | 1358 | -3.0% πŸ”΄ | | handle_800_max_ongoing_requests_throughput_std | 26.0 | 26.0 | -0.1% β€” | | handle_model_comp_800_max_ongoing_requests_avg_rps | 709.0 | 686.0 | -3.2% πŸ”΄ | | handle_model_comp_800_max_ongoing_requests_throughput_std | 13.2 | 15.1 | +14.2% β€” | | handle_streaming_avg_tps | 10226 | 12251 | +19.8% 🟒 | | handle_streaming_throughput_std | 847.1 | 164.5 | -80.6% β€” | | handle_streaming_p50_latency | 14705 | 12181 | -17.2% 🟒 | | handle_streaming_p90_latency | 16185 | 12434 | -23.2% 🟒 | | handle_streaming_p95_latency | 16976 | 12480 | -26.5% 🟒 | | handle_streaming_p99_latency | 16989 | 12488 | -26.5% 🟒 |", "url": "https://github.com/ray-project/ray/pull/64462", "timestamp": "2026-08-12T12:30:53Z", "metrics": { "reactions": 1, "comments": 2 }, "labels": [ "stale", "core", "go" ], "author": "Yicheng-Lu-llll", "assignees": [], "change": "new" }, { "id": "github:ray-project/ray:pull_request:64540", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[data] Add orc datasource for V2", "text": "## Description Support read orc format ## Related issues > Link related issues: \"Fixes #1234\", \"Closes #1234\", or \"Related to #1234\". ## Additional information > Optional: Add implementation details, API changes, usage examples, screenshots, etc.", "url": "https://github.com/ray-project/ray/pull/64540", "createdAt": "2026-07-06T03:31:43Z", "updatedAt": "2026-08-13T04:07:18Z", "timestamp": "2026-08-13T04:07:18Z", "metrics": { "reactions": 1, "comments": 7 }, "labels": [ "data", "go", "community-contribution" ], "author": "laysfire", "state": "open", "assignees": [ "ayushk7102" ] }, { "id": "github:ray-project/ray:pull_request:64542", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[core][rdt] Application-level exclusive NIC assignment for NIXL/UCX t…", "text": "## Why Ray actors using RDT/NIXL over UCX don't control which RDMA NIC they use β€”UCX picks devices on its own. On multi-NIC nodes this means several actors can silently pile onto the same NIC while others sit idle, causing unpredictable tensor-transfer performance with no visible error. Closes/Addresses #64426. ## What Per discussion on the issue (thanks @Sparks0219 @stephanie-wang), this implements NIC pinning entirely at the application level inside `ray.experimental.rdt` β€” no Ray Core / scheduler / resource-model changes.This keeps the change small, fully opt-in, and iterable, while leaving the door open to promote parts of it into core later if the design holds up. ### `ray/experimental/rdt/nic_allocator.py` (new) - `discover_rdma_nics()` β€” enumerates port-qualified RDMA device names (e.g. `mlx5_0:1`) from `/sys/class/infiniband`, the same sysfs surface `_is_efa_available()` already relies on (works inside containers/pods where netdevs are namespaced away). Returns `[]` on non-RDMA/non-Linux hosts. - `NICAllocator` β€” a small `@ray.remote` actor acting as a cluster-wide registry of `(node, NIC) -> owning actor`. Runs as a single detached singleton; single-threaded actor execution gives us race-free acquire/release for free. Supports idempotent node registration, re-entrant acquisition (an actor re-acquiring gets the same NIC back instead of leaking a second one), and a `snapshot()` for observability. - `acquire_nic_for_current_actor()` / `release_nic_for_current_actor()` β€” the public entry points. Both are no-ops unless `RAY_RDT_NIC_PINNING=1` is set, and fail open on any error (discovery failure, allocator unreachable, timeout) by returning `None` / logging-and-returning, so a broken allocator degrades to today's behavior rather than breaking transfers. ### `ray/experimental/rdt/nixl_tensor_transport.py` `_init_nixl_agent()` now calls `acquire_nic_for_current_actor()` and, if it gets a NIC back, sets `UCX_NET_DEVICES` **before** constructing the NIXL agent (UCX reads this at context-init time). Scoped to the `UCX` backend only β€” `LIBFABRIC`/EFA is untouched. A user-supplied `UCX_NET_DEVICES` always takes precedence and is never overwritten. ### `ray/experimental/rdt/rdt_manager.py` `RDTManager.shutdown()` now calls `release_nic_for_current_actor()` to return the actor's NIC to the pool on clean shutdown. (Known gap: a SIGKILLed actor leaks its NIC until `release_all_for_node` is called β€” `snapshot()` gives operators visibility into this; a reaper is a natural follow-up.) ### `ray/tests/test_rdt_nic_allocator.py` (new) - `discover_rdma_nics` against a fake sysfs tree (parametrized: no devices / single device / multi-port / multi-device, verifying sorted determinism). - `NICAllocator` logic exercised directly as a plain class (no cluster): acquire/release cycle, exhaustion, re-entrancy, idempotent registration, unknown node, `release_all_for_node`. - Fail-open checks: flag unset, and flag set but no NICs discovered β€”both return `None` before any Ray call is made. - One end-to-end test with `ray_start_regular`: 4 actors contend for 2 fake NICs, exactly 2 succeed, and releasing one frees a slot for a waiter. ## Design notes / open questions carried from the issue discussion - **Opt-in and fail-open by design** β€” with `RAY_RDT_NIC_PINNING` unset, every code path here is a no-op; existing RDT/NIXL users see zero behavior change. - **Not scheduler-aware** β€” the allocator can refuse a NIC when the pool is exhausted, but it does not (and cannot, from application code) prevent Ray from *placing* another actor on an already-full node. That trade-off is what keeps this out of Ray Core for v1, per @stephanie-wang's suggestion. - **GPU/NIC PCIe affinity** (e.g. matching topology via `nvidia-smi topo -m`) is intentionally out of scope here β€” this PR only does exclusive assignment. Happy to iterate on affinity-aware allocation as a follow-up once the basic mechanism is validated. - **NIC granularity**: one allocatable unit = one (device, port) pair, not one HCA. Flagging this in case multi-port HCAs should be treated differently. ## Test plan - [x] New unit tests in `test_rdt_nic_allocator.py` (sysfs discovery, allocator logic, fail-open paths, end-to-end multi-actor contention) - [ ] Manual validation on real RDMA hardware (multi-NIC node) β€” checking `UCX_NET_DEVICES` is set correctly per actor and transfers no longer collide on one NIC - [x] Confirmed default behavior (flag unset) is unchanged for existing NIXL tests", "url": "https://github.com/ray-project/ray/pull/64542", "timestamp": "2026-08-12T12:30:55Z", "metrics": { "reactions": 1, "comments": 4 }, "labels": [ "core", "community-contribution", "unstale" ], "author": "AyushKashyapII", "assignees": [], "change": "new" }, { "id": "github:ray-project/ray:pull_request:64601", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[Data] Fix ResourceBudget backpressure causing pipeline stall", "text": "## Why are these changes needed? Two interacting bugs in `ResourceBudgetBackpressurePolicy` can cause a Ray Data pipeline to stall indefinitely β€” CPU/GPU idle, Object Store filling up, no error logged, no progress. Reproducing this requires enough intermediate output to pressure the Object Store; small pipelines won't hit it. ### Background `ResourceBudgetBackpressurePolicy` blocks an operator from submitting new tasks when **either**: (a) the operator's `incremental_resource_usage` (CPU/GPU/memory) exceeds its budget, or (b) the operator's Object Store budget for task outputs is below `obj_store_mem_max_pending_output_per_task`. When either fires, the operator enters `backpressured:tasks(ResourceBudget)` and stops scheduling. ### The two bugs **Bug 1 β€” `OutputBackpressureGuard.should_unblock` doesn't distinguish Object Store–blocked from CPU/GPU-blocked.** The liveness relaxation in `should_unblock` relaxes an upstream operator's output backpressure when a downstream operator has no active tasks and can't submit β€” so upstream can finish and release CPU/GPU/memory. This helps when downstream is blocked on condition (a). But when downstream is blocked on condition (b) β€” its Object Store budget for task outputs is exhausted β€” relaxing upstream only writes *more* output into the already-full store while downstream still can't schedule. The relaxation feeds the stall it's meant to prevent. **Bug 2 β€” `update_budgets` unconditionally allocates shared resources to over-reserved operators.** Each scheduling iteration, `update_budgets` recomputes every eligible operator's budget as its unused reservation plus a share of the remaining shared resources. Even when an operator's Object Store usage already exceeds its reserved resources (unused reservation is zero), it still receives a share of the remaining shared resources β€” restoring a positive budget, letting it submit more tasks, and further draining the shared resources that other operators depend on. **How they compound:** Bug 2 keeps the upstream operator's budget alive so it keeps producing output. Bug 1 relaxes the upstream's output backpressure so that output flows unchecked. Together, the Object Store fills until every operator's budget is exhausted and the pipeline stalls. ## Fix 1. In `should_unblock`, skip the liveness relaxation when the downstream operator is specifically blocked on Object Store budget for task outputs (new predicate `is_task_submission_blocked_on_object_store`). When downstream is blocked on CPU/GPU (condition a), the original relaxation is preserved. `IdleDetector` remains the bounded liveness fallback for a genuinely stalled pipeline. Uses `continue` (not `return False`) so that when multiple downstream operators exist, an Object Store–blocked one is skipped while a CPU/GPU-blocked one can still trigger relaxation. 2. Optionally withhold shared Object Store resources from operators that satisfy **both**: (a) per-operator usage exceeds `object_store_reservation_overshoot_ratio` Γ— its total reserved resources (task-side + output-side), AND (b) global Object Store usage exceeds `object_store_pool_pressure_fraction` of the cluster limit. Both thresholds must be set to enable the throttle; either being `None` (default) preserves legacy behavior. Requiring both conditions avoids throttling when the Object Store is nearly empty (a single overshooting operator is harmless there) and avoids letting one operator drain the shared pool when the store is under pressure β€” a single threshold would misfire in one direction or the other. Fix 2 pre-filters over-reserved operators before the shared-resource allocation loop (rather than skipping with `continue` inside the loop), keeping the per-operator share fraction aligned with the number of operators actually receiving resources. The fallback branch also iterates the pre-filtered list, preventing leftover budget from being routed back to an over-reserved operator. ## Related issues None. ## Additional information **Enabling the throttle (Fix 2)** β€” off by default. To enable, set both thresholds on `DataContext`: ```python ctx = ray.data.DataContext.get_current() ctx.object_store_reservation_overshoot_ratio = 1.5 # per-op overshoot headroom ctx.object_store_pool_pressure_fraction = 0.8 # engage at 80% pool usage ``` Or via environment variables (equivalent): ```bash export RAY_DATA_OBJECT_STORE_RESERVATION_OVERSHOOT_RATIO=1.5 export RAY_DATA_OBJECT_STORE_POOL_PRESSURE_FRACTION=0.8 ``` **Tests** (`test_resource_manager.py`): `TestOutputBackpressureGuard` (Fix 1): - `test_no_unblock_when_downstream_blocked_on_object_store` - `test_is_task_submission_blocked_on_object_store` `TestReservationOpResourceAllocator` (Fix 2): - `test_is_op_over_reserved_on_object_store` - `test_update_budgets_splits_shared_evenly_when_op_over_reserved`", "url": "https://github.com/ray-project/ray/pull/64601", "createdAt": "2026-07-08T15:23:34Z", "updatedAt": "2026-08-13T13:38:53Z", "timestamp": "2026-08-13T13:38:53Z", "metrics": { "reactions": 1, "comments": 2 }, "labels": [ "data", "go", "community-contribution" ], "author": "dragongu", "state": "open", "assignees": [] }, { "id": "github:ray-project/ray:pull_request:64670", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[core][metrics] Add MetricRegistry: idempotent facade over ray.util.metrics", "text": "## Description Adds `ray.util.metric_registry.MetricRegistry` (DeveloperAPI): a thread-safe, get-or-create factory over the three custom-metric primitives (`Counter`/`Gauge`/`Histogram`). It owns the quirks every caller currently re-solves by hand: - **Dedup by name** β€” the same name returns a handle to the same underlying Ray metric, so callers don't manage their own caches. - **Name sanitizing** β€” `:` β†’ `_` (illegal in Ray metric names), optional namespace prefix. - **The Counter `_total` quirk** β€” Ray's `Counter` re-appends `_total` on export; a trailing `_total` in the requested name is folded in rather than doubled. - **Fixed-`tag_keys` handling** β€” Ray fixes `tag_keys` at construction and validates strictly at record time. Handles pad declared-but-missing keys with `\"\"` (without clobbering default tags), and a label key first seen after creation warns once and is dropped instead of raising (Ray cannot widen `tag_keys`). Handles keep the kind-native verbs (`inc`/`set`/`observe`) with Ray semantics, plus a kind-agnostic `record()` for mirroring pipelines (counter `record()` no-ops on non-positive deltas) and a histogram `timer()` context manager. This is PR 1 of a 3-PR stack: 1. **This PR** β€” `MetricRegistry` 2. `PrometheusCollector` β€” mirror any Prometheus /metrics exposition into Ray metrics via the registry 3. Migrate serve's `haproxy_metrics.py` onto the registry ## Related issues N/A ## Additional information Tests are pure wrapper tests (no cluster) following the `metric_mock` pattern from `test_metrics_agent.py`: ``` python -m pytest python/ray/tests/test_metric_registry.py # 15 passed ```", "url": "https://github.com/ray-project/ray/pull/64670", "createdAt": "2026-07-10T18:57:37Z", "updatedAt": "2026-08-13T12:31:22Z", "timestamp": "2026-08-13T12:31:22Z", "metrics": { "reactions": 1, "comments": 2 }, "labels": [ "stale", "core", "go" ], "author": "xyuzh", "state": "closed", "assignees": [] }, { "id": "github:ray-project/ray:pull_request:64672", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[serve] Build serve_haproxy_* metrics through MetricRegistry", "text": "## Description Replaces the seven raw `ray.util.metrics` constructions in `HAProxyMetricsCollector` with handles from a shared, module-level `MetricRegistry(namespace=\"serve_haproxy\")`. The registry get-or-creates by name, so re-constructing a collector in the same process (tests, actor restarts) reuses the existing Ray metric objects instead of re-creating them. **No behavior change**: exported metric names, descriptions, tag keys, histogram buckets, and default tags are byte-identical (the requests counter still gains `_total` from Ray on export), and the record paths are untouched β€” registry handles expose the same `inc`/`set`/`observe` verbs with Ray semantics. This is PR 3 of a 3-PR stack (stacked on #64670 and #64671; only the last commit is new here): 1. #64670 β€” `MetricRegistry` 2. #64671 β€” `PrometheusCollector` 3. **This PR** β€” serve HAProxy metrics on the registry ## Related issues N/A ## Additional information The existing pure parse/record unit tests pass unmodified; adds one regression test asserting handle dedup across collector constructions and the unchanged exported names. ``` python -m pytest python/ray/serve/tests/test_haproxy_metrics.py # 58 passed ``` (3 pre-existing `AF_UNIX path too long` failures on macOS reproduce identically on the base branch β€” environmental, unrelated.)", "url": "https://github.com/ray-project/ray/pull/64672", "createdAt": "2026-07-10T18:58:09Z", "updatedAt": "2026-08-13T12:31:24Z", "timestamp": "2026-08-13T12:31:24Z", "metrics": { "reactions": 1, "comments": 2 }, "labels": [ "serve", "stale" ], "author": "xyuzh", "state": "closed", "assignees": [] }, { "id": "github:ray-project/ray:pull_request:64738", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[Autoscaler][AWS] Retry key pair creation after duplicate", "text": "## Description Concurrent `ray up` processes can both observe the same generated EC2 key-pair name as available and then race in `CreateKeyPair`. The losing request receives `InvalidKeyPair.Duplicate`, which currently aborts AWS bootstrap even though the existing loop can safely try another generated name. Catch only `InvalidKeyPair.Duplicate` and continue with the next key-pair name. Other `ClientError` codes still propagate unchanged, and private key material is written only for the successful `CreateKeyPair` response. ## Related issues Fixes #64673 ## Additional information Added a botocore `Stubber` regression test that reproduces the `DescribeKeyPairs`/`CreateKeyPair` race. It verifies that bootstrap selects the next generated name, writes only its private key material, and applies that name to every node type. Validation performed: - Negative control on unmodified `master`: failed with `InvalidKeyPair.Duplicate` as expected. - Focused regression test: 20/20 consecutive passes. - `python -m pytest -q python/ray/tests/aws`: 61 passed. - Ruff, Black, Python AST, and `git diff --check`: passed. OpenAI Codex assisted with the investigation, implementation, and local validation. I reviewed the final change and test behavior and take responsibility for the contribution.", "url": "https://github.com/ray-project/ray/pull/64738", "createdAt": "2026-07-14T02:57:38Z", "updatedAt": "2026-08-13T00:38:46Z", "timestamp": "2026-08-13T00:38:46Z", "metrics": { "reactions": 1, "comments": 2 }, "labels": [ "core", "community-contribution", "unstale" ], "author": "hogeheer499-commits", "state": "open", "assignees": [ "rueian" ] }, { "id": "github:ray-project/ray:pull_request:64828", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[Data][1/N] add external shuffle runtime library", "text": "Adds the file-transport runtime for a new external (on-disk) shuffle variant: TCP wire protocol, per-node ShuffleFileServer actor, connection / fetch primitives, and error classification. This PR lands only the runtime library and its testers required by external shuffle tasks. It's not wired into the plan yet and there's no user-visible behavior change. ## description - Reducers open one arrow flight connection per source ShuffleFileServer, authenticate with a per-shuffle token, and fetch data straight to disk. - ShuffleFileServer will be started by map task to serve reducer's file read request. Upon init, ShuffleFileServer will bind file serving function to a random available port. - Fault tolerance is driven by Ray's actor state (ActorDied / ActorUnavailable / ActorUnschedulable). On mid-fetch TCP failure, the reducer compares endpoints: if the manager was restarted (where the endpoint are likely to be different) we retry in place; if the endpoint is unchanged (which is Ray RPC still works but our TCP path doesn't) it's almost always a network-configuration problem (NetworkPolicy, firewall, routing) that retries to the same manager can't fix, so we surface a terminal ShuffleFileServerAnomalyError. ## Additional information The complete implementation of shuffle is in https://github.com/ray-project/ray/pull/64733, including runtime libraries, external shuffle task definition, plan wiring and a user-visible flag. I plan to do three PRs (1 is runtime libraries, 2 is task definition and wiring, and 3 is the user-visible flag) to gradually merge the complete implementation.", "url": "https://github.com/ray-project/ray/pull/64828", "createdAt": "2026-07-17T01:39:05Z", "updatedAt": "2026-08-13T09:38:04Z", "timestamp": "2026-08-13T09:38:04Z", "metrics": { "reactions": 1, "comments": 4 }, "labels": [ "data", "go" ], "author": "ShockYoungCHN", "state": "open", "assignees": [] }, { "id": "github:ray-project/ray:pull_request:64885", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[data] Fix text_embedding release test via raydepsets python depset", "text": "The `text_embedding_{fixed_size,autoscaling,fixed_size_chaos}` release tests were jailed with `ModuleNotFoundError: No module named 'tf_keras'` / `ValueError: ... Keras 3 ... is not yet supported in Transformers`. Root cause: the imperative `byod_install_text_embedding.sh` pip-installed transformers/sentence-transformers/torch onto the raw cu123 base image, which ships TensorFlow + Keras 3 but no tf-keras. At import time `sentence_transformers` imports `transformers`, which refuses to load its TF integration against Keras 3 without the backwards-compatible tf-keras shim, crashing before any Ray Data runs. Replace the byod script with a raydepsets-managed python depset: - Add release/nightly_tests/dataset/text_embedding/requirements.in with the test's deps plus tf-keras==2.20.0 (matching requirements_compiled). - Add a text_embedding_py310 depset that expands the GPU base testdeps, and generate release/ray_release/byod/text_embedding_py3.10.lock. - Point the release test at `python_depset: text_embedding_py3.10.lock` and delete byod_install_text_embedding.sh. The resolved lock stays on numpy 2.2.6 (no downgrade); tf-keras lets transformers load cleanly against the base image's Keras 3. AI assistance (Claude Code) was used for this change. Testing: - `bazelisk run //ci/raydepsets:raydepsets -- build \\ ci/raydepsets/configs/release_multimodal_inference_benchmarks_tests.depsets.yaml \\ --name text_embedding_py310 --check` -> \"Lock files are up to date.\" - Verified resolved versions: transformers==4.56.2, sentence-transformers==5.1.0, torch==2.8.0+cu128, tf-keras==2.20.0, tensorflow==2.20.0, numpy==2.2.6. Not a duplicate: no open PR/issue touches the text_embedding byod deps (checked `gh pr list` for text_embedding and tf-keras). > Thank you for contributing to Ray! πŸš€ > Please review the [Ray Contribution Guide](https://docs.ray.io/en/master/ray-contribute/getting-involved.html) before opening a pull request. > ⚠️ Remove these instructions before submitting your PR. > πŸ’‘ Tip: Mark as draft if you want early feedback, or ready for review when it's complete. ## Description > Briefly describe what this PR accomplishes and why it's needed. ## Related issues > Link related issues: \"Fixes #1234\", \"Closes #1234\", or \"Related to #1234\". ## Additional information > Optional: Add implementation details, API changes, usage examples, screenshots, etc.", "url": "https://github.com/ray-project/ray/pull/64885", "timestamp": "2026-08-12T12:31:12Z", "metrics": { "reactions": 1, "comments": 1 }, "labels": [ "stale", "data", "devprod", "release-test", "go" ], "author": "elliot-barn", "assignees": [], "change": "new" }, { "id": "github:ray-project/ray:pull_request:64895", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "fix(core): fold runtime-env hash instead of truncating to int", "text": "## Description `CalculateRuntimeEnvHash` previously did `return static_cast(hash)` on a 64-bit `size_t` hash. That drops the upper 32 bits and can produce a negative key. The value is used as a worker-pool reuse key, so a low-bit collision can silently run a task under the wrong runtime env. This folds with `hash ^ (hash >> 32)` and keeps a non-negative 31-bit key (`& 0x7fffffff`) so existing `int`-typed call sites stay compatible while using more of the original hash entropy. ## Related issues Fixes #64836 ## Additional information Minimal change confined to `CalculateRuntimeEnvHash`; no API surface type widen (full `int64_t` migration left as follow-up if maintainers want it).", "url": "https://github.com/ray-project/ray/pull/64895", "timestamp": "2026-08-12T12:57:39Z", "metrics": { "reactions": 1, "comments": 2 }, "labels": [ "stale", "core", "community-contribution" ], "author": "Solaris-star", "assignees": [], "change": "new" }, { "id": "github:ray-project/ray:pull_request:64928", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[Train] Add NCCL RAS health callback", "text": "## Description This PR adds an opt-in callback that detects hanging distributed training jobs by polling NCCL's RAS (Reliability/Availability/Serviceability) subsystem. NCCL β‰₯ 2.24 runs a monitoring thread inside every NCCL process that forms a peer mesh tracking per-rank health and collective op-counts. By querying the `ncclras` client and diffing collective-counts between successive polls, Ray Train can automatically detect and act on hangs that would otherwise stall a job until a torch process group timeout occurs. As RAS gives a snapshot of the current NCCL state, to determine if the mismatch is a temporary issue or a permanent one, the callback polls RAS every N seconds. If a communicator is mismatched *and* makes no progress at all across multiple sequential polls, then this most likely indicates that a deadlock / wedge / hang is in progress for that communicator. Its important to note that its possible for NCCL to continue (not hang) with incorrect data depending on the nccl operation, tensor size, nccl version, etc. In these cases, users will observe the collective-count continue to increase but often with a constant mismatch. In this version of the callback, we are not detecting those issues and focusing on purely NCCL collective mismatches which cause hangs. The callback is disabled by default and enabled with `RAY_TRAIN_ENABLE_NCCL_HANG_DETECTOR=1`, as the detection heuristics still need validation against real-world hangs (spot preemptions, pipeline-parallel send/recv comms that legitimately sit skewed for long periods) before it can be considered for default-on. ### Requirements - RAS itself landed in 2.24, but the poller parses `ncclras -f json`, and the `-f` flag requires NCCL 2.28+. If the binary rejects `-f`, or isn't on PATH, the callback logs a warning once and disables itself for the rest of the run rather than retrying forever. - `ci/docker/base.gpu.Dockerfile` pins and holds `libnccl2`/`libnccl-dev` at `2.28.9-1+cuda12.9` (previously whatever the CUDA base image shipped) so `ncclras` is present and JSON-capable in GPU CI. The build fails fast if the pin doesn't stick or the client binary is missing. This affects all GPU CI images, not just Train tests. - `py-spy` (and `SYS_PTRACE` in the container) for native stack traces at hang time. When unavailable, the callback falls back to a Python-only traceback of every thread. ### How it works - `NCCLRASCallback` lives on the controller and polls `ncclras` on a worker (ras forms a mesh-network between ranks requiring that only one of the workers to be polled and receive the whole network's state). Workers are tried in turn until one returns a usable report, and the query runs on a background thread so it never blocks the controller's poll loop. - A communicator's frozen-streak only advances when the RAS report shows a collective-count mismatch, every rank in that communicator is `RUNNING`, and *no* rank advanced *any* collective since the previous poll. Any progress on any op resets that communicator's streak; streaks are tracked per communicator so each one is confirmed on its own. - A confirmed hang requires the streak to persist for `RAY_TRAIN_NCCL_RAS_CONFIRM_DURATION_S` (default 600s), converted to a consecutive-poll count using `RAY_TRAIN_NCCL_RAS_MIN_POLL_INTERVAL_S` (default 15s), 40 polls by default, to ensure that a mismatch isn't a snapshot issue. - Before confirmation there is an escalation ladder: a `WARNING` when a communicator first crosses ~60s frozen, then a periodic `WARNING` every ~120s naming every still-frozen communicator, its stalled duration, and (in `fail` mode) the time remaining until `NCCLHangError`, alongside the human-readable `ncclras -f text` report. A communicator that resumes progress after being suspected logs an explicit recovery message. - On a confirmed hang, the callback logs the `ncclras -f text` report and captures py-spy stack traces from all workers (uploaded to `/nccl_ras_hang_stack_traces/` as `rank_.log`), then depending on the callback mode, in \"fail\" a `NCCLHangError` is raised and in \"observe\", a detailed log message is produced but no more. `observe` is the default; set `RAY_TRAIN_NCCL_RAS_ACTION=fail` to fail the run. - `NCCLHangError` (a `WorkerGroupError` subclass, `DeveloperAPI`, exported as `ray.train.NCCLHangError`) is treated as non-retryable by the `DefaultFailurePolicy` regardless of the `max_failures` budget, since a desync hang is usually deterministic and a restart would just hang again. ### Configuration | Env var | Default | Read on | | --- | --- | --- | | `NCCL_RAS_ADDR` | `localhost:28028` | worker | | `RAY_TRAIN_ENABLE_NCCL_HANG_DETECTOR` | `0` | driver | | `RAY_TRAIN_NCCL_RAS_ACTION` | `observe` | driver | | `RAY_TRAIN_NCCL_RAS_MIN_POLL_INTERVAL_S` | `15` | driver | | `RAY_TRAIN_NCCL_RAS_CONFIRM_DURATION_S` | `600` | driver | | `RAY_TRAIN_NCCLRAS_PATH` | `ncclras` (PATH lookup) | driver | The `RAY_TRAIN_*` knobs are read once when the callback is constructed on the driver validated eagerly (invalid values raise at construction); `NCCL_RAS_ADDR` is NCCL's variable and is read on the worker at query time. ### Known limitations - **Symmetric in-collective hangs are not detected.** RAS reports collective *launch* counts, so if every rank launches the same collective and the fabric wedges mid-op, all counts match and skew is zero. - **A rank that exited early is not detected.** A communicator is only considered for detection when all of its ranks report `RUNNING`, so a `FINALIZE`/`ABORT` rank alongside frozen peers is currently skipped. This could be considered in this implementation but I believe the Ray health checker should catch this first. - `missing_ranks` (unresponsive / considered-dead peers) is parsed around but not yet used as a signal. - No metrics are emitted yet, and there is no culprit-rank attribution β€” `NCCLHangError.worker_failures` is empty; the RAS text report and stack traces are the only diagnostics. ### Testing - `test_nccl_ras_callback.py` β€” CPU unit coverage of RAS JSON parsing (including the malformed `missing_ranks[]` comma NCCL 2.28.9 emits), `NCCL_RAS_ADDR` parsing, frozen-vs- advancing classification, per-communicator streak independence, communicators appearing and disappearing between polls, poll throttling, confirm-duration β†’ poll-count conversion, config validation, the escalation/observe messaging, and that a failed stack dump does not suppress the hang error. - `test_nccl_ras_hang_detection.py` β€” GPU end-to-end scenarios (`train_v2_gpu`) inducing real NCCL desyncs in a `TorchTrainer` with the detector configured for fast confirmation. Requires β‰₯ 2 visible GPUs and `ncclras` on PATH; the multi-communicator subset test requires 4 GPUs and skips otherwise. - `test_failure_policy.py` β€” added `test_nccl_hang_error_is_non_retryable`.", "url": "https://github.com/ray-project/ray/pull/64928", "createdAt": "2026-07-22T10:44:09Z", "updatedAt": "2026-08-13T17:20:01Z", "timestamp": "2026-08-13T17:20:01Z", "metrics": { "reactions": 1, "comments": 2 }, "labels": [ "train" ], "author": "pseudo-rnd-thoughts", "state": "open", "assignees": [] }, { "id": "github:ray-project/ray:pull_request:64986", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[Core] Support dynamic node labeling at runtime", "text": "## Description Ray only lets you set node labels at startup, through `ray start --labels`, `RAY_OVERRIDE_LABELS`, or `--labels-file`. After a node is up, its labels are fixed for the life of the raylet. On a long-running cluster that is a poor fit: the labels you want on a node change over time, and the only way to change them today is to restart the node. Without a runtime update, the workaround is to pre-declare every worker group with every label it might ever need. That makes cluster setup awkward and still cannot express labels that depend on runtime state. This PR adds `ray.experimental.update_node_labels(node_id, labels)` to change a node's user-defined labels while the cluster runs, so `label_selector`-based scheduling can be redirected toward or away from a node as its situation changes. ## Related issues Related to #45375 ## Additional information ### End-to-end run #### demo.py ```py import ray import ray.experimental ray.init(address=\"auto\") node_id = ray.nodes()[0][\"NodeID\"] print(\"node id:\", node_id) print(\"labels at ray start :\", ray.nodes()[0][\"Labels\"]) # Replace user labels: change tier, drop zone, add team. ray.experimental.update_node_labels(node_id, {\"tier\": \"gpu\", \"team\": \"ml\"}) print(\"labels after update :\", ray.nodes()[0][\"Labels\"]) # Clearing keeps the reserved ray.io/ labels. ray.experimental.update_node_labels(node_id, {}) print(\"labels after clear :\", ray.nodes()[0][\"Labels\"]) ``` #### Result \"image\"", "url": "https://github.com/ray-project/ray/pull/64986", "timestamp": "2026-08-12T12:31:19Z", "metrics": { "reactions": 1, "comments": 2 }, "labels": [ "core", "community-contribution", "unstale" ], "author": "nadongjun", "assignees": [], "change": "new" }, { "id": "github:ray-project/ray:pull_request:65015", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[core][dashboard] Return 4xx from node and actor detail APIs", "text": "## Description Makes three node/actor dashboard endpoints report client errors with 4xx status codes instead of 200/500. [#51417](https://github.com/ray-project/ray/pull/51417) added the `HTTPStatusCode` enum and the `rest_response(status_code=...)` plumbing, but adoption stalled after one endpoint. Across non-test dashboard source there are currently 28 `OK` and 13 `INTERNAL_ERROR` usages versus only 4 non-500 error codes, all confined to `state_head.py` / `state_api_utils.py`. Backend (`python/ray/dashboard/modules/node/node_head.py`): - `GET /nodes/{node_id}` returned **200 with a near-empty payload** for an unknown node ID, because `DataOrganizer.get_node_info` falls back to `.get(node_id, {})`. Now returns **404**. - `GET /logical/actors/{actor_id}` returned **200 with `detail: null`** for an unknown actor ID, because `get_actor_infos` maps unknown IDs to `None`. Now returns **404**. - `GET /nodes` with an unsupported `view` returned **500**. An unsupported query parameter is a client error, so it now returns **400**. ## Related issues Related to [#51442](https://github.com/ray-project/ray/issues/51442) (umbrella: revisit Ray dashboard API status codes). Since that issue is an umbrella, this PR covers `node_head.py` only, so each module can be reviewed independently. ## Additional information ### Frontend change is required, not incidental The node and actor detail pages surfaced these cases by reading `result === false` out of the **200 response body**. axios rejects on 4xx, so those branches go dead the moment the status code changes, and the pages would sit on \"Loading the node infos...\" indefinitely with `msg` never updated. The SWR fetchers in `useNodeDetail.ts` and `useActorDetail.ts` now catch the 404 and set the same message they set before, preserving existing UI behavior. Consumers of `useFetchActor` (the Serve pages) already use optional chaining on the result, so returning `undefined` there is unchanged behavior. ### Tests Added `test_node_api_status_codes` (single cluster, looping over all three error paths plus the three success paths β€” not parametrized, since `ray_start_with_dashboard` starts a fresh cluster per invocation) and `test_actor_not_found_status_code`. Test environment: because Ray's core is compiled, these were run against the Ray Windows nightly wheel built from `3fb63d965f` β€” the exact base commit of this branch β€” with the patched `node_head.py` overlaid, so the patch was the only delta. ``` pytest -v python/ray/dashboard/modules/node/tests/test_node.py -k test_node_api_status_codes 6 passed in 122.16s pytest -v python/ray/dashboard/modules/node/tests/test_actor.py -k test_actor_not_found_status_code 1 passed in 19.74s pytest -q python/ray/dashboard/modules/node/tests/test_actor.py 5 passed in 142.51s pytest -q python/ray/dashboard/modules/node/tests/test_node.py 7 passed, 4 skipped, 2 failed in 286.73s ``` The two failures in the full `test_node.py` run are `test_node_info` and `test_worker_pids_reported`. Both **fail identically against unpatched `node_head.py`** and are pre-existing Windows-environment issues, not regressions: worker cmdlines don't match `\"ray::Actor\"` on Windows, and runtime_env's uv virtualenv setup hits `WinError 206` (path length). Control run confirming the new tests are not vacuous β€” against unpatched code: - `/nodes/{unknown}` β†’ `assert 200 == 404` (fails) - `/nodes?view=unknown_view` β†’ `assert 500 == 400` (fails) - `/nodes` β†’ `assert 500 == 400` (fails) - the three success paths β†’ still pass - actor test β†’ `assert 200 == 404`, body `{\"result\": true, \"msg\": \"Actor details fetched.\", \"data\": {\"detail\": null}}` (fails) Frontend checks: ``` npx tsc --noEmit # clean, exit 0 npx eslint # clean npx prettier -c # clean ``` Python lint: `ruff check` clean; `black` leaves the changed files unmodified. Caveats worth reviewer attention: all of the above ran on Windows rather than Ray's Linux CI, and against a nightly wheel plus overlaid file rather than a from-source build. Linux CI remains the authoritative check. ### Not a duplicate No open PR references [#51442](https://github.com/ray-project/ray/issues/51442), and the issue timeline has no cross-referenced PRs. The issue has been assigned since 2025-03-18 with no linked work; I commented on it before starting ([comment](https://github.com/ray-project/ray/issues/51442#issuecomment-5084257594)). ### AI assistance AI assistance (Claude Code) was used to produce this change.", "url": "https://github.com/ray-project/ray/pull/65015", "createdAt": "2026-07-26T17:41:02Z", "updatedAt": "2026-08-12T18:52:12Z", "timestamp": "2026-08-12T18:52:12Z", "metrics": { "reactions": 1, "comments": 3 }, "labels": [ "dashboard", "core", "go", "community-contribution" ], "author": "chenyuan99", "state": "open", "assignees": [ "MortalHappiness" ] }, { "id": "github:ray-project/ray:pull_request:65020", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "Bump transformers from 4.36.2 to 5.5.0 in /python", "text": "Bumps [transformers](https://github.com/huggingface/transformers) from 4.36.2 to 5.5.0.
Release notes

Sourced from transformers's releases.

Release v5.5.0

New Model additions

Gemma4

Gemma 4 is a multimodal model with pretrained and instruction-tuned variants, available in 1B, 13B, and 27B parameters. The architecture is mostly the same as the previous Gemma versions. The key differences are a vision processor that can output images of fixed token budget and a spatial 2D RoPE to encode vision-specific information across height and width axis.

You can find all the original Gemma 4 checkpoints under the Gemma 4 release.

The key difference from previous Gemma releases is the new design to process images of different sizes using a fixed-budget number of tokens. Unlike many models that squash every image into a fixed square (like 224Γ—224), Gemma 4 keeps the image's natural aspect ratio while making it the right size. There a a couple constraints to follow:

  • The total number of pixels must fit within a patch budget
  • Both height and width must be divisible by 48 (= patch size 16 Γ— pooling kernel 3)

[!IMPORTANT] Gemma 4 does not apply the standard ImageNet mean/std normalization that many other vision models use. The model's own patch embedding layer handles the final scaling internally (shifting values to the [-1, 1] range).

The number of "soft tokens" (aka vision tokens) an image processor can produce is configurable. The supported options are outlined below and the default is 280 soft tokens per image.

Soft Tokens Patches (before pooling) Approx. Image Area
70 630 ~161K pixels
140 1,260 ~323K pixels
280 2,520 ~645K pixels
560 5,040 ~1.3M pixels
1,120 10,080 ~2.6M pixels

To encode positional information for each patch in the image, Gemma 4 uses a learned 2D position embedding table. The position table stores up to 10,240 positions per axis, which allows the model to handle very large images. Each position is a learned vector of the same dimensions as the patch embedding. The 2D RoPE which Gemma 4 uses independently rotate half the attention head dimensions for the x-axis and the other half for the y-axis. This allows the model to understand spatial relationships like "above," "below," "left of," and "right of."

NomicBERT

NomicBERT is a BERT-inspired encoder model that applies Rotary Position Embeddings (RoPE) to create reproducible long context text embeddings. It is the first fully reproducible, open-source text embedding model with 8192 context length that outperforms both OpenAI Ada-002 and OpenAI text-embedding-3-small on short-context MTEB and long context LoCo benchmarks. The model generates dense vector embeddings for various tasks including search, clustering, and classification using specific instruction prefixes.

Links: Documentation | Paper

MusicFlamingo

Music Flamingo is a fully open large audio–language model designed for robust understanding and reasoning over music. It builds upon the Audio Flamingo 3 architecture by including Rotary Time Embeddings (RoTE), which injects temporal position information to enable the model to handle audio sequences up to 20 minutes. The model features a unified audio encoder across speech, sound, and music with special sound boundary tokens for improved audio sequence modeling.

Links: Documentation | Paper

... (truncated)

Commits
  • c1c3424 update
  • 20bff68 update release workflow
  • 8956441 v5.5.0
  • 5135e5e casually dropping the most capable open weights on the planet (#45192)
  • a594e09 Internalise the NomicBERT model (#43067)
  • 4932e97 Fix resized LM head weights being overwritten by post_init (#45079)
  • 57e8413 [Qwen3.5 MoE] Add _tp_plan to ForConditionalGeneration (#45124)
  • b10552e Fix TypeError: 'NoneType' object is not iterable in GenerationMixin.generate ...
  • 423f2a3 fix(models): Fix dtype mismatch in SwitchTransformers and TimmWrapperModel (#...
  • ade7a05 Generalize gemma vision mask to videos (#45185)
  • Additional commits viewable in compare view

", "url": "https://github.com/ray-project/ray/pull/65020", "timestamp": "2026-08-12T12:31:22Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "stale", "dependencies", "python", "train", "core" ], "author": "dependabot[bot]", "assignees": [], "change": "new" }, { "id": "github:ray-project/ray:pull_request:65031", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[Serve] Optimize RollingWindow metrics using monotonic deque in O(1)", "text": "### What is the problem this PR is trying to solve? This PR optimizes the metrics calculations in `RollingWindowMax` and `RollingWindowMin` by replacing the `O(num_buckets)` array scan with an amortized `O(1)` monotonic deque. It also fixes a correctness bug in the legacy implementation where the currently accumulating active bucket was ignored during `get_max()` / `get_min()` evaluations until it rolled over, which could blind autoscaler metrics to immediate traffic spikes. **Performance Improvements** Microbenchmarks measuring 2,000,000 concurrent writes (`add()`) and 500,000 reads (`get_max()`) across 10 threads yielded massive query speedups with no space complexity penalty (still strictly bounded to `O(num_buckets)`): - **Reads (`get_max`)**: 30x faster (0.58s vs 17.86s) - **Mixed Concurrent Workload**: 14x faster (0.77s vs 10.84s) - **Writes**: Marginally faster (0.74s vs 0.83s) ### AI Assistance AI assistance was used to analyze the codebase, implement the monotonic deque, and run performance validations. - **Non-duplication**: Verified via `gh pr list` and `gh issue list` that no existing PR or issue addresses this optimization or the uncommitted-bucket visibility bug. - **Tests run & results**: - `pytest python/ray/serve/tests/unit/test_rolling_window.py`: All 80 concurrent edge-case tests (including newly added parity tests for `RollingWindowMin`) passed successfully. - `pre-commit run --files python/ray/serve/_private/rolling_window.py python/ray/serve/tests/unit/test_rolling_window.py`: Formatting (ruff/black) and static analysis (mypy) passed.", "url": "https://github.com/ray-project/ray/pull/65031", "createdAt": "2026-07-27T09:48:46Z", "updatedAt": "2026-08-13T00:39:08Z", "timestamp": "2026-08-13T00:39:08Z", "metrics": { "reactions": 1, "comments": 2 }, "labels": [ "serve", "community-contribution", "unstale" ], "author": "pavan51", "state": "open", "assignees": [] }, { "id": "github:ray-project/ray:pull_request:65071", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[train] set Neuron rendezvous env vars in the XLA backend", "text": "## Description `_TorchAwsNeuronXLABackend` cannot bring up a Neuron collective that spans instances. On two `trn1.32xlarge` hosts with 64 workers at one NeuronCore each, backend startup fails inside `_setup_xla_torch_process_group` with ``` Nrt::BuildGlobalComm failed on NeuronCores 0-1(2): nrt_status=1, message=\"Non specific failure\". ``` The Neuron runtime builds a cross-instance communicator by having every rank meet at the endpoint named in `NEURON_RT_ROOT_COMM_ID`, and nothing in Ray sets it. `on_start` already asks worker 0 for `get_address_and_port` and uses the answer for `MASTER_ADDR` and `MASTER_PORT`, so the address is already in hand; it is just never exported under the name the runtime reads. A single instance has nothing to rendezvous with, which is why this is invisible until the job crosses a host boundary. This patch takes a second free port from worker 0 and sets `NEURON_RT_ROOT_COMM_ID=:` in the existing `set_env_vars` closure, so every worker gets the same string. A second `get_address_and_port` call rather than `MASTER_PORT` plus an offset, because the runtime needs an endpoint of its own and `get_address_and_port` returns a port it has just verified free. Setting that alone was not sufficient in my environment, which is the part I did not expect. With the root comm id correct on every rank, the run then died in the bootstrap: ``` CCOM WARN Net : No interface found in the same subnet as remote address <48820> CCOM WARN NET/Socket : No usable listening interface found ``` CCOM selects its bootstrap socket by enumerating local interfaces and looking for one whose subnet contains the peer address. Under the AWS VPC CNI each pod holds a /32, so no local interface is ever in the same subnet as a peer and the search returns nothing. Naming the interface skips the search. I want to be clear that this half is deployment specific: the /32 comes from the CNI, and on bare EC2 instances where the host address sits in a real subnet I would expect the match to succeed and this failure never to appear. I have not tested that and cannot confirm it. The `NEURON_RT_ROOT_COMM_ID` half is not conditional on any of that. So the interface is an opt-in `socket_ifname` field on `TorchXLAConfig` defaulting to `None`, not a hardcoded `eth0`. `eth0` is correct for a VPC CNI pod and wrong in general, and #42808 already removed `DEFAULT_NCCL_SOCKET_IFNAME` from Ray Train because users prefer these left alone; defaulting to `None` keeps that promise while making the knob discoverable next to `neuron_parallel_compile`. When the field is set the backend uses `os.environ.setdefault`, so a value the user already exported wins, matching the `TORCH_NCCL_ASYNC_ERROR_HANDLING` guard in `python/ray/train/torch/config.py`. Open question for the reviewer, since I can see the argument both ways. The two interface variables can already be set through `runtime_env` `env_vars`, so the field buys discoverability rather than capability, and you may prefer to drop it and document the `runtime_env` route instead. `NEURON_RT_ROOT_COMM_ID` genuinely cannot be set that way, because its correct value is not known until the worker group exists. Happy to cut the field if you would rather keep the config surface small. `NEURON_RT_ROOT_COMM_ID` itself is set unconditionally, matching how `MASTER_ADDR` and `MASTER_PORT` are handled two lines above. If you would rather it also yield to a pre-existing value, say so and I will change it. ## Related issues Closes #65070 ## Additional information Tests: - Verified on 2 x `trn1.32xlarge` (32 NeuronCores each, 64 Ray Train workers at one core per worker) on Kubernetes with the AWS VPC CNI and EFA enabled, running ray 2.56.1, torch-neuronx 2.8.0.2.10.16998, torch-xla 2.8.1, neuronx-cc 2.21.33363.0, Python 3.11.11. Before the change, backend startup fails with `Nrt::BuildGlobalComm failed`. After it, the process group forms and a 64 rank job trains across both hosts: 24 steady state steps, no NaN, and throughput around 90 percent of twice the single instance rate measured on the same two hosts in the same session. - The equivalent of this change was carried as a `TorchXLAConfig` subclass overriding `on_start` before being written as a patch, and that is what the run above used. The in-tree version differs only in taking the interface name from the config instead of hardcoding it. - `black` and `ruff` clean on the changed file, at the repo's 88 column line length. - No unit test added. There is no existing test module for this backend, and the behaviour that matters here is not observable without Trainium hardware: the env vars are trivially assertable with a mocked worker group, but that only tests that the strings are spelled correctly. I did not find a Trainium multi-instance runner in Ray's CI, so this path cannot be covered there. If you want a mock-based unit test asserting the env vars land on every worker, say the word and I will add one. - Single instance behaviour is unchanged in substance. `NEURON_RT_ROOT_COMM_ID` is set but unused when there is nothing to rendezvous with, and `socket_ifname` defaults to `None` so no interface variable is touched unless asked for.", "url": "https://github.com/ray-project/ray/pull/65071", "createdAt": "2026-07-28T08:22:51Z", "updatedAt": "2026-08-13T00:39:12Z", "timestamp": "2026-08-13T00:39:12Z", "metrics": { "reactions": 1, "comments": 3 }, "labels": [ "stale", "train", "community-contribution" ], "author": "violivei", "state": "open", "assignees": [] }, { "id": "github:ray-project/ray:pull_request:65072", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[core] feat(rdt): enable driver-side ray.put with NIXL tensor transport", "text": "Allow ray.put(value, _tensor_transport='nixl') from a driver process. Previously this failed because get_runtime_context().current_actor raises RuntimeError in a non-actor context. Changes: - Make RDTMeta.src_actor and TransferMetadata.src_actor Optional - Catch RuntimeError in put_object and set src_actor=None for driver - Reject two-sided transports (NCCL/GLOO) from driver with clear error - Guard src_actor dereferences in _abort_transport and transfer paths - Handle free_object_primary_copy locally for driver-owned objects - Raise ValueError for _use_object_store=True on driver-owned RDT objects - Align worker.py error to RuntimeError with 'one-sided transport' message Fixes ray-project/ray#64714 ## Tests Three new tests in `python/ray/tests/rdt/test_rdt_nixl.py`: ### `test_driver_put_nixl` End-to-end driver-side NIXL flow: 1. `ray.put(tensor, _tensor_transport=\"nixl\")` from the driver 2. Verify `src_actor is None` in RDT metadata 3. Verify tensor registered in NIXL transport 4. Actor consumes the driver-created ref via `actor.sum.remote(ref)` 5. Driver-local `ray.get(ref)` returns the original tensor 6. After `del ref` + `gc.collect()`, verify cleanup: tensor freed, RDT metadata removed, NIXL metadata removed, tensor descriptor cache cleared ### `test_driver_owned_rdt_rejects_object_store_fallback` Verify `_use_object_store=True` on a driver-owned RDT object raises `ValueError` matching `_use_object_store=True` instead of crashing with `AttributeError` on `None.__ray_call__`. ### `test_driver_rejects_two_sided_transport` Verify `ray.put(tensor, _tensor_transport=\"nccl\")` from the driver raises `RuntimeError` matching `one-sided transport`. Two-sided transports require an actor context to coordinate the send. ### Running ```bash # New driver tests pytest -sv python/ray/tests/rdt/test_rdt_nixl.py \\ -k \"test_driver_put_nixl or test_driver_rejects_two_sided_transport or test_driver_owned_rdt_rejects_object_store_fallback\" ``` ### Results | Test | Result | |------|--------| | `test_driver_put_nixl` | PASSED | | `test_driver_owned_rdt_rejects_object_store_fallback` | PASSED | | `test_driver_rejects_two_sided_transport` | PASSED |", "url": "https://github.com/ray-project/ray/pull/65072", "createdAt": "2026-07-28T10:28:32Z", "updatedAt": "2026-08-13T17:02:53Z", "timestamp": "2026-08-13T17:02:53Z", "metrics": { "reactions": 1, "comments": 2 }, "labels": [ "core", "community-contribution" ], "author": "n3sfan", "state": "open", "assignees": [ "stephanie-wang", "Sparks0219" ] }, { "id": "github:ray-project/ray:pull_request:65077", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "fix json bug", "text": "## What happened + What you expected to happen Deploying a Ray Serve LLM (`ray.serve.llm.build_openai_app`) with a nested dataclass field in `engine_kwargs` β€” e.g. `structured_outputs_config` β€” crashes the vLLM engine on startup with: ``` AttributeError: 'dict' object has no attribute 'reasoning_parser' ``` or, depending on which field is accessed first: ``` AttributeError: 'dict' object has no attribute 'compute_hash' ``` **Expected:** the engine starts successfully and `structured_outputs_config` is applied to the model, exactly as it would with vLLM's own CLI (`--structured-outputs-config.enable_in_reasoning=True`). ### Why this matters This blocks the *documented* upstream vLLM fix for broken structured outputs on Qwen3 Coder + reasoning models ([vllm-project/vllm#18819](https://github.com/vllm-project/vllm/issues/18819)), which requires setting `structured_outputs_config.enable_in_reasoning=True`. Ray Serve LLM users deploying via YAML (`serveConfigV2` / `LLMConfig.engine_kwargs`) have no working way to set this flag β€” the only way to express a nested vLLM config field from YAML is as a plain dict, and that dict reaches `AsyncEngineArgs` unconverted. ## Root cause `_get_vllm_engine_config()` in `python/ray/llm/_internal/serve/engines/vllm/vllm_engine.py` builds the engine config like this: ```python async_engine_args = vllm.engine.arg_utils.AsyncEngineArgs( **engine_config.get_initialization_kwargs() ) vllm_engine_config = async_engine_args.create_engine_config( usage_context=UsageContext.OPENAI_API_SERVER ) ``` `engine_config.get_initialization_kwargs()` returns a plain `dict` for every `engine_kwargs` entry, including nested ones like `structured_outputs_config`. `vllm.engine.arg_utils.EngineArgs.__post_init__` only coerces a fixed allowlist of fields from dict to their dataclass type (`compilation_config`, `attention_config`, `mamba_config`, `kernel_config`, `eplb_config`, `weight_transfer_config`, `ir_op_priority`). `structured_outputs_config` and `reasoning_config` are **not** in that list, so they stay as plain `dict` objects on the `AsyncEngineArgs` instance. `create_engine_config()` then does (vLLM `engine/arg_utils.py`): ```python if self.reasoning_parser: self.structured_outputs_config.reasoning_parser = self.reasoning_parser ``` β€” which crashes immediately with `AttributeError: 'dict' object has no attribute 'reasoning_parser'` whenever a top-level `engine_kwargs.reasoning_parser` is also set (a very common config for Qwen3 reasoning models). Even without that top-level field, `VllmConfig.compute_hash()` later calls `self.structured_outputs_config.compute_hash()`, which crashes the same way. ### There already is a fix for a *similar* symptom β€” but it doesn't cover this path #60380 (`[LLM] Fix nested dict to Namespace conversion in vLLM engine initialization`, merged) introduced a helper, `_convert_config_dicts()`, that converts known `AsyncEngineArgs` dict fields into their proper dataclass instances using field type hints. It is called in `VLLMEngine.start()`: ```python merged = vllm_frontend_args.__dict__ | vllm_engine_args.__dict__ merged = _convert_config_dicts(merged) args = _dict_to_namespace(merged) self._vllm_args = args ``` This fixes `AttributeError` in the **serving/frontend layer** (`build_asgi_app()` / `init_app_state()`, which reads `args.structured_outputs_config.backend` etc. to configure the OpenAI-compatible endpoint). But `self._vllm_args` is only built *after* the engine itself has already been started (`self._start_async_llm_engine(vllm_engine_args, ...)`), using the args that came out of `_get_vllm_engine_config()`. That function builds `AsyncEngineArgs` and calls `create_engine_config()` **before** `_convert_config_dicts()` ever runs β€” so the crash described above still reproduces on top of #60380. ## Reproduction ```python from ray.llm._internal.serve.core.configs.llm_config import LLMConfig, ModelLoadingConfig from ray.llm._internal.serve.engines.vllm.vllm_engine import _get_vllm_engine_config llm_config = LLMConfig( model_loading_config=ModelLoadingConfig( model_id=\"qwen-test\", model_source=\"Qwen/Qwen3-4B\" ), engine_kwargs={ \"reasoning_parser\": \"qwen3\", \"structured_outputs_config\": {\"enable_in_reasoning\": True}, }, ) _get_vllm_engine_config(llm_config) # AttributeError: 'dict' object has no attribute 'reasoning_parser' ``` Or via a RayService `serveConfigV2`: ```yaml applications: - name: llms import_path: ray.serve.llm:build_openai_app route_prefix: \"/\" args: llm_configs: - model_loading_config: model_id: qwen3-4b-thinking model_source: /models/Qwen3-4B engine_kwargs: tensor_parallel_size: 2 reasoning_parser: qwen3 structured_outputs_config: backend: xgrammar enable_in_reasoning: true deployment_config: autoscaling_config: min_replicas: 1 max_replicas: 1 ``` The vLLM engine actor crashes on startup (`CrashLoopBackOff` under KubeRay). ## Related - vllm-project/vllm#18819 β€” the upstream fix this blocks (`enable_in_reasoning`) - ray-project/ray#60380 β€” fixes the same class of bug in the serving/frontend layer, but not this path - ray-project/ray#60430 β€” regression test for #60380", "url": "https://github.com/ray-project/ray/pull/65077", "createdAt": "2026-07-28T11:40:21Z", "updatedAt": "2026-08-13T12:31:43Z", "timestamp": "2026-08-13T12:31:43Z", "metrics": { "reactions": 1, "comments": 2 }, "labels": [ "serve", "stale", "community-contribution" ], "author": "MiXaiLL76", "state": "open", "assignees": [] }, { "id": "github:ray-project/ray:pull_request:65098", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[core][cgraph] Deprecate Ray Compiled Graphs", "text": "## Why are these changes needed? Marks Ray Compiled Graphs (cgraph / aDAG) as deprecated, pointing users to **Ray Direct Transport (RDT)** as the successor for direct GPU-to-GPU tensor transfer between actors. The wind-down has already informally started (the GPU cgraph test target is disabled with a \"will be deprecated/removed soon\" note, and all `compiled_graphs*` release benchmarks are `frequency: manual`); this makes it official and user-visible. **What this does:** - `DAGNode.experimental_compile()` emits a `RayDeprecationWarning` β€” the single funnel all cgraph usage flows through. - Deprecation notes on the `CompiledDAG`, `DAGContext`, and `CompiledDAGRef` docstrings. - Docs: the compiled-graph landing page becomes \"Ray Compiled Graph (deprecated)\" with a banner linking to :ref:`direct-transport`; the same banner is added to the quickstart/profiling/overlap/troubleshooting/API sub-pages; the recommendation note in `ray-dag.rst` is updated. - Adds `test_experimental_compile_deprecation_warning` asserting the warning fires. **What this deliberately does NOT do:** - The feature is unchanged and fully functional. `compiled_graphs` CI targets and the manual-frequency `compiled_graphs*` release benchmarks keep working; the warning fires once at compile time, outside benchmark timing loops. - Shared infra reused by RDT and Serve is untouched: `ray.experimental.channel`, `ray.experimental.collective`, the base `ray.dag` nodes (`InputNode`/`DAGNode`/`MultiOutputNode`, `py_obj_scanner` β€” used by Serve), and `with_tensor_transport`. ## Related issue number N/A ## Checks - Not a duplicate: searched open PRs for compiled-graph deprecation work; none exists. - AI assistance was used for this PR (Claude Code); the change has been reviewed line-by-line. - Tests: verified against a real Ray build (edited module overlaid onto ray 2.56.1): the `RayDeprecationWarning` fires with the expected message, `pytest.warns` in the new test passes, and a compiled DAG still executes end-to-end and tears down cleanly afterward. Ray's root `pytest.ini` ignores warnings globally, so existing cgraph tests are unaffected. πŸ€– Generated with [Claude Code](https://claude.com/claude-code)", "url": "https://github.com/ray-project/ray/pull/65098", "timestamp": "2026-08-12T12:31:27Z", "metrics": { "reactions": 1, "comments": 1 }, "labels": [ "stale", "docs", "core", "deprecation" ], "author": "elliot-barn", "assignees": [], "change": "new" }, { "id": "github:ray-project/ray:pull_request:65107", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[doc][KubeRay] Add mTLS for RayClusters user guide", "text": "## Description Adding user guide for mTLS configuration with RayClusters via KubeRay as per the new alpha feature in KubeRay 1.7 ## Related issues Relates to https://github.com/ray-project/kuberay/issues/5048 ## Additional information N/A", "url": "https://github.com/ray-project/ray/pull/65107", "createdAt": "2026-07-29T16:40:10Z", "updatedAt": "2026-08-13T11:15:12Z", "timestamp": "2026-08-13T11:15:12Z", "metrics": { "reactions": 1, "comments": 7 }, "labels": [ "docs", "core", "go", "community-contribution" ], "author": "chipspeak", "state": "open", "assignees": [ "machichima" ] }, { "id": "github:ray-project/ray:pull_request:65132", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[core][gcs][Actice-Passive] Phase 2.1 Add Active-Passive leader-election interface (protocol, status, client cache)", "text": "## Description This is the foundational, interface-only PR for the Active-Passive GCS feature (split from the larger #64422). It introduces the wire-protocol field, status code, config flag, and client-side leadership-caching plumbing β€” all feature-gated behind `LEADER_ELECT` / `RAY_LEADER_ELECT`, which **defaults to OFF**. No server-side passive behavior is included here; this PR is pure additive scaffolding that later PRs build on. ## Issue https://github.com/ray-project/ray/issues/63643 ## High-level changes | Layer | Change | | :--- | :--- | | **Protocol** | Add `optional bool is_leader = 4` to `CheckAliveReply`. Marked `optional` so a new client can distinguish \"field absent\" (an old GCS during a rolling upgrade) from an explicit `false`. | | **Status** | New `StatusCode::GcsPassive` (38) with `Status::GcsPassive()` and `IsGcsPassive()`, registered in the code↔string maps. No code path returns it yet. | | **Config** | New `RAY_CONFIG(bool, LEADER_ELECT, false)` (C++) and matching `ray_constants.RAY_LEADER_ELECT` (Python), kept in sync via the `RAY_LEADER_ELECT` env var. | | **C++ client** | `NodeInfoAccessor` caches an `atomic is_gcs_leader_`, updated from each `CheckAlive` reply (`has_is_leader() ? is_leader() : true` β€” absence is treated as the legacy leader). Exposes `IsGcsLeader()`. When `LEADER_ELECT` is off, it initializes to `true` and never changes (legacy behavior). The test-only default constructor sets `client_impl_ = nullptr`. | | **Python bindings** | `GcsClient.is_gcs_leader_local()` (reads the cache) and `is_gcs_leader()` (triggers a `CheckAlive`, then reads the cache). Both return `True` when `RAY_LEADER_ELECT` is off. `check_status` maps `GcsPassive β†’ RpcError(UNAVAILABLE)`. | ## Behavior change / backward compatibility - **With `LEADER_ELECT` off (the default): zero observable change.** `is_gcs_leader()` / `is_gcs_leader_local()` always return `True`; the `GcsPassive` status code and the `is_leader` field are never produced. - **Wire compatibility:** the new proto field is `optional` and defaults to absent, so old/new clients and servers interoperate during rolling upgrades (absence = legacy leader). - **No public API signatures change**; all additions are new symbols. ## Test plan **Unit (C++)** β€” accessor_test.cc - `TestIsGcsLeaderCaching` β€” cache defaults to `true`; `store`/`load` behave correctly. - `TestLeaderElectionDisabledByDefault` β€” with `LEADER_ELECT` off, the accessor always reports itself as leader regardless of any `is_leader` value (rolling-upgrade / legacy guard). - Command: `bazel test //src/ray/gcs_rpc_client/tests:accessor_test` β†’ **PASS** **Integration (Python)** β€” test_gcs_utils.py - `test_is_gcs_leader_defaults` β€” default path: both methods return `True`; and in a spawned subprocess with `RAY_LEADER_ELECT=true`, `is_gcs_leader_local()` starts as `False` (client-side cache init). The cross-layer server assertion is intentionally deferred to a later PR. - Command: `python -m pytest python/ray/tests/test_gcs_utils.py::test_is_gcs_leader_defaults` β†’ **PASS**", "url": "https://github.com/ray-project/ray/pull/65132", "createdAt": "2026-07-30T17:19:02Z", "updatedAt": "2026-08-12T16:47:45Z", "timestamp": "2026-08-12T16:47:45Z", "metrics": { "reactions": 1, "comments": 2 }, "labels": [ "core", "go", "community-contribution" ], "author": "YoyinZyc", "state": "closed", "assignees": [ "rueian" ] }, { "id": "github:ray-project/ray:pull_request:65173", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[dashboard] Stream job log responses", "text": "## Description `GET /api/jobs/{job_or_submission_id}/logs` currently materializes the full driver log at each dashboard hop. Large or concurrent requests can exhaust dashboard memory. This change: - reads driver logs in 64 KiB chunks - JSON-escapes chunks incrementally - streams the response through JobAgent, JobHead, and the parent dashboard proxy with backpressure - keeps stream duration unlimited while bounding connection setup and pool waits - preserves the existing `{\"logs\": string}` response and `JobSubmissionClient.get_job_logs()` behavior The SDK still returns one complete string, so client memory remains proportional to returned log size. ## Related issues Fixes #28336. No open PR implements request-time streaming. #61537 was closed and truncated large responses. #65006 rotates logs on disk, which is complementary. ## Additional information Testing: - `.venv/bin/pytest -q python/ray/dashboard/modules/job/tests/test_job_log_storage_client.py python/ray/dashboard/tests/test_utils.py`: 4 passed - `PATH=$PWD/.venv/bin:$PATH .venv/bin/pytest -q python/ray/dashboard/modules/job/tests/test_http_job_server.py::test_submit_job_with_exception_in_driver python/ray/dashboard/modules/job/tests/test_http_job_server.py::test_missing_resources`: 2 passed - `pre-commit run --files `: passed - `npm run build` in `python/ray/dashboard/client`: passed - Local Ray head and Jobs CLI/API smoke with an 8.39 MiB log: complete output, valid JSON, chunked transfer, healthy dashboard - Three concurrent throttled full-log requests: complete valid JSON and bounded server RSS growth AI assistance was used. I reviewed every changed line and ran the tests above locally.", "url": "https://github.com/ray-project/ray/pull/65173", "createdAt": "2026-08-03T16:29:57Z", "updatedAt": "2026-08-13T01:48:31Z", "timestamp": "2026-08-13T01:48:31Z", "metrics": { "reactions": 1, "comments": 2 }, "labels": [ "dashboard", "core", "go", "community-contribution" ], "author": "bvolpato", "state": "open", "assignees": [ "rueian" ] }, { "id": "github:ray-project/ray:pull_request:65202", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[Data] Throttle OutputBackpressureGuard releases with a per-op interval", "text": "## Description **Summary:** Add an opt-in, per-operator minimum interval between `OutputBackpressureGuard` releases so the liveness escape hatch can't defeat backpressure on large-block workloads. Disabled by default; no behavior change unless configured. `OutputBackpressureGuard` is a liveness escape hatch: when backpressure policies clamp an operator's output budget to 0 bytes, the guard flips it to 1 byte so the executor emits one more block and the pipeline can't deadlock. This check runs on every scheduling iteration, so whenever the guard keeps voting to release, an operator effectively emits a block per iteration. On workloads with very large blocks (e.g. rows carrying raw audio buffers), that per-iteration release rate can outpace how fast downstream drains. At that point the escape hatch effectively defeats the backpressure it's meant to be a narrow exception to: the policies say \"0 bytes\", but the operator keeps emitting a large block every iteration anyway. Object store usage then grows unbounded even though the pipeline is nominally under backpressure, pushing the store toward spilling / OOM. This PR adds an opt-in, per-operator **minimum interval between successive guard releases**. With a positive interval, the guard releases an operator at most once per interval, giving downstream time to drain between unblocks. The first release for each operator is never throttled, and idle-detection remains the liveness fallback, so pipeline liveness is preserved. The feature is disabled by default (`None`), so existing behavior is unchanged. ## Related issues ## Additional information **New configuration** - `DataContext.output_backpressure_guard_release_interval_s: Optional[float]` - Env var: `RAY_DATA_OUTPUT_BACKPRESSURE_GUARD_RELEASE_INTERVAL_S` - Default: `None` (no throttling β€” legacy behavior). Non-positive values also disable it. **Usage** ```python import ray ctx = ray.data.DataContext.get_current() # Release a fully-throttled operator at most once every 300s. ctx.output_backpressure_guard_release_interval_s = 300 ```", "url": "https://github.com/ray-project/ray/pull/65202", "createdAt": "2026-08-04T14:18:54Z", "updatedAt": "2026-08-12T21:14:02Z", "timestamp": "2026-08-12T21:14:02Z", "metrics": { "reactions": 1, "comments": 3 }, "labels": [ "data", "go", "community-contribution" ], "author": "dragongu", "state": "open", "assignees": [] }, { "id": "github:ray-project/ray:pull_request:65215", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[serve] remove deprecated Serve APIs", "text": "## Changes - Remove deploy_mode from ServeInstanceDetails, plus the single member ServeDeployMode enum and doc example - Remove the RAY_AGENT_ADDRESS deprecation warning and its three CLI call sites - Remove use_new_handle_api from DeploymentHandle.options - Raise ValueError instead of warning on '#' in deployment names, since '#' is the replica ID delimiter - Remove _deprecation_warning, marked for removal in 3.0.0", "url": "https://github.com/ray-project/ray/pull/65215", "createdAt": "2026-08-04T21:44:37Z", "updatedAt": "2026-08-12T19:34:19Z", "timestamp": "2026-08-12T19:34:19Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "serve", "go" ], "author": "YashwanthRanjanSingaravel", "state": "closed", "assignees": [] }, { "id": "github:ray-project/ray:pull_request:65220", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[Data] [Core] Deferring gRPC Frees and OSS callbacks to Reduce OSM Usage Update Latency", "text": "## Description Move `free_object_on_nodes_async_` (gRPC free broadcast) and out-of-scope callbacks outside of `mutex_` in `ReferenceCounter::OnObjectOutOfScopeOrFreed`. Previously, when many `ObjectRefs` are freed simultaneously (e.g., LIMIT operators, UDF failures), the gRPC calls (~70% of `RemoveLocalReference` runtime) and callbacks (~7%) executed under `mutex_`, forcing 10,000 concurrent frees to serialize into ~1s of lock contention. This change: 1. Collects gRPC and callback work under the lock into a deferred work list; 2. Swaps it out before releasing the lock; 3. Fires callbacks first (so Data-side `BlockRefCounter` updates ASAP); 4. Posts gRPC free calls to a dedicated `object_free_rpc_service_` thread, keeping them off both `io_service_` (gRPC handlers, GCS) and `object_freed_callback_service_` (lightweight BRC callbacks). ## How the speedup works `FreeObjectOnNodesAsync` sends gRPC RPCs to remote nodes to free their copies of an object. Previously it ran under `mutex_` inside `OnObjectOutOfScopeOrFreed`, so each `RemoveLocalReference` held the lock for ~80us (18us ref counting + 56us gRPC). When Python frees 10k refs, each ref's `__dealloc__` calls `RemoveLocalReference` sequentially, and the next ref can't start until the current ref's gRPC finishes. With this change, the lock is released after ref counting (~18us) and the gRPC is posted to a dedicated thread, so the main thread immediately moves to the next ref while gRPC calls run in parallel. ## Benchmark results Compared against baseline (master, averaged across 4 builds). Each latency metric measures the time from ref drop to individual callback firing. p50, p95, p99, and max are all recorded by the benchmark, but p95 is selected here for horizontal comparison. | Metric | Scale | Baseline | This PR | Speedup | |--------|-------|----------|---------|---------| | Burst drain p95 | 100 | 7.80ms | 3.60ms | 2.2x | | Burst drain p95 | 1k | 76.98ms | 18.90ms | 4.1x | | Burst drain p95 | 5k | 574.63ms | 143.70ms | 4.0x | | Burst drain p95 | 10k | 986.28ms | 266.60ms | 3.7x | | BRC burst p95 | 100 | 7.83ms | 3.60ms | 2.2x | | BRC burst p95 | 1k | 77.45ms | 39.70ms | 2.0x | | BRC burst p95 | 5k | 470.73ms | 137.70ms | 3.4x | | BRC burst p95 | 10k | 977.90ms | 226.10ms | 4.3x | Pipeline p95 (steady-state, one ref at a time) is unchanged, as expected. Per-callback p95 (individual `del ref` to callback latency) regressed at large scales (0.7ms to 3.1ms at 10k) because callback posting now happens after mutex release + deferred work swap instead of immediately under mutex. This is a tradeoff: individual callback latency is slightly higher, but total burst drain time is 3-4x faster. ## Related issues Related to ray-project/ray#64521 (callback throughput benchmark). Related to ray-project/ray#64456 (BlockRefCounter-based memory tracking).", "url": "https://github.com/ray-project/ray/pull/65220", "createdAt": "2026-08-05T00:23:53Z", "updatedAt": "2026-08-13T00:53:18Z", "timestamp": "2026-08-13T00:53:18Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "core", "data", "go" ], "author": "rayhhome", "state": "open", "assignees": [ "Yicheng-Lu-llll", "rayhhome" ] }, { "id": "github:ray-project/ray:pull_request:65228", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[Data] Deprecate `ray_remote_args` for Dataset transformations", "text": "### What changed This PR adds named worker options to: - `map` - `map_batches` - `flat_map` - `with_columns` - `with_column` - `filter` - `add_column` - `drop_columns` - `select_columns` - `rename_columns` The new parameters are: - `label_selector` - `fallback_strategy` - `max_calls` - `resources` - `accelerator_type` - `runtime_env` - `max_concurrency` Options passed through `**ray_remote_args` continue to work, but now emit a `RayDeprecationWarning`. `ray_remote_args` will be removed in Ray 2.64. `placement_group_bundles` and `placement_group_strategy` need additional execution support, so they will be added separately after #64090. Reads, writes, `random_shuffle`, and `join` are out of scope. --- Closes https://github.com/ray-project/ray/issues/65227", "url": "https://github.com/ray-project/ray/pull/65228", "createdAt": "2026-08-05T11:48:52Z", "updatedAt": "2026-08-13T12:02:15Z", "timestamp": "2026-08-13T12:02:15Z", "metrics": { "reactions": 1, "comments": 1 }, "labels": [ "data", "deprecation", "go", "community-contribution" ], "author": "yuhuan130", "state": "open", "assignees": [] }, { "id": "github:ray-project/ray:pull_request:65230", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[core] Add RAY_DISABLE_WORKER_LOG_PREFIX to control log prefix behavior", "text": "## Description As mentioned in the issue 64992. `log_to_driver=True` (the default) prepends a `(name pid=...)` prefix to every worker/actor stdout/stderr line forwarded to the driver. There was previously no way to disable *only* that prefix: `RAY_COLOR_PREFIX=0` only strips ANSI color, `log_to_driver=False` drops forwarding entirely (logs never reach the driver's stdout), and `logging_config=LoggingConfig(...)` strips the prefix but also seizes the root logger for the driver/actors/tasks, which isn't viable for applications with their own logging configuration. The `ignore_prefix` existed , but was only dependent on job_logging_config, this fix adds another env-var to decide the switch. ## Related issues Closes #64992. ## Additional information Testing performed: - Full `python/ray/tests/test_logging_2.py` suite: 12/12 passed, including the new `test_disable_worker_log_prefix_env_var` and the pre-existing prefix-related tests. - Manually verified default behavior is unchanged when the env var is unset . > === WITHOUT the flag (default behavior) === // Some warnings 2026-08-05 14:23:04,148 INFO worker.py:2024 -- Started a local Ray instance. Usage stats collection is enabled by default for nightly wheels. To disable this, run the following command: `ray disable-usage-stats` before starting Ray. See https://docs.ray.io/en/master/cluster/usage-stats.html for more details. (SegmentationActor pid=558) {\"event\": \"segmenting batch\", \"job_id\": \"abc123\", \"level\": \"info\"} > > === WITH RAY_DISABLE_WORKER_LOG_PREFIX=1 === > 2026-08-05 14:24:09,569 INFO worker.py:2024 -- Started a local Ray instance. > Usage stats collection is enabled by default for nightly wheels. To disable this, run the following command: `ray disable-usage-stats` before starting Ray. See https://docs.ray.io/en/master/cluster/usage-stats.html for more details. > {\"event\": \"segmenting batch\", \"job_id\": \"abc123\", \"level\": \"info\"} **AI assistance disclosure** : This change was tested with the help of AI (Claude). I have reviewed the tests and did manually as well to verify the same.", "url": "https://github.com/ray-project/ray/pull/65230", "createdAt": "2026-08-05T14:39:11Z", "updatedAt": "2026-08-13T13:18:05Z", "timestamp": "2026-08-13T13:18:05Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "core", "community-contribution" ], "author": "imtherealnaska", "state": "open", "assignees": [] }, { "id": "github:ray-project/ray:pull_request:65239", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[docs] Add Kubernetes and KubeRay conventions to the style guide", "text": "## Why this change Ray on Kubernetes docs describe Kubernetes API objects alongside Ray concepts, and the two vocabularies collide. Deployment, Job, and Service each name both a Kubernetes API object and a distinct Ray concept, so capitalization is the only thing telling them apart, and the style guide didn't say which form to use where. A contributor asking \"RayCluster or Ray cluster?\" had to derive the answer from two guides and adjudicate between them. This adds a `Writing Ray on Kubernetes and KubeRay docs` section to the style guide, scoped to the user-facing pages under `doc/source/cluster/kubernetes/` and the contributor-facing pages in [ray-project/kuberay](https://github.com/ray-project/kuberay). The approach is to extend this guide with a Kubernetes deference section rather than stand up a separate KubeRay style guide. The two guides already overlap heavily, and a forked guide would drift within a release or two. ## What the casing scan found The term list isn't invented. It comes from a prose-only scan of three corpora: 79 user-facing pages in this repo, 12 contributor pages in `ray-project/kuberay`, and 128 pages of `kubernetes/website` (`content/en/docs/concepts/`, commit `bb38117c0`). The scan strips front matter, fenced code blocks, inline code, MyST roles, and link targets before counting, so YAML field names and CLI output don't inflate the figures. Headline: **these docs already follow the Kubernetes casing rule.** Upstream Kubernetes capitalizes \"Pod\" in 71% of prose uses; the Ray-side KubeRay docs do so in 70%. So this PR documents a rule the docs mostly already follow, and does not propose a capitalization sweep. | Term | Ray-side KubeRay docs | Upstream Kubernetes | Read | |---|---|---|---| | Pod | 371/527 (70%) | 3712/5195 (71%) | Matches upstream almost exactly. | | ConfigMap | 20/21 (95%) | 119/130 (91%) | Healthy. | | Ingress | 8/15 (53%) | 148/242 (61%) | Roughly aligned. | | Service | 18/114 (15%) | 515/946 (54%) | Diverges by design. | | Job | 17/143 (11%) | 522/610 (85%) | Diverges by design. | | Deployment | 6/53 (11%) | 355/461 (77%) | Diverges by design. | The three \"diverges by design\" rows are the reason this section exists. Sampling shows the lowercase uses are Ray Serve deployments, Ray jobs, and Ray services, not Kubernetes API objects. That's a naming collision, not drift, so the fix is a disambiguation rule rather than a sweep. Kubernetes's own rule expects a mix (UpperCamelCase for the API object, sentence case for the general concept), which is why the upstream column isn't 100% either. The one genuine inconsistency the scan found is **\"Ray Autoscaler\" (44 uses) against \"Ray autoscaler\" (35)**, spread across at least eight files. This guide already says to lowercase generic nouns even when they name a Ray concept, so lowercase wins. Recording the rule here is the prerequisite for a mechanical follow-up cleanup. I've left that out of this PR to keep the rule change reviewable on its own. ## Open questions for reviewers I've deliberately written only the rules I can support from the two published guides plus the scan. These five are genuinely open, and I'd rather resolve them in review than assert them: 1. ~~**Confirm the custom-resource split.**~~ **Resolved** by @win5923: the split is intentional and consistent with Kubernetes terminology. Original question: The section documents `RayCluster` (the resource) versus \"Ray cluster\" (the running thing it produces), currently 335 against 106 uses in this repo. I read it as intentional and matching the Kubernetes API-object rule. Worth confirming rather than normalizing to one form. 1. ~~**\"head Pod\" versus \"head node.\"**~~ **Resolved** by @win5923: use \"head Pod\" consistently when a Ray cluster runs on Kubernetes. Original question: The section prefers \"head Pod\" for the Kubernetes-hosted case, since a node is a machine on Kubernetes. `doc/source/cluster/kubernetes/index.md` currently says \"a head node pod and a collection of worker node pods,\" which stacks both vocabularies. Reasonable, or is \"head node\" worth keeping for continuity with Ray's core architecture docs? 1. **Feature graduation phases.** Kubernetes uses start case (\"DRA is Beta\"); Ray docs lowercase feature stages. This bites on pages discussing a Kubernetes feature's stage and a Ray feature's stage in one paragraph. I left it out of the section rather than pick. Which wins? 1. **Field values.** Kubernetes wants unquoted, unstyled values (\"set `imagePullPolicy` to Always\"); this guide says to use backticks for configuration values and gives `num_replicas: 1`. These are closer than they look, since one is a bare value and the other a key-value pair, but the guidance should say which form applies where. 1. **\"Ray dashboard\" versus \"Ray Dashboard\"** (43 title-case against 18 lowercase). This guide implies lowercase, but the dashboard is arguably a named UI surface. Not addressed in the section pending a call. Two further items are out of scope here but worth flagging: - **Line wrapping in the `kuberay` repo.** This guide requires soft wrap. The Kubernetes guide requires manual wrap, justified by localization teams diffing per line, which KubeRay doesn't have. Nothing in this PR changes that, since this guide already governs `doc/source/`. But `kuberay`'s `.markdownlint.yaml` sets `MD013` to 120 characters via pre-commit, which is a latent gate: 266 lines in its `docs/` already exceed it, it isn't enforced in CI, and it isn't auto-fixable, so contributors hit it unpredictably on whichever file they touch. Disabling it there would align the two repos, but that's a call for KubeRay maintainers in their own repo. - **Turning Vale enforcement on.** This PR now adds the ruleset but doesn't enforce it. See the section below. ## Staged Vale ruleset A later commit adds a Vale ruleset for these pages, following the pattern already used for the Ray Data docs. **Nothing is enforced yet.** The Vale hook in `.pre-commit-config.yaml` stays scoped to `^doc/source/data/.*`, so CI checks exactly what it checked before. - `.vale/styles/config/vocabularies/KubeRay/accept.txt` β€” the KubeRay custom resources, the Kubernetes and ecosystem API objects these pages reference, and project names. Skips what the `General` vocabulary already accepts (`Kubernetes`, `kubectl`, `CRD(s)?`, `namespace`). - `.vale/styles/KubeRay/Terms.yml` β€” a `substitution` rule at `warning` level covering only the casing forms the new style-guide section settles, with `vocab: false` (see below). The open questions above are deliberately absent. - `.vale.ini` β€” registers the vocabulary and scopes the style to `doc/source/cluster/kubernetes/`. Registering a vocabulary that the hook doesn't exercise matches existing practice here: `RLlib` and `Train` are already in `Vocab` while the hook only runs on `doc/source/data/`. Contributors can run it by hand today: ``` vale doc/source/cluster/kubernetes/ ``` To enforce later, widen the `files` pattern on the `vale` hook. I'd still hold that until the open questions land. ### What I verified Vale 3.12.0 locally. **Path scoping works.** The KubeRay rules fire on a file under `doc/source/cluster/kubernetes/` and stay silent on identical content at the repo root. **Current state of these pages** β€” 146 `KubeRay.Terms` alerts, which lines up with the casing scan: | Alert | Count | |---|---| | Use 'Ray autoscaler' instead of 'Ray Autoscaler' | 42 | | Use 'head Pod' instead of 'head pod' | 20 | | Use 'worker Pod' instead of 'worker pod' | 7 | | Use 'Ray job' instead of 'ray job' | 2 | | Use 'Ray Serve' instead of 'Ray serve' | 1 | | Use 'Ray cluster' instead of 'ray cluster' | 1 | | Use 'KubeRay' instead of 'Kuberay' | 1 | | Use 'GCS fault tolerance' instead of 'GCS FT' | 1 | The 42 matches the scan's count for these pages exactly, which is a useful cross-check on both. None of it is enforced, so nothing here has to be fixed in this PR. **No regression on the gated path.** Alert count on `doc/source/data/` goes 174 β†’ 173, deterministic across repeated runs in both states, with no new alerts. A `Vocab` list is global, so it can only widen what's accepted. The single alert that disappears is a `Google.Parens` hit in `concurrent-dataset-execution.md`; no term in the new vocabulary appears on that line, so I'm reporting that as observed rather than claiming I understand the mechanism. It's a reduction either way, so it can't break the gate. ### The vocabulary shadowing trap Worth knowing if you write Vale rules here: a `Vocab` accept entry shadows its token for **every** rule, case-insensitively. A substitution whose target appears in an accept list silently never fires, and nothing warns you. This bit the `Ray Autoscaler` swap, because `autoscaler` is accepted in `config/vocabularies/Data/accept.txt`. The fix is `vocab: false` on the rule, which opts out of the filtering without touching the Data vocabulary. The rule carries a comment saying so. Credit to Bugbot for catching it. I'd first shipped that swap commented out, having concluded the only way to enable it was dropping `autoscaler` from the Data vocabulary and accepting new spelling alerts on the gated path. That was wrong, and `vocab: false` avoids the tradeoff entirely. ## Added after review Two rules that review surfaced, both in the style-guide section plus the Vale ruleset. **Node stays lowercase.** Node is a Kubernetes API object, so \"capitalize Kubernetes API objects\" read literally tells contributors to write \"Kubernetes Node.\" These pages almost always mean the machine, and all 24 uses are correctly lowercase today, so a literal reading of the rule would have someone \"fix\" working prose into wrongness. Kubernetes applies the same two-part rule it applies elsewhere, capitalizing Node only for the resource, and the section now says so. **Don't write \"Ray Pod.\"** This answers the casing question raised in review, and the answer turned out to be neither casing. A Pod is a Kubernetes host that contains a Ray node plus any sidecars, so naming it after Ray overstates what it is. \"TPU Pod\" also already occupies the qualified-Pod form in these pages and means a group of TPU chips rather than a Kubernetes object. The term had no settled form to preserve either: across the 55 prose lines using it, casing split four ways at 20, 16, 16, and 15. Stating that alone would be a trap, because a bare \"Pod\" isn't always an adequate replacement, so the section gives the test that separates the cases: - Qualify when the qualifier distinguishes two referents. That's why \"Ray node\" against \"Kubernetes node\" is correct and should stay. - Don't qualify when it relabels one thing. - Keep the scope explicit where a page also describes Pods the RayCluster doesn't own, such as the operator, Redis, or a `curl` Pod, and where a nearby command filters by label. \"List all Pods\" and \"List the Ray cluster's Pods\" describe different commands and shouldn't read the same way. The cleanup is #65423, which removes all 89 uses from the docs and is reviewable on its own. This PR is the rule; that one is the sweep. Vale picks up four swaps for the retired forms. Plurals get their own entries, for the word-boundary reason @win5923 identified on the `head pod` swaps. Verified that they don't fire on \"Ray head Pod\" or \"Ray worker Pods\", and that the gated `doc/source/data/` path holds at 173 alerts. ## Not a duplicate Checked open PRs for this area before starting: ``` gh pr list --repo ray-project/ray --state open --search \"writing-style\" gh pr list --repo ray-project/ray --state open --search \"style guide in:title\" gh pr list --repo ray-project/ray --state open --search \"kuberay style\" ``` No open PR touches `doc/source/ray-contribute/writing-style.md` or proposes KubeRay style guidance. The `writing-style` matches were all false positives on `write_delta`. This extends the guide added in #64518. ## Testing ``` pre-commit run --files doc/source/ray-contribute/writing-style.md ``` Passes, but as a no-op on the Markdown: every hook reports \"no files to check,\" because no pre-commit hook applies to Markdown under `doc/source/` and Vale is scoped to `doc/source/data/`. Stating that plainly instead of claiming coverage it doesn't give. The Vale config files do have hook coverage, and those checks ran for real: ``` pre-commit run --files .vale.ini .vale/styles/KubeRay/Terms.yml .vale/styles/config/vocabularies/KubeRay/accept.txt ``` `trim trailing whitespace`, `fix end of files`, `check for added large files`, and `semgrep` all pass. Vale verification for the ruleset is in the section above: path scoping, the 31-alert current state, and the 174 β†’ 173 no-regression check on the gated path. Verified by hand: the new `(kubernetes-docs-style)=` anchor doesn't collide with any existing target in `doc/source/`, and the section uses the same MyST anchor form and Markdown pipe-table style already used elsewhere in this file. Rendering is verified by the Read the Docs preview build on this PR. No code, no API, no behavior change. The prose is a contributor guide; the Vale files add configuration without widening what CI enforces. ## AI assistance Written with AI assistance (Claude). I've reviewed every changed line, generated and inspected the casing counts myself, and can defend the change end to end. The scan scripts that produced the numbers live outside this repo; happy to share the methodology or re-run against a different corpus if a reviewer wants to check a figure. --- ### Update: the vocabulary now covers the ecosystem terms these pages use (555ebdad) Running `vale doc/source/cluster/kubernetes/`, which this PR's `.vale.ini` comment tells contributors to do, produced 215 `Vale.Spelling` alerts on correctly spelled terms. The accept list covered API objects, the custom resources, and project names, but not the wider ecosystem vocabulary the pages actually contain. Left as is, a contributor following the instruction learns to ignore Vale on exactly the paths this PR wants gated. Added: the ecosystem projects (Istio, Kiali, gVisor, Kata, Krew, Kustomize, Bottlerocket, Aliyun, GCSFuse, Promtail, Fluentd, Filebeat, Alertmanager), the Gateway API `HTTPRoute`, and the Kubernetes and infrastructure terms in use (cgroup, finalizer, kubeconfig, kubelet, namespaced, namespaces, toleration, upscaling, downscaling, vCPU, FQDN, AMI, subnet, syscall, hostname, cron, CA, autoscalers). **Excluded on purpose, since an accept list shouldn't launder a defect.** `CR` and `CRs`, 12 uses, because this PR's own guidance says to write \"custom resource\" and to define CustomResourceDefinition then use CRD. Real misspellings: `heterogenous`, `replicaset` for ReplicaSet, `Deepseek` for DeepSeek, `nodepool` for node pool. `gsutil`, which `gcloud storage` supersedes. Unbackticked code identifiers, which are a formatting defect rather than a vocabulary gap: `podSpec`, `podSelector`, `nodeSelector`, `jsonPointers`, `intervalSeconds`, `workerGroup`, `ignoreDifferences`, `HTTPMode`, `args`, `yaml`. And generic English that Vale's dictionary lacks, which belongs in the General vocabulary rather than a KubeRay-specific one. Three entries are case-tolerant (`[Tt]oleration(s)?`, `[Uu]pscaling`, `[Dd]ownscaling`). A lowercase-only entry makes `Vale.Terms` flag the legitimate sentence-start capitalization. **Verification**, deterministic across repeated runs: - `doc/source/cluster/kubernetes/`: 1329 alerts to 1178. `Vale.Spelling` drops from 215 to 72. `KubeRay.Terms` is unchanged at 160, confirming no swap got shadowed by a new accept entry, which is the trap the `Ray Autoscaler` comment in `Terms.yml` documents. - `doc/source/data/`, the only path the pre-commit hook gates: 173 to 170, with **no new alerts**. A `Vocab` is global, so it can only widen what's accepted. - One candidate entry was dropped for exactly that reason. A `runtime(s)?` entry made `Vale.Terms` flag \"Runtime\" in `doc/source/data/monitoring-your-workload.rst`, a new alert on the gated path. Four new `Vale.Terms` alerts do appear on the KubeRay paths, and all four are genuine casing defects in existing pages rather than noise: \"Kubelet\" twice in `configuring-ippr.md`, lowercase \"istio\" naming the project in `rayservice-incremental-upgrade.md`, and \"Cgroups\" in the `resource-isolation-with-writable-cgroups.md` H1, which is also title case. Those are content fixes for a separate change, not this one. AI assistance was used for this commit, including the Vale sweep and the before-and-after verification above.", "url": "https://github.com/ray-project/ray/pull/65239", "createdAt": "2026-08-05T19:50:41Z", "updatedAt": "2026-08-13T17:43:42Z", "timestamp": "2026-08-13T17:43:42Z", "metrics": { "reactions": 1, "comments": 3 }, "labels": [ "docs", "core", "go" ], "author": "dstrodtman", "state": "open", "assignees": [] }, { "id": "github:ray-project/ray:pull_request:65245", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[core] Add local:// runtime_env URIs for code already in node", "text": "## Description - Adds a `local://` runtime_env URI: `working_dir: local:///app` - Used in place with no uploading or unpacking ## Changes - `Protocol.LOCAL`: `parse_uri` returns the path itself, rejects non absolute paths. - `working_dir`/`py_modules`: no upload or download, 0 bytes cached, never deleted. - new section in the docs.", "url": "https://github.com/ray-project/ray/pull/65245", "createdAt": "2026-08-06T00:50:03Z", "updatedAt": "2026-08-12T19:35:14Z", "timestamp": "2026-08-12T19:35:14Z", "metrics": { "reactions": 1, "comments": 1 }, "labels": [ "docs", "core", "go" ], "author": "YashwanthRanjanSingaravel", "state": "open", "assignees": [] }, { "id": "github:ray-project/ray:pull_request:65263", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[docs] Add a Kubernetes and Ray scheduling orientation guide", "text": "## Description Running Ray on Kubernetes involves two schedulers. Kubernetes places pods on machines, and Ray places tasks and actors on pods. The docs cover each layer well on its own, but nothing explains how they relate or which one to investigate when a workload doesn't start. The closest existing page is `k8s-autoscaler.md`, but it's scoped to autoscaling and marked advanced, so a user debugging a pending actor is unlikely to find it. This PR adds a user guide that: - Names the two layers and what each one places. - Documents how KubeRay derives Ray's logical resource capacity from the container spec, which is where the two layers most often disagree. - Maps pod state to the layer worth investigating. - Defers all mechanism to the existing Ray Core scheduling, RayCluster config, and autoscaler pages rather than restating them. It also adds a pointer to the new page from the autoscaler guide. The KubeRay resource-derivation details are drawn from `user-guides/config.md` (container limits, the CPU-request fallback as of KubeRay 1.3.0, integer rounding, and ignored memory and GPU requests). ## Related issues None. This came out of a documentation gap audit, not a filed issue. ## Additional information ### Not a duplicate Searched open PRs against `ray-project/ray` for \"kubernetes scheduling\", \"scheduling overview\", and related area keywords. No open PR covers this. No existing page in `doc/source` joins the two scheduling layers. ### Testing **Source verification against the KubeRay implementation.** Every claim about how KubeRay derives Ray's logical capacity was checked against `ray-operator/controllers/ray/common/pod.go` in `ray-project/kuberay`, not just against the existing docs: - `generateRayStartCommand` reads `resource.Limits[ResourceCPU]` for `num-cpus`, and falls back to `resource.Requests[ResourceCPU]` when the limit is zero. - The CPU-request fallback landed in kuberay commit `ba50bfa8` (#2365). It's absent from `v1.2.1` and present in `v1.3.0`, which confirms the \"starting with KubeRay 1.3.0\" attribution. - Integer rounding follows from `Quantity.Value()`, which rounds up away from zero, so a `500m` limit becomes one logical CPU. - Memory reads `Limits[ResourceMemory]` only, with no request fallback, and accelerators go through `addWellKnownAcceleratorResources(rayStartParams, resource.Limits)`. Both confirm that requests are ignored for memory and GPU. **Other checks:** - `vale doc/source/cluster/kubernetes/user-guides/scheduling.md` β€” remaining errors are `Vale.Spelling` on \"tolerations\" and \"autoscalers\" (standard terms used throughout the existing KubeRay docs, absent from the Vale vocabulary) and one `Google.OxfordComma` false positive on a two-item disjunction. The existing `k8s-autoscaler.md` reports a comparable baseline. - `pre-commit run` on the staged files β€” every hook reports \"no files to check\". No pre-commit hook currently covers Markdown under `doc/source/`. - Every `{ref}` target used on the page was verified to exist by grepping for its label definition. - **A full Sphinx docs build was not run locally.** Link resolution and toctree wiring rely on CI. Flagging this explicitly rather than implying broader verification than was performed. ### Self-review findings already fixed A critical pass before requesting review caught that the page claimed Ray integrates with four batch schedulers. There are five: the `kubernetes-sigs/scheduler-plugins` integration provides gang scheduling for RayCluster through the PodGroup API as of KubeRay v1.4.0. Corrected, with the fifth link added. ### AI assistance AI assistance was used to draft this page. The content is grounded in the repository sources cited above rather than generated from prior knowledge. It still needs a final human review pass before it should be considered ready, which is why this is opened as a draft. --- ### Update: review feedback addressed (983186c) No longer a draft, so the \"final human review pass\" caveat above is satisfied. @andrewsykim asked for the Ray node to Kubernetes Pod mapping stated in the intro, since it isn't obvious to readers new to Ray or KubeRay. It was present but buried as a subordinate clause. It now opens the page in its own paragraph, with the head and worker corollary and a note on why the page switches between \"Pod\" and \"Ray node\". The resource table also said a Ray node \"is\" a Pod; it runs as one, and the Pod is a superset that can hold the autoscaler sidecar, a log shipper, or a RayJob submitter. Both bot comments applied. One added a technical claim, verified the same way as the rest: - **The limits come from the main Ray container, which must be first in the `containers` list.** `utils.RayContainerIndex` is `0` in `ray-operator/controllers/ray/utils/constant.go`, and `BuildPod` passes `pod.Spec.Containers[utils.RayContainerIndex].Resources` into `generateRayStartCommand`. So limits set on a sidecar don't contribute to the Pod's logical capacity, which is the reason the detail is worth stating. - Added a `` placeholder to the `kubectl describe pod` command. `pre-commit run` again reports \"no files to check\" for every hook, consistent with the note above that nothing covers Markdown under `doc/source/`. Still no local Sphinx build.", "url": "https://github.com/ray-project/ray/pull/65263", "createdAt": "2026-08-06T16:54:19Z", "updatedAt": "2026-08-12T19:27:42Z", "timestamp": "2026-08-12T19:27:42Z", "metrics": { "reactions": 1, "comments": 1 }, "labels": [ "docs", "core", "go", "docs-go" ], "author": "dstrodtman", "state": "closed", "assignees": [] }, { "id": "github:ray-project/ray:pull_request:65264", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[docs] Add a scheduling overview with defaults to the Ray Core scheduling page", "text": "## Description `doc/source/ray-core/scheduling/index.rst` carried a standing note that the page lacked an overview of the features and configs that affect scheduling, along with their default values. Most of those defaults are already documented, but they're spread across the resources, labels, and placement group pages, so a reader has no single place to see what Ray does before any configuration. This PR adds a \"Scheduling at a glance\" section that: - Describes the two-step placement model: narrow to feasible nodes, then pick one. - Tabulates each control with its default, linking to the page that documents it rather than restating the mechanism. - Tabulates the three environment variables that tune the `\"DEFAULT\"` strategy. - Calls out that the non-zero actor scheduling default means an actor needs a node with a free CPU to start, which is a recurring source of confusion. It removes the standing note, since this addresses it. Happy to restore it if maintainers consider the gap only partly closed. One genuine addition: `RAY_scheduler_top_k_absolute` was named on the page without its value. It defaults to `1` per `src/ray/common/ray_config_def.h`. ## Related issues None. The work is driven by the in-file note. ## Additional information ### Not a duplicate Searched open PRs against `ray-project/ray` for \"scheduling overview\" and related area keywords. Nothing open touches this page. ### Testing **Every default in the new tables was verified against source, not against other docs pages:** - The three scheduler environment variables come from `src/ray/common/ray_config_def.h`: `scheduler_spread_threshold` 0.5, `scheduler_top_k_fraction` 0.2, `scheduler_top_k_absolute` 1. - The actor defaults were checked in `python/ray/actor.py`. A bare actor takes the simple branch, giving it 0 lifetime CPU and an `actor_method_cpu` of 1. Because method CPU is 1, `actor_placement_resources` becomes the lifetime resources with CPU incremented by 1. That's what makes \"1 CPU for scheduling, 0 for running\" true, and it's why the constants alone read as if they say the opposite. - Node auto-detection and the task default come from `scheduling/resources.rst`; the default labels come from `scheduling/labels.md`. - Every `:ref:` and `:doc:` target in the new section was verified to exist by grepping for its label definition. **Other checks:** - `vale doc/source/ray-core/scheduling/index.rst` β€” the added content introduces no new class of Vale error. The message-set diff against the pre-change file is empty. - `pre-commit run` on the staged file β€” every hook reports \"no files to check\". - **A full Sphinx docs build was not run locally.** Cross-reference resolution relies on CI. ### Self-review findings already fixed A critical pass before requesting review caught three problems in the first draft: - Data locality was described as breaking ties for tasks. The page's own `\"DEFAULT\"` section says locality takes precedence over utilization scoring, so Ray prefers the locality node regardless of how utilized it is. \"Tie-breaking\" understated it. - The gang placement row referred to bundles being scheduled independently outside a placement group, but bundles only exist inside one, so it described something that can't happen. - A sentence said \"both defaults above are non-zero\" when the two actor defaults are 1 and 0, contradicting the row directly above it. ### AI assistance AI assistance was used to draft this section. Every default stated was read out of the source tree rather than recalled. It still needs a final human review pass, which is why this is opened as a draft.", "url": "https://github.com/ray-project/ray/pull/65264", "createdAt": "2026-08-06T16:54:31Z", "updatedAt": "2026-08-12T19:26:51Z", "timestamp": "2026-08-12T19:26:51Z", "metrics": { "reactions": 1, "comments": 1 }, "labels": [ "docs", "core" ], "author": "dstrodtman", "state": "open", "assignees": [] }, { "id": "github:ray-project/ray:pull_request:65267", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[serve] Env-gated observe layer4 mark-down on HAProxy servers", "text": "## Summary When a node is drained (spot reclaim, compaction, scale-down), its displaced (soft-stopped) HAProxy processes keep serving established connections with a frozen config in which torn-down replicas remain nominally UP. Requests those processes accept are black-holed until `hard-stop-after` instead of failing over. This PR adds an env-gated `observe layer4 error-limit on-error mark-down` to backend replica servers so live traffic itself marks dead servers DOWN and the existing `backup` fallback takes over. Default off; the rendered config is byte-identical when unset. ## Chain of events 1. A node starts draining (2026-08-04 incident: spot reclaim; 2026-07-23: replica compaction). 2. The controller migrates the node's replicas; replacements are healthy elsewhere in ~2s. 3. Updated backend configs are pushed to every proxy. 4. Each HAProxy applies the new config by reloading: a new process starts, and the old one is soft-stopped (`-sf`) but keeps serving its established connections. 5. The old process's view is frozen: the stopped replicas stay UP in its map, and the replacements don't exist in it. 6. A request arrives on a connection held by the old process; the frontend accepts it. 7. Dispatch picks a stale server; the connect fails β€” the replica is gone. 8. Retries and redispatch only consider servers believed UP: the same dead ones. `nbsrv > 0`, so the `backup` fallback stays ineligible. 9. No error is ever returned; the client burns its full timeout (150s in our fleet). 10. Steps 6-9 repeat for every request landing on that process until `hard-stop-after` (400s) kills it: queued requests die as 502s (last client failures at drain+401s, to the second), and traffic then flows only to healthy processes. Impact: 8 failed client requests over ~6.5 minutes, exactly on the apps whose replicas all sat on the drained node (echo 1/1 replica, highscale 2/2). ## Fix `RAY_SERVE_HAPROXY_OBSERVE_MARK_DOWN_ENABLED` (default off) renders `observe layer4 error-limit on-error mark-down` (`RAY_SERVE_HAPROXY_OBSERVE_ERROR_LIMIT`, default 3) on HTTP and gRPC replica server lines. Connection-level failures observed by live traffic mark the server DOWN synchronously in the request path β€” verified in HAProxy source: `__health_adjust()` runs inline with no `stopping` guard, and soft-stop does not destroy check tasks, so this works inside displaced processes, the one place nothing else can. With all primaries DOWN, `option redispatch` + the existing `backup` fallback serve the request. Recovery is automatic: DOWN servers keep being health-probed and `rise` passes restore them in ~0.5s, so a false positive self-heals. Backup/fallback servers are never observed; router-path servers inherit state via `track`.", "url": "https://github.com/ray-project/ray/pull/65267", "createdAt": "2026-08-06T18:56:27Z", "updatedAt": "2026-08-13T12:08:26Z", "timestamp": "2026-08-13T12:08:26Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "serve", "go" ], "author": "harshit-anyscale", "state": "closed", "assignees": [ "harshit-anyscale" ] }, { "id": "github:ray-project/ray:pull_request:65282", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[autoscaler] Precompute SerializeToString keys to eliminate redundant serialization in scheduling loop", "text": "## Why are these changes needed? After merging #64175, benchmark testing showed that scaling to 3000 nodes (15000 tasks Γ— 0.2 CPU + 30MB memory) still takes ~51 minutes. Profiling with py-spy revealed two independent hotspots in the scheduling loop: 1. **Pre-filter logic flaw** β€” fixed in #65171 (OR-across-dimensions β†’ AND-within-shape/OR-across-shapes) 2. **Redundant `SerializeToString` calls** β€” fixed in this PR The `UnschedulableRequestCache.contains()` serializes each request via `SerializeToString(deterministic=True)` once per candidate node inside `try_schedule`, resulting in **N_nodes Γ— M_requests** redundant serializations per scheduling round. For 15000 requests across 3000 nodes, this produces ~45M repeated protobuf serializations. ### Profiling evidence (py-spy, 30s sample during baseline run): - `contains()` in scheduler.py: **55% Own CPU** (7.77s / 14s) - `_audit_fork_safety` triggered by SerializeToString inside `contains()`: **18% Own CPU** (2.43s / 14s) - Combined: **73% of scheduler CPU** spent on redundant serialization \"image\" ![py-spy top](https://github.com/user-attachments/assets/8b5b56bb-0f87-445c-a5c9-9346bd3ce0d9) ### Fix Precompute a `{id(request): bytes}` dictionary **once** before entering the scheduling loop, and pass it through `_sched_best_node` β†’ `try_schedule`. The `UnschedulableRequestCache` interface is simplified to accept raw `bytes` keys directly. Includes a defensive `.get()` fallback: if a request is not in the precomputed dict, it falls back to on-the-fly serialization rather than raising a `KeyError`. ## Benchmark Cluster: 3000 max workers (1 CPU each), 15000 tasks Γ— 0.2 CPU. | Condition | Time to 3000 nodes | |-----------|-------------------| | After #64175 (baseline) | ~51 min | | + #65171 only (pre-filter AND/OR) | ~18 min | | + #65171 + this fix combined | ~14 min | ## Related PRs - #64175 β€” baseline optimization that this PR builds on (benchmark results from that PR revealed this hotspot) - #65171 β€” parallel fix for pre-filter logic (independent, can merge in either order) ## Checks - [x] I've signed all my commits with DCO sign-off. - [x] I've run the existing tests β€” all pass. The `try_schedule` interface change is backward-compatible via `Optional` parameter with on-the-fly fallback. - [x] Changes are confined to `python/ray/autoscaler/v2/scheduler.py` (single file, low blast radius). - [x] No duplicate PR exists for this optimization (searched open PRs for SerializeToString, precompute, UnschedulableRequestCache β€” zero matches). - [x] AI assistance was used. I (human submitter) have reviewed every changed line and understand the change end-to-end.", "url": "https://github.com/ray-project/ray/pull/65282", "createdAt": "2026-08-07T08:25:27Z", "updatedAt": "2026-08-13T13:19:08Z", "timestamp": "2026-08-13T13:19:08Z", "metrics": { "reactions": 1, "comments": 1 }, "labels": [ "core", "community-contribution" ], "author": "Jade07-1", "state": "open", "assignees": [ "rueian" ] }, { "id": "github:ray-project/ray:pull_request:65287", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[doc] Style pass on the JAX TPU profiling user guide", "text": "## Description A documentation style and grammar pass over the JAX TPU profiling user guide added in #64735, plus the TPU profiling section that PR added to `profiling.md`. #64735 merged before a docs-team review landed (@rueian had asked for one on the thread). This follow-up applies [the Ray documentation style guide](https://docs.ray.io/en/master/ray-contribute/writing-style.html) to the new page. The first commit is style and grammar only. The second commit carries two factual corrections that came out of review on this PR, both verified against the source and described below. ### Corrections (commit 2) - **Port discovery.** The page implied that passing a custom port to `init_jax_profiler()` means you must also pass `port=` to the dashboard endpoint. `init_jax_profiler()` registers whatever port it binds to under `jax_profiler_port::` in the GCS internal KV store (`python/ray/util/tpu.py`), and the `/worker/jax_profile` handler reads that key only when the request omits `port=`, returning a 400 when the lookup is empty (`python/ray/dashboard/modules/reporter/reporter_head.py`). So `port=` bypasses the lookup rather than being required for custom ports. The rewrite also documents `JAX_PROFILER_PORT`, which the same function reads and which the page didn't mention. - **Profile plugin name.** `tensorboard-plugin-profile` publishes on PyPI as the XProf Profiler Plugin ([openxla/xprof](https://github.com/openxla/xprof)), covering TPU, GPU, and CPU. Dropped the inaccurate \"Google TPU\" qualifier. ### Style changes (commit 1) - **Added the required `myst.html_meta` `description` front matter.** The guide asks for one on every page; the new page had none. - **Tagged the untagged fence.** The expected `ls` output block had no language; it's now `text`. - **Active voice where there's a nameable actor.** \"`tensorflow` is required by the Ray Dashboard `ReporterAgent`\" becomes the agent using it. \"The JAX trace file was captured and saved\" becomes the profiler capturing and saving it. - **Cut filler and hedging.** Dropped \"Note that.\" `ensure X is set to \"1\"` becomes `set X to \"1\"`. \"Ensure you have TensorBoard installed\" becomes \"Install TensorBoard.\" - **`Ray 2.57+` spelled out as `Ray 2.57 or later`**, per the rule against symbols standing in for words in prose. - **Consistent product naming.** \"the dashboard head,\" \"the dashboard profiling endpoints,\" and \"the head node dashboard port\" now all say **Ray Dashboard**. - **Split one long multi-clause sentence** in the profiler-initialization section, which carried a mid-sentence \"which defaults to port 9999\" clause. - **Added the missing blank line between each H3 and its lead-in prose.** The H2s in the file already had them. - **`profiling.md`:** rewrote the new TPU profiling paragraph, which used the \"Ray provides integration for\" framing the guide steers away from. Four H3 headings changed wording slightly, so their generated anchors change. The page is a day old and nothing links to those anchors yet. ### Left alone on purpose The `` placeholders. The style guide's example uses lowercase-hyphenated names inside angle brackets, but a survey of `doc/source` shows uppercase-in-brackets is the established convention across Ray docs. Not worth churning this page against the rest. ## Questions for the author and reviewers Two technical points I noticed but didn't touch, since they're outside a style reviewer's scope: 1. **The `ls` command and its expected output don't line up.** The guide runs: ```bash kubectl exec -it -c ray-worker -- ls -la /tmp/ray/session_latest/logs/profiles ``` and shows the output as: ```text -rw-r--r-- 1 ray users 79526813 Jun 2 15:26 /tmp/ray/session_latest/logs/profiles/plugins/profile/2026_06_02_15_26_15/localhost_9999.xplane.pb ``` A plain `ls -la` on that directory would list its immediate children (`plugins`), not a nested file with its full path. Was the sample output captured with `ls -laR`, a `find`, or an `ls` against the full nested path? Happy to send the fix once you confirm which. 2. **`-it` on that `exec` looks unnecessary** for a non-interactive `ls`. It also fails in contexts without a TTY, such as a CI script. Worth dropping to just `kubectl exec -c ray-worker -- ls ...`? ## Not a duplicate No open PR or issue targets `doc/source/ray-observability/user-guides/jax-tpu-profiling.md`. Checked with `gh pr list --repo ray-project/ray --state open --search \"jax-tpu-profiling in:title,body\"` and `gh search prs --repo ray-project/ray --state open \"jax tpu profiling\"`; both empty. All of the review feedback on #64735 (the hardcoded personal paths and registry names from the Gemini review, @andrewsykim's questions on the custom image, the sample RayJob, and pod scoping) was addressed by the author before merge, so nothing here reopens a settled thread. ## Verification - `vale --config=.vale.ini --output=line doc/source/ray-observability/user-guides/jax-tpu-profiling.md` β€” clean apart from `Google.Acronyms` on \"JAX\" and the two `Google.Passive` hits this PR fixes. Advisory only: per `.vale.ini`, Vale is scoped to the Ray Data docs and the example gallery, so it doesn't gate `ray-observability`. - `pre-commit run --files ` β€” no hook covers Markdown under `doc/source/`, so every check reports \"no files to check.\" - The two corrections in commit 2 were read out of `python/ray/util/tpu.py` and `python/ray/dashboard/modules/reporter/reporter_head.py`, and the plugin's identity from the PyPI metadata for `tensorboard-plugin-profile`. No behavior claim here comes from the page's own prose. - No code changed, so no unit tests apply. The Sphinx build is the real gate. Relying on the Read the Docs preview build on this PR to confirm the new front matter parses and the `{ref}` targets still resolve. AI assistance (Claude Code) was used to draft this change. I reviewed every changed line and I stand behind it.", "url": "https://github.com/ray-project/ray/pull/65287", "createdAt": "2026-08-07T16:02:21Z", "updatedAt": "2026-08-12T18:07:35Z", "timestamp": "2026-08-12T18:07:35Z", "metrics": { "reactions": 1, "comments": 2 }, "labels": [ "docs", "core", "go", "docs-go" ], "author": "dstrodtman", "state": "closed", "assignees": [] }, { "id": "github:ray-project/ray:pull_request:65298", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[core][2/N] Reconnect to Redis in place when the connection drops", "text": "## Description When the async connection to Redis drops, hiredis frees the raw `redisAsyncContext` and nothing puts it back. Every subsequent GCS command fails with `Status::Disconnected` until `num_redis_request_retries` runs out (~3.5s), and gcs_server aborts with `Failed to run redis cmds`. A Redis restart, an idle timeout, a proxy dropping its backend, or a Sentinel failover is enough to take the head node down, which defeats the purpose of GCS fault tolerance. [#64299](https://github.com/ray-project/ray/pull/64299) (1/N) made the connection-failure paths return `Status` instead of crashing. This PR adds the reconnect itself, as a stack of three commits that each build and fix something on their own. The commit messages carry the design details; in short: **1/3: Rebind the async context in place.** In-flight `RedisRequestContext`s hold a raw `RedisAsyncContext*`, so the reconnect preserves object identity: `Reset()` swaps the raw hiredis handle while the wrapper stays put. Success is only declared from the hiredis connect callback, since `redisAsyncConnect` reports a healthy context even when nothing is listening. **2/3: Re-resolve the address on every reconnect attempt.** The recorded address survives a failover behind a proxy or Service, but for Sentinel it is the old primary's. Sentinel setups now re-query the Sentinel on each attempt and follow the promotion; everything else re-resolves through DNS. Redis Cluster addressed directly has no lookup point and stays out of scope (documented: front it with a Service or proxy). **3/3: Don't spend a command's retries on attempts that never reached Redis.** A Sentinel promotion takes ~4s, the command retry budget ~3.5s, so gcs_server aborted moments before the reconnect landed. Null replies (never reached Redis) are refunded within the new `redis_reconnect_grace_period_ms` (default 60s; `0` restores the old behaviour exactly). Error replies (`-READONLY`, …) drain the budget as before. `test_redis_with_sentinel_failureover` asserted the old behaviour and carried the TODO for this work in its docstring. It now asserts the same process survives the failover and that the old cold-restart coverage still holds. How the three commits interact, on the case that needs all of them (Sentinel direct, measured timings): ```mermaid sequenceDiagram participant REQ as RedisRequestContext participant RC as RedisContext participant S as Sentinel participant P1 as primary :7001 participant P2 as replica :7002 Note over REQ,P1: connected. command budget: 6 tries in ~3.5s P1--xRC: connection drops (t=0) Note over REQ: 3/3: null replies are refunded within the 60s grace,
so the command is not spending its 6 tries yet loop backoff 100ms..1s, up to redis_db_connect_retries=120 RC->>S: 2/3: SENTINEL MASTERS S-->>RC: primary is :7001 RC->>P1: async connect P1--xRC: refused end RC->>S: SENTINEL MASTERS S-->>RC: primary is :7002 (promoted at t=4s) RC->>P2: async connect P2-->>RC: connect callback OK Note over RC: 1/3: Reset() rebinds the same RedisAsyncContext,
so in-flight requests keep their raw pointer valid REQ->>P2: retry succeeds ``` Before this PR the timeline ends at t=3.5s with `Failed to run redis cmds: [PING] for 5 times`. ### Behaviour matrix | Topology | Before | After | |---|---|---| | Plain Redis, drop/restart | FATAL in ~3.5s / SIGSEGV | survives, reconnects in ~0ms | | Proxy / K8s Service, backend failover | dies before proxy recovers | survives (~1ms, same VIP) | | Sentinel direct, primary failover | dies ~0.5s before promotion | survives, follows the promotion | | Redis Cluster direct, failover | dies | still unsupported; documented, use a Service/proxy in front | | Redis permanently unreachable | FATAL ~3.5s | still exits (bounded by grace + retry drain), deliberately | ### Verification Everything below ran against real Redis processes and containers (killed, restarted, demoted; no mocks). Each scenario runs a task (`ray.get`) before and after the failure, so \"survives\" means the cluster still schedules work, not just that the process exists.
Plain Redis (single node) β€” 9 scenarios | Scenario | Before | After | |---|---|---| | Connection drop | FATAL in ~3.5s | survives, 0ms reconnect, tasks run | | Process restart (no persistence) | SIGSEGV | survives | | Restart while loading a 400k-key AOF | dies | survives, no `-LOADING` reply leaks | | Never comes back | FATAL ~3.5s | bounded FATAL after grace + drain, no hang | | Rapid flapping (drop every 200ms for 10s) | dies on first drop | survives, 88 reconnects, 0 crash markers | | Concurrent load across a drop | dies | 160,140 tasks, 0 errors | | Soak: 60 drops | β€” | RSS +0.7MB, fd +2, 120 reconnects | | Worker node attached across a drop | β€” | both nodes stay alive, tasks on both | | Detached actor across a drop | β€” | callable after reconnect |
HAProxy / Kubernetes Service + Sentinel-managed failover β€” 2 scenarios | Scenario | Before | After | |---|---|---| | Kill primary container β†’ Sentinel promotes β†’ proxy repoints | dies | survives, reconnects through the same VIP, tasks run | | Second backend churn 10s later | β€” | same recovery path |
Sentinel addressed directly β€” 8 scenarios | Scenario | Before | After | |---|---|---| | Primary killed β†’ replica promoted (~4s) | dies at ~3.5s | survives, logs `Sentinel now reports the primary at :7002, was :7001`, reconnects to the promoted node | | Two failovers in a row | β€” | follows both promotions, both directions | | Old primary rejoins as replica | β€” | stays on the new primary | | Sentinel dies while the primary is down | β€” | keeps retrying, no crash, no false \"Reconnected\" | | Sentinel reports a dead address, then a live one | β€” | adopts the good address once offered | | Password auth across the failover | β€” | 0 auth errors | | Promoted node rejects AUTH | β€” | keeps retrying, no false success | | GCS cold restart after the failover | covered by the old test | still covered; detached actor recovered from the promoted primary |
Redis Cluster / TLS β€” 4 scenarios | Scenario | Result | |---|---| | Single-shard cluster, drop | survives, reconnects, tasks run | | `CLUSTER FAILOVER`, GCS pointed at a node directly | does not recover; measured, documented with the workaround | | `rediss://` drop + reconnect | survives, reconnect re-initiates TLS | | TLS + Sentinel re-query | re-query initiates TLS like the startup path |
Config knobs β€” 6 scenarios | Knob | Verified | |---|---| | `redis_reconnect_grace_period_ms=0` | restores pre-change behaviour exactly (~3.5s FATAL) | | `redis_reconnect_grace_period_ms=5000` | bounded exit at grace + drain | | `redis_db_connect_retries=0` / `=1` | exits after exactly 0 / 1 attempts | | `redis_retry_base_ms=1, max_ms=10` | no hot spin, CPU flat | | `num_redis_request_retries=0` | idle drop survives, 0 wasted connects | | `retries=3` + `grace=60s` | the smaller bound wins |
Unit tests: 4 new cases in `redis_async_context_test` (in-place `Reset`, never-connected release, teardown during reconnect). All store-client tests pass at each commit of the stack independently. Not verified locally: Linux CI (this PR is its first run), ASAN/TSAN (macOS toolchain), IPv6. ### Known limitations (deliberately out of scope) - The reconnect handshake (Sentinel re-query, DNS resolve) runs synchronously on the io_service thread. Each step is bounded by the probe timeout, but a fully async handshake is follow-up work. - Error replies are not refunded: a Service that keeps routing writes to a demoted (read-only) primary for longer than the command budget still brings GCS down, exactly as before this PR. - Against a SYN-blackholed address, each async connect attempt settles only after the kernel TCP timeout, which stretches the `redis_db_connect_retries` budget in wall-clock terms. ## Related issues Closes #48781. Related: #53475, #64204 (SIGSEGV on failover, fixed earlier), #64299 (1/N prerequisite). ## Checks - [x] I've signed off every commit(by using the -s flag, i.e., `git commit -s`) in this PR. - [x] I've run `scripts/format.sh` to lint the changes in this PR. - [x] I've included any doc changes needed for https://docs.ray.io/en/master/. (`kuberay-gcs-ft.md`: new section on what happens when the Redis connection drops, plus the tuning knobs.) - [ ] I've added any new APIs to the API Reference. N/A: no public API change; one new internal config (`RAY_redis_reconnect_grace_period_ms`). - [x] I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/ - Testing Strategy - [x] Unit tests - [ ] Release tests - [ ] This PR is not tested :( ## AI assistance disclosure AI assistance (Claude) was used for the implementation, the verification harnesses, and this description. I (the human submitter) have reviewed every changed line and understand the change end-to-end. - **Not duplicating existing work:** the previous attempt (#48781) was auto-closed as stale in July 2025, and a search of open PRs on 2026-08-08 found none implementing Redis reconnection (#64847 and #62462 touch gRPC reconnect settings, not the Redis connection). The adjacent fixes #64204 and #64299 landed separately and this PR stacks on them. - **Test commands run locally** (macOS source build; results in the Verification section above): - `bazel test //src/ray/gcs/store_client/tests:redis_async_context_test //src/ray/gcs/store_client/tests:redis_store_client_test` β€” pass at each commit of the stack - `pytest python/ray/tests/test_gcs_fault_tolerance.py::test_redis_with_sentinel_failureover` β€” passed (16.4s), with `TEST_EXTERNAL_REDIS=1` - a 29-scenario topology harness against real Redis/Sentinel/HAProxy processes and containers, run before and after the change", "url": "https://github.com/ray-project/ray/pull/65298", "createdAt": "2026-08-08T14:08:57Z", "updatedAt": "2026-08-13T06:18:16Z", "timestamp": "2026-08-13T06:18:16Z", "metrics": { "reactions": 1, "comments": 3 }, "labels": [ "docs", "core", "community-contribution" ], "author": "nadongjun", "state": "open", "assignees": [ "rueian" ] }, { "id": "github:ray-project/ray:pull_request:65301", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[Core] Enforce the task_id/put_index contract in GetGeneratorReturnId", "text": "## Description `WorkerContext::GetGeneratorReturnId` documents that the caller supplies both `task_id` and `put_index` or neither, because deducing only one of them keys the ObjectID to one task while drawing the index from another. The `RAY_CHECK` meant to enforce that is a tautology: ```cpp RAY_CHECK((task_id.IsNil() && !put_index.has_value()) || (!task_id.IsNil() || put_index.has_value())); ``` With A = `task_id.IsNil()` and B = `put_index.has_value()`, the second term `!A || B` is false only when `A && !B`, and in exactly that case the first term `A && !B` is true. The disjunction holds for all four inputs, so the check has never fired since it was added in #35584. The unguarded combination that matters is a real `task_id` with no `put_index`: it takes the index from this thread's put counter, which belongs to whatever task this thread is running, so the resulting ObjectID can collide with one that task mints. Using `&&` for the second term makes the check the both-or-neither predicate the doc describes. `AllocateDynamicReturnId`'s `put_index` default was `-1`. `ObjectIDIndexType` is `uint32_t`, so that default is an optional holding `UINT32_MAX`, not the `std::nullopt` its docstring describes. A caller using the default would abort: the two defaults together are the illegal Nil-plus-index combination, and even past the check, 4294967295 exceeds the 1e8 generator-return limit. Defaulting it to `std::nullopt` makes the two defaults together mean \"deduce both\", which is what the docstring promises. This also compares the unwrapped `current_put_index` against the generator-return limit instead of the optional. `std::optional`'s mixed comparison yields false when empty, so an empty optional would skip that check rather than trip it. The optional is engaged at that point today, so this is only to keep the guard from silently going dead if the surrounding branches are ever merged. Neither defect is reachable today: `libcoreworker.pxd` declares all three parameters without defaults so Cython cannot omit one, there is no C++ caller, and all four call sites in `_raylet.pyx` pass a legal combination. So this makes the contract hold for the next caller rather than fixing something users hit. ## Related issues Fixes #65300 ## Additional information Adds `context_test.cc`; `WorkerContext` had no test file. Two death tests cover the illegal combinations and two cases cover the legal ones, including the Nil-plus-nullopt form that both production Python call sites use. The death tests match the check message rather than `\"\"` or `\".*\"` deliberately. Before the fix, Nil with an index slipped past the tautology and segfaulted dereferencing a null current task, and since a segfault also counts as death, a permissive matcher would have accepted it as the expected failure. I verified both directions by reverting only the predicate and re-running: the two death tests then fail in two different ways (one never dies, the other dies with the segfault instead of the check), and both pass with the fix. `bazel test //src/ray/core_worker/tests:context_test` passes, and `//src/ray/core_worker:core_worker_lib` builds. I used AI assistance to investigate and draft this change. I reviewed every changed line and ran the build and tests myself.", "url": "https://github.com/ray-project/ray/pull/65301", "createdAt": "2026-08-08T18:39:02Z", "updatedAt": "2026-08-12T20:02:54Z", "timestamp": "2026-08-12T20:02:54Z", "metrics": { "reactions": 1, "comments": 1 }, "labels": [ "core", "go", "community-contribution" ], "author": "LuciferYang", "state": "closed", "assignees": [ "martinlhw" ] }, { "id": "github:ray-project/ray:pull_request:65351", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[llm][ci] Upgrade to vllm 0.27.0", "text": "## Description > Briefly describe what this PR accomplishes and why it's needed. ## Related issues > Link related issues: \"Fixes #1234\", \"Closes #1234\", or \"Related to #1234\". ## Additional information > Optional: Add implementation details, API changes, usage examples, screenshots, etc.", "url": "https://github.com/ray-project/ray/pull/65351", "createdAt": "2026-08-10T23:53:04Z", "updatedAt": "2026-08-13T06:39:35Z", "timestamp": "2026-08-13T06:39:35Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "go" ], "author": "jeffreywang88", "state": "open", "assignees": [] }, { "id": "github:ray-project/ray:pull_request:65363", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[doc] Update the number of index.rst document", "text": "## Description documents is five, the code is 2 the number is incorrect ## Related issues > Link related issues: \"Fixes #65364\" ## Additional information update the number", "url": "https://github.com/ray-project/ray/pull/65363", "createdAt": "2026-08-11T08:10:00Z", "updatedAt": "2026-08-13T06:33:03Z", "timestamp": "2026-08-13T06:33:03Z", "metrics": { "reactions": 1, "comments": 1 }, "labels": [ "core", "community-contribution" ], "author": "xubo245", "state": "open", "assignees": [ "Yicheng-Lu-llll" ] }, { "id": "github:ray-project/ray:pull_request:65366", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[core][runtime_env] Support isolated and cached pip dependencies with image_uri containers", "text": "## Description Allow combining the `image_uri` runtime environment field with pip. The runtime env agent prepares one virtual environment per cache key (image digest, Python ABI, Ray version, normalized pip config, and pip install environment) using the image's own interpreter, publishes it atomically under a per-key file lock, and mounts it read-only into every equivalent worker container instead of reinstalling per worker. Highlights: - Node-resolved cache URIs (resolve_uris) with dynamic reference counting in the agent, including release of orphaned references when an env is deleted while its creation is still in flight. - Atomic publish: staging dir + fsynced manifest + rename; failed or cancelled installs are cleaned up and never published. - Cross-process flock protocol shared by create, eviction, and the startup staging sweep; eviction renames to trash and removes the tree off the event loop. - Workers with pip run the exact image the environment was built against; image-only envs keep tag references and podman auto-pull. - Python/Ray version match enforced when pip is used, warned otherwise. ## Related issues Closes #65322", "url": "https://github.com/ray-project/ray/pull/65366", "createdAt": "2026-08-11T11:39:54Z", "updatedAt": "2026-08-13T10:36:32Z", "timestamp": "2026-08-13T10:36:32Z", "metrics": { "reactions": 1, "comments": 1 }, "labels": [ "core", "community-contribution" ], "author": "zzchun", "state": "open", "assignees": [] }, { "id": "github:ray-project/ray:pull_request:65368", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[ci] Remove the banned-words lint check in favor of Vale", "text": "## Summary Removes the `banned_words` lint check and its Buildkite matrix entry. `ci/lint/check-banned-words.sh` is a `git grep` for three fixed strings across `*.py`, `*.rst`, `*.md`, and `*.ipynb`. Vale already flags all three, at `error` severity, with better messages. This PR deletes the redundant check. ## The three terms, and what catches them today The check enforces exactly three case-sensitive strings. Every one is already covered by a Vale rule in this repository, with no configuration change required by this PR: | Term removed from the grep | Correct form | Vale rule that catches it | Source | | --- | --- | --- | --- | | `Kuberay` | `KubeRay` | `Google.WordList` (error) | `\"[Kk]ube[Rr]ay\": KubeRay` in `.vale/styles/Google/WordList.yml` | | `RLLib` | `RLlib` | `Vale.Terms` (error) | `(RL\\|rl)lib` in `.vale/styles/config/vocabularies/RLlib/accept.txt` | | `Rllib` | `RLlib` | `Vale.Terms` (error) | same entry | Verified against a fixture page containing both the correct and incorrect forms: ``` t.md:5:18:Google.WordList:Use 'KubeRay' instead of 'Kuberay'. t.md:5:27:Vale.Terms:Use 'RLlib' instead of 'RLLib'. t.md:5:38:Vale.Terms:Use 'RLlib' instead of 'Rllib'. ``` The correct spellings `KubeRay` and `RLlib` produce no alerts, so the rules discriminate on capitalization the same way the case-sensitive grep did. Vale's messages name the correct form; the grep only says a misspelled word was found and asks the author to go fix it. The `Vale.Terms` message improves further on Vale 3.17.1 (#65375), which renders the vocabulary entry as `RLlib` rather than echoing the raw `(RL|rl)lib` pattern. Because coverage already exists, this PR deliberately adds nothing to the `reject.txt` vocabulary. A reject entry would fire `Vale.Avoid` *in addition to* the rule above, double-reporting the same token with a strictly less useful message (\"Avoid using 'Kuberay'\" versus \"Use 'KubeRay' instead of 'Kuberay'\"). ## What the check costs The step carries the `always` tag in `.buildkite/lint.rayci.yml`, so it runs on every pull request, including documentation-only ones. Measured across 89 `lint: banned_words` jobs from recent passing `premerge` builds: | Measure | p50 | p90 | Range | | --- | --- | --- | --- | | Job run time | 111s | 116s | 108s – 642s | | Queue wait before start | 42s | 51s | 8s – 146s | | Wall clock, queue plus run | 153s | β€” | β€” | Nearly all of that is fixed overhead rather than work. Taking build 71819 as a representative sample, the job started at 10:59:01.6 and finished at 11:00:52.8, and `Checking for common mis-spellings...` printed at 11:00:52. So roughly 110 seconds went to the repository clone and the forge image pull, and the `git grep` itself accounted for under a second of it. At 391 `premerge` builds over the last seven days, the step consumes on the order of 12 agent-hours per week to run three sub-second greps. ## Why not keep it anyway Beyond the cost, the grep can't express the rule properly: - **No scoping.** Vale rules apply per path glob. The grep is repo-wide, with no way to hold a stricter bar in one tree than another. - **No exceptions.** Vale has `TokenIgnores`, `BlockIgnores`, and vocabularies. The grep has none, so a page that needs to *name* an incorrect spelling can't. That's not hypothetical: #65239 adds a Kubernetes and KubeRay terminology table to the style guide, and the row telling contributors to write `KubeRay` rather than `Kuberay` fails the check. There is no way to satisfy both. - **No severity.** Vale has `suggestion`, `warning`, and `error`. The grep is a hard CI failure or nothing. - **No extensibility.** Adding a term means editing a shell string that ships its own bespoke failure output. Adding a term to Vale means one line in a vocabulary or `swap` map, and it inherits the shared message, link, and reporting. ## The remaining gap, stated plainly The gap this PR opens is **scope, not rule coverage**. Vale runs today only against `doc/source/data/` and `doc/source/ray-overview/examples`, both in CI and in the pre-commit hook. Within those trees the three spellings are enforced today and stay enforced after this PR. Outside them, the repo-wide grep was the only thing checking, and nothing replaces it until Vale's scope widens. Adding the terms to a Vale vocabulary would not close that gap, because the limit is which paths Vale is pointed at, not which rules it has. The docs team is widening that scope over the coming weeks. ## Related changes - Drops the `banned_words` entry from the always-on lint matrix in `.buildkite/lint.rayci.yml`. - Removes the local-run instructions from `doc/source/ray-contribute/getting-involved.md`. - Updates the prose-only lint step count in `doc/source/ray-contribute/ci.md` from four to three. ## Checks - [x] I've signed off every commit (`git commit -s`). - [x] I've made sure the tests are passing. - [x] Release tests and any other tests I've added and changed pass. - [x] This PR is not tested :(", "url": "https://github.com/ray-project/ray/pull/65368", "createdAt": "2026-08-11T12:30:49Z", "updatedAt": "2026-08-12T16:33:29Z", "timestamp": "2026-08-12T16:33:29Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "docs", "core", "devprod", "go" ], "author": "dstrodtman", "state": "open", "assignees": [] }, { "id": "github:ray-project/ray:pull_request:65370", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "serve: expose multiplexed model IDs in ReplicaDetails", "text": "## Summary Adds a backward-compatible `multiplexed_model_ids` field to `ReplicaDetails` and keeps it synchronized when the controller receives updated multiplexed model IDs. Extends the existing multiplexing test to verify the values through `ServeInstanceDetails` returned by `serve_instance.get_serve_details()`. Closes #65347 ## Duplicate check I checked open PRs for #65347 and for `multiplexed_model_ids ReplicaDetails`; no matching open PR existed. ## Testing - `python -m compileall` on all four changed Python files β€” passed. - Direct execution of `test_record_multiplexed_model_ids_updates_actor_details` β€” passed. - `git diff --check` β€” passed. - Ruff, pydoclint, Ray Serve mypy, Pyrefly, import-order, and Black checks β€” passed. - The full pytest integration test could not be collected locally because Windows `setup-dev.py` junctions caused duplicate `conftest.py` registration. Ray CI will run the added integration test on Linux. - `docstyle` could not run because `/bin/bash` is unavailable in this Windows environment; Semgrep's pre-commit environment had an invalid cached Python path. ## AI assistance I used AI assistance to help implement and validate this change. I reviewed and understand every changed line.", "url": "https://github.com/ray-project/ray/pull/65370", "createdAt": "2026-08-11T13:32:40Z", "updatedAt": "2026-08-13T14:46:49Z", "timestamp": "2026-08-13T14:46:49Z", "metrics": { "reactions": 1, "comments": 3 }, "labels": [ "serve", "community-contribution" ], "author": "upadhyay74aman", "state": "open", "assignees": [] }, { "id": "github:ray-project/ray:pull_request:65375", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[CI] Make the Vale pre-commit hook run, and upgrade Vale to 3.17.1", "text": "## Description Two related fixes to the Vale setup. ### 1. The `vale` pre-commit hook has never run The top-level `exclude` in `.pre-commit-config.yaml` lists `doc/source/`, and pre-commit ANDs that global `exclude` with each hook's own `files` pattern. The Vale hook's pattern, `^doc/source/data/.*\\.(md|rst)$`, lives entirely inside `doc/source/`, so the two can never both be satisfied and the hook matches zero files: ``` $ pre-commit run vale --all-files vale.................................................(no files to check)Skipped ``` Committing a change under `doc/source/data` skips it silently too. Vale still ran in CI through `ci/lint/check-documentation-style.sh`, so the gate existed, but contributors got no local signal, and a green pre-commit run said nothing at all about Vale. This narrows the `doc/source/` entry to `doc/source/(?!data/.*[.](md|rst)$)`, so it keeps excluding the tree except the prose the hook targets. Narrowing it also exposes those 48 files to hooks that had never seen them. Three needed exemptions, all recorded in the config with comments: * `trailing-whitespace` and `end-of-file-fixer` would rewrite 17 unrelated files. I've exempted those two hooks from `doc/source/data` so that enabling Vale stays a config-only change. **The whitespace cleanup is worth doing, and I'm happy to send it as its own PR** rather than bury it here, partly because three of those 17 files are also touched by the open #65372. * `semgrep` would fail. Caught by Cursor Bugbot, and it was right. `semgrep.yml`'s `code-block-python` rule lists `doc/source/data/**/*.rst` in its `paths.include` at `severity: ERROR`, there are 8 pre-existing violations across 5 files, and `ci/lint/lint.sh` runs `pre-commit run semgrep --all-files` as the `semgrep_lint` step. Converting those code blocks to `testcode` is real work that belongs in its own PR, not in a Vale change. The two rST hooks that also become eligible pass as-is. If you'd rather just take the whitespace churn here, say so and I'll fold it in. **A bigger finding, which I'm raising rather than fixing here.** That `semgrep` exemption is a second instance of the same bug, not a new one. Because the top-level `exclude` has covered `doc/source/`, the `code-block-python` rule has never run on **any** of the five doc paths it targets: 31 occurrences across 16 files in `doc/source/data`, `ray-core`, `ray-observability`, and `train`. Every one is an untested documentation snippet the rule was written to catch. Happy to take that on as a follow-up, but converting snippets means they start executing in CI, so it wants a maintainer opinion on scope and on whether the rule is still wanted for docs at all. ### 2. Vale was pinned to 3.4.1, from June 2024 This upgrades both the hook `rev` and the CI script to 3.17.1. The bump isn't inert, so three supporting changes come with it. `Vale.Spelling` produces **no alerts at all** under 3.4.1, but 11 error-severity alerts under 3.17.1, because the newer engine word-boundary-anchors vocabulary patterns. Existing entries such as `[Bb]ackpressure`, `dtype`, and `namespace` therefore stopped covering `backpressured`, `dtypes`, and `namespaces`. This widens those three and adds terms that were never listed at all: `Databricks`, `Datasink`, `Multimodality`, `preprocess`, `resample`, `vCPU`, `Vectorizer`. Two Google rules also became error-severity on content where they're wrong, so they're set to `warning` rather than editing correct prose: * `Google.HeadingPunctuation` reads the period in a numbered heading such as `### 1. Label your worker nodes` as a heading-ending period. The match it reports is literally `1.`. * `Google.OxfordComma` fires on \"X, A and B\" where A and B modify X rather than continuing a list, as in \"resources used by all operators, active and requested\". Both follow the existing precedent of tuning Google rules in `.vale.ini` alongside `Google.Colons` and `Google.Headings`, and both preserve 3.4.1's effective behavior, where neither rule blocked. ## Related issues None. Cross-referencing two of my own open PRs that touch adjacent surfaces, neither of which shares a file with this one: * #65368 removes the banned-words lint in favor of Vale. That increases how much Ray leans on Vale, which makes a dead local hook and a two-year-old pin more worth fixing. Touches `ci/lint/check-banned-words.sh`, `ci/lint/lint.sh`, and `.buildkite/lint.rayci.yml`. * #65342 makes the docs-go scope guard count Vale configuration as documentation content. Worth noting that after it merges, a Vale-config-only PR like this one would classify as documentation. Touches `ci/lint/validate_docs_go_scope.sh`. ## Additional information **AI assistance was used.** I used Claude Code to investigate and draft this. I reviewed every changed line, ran the measurements below myself, and can defend the change end to end. **Not a duplicate.** `gh pr list --repo ray-project/ray --state open --search \"vale in:title\"` and the same for `pre-commit` return only #65368 and #65342 above, which change different files and have different goals. **Measurements**, on the exact set `ci/lint/check-documentation-style.sh` lints (`doc/source/data` and `doc/source/ray-overview/examples`): | Vale | Total alerts | Error-severity | Exit code | Lint step | |---|---|---|---|---| | 3.4.1 (current pin) | 162 | 0 | 0 | green | | 3.17.1, bump alone | 208 | 14 | 1 | **red** | | 3.17.1, with this PR | 196 | 0 | 0 | green | That middle row is the reason the vocabulary and `.vale.ini` changes are in the same PR as the version bump: without them the bump turns `lint: documentation_style` red. **Checks run locally:** ``` # the CI script's exact target set, both versions vale doc/source/data doc/source/ray-overview/examples # 3.4.1 -> exit 0; 3.17.1 pre-fix -> exit 1; post-fix -> exit 0 # the hook now actually matches files and passes pre-commit run vale --files doc/source/data/data-internals.rst # vale.....................................................................Passed # and genuinely fails, rather than passing vacuously: appended a line # containing \"This sentance has a deliberate spelling mistake.\" pre-commit run vale --files doc/source/data/data-internals.rst # 349:6 error Did you really mean 'sentance'? Vale.Spelling # 1 error, 0 warnings and 17 suggestions in 1 file. # the two hooks I exempted stay exempt, so no file churn pre-commit run trailing-whitespace --files $(git ls-files 'doc/source/data' | grep -E '\\.(md|rst)$') pre-commit run end-of-file-fixer --files $(git ls-files 'doc/source/data' | grep -E '\\.(md|rst)$') # both: (no files to check)Skipped # the two rST hooks that newly become eligible pre-commit run rst-directive-colons --files ... # Passed pre-commit run rst-inline-touching-normal --files ... # Passed ``` I also confirmed the 3.17.1 release publishes the two asset filenames the CI script downloads, `vale_3.17.1_Linux_64-bit.tar.gz` and `vale_3.17.1_macOS_arm64.tar.gz`. **Net effect:** contributors get Vale locally, CI runs a Vale two years newer, and `Vale.Spelling` becomes an enforced check instead of a silent one.", "url": "https://github.com/ray-project/ray/pull/65375", "createdAt": "2026-08-11T16:20:57Z", "updatedAt": "2026-08-12T14:45:09Z", "timestamp": "2026-08-12T14:45:09Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "docs", "core", "devprod", "go" ], "author": "dstrodtman", "state": "open", "assignees": [], "change": "updated" }, { "id": "github:ray-project/ray:pull_request:65388", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[doc] State that the ray-ml images are deprecated, and stop recommending them", "text": "## Why Ray announced the `rayproject/ray-ml` deprecation in #46378 and, in a follow-up comment on that issue, stopped publishing the images entirely as of Ray 2.50 β€” \"not even with `-deprecated` tags.\" Nothing a reader can see says so. That issue went stale and was closed, and the deprecation never reached the docs. The gap costs the reader real time, because the frozen tags don't fail loudly. Verified against Docker Hub on 2026-08-11: | Tag | Status | |---|---| | `ray-ml:latest`, `ray-ml:latest-gpu` | 200, frozen at the `2.30.0` digest, last pushed 2024-06-20 | | `ray-ml:2.31.0` … `2.57.0` | 404 | | `ray-ml` newest push of any kind | `nightly-*` on 2025-10-01 | A 404 teaches the reader something. A 200 serving a two-year-old Ray teaches them nothing. `ray-overview/installation.rst`, the canonical image reference, documents `rayproject/ray` only and has never mentioned `ray-ml`, so a reader arriving with an existing `ray-ml` reference finds no explanation for its absence. ## What changed **`ray-overview/installation.rst`** β€” adds the deprecation statement to the `docker-images` section, where a reader looking for image guidance lands. States what replaces it: build on a `rayproject/ray` base, or declare the packages in a runtime environment. Documents the constraint file every `rayproject/ray` image ships at `/home/ray/requirements_compiled.txt`, which installs the exact library version Ray tested against for that release. Links #46378 for the rationale rather than restating it. **`cluster/kubernetes/user-guides/gpu.rst`** β€” recommended `ray-ml:2.6.3-gpu` as \"ideal for running GPU-based ML workloads,\" in both prose and a config snippet. Now recommends `ray:2.57.0-gpu` and says plainly that the image doesn't include TensorFlow or PyTorch. The page already pointed at custom images and runtime environments as the way to add dependencies; that guidance just contradicted the recommendation above it. **`cluster/vms/configs/xgboost-benchmark.yaml`** β€” ran the XGBoostTrainer benchmark on `ray-ml:2.0.0`. Now uses `ray:2.57.0`, plus a `setup_commands` entry that installs `xgboost` against the image's constraint file. `xgboost` appears nowhere in `python/setup.py`, so `ray[all]` doesn't include it and a bare image swap would have broken the benchmark. ## Deliberately not in this PR Four other `ray-ml` references remain in `doc/source`, each blocked on something outside this change: - `cluster/vms/references/ray-cluster-configuration.rst` and `cluster/kubernetes/user-guides/config.md` β€” already covered by #65339 and #65336 respectively. Excluded to avoid conflicts. - `cluster/kubernetes/examples/mobilenet-rayservice.md`, `stable-diffusion-rayservice.md`, and `rayjob-batch-inference-example.md` β€” these describe sample manifests that live in `ray-project/kuberay`, and the prose accurately reports what those manifests do. Rewriting the prose alone would put the docs at odds with the file the reader applies. That last group needs a heads-up, because those three examples are broken today rather than merely stale. On both kuberay `v1.6.0` and `master`, `ray-service.mobilenet.yaml`, `ray-service.stable-diffusion.yaml`, and `ray-job.batch-inference.yaml` all pin `rayproject/ray-ml:2.46.0.0e19ea-py39-{cpu,gpu}`, which 404s. Following any of those three examples ends in `ImagePullBackOff`. Fixing them isn't a tag bump either: `mobilenet` needs `tensorflow` and the other two need `torch`, all of which came from the `ray-ml` image, so each manifest needs `runtime_env` additions. Happy to take that on in kuberay if maintainers agree on the approach, or to defer to whoever owns those samples. Separately, `ray-job.batch-inference.yaml`'s image tag is already out of sync with the snippet quoted in `rayjob-batch-inference-example.md` (`2.6.3-gpu` in the docs vs. `2.46.0.0e19ea-py39-gpu` in the manifest), independent of the deprecation. ## One thing outside docs scope The frozen `ray-ml:latest` and `latest-gpu` tags are the actual trap. A deprecation notice helps readers who come to the docs; removing or re-tagging those two on Docker Hub would help everyone with an existing Dockerfile or manifest that references them. Not something I can do from a docs PR, but worth a decision from whoever owns the registry. ## Testing Docs-only prose, RST, and cluster-config YAML; no code paths touched. - `pre-commit run --files ` β€” every hook reports \"no files to check.\" No pre-commit hook currently covers prose under `doc/source`. - Every replacement tag verified live against the Docker Hub registry API: `rayproject/ray` `2.57.0`, `2.57.0-gpu`, `2.57.0-cpu`, `2.57.0-py311-gpu`, `2.57.0-py311-cpu` all return 200, pushed 2026-08-11. - `xgboost-benchmark.yaml` parses under `yaml.safe_load`. - `xgboost==2.1.0` confirmed pinned in `python/requirements_compiled.txt`, so the constraint comment in that file is accurate. - Confirmed `setup_commands` run inside the container when `docker.image` is set: `DockerCommandRunner.run` with the default `run_env=\"auto\"` routes any command not beginning with `docker` through `docker exec`. - Confirmed the constraint file's in-image path: `docker/base-deps/Dockerfile` copies it to `/home/ray/requirements_compiled.txt`. - Cross-reference targets confirmed present: `runtime-environments` (`ray-core/handling-dependencies.rst:52`) and `docker-images` (`ray-overview/installation.rst:313`). - `installation.rst` parsed with docutils; the only diagnostics in the changed range are the expected Sphinx-only `:ref:` role. I did not run the 100Gi XGBoost benchmark or apply the KubeRay snippet, which need cloud resources I don't have here. ## AI assistance AI assistance (Claude Code) was used to inventory the `ray-ml` references and draft these edits. I reviewed every changed line and ran the verification above.", "url": "https://github.com/ray-project/ray/pull/65388", "createdAt": "2026-08-11T19:27:45Z", "updatedAt": "2026-08-12T18:07:21Z", "timestamp": "2026-08-12T18:07:21Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "docs", "core", "go" ], "author": "dstrodtman", "state": "closed", "assignees": [] }, { "id": "github:ray-project/ray:pull_request:65389", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[Data] OpTask._cancel never passes force=True", "text": "## Description Current behavior of `OpTask._cancel` never passes `force=True` to `ray.cancel`. It is because whether the task is an actor task is determined by checking if the actor id is nil, which is false for both normal tasks and actor tasks (refer to the issue). Slicing the hex method was withdrawn through discussions with maintainers. Instead, removed the check and lets Ray Core classify the task, with a fallback to force=False when Core rejects it for an actor task. ## Related issues Closes #65280 ## Additional information Two tests added in `TestOpTaskCancel` (`python/ray/data/tests/test_streaming_executor.py`) - One test for a normal task - One test for an actor task", "url": "https://github.com/ray-project/ray/pull/65389", "createdAt": "2026-08-11T19:46:54Z", "updatedAt": "2026-08-12T16:02:40Z", "timestamp": "2026-08-12T16:02:40Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "data", "community-contribution" ], "author": "Hyunoh-Yeo", "state": "open", "assignees": [], "change": "updated" }, { "id": "github:ray-project/ray:pull_request:65390", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "Fix/tls server cert hot reload", "text": "## Description Fixes the **server-side** half of #65365. Ray's Python gRPC servers previously read TLS cert/key/CA files once at startup via `grpc.ssl_server_credentials()`. When cert-manager rotates files on disk (e.g. KubeRay mTLS), running servers kept presenting the old certificate until process restart. This PR switches `add_port_to_grpc_server()` to `grpc.dynamic_ssl_server_credentials()` with a `_ReloadableServerCertConfig` fetcher that: - Re-reads cert files when their mtimes change - Seeds the mtime cache after the initial load to avoid redundant reloads - Falls back to the last known-good configuration on stat/read/parse errors (raising from the fetcher would break new handshakes) **Affected Python gRPC servers:** Ray Client server, dashboard agent gRPC, Serve replica internal gRPC. ### Known gaps (out of scope) | Area | Status | Notes | |------|--------|-------| | **Client/channel side** (`init_grpc_channel` in `grpc_utils.py`) | Not in this PR | Still reads TLS material once via `grpc.ssl_channel_credentials()`. Blocked on [grpc/grpc#41779](https://github.com/grpc/grpc/issues/41779) β€” Python gRPC has no dynamic reload API for channels. | | **C++ core** (`src/ray/rpc/grpc_server.cc`) | Not in this PR | GCS, raylet, object manager, etc. load certs once at startup via `grpc::SslServerCredentials()`. Separate path, unaffected by this change. | ## Related issues Related to #65365 (server-side scope). ## Additional information ### Test plan - [x] `python -m pytest -v -s python/ray/_common/tests/test_tls_utils.py` β€” 7/7 passed (macOS) - [ ] `python -m pytest -v -s python/ray/tests/test_tls_auth.py::test_tls_server_cert_rotation_without_restart` β€” skipped locally on darwin; runs in CI on Linux - [x] `pre-commit run --from-ref master --to-ref HEAD` β€” passed ### Contribution notes - No duplicate open PR found for this change. - AI assistance was used; all changes reviewed by the submitter. - Every commit includes DCO `Signed-off-by`.", "url": "https://github.com/ray-project/ray/pull/65390", "createdAt": "2026-08-11T19:55:54Z", "updatedAt": "2026-08-13T11:21:33Z", "timestamp": "2026-08-13T11:21:33Z", "metrics": { "reactions": 1, "comments": 3 }, "labels": [ "core", "community-contribution" ], "author": "Divyam19", "state": "open", "assignees": [] }, { "id": "github:ray-project/ray:pull_request:65394", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[Data] Chunk DataFrame emission in WebDatasetDatasource._read_stream", "text": "## Description `WebDatasetDatasource._read_stream` emits one single-row DataFrame per sample, which means the output buffer runs `PandasBlockAccessor.size_bytes` once per sample. On large shards this dominates read-side CPU (~63% of the `ReadWebDataset` operator's CPU). This changes the behavior of readwebdataset and let it accumulates 512 samples per emitted DataFrame. ## Related issues Closes #65350 ## Additional information Benchmark specs are specified in #65350 - It was cross-validated with a reporter Added a parametrized test (`test_read_webdataset_chunked_samples`) that verifies the output is identical to the original per-sample behavior (row count, values, and order) - test with different sample sizes (1 / 511 / 512 / 513 / 1000 samples).", "url": "https://github.com/ray-project/ray/pull/65394", "createdAt": "2026-08-11T20:35:01Z", "updatedAt": "2026-08-12T20:34:49Z", "timestamp": "2026-08-12T20:34:49Z", "metrics": { "reactions": 1, "comments": 1 }, "labels": [ "data", "go", "community-contribution" ], "author": "Hyunoh-Yeo", "state": "closed", "assignees": [] }, { "id": "github:ray-project/ray:pull_request:65398", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[serve] Add separate fast path for unary gRPC direct ingress", "text": "## What Currently, a unary request returns its single response through an interposed async generator instead of returning directly. Instead we split the handler into three pieces to maintain the speed of the unary-unary hot path. 1. `_prepare_direct_ingress_grpc_request` does the setup every request needs regardless of shape. 2. `_direct_ingress_grpc_unary_response` handles calls that return exactly one response. Returns the response directly. 3. `_direct_ingress_grpc_streaming_response` stays a generator and handles calls that yield a stream. Setup and teardown still are shared. The only difference is how they consume the response. ## Why Unary gRPC throughput on `pytest_serve_throughput_optimized_microbenchmarks` dropped when the handlers were unified, and this recovers most of it. Numbers are average rps from the release test. | metric | before | now | with this change | |---|---:|---:|---:| | `grpc_avg_rps` | 4710.6 | 4601.1 | 4684.1 | | `grpc_100_max_ongoing_requests_avg_rps` | 4726.2 | 4641.3 | 4678.3 | | `grpc_800_max_ongoing_requests_avg_rps` | 4759.7 | 4675.7 | 4731.0 |", "url": "https://github.com/ray-project/ray/pull/65398", "createdAt": "2026-08-11T21:14:00Z", "updatedAt": "2026-08-12T18:24:46Z", "timestamp": "2026-08-12T18:24:46Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "serve", "go" ], "author": "YashwanthRanjanSingaravel", "state": "open", "assignees": [] }, { "id": "github:ray-project/ray:pull_request:65403", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[TPU] Add subslice_index to subslice_placement_group", "text": "## Description When requesting multiple subslices simultaneously, it can be helpful to allow selecting specific subslices to guarantee a spread of jobs. This change adds an optional subslice_index parameter to the subslice_placement_group builder which restricts placement group selection. ## Related issues Follow on to #64578 ## Additional information n/a", "url": "https://github.com/ray-project/ray/pull/65403", "createdAt": "2026-08-11T23:41:38Z", "updatedAt": "2026-08-13T10:15:49Z", "timestamp": "2026-08-13T10:15:49Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "train", "core", "community-contribution" ], "author": "spencer-p", "state": "open", "assignees": [] }, { "id": "github:ray-project/ray:pull_request:65404", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[doc][TPU] Document TPU subslicing options with Ray", "text": "## Description This PR documents the three methods available for TPU subslicing with Ray: 1. Dynamic slicing with Kueue (provisions subslices via GKE) 2. RayClusters with a subslice annotation 3. SubslicePlacementGroup from the Ray TPU driver. ## Related issues - [Dynamic slicing docs](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/dynamic-slicing) - https://github.com/ai-on-gke/kuberay-tpu-webhook/pull/32 - #64578 ## Additional information May need updates pending https://github.com/ai-on-gke/kuberay-tpu-webhook/pull/32 Also included some minor code changes to make SubslicePlacementGroup compatible with tpu.dispatch that were previously missed.", "url": "https://github.com/ray-project/ray/pull/65404", "createdAt": "2026-08-11T23:49:34Z", "updatedAt": "2026-08-12T23:42:21Z", "timestamp": "2026-08-12T23:42:21Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "docs", "core", "community-contribution" ], "author": "spencer-p", "state": "open", "assignees": [] }, { "id": "github:ray-project/ray:pull_request:65405", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "fix(jobs): return structured 503 when job logs are unavailable", "text": "## Summary - return a structured `503 Service Unavailable` when a submission job still exists but its driver agent is no longer registered or reachable - preserve successful log retrieval, unknown-job `404` behavior, and the public SDK's existing `RuntimeError` contract - document the JSON error response and bump the Jobs API version The response does not imply that node-local logs are persisted or recoverable. This is a draft pending maintainer confirmation that `503` is the preferred API contract. Fixes #65402 ## Test plan - [x] `pytest -q python/ray/dashboard/modules/job/tests/test_http_job_server.py -k 'get_job_logs_from_live_agent or get_job_logs_when_driver_node_is_missing or get_job_logs_when_driver_agent_is_unreachable or get_job_logs_unknown_job_preserves_404'` - [x] `pytest -q python/ray/dashboard/modules/job/tests/test_sdk.py -k job_logs_after_driver_node_removed` - [x] `pytest -vv python/ray/dashboard/modules/job/tests/test_job_agent.py -k job_log_in_multiple_node --maxfail=1` - [x] `ruff check` and `ruff format --check` on changed Python files - [x] Python compilation and OpenAPI YAML parsing - [x] Upstream [microcheck](https://buildkite.com/ray-project/microcheck/builds/51960) passed. - [ ] The two-node E2E was not independently rerun locally because this checkout lacks compiled `ray._raylet`. The public microcheck summary does not expose individual job execution, so the PR remains draft pending CI or maintainer confirmation that it ran.", "url": "https://github.com/ray-project/ray/pull/65405", "createdAt": "2026-08-12T00:32:39Z", "updatedAt": "2026-08-13T13:35:55Z", "timestamp": "2026-08-13T13:35:55Z", "metrics": { "reactions": 1, "comments": 3 }, "labels": [ "docs", "core", "community-contribution" ], "author": "nataliekung", "state": "open", "assignees": [] }, { "id": "github:ray-project/ray:pull_request:65406", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[Data] [DO NOT MERGE] arrow-rs Parquet reader A/B β€” treatment arm (release-test trigger)", "text": "Draft PR whose only purpose is to trigger the release pipeline for the multi-node arrow-rs vs PyArrow A/B (the reader from #65117, ported onto #64985's planner). This branch is stacked on unmerged #64985, so its diff includes that PR's commits β€” do not review or merge. Baseline arm: see the companion draft PR for `arrow-rs-ab-baseline-64985`. πŸ€– Generated with [Claude Code](https://claude.com/claude-code)", "url": "https://github.com/ray-project/ray/pull/65406", "createdAt": "2026-08-12T00:47:40Z", "updatedAt": "2026-08-13T00:30:32Z", "timestamp": "2026-08-13T00:30:32Z", "metrics": { "reactions": 0, "comments": 2 }, "labels": [ "data", "release-test" ], "author": "AarryaSaraf", "state": "open", "assignees": [] }, { "id": "github:ray-project/ray:pull_request:65410", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[serve] Enforce type checking on the scheduling/routing modules", "text": "Add deployment_scheduler.py, router.py, and request_router.py to the mypy + pyrefly serve allowlists and fix their type errors. - request_router: declare the members MultiplexMixin/FIFOMixin use from the composed RequestRouter (bare annotations, Callable for methods, so they don't shadow the real methods via MRO), fix the ReplicaID/str key confusion at its sources, assert-narrow where a None would already crash, ignore the asyncio.Task `.replica` monkey-patch. - router: add an asserting `_active_request_router` property (through the lazy-loading `request_router`) for the guaranteed-non-None accesses, keeping the Optional accessor for the None-checks, narrow the singleton `_asyncio_loop`, the async-router variant returns asyncio.Future by design (override), cast the LongPoll (namespace, DeploymentID) key dicts. - deployment_scheduler: fix the ReplicaID-keyed dicts (str -> ReplicaID), import IMPLICIT_RESOURCE_PREFIX/node_labels_match_selector with one ignore, annotate the scheduling-strategy union, give the launching/running loops distinct vars.", "url": "https://github.com/ray-project/ray/pull/65410", "createdAt": "2026-08-12T02:17:22Z", "updatedAt": "2026-08-13T01:59:22Z", "timestamp": "2026-08-13T01:59:22Z", "metrics": { "reactions": 1, "comments": 1 }, "labels": [ "serve", "go" ], "author": "johntaylor-cell", "state": "open", "assignees": [ "johntaylor-cell" ] }, { "id": "github:ray-project/ray:pull_request:65417", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[dashboard] Safely skip unsupported process GPU utilization API", "text": "## Description Some third-party NVML-compatible libraries export 'nvmlDeviceGetProcessesUtilizationInfo' but terminate the calling process when the API is invoked. Because this happens in native code, Python exception handling cannot prevent the Reporter Agent from exiting. ## What changed - Detect incompatible PPU devices before calling the process utilization API. - Use strict, case-insensitive standalone 'PPU' name matching. - Skip the API per GPU index so compatible NVIDIA GPUs on mixed nodes continue reporting per-process SM utilization. - Fail closed for the current collection cycle when device detection fails, and retry detection on the next cycle. - Add 'RAY_SKIP_PROCESS_UTIL_API=true as a node-wide operator escape hatch. - Preserve process IDs, allocated GPU memory, and device-level metrics when process utilization is skipped. - Document the escape hatch and its metric impact. ## Related issues Fixes #65414 ## Additional information > Optional: Add implementation details, API changes, usage examples, screenshots, etc.", "url": "https://github.com/ray-project/ray/pull/65417", "timestamp": "2026-08-12T13:32:14Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "docs", "core", "community-contribution" ], "author": "zzchun", "assignees": [], "change": "new" }, { "id": "github:ray-project/ray:pull_request:65418", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[core][rdt] nixl: allow pool to serve tensors on a different device", "text": "## Why Fixes #64712. The `pool_eligible` check in `NixlTensorTransport` rejected pools whose device differed from the source tensors. `MemoryPoolManager.allocate_for_tensors` already handles cross-device copies via `.copy_()`, so the constraint was unnecessary. Removing it lets a CPU memory pool back GPU tensor sends, which is useful when GPU memory is scarce and long-lived streaming pools are needed. ## Changes - `nixl_tensor_transport.py`: remove `t.device == self._memory_pool.get_pool_tensor().device` from `pool_eligible`.", "url": "https://github.com/ray-project/ray/pull/65418", "timestamp": "2026-08-12T13:31:08Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "core", "community-contribution" ], "author": "BatshevaBlack", "assignees": [], "change": "new" }, { "id": "github:ray-project/ray:pull_request:65419", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "fix(autoscaler): deduplicate cloud instances during termination", "text": "Fixes #65416 ## Summary Multiple instance-manager records can refer to the same cloud instance after a Raylet restarts in place. The cloud-instance subscriber previously forwarded every record's ID to the provider, so KubeRay could decrement the same worker more than once while building a scale request and fail with an assertion before submitting the deletion. This change de-duplicates cloud instance IDs at the existing subscriber/provider boundary while preserving first-seen order. Each cloud instance is therefore requested for termination at most once per update batch. ## Tests - Added a regression test covering duplicate termination events for one cloud instance and a distinct instance. - `python -m py_compile python/ray/autoscaler/v2/instance_manager/subscribers/cloud_instance_updater.py python/ray/autoscaler/v2/tests/test_subscribers.py` - `ruff check` on the two changed files - `black --check` on the two changed files - `git diff --check` - Executed a lightweight stub regression against the changed module; duplicate IDs produced exactly `[c1, c2]`. The focused pytest module could not run in this Windows environment because Ray has no installable Windows wheel and the source tree requires generated/runtime Ray modules. The KubeRay kind/Kubernetes end-to-end test was not run because no cluster tooling is available here. AI assistance was used for issue screening, investigation, implementation, and test drafting. Per the repository contribution policy, the human submitter must review and be able to defend every changed line.", "url": "https://github.com/ray-project/ray/pull/65419", "createdAt": "2026-08-12T08:37:09Z", "updatedAt": "2026-08-12T18:22:57Z", "timestamp": "2026-08-12T18:22:57Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "core", "community-contribution" ], "author": "mikemikimike", "state": "open", "assignees": [ "martinlhw" ] }, { "id": "github:ray-project/ray:pull_request:65420", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[core] Tombstone lease ids on CancelWorkerLease", "text": "Following up on #56195 and making CancelWorkerLease truly idempotent by tombstoning lease ids. There's currently a race with RequestWorkerLease where if the RequestWorkerLease RPC arrives AFTER CancelWorkerLease RPC then we just repeatedly trigger additional CancelWorkerLease RPC requests. We've seen a couple cases where there's a large amount of CancelWorkerLease RPC requests and this PR roughly bounds the number of CancelWorkerLease RPCs to O(number of leases), and this should allow us to ignore the message reordering case completely. There is still one potential race on the other side where if for some reason the lease is granted from the perspective of the raylet, but the reply takes a while to reach the core worker for some reason, the core worker could potentially spam additional CancelWorkerLease requests for the same lease id. Leaving this as a follow up to verify if it actually matters.", "url": "https://github.com/ray-project/ray/pull/65420", "createdAt": "2026-08-12T08:56:34Z", "updatedAt": "2026-08-12T20:04:18Z", "timestamp": "2026-08-12T20:04:18Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "core", "go" ], "author": "Sparks0219", "state": "open", "assignees": [ "Yicheng-Lu-llll" ] }, { "id": "github:ray-project/ray:pull_request:65422", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[Data][LLM] Add multi-host TPU batch inference for Ray Data LLM", "text": "## Summary This PR adds multi-host TPU support to Ray Data LLM batch inference (`vLLMEngineProcessorConfig` / `build_processor`). With a TPU `accelerator_type` and a topology in `accelerator_config`, `build_processor` reserves one slice placement group for the processor, places the engine actor on bundle 0 (CPU only, `capture_child_tasks=True`), and releases the slice via `Processor.close()`. Accelerator config/backends live in `python/ray/llm/_internal/common/accelerators.py` and are shared with Serve. Batch scheduling now resolves through `AcceleratorBackend.build_batch_scheduling_options`, so the builder calls the backend unconditionally and `vLLMEngineStage.post_init` is removed. The GPU path keeps its existing behavior (same bundles, same PACK placement group per actor), just built in `GPUAccelerator` instead of the stage. ```python config = vLLMEngineProcessorConfig( model_source=\"...\", accelerator_type=\"TPU-V6E\", accelerator_config={\"kind\": \"tpu\", \"topology\": \"4x4\"}, # TP * PP * DP must equal topology chip count (PP/DP default to 1). engine_kwargs={\"tensor_parallel_size\": 16}, concurrency=1, ) with build_processor(config) as processor: out = processor(ds) out.materialize() # finish before close / leaving the with-block ``` A `chips_per_vm` field is also added for the TPU backend to support ambiguous topologies (v6e `2x4`, etc.). This field is already accepted by `SlicePlacementGroup` so I just added it to the parameters users can specify for TPU in their config. ## Behavior changes - `accelerator_type=\"CPU\"` is rejected for vLLM batch inference - `get_accelerator_backend` raises on unrecognized configs - TPU fill constraint is `tensor_parallel_size * pipeline_parallel_size * data_parallel_size == topology chips` (PP/DP default to 1). This diverges from GPU (`tp*pp` only) because the TPU slice size is fixed by topology and equality must count every device dimension. - An omitted `strategy` in `placement_group_config` resolves to `PACK`. Strategy has no practical effect on TPU slice placement β€” `slice_placement_group` derives the bundle count from capacity and label-pins bundles to the slice, so demand is exact-fit β€” but it is still passed through. - TPU slice bundles carry `accelerator_type:: 0.001`, matching Serve's `default_bundles`. This constrains single-host topologies to the requested generation and satisfies the actor's admission onto bundle 0. - `build_processor` no longer mutates the caller's `engine_kwargs`; `distributed_executor_backend` is set on a private copy. ## Testing - [x] Unit tests - [x] Integration test against a simulated multi-host v6e-16 cluster - [x] Manual test - converted a GPU example from ray docs to TPU, process in comments ## Follow-ups - Docs for TPU batch inference - Configurable `DEFAULT_PG_READY_TIMEOUT_S` β€” reservation is bounded by `head_reservation_timeout_s` (multi-host only), but the worker PG is not waited on - Multi-slice `concurrency > 1`", "url": "https://github.com/ray-project/ray/pull/65422", "createdAt": "2026-08-12T11:15:33Z", "updatedAt": "2026-08-13T13:34:53Z", "timestamp": "2026-08-13T13:34:53Z", "metrics": { "reactions": 1, "comments": 2 }, "labels": [ "data", "community-contribution" ], "author": "ryanaoleary", "state": "open", "assignees": [] }, { "id": "github:ray-project/ray:pull_request:65423", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[docs] Retire the \"Ray Pod\" term from the Kubernetes docs", "text": "## Why this change Ray on Kubernetes docs use \"Ray Pod\" 89 times, and this PR removes the term. It's the concrete counterpart to the discussion in #65239, where a reviewer asked whether the Kubernetes casing rule should capitalize Pod in \"Ray pod\". Investigating that question produced a better answer than either casing. **A Pod isn't a Ray node, it's a host that contains one.** Kubernetes defines a Pod as an \"application-specific logical host\" holding one or more containers. KubeRay indexes exactly one Ray container per Pod through `utils.RayContainerIndex`, and routinely places non-Ray containers alongside it: the autoscaler sidecar, the history-server collector, the RayJob submitter in `SidecarMode`, and the Fluent Bit sidecars that `persist-kuberay-custom-resource-logs.md` documents. The Pod is a strict superset of the Ray node, so naming it after Ray overstates what it is. It's the same category error as \"head node pod\", one level up. **\"Pod\" is already taken in these docs, by something that isn't a Kubernetes object.** A TPU Pod is a group of interconnected TPU chips. `user-guides/tpu.md` writes \"The name of the TPU Pod or slice\" and \"The TPU pod type\" three lines apart, and `ray.util.tpu.get_current_pod_name()` is a public API whose docstring reads \"Return the name of the TPU pod that the worker is a part of\". Standardizing on a capitalized `Ray Pod` would put it beside a capitalized `TPU Pod` on the same pages, distinguished only by the preceding word. **The mapping is already documented.** `user-guides/k8s-autoscaler.md` says \"each Ray node is run as a Kubernetes Pod\", so two defined terms already cover what the compound names. **There was no convention to preserve.** Across the 55 prose lines using the term, casing split four ways: | Form | Occurrences | |---|---| | Ray Pods | 20 | | Ray pods | 16 | | Ray pod | 16 | | Ray Pod | 15 | Meanwhile \"head Pod\" and \"worker Pod\" already outnumber it 284 to 86 under `doc/source/cluster/kubernetes/`, so the replacement vocabulary is the one these docs mostly use. ## What changed 83 lines across 31 files. The term becomes \"Pod\" where context is unambiguous, which is most of them. Where a bare \"Pod\" would lose meaning I used an explicit form instead: - Headings, which have no surrounding context: \"Check logs of the head and worker Pods\", \"Verify the Ray cluster's Pods and PodGroup\", \"Configuring Kubernetes Pods for TPU usage\". - Pages where other kinds of Pod are in scope: \"the Ray cluster's Pods\" in `network-policy.md`, `prometheus-grafana.md`, `k8s-events.md`, and `scheduler-plugins.md`. - `rayservice-high-availability.md` listed \"Ray Pod: This Pod sends requests to the RayService\". That Pod is a client that isn't part of the RayCluster, and it's named `ray-pod` in the sample manifest, so it's now listed by its actual object name. ## What replaces it on the orientation pages Removing the compound takes away the handle a Ray-first reader was using for the node-and-Pod relationship, so the relationship now gets stated where readers form their mental model. It was only in `user-guides/k8s-autoscaler.md`, whose title begins \"(Advanced)\", and absent from `index.md`, `getting-started.md`, and `user-guides/config.md`. One sentence each on the two orientation pages: - `index.md`: \"KubeRay runs each Ray node as a Kubernetes Pod, so each Ray cluster consists of a head Pod and a collection of worker Pods.\" - `getting-started.md`: \"It runs each Ray node as a Kubernetes Pod, so a Ray cluster's head node is its head Pod and its worker nodes are its worker Pods.\" `getting-started/raycluster-quick-start.md` is the natural third location, since line 53 is where the two vocabularies genuinely first meet for a reader following the quickstart. It's left alone deliberately: that paragraph writes \"pods\" lowercase throughout, and adding a capitalized sentence into it would either read inconsistently or drag the casing sweep into this PR. It belongs to whoever runs that pass. The rules behind both changes are in #65239, which now carries a \"State the node-to-Pod mapping once, early\" section and a \"Keep Pods out of Ray-general pages\" section. The second one is why this PR touches `ray-core/scheduling/placement-group.rst` and `serve/production-guide/fault-tolerance.md`. ## What this deliberately doesn't touch - **`Ray head Pod` and `Ray worker Pod` stay.** Those qualify by role, which is the form these docs already prefer and the one Google's Ray on GKE docs use. Only the unqualified compound goes. - **No casing sweep.** Lowercase \"head pod\" occurrences are left alone. That's the Vale rule in #65239, not this PR. - **`python/` code comments and docstrings keep the term.** About 10 occurrences under `python/ray/autoscaler`, `python/ray/dashboard`, and the KubeRay tests identify which Pods the operator manages, which is reasonable for a code comment. This PR is scoped to documentation prose. - **The `kuberay` repo is out of scope.** Its authored prose has a handful of uses and its `docs/reference/api.md` is generated from Go doc comments. That's a maintainer call for that repo, and I'll raise it there separately rather than assume the same decision carries. ## Two fixes on lines this change already rewrites - `k8s-autoscaler.md` linked to `.../horizontal-Pod-autoscale/`, which returns **404**. The correct URL is all-lowercase and returns 200. An earlier Pod-casing sweep appears to have capitalized inside the URL. - `index.md` described a Ray cluster as \"a head node pod and a collection of worker node pods\", stacking both vocabularies in one phrase. Now \"a head Pod and a collection of worker Pods\". - `index.md` also wrote \"heterogenous compute nodes\" on that same line. Now \"heterogeneous\". ## One casing fix folded in `k8s-ecosystem/kai-scheduler.md` wrote \"KAI scheduler deployment\" in one place and \"KAI Scheduler\" everywhere else. The KubeRay Vale vocabulary proposed in #65239 surfaced it, and @win5923 confirmed the two-word title-case form on that PR. Folded in here rather than opened as a one-line PR, per the guidance in `AGENTS.md` about standalone trivial changes. The other hyphenated spellings in that file stay as they are. They sit inside links to `NVIDIA/KAI-Scheduler`, the Helm release name, and the `` placeholder, so none of them are prose. ## Testing ``` pre-commit run --files $(git diff --name-only) ``` Every hook reports \"no files to check\". Stating that plainly rather than implying coverage it doesn't give: no pre-commit hook applies to Markdown or reStructuredText under `doc/source/`, and the `vale` hook is scoped to `^doc/source/data/.*`, so none of these files are linted in CI today. What I verified by hand instead: - **No residual term.** `grep -rIniE '\\bray [*_`]* ?pods?\\b' .` over the whole repo returns only the `python/` code comments listed above as intentionally out of scope. This includes a markup-split case, \"Ray \\*\\*Pod quantities\\*\\*\", that a naive search misses. - **reStructuredText heading integrity.** `gpu.rst` has a section whose title shortened, so its underline was resized to match. Title and underline are both 30 characters. - **No broken anchors.** Four headings changed text. Searched for inbound references to their generated anchors and found none, and none carries an explicit MyST anchor. - **The URL fix.** `curl` returns 404 for the old casing and 200 for the new one. - **Config comments included.** `configs/static-ray-cluster.tls.yaml` carried the term in two YAML comments mirroring `config.md` prose, so the removal is complete rather than Markdown-only. Rendering is verified by the Read the Docs preview build on this PR. No code, no API, no behavior change. ## Not a duplicate ``` gh pr list --repo ray-project/ray --state open --search \"Ray Pod\" gh pr list --repo ray-project/ray --state open --search \"pod terminology\" ``` No open PR proposes this change. #65239 is the style-guide PR where the question came up and is the discussion surface for the rule; this PR is the cleanup and stands on its own against master. Two of my open PRs touch nearby files: #65287 rewrites lines in `jax-tpu-profiling.md`, which this PR doesn't touch, and #65263 is a draft adding `scheduling.md`, which introduces one new use of the term that I'll fix there rather than here. Per `AGENTS.md`, mechanical cleanups need maintainer coordination. That's the thread on #65239 with @win5923, who reviewed and approved the style-guide change and raised the question this PR answers. ## AI assistance Written with AI assistance (Claude). I've reviewed every changed line, generated and inspected the counts myself, and can defend the change end to end. --- ### Update: four casing defects folded in (f9fc8ea) Extending the KubeRay Vale vocabulary in #65239 turned four latent casing errors into alerts. They belong with the rest of the terminology cleanup rather than in a PR of their own. - `configuring-ippr.md`, two uses: \"Kubelet\" to \"kubelet\". Kubernetes writes the component lowercase in prose. - `rayservice-incremental-upgrade.md`: \"Install istio\" to \"Install Istio\" in a step label, where the word names the project. The lowercase `istio` elsewhere on that page is correct and stays, since it's a GatewayClass `metadata.name` and a `controllerName` value rather than the project name. - `resource-isolation-with-writable-cgroups.md`: the H1 read \"Resource Isolation with Writable Cgroups on Google Kubernetes Engine (GKE)\". Lowercasing only \"Cgroups\" would leave a title-case heading with one lowercase word, so the whole heading moves to sentence case, which the style guide requires anyway. **One coupling worth flagging for review.** The explicit MyST label above that H1 is unchanged, so every `{ref}` to the page still resolves, and the bare `{ref}` in `cluster/kubernetes/user-guides.md` picks up the new title automatically. But `ray-core/resource-isolation-with-cgroupv2.rst` passes custom link text that mirrored the old title case, so that text moves to sentence case too. That's the one file this PR touches outside `doc/source/cluster/kubernetes/`, and it's a one-line consistency fix caused directly by the heading change. Verified against the #65239 ruleset by diffing the alert set for these files before and after the change: the four alerts clear and no new alert appears. AI assistance was used for this commit and for that verification. ### Update: a bad replacement in this PR, plus three defects in the same files (89b0e49) **This PR had a bug.** The term replacement turned \"KubeRay pods\" into \"KubePods\" in `tpu.md`: the pattern matched \"Ray pods\" inside \"KubeRay pods\" and left the \"Kube\" prefix glued to the replacement. `git log -S` traces it to the retirement commit, so it isn't pre-existing. It's now \"Pods\", which is what the rule asks for when the qualifier only names the framework. I swept every added line in this PR for the same signature, a lowercase letter glued to \"Pod\" or \"Pods\", and this was the only instance. The other repo-wide matches are Kubernetes event reasons such as `CreatedHeadPod`, correct as literal strings. Three more defects, in files this PR already edits: - `tpu.md`: \"GKE nodepool\" to \"GKE node pool\" in prose. The `az aks nodepool` spelling elsewhere in the docs is a CLI subcommand and stays. - `prometheus-grafana.md`: \"spawned by replicaset controller\" to \"spawned by a ReplicaSet controller\", fixing the API object casing and a missing article. - `rayservice-incremental-upgrade.md`: \"utilizing\" to \"using\", \"like\" to \"such as\" for an example, and \"(e.g. `istio`)\" to \"for example `istio`\", plus the imperative in place of \"You will need to\". Verified against the #65239 ruleset with a column-independent diff of the alert set for these files: five alerts clear, no new finding. The two passive-voice alerts that change column position are pre-existing and unchanged in substance. **Deliberately not in this PR.** These sit in files this PR doesn't otherwise touch, so folding them in would widen its file scope for unrelated reasons: \"Deepseek\" for DeepSeek in `examples/rayserve-deepseek-example.md` (an H1) and `serve/llm/architecture/overview.md`; `gsutil` in `examples/distributed-checkpointing-with-gcsfuse.md`, where `gcloud storage` supersedes it and swapping a documented command needs someone who can verify the equivalent invocation; and \"nodepool\" in the prose of `user-guides/azure-aks-gpu-cluster.md`. AI assistance was used for this commit and for the verification above.", "url": "https://github.com/ray-project/ray/pull/65423", "createdAt": "2026-08-12T13:14:06Z", "updatedAt": "2026-08-13T12:25:43Z", "timestamp": "2026-08-13T12:25:43Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "docs", "core", "go", "docs-go" ], "author": "dstrodtman", "state": "open", "assignees": [] }, { "id": "github:ray-project/ray:pull_request:65424", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[core] Enable TCP keepalive on GCS<->Redis connections", "text": "## Description Idle GCS↔Redis flows get silently removed by NAT, proxies, load balancers, and managed-service gateways. The GCS only finds out on its next command, which then hangs until a timeout instead of failing fast β€” and on a managed Redis behind an LB with a 60–350s idle timeout, this is a routine event, not an edge case. This PR enables TCP keepalive on every external Redis socket, applied from a single place in `ConnectWithoutRetries` so that sync, async, Sentinel, Cluster-redirect, and cleanup connections all get the same policy before TLS, AUTH, or any command runs. Two knobs, deliberately decoupled: | Config | Default | Purpose | | --- | --- | --- | | `redis_tcp_keepalive_interval_seconds` | `30` | Keeps an idle flow alive. Must stay **below** the shortest idle timeout on the network path. `0` disables keepalive entirely. | | `redis_tcp_keepalive_probes` | `9` | How many unanswered probes declare the connection dead, so detection β‰ˆ `interval * (1 + probes)`. | These two need opposite tuning β€” probes must be frequent enough to keep a flow alive, while declaring a connection dead must be slow enough to ride out transient congestion. hiredis ties them together (`TCP_KEEPINTVL = interval/3`, `TCP_KEEPCNT = 3`, so dead at `2 * interval` β‰ˆ 60s), so this PR overrides both on glibc Linux to get ~5 minutes instead. That margin matters because in-place reconnect has not landed yet: a connection declared dead escalates to a GCS crash once the request retry budget is exhausted (`RedisRequestContext::Run` β†’ `RAY_LOG(FATAL)`, and the periodic health check's `RAY_CHECK_OK` in `gcs_server.cc`). The docs carry an explicit warning about this, and the default is chosen conservatively for the same reason. **Platform support.** Only glibc Linux honors the probe count, because hiredis guards `TCP_KEEPIDLE`/`TCP_KEEPINTVL`/`TCP_KEEPCNT` on `__GLIBC__`. On musl only `SO_KEEPALIVE` is applied and kernel defaults govern the timing; macOS applies the interval alone; Windows uses a system-fixed probe count. The code, the runtime log, the config comments, the docs, and the test guards all agree on this β€” the log prints the *effective* values, not the configured ones, so a post-mortem shows which applied. ## Related issues Related to #65298 (`[core][2/N] Reconnect to Redis in place when the connection drops`). **This is not a duplicate.** I checked the open PR list before starting: - `gh pr list --repo ray-project/ray --state open --search \"redis keepalive\"` β†’ no results - `gh pr list --repo ray-project/ray --state open --search \"tcp keepalive in:title,body\"` β†’ only unrelated dependabot PRs - `gh pr diff 65298 | grep -iE \"keepalive|SO_KEEPALIVE|KEEPIDLE\"` β†’ **zero hits** #65298 is *recovery* (reconnect once a connection is known dead). This PR is *detection and prevention* (keep the flow alive; find out promptly when it isn't). They are complementary, and this PR does not implement or pre-empt any part of #65298. **Coordination note for reviewers:** #65298 is open and touches four of the same files (`redis_context.cc`, `redis_context.h`, `ray_config_def.h`, and `kuberay-gcs-ft.md`), so whichever lands second will need a rebase. Two specific follow-ups once #65298 merges: 1. The `{warning}` block added here (\"Ray does not yet reconnect to Redis in place… the GCS process exits\") becomes false and must be rewritten. 2. The `redis_tcp_keepalive_probes` default of 9 is conservative *because* a dead connection currently means a crash. With reconnect in place, a shorter detection window becomes safe and probably desirable. Happy to rebase on top of #65298 instead, or to hold this until it merges β€” reviewer's call. Builds on #64299 (`[core][1/N] Make RedisContext::Connect non-fatal on connection failure`), already merged: this PR relies on `Connect()` returning a `Status`, and extends the same contract by making configuration errors non-retryable.", "url": "https://github.com/ray-project/ray/pull/65424", "createdAt": "2026-08-12T13:52:19Z", "updatedAt": "2026-08-13T08:14:18Z", "timestamp": "2026-08-13T08:14:18Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "docs", "core", "community-contribution" ], "author": "zzchun", "state": "open", "assignees": [] }, { "id": "github:ray-project/ray:pull_request:65425", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[CI] Clear whitespace debt in doc/source/data and drop the pre-commit carve-out", "text": "## Why Stacked on #65375, and should merge after it. #65375 fixes a pre-commit `exclude` bug that had kept the `vale` hook from matching any files. Narrowing that exclude so Vale can see `doc/source/data` prose also exposed those files to `trailing-whitespace` and `end-of-file-fixer` for the first time. Rather than make a config-only change rewrite 17 unrelated prose files, #65375 exempted the two hooks from the directory and deferred the cleanup. This is that cleanup. ## What 1. Removes both `exclude: ^doc/source/data/` lines that #65375 added, along with the comments explaining them. 2. Runs `trailing-whitespace` and `end-of-file-fixer` over the 48 Markdown and rST files under `doc/source/data`. That rewrites 17 of them: trailing spaces trimmed, missing final newlines added. No content changes. `git diff --ignore-all-space --ignore-blank-lines -- doc/source/data` is empty, so the whole prose diff is whitespace. Both hooks now pass `--all-files`. ## Not in scope The `semgrep` exemption on the same directory stays. It covers 8 pre-existing `code-block-python` violations, which want a real fix rather than a whitespace pass. ## Sequencing Draft until #65375 merges. **Reviewers: the diff currently includes #65375's two commits.** This branch is built on top of that PR's branch, which lives on a fork, so GitHub can't take it as the base ref and the shared history shows up here. Once #65375 lands, a rebase drops those commits and this becomes the 18-file whitespace change described above. Until then, the commit to review here is the last one. Two open PRs touch files in this diff and will need a rebase here once they land. Resolution is mechanical: rebase, re-run the two hooks, amend. - #65372 touches `aggregating-data.rst`, `saving-data.rst`, and `data-internals.rst`. - #65374 touches `data-internals.rst`. ## Reviewing this A pure-whitespace diff is only reviewable when nothing else is in it, which is the point of keeping it as its own PR. `git diff -w` showing nothing is the check that matters.", "url": "https://github.com/ray-project/ray/pull/65425", "createdAt": "2026-08-12T14:21:20Z", "updatedAt": "2026-08-12T14:21:20Z", "timestamp": "2026-08-12T14:21:20Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [], "author": "dstrodtman", "state": "open", "assignees": [], "change": "updated" }, { "id": "github:ray-project/ray:pull_request:65426", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[docs] vendor the KubeRay CRD API reference into the Ray docs", "text": "## Why are these changes needed? The KubeRay CRD API reference β€” the field-level reference for `RayCluster`, `RayJob`, `RayService`, and `RayCronJob` β€” is published only on the MkDocs site at `ray-project.github.io/kuberay`. That site has two problems that compound each other: - Its `mkdocs.yml` `nav:` block contains only `index.md`. The API reference builds and serves but appears nowhere in site navigation, so it's reachable only by direct URL, site search, or a search engine. - Its landing page states that documentation has moved to `ray-project/ray`. Meanwhile `doc/source/cluster/kubernetes/references.md` here is a ~14-line stub whose entire substance is an outbound link to that page. So Ray's documented answer for \"what fields does a RayCluster support?\" is a link off-site to a nav-hidden page on a site that says it holds no documentation. It's also the most actively updated file in the KubeRay `docs/` tree β€” recent CRD features (`AuthOptions`, `GcsEmbeddedStorage`, `NetworkPolicyRules`, `IngressOptions`, `HistoryServerOptions`, `RayCronJob`) all extend it. This vendors the generated reference into the Ray docs and repoints the inbound links at it. ## What changed | File | Change | | --- | --- | | `doc/source/cluster/kubernetes/references/api.md` | **New.** Vendored generated reference, `ray.io/v1` only. | | `doc/source/cluster/kubernetes/references.md` | Toctree entry; the outbound link becomes a `{ref}`; added a note that v1alpha1 is deprecated. | | `doc/source/cluster/kubernetes/getting-started/rayjob-quick-start.md` | Outbound link β†’ `{ref}`. | | `doc/source/cluster/kubernetes/user-guides/gcp-gke-tpu-cluster.md` | Outbound link β†’ the operator installation page (see below). | | `.github/CODEOWNERS` | Entry for the generated file. | ### How the source artifact is produced `docs/reference/api.md` in `ray-project/kuberay` is a committed build artifact, not a hand-written page. `make api-docs` runs `elastic/crd-ref-docs` (pinned v0.2.0) over the Go CRD types in `ray-operator/apis/ray/`. The `ray-operator-verify-api-docs` job in `.github/workflows/consistency-check.yaml` regenerates and diffs it on every push and PR to `master` and `release-*`, failing on drift. That CI job is what makes a vendored copy viable: freshness is already enforced upstream, so the copy has a single well-defined source of truth. The file carries a `GENERATED FILE -- DO NOT EDIT` header naming the upstream commit it came from, with a permalink. ### Why `ray.io/v1` only The upstream artifact documents both `ray.io/v1` and `ray.io/v1alpha1`. This copy carries only v1, for two reasons. **v1alpha1 is deprecated and on a removal runway.** It hasn't been the storage version since v1.0 (October 2023), receives no new fields, and ray-project/kuberay#5124 tracks its removal. Every CRD feature added since December 2023 is v1-only. **Carrying it would import a silent correctness bug.** `crd-ref-docs` derives anchors from the type name alone, and eleven type names are defined in both API versions (`RayClusterSpec`, `WorkerGroupSpec`, `RayJob`, `RayService`, `AutoscalerOptions`, ...). Only the v1 heading owns the slug, so every v1alpha1 cross-link resolves to the v1 definition β€” with no warning at generation time. The definitions differ substantially (`RayClusterSpec` has 14 fields in v1, 7 in v1alpha1), so a reader following a v1alpha1 link lands on a table advertising fields that don't exist in that version. Filed upstream as ray-project/kuberay#5091. Dropping the section resolves the collision rather than papering over it. The stripping step is written to become a silent no-op once upstream drops v1alpha1 from generation. ### The gcp-gke-tpu-cluster.md link This one pointed at the MkDocs `/deploy/helm/` page, not the API reference. That page documents Helm *chart values*, while the sentence introducing it is about *installing* KubeRay. It's repointed at `kuberay-operator-installation.md`, which is the maintained page for what the sentence actually promises. The `helm repo add kuberay https://ray-project.github.io/kuberay-helm/` commands elsewhere in the docs are untouched β€” that's the Helm chart repository, a different host from the docs site. ## Checks **Anchor resolution.** The vendored page has 46 h1–h4 headings producing 46 distinct slugs (zero collisions, confirming the v1-only decision eliminates that class) and 93 in-page anchor links across 43 distinct targets, all resolving. **Build.** Built the page in a minimal Sphinx project against the pinned docs versions β€” sphinx 8.2.3, myst-parser 5.1.0 β€” with `myst_heading_anchors = 4`, `nitpicky = True`, and `-W --keep-going`: ``` build succeeded. exit 0 ``` Zero warnings. Inspected the rendered HTML: all 46 internal `href`s have a matching `id`, and no link resolves to a docutils fallback `idN`. This page depends on #65240 (h4 heading anchors), which has merged. `crd-ref-docs` renders every CRD type as an h4 and cross-links types by anchor, so the page could not build under the previous `myst_heading_anchors = 3`. A full `make rtd` build was not run locally; the docs CI build on this PR is the check for that. ## Follow-ups (not in this PR) - A sync job to keep the vendored copy current. It diffs the processed upstream output against the copy committed here β€” not against the previous upstream version β€” so it's idempotent and self-healing: an unmerged sync PR gets updated rather than duplicated, and a hand-edit here gets corrected on the next run. - Several other pages on the MkDocs site have no equivalent in the Ray docs β€” the Helm chart values tables, additional `rayStartParams` coverage (`--no-monitor` and the autoscaler READONLY rationale, `--block`, `--address` FQDN construction), and the image tagging and registry scheme. Being handled separately. ## Related issue number Depends on #65240 (merged). Related upstream: ray-project/kuberay#5091, ray-project/kuberay#5124. ## AI assistance AI assistance (Claude Code) was used for the anchor-collision analysis, the build verification, the post-processing script, and drafting this description. I reviewed the vendored content and the link changes and can defend both.", "url": "https://github.com/ray-project/ray/pull/65426", "createdAt": "2026-08-12T14:46:49Z", "updatedAt": "2026-08-12T14:55:01Z", "timestamp": "2026-08-12T14:55:01Z", "metrics": { "reactions": 1, "comments": 1 }, "labels": [], "author": "dstrodtman", "state": "closed", "assignees": [], "change": "updated" }, { "id": "github:ray-project/ray:pull_request:65428", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[docs] vendor the KubeRay CRD API reference into the Ray docs", "text": "## Why are these changes needed? The KubeRay CRD API reference β€” the field-level reference for `RayCluster`, `RayJob`, `RayService`, and `RayCronJob` β€” is published only on the MkDocs site at `ray-project.github.io/kuberay`. That site has two problems that compound each other: - Its `mkdocs.yml` `nav:` block contains only `index.md`. The API reference builds and serves but appears nowhere in site navigation, so it's reachable only by direct URL, site search, or a search engine. - Its landing page states that documentation has moved to `ray-project/ray`. Meanwhile `doc/source/cluster/kubernetes/references.md` here is a ~14-line stub whose entire substance is an outbound link to that page. So Ray's documented answer for \"what fields does a RayCluster support?\" is a link off-site to a nav-hidden page on a site that says it holds no documentation. It's also the most actively updated file in the KubeRay `docs/` tree β€” recent CRD features (`AuthOptions`, `GcsEmbeddedStorage`, `NetworkPolicyRules`, `IngressOptions`, `HistoryServerOptions`, `RayCronJob`) all extend it. This vendors the generated reference into the Ray docs and repoints the inbound links at it. ## What changed | File | Change | | --- | --- | | `doc/source/cluster/kubernetes/references/api.md` | **New.** Vendored generated reference, `ray.io/v1` only, from the `v1.6.2` release tag. | | `doc/source/cluster/kubernetes/references.md` | Toctree entry; the outbound link becomes a `{ref}`; added a note that v1alpha1 is deprecated. | | `doc/source/cluster/kubernetes/getting-started/rayjob-quick-start.md` | Outbound link β†’ `{ref}`. | | `doc/source/cluster/kubernetes/user-guides/gcp-gke-tpu-cluster.md` | Outbound link β†’ the operator installation page (see below). | Content-only: every changed file is under `doc/`, so the `docs-go` label's scope guard (`ci/lint/validate_docs_go_scope.sh`) passes. An earlier revision of this PR also added a `.github/CODEOWNERS` entry for the generated file; it's dropped, both because it would have failed that guard and because the existing `/doc/source/cluster/kubernetes/` rule already assigns `@andrewsykim @ray-project/ray-core @ray-project/ray-docs` β€” a more specific rule would have *narrowed* ownership by dropping ray-core. The generated-file header is what deters hand-edits. ### How the source artifact is produced `docs/reference/api.md` in `ray-project/kuberay` is a committed build artifact, not a hand-written page. `make api-docs` runs `elastic/crd-ref-docs` (pinned v0.2.0) over the Go CRD types in `ray-operator/apis/ray/`. The `ray-operator-verify-api-docs` job in `.github/workflows/consistency-check.yaml` regenerates and diffs it on every push and PR to `master` **and `release-*`**, failing on drift. That CI job is what makes a vendored copy viable: freshness is already enforced upstream, so the copy has a single well-defined source of truth. Because the job covers `release-*` too, a release tag's `api.md` is equally guaranteed consistent with that branch's CRD types. ### Why the `v1.6.2` release tag, not `master` Per @andrewsykim's review, this now vendors from the latest KubeRay release rather than from `master`. `master` carries CRD surface that no release ships β€” diffing `docs/reference/api.md` at `v1.6.2` against `master` is **+247 lines**, adding twelve types that exist in no release: `CollectorOptions`, `GcsEmbeddedStorage`, `GCSStorageDeletionPolicy`, `GcsFaultToleranceBackend`, `HistoryServerOptions`, `IngressOptions`, `IngressPathType`, `NetworkPolicyConfig`, `NetworkPolicyMode`, `NetworkPolicyRules`, `TLSOptions`, `WorkerGroupNetworkPolicyRules` plus field-level additions such as `priority`, `suspend`, `timeZone`, and `command`/`args`. The source ref is the **tag**, not the release branch head: `release-1.6` is currently one commit ahead of `v1.6.2`. The page body now states the KubeRay version, so the fields a reader sees are attributable to a version they can install; the generated-file comment records the tag and its commit SHA. ### Why `ray.io/v1` only The upstream artifact documents both `ray.io/v1` and `ray.io/v1alpha1`. This copy carries only v1, for two reasons. **v1alpha1 is deprecated and on a removal runway.** It hasn't been the storage version since v1.0 (October 2023), receives no new fields, and ray-project/kuberay#5124 tracks its removal. Every CRD feature added since December 2023 is v1-only. **Carrying it would import a silent correctness bug.** `crd-ref-docs` derives anchors from the type name alone, and eleven type names are defined in both API versions (`RayClusterSpec`, `WorkerGroupSpec`, `RayJob`, `RayService`, `AutoscalerOptions`, ...). Only the v1 heading owns the slug, so every v1alpha1 cross-link resolves to the v1 definition β€” with no warning at generation time. The definitions differ substantially (`RayClusterSpec` has 14 fields in v1, 7 in v1alpha1), so a reader following a v1alpha1 link lands on a table advertising fields that don't exist in that version. Filed upstream as ray-project/kuberay#5091. Dropping the section resolves the collision rather than papering over it. The stripping step is written to become a silent no-op once upstream drops v1alpha1 from generation. ### The gcp-gke-tpu-cluster.md link This one pointed at the MkDocs `/deploy/helm/` page, not the API reference. That page documents Helm *chart values*, while the sentence introducing it is about *installing* KubeRay. It's repointed at `kuberay-operator-installation.md`, which is the maintained page for what the sentence actually promises. The `helm repo add kuberay https://ray-project.github.io/kuberay-helm/` commands elsewhere in the docs are untouched β€” that's the Helm chart repository, a different host from the docs site. ## Checks **Anchor resolution.** The vendored page has 34 h1–h4 headings producing 34 distinct slugs (zero collisions, confirming the v1-only decision eliminates that class) and 67 in-page anchor links across 31 distinct targets, all resolving. **Build.** Built the page in a minimal Sphinx project against the pinned docs versions β€” sphinx 8.2.3, myst-parser 5.1.0 β€” with `myst_heading_anchors = 4`, `nitpicky = True`, and `-W --keep-going`: ``` build succeeded. exit 0 ``` Zero warnings. Inspected the rendered HTML: all 134 internal `href`s have a matching `id`, and none resolves to a docutils fallback `idN`. This page depends on #65240 (h4 heading anchors), which has merged. `crd-ref-docs` renders every CRD type as an h4 and cross-links types by anchor, so the page could not build under the previous `myst_heading_anchors = 3`. A full `make rtd` build was not run locally; the docs CI build on this PR is the check for that. ## Follow-ups (not in this PR) **The sync job.** It diffs the processed upstream output against the copy committed here β€” not against the previous upstream version β€” so it's idempotent and self-healing: an unmerged sync PR gets updated rather than duplicated, and a hand-edit here gets corrected on the next run. Two open design points, discussed in the review thread: - *What triggers it.* The semantically correct signal is the KubeRay `release` event with `types: [released]`, which fires only for non-prerelease publishes (every `-rc.N` tag is flagged `prerelease: true`). A `push: tags` trigger would fire at Step 3 of the release process, before images, charts, and validation. Until the workflow can live in a repo that receives KubeRay events, the equivalent is a scheduled poll of `GET /repos/ray-project/kuberay/releases/latest`, which returns the maintainer-designated latest stable and so isn't fooled by interleaved release lines (`v1.5.2` and `v1.6.1` published 22 seconds apart). - *Whether Ray master should also carry a dev reference* built from KubeRay `master`, alongside the stable one. Open question for the maintainers. **Remaining MkDocs-only content.** Several other pages on the MkDocs site have no equivalent in the Ray docs β€” the Helm chart values tables, additional `rayStartParams` coverage (`--no-monitor` and the autoscaler READONLY rationale, `--block`, `--address` FQDN construction), and the image tagging and registry scheme. Being handled separately. ## Related issue number Depends on #65240 (merged). Related upstream: ray-project/kuberay#5091, ray-project/kuberay#5124. ## AI assistance AI assistance (Claude Code) was used for the anchor-collision analysis, the release-mechanics analysis, the build verification, the post-processing script, and drafting this description. I reviewed the vendored content and the link changes and can defend both.", "url": "https://github.com/ray-project/ray/pull/65428", "createdAt": "2026-08-12T14:54:47Z", "updatedAt": "2026-08-12T20:40:21Z", "timestamp": "2026-08-12T20:40:21Z", "metrics": { "reactions": 1, "comments": 6 }, "labels": [ "docs", "go", "docs-go" ], "author": "dstrodtman", "state": "open", "assignees": [] }, { "id": "github:ray-project/ray:pull_request:65429", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[doc] Pin the RocksDB GCS example to the release that introduced it", "text": "## Why The embedded RocksDB GCS backend page told readers the feature wasn't in a Ray release yet and pointed its manifest at `rayproject/ray:nightly`. The Ray half of that is no longer true: the backend shipped in **2.57.0** (`_is_rocksdb_gcs` is present in `ray-2.57.0`, absent in `ray-2.56.1`), and `rayproject/ray:2.57.0` resolves on Docker Hub. Beyond the expired claim, a floating tag isn't the right way to flag a preview feature. Docs land on `master` immediately but aren't published until they ship to `latest`, which tracks a Ray release, so by the time a reader sees this page the feature is in a release by construction. Naming the release that introduced it is both accurate and more useful than a nightly build. ## What changed - Pin both container images to `rayproject/ray:2.57.0` instead of `nightly`. - Replace the \"isn't in a stable Ray release yet\" prerequisite with the concrete floor: Ray 2.57.0 or later. - State the KubeRay prerequisite as KubeRay v1.7 or later, per @andrewsykim's review. `GCSFaultToleranceEmbeddedStorage` is on KubeRay master and absent from `v1.6.2`, the current release, so v1.7 is the release that ships the gate. The alpha admonition stays. The feature is alpha and gated off by default, which is independent of which release contains it. ## Verification - `_is_rocksdb_gcs` present in `git show ray-2.57.0:python/ray/_private/node.py`, absent in `ray-2.56.1`. - `rayproject/ray:2.57.0` returns 200 from the Docker Hub tag API. - `GCSFaultToleranceEmbeddedStorage` present in `ray-operator/pkg/features/features.go` on kuberay master, absent at tag `v1.6.2`.", "url": "https://github.com/ray-project/ray/pull/65429", "createdAt": "2026-08-12T15:37:58Z", "updatedAt": "2026-08-12T19:24:14Z", "timestamp": "2026-08-12T19:24:14Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "docs", "core", "go", "docs-go" ], "author": "dstrodtman", "state": "open", "assignees": [] }, { "id": "github:ray-project/ray:pull_request:65430", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[doc] Correct the fake multi-node docker image guidance", "text": "## Why Three claims in this paragraph are stale or wrong: 1. **The default image is misstated.** The doc says `rayproject/ray:nightly`. The code says `DEFAULT_DOCKER_IMAGE = \"rayproject/ray:nightly-py{major}{minor}-cpu\"` (`python/ray/autoscaler/_private/fake_multi_node/test_utils.py:26`), formatted with the interpreter's own version at `test_utils.py:195`. 2. **`rayproject/ray:nightly-py38-cpu` is frozen.** Docker Hub last pushed it on 2024-03-13, when py38 support was dropped. It still returns 200, so a contributor who copies it gets a two-year-old image with no error. 3. **The Buildkite image it points at doesn't exist.** `rayproject/ray:multinode-py38` 404s on Docker Hub, and no step in `.buildkite/`, `ci/`, or `docker/` builds it. The paragraph tells contributors to set `RAY_DOCKER_IMAGE` to an image nothing produces. ## What changed - State the actual default, `rayproject/ray:nightly-py-cpu`, and why nightly is the right default here: this provider tests the autoscaler on master, not on a release. - Drop the dead Buildkite/`multinode-py38` instructions and keep the reusable advice they were wrapping: bring an image with `openssh-server` already installed and set `RAY_HAS_SSH=1`, otherwise every node installs the package on startup (`node_provider.py:148-155`). `RAY_DOCKER_IMAGE`, `RAY_HAS_SSH`, `RAY_TEMPDIR`, and `RAY_TESTHOST` are all still read by the provider, so the surrounding guidance stands as written.", "url": "https://github.com/ray-project/ray/pull/65430", "createdAt": "2026-08-12T15:39:02Z", "updatedAt": "2026-08-12T19:23:24Z", "timestamp": "2026-08-12T19:23:24Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "docs", "core", "go", "docs-go" ], "author": "dstrodtman", "state": "open", "assignees": [] }, { "id": "github:ray-project/ray:pull_request:65431", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[ci] Scope the doc_readme lint check to the files it actually checks", "text": "## Why this change is needed `lint: doc_readme` carries the `always` tag, so it runs on every pull request, including documentation-prose-only ones. Its inputs are the repo-root `README.rst` and `python/setup.py`. Nothing under `doc/` feeds it, so a prose change cannot change its result. What it actually checks, from `ci/lint/lint.sh`: ```bash doc_readme() { /usr/bin/python -m pip install -c python/requirements_compiled.txt docutils cd python && /usr/bin/python setup.py check --restructuredtext --strict --metadata } ``` `--restructuredtext` validates `long_description`, which `python/setup.py` reads from the repo-root `README.rst`. `--metadata` validates `setup.py`'s own metadata fields. So the check guards how Ray's PyPI long description renders. It's a useful gate and this PR doesn't touch what it does; it's pointed at the wrong trigger. It ended up in the `lint-small-prose` matrix, which exists for the opposite reason: that matrix holds the checks a Markdown or reStructuredText change *can* fail, so they survive the prose filter that strips the code-oriented `lint-small` matrix. The name `doc_readme` reads like documentation. The other three steps in that matrix belong there; this one is packaging. ## What this PR does 1. Moves `doc_readme` into its own step tagged `pypi_readme` alone, with no `lint` and no `always`. 2. Adds a rule in `.buildkite/always.rules.txt` emitting `pypi_readme` for `README.rst` and `python/setup.py`. Both files keep `always lint`, which the catch-all would otherwise have given them, so no other step's trigger narrows. 3. Adds test cases to `.buildkite/always.rules.test.txt` for both inputs and for neighbors that must not emit the tag. 4. Rewrites the matrix comment to state the rule rather than imply it: adding a step to the prose matrix asserts a prose-only diff can fail it. Prose pull requests and code pull requests that leave both files alone both stop selecting the step. The second half of that is broader than strictly necessary to fix the prose case, and it's the point: a check that cannot fail from your change shouldn't be in your PR's status list. ## Two consequences worth flagging **Microcheck eligibility.** Dropping `lint` from the step means the microcheck pipeline's hardcoded `tag:lint` selector no longer makes it eligible there, so it runs on premerge only. Selects and rule-emitted tags are ANDed in rayci's step filter, so this is the expected outcome rather than a surprise. Microcheck is non-blocking; premerge is the gate this check has to hold. Noted in the step comment. **Step IDs shift within the lint group.** Step IDs are positional (`fmt.Sprintf(\"g%d_s%d\", i, j)`), so inserting a step moves `api-param-coverage` from `g11_s3` to `g11_s4` and `docs-go-scope` from `g11_s4` to `g11_s5`. `Test.gen_microcheck_step_ids` derives step IDs from recent Bazel test results, and group `g11` contains no Bazel test steps, so nothing keyed on step ID is affected. ## Not a duplicate No open PR touches `doc_readme` (`gh pr list --repo ray-project/ray --state open --search doc_readme` returns nothing). #65368 edits the same matrix to remove `banned_words`, so these two will conflict textually. They're independent changes and either merge order works; whichever lands second needs a trivial rebase. ## Testing Rules layer, from the branch: ``` $ rayci test-rules $ echo $? 0 ``` 125 cases pass. Negative control: reverting the `README.rst` expectation to `always lint` fails with `+pypi_readme (unexpected)`, confirming the test is reading the new rule rather than passing vacuously. Step selection, generating the pipeline that the pinned rayci binary (v0.46.0, from `.rayciversion`) actually emits for a probe diff, rather than reasoning about tags: | Probe diff | Selected tags | `lint: doc_readme` | | --- | --- | --- | | `doc/source/index.rst`, at `master` | `always` | runs, inside the `g11_s0` matrix | | `doc/source/index.rst`, on this branch | `always` | not emitted | | `python/ray/actor.py`, on this branch | `always lint …` | not emitted | | `README.rst`, on this branch | `always lint pypi_readme` | emitted, `g11_s1` | | `python/setup.py`, on this branch | `… pypi_readme …` | emitted, `g11_s1` | The `master` row is the before state: on a prose-only diff the emitted prose matrix contains `banned_words`, `doc_readme`, `documentation_style`, `doc_no_new_rst`. On this branch that matrix has three entries and no separate `doc_readme` step appears. Not verifiable outside Buildkite: `if:` expression evaluation, which this PR doesn't touch. ## AI assistance AI assistance was used for this change. Every changed line was reviewed by a human before this PR was opened, and the commands above were run locally.", "url": "https://github.com/ray-project/ray/pull/65431", "createdAt": "2026-08-12T15:44:20Z", "updatedAt": "2026-08-12T19:22:17Z", "timestamp": "2026-08-12T19:22:17Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "core", "devprod", "go" ], "author": "dstrodtman", "state": "open", "assignees": [] }, { "id": "github:ray-project/ray:pull_request:65432", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[serve] test_cli deflake by adding explicit wait timeouts and stopping tracing config leak", "text": "Two independent cleanups to `python/ray/serve/tests/test_cli.py`. **1. Explicit wait timeouts.** 17 of the file's 41 `wait_for_condition` calls had an effective timeout of 10s, and 12 of those were bare calls inheriting the default rather than stating one. Every call now carries an explicit timeout, and existing values are doubled, matching the convention already used elsewhere in this file. **2. Stop global tracing config leaking across tests.** `test_deploy_config_tracing_config_declarative_flow` deploys a config with `tracing_config: {enabled: true, sampling_ratio: 1.0}`. `ServeController.deploy_config` only ever *sets* `global_tracing_config` β€” there is no reset path β€” and the `serve_instance` fixture is session-scoped, its teardown only deleting apps. Every subsequent test in the file therefore ran with tracing enabled at ratio 1.0, exporting a span per request. The test now resets tracing to disabled in its teardown. ## Testing - 114 unloaded full-file runs, all green. - Under induced CPU contention, the timeout change fixed 3 previously-failing tests (`test_deploy_multi_app_basic`, `test_deploy_multi_app_builder_with_args`, `test_status_multi_app`); tests that wait for a terminal `DEPLOY_FAILED` state are unaffected, as that duration is governed by replica retry exhaustion rather than the wait timeout. - Tracing leak fix verified directly: a probe test in the same session now observes `enabled=False` where it previously saw `enabled=True`.", "url": "https://github.com/ray-project/ray/pull/65432", "createdAt": "2026-08-12T16:34:25Z", "updatedAt": "2026-08-12T16:39:56Z", "timestamp": "2026-08-12T16:39:56Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "serve", "go" ], "author": "johntaylor-cell", "state": "open", "assignees": [ "johntaylor-cell" ] }, { "id": "github:ray-project/ray:pull_request:65435", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[data] Cap task pool resource demand by runnable inputs", "text": "## Description `TaskPoolMapOperator.min_max_resource_requirements()` reports infinite maximum demand when `max_concurrency` is unset. That is a useful capacity bound, but it is not always a useful fair-share bound: an operator with one active or queued input can run at most one task. This matters for pipelines where one task fans out into many rows. The following is a complete Ray Data example matching that shape: ```python from dataclasses import dataclass import ray @dataclass class FilePartitioningTask: files: list[str] def partition_files(_): return { \"item\": [ FilePartitioningTask(files=[f\"file-{i}.jsonl\"]) for i in range(20) ] } class ProcessFile: def __call__(self, batch): return batch ray.init(num_cpus=8) ds = ( ray.data.from_items([None]) .map_batches( partition_files, batch_size=1, compute=ray.data.TaskPoolStrategy(size=1), num_cpus=1, ) .repartition(target_num_rows_per_block=1) .map_batches( ProcessFile, batch_size=1, compute=ray.data.ActorPoolStrategy(size=7), num_cpus=1, ) ) assert ds.materialize().count() == 20 ``` The first map runs once and returns 20 dataclass rows in one block. `StreamingRepartition` receives one input block, so it can launch one task, then emits 20 one-row blocks for the actor pool. On an 8-CPU local cluster, master allocated as many as 4 CPUs to that single runnable repartition task while the downstream actor pool received 6. With this change, repartition stayed at 1 CPU and the downstream pool received 7 once it had queued work. This PR caps a task-pool operator's current maximum demand at its active tasks plus queued input bundles. The configured `max_concurrency` remains the upper bound. Resource budgets are recomputed as inputs arrive, so the cap grows with runnable work. Actor pools continue to report their configured pool bounds. Their actors may need to be created before input arrives, and applying the same input-based cap would change `min_size` and `initial_size` prewarming behavior. The change applies to task-pool operators generally; it does not inspect operator names or special-case `StreamingRepartition`. Capacity left by an operator's current-demand cap is offered to downstream operators that can use it. ## Trade-offs Benefits: - Fair-share allocation reflects work the task pool can run now. - Otherwise-unused CPU or GPU capacity can serve runnable downstream work. - Allocation expands on later scheduler updates when more input becomes available. - Actor-pool startup and autoscaling behavior is unchanged. Costs: - Ray tasks are not preemptive. If downstream borrows the capacity and more upstream input later arrives, the allocator stops admitting additional downstream work, but already-running downstream tasks keep their resources until they finish. - The allocator does not reserve speculative headroom for inputs that an upstream operator has not produced yet. - This changes fair-share accounting for every task-pool map operator, not only streaming repartition. The first two costs are the work-conserving trade-off: idle capacity is used now in exchange for reacquisition waiting on already-running work if demand returns. I searched open issues and PRs for the same change. #62232 is related to StreamingRepartition concurrency, but it enforces an explicit user-configured ceiling. This PR instead changes fair-share accounting for currently runnable work. Short local wall-clock results varied with actor startup and autoscaling, so the local validation above is evidence for the allocation change, not a throughput claim. AI assistance was used to help investigate, implement, and test this change. ## Related issues Fixes #65433 ## Tests - `.venv/bin/python -m pytest -q python/ray/data/tests/test_reservation_based_resource_allocator.py` β€” 18 passed - `.venv/bin/python -m pytest -q python/ray/data/tests/test_task_pool_map_operator.py::test_min_max_resource_requirements` β€” 1 passed - `.venv/bin/python -m pytest -q python/ray/data/tests/test_actor_pool_map_operator.py -k 'test_min_max_resource_requirements or test_min_max_resource_requirements_unbounded'` β€” 2 passed - Standalone example above β€” passed with 20 output rows - Ray pre-commit hooks on all changed files β€” passed", "url": "https://github.com/ray-project/ray/pull/65435", "createdAt": "2026-08-12T19:53:17Z", "updatedAt": "2026-08-12T23:38:55Z", "timestamp": "2026-08-12T23:38:55Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [], "author": "praateekmahajan", "state": "open", "assignees": [] }, { "id": "github:ray-project/ray:pull_request:65436", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[data] Avoid rebuilding row-sized output buffers", "text": "## Description When only `target_num_rows_per_block` is set, `BlockOutputBuffer` currently builds the buffered block, emits one slice, and puts the remainder into a new builder. A StreamingRepartition with a one-row target repeats that work for every output block. This PR keeps the built block and advances through it by row offset. A builder is only needed again when a partial tail must be combined with later input. Byte-based sizing, combined row-and-byte sizing, disabled block shaping, ordering, and final partial blocks retain their existing behavior. I searched open issues and PRs for the same change. This complements the remainder-copying fix in #48266: that change avoided copying the remainder, while this change avoids rebuilding it for every emitted block. #62232 addresses user-configured StreamingRepartition concurrency and is independent of this slicing path. AI assistance was used to help investigate, implement, and test this change. ## Related issues Fixes #65434 ## Additional information I tested the change with a standalone version of the pipeline that motivated it, without importing Curator: ``` one seed -> MapBatches(generate_files_to_process) -> StreamingRepartition(target_num_rows_per_block=1) -> downstream map ``` `generate_files_to_process` returns a list of `FilePartitioningTask(files: list[str], metadata: dict)` objects from one task. With the default `DataContext`, Ray stores these rows in an `ArrowPythonObjectType` column. For 30,000 tasks, draining the row-sized output buffer changed from 30,000 builder builds in 0.281s to one build in 0.105s (2.7x). A full 1,000-task pipeline improved from a median 18.34s to 18.05s, while median time to the first downstream task was 1.383s versus 1.374s and time to the 100th was 2.825s in both cases. This shows the repeated build is removed for Python object rows, but it is not by itself an explanation for a large downstream scale-up delay. Tests run: - `.venv/bin/python -m pytest -q python/ray/data/tests/test_map_operator.py` β€” 68 passed - `.venv/bin/python -m pytest -q python/ray/data/tests/test_repartition_e2e.py -k 'target_num_rows_per_block or guarantee_row_num or streaming_repartition or partial_last_block'` β€” 21 passed, 6 deselected - Ray pre-commit hooks on all changed files β€” passed", "url": "https://github.com/ray-project/ray/pull/65436", "createdAt": "2026-08-12T19:53:17Z", "updatedAt": "2026-08-12T20:42:43Z", "timestamp": "2026-08-12T20:42:43Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [], "author": "praateekmahajan", "state": "open", "assignees": [] }, { "id": "github:ray-project/ray:pull_request:65437", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[Serve][2/2] Propagate TracingConfig to proxies via long poll", "text": "Follow-up to #63273 ([1/2]) completing the TracingConfig feature. - Added a `GLOBAL_TRACING_CONFIG` long-poll namespace. The controller broadcasts the global tracing config which is subscribed by the proxies, which runs `setup_tracing` on delivery. This wires proxy tracing without the init-time controller callback that cannot work (proxies can start before the controller). This broadcast mechanism also lets users change the tracing config at runtime through the existing serve-config API. - Controller: - checkpoint-ing the tracing config (So that it can be restored during the recovery). - Default it to an env-var-sourced TracingConfig() so it is never `None`, else it will cause attribute error. - Publishing the initial snapshot, and broadcasting changes from apply_config. get_tracing_config(). - reconfigure validates its input since the value is broadcasted cluster-wide. - `tracing_utils.setup_tracing`: - dropped the env-var fallback and the empty-path check, - `tracing_config` is now required (single source of truth). Its fields already default from the RAY_SERVE_TRACING_* env vars. - Proxy sets tracing up only from the long-poll callback, so `set_tracer_provider` is called once per process. - Replicas continue to read the config when they start. - Tests: assert proxy traces are produced via the imperative flow, and that a tracing config changed after startup survives controller recovery (so the checkpoint, not the constructor arg, is exercised). This PR used Claude Code Assistant for writing code.", "url": "https://github.com/ray-project/ray/pull/65437", "createdAt": "2026-08-12T20:09:27Z", "updatedAt": "2026-08-13T06:41:01Z", "timestamp": "2026-08-13T06:41:01Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "serve", "go", "community-contribution" ], "author": "suppagoddo", "state": "open", "assignees": [ "jeffreywang88" ] }, { "id": "github:ray-project/ray:pull_request:65438", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[doc] Add a KubeRay nav section and generalize the section sidebar", "text": "## Why The KubeRay guides are one of the most-visited parts of the Ray docs, but they're buried two levels down under Ray Clusters and share the global sidebar with the entire site. A reader working through RayService configuration gets every top-level Ray section in the left nav and no KubeRay-specific orientation. This promotes KubeRay to a top-level nav tab with a sidebar scoped to just KubeRay, reusing the mechanism the APIs tab already ships. ## What changed **Navigation (`doc/source/navbar.yml`)** `Use Cases` is replaced by a `KubeRay` tab in the #2 position. Tab order is now Get Started, KubeRay, Example Gallery, Library, APIs, Resources. `ray-overview/use-cases` still exists and stays in the root toctree and global sidebar; it only loses its tab. KubeRay also stays in the global sidebar under Ray Clusters, so `cluster/getting-started` is unchanged and readers browsing Ray Clusters still find it. No page moved, so no URLs change and no redirects are needed. **Section sidebars (`doc/source/_ext/api_sidebar.py`)** The APIs tab's client-side sidebar mechanism is generalized from one hardcoded section to an N-section table. Adding a section is now a `SECTIONS` entry (root docname, fragment filename, path prefixes, label, maxdepth) plus a `navbar.yml` tab β€” no template or loader change. That also covers the \"document the extension's config contract\" follow-up from the original APIs work. Two changes were needed to make it general: - **Subtree isolation.** The old code assumed the section root was a top-level `li.toctree-l1` marked `current`. `cluster/kubernetes/index` is nested two levels deep, so it isn't. It now walks to the *deepest* `li.current`, which is the section root by construction since the toctree was resolved for it. This reduces to the identical result for APIs. - **Href rebasing.** `_root_relative` stripped exactly one leading `../`, hardcoding `apis/index`'s depth-1 position. It now rebases onto `dirname(root_docname)` via `posixpath.normpath`, correct at any depth and byte-identical for APIs. `_templates/api-sidebar.html` β†’ `section-sidebar.html` and `_static/api-nav-loader.js` β†’ `section-nav-loader.js`; the fragment is per-section (`api-nav.html`, `kuberay-nav.html`). **Tab highlighting (`doc/source/custom_directives.py`)** `render_header_nodes` compared `item[\"file\"] == pagename`, so a tab highlighted only on its exact landing page β€” the APIs tab went dark as soon as you clicked into `data/api/…`. It now reads `context.get(\"navbar_active_file\") or pagename`, which the extension sets to the matched section's root. Both the APIs and KubeRay tabs now stay highlighted across their whole section. Tabs outside a section keep the previous exact-match behavior. **No-JS and accessibility fallback** Previously the section sidebar rendered `Loading API navigation…` and populated entirely from JavaScript, so no-JS readers got no navigation at all. The mount now server-renders the section's top-level pages as a real list of links (from `env.toctree_includes`, via `pathto`). The loader upgrades it to the full tree on success and, on failure, leaves it in place rather than replacing it with an error β€” a failed fetch now degrades to a shallower nav instead of none. `aria-busy` flips to `false` on both paths. This is a meaningful improvement, not full parity: the fallback is the section's top level only, not the whole tree. ## Design note: the sidebar swap is intentional Worth stating up front, because it's the obvious review question. KubeRay keeps its existing position in the global sidebar under Ray Clusters, so nothing moves for readers browsing that section. But once you're on a `cluster/kubernetes/**` page, the sidebar is *replaced* by the KubeRay-only tree rather than the KubeRay subtree being highlighted inside the full site nav. Key Concepts, Deploying on VMs, the Applications Guide, and the rest of `cluster/` drop out of the left nav until you navigate back out. That's deliberate. KubeRay is the recommended path for running Ray on Kubernetes and it's a large body of content β€” around 80 pages across getting-started, user guides, examples, ecosystem integrations, benchmarks, and troubleshooting β€” currently reachable only by drilling two levels down through Ray Clusters. Surfacing it at the top level and giving it undivided navigation is the point of the change; keeping the whole-site tree alongside it would preserve the discoverability problem this is meant to fix. Cross-links out of the section (roughly 32 of them, mostly into `serve/`, `ray-observability/`, and `ray-core/`) still work normally and land the reader back on the global sidebar. ## Notes for reviewers - **`_ext/api_sidebar.py` keeps its name even though the mechanism is no longer APIs-specific.** It's referenced by `.buildkite/test.rules.txt`, `.buildkite/test.rules.test.txt`, `doc/BUILD.bazel`, `ci/pipeline/BUILD.bazel`, and `ci/pipeline/test_doc_api_rules_sync.py`. Renaming it would pull the shared CI tag-rules surface into this PR. `API_PATH_PREFIXES` is preserved as a module-level literal tuple so that sync test still parses it out of the AST. There's a docstring note; happy to do the rename as a separate CI-rules PR. - **Two unrelated `black` hunks in `custom_directives.py`.** Pre-existing formatting drift in `collect_example_orphans`. The file isn't in black's exclude list, so pre-commit reformats the whole file on any commit touching it. - **The fallback nav's titles come from `env.titles`**, i.e. each target page's own title, not any custom label the section root's toctree gives the entry. So the APIs fallback reads \"Ray Data API\" where the hydrated tree reads \"Ray Data\". Recovering custom labels needs `env.get_doctree(root)` per page, too expensive for a fallback. KubeRay uses no custom labels and is unaffected. - The `no toctree children` warning is suppressed when the section root isn't in `env.all_docs`, so scoped `DOC_LIB` / `build_one_lib` builds don't fail under `fail_on_warning` on an out-of-scope section. ## Testing Opened as a draft because **this has not been through a Sphinx build yet** β€” I'm looking to the RtD preview for that. What's verified locally: - `black` clean at the repo's pinned 22.10.0; modules compile; `node --check` on the loader. - `ci/pipeline/test_doc_api_rules_sync.py` passes (2/2), confirming `API_PATH_PREFIXES` still parses and stays in sync with the `doc_api` tag rule. - Subtree isolation and href rebasing checked against synthetic toctree HTML for both the depth-1 (APIs) and depth-2 (KubeRay) cases: APIs output unchanged, KubeRay correctly excludes Key Concepts, VMs, and the Ray Clusters parent. - `section-sidebar.html` rendered at three page depths, confirming correct relative fragment URLs, correct relative link targets, and current-page marking. The main thing the preview needs to confirm is the runtime assumption that `global_toctree_for_doc` marks `current` down the nested KubeRay path the way the fixture assumes. Worth a look on the preview: - KubeRay tab present at position 2, sidebar scoped to KubeRay on `cluster/kubernetes/**`. - APIs sidebar unchanged from today. - Both tabs stay highlighted on section sub-pages. - With JavaScript disabled, both sections still show a top-level nav.", "url": "https://github.com/ray-project/ray/pull/65438", "createdAt": "2026-08-12T20:41:51Z", "updatedAt": "2026-08-12T20:50:16Z", "timestamp": "2026-08-12T20:50:16Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [], "author": "dstrodtman", "state": "open", "assignees": [] }, { "id": "github:ray-project/ray:pull_request:65439", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "refactor(setup): modernize string formatting and ensure explicit file encoding", "text": "## Description This PR modernizes the `setup.py` build configuration script by refactoring legacy `.format()` string formatting to modern Python f-strings. Additionally, it explicitly enforces `encoding=\"utf-8\"` in `find_version()` to ensure cross-platform consistency and prevent encoding errors on non-UTF-8 default environments (e.g., Windows). ## Related issues Related to code cleanup and build script maintenance. ## Additional information - Refactored `.format()` calls to f-strings across `setup.py`. - Added explicit `encoding=\"utf-8\"` when reading version files. - Replaced slice-based extension checking (`[-3:] == \".py\"`) with `.endswith(\".py\")`. - No functional or breaking changes introduced to the build pipeline.", "url": "https://github.com/ray-project/ray/pull/65439", "createdAt": "2026-08-12T21:12:50Z", "updatedAt": "2026-08-13T01:33:13Z", "timestamp": "2026-08-13T01:33:13Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "core", "community-contribution" ], "author": "ardozcnn", "state": "open", "assignees": [] }, { "id": "github:ray-project/ray:pull_request:65440", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[serve][llm] Surface engine errors on the direct-streaming ASGI app", "text": "## Description vLLM's **engine error handler** reads `app.state.server` to flip `should_exit` and stop the uvicorn server its launcher created. `VLLMEngine.build_asgi_app` calls only `build_app` and `init_app_state`, neither of which sets it. So on the direct-streaming path, every `EngineGenerateError` / `EngineDeadError` raises `AttributeError` inside the handler, and that replaces the engine error, and therefore clients get 'State' object has no attribute 'server' instead of the real failure. ## Solution Supply a dummy object that just tolerates an attribute write `types.SimpleNamespace()`. Nothing reads `should_exit`; `terminate_if_errored` only assigns it. With the read satisfied, the handler runs on to `create_error_response(exc)` and returns vLLM's real error. Discarding the write is safe: `should_exit` only stops vLLM's uvicorn loop, which Ray never runs β€” Serve restarts replicas via `LLMServer.check_health`. ## Reproduction Stand up a server with the following script and environment variables `RAY_SERVE_LLM_ENABLE_DIRECT_STREAMING=1` and `RAY_SERVE_ENABLE_HA_PROXY=1`. ``` # serve_app.py from ray import serve from ray.serve.llm import LLMConfig, LLMServingArgs, build_openai_app llm_config = LLMConfig( model_loading_config=dict( model_id=\"qwen\", model_source=\"Qwen/Qwen3-0.6B\", ), deployment_config=dict( autoscaling_config=dict(min_replicas=1, max_replicas=1), ), ) serve.run(build_openai_app(LLMServingArgs(llm_configs=[llm_config])), blocking=True) ``` - Before ``` { \"error\": { \"message\": \"'State' object has no attribute 'server'\", \"type\": \"InternalServerError\", \"param\": null, \"code\": 500 } } ``` - After ``` { \"error\": { \"message\": \"EngineCore encountered an issue. See stack trace (above) for the root cause.\", \"type\": \"InternalServerError\", \"param\": null, \"code\": 500 } } ``` ## Related issues > Link related issues: \"Fixes #1234\", \"Closes #1234\", or \"Related to #1234\". ## Additional information > Optional: Add implementation details, API changes, usage examples, screenshots, etc.", "url": "https://github.com/ray-project/ray/pull/65440", "createdAt": "2026-08-12T21:52:52Z", "updatedAt": "2026-08-13T01:32:35Z", "timestamp": "2026-08-13T01:32:35Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "serve", "llm", "go" ], "author": "jeffreywang88", "state": "open", "assignees": [] }, { "id": "github:ray-project/ray:pull_request:65441", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[doc][History server] Update doc for history server for `RAY_ROOT_DIR` -> `STORAGE_ROOT_DIR`", "text": "## Description https://github.com/ray-project/kuberay/pull/5121 updated the env var from `RAY_ROOT_DIR` to `STORAGE_ROOT_DIR`. The default storage root dir is now the bucket root itself, so updated example and file structure in the documentation. ## Related issues Related to https://github.com/ray-project/kuberay/pull/5121 ## Additional information", "url": "https://github.com/ray-project/ray/pull/65441", "createdAt": "2026-08-12T22:15:06Z", "updatedAt": "2026-08-13T17:19:26Z", "timestamp": "2026-08-13T17:19:26Z", "metrics": { "reactions": 1, "comments": 1 }, "labels": [ "core", "go", "community-contribution" ], "author": "chiayi", "state": "open", "assignees": [] }, { "id": "github:ray-project/ray:pull_request:65442", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[TPU] Rename dispatch to run_on_slice", "text": "# Description We chose a terse name to be less redundant originally. However, when imported like `from ray.util.tpu import dispatch`, the function is taking a commonly used word away as an identifier and is also ambiguously named at call sites. For readability, rename dispatch to run_on_slice, and keep the former name as an alias for potential existing uses. ## Related issues Function introduced in #64493", "url": "https://github.com/ray-project/ray/pull/65442", "createdAt": "2026-08-12T23:50:50Z", "updatedAt": "2026-08-13T01:30:51Z", "timestamp": "2026-08-13T01:30:51Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "train", "core", "community-contribution" ], "author": "spencer-p", "state": "open", "assignees": [] }, { "id": "github:ray-project/ray:pull_request:65445", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[docs][core] Document JVM signal conflicts with PyArrow HDFS", "text": "## Why are these changes needed? PyArrow HDFS loads `libhdfs`, which embeds a HotSpot JVM in the Python process. HotSpot uses some `SIGSEGV` events as recoverable VM control flow, while Ray's CoreWorker installs an Abseil failure-signal handler that can interpret the same signal as a fatal native crash. The conflict can be established during handler installation and surface later as an intermittent JVM crash. This PR documents: - the symptoms and root cause of the Ray/PyArrow HDFS signal-handler conflict; - HotSpot signal chaining with `libjsig.so` as the preferred mitigation because it keeps Ray's native crash diagnostics enabled; - `RAY_DISABLE_FAILURE_SIGNAL_HANDLER=1` as a fallback and its diagnostic trade-off; - the process-start timing requirement for `LD_PRELOAD` and a KubeRay configuration example; - discoverable entry points from the Ray Data and Ray Train HDFS documentation. Related to #36415. ## Duplicate-work check I reviewed #36415 and searched open PRs for the issue number, `libjsig`, and PyArrow/HDFS signal-handler documentation. I found no open PR covering this documentation work. #58984 is already merged and added `RAY_DISABLE_FAILURE_SIGNAL_HANDLER`, but it didn't document the option, its diagnostic trade-off, or HotSpot signal chaining. ## Validation Documentation checks: - `git diff --check` - `python -m pre_commit run --files doc/source/ray-observability/user-guides/debug-apps/debug-failures.rst doc/source/train/user-guides/persistent-storage.rst doc/source/data/loading-data.rst` - `sphinx-lint --disable trailing-whitespace doc/source/ray-observability/user-guides/debug-apps/debug-failures.rst doc/source/train/user-guides/persistent-storage.rst doc/source/data/loading-data.rst` - Verified the linked Oracle signal-handling pages and #36415 return HTTP 200. - Validated the documented `find`/`LD_PRELOAD` shell snippet with the local OpenJDK installation. Runtime validation behind the guidance: - Reproduced the unmitigated JVM termination and generated multiple `hs_err_pid*.log` files. - Completed 5,000- and 10,000-operation HDFS stress runs with `RAY_DISABLE_FAILURE_SIGNAL_HANDLER=1` without reproducing the crash. - Completed a 5,000-operation HDFS stress run with only `libjsig.so` preloaded and Ray's handler left enabled. - Completed a separate 1,000-operation `libjsig.so` run, followed by a clean `ray.shutdown()` and process exit code 0. The full Sphinx documentation site wasn't built locally because the existing Python environment doesn't include the Ray documentation dependency set. ## AI assistance AI assistance was used to investigate the failure, draft the documentation, and run the validation commands. I reviewed every changed line and the reported test results.", "url": "https://github.com/ray-project/ray/pull/65445", "createdAt": "2026-08-13T03:06:39Z", "updatedAt": "2026-08-13T07:31:01Z", "timestamp": "2026-08-13T07:31:01Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "docs", "core", "community-contribution" ], "author": "OneSizeFitsQuorum", "state": "open", "assignees": [] }, { "id": "github:ray-project/ray:pull_request:65446", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[Data] Cluster autoscaler requests a bundle larger than the node it was copied from, because node memory is rounded up", "text": "# Issue #65237 β€” autoscaler requests a bundle larger than the node it was copied from ## 1. Root cause `DefaultClusterAutoscalerV2` builds its autoscaler request out of node *shapes*: `_get_node_resource_spec_and_count()` turns every live worker node's resource dict into a `_NodeResourceSpec`, and `try_trigger_scaling()` sends `spec.to_bundle()` back to the autoscaler as active bundles (one per existing node) and pending bundles (the scale-up delta). `_NodeResourceSpec.of()` quantized memory with a *nearest*-multiple rounding (`python/ray/data/_internal/cluster_autoscaler/default_cluster_autoscaler_v2.py:64` before the fix): ```python mem = int(round(mem / GiB, 1) * GiB) if mem > 0 else 0 ``` The quantization exists so that two nodes of the same instance type that report slightly different physical memory (Ray derives the `memory` resource from memory available at init time, so it jitters) collapse into one shape instead of two. Rounding to the *nearest* 0.1 GiB rounds **up** for anything in the upper half of a bucket. A node reporting 14.87 GiB becomes a 14.9 GiB bundle: strictly larger than the node it was copied from. That bundle does not fit on any node of that type, so the autoscaler treats the demand as infeasible rather than launching the node, and the cluster never scales up (CPU and GPU were already floored, so memory was the only resource with this defect). ## 2. The fix and why Quantize memory **down** instead of to the nearest multiple: ```python mem = math.floor(mem / _MEMORY_QUANTIZATION_BYTES) * _MEMORY_QUANTIZATION_BYTES ``` Rounding down makes `spec.mem <= node_memory` an invariant, which is the property the request actually needs: a bundle must be schedulable on the node shape it describes. The error is now asymmetric in the safe direction β€” under-requesting memory still lands on a node of that type, over-requesting can never be satisfied. The granularity is a named constant `_MEMORY_QUANTIZATION_BYTES = GiB`. It was widened from 0.1 GiB to 1 GiB because flooring at 0.1 GiB would have *lost* the grouping property the quantization was added for: 14.87 GiB and 14.93 GiB floor to 14.8 and 14.9, i.e. two shapes for one instance type (this is exactly what `test_nodes_with_similar_memory_grouped` covers). A 1 GiB bucket keeps both nodes on the same shape and gives 10x more slack against reported-memory jitter than the original rounding did, while never exceeding the source node. Matching CPU/GPU are still exact, so the coarser memory bucket does not realistically make the request match a different (smaller) instance type. ## 3. Files changed - `python/ray/data/_internal/cluster_autoscaler/default_cluster_autoscaler_v2.py` β€” added `_MEMORY_QUANTIZATION_BYTES`; `_NodeResourceSpec.of()` now floors memory to it. - `python/ray/data/tests/test_default_cluster_autoscaler_v2.py` β€” added `test_spec_never_exceeds_node_memory` (parametrized invariant check) and `test_bundles_fit_on_the_nodes_they_were_derived_from` (end-to-end through `_get_node_resource_spec_and_count`, asserting every resource in the emitted bundle fits the source node). ## 4. Risk / uncertainty - Memory demand per node is now under-reported by up to 1 GiB (previously by up to 0.05 GiB, or over-reported by up to 0.05 GiB). Under-reporting is safe for feasibility. It does slightly loosen `cap_resource_request_to_limits`: with a user-configured `resource_limits.memory`, pending bundles are measured with the floored value, so a scale-up can overshoot the configured memory limit by up to 1 GiB per pending bundle (bounded by `cluster_scaling_up_delta`, default 1, times the number of node shapes). The old 0.1 GiB rounding had the same failure mode, an order of magnitude smaller. - Nodes with less than 1 GiB of Ray `memory` now produce `memory: 0` bundles (the old code did this below 0.05 GiB). Such worker nodes are not realistic in an autoscaling Ray Data cluster, and CPU/GPU still carry the demand. - The choice of 1 GiB is a judgement call: it is the smallest round bucket that keeps the existing same-node-type grouping test passing under flooring. A maintainer may prefer a different constant; it is isolated in one named variable. - Head-node-group detection (`node_group_config` shape vs. the running head node's shape) also compares quantized specs. Coarser buckets make that comparison neither better nor worse in a way I could establish, since a configured shape and the reported shape can straddle a bucket boundary at any granularity. ## 5. How I verified it Ray is not installed in this environment (`import ray` fails), so the test suite could not be executed here. Verification done: - Extracted the real `_NodeResourceSpec` class from the patched source with `ast` and exec'd it against stubs, then checked, for `mem` in `{0, 1000 B, 0.5, 14.87, 14.93, 32, 8, 1.999, 2 GiB}`: the quantized value is always `<= mem`, is an `int` (the frozen dataclass asserts this in `__post_init__`), and `of(mem=14.87 GiB) == of(mem=14.93 GiB)` still holds. All passed; the pre-fix code returns 14.9 GiB for a 14.87 GiB node, reproducing the issue. - Hand-checked every existing memory value in `test_default_cluster_autoscaler_v2.py` against the new quantization: the small byte-valued fixtures (1000/2000/3000) already quantized to 0 under the old rounding and still do; the GiB-valued fixtures (2, 4, 8, 32 GiB) are exact multiples and are unchanged, including the `memory: 8.0GiB` log-message assertion. - `grep` confirmed `_NodeResourceSpec` / `_get_node_resource_spec_and_count` have no consumers outside this module and its test file. - `python -m py_compile` on both changed files; no line exceeds the configured ruff `line-length = 88`. Suggested command for a human to run before submitting: ```bash pytest -q python/ray/data/tests/test_default_cluster_autoscaler_v2.py ```", "url": "https://github.com/ray-project/ray/pull/65446", "createdAt": "2026-08-13T03:17:12Z", "updatedAt": "2026-08-13T07:30:28Z", "timestamp": "2026-08-13T07:30:28Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "data", "community-contribution" ], "author": "pjdurden", "state": "open", "assignees": [] }, { "id": "github:ray-project/ray:pull_request:65447", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[train] Share PlacementGroupCleaner across Train runs", "text": "## Description Ray Train v2 currently creates one detached `PlacementGroupCleaner` actor per Train controller. Since cleaners are pinned to the head node, their memory use grows linearly with concurrent Train runs. In the workload described in #65443, 24 cleaners use approximately 14.4 GB on the head node. This PR replaces the per-controller actors with one cluster-wide shared cleaner. The cleaner tracks controllers independently, so a failed controller only triggers cleanup for its own placement groups. The shared cleaner: - uses a fixed, versioned name and internal namespace with `get_if_exists=True`; - remains detached, pinned to the head node, and scheduled outside captured placement groups; - persists controller and placement-group registrations in GCS internal KV; - restores active and incomplete cleanup records after actor restart; - rejects late placement-group registration after controller death; - isolates State API and placement-group cleanup failures between controllers; - scopes State API queries to its own cluster when multiple clusters are active. ## Related issues Fixes #65443. ## Testing ```text PYTHONPATH=\"$PWD/python\" python -m pytest -q \\ python/ray/train/v2/tests/test_placement_group_cleaner.py # 15 passed PYTHONPATH=\"$PWD/python\" python -m pytest -q \\ python/ray/train/v2/tests/test_controller_callback_behaviour.py # 10 passed pre-commit run --files \\ python/ray/train/v2/_internal/callbacks/placement_group_callback.py \\ python/ray/train/v2/_internal/execution/controller/placement_group_cleaner.py \\ python/ray/train/v2/_internal/state/util.py \\ python/ray/train/v2/tests/test_placement_group_cleaner.py # Passed git diff --check origin/master...HEAD # Passed ``` A local end-to-end multi-job scenario used two independent drivers, each running a real Train v2 `DataParallelTrainer` with a controller, worker, and placement group. Both runs shared one cleaner. After one driver was killed, only its placement group was removed; the other run remained active and completed normally. The cleaner stayed alive throughout. ## Contribution notes No other open PR referencing #65443 or matching this PlacementGroupCleaner change was found before requesting review. AI assistance was used for implementation, testing, and self-review of this change.", "url": "https://github.com/ray-project/ray/pull/65447", "createdAt": "2026-08-13T05:34:32Z", "updatedAt": "2026-08-13T13:34:06Z", "timestamp": "2026-08-13T13:34:06Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "train", "community-contribution" ], "author": "OneSizeFitsQuorum", "state": "open", "assignees": [] }, { "id": "github:ray-project/ray:pull_request:65448", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[Core] Randomize worker port allocation", "text": "## Description When multiple raylets share a network namespace and use the same worker port range, they currently allocate ports in the same ascending order. This causes deterministic collisions near the beginning of the range, even when most ports remain available. This PR builds and validates the worker port pool once during raylet startup, then shuffles it before constructing the free-port queue. It: - Shuffles both configured ranges and explicit `--worker-port-list` values. - Rejects out-of-range and duplicate explicit ports with errors that identify the offending port. - Preserves explicit-list precedence and the existing `max_worker_port=0` behavior. - Adds `worker_port_shuffle_enabled`, defaulting to `true`, with `false` restoring the previous allocation order. - Leaves `GetNextFreePort` and `MarkPortAsFree` unchanged. - Documents that randomization reduces collision probability but does not provide cross-raylet port reservation. Non-overlapping ranges or `--min-worker-port=0 --max-worker-port=0` remain the robust solutions. ## Related issues Closes #65444. ## Additional information The shuffle occurs only once, during `WorkerPool` construction. It does not change the configured set of ports or the existing occupied-port retry and port-return behavior.", "url": "https://github.com/ray-project/ray/pull/65448", "createdAt": "2026-08-13T06:11:56Z", "updatedAt": "2026-08-13T09:56:46Z", "timestamp": "2026-08-13T09:56:46Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "core", "community-contribution" ], "author": "zzchun", "state": "open", "assignees": [] }, { "id": "github:ray-project/ray:pull_request:65450", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[Data] Dereference ObjectRefs passed through map UDF `fn_args`", "text": "## Description Ray Data currently captures UDF `fn_args` in the serialized UDF wrapper. As a result, direct `ObjectRef` values aren't passed as top-level Ray task arguments: - UDFs must manually call `ray.get()`. - Ray's scheduler can't wait for these dependencies before scheduling map work. - Map workers may reserve CPU or GPU resources and then block inside user code. Passing direct `ObjectRef` values as task arguments aligns this behavior with Ray Core and enables dependency-aware scheduling. However, changing the default immediately would break existing UDFs that expect to receive an `ObjectRef` and call `ray.get()` themselves. This PR therefore introduces the new behavior through a backward-compatible migration path. This PR adds `DataContext.enable_dereference_object_refs_in_fn_args`, which defaults to `False`. With the default legacy behavior: - Direct `ObjectRef` values continue to reach the UDF unchanged. - Ray Data emits a `RayDeprecationWarning` describing the migration. Users can opt in before creating the Dataset: ```python ray.data.DataContext.get_current().enable_dereference_object_refs_in_fn_args = True ``` or set: ```shell RAY_DATA_ENABLE_DEREFERENCE_OBJECT_REFS_IN_FN_ARGS=1 ``` When enabled: - Direct `ObjectRef` values in `fn_args` are passed as top-level task arguments. - Ray Core waits for and dereferences them before invoking the UDF. - Transform adapters reconstruct the positional argument tuple once per task. - Sync, async, async-generator, row, batch, flat-map, filter, and callable-class `__call__` paths use the resolved arguments consistently. - Failed dependencies can prevent user code from starting. Users who intentionally need reference semantics after the default changes can wrap the `ObjectRef` in a container. Only direct top-level `ObjectRef` values are automatically dereferenced. ## Related issues closes #65449 and related to #49207 (`fn_kwargs` and callable-class constructor arguments remain separate follow-up surfaces.) ## Additional information The download planner is updated because it also consumes the private `_get_udf` helper. The batch-transform helper retains its existing raw one-argument UDF calling convention when no argument resolver is supplied. Observable behavior changes only for direct `ObjectRef` values in `fn_args`: - The UDF now receives the resolved value. - Failed dependencies can stop a map task before user code begins. - Map-to-all-to-all execution remains unfused while task kwargs are present.", "url": "https://github.com/ray-project/ray/pull/65450", "createdAt": "2026-08-13T06:40:45Z", "updatedAt": "2026-08-13T10:51:14Z", "timestamp": "2026-08-13T10:51:14Z", "metrics": { "reactions": 1, "comments": 2 }, "labels": [ "data", "community-contribution" ], "author": "viiccwen", "state": "open", "assignees": [] }, { "id": "github:ray-project/ray:pull_request:65452", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[Core] Release RuntimeEnv agent per-job logger file handles after setup", "text": "## Description The RuntimeEnv agent caches one logger per job (`_per_job_logger_cache`) and never closes it, and `logging.getLogger` holds a second reference in the global registry. Each logger owns `1 + len(log_files)` file descriptors, so open FDs grow with the cumulative number of jobs the node has handled until every setup fails with `OSError: [Errno 24] Too many open files`. Fix: scope the descriptors to the in-flight setup. - `SetupLoggerFactory` pools file handlers by absolute log path with a > reference count; concurrent setups writing the same file share one handler, > and the file closes when the last one exits. - A context manager wraps `_setup_runtime_env`, releasing on every exit path > (success, failure, exception, timeout, cancellation). - The per-setup logger is a detached `logging.Logger` (not in the global > registry) with the same level/format/propagate as before. - Late writes from executor threads can't re-leak: release attaches a > `NullHandler` before detaching (no empty-handler-list window β†’ > `logging.lastResort`), and `_SetupFileHandler.emit` refuses to write once > closed (`FileHandler.emit` otherwise silently reopens the file in append > mode; `handle()`/`close()` share the handler lock, so the check is > race-free). Both races were reproduced with failing tests first. Log files on disk are untouched β€” retention stays decoupled from FD release. **Not a duplicate:** #54935 is open with no linked PR. The prior attempt #55613 (LRU cache) was auto-closed as stale; an LRU cap keeps up to `maxsize` FDs open and its eviction deleted log files out from under the log monitor, so this takes the setup-scoped-release approach instead. ## Related issues Closes #65451 Related to #55613", "url": "https://github.com/ray-project/ray/pull/65452", "createdAt": "2026-08-13T08:32:52Z", "updatedAt": "2026-08-13T13:33:17Z", "timestamp": "2026-08-13T13:33:17Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "core", "community-contribution" ], "author": "zzchun", "state": "open", "assignees": [] }, { "id": "github:ray-project/ray:pull_request:65453", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[Data] Add support for writing ORC files", "text": "## Description Ray Data can read ORC files, but it can't write them. This adds `Dataset.write_orc()`, backed by a new `ORCDatasink` that writes each output block with `pyarrow.orc.write_table`. The API forwards PyArrow ORC writer options and supports `min_rows_per_file` through the existing block-based file datasink infrastructure. ## Related issues Follow-up to #64600 ## Additional information Mirrors the `write_csv` / `write_json` APIs. PyArrow ORC writer options can be passed directly to `write_orc()`. No new dependency (`pyarrow` already ships `pyarrow.orc`). Adds write tests in `test_orc.py` and API docs in `saving_data.rst`. ```python import ray ds = ray.data.range(100) ds.write_orc(\"s3://bucket/path/\", compression=\"zstd\")", "url": "https://github.com/ray-project/ray/pull/65453", "createdAt": "2026-08-13T09:53:45Z", "updatedAt": "2026-08-13T16:44:39Z", "timestamp": "2026-08-13T16:44:39Z", "metrics": { "reactions": 1, "comments": 1 }, "labels": [ "docs", "data", "community-contribution" ], "author": "WFY123wfy", "state": "open", "assignees": [] }, { "id": "github:ray-project/ray:pull_request:65455", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[Core][KubeRay] Route autoscaler INFO logs to stdout instead of stderr", "text": "## Description `_setup_logging()` in the KubeRay autoscaler entry point attached a single `logging._StderrHandler` to the root logger, so every record β€” INFO included β€” went to the container's **stderr**. Kubernetes' CRI log format tags each line with the stream it arrived on (`TIMESTAMP STREAM TAG CONTENT`), and log collectors derive severity from that field rather than parsing the `INFO` token out of the message. On a KubeRay cluster with log collection, that makes 100% of routine autoscaler output arrive as errors β€” measured 2392 `status:error` / 0 `status:info` over 20 minutes on an idle production cluster, where the container's only output was `skipping idle termination`. This PR splits the root logger's console output by severity: sub-WARNING records to stdout, WARNING and above to stderr. This restores the behaviour the function's docstring still describes (\"Also log to pod stdout\") and that the code's own comment described before #34182 (\"Logs will also be written to the container's stdout\"). ### Behaviour change **This changes the default.** Anyone parsing the autoscaler container's stderr for INFO-level lines will now find them on stdout instead. `kubectl logs` is unaffected, since it interleaves both streams. WARNING/ERROR/CRITICAL stay on stderr. `monitor.log` is untouched β€” it still receives every level at `RAY_LOGGER_LEVEL`, because it backs the Ray dashboard and the State API (`/api/v0/logs`). Neither its level nor its gating changed; there is a regression test for this. ### The stale comment The removed comment claimed: ```python # The stdout handler was set up in the Ray CLI entry point. # See ray.scripts.scripts::cli(). ``` That is not true on master, in two independent ways: 1. `cli()` calls `ray._private.ray_logging.setup_logger()`, which re-sets the level and re-formats handlers already on the `ray` logger. It never calls `addHandler`. 2. Even if it did, `setup_component_logger()` β€” called earlier in this same function β€” calls `ray._private.log.clear_logger(\"ray\")`, which empties that logger's handlers. Verified by running the real CLI path (`setup_logger()` then `_setup_logging()`) and inspecting the handler list: `handlers writing to sys.stdout = []`. The comment is removed. ### Design note: `_StderrHandler` and Rich compatibility #33652 deliberately chose `logging._StderrHandler` over a plain `StreamHandler` because it \"always uses whatever `sys.stderr` is currently set to rather than the value of `sys.stderr` at handler construction time,\" which was needed for Rich's `FileProxy` redirect to work when a logger is instantiated outside vs. inside `rich.live.Live()`. **This is the main design risk in the change and I did not want to quietly drop that property.** A naive swap to `logging.StreamHandler(sys.stdout)` would capture the stream at construction time and break under any post-setup redirect. Two things are worth noting: - #33652 modified only `python/ray/_private/ray_logging.py`. It never touched `run_autoscaler.py`; the `_StderrHandler` here arrived separately via #34182, as part of a formatting refactor rather than a stream-routing decision. - `FileProxy` no longer exists anywhere in the repo (`grep -rn \"FileProxy\" python/` returns nothing), and `rich` is not imported anywhere under `python/ray/autoscaler/`. So the original rationale arguably no longer applies here. Rather than rely on that, this PR keeps the lazy-stream property for **both** streams via a small `_StderrHandler`-equivalent that resolves its target at emit time: ```python class _LazyStreamHandler(logging.StreamHandler): def __init__(self, stream_name: str): logging.Handler.__init__(self) # skip StreamHandler.__init__; it assigns self.stream self._stream_name = stream_name @property def stream(self): return getattr(sys, self._stream_name) ``` This is strictly no worse than the status quo under stream redirection, and there is a test asserting the handler follows a `sys.stdout` reassignment made after construction. Happy to move this into `ray/_private/log.py` next to `PlainRayHandler` if reviewers would prefer it shared rather than local to the KubeRay entry point. The stderr handler's level is floored at `WARNING` rather than hardcoded to it, so a stricter `RAY_LOGGER_LEVEL=error` still suppresses warnings on stderr. The floor is computed by reading the level back off the handler after `setLevel()`, because `logging.getLevelName()` returns a *string* (`\"Level FOO\"`) for names it does not recognize, and `RAY_LOGGER_LEVEL` is read straight from the environment without validation. `setLevel()` is what rejects those, raising `ValueError` exactly as this function already did for an invalid `RAY_LOGGER_LEVEL`; there is a test pinning that. ### Unrelated pre-existing issue noticed while testing (not fixed here) `RAY_LOGGER_LEVEL=debug` does not actually produce DEBUG output from the autoscaler, before or after this PR. `_setup_logging` calls `logging.root.setLevel(level)`, but the `ray` logger has its own explicit `INFO` level set during `import ray`, and the autoscaler's logger is a descendant of it β€” so the effective level stays `INFO` and DEBUG records are never created. `clear_logger(\"ray\")` resets handlers and `propagate` but not the level. I left this alone to keep the PR focused. It does mean the level-related tests here assert handler *configuration* rather than emitted output for sub-INFO levels; that is called out in a docstring so the next reader does not mistake it for an oversight. Happy to file it separately if it is worth fixing. ## Related issues Closes #65454 Related to #48651 (directing logs to both stdout/stderr and log files) β€” that issue is broader and framework-wide; this is a targeted fix for the KubeRay autoscaler's own handler setup and does not conflict with it. Same bug class as #32339, fixed one component over. Regression from #34182. ## Additional information ### Tests Added `python/ray/tests/kuberay/test_run_autoscaler_logging.py` (14 tests, registered in `python/ray/tests/BUILD.bazel` in the existing small/`team:core` `py_test_module_list`): - `test_info_goes_to_stdout` β€” INFO on stdout, not stderr. - `test_warning_and_above_go_to_stderr` β€” WARNING/ERROR/CRITICAL on stderr, not stdout. - `test_monitor_log_still_receives_all_levels` β€” `monitor.log` keeps every level. - `test_default_level_filters_debug` β€” DEBUG dropped everywhere at the default level. - `test_handler_levels_track_logger_level` β€” parametrized over all five `LOGGER_LEVEL_CHOICES`, pinning the stderr WARNING floor and the stricter-level case. - `test_level_stricter_than_warning_suppresses_warnings` β€” `error`/`critical` keep warnings off both streams. - `test_split_is_numeric_not_name_based` β€” a custom level 25 between INFO and WARNING routes to stdout. - `test_invalid_logger_level_raises` β€” invalid `RAY_LOGGER_LEVEL` still raises `ValueError`. - `test_lazy_stream_handler_follows_stream_reassignment` β€” preserves the #33652 property under a post-construction `sys.stdout` swap. The tests snapshot and restore global logging state (root handlers/level, `ray` logger handlers/propagate) so they cannot leak into other tests in the same process, and close the handlers on teardown so `monitor.log`'s `RotatingFileHandler` does not leak an fd. Confirmed the tests actually catch the bug: reverting just the handler setup to the old single-`_StderrHandler` version fails 7 of the 14, including `test_info_goes_to_stdout` with `AssertionError: assert 'INFO_RECORD' in ''`. The `monitor.log` test passes both before and after, which is the check that the fix leaves `monitor.log` alone. ### Commands run ``` $ python -m pytest -q python/ray/tests/kuberay/test_run_autoscaler_logging.py 14 passed in 0.08s $ python -m pytest -q python/ray/tests/kuberay/test_autoscaling_config.py \\ python/ray/tests/kuberay/test_kuberay_node_provider.py \\ python/ray/tests/kuberay/test_run_autoscaler_logging.py 90 passed in 0.75s $ pre-commit run ruff --files # Passed $ pre-commit run black --files # Passed $ pre-commit run pydoclint --files # Passed $ pre-commit run buildifier --files python/ray/tests/BUILD.bazel # Passed $ pre-commit run buildifier-lint --files python/ray/tests/BUILD.bazel # Passed $ pre-commit run check-import-order docstyle python-no-log-warn ... # Passed ``` Environment: `3.0.0.dev0` nightly wheel (`eb5882b7ba`) with the local source tree symlinked in via `python/ray/setup-dev.py`, per the development guide. The `prettier` hook could not install in my environment (blocked npm registry); it only applies to `doc/` JS/CSS and does not cover any file in this PR. ### AI assistance AI assistance was used for this change. I reviewed every changed line, ran the tests and linters above locally, and can defend the change end-to-end. ### Duplicate check Searched `ray-project/ray` and `ray-project/kuberay` issues and PRs for this symptom before filing (`gh api search/issues` across several phrasings covering autoscaler + stderr/stdout + severity/log level). No existing issue or open PR covers it. The closest items are kuberay#4095 (closed by documenting `RAY_LOGGER_LEVEL`, which is not a fix here β€” it silences `monitor.log` as well, see the issue) and ray#48651 (broader, still open).", "url": "https://github.com/ray-project/ray/pull/65455", "createdAt": "2026-08-13T10:28:36Z", "updatedAt": "2026-08-13T13:31:22Z", "timestamp": "2026-08-13T13:31:22Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "kuberay", "core", "community-contribution", "kubernetes" ], "author": "vivekmahajan", "state": "open", "assignees": [] }, { "id": "github:ray-project/ray:pull_request:65456", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[doc] Pin the ReDoc bundle to v2.5.3 instead of tracking the CDN latest tag", "text": "## Why this change is needed `doc/source/conf.py` points `redoc_uri` at the ReDoc CDN's unpinned `latest` tag: ```python redoc_uri = \"https://cdn.redoc.ly/redoc/latest/bundles/redoc.standalone.js\" ``` `sphinxcontrib-redoc` injects that script tag alongside the spec; it doesn't bundle a renderer. So the published Jobs API page at `cluster/running-applications/job-submission/api.html` is rendered client-side, at page-view time, by whatever Redocly currently serves at `latest`. If a breaking change lands in that bundle, the live page degrades with no build-time signal at all, because the Sphinx build never executes the script. The build stays green while the page is broken. This pins the version so the renderer stops moving underneath the docs unobserved. ## Related changes None. This is deliberately scoped to the one line plus a comment. ## What changes - Pin `redoc_uri` to `v2.5.3`. - Add a comment recording why it's pinned and that bumping it requires a visual check on the Read the Docs preview, so a future dependency sweep doesn't helpfully restore `latest`. ## Why v2.5.3, and why this is a no-op today `v2.5.3` is what `latest` currently resolves to, so the pin doesn't silently downgrade the rendering. The two bundles are byte-identical, verified 2026-08-13: | URL | HTTP | Bytes | sha256 | | --- | --- | --- | --- | | `.../redoc/latest/bundles/redoc.standalone.js` | 200 | 1,097,271 | `1320f442151c57c447d3b70c7ffc6c4f86d08464020fe34c8cc5d3164e9944f0` | | `.../redoc/v2.5.3/bundles/redoc.standalone.js` | 200 | 1,097,271 | `1320f442151c57c447d3b70c7ffc6c4f86d08464020fe34c8cc5d3164e9944f0` | `2.5.3` is also the current `latest` dist-tag on npm. The next release line is `3.0.0-rc.0`, which is exactly the kind of bundle that would arrive through this URL unannounced and unverified today. For context on why the unpinned tag isn't hypothetical, `latest` has already moved well away from older 2.x builds: `v2.5.0` is 910,994 bytes against the current 1,097,271. ## Checks - [x] `doc/source/conf.py` parses; `pre-commit run` clean on the staged change. - [ ] Loaded `api.html` in the Read the Docs preview for this PR and confirmed the three-panel layout and the endpoint list render. Will confirm here once the preview build finishes β€” a green Sphinx build proves nothing for a client-rendered page. ## Out of scope Replacing the unmaintained `sphinxcontrib-redoc` extension, and the separate question of the spec's declared OpenAPI version. Both are tracked separately and neither blocks this pin, which is worth having regardless of how they land.", "url": "https://github.com/ray-project/ray/pull/65456", "createdAt": "2026-08-13T12:34:15Z", "updatedAt": "2026-08-13T13:30:14Z", "timestamp": "2026-08-13T13:30:14Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "core", "go", "docs-go" ], "author": "dstrodtman", "state": "open", "assignees": [] }, { "id": "github:ray-project/ray:pull_request:65457", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[doc] Remove invalid summary keys from Jobs API spec Response Objects", "text": "## Why this change is needed `summary` isn't a valid field on an OpenAPI Response Object, in either 3.0.x or 3.1. The valid fields are `description`, `headers`, `content`, and `links`. `doc/source/cluster/running-applications/job-submission/openapi.yml` carries 21 of them, and they make the document fail validation against the `3.1.0` version it declares: ``` $ openapi-spec-validator openapi.yml openapi.yml: Validation Error: Unevaluated properties are not allowed ('summary' was unexpected) Failed validating 'unevaluatedProperties' in schema['properties']['paths'][...]['responses'][...] $comment: https://spec.openapis.org/oas/v3.1.0#response-object On instance['paths']['/api/version']['get']['responses']['200'] ``` So the spec is invalid against its own declared version today. That matters because `rest.rst` explicitly invites readers to generate client libraries from this spec, and it blocks wiring a validation check into CI, since a validator that has to be run with known-failing output is one nobody will adopt. ## Related changes None. This is scoped to removing the invalid keys. ## What changes Removes the 21 `summary` keys from Response Objects. Nothing else β€” the diff is 21 deletions and zero insertions. The 8 operation-level `summary` fields are untouched. Those are valid on an Operation Object and they do render, as the endpoint titles in the sidebar and section headings. Only the response-level ones go. ## This does not change the published page ReDoc ignores unknown keys, so the rendered output is unaffected. Rather than assert that, I verified it against the live page by probing the rendered DOM after the ReDoc bundle ran: - All 21 summary values are **absent** from the rendered body text. The six that are unique to a response `summary` and appear nowhere else in the spec β€” `Job Submit Response`, `Job Submit Validation Error`, `Job Submit Internal Error`, `List of Job Details`, `Job Logs Internal Error`, `Wrong Job Type` β€” return no match. - They **are** present in the raw page HTML, because the spec is embedded in the page as JSON. The data reaches the browser; ReDoc simply doesn't render it. - The sibling `description` of those same Response Objects **does** render, which rules out the alternative explanation that those responses were never reached or stayed collapsed. For example, the rendered Responses block for `POST /api/jobs` reads: ``` Responses 200 The ID of the submitted job. 400 A TypeError or ValueError was raised when submitting the job. 500 An internal error occurred when submitting the job. ``` Status code and `description`, with the three `summary` values for those exact responses nowhere in the output. ## No documented content is lost Every one of the 21 affected responses already has a `description` that states the same thing in prose, and `description` is the required field on a Response Object. A few examples: | Path | Removed `summary` | Retained `description` | | --- | --- | --- | | `GET /api/version` 200 | `Version` | The Ray Jobs API version and the Ray version running on the cluster. | | `POST /api/jobs` 400 | `Job Submit Validation Error` | A TypeError or ValueError was raised when submitting the job. | | `DELETE /api/jobs/{submission_id}` 400 | `Wrong Job Type` | The job was not submitted via the Ray Jobs API, so it cannot be deleted. | Verified programmatically that all 21 responses retain a `description` after the edit, so none becomes invalid for the opposite reason. ## Checks - [x] Spec parses; the parsed document is identical to the previous one with exactly those 21 keys removed and no other difference. - [x] `openapi-spec-validator` now reports `OK` against the declared 3.1.0. It failed before this change. - [x] Rendered-DOM probe of the live page, described above. - [x] `pre-commit run` clean on the staged change. - [ ] Confirm `api.html` is unchanged in the Read the Docs preview for this PR. Will confirm here once the preview build finishes β€” this is a client-rendered page, so a green Sphinx build proves nothing about it. ## Note for reviewers The invalidity is pre-existing and isn't caused by any version change. It fails under 3.0.x too, so this cleanup is worth having independently of the separate open question about which OpenAPI version this document should declare. After this change the spec validates against both 3.1.0 and 3.0.3, which leaves that question free to be decided on its own merits.", "url": "https://github.com/ray-project/ray/pull/65457", "createdAt": "2026-08-13T12:43:16Z", "updatedAt": "2026-08-13T13:29:24Z", "timestamp": "2026-08-13T13:29:24Z", "metrics": { "reactions": 1, "comments": 1 }, "labels": [ "docs", "core", "go", "docs-go" ], "author": "dstrodtman", "state": "open", "assignees": [] }, { "id": "github:ray-project/ray:pull_request:65458", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[doc] Declare OpenAPI 3.0.3 in the Jobs API spec instead of 3.1.0", "text": "> **Stacked PR.** This targets `doc-1513-remove-invalid-response-summary` (#65457), not `master`, because that PR makes the spec valid and this one shouldn't inherit a failing validator run. Review #65457 first; GitHub retargets this to `master` automatically when it merges. The two changes are independent in substance and can be split if you'd rather take them separately. ## Why this change is needed `doc/source/cluster/running-applications/job-submission/openapi.yml` declares `openapi: 3.1.0` and uses not one construct that requires 3.1. Audited the full spec for 3.1-only constructs. Every check came back empty: | 3.1-only construct | Present? | | --- | --- | | `jsonSchemaDialect` | no | | top-level `webhooks` | no | | `$schema` | no | | `const` | no | | `examples` (the plural, JSON-Schema-aligned form) | no | | type arrays (`type: [string, \"null\"]`) | no | | `null` type | no | | `info.summary` | no | It also uses no `nullable`, the 3.0-only keyword it would need if it had ever expressed nullability. The complete set of keywords in the file is `description`, `type`, `summary`, `schema`, `content`, `title`, `required`, `properties`, `parameters`, `in`, `$ref`, `allOf`, `additionalProperties`, `enum`, `deprecated`, `items`, `requestBody`, `responses`, `get`, `post`, `delete`, `components`, `schemas`, `info`, `paths`, and `version`. Every one is valid in 3.0.x with identical semantics. So the document is already a valid 3.0.3 document that happens to declare 3.1.0. **Declaring the oldest version the document actually conforms to widens the set of tools that can consume it.** 3.1 support across the OpenAPI ecosystem still lags 3.0 β€” client generators, validators, and renderers handle 3.0.x completely and 3.1 partially. `rest.rst` explicitly invites readers to generate client libraries from this spec, so the declared version directly affects whether that works for them. Nothing is gained by declaring 3.1 while using no 3.1 feature. There's a concrete in-repo case for this too. The doc build currently renders this spec with `sphinxcontrib-redoc`, unmaintained since 2020, and the maintained alternative from the same org, `sphinxcontrib-openapi`, has only partial 3.1 support: - The PR that added it, [sphinx-contrib/openapi#138](https://github.com/sphinx-contrib/openapi/pull/138) (merged January 2023), is titled \"Add **initial** OpenAPI 3.1 support\" and its description says \"we focus on the MVP here\" with \"a whole lot more to support here including things like webhooks.\" - [sphinx-contrib/openapi#160](https://github.com/sphinx-contrib/openapi/issues/160) is an open, uncommented 3.1 rendering bug filed October 2024 β€” an unhandled `TypeError` in the `openapi31.py` code path. Declaring 3.0.3 removes that whole risk class from any future evaluation of that extension, at the cost of one line and no lost expressiveness. ## Related changes Stacked on #65457, which removes 21 invalid `summary` keys from Response Objects. That's a separate, version-independent validity fix; this PR is about the declared version only. ## What changes One line: `openapi: 3.1.0` β†’ `openapi: 3.0.3`. **`info.version` deliberately stays at `4.0.0`.** It tracks the Ray Jobs API version, not the OpenAPI document version. The two are easy to conflate, so to be explicit: this PR changes only the OpenAPI specification version the document claims to conform to, and nothing about the API itself. ## The published page is unaffected, verified ReDoc handles both versions, so no visual change is expected β€” but this is a client-rendered page, and a green Sphinx build proves nothing about it. So I rendered the spec through the same pinned ReDoc v2.5.3 bundle the build uses, at both declared versions, and compared: | Check | Result | | --- | --- | | Rendered body text | **Byte-identical** (10,561 chars each) | | Full-page screenshot | **Pixel-identical** (same sha256) | | JavaScript errors or warnings | **None**, at either version | | Endpoint content actually rendered | Confirmed at both, so \"identical\" isn't two identical failures | ## Checks - [x] `openapi-spec-validator` reports `OK` against the declared 3.0.3. - [x] Rendered-output comparison above. - [x] `pre-commit run` clean on the staged change. - [ ] Confirm `api.html` is unchanged in the Read the Docs preview for this PR. ## If you'd rather not take this Reasonable outcomes here include declining it. If the intent is that this spec should eventually use 3.1 features, say so and I'll close this β€” the useful part is then the recorded finding that nothing in the document depends on 3.1 today.", "url": "https://github.com/ray-project/ray/pull/65458", "createdAt": "2026-08-13T12:57:00Z", "updatedAt": "2026-08-13T13:28:34Z", "timestamp": "2026-08-13T13:28:34Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "docs", "core", "go", "docs-go" ], "author": "dstrodtman", "state": "open", "assignees": [] }, { "id": "github:ray-project/ray:pull_request:65459", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[doc] Record what blocks lifting the setuptools ceiling and replacing sphinxcontrib-redoc", "text": "> **Stacked PR.** Fourth in a chain: #65456 (pin ReDoc) β†’ #65457 (spec validity) β†’ #65458 (declare 3.0.3) β†’ this. It's independent in substance and touches only `doc/requirements-doc.txt`, so it can be retargeted to `master` if you'd rather take it on its own. ## Why this change is needed The `setuptools==80.9.0` pin in `doc/requirements-doc.txt` exists because `sphinxcontrib-redoc` imports `pkg_resources` at module load, and setuptools 82 removed it. The comment ends with \"Lifting this ceiling means replacing or vendoring that extension,\" which leaves two questions open: 1. Is `sphinxcontrib-redoc` actually the *only* thing in the doc environment that needs `pkg_resources`? If not, removing it wouldn't lift the ceiling anyway. 2. Is there a usable replacement? I evaluated both. This records the answers so the next person hitting this pin doesn't re-derive them. ## Related changes Fourth in the chain above. No functional dependency on the three below it. ## What changes A comment in `doc/requirements-doc.txt`. No behavior change, no dependency change, no build change. ## Finding 1: `sphinxcontrib-redoc` is the sole blocker Installed every extension `conf.py` loads into an environment with no setuptools and no importable `pkg_resources`, then imported each one: | Result | Extensions | | --- | --- | | Import cleanly | `sphinx.ext.autodoc`, `viewcode`, `napoleon`, `doctest`, `coverage`, `autosummary`, `intersphinx`, `sphinx_click.ext`, `sphinxemoji`, `sphinx_copybutton`, `sphinx_sitemap`, `myst_nb`, `sphinxcontrib.autodoc_pydantic`, `sphinx_remove_toctrees`, `sphinx_design`, `sphinx_docsearch`, `sphinx_collections`, `sphinxext.opengraph` | | **Fails** | `sphinxcontrib.redoc` β€” `ModuleNotFoundError: No module named 'pkg_resources'` | That failure is the control: it reproduces the exact documented breakage, which confirms the test has power rather than passing vacuously. So dropping that one extension is sufficient to lift the ceiling. ## Finding 2: the maintained replacement works, in exactly one configuration `sphinxcontrib-openapi` 0.9.0 (released 2026-02-10, `requires_python >=3.10`) is the actively maintained alternative from the same `sphinx-contrib` org. It needs no setuptools, and it **does** render this spec with full request and response schemas and **zero warnings under `-W`** β€” but only with both of these: - `openapi_default_renderer = \"httpdomain\"` in `conf.py`, then a plain `.. openapi::` directive. - Operation `summary` fields as plain scalars rather than YAML block scalars. Neither is obvious, and each failure mode is silent or fatal rather than self-explaining: | Configuration | Schema content | Warnings under `-W` | | --- | --- | --- | | default renderer (`httpdomain:old`) | **none** β€” paths, parameters, status codes only | 0 | | `httpdomain:old` + `:request:` | request bodies only, as a raw JSON blob | 1 β€” [#171](https://github.com/sphinx-contrib/openapi/issues/171) | | `httpdomain:old` + `:examples:` | full | 1 β€” [#172](https://github.com/sphinx-contrib/openapi/issues/172) | | `httpdomain`, block-scalar summaries | full | 2 β€” [#173](https://github.com/sphinx-contrib/openapi/issues/173), plus literal `**` on the page | | **`httpdomain`, plain-scalar summaries** | **full** | **0** | The literal-asterisk defect is worth calling out because it's triggered by this repo's own spec: two operations use `summary: |`, which leaves a trailing newline in the value that the renderer interpolates straight into `**...**`. Changing those two lines to plain scalars is sufficient. The three configurations that genuinely don't work are filed upstream with minimal reproducers. ## So what actually gates migrating Not dependency hygiene, and not warnings. It's that ReDoc's standalone three-panel page would be replaced by output rendered inline in the site theme. That's a visible change to a published reference page, so it's a maintainer call rather than a docs-tooling cleanup, and this PR doesn't make it. Two things worth weighing whenever that call happens. In favor: the current page is a client-rendered JavaScript island. It's in no toctree, it's reachable only through a hardcoded `` link at `rest.rst:94`, and its endpoint content isn't in the built HTML, so it doesn't participate in Sphinx search or cross-referencing. The trial build produced real Sphinx nodes and an `http-routingtable` index. Migrating would also let the `setuptools` ceiling lift, per finding 1. Against: it's a published reference page that readers may have bookmarked in its current form, and the inline rendering is a different reading experience rather than a strictly better one. My trial builds used the `basic` theme, so I can't show what it looks like under this site's actual theme without a full doc build. ## Checks - [x] Comment-only change; `pre-commit run` clean. - [x] Every claim in the comment verified by the trial builds described above.", "url": "https://github.com/ray-project/ray/pull/65459", "createdAt": "2026-08-13T13:30:11Z", "updatedAt": "2026-08-13T14:03:44Z", "timestamp": "2026-08-13T14:03:44Z", "metrics": { "reactions": 1, "comments": 1 }, "labels": [], "author": "dstrodtman", "state": "closed", "assignees": [] }, { "id": "github:ray-project/ray:pull_request:65460", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[doc] Render the Jobs API spec with sphinxcontrib-openapi instead of ReDoc", "text": "> **Stacked PR.** Targets `doc-1513-declare-openapi-303` (#65458), which targets #65457. Review order is #65457 β†’ #65458 β†’ this. GitHub retargets automatically as each merges. > > **Conflicts with #65456 by design.** That PR pins `redoc_uri` to a specific ReDoc version; this one deletes `redoc_uri` entirely. If #65456 lands first I'll resolve the conflict here. If this lands, #65456 becomes moot and should be closed. Either order works β€” #65456 is worth keeping as the safe fix in case this one stalls in review. ## Why this change is needed `sphinxcontrib-redoc` is unmaintained. 1.6.0 is the latest release and dates from 2020; the `2.0.0a1` alpha still imports `pkg_resources`. Keeping it costs this repo three things: | Cost | Location | | --- | --- | | A `setuptools<82` ceiling, because the extension imports `pkg_resources` at module load and setuptools 82 removed it | `doc/requirements-doc.txt` | | A `misc.copy_overwrite` entry in `suppress_warnings`, because the extension redundantly copies its own bundled asset | `doc/source/conf.py` | | The renderer fetched from a third-party CDN at page-view time | `doc/source/conf.py` | All of that buys exactly one page. The Jobs API spec is the only consumer: `conf.py` had one `redoc` entry and there's no other OpenAPI document in the repo. There's a second, larger reason. **The current page is a client-rendered JavaScript island.** ReDoc has no source file at all β€” the extension synthesizes `api.html` from `conf.py`. So today the page: - is in **no toctree**, and - is reachable only through a hardcoded `` `... `_ `` link at `rest.rst:94`, and - has **none of its endpoint content in the built HTML**, so it can't participate in Sphinx search or cross-referencing. ## Related changes Fourth in the chain above. Supersedes the \"record why we stay on ReDoc\" PR (#65459), which I'll close β€” its conclusion is obsolete now that this works. ## What changes - Swap `sphinxcontrib.redoc` β†’ `sphinxcontrib.openapi` (0.9.0) in `conf.py` and `doc/requirements-doc.txt`, and regenerate the deplock. - Add a real `api.md`, put it in the `job-submission` toctree, and replace the raw `` link with a `:ref:`. - Drop the now-unneeded `misc.copy_overwrite` suppression. - Change two operation `summary` fields from YAML block scalars to plain scalars (see below). - Update the `setuptools` comment: the `<82` ceiling's cause is gone. The pin itself is left in place, since unpinning is a separate resolve worth verifying on its own. `sphinxcontrib-openapi` is pure Python, from the same `sphinx-contrib` org, and needs no `pkg_resources`. The deplock diff is `-sphinxcontrib-redoc` and `+sphinxcontrib-openapi` plus its five dependencies (`sphinxcontrib-httpdomain`, `sphinx-mdinclude`, `deepmerge`, `mistune`, `picobox`). ## Verification Built with `make -C doc rtd-build`, which runs the RtD-faithful clean build with `-W`, matching `fail_on_warning: true`: ``` build succeeded. ``` Exit 0, and **zero `WARNING:`/`ERROR:` lines in the whole log**. `raydepsets --check` on `docbuild_depset_311` reports \"Lock files are up to date.\" Checked against the built `api.html` rather than inferred: | Check | Result | | --- | --- | | Request/response schemas rendered | Yes β€” `Request JSON Object` and `Response JSON Object` field lists, with `entrypoint`, `runtime_env`, `metadata`, and the rest, including types and required flags | | `$ref` and `allOf` resolution | Resolved, including through `JobDetails` | | Literal `**` on the page | None β€” zero docutils `problematic` nodes | | `http-routingtable` index | Generated | | Page in the sidebar, with prev/next pagination | Yes | | MyST page equivalent to the `.rst` draft it replaced | Yes β€” 8 httpdomain operation objects, identical `Parameters` / `Request JSON Object` / `Response JSON Object` / `Status Codes` field lists, no leaked rST, and both `:ref:` directions resolving with the right link text | ## Three non-obvious configuration details All three are commented at their sites in the diff, because all three fail quietly. **The renderer is selected via `openapi_default_renderer`, not by the directive spelling.** Writing `.. openapi:httpdomain::` in the page adds an `unknown directive name` warning, which is fatal under `fail_on_warning`. Selecting it in `conf.py` and using a plain `.. openapi::` avoids that. **The default renderer emits no request or response schemas at all** β€” just paths, parameters, and status codes. So leaving `openapi_default_renderer` unset would silently drop most of the reference content while still building green. That's the trap worth knowing about if anyone touches this later. **The `openapi` directive can't be a bare MyST fence; it has to be wrapped in `eval-rst`.** `sphinxcontrib-openapi` emits reStructuredText into a `ViewList` and hands it to `nested_parse`, but MyST's `nested_parse` renders that content as Markdown. A ```` ```{openapi} ```` fence therefore builds green with zero warnings while silently degrading the entire page: every `.. http:get::` and `:resjson:` line renders as literal text, so nothing becomes an httpdomain object and the page has no field lists, no anchors, and no search or cross-referencing. That's a worse failure than the one above, and just as quiet. ## Why the summary fields change `sphinxcontrib-openapi`'s renderer interpolates an operation's `summary` into `**...**` without stripping it. A YAML block scalar (`summary: |`) leaves a trailing newline in the value, so the markup opens on one line and closes on the next, docutils can't pair it, and readers see literal asterisks. Filed upstream as [sphinx-contrib/openapi#173](https://github.com/sphinx-contrib/openapi/issues/173). Two operations used block scalars (`GET /api/version` and `POST /api/jobs`); the other six already used plain scalars. Converting those two is the whole fix and changes no rendered text. ## What this gives up, honestly **Request and response samples.** ReDoc rendered example payloads; this doesn't. It isn't a configuration choice β€” both routes to examples are blocked by upstream bugs, and I've confirmed each against this spec: - `:generate-examples-from-schemas:` crashes with `ValueError: dictionary update sequence element #0 has length 1; 2 is required` ([#166](https://github.com/sphinx-contrib/openapi/issues/166)). Example generation can't handle an `allOf` that targets a non-object schema, which is exactly how `JobDetails` attaches descriptions to the `JobType` and `JobStatus` string enums. - `:response-examples-for:` is rejected as an unknown option ([#165](https://github.com/sphinx-contrib/openapi/issues/165)). `api.md` carries a comment recording this so nobody re-derives it, and I've added the reproduction detail to both upstream issues. Weighed against gaining search, cross-referencing, navigation, a dependency that's actually maintained, and no third-party CDN at page-view time, losing sample payloads seemed like the right trade β€” but it is a real loss, and it's reasonable to want the upstream fixes first instead. I've also filed [#171](https://github.com/sphinx-contrib/openapi/issues/171), [#172](https://github.com/sphinx-contrib/openapi/issues/172), and [#173](https://github.com/sphinx-contrib/openapi/issues/173), each with a minimal reproducer, and am planning PRs for them. ## Checks - [x] `make -C doc rtd-build`: build succeeded, zero warnings. - [x] `raydepsets --check`: lock files up to date. - [x] Built page inspected for schema content, `allOf`/`$ref` resolution, and stray markup. - [x] `pre-commit run` clean. - [ ] Read the Docs preview reviewed for the appearance change.", "url": "https://github.com/ray-project/ray/pull/65460", "createdAt": "2026-08-13T14:03:32Z", "updatedAt": "2026-08-13T16:25:52Z", "timestamp": "2026-08-13T16:25:52Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [], "author": "dstrodtman", "state": "open", "assignees": [] }, { "id": "github:ray-project/ray:pull_request:65461", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[doc] llms.txt: tell agents the page links also serve Markdown", "text": "## Summary Follow-up to #64458. Adds one line to `llms.txt` telling agents that the page links also serve Markdown. Read the Docs serves a Markdown rendering of any page **from that page's own `.html` URL**, under an `Accept: text/markdown` request. There is no separate `.md` file to link to β€” `page.md`, `page.html.md`, `/md/.md`, and `?format=md` all 404, and no `Link: rel=alternate` header advertises one. So the `.html` links `llms.txt` already emits *are* the Markdown links. Nothing in the file says so, and an agent has no way to discover it. ``` # Ray > Ray is an open-source unified compute framework for scaling AI and Python workloads… Full page text, grouped by section, is in [llms-full.txt](https://docs.ray.io/en/master/llms-full.txt). Request the links that follow with the HTTP header `Accept: text/markdown` to retrieve a Markdown rendering of the page. ## Overview … ``` ## The header has to be explicit Measured against `docs.ray.io/en/master/data/data.html`: | `Accept:` sent | returns | |---|---| | `*/*` β€” the default for curl, `requests`, `httpx` | `text/html` | | `text/html,application/xhtml+xml,*/*;q=0.8` β€” browser default | `text/html` | | `text/plain` | `text/html` | | `text/markdown` | **`text/markdown`** | | `text/markdown, */*` | **`text/markdown`** | | `text/markdown;q=1.0, text/html;q=0.9` | **`text/markdown`** | A wildcard isn't enough; the type has to be named. A client gets Markdown only by asking deliberately, so the pointer has a real audience. ## Config-gated New `llms_txt_markdown_hint`, **default `False`**. Content negotiation is a property of the host, not of the generator β€” a build served from somewhere that doesn't negotiate Markdown would otherwise advertise a representation that doesn't exist. `conf.py` sets it `True` for Ray. ## Wording note It says \"a Markdown rendering of **the page**\", deliberately not \"the source\". What comes back is the built page converted to Markdown β€” front matter, then site chrome (`Skip to main content`, theme switcher, version picker), then the content. That is not what `_sources/..txt` returns, and `_sources` genuinely exists, so calling it \"the source\" would point agents at the wrong artifact. ## Why not link `_sources/..txt` instead Considered and rejected: - **It leaks the source format to the consumer.** master is **363 `.rst` to 232 `.md`**, so most pages would hand back reStructuredText, not Markdown β€” noisier for an agent than the HTMLβ†’Markdown conversion. - **The URL isn't derivable from the page URL** β€” it depends on each page's source suffix (`index.md.txt` vs `installation.rst.txt`). - **It isn't human-clickable.** - **It would churn.** As pages migrate to MyST, each link would change from `.rst.txt` to `.md.txt`. Negotiation happens on the rendered HTML, so it's identical for both source formats and survives the migration. - **It's redundant** β€” the verbatim source is already in the `llms-full.txt` shards. ## Testing ``` $ python doc/source/_ext/test_llms_txt.py PASS: llms_txt extension produced a correct index, Optional section, llms-full shards, excludes notebooks by source type, gates the Markdown pointer, and honors the llms_txt_build gate. $ ruff check doc/source/_ext/llms_txt.py doc/source/_ext/test_llms_txt.py All checks passed! ``` New `test_markdown_hint` covers both directions: the pointer appears in the header (asserted to sit before the first `##` section) when the config is on, and is absent when it's off. The header table above was produced by probing the live `docs.ray.io`. All three changed files route to `@ doc doc_api` under the current `.buildkite/test.rules.txt`, so `doc: test llms.txt extension` runs on this PR. Not a duplicate: `gh pr list --state open` finds no other open PR touching `llms.txt` or Markdown content negotiation. ## AI assistance Drafted with Claude Code. The behavioural claims here (which `Accept` values return what, which `.md` URL forms 404, the negotiated output including page chrome) were each verified against the live site rather than assumed.", "url": "https://github.com/ray-project/ray/pull/65461", "createdAt": "2026-08-13T15:06:22Z", "updatedAt": "2026-08-13T15:31:51Z", "timestamp": "2026-08-13T15:31:51Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "go", "docs-go" ], "author": "ronny-anyscale", "state": "open", "assignees": [] }, { "id": "github:ray-project/ray:pull_request:65462", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[docs] Sync the vendored KubeRay CRD API reference", "text": "## Why are these changes needed? Vendors the generated KubeRay CRD API reference at `doc/source/cluster/kubernetes/references/api.md` from `ray-project/kuberay` release tag v1.6.2, pinned at commit [`0a9eabd`](https://github.com/ray-project/kuberay/blob/0a9eabde561ab59ab4bbe626072c359da9214484/docs/reference/api.md). Opened automatically. The upstream file is a build artifact that `elastic/crd-ref-docs` generates from the Go CRD types in `ray-operator/apis/ray/`, and KubeRay's `ray-operator-verify-api-docs` CI job fails on drift between the two, so the content is verified upstream before it reaches here. That job covers `release-*` branches as well as `master`, so a release tag's artifact carries the same guarantee. This syncs from the latest KubeRay *release*, not from `master`. Master carries CRD surface that no release ships, and documenting it here would advertise fields a reader can't use. The `ray.io/v1alpha1` section is stripped on the way in. That API version is deprecated and on a removal runway (ray-project/kuberay#5124), and its cross-links collide with the v1 anchors (ray-project/kuberay#5091). With one package left, the `Packages` index and the `ray.io/v1` heading restate the page's own subject, so both are dropped and the headings under them move up to h2. Sphinx's right-hand page nav lists h2 only, so this is what puts the CRD types in it. Heading text is untouched, so the anchors cross-links resolve against don't move. A page intro replaces the generated package blurb those headings carried, naming the resource types read off the page's own index. ## Checks Content-only: the single changed file is documentation under `doc/`, so the `docs-go` scope guard applies cleanly. The Read the Docs preview build on this PR is the gate β€” a green build means the regenerated reference still renders and all of its in-page anchors resolve. Nothing else in this PR. If it contains any other change, something is wrong with the sync job.", "url": "https://github.com/ray-project/ray/pull/65462", "createdAt": "2026-08-13T16:20:13Z", "updatedAt": "2026-08-13T16:23:14Z", "timestamp": "2026-08-13T16:23:14Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "docs", "docs-go" ], "author": "dstrodtman", "state": "open", "assignees": [] }, { "id": "github:ray-project/ray:pull_request:65463", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "core: release runtime env per-job loggers", "url": "https://github.com/ray-project/ray/pull/65463", "createdAt": "2026-08-13T17:27:57Z", "updatedAt": "2026-08-13T17:32:13Z", "timestamp": "2026-08-13T17:32:13Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [], "author": "rohitsalla", "state": "open", "assignees": [] } ], "events": [ { "id": "event:bd946fe3fc031f69fac7", "signalId": "github:ray-project/ray:pull_request:65459", "event": "discovered", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65459", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[doc] Record what blocks lifting the setuptools ceiling and replacing sphinxcontrib-redoc", "text": "> **Stacked PR.** Fourth in a chain: #65456 (pin ReDoc) β†’ #65457 (spec validity) β†’ #65458 (declare 3.0.3) β†’ this. It's independent in substance and touches only `doc/requirements-doc.txt`, so it can be retargeted to `master` if you'd rather take it on its own. ## Why this change is needed The `setuptools==80.9.0` pin in `doc/requirements-doc.txt` exists because `sphinxcontrib-redoc` imports `pkg_resources` at module load, and setuptools 82 removed it. The comment ends with \"Lifting this ceiling means replacing or vendoring that extension,\" which leaves two questions open: 1. Is `sphinxcontrib-redoc` actually the *only* thing in the doc environment that needs `pkg_resources`? If not, removing it wouldn't lift the ceiling anyway. 2. Is there a usable replacement? I evaluated both. This records the answers so the next person hitting this pin doesn't re-derive them. ## Related changes Fourth in the chain above. No functional dependency on the three below it. ## What changes A comment in `doc/requirements-doc.txt`. No behavior change, no dependency change, no build change. ## Finding 1: `sphinxcontrib-redoc` is the sole blocker Installed every extension `conf.py` loads into an environment with no setuptools and no importable `pkg_resources`, then imported each one: | Result | Extensions | | --- | --- | | Import cleanly | `sphinx.ext.autodoc`, `viewcode`, `napoleon`, `doctest`, `coverage`, `autosummary`, `intersphinx`, `sphinx_click.ext`, `sphinxemoji`, `sphinx_copybutton`, `sphinx_sitemap`, `myst_nb`, `sphinxcontrib.autodoc_pydantic`, `sphinx_remove_toctrees`, `sphinx_design`, `sphinx_docsearch`, `sphinx_collections`, `sphinxext.opengraph` | | **Fails** | `sphinxcontrib.redoc` β€” `ModuleNotFoundError: No module named 'pkg_resources'` | That failure is the control: it reproduces the exact documented breakage, which confirms the test has power rather than passing vacuously. So dropping that one extension is sufficient to lift the ceiling. ## Finding 2: the maintained replacement works, in exactly one configuration `sphinxcontrib-openapi` 0.9.0 (released 2026-02-10, `requires_python >=3.10`) is the actively maintained alternative from the same `sphinx-contrib` org. It needs no setuptools, and it **does** render this spec with full request and response schemas and **zero warnings under `-W`** β€” but only with both of these: - `openapi_default_renderer = \"httpdomain\"` in `conf.py`, then a plain `.. openapi::` directive. - Operation `summary` fields as plain scalars rather than YAML block scalars. Neither is obvious, and each failure mode is silent or fatal rather than self-explaining: | Configuration | Schema content | Warnings under `-W` | | --- | --- | --- | | default renderer (`httpdomain:old`) | **none** β€” paths, parameters, status codes only | 0 | | `httpdomain:old` + `:request:` | request bodies only, as a raw JSON blob | 1 β€” [#171](https://github.com/sphinx-contrib/openapi/issues/171) | | `httpdomain:old` + `:examples:` | full | 1 β€” [#172](https://github.com/sphinx-contrib/openapi/issues/172) | | `httpdomain`, block-scalar summaries | full | 2 β€” [#173](https://github.com/sphinx-contrib/openapi/issues/173), plus literal `**` on the page | | **`httpdomain`, plain-scalar summaries** | **full** | **0** | The literal-asterisk defect is worth calling out because it's triggered by this repo's own spec: two operations use `summary: |`, which leaves a trailing newline in the value that the renderer interpolates straight into `**...**`. Changing those two lines to plain scalars is sufficient. The three configurations that genuinely don't work are filed upstream with minimal reproducers. ## So what actually gates migrating Not dependency hygiene, and not warnings. It's that ReDoc's standalone three-panel page would be replaced by output rendered inline in the site theme. That's a visible change to a published reference page, so it's a maintainer call rather than a docs-tooling cleanup, and this PR doesn't make it. Two things worth weighing whenever that call happens. In favor: the current page is a client-rendered JavaScript island. It's in no toctree, it's reachable only through a hardcoded `` link at `rest.rst:94`, and its endpoint content isn't in the built HTML, so it doesn't participate in Sphinx search or cross-referencing. The trial build produced real Sphinx nodes and an `http-routingtable` index. Migrating would also let the `setuptools` ceiling lift, per finding 1. Against: it's a published reference page that readers may have bookmarked in its current form, and the inline rendering is a different reading experience rather than a strictly better one. My trial builds used the `basic` theme, so I can't show what it looks like under this site's actual theme without a full doc build. ## Checks - [x] Comment-only change; `pre-commit run` clean. - [x] Every claim in the comment verified by the trial builds described above.", "url": "https://github.com/ray-project/ray/pull/65459", "createdAt": "2026-08-13T13:30:11Z", "updatedAt": "2026-08-13T13:42:55Z", "timestamp": "2026-08-13T13:42:55Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [], "author": "dstrodtman", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:04ffe6c4a62803f1ce74", "signalId": "github:ray-project/ray:pull_request:64601", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:64601", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[Data] Fix ResourceBudget backpressure causing pipeline stall", "text": "## Why are these changes needed? Two interacting bugs in `ResourceBudgetBackpressurePolicy` can cause a Ray Data pipeline to stall indefinitely β€” CPU/GPU idle, Object Store filling up, no error logged, no progress. Reproducing this requires enough intermediate output to pressure the Object Store; small pipelines won't hit it. ### Background `ResourceBudgetBackpressurePolicy` blocks an operator from submitting new tasks when **either**: (a) the operator's `incremental_resource_usage` (CPU/GPU/memory) exceeds its budget, or (b) the operator's Object Store budget for task outputs is below `obj_store_mem_max_pending_output_per_task`. When either fires, the operator enters `backpressured:tasks(ResourceBudget)` and stops scheduling. ### The two bugs **Bug 1 β€” `OutputBackpressureGuard.should_unblock` doesn't distinguish Object Store–blocked from CPU/GPU-blocked.** The liveness relaxation in `should_unblock` relaxes an upstream operator's output backpressure when a downstream operator has no active tasks and can't submit β€” so upstream can finish and release CPU/GPU/memory. This helps when downstream is blocked on condition (a). But when downstream is blocked on condition (b) β€” its Object Store budget for task outputs is exhausted β€” relaxing upstream only writes *more* output into the already-full store while downstream still can't schedule. The relaxation feeds the stall it's meant to prevent. **Bug 2 β€” `update_budgets` unconditionally allocates shared resources to over-reserved operators.** Each scheduling iteration, `update_budgets` recomputes every eligible operator's budget as its unused reservation plus a share of the remaining shared resources. Even when an operator's Object Store usage already exceeds its reserved resources (unused reservation is zero), it still receives a share of the remaining shared resources β€” restoring a positive budget, letting it submit more tasks, and further draining the shared resources that other operators depend on. **How they compound:** Bug 2 keeps the upstream operator's budget alive so it keeps producing output. Bug 1 relaxes the upstream's output backpressure so that output flows unchecked. Together, the Object Store fills until every operator's budget is exhausted and the pipeline stalls. ## Fix 1. In `should_unblock`, skip the liveness relaxation when the downstream operator is specifically blocked on Object Store budget for task outputs (new predicate `is_task_submission_blocked_on_object_store`). When downstream is blocked on CPU/GPU (condition a), the original relaxation is preserved. `IdleDetector` remains the bounded liveness fallback for a genuinely stalled pipeline. Uses `continue` (not `return False`) so that when multiple downstream operators exist, an Object Store–blocked one is skipped while a CPU/GPU-blocked one can still trigger relaxation. 2. Optionally withhold shared Object Store resources from operators that satisfy **both**: (a) per-operator usage exceeds `object_store_reservation_overshoot_ratio` Γ— its total reserved resources (task-side + output-side), AND (b) global Object Store usage exceeds `object_store_pool_pressure_fraction` of the cluster limit. Both thresholds must be set to enable the throttle; either being `None` (default) preserves legacy behavior. Requiring both conditions avoids throttling when the Object Store is nearly empty (a single overshooting operator is harmless there) and avoids letting one operator drain the shared pool when the store is under pressure β€” a single threshold would misfire in one direction or the other. Fix 2 pre-filters over-reserved operators before the shared-resource allocation loop (rather than skipping with `continue` inside the loop), keeping the per-operator share fraction aligned with the number of operators actually receiving resources. The fallback branch also iterates the pre-filtered list, preventing leftover budget from being routed back to an over-reserved operator. ## Related issues None. ## Additional information **Enabling the throttle (Fix 2)** β€” off by default. To enable, set both thresholds on `DataContext`: ```python ctx = ray.data.DataContext.get_current() ctx.object_store_reservation_overshoot_ratio = 1.5 # per-op overshoot headroom ctx.object_store_pool_pressure_fraction = 0.8 # engage at 80% pool usage ``` Or via environment variables (equivalent): ```bash export RAY_DATA_OBJECT_STORE_RESERVATION_OVERSHOOT_RATIO=1.5 export RAY_DATA_OBJECT_STORE_POOL_PRESSURE_FRACTION=0.8 ``` **Tests** (`test_resource_manager.py`): `TestOutputBackpressureGuard` (Fix 1): - `test_no_unblock_when_downstream_blocked_on_object_store` - `test_is_task_submission_blocked_on_object_store` `TestReservationOpResourceAllocator` (Fix 2): - `test_is_op_over_reserved_on_object_store` - `test_update_budgets_splits_shared_evenly_when_op_over_reserved`", "url": "https://github.com/ray-project/ray/pull/64601", "createdAt": "2026-07-08T15:23:34Z", "updatedAt": "2026-08-13T13:38:53Z", "timestamp": "2026-08-13T13:38:53Z", "metrics": { "reactions": 1, "comments": 2 }, "labels": [ "data", "go", "community-contribution" ], "author": "dragongu", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:0164561a2073036bb21f", "signalId": "github:ray-project/ray:issue:65451", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:issue:65451", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "issue", "title": "[Core][runtime_env] RuntimeEnv agent leaks one logger (and its file descriptors) per job, eventually failing all setups with EMFILE`", "text": "### What happened The RuntimeEnv Agent caches one logger per job in `_per_job_logger_cache` (`python/ray/_private/runtime_env/agent/runtime_env_agent.py:209`), created via `setup_component_logger` in `get_or_create_logger` (`:304-313`). Each logger eagerly opens one `RotatingFileHandler` for `runtime_env_setup-.log` plus one per entry in the runtime env config's `log_files` β€” i.e. `1 + len(log_files)` file descriptors per job. Nothing ever releases them: - The cache has no eviction and no `close()` call anywhere in the file. - The logger is created through `logging.getLogger` (`ray_logging/__init__.py:80`), so the global logging registry holds a second strong reference β€” even clearing the cache would not free the handlers. - The env-level result cache does not bound the growth: `ReferenceTable` deletes a cached env when its last reference goes away (`runtime_env_agent.py:286-302`), so the next job re-runs setup under a new `job_id` and opens a fresh logger. Open FDs therefore grow with the **cumulative number of jobs the node has ever handled**, not with current setup concurrency. Restarting the agent is the only reset. Once the process `nofile` limit is reached, the agent can no longer create log files, download packages, create temp files, or start subprocesses, and every subsequent runtime env setup fails with: ``` OSError: [Errno 24] Too many open files: '/tmp/ray/session_.../logs/runtime_env_setup-.log' ```", "url": "https://github.com/ray-project/ray/issues/65451", "createdAt": "2026-08-13T08:07:01Z", "updatedAt": "2026-08-13T13:38:03Z", "timestamp": "2026-08-13T13:38:03Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "core", "stability", "community-backlog" ], "author": "zzchun", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:874dd7c6107cdec421b6", "signalId": "github:ray-project/ray:issue:65454", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:issue:65454", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "issue", "title": "[Core][KubeRay] Autoscaler sends all log records, including INFO, to stderr", "text": "## What happened + What you expected to happen `_setup_logging()` in `python/ray/autoscaler/_private/kuberay/run_autoscaler.py` attaches a single `logging._StderrHandler` to the root logger, so **every** autoscaler log record β€” INFO included β€” is written to the container's **stderr**. The function's own docstring says it should be stdout: ```python def _setup_logging(log_dir: str) -> None: \"\"\"Log to autoscaler log file (typically, /tmp/ray/session_latest/logs/monitor.*) Also log to pod stdout (logs viewable with `kubectl logs -c autoscaler`). ``` but the implementation writes stderr, and nothing writes to stdout at all: ```python level = logging.getLevelName(ray_constants.LOGGER_LEVEL.upper()) # default \"info\" stderr_handler = logging._StderrHandler() stderr_handler.setFormatter(logging.Formatter(ray_constants.LOGGER_FORMAT)) stderr_handler.setLevel(level) logging.root.setLevel(level) logging.root.addHandler(stderr_handler) # The stdout handler was set up in the Ray CLI entry point. # See ray.scripts.scripts::cli(). ``` **That trailing comment is stale.** `ray.scripts.scripts::cli()` calls `ray._private.ray_logging.setup_logger()`, which only re-sets the level and re-formats handlers already present on the `ray` logger β€” it never calls `addHandler`. And `setup_component_logger()`, called a few lines earlier in this same function, calls `ray._private.log.clear_logger(\"ray\")`, which *empties* the `ray` logger's handlers. The reproduction script below prints the resulting handler list; no handler targets stdout. ### Why this matters Kubernetes' CRI log format records the stream each line arrived on, per line: ``` TIMESTAMP STREAM TAG CONTENT 2016-10-06T00:17:09.669794202Z stdout P log content 1 ``` () Log collectors commonly derive a line's severity from that `STREAM` field rather than parsing the `INFO` token out of the message body, so on a KubeRay cluster with log collection routine autoscaler output is ingested as errors. The measurement below is consistent with that. Measured on a production KubeRay cluster (Datadog), autoscaler sidecar, **idle** cluster, 20 minutes: | | count | |---|---| | `status:error` | 2392 | | `status:info` | 0 | | `status:warn` | 0 | Over 24h the same container logged 63,048 `skipping idle termination` lines and zero scale events, drains, patches or exceptions. Every one of those lines was classified as an error, so the container's error rate carries no signal and cannot be alerted on. ### Expected Sub-WARNING records on stdout, WARNING and above on stderr β€” the conventional split β€” so severity survives log collection. `monitor.log` should keep receiving every level, since it backs the dashboard and the State API (`/api/v0/logs`). ### Origin This looks like an unintended regression from #34182 (\"Global logging format changes\"). Before that PR the code read: ```python logger_name=\"ray\", # Root of the logging hierarchy for Ray code. ) # Logs will also be written to the container's stdout. # The stdout handler was set up in the Ray CLI entry point. ``` #34182 removed the `# Logs will also be written to the container's stdout.` line, added the `_StderrHandler`, and left the `# The stdout handler was set up in the Ray CLI entry point.` line dangling. The intent recorded in the original comment (and still in the docstring) was stdout. Note that #33652, which introduced `logging._StderrHandler` for Rich `FileProxy` compatibility, touched only `python/ray/_private/ray_logging.py` β€” it never touched `run_autoscaler.py`, and it was not a severity-routing decision. Same bug class as #32339 (\"[Serve][Dashboard] Ray Serve actor logs show up in .err files instead of .out files\"), which was accepted as `bug`/`P2`/`observability`. Adjacent to #48651. ## Versions / Dependencies - Ray: reproduced on `master` (`9c6ffee9a5`) and on the `3.0.0.dev0` nightly (`eb5882b7ba`). Present since #34182 (Ray 2.4). - Python: 3.12 - OS: reproduced on macOS locally; observed in production on Linux / KubeRay. ## Reproduction script No cluster required β€” this calls the logging setup directly and reports which stream each level lands on. ```python import io, logging, os, sys, tempfile from ray.autoscaler._private.kuberay.run_autoscaler import _setup_logging logging.root.handlers.clear() tmp = tempfile.mkdtemp() out, err = io.StringIO(), io.StringIO() real_out, real_err = sys.stdout, sys.stderr sys.stdout, sys.stderr = out, err try: _setup_logging(tmp) log = logging.getLogger(\"ray.autoscaler._private.kuberay.run_autoscaler\") log.info(\"INFO-MARKER skipping idle termination\") log.warning(\"WARN-MARKER\") for h in logging.root.handlers: h.flush() finally: sys.stdout, sys.stderr = real_out, real_err print(\"root handlers:\", [type(h).__name__ for h in logging.root.handlers]) print(\"stdout bytes:\", len(out.getvalue())) print(\"INFO on stdout?\", \"INFO-MARKER\" in out.getvalue()) print(\"INFO on stderr?\", \"INFO-MARKER\" in err.getvalue()) print(\"INFO in monitor.log?\", \"INFO-MARKER\" in open(os.path.join(tmp, \"monitor.log\")).read()) ``` Actual output: ``` root handlers: ['RotatingFileHandler', '_StderrHandler'] stdout bytes: 0 INFO on stdout? False INFO on stderr? True INFO in monitor.log? True ``` Expected: `INFO on stdout? True`, `INFO on stderr? False`, `monitor.log` unchanged. ### Why `RAY_LOGGER_LEVEL=warning` is not a workaround Raising the level silences `monitor.log` too. `setup_component_logger()` is gated on the same `ray_constants.LOGGER_LEVEL`, and `logging.root.setLevel(level)` runs after it, so INFO calls short-circuit in `Logger.isEnabledFor` and never reach the rotating file handler. That removes the dashboard/State API copy of the logs, which is the copy you actually want to keep. (Raised in ray-project/kuberay#4095, documented in ray-project/kuberay#4104.) ## Issue Severity Medium: It is a significant difficulty but I can work around it.", "url": "https://github.com/ray-project/ray/issues/65454", "createdAt": "2026-08-13T10:24:49Z", "updatedAt": "2026-08-13T13:37:06Z", "timestamp": "2026-08-13T13:37:06Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "usability", "core", "observability", "stability", "community-backlog" ], "author": "vivekmahajan", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:fda3993a2ced6a9a0a24", "signalId": "github:ray-project/ray:pull_request:65405", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65405", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "fix(jobs): return structured 503 when job logs are unavailable", "text": "## Summary - return a structured `503 Service Unavailable` when a submission job still exists but its driver agent is no longer registered or reachable - preserve successful log retrieval, unknown-job `404` behavior, and the public SDK's existing `RuntimeError` contract - document the JSON error response and bump the Jobs API version The response does not imply that node-local logs are persisted or recoverable. This is a draft pending maintainer confirmation that `503` is the preferred API contract. Fixes #65402 ## Test plan - [x] `pytest -q python/ray/dashboard/modules/job/tests/test_http_job_server.py -k 'get_job_logs_from_live_agent or get_job_logs_when_driver_node_is_missing or get_job_logs_when_driver_agent_is_unreachable or get_job_logs_unknown_job_preserves_404'` - [x] `pytest -q python/ray/dashboard/modules/job/tests/test_sdk.py -k job_logs_after_driver_node_removed` - [x] `pytest -vv python/ray/dashboard/modules/job/tests/test_job_agent.py -k job_log_in_multiple_node --maxfail=1` - [x] `ruff check` and `ruff format --check` on changed Python files - [x] Python compilation and OpenAPI YAML parsing - [x] Upstream [microcheck](https://buildkite.com/ray-project/microcheck/builds/51960) passed. - [ ] The two-node E2E was not independently rerun locally because this checkout lacks compiled `ray._raylet`. The public microcheck summary does not expose individual job execution, so the PR remains draft pending CI or maintainer confirmation that it ran.", "url": "https://github.com/ray-project/ray/pull/65405", "createdAt": "2026-08-12T00:32:39Z", "updatedAt": "2026-08-13T13:35:55Z", "timestamp": "2026-08-13T13:35:55Z", "metrics": { "reactions": 1, "comments": 3 }, "labels": [ "docs", "core", "community-contribution" ], "author": "nataliekung", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:e0ce62fd14461240c3aa", "signalId": "github:ray-project/ray:pull_request:65422", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65422", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[Data][LLM] Add multi-host TPU batch inference for Ray Data LLM", "text": "## Summary This PR adds multi-host TPU support to Ray Data LLM batch inference (`vLLMEngineProcessorConfig` / `build_processor`). With a TPU `accelerator_type` and a topology in `accelerator_config`, `build_processor` reserves one slice placement group for the processor, places the engine actor on bundle 0 (CPU only, `capture_child_tasks=True`), and releases the slice via `Processor.close()`. Accelerator config/backends live in `python/ray/llm/_internal/common/accelerators.py` and are shared with Serve. Batch scheduling now resolves through `AcceleratorBackend.build_batch_scheduling_options`, so the builder calls the backend unconditionally and `vLLMEngineStage.post_init` is removed. The GPU path keeps its existing behavior (same bundles, same PACK placement group per actor), just built in `GPUAccelerator` instead of the stage. ```python config = vLLMEngineProcessorConfig( model_source=\"...\", accelerator_type=\"TPU-V6E\", accelerator_config={\"kind\": \"tpu\", \"topology\": \"4x4\"}, # TP * PP * DP must equal topology chip count (PP/DP default to 1). engine_kwargs={\"tensor_parallel_size\": 16}, concurrency=1, ) with build_processor(config) as processor: out = processor(ds) out.materialize() # finish before close / leaving the with-block ``` A `chips_per_vm` field is also added for the TPU backend to support ambiguous topologies (v6e `2x4`, etc.). This field is already accepted by `SlicePlacementGroup` so I just added it to the parameters users can specify for TPU in their config. ## Behavior changes - `accelerator_type=\"CPU\"` is rejected for vLLM batch inference - `get_accelerator_backend` raises on unrecognized configs - TPU fill constraint is `tensor_parallel_size * pipeline_parallel_size * data_parallel_size == topology chips` (PP/DP default to 1). This diverges from GPU (`tp*pp` only) because the TPU slice size is fixed by topology and equality must count every device dimension. - An omitted `strategy` in `placement_group_config` resolves to `PACK`. Strategy has no practical effect on TPU slice placement β€” `slice_placement_group` derives the bundle count from capacity and label-pins bundles to the slice, so demand is exact-fit β€” but it is still passed through. - TPU slice bundles carry `accelerator_type:: 0.001`, matching Serve's `default_bundles`. This constrains single-host topologies to the requested generation and satisfies the actor's admission onto bundle 0. - `build_processor` no longer mutates the caller's `engine_kwargs`; `distributed_executor_backend` is set on a private copy. ## Testing - [x] Unit tests - [x] Integration test against a simulated multi-host v6e-16 cluster - [x] Manual test - converted a GPU example from ray docs to TPU, process in comments ## Follow-ups - Docs for TPU batch inference - Configurable `DEFAULT_PG_READY_TIMEOUT_S` β€” reservation is bounded by `head_reservation_timeout_s` (multi-host only), but the worker PG is not waited on - Multi-slice `concurrency > 1`", "url": "https://github.com/ray-project/ray/pull/65422", "createdAt": "2026-08-12T11:15:33Z", "updatedAt": "2026-08-13T13:34:53Z", "timestamp": "2026-08-13T13:34:53Z", "metrics": { "reactions": 1, "comments": 2 }, "labels": [ "data", "community-contribution" ], "author": "ryanaoleary", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:746c0c318439c070fc79", "signalId": "github:ray-project/ray:pull_request:65447", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65447", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[train] Share PlacementGroupCleaner across Train runs", "text": "## Description Ray Train v2 currently creates one detached `PlacementGroupCleaner` actor per Train controller. Since cleaners are pinned to the head node, their memory use grows linearly with concurrent Train runs. In the workload described in #65443, 24 cleaners use approximately 14.4 GB on the head node. This PR replaces the per-controller actors with one cluster-wide shared cleaner. The cleaner tracks controllers independently, so a failed controller only triggers cleanup for its own placement groups. The shared cleaner: - uses a fixed, versioned name and internal namespace with `get_if_exists=True`; - remains detached, pinned to the head node, and scheduled outside captured placement groups; - persists controller and placement-group registrations in GCS internal KV; - restores active and incomplete cleanup records after actor restart; - rejects late placement-group registration after controller death; - isolates State API and placement-group cleanup failures between controllers; - scopes State API queries to its own cluster when multiple clusters are active. ## Related issues Fixes #65443. ## Testing ```text PYTHONPATH=\"$PWD/python\" python -m pytest -q \\ python/ray/train/v2/tests/test_placement_group_cleaner.py # 15 passed PYTHONPATH=\"$PWD/python\" python -m pytest -q \\ python/ray/train/v2/tests/test_controller_callback_behaviour.py # 10 passed pre-commit run --files \\ python/ray/train/v2/_internal/callbacks/placement_group_callback.py \\ python/ray/train/v2/_internal/execution/controller/placement_group_cleaner.py \\ python/ray/train/v2/_internal/state/util.py \\ python/ray/train/v2/tests/test_placement_group_cleaner.py # Passed git diff --check origin/master...HEAD # Passed ``` A local end-to-end multi-job scenario used two independent drivers, each running a real Train v2 `DataParallelTrainer` with a controller, worker, and placement group. Both runs shared one cleaner. After one driver was killed, only its placement group was removed; the other run remained active and completed normally. The cleaner stayed alive throughout. ## Contribution notes No other open PR referencing #65443 or matching this PlacementGroupCleaner change was found before requesting review. AI assistance was used for implementation, testing, and self-review of this change.", "url": "https://github.com/ray-project/ray/pull/65447", "createdAt": "2026-08-13T05:34:32Z", "updatedAt": "2026-08-13T13:34:06Z", "timestamp": "2026-08-13T13:34:06Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "train", "community-contribution" ], "author": "OneSizeFitsQuorum", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:dbe16b925af4d5347c95", "signalId": "github:ray-project/ray:pull_request:65452", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65452", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[Core] Release RuntimeEnv agent per-job logger file handles after setup", "text": "## Description The RuntimeEnv agent caches one logger per job (`_per_job_logger_cache`) and never closes it, and `logging.getLogger` holds a second reference in the global registry. Each logger owns `1 + len(log_files)` file descriptors, so open FDs grow with the cumulative number of jobs the node has handled until every setup fails with `OSError: [Errno 24] Too many open files`. Fix: scope the descriptors to the in-flight setup. - `SetupLoggerFactory` pools file handlers by absolute log path with a > reference count; concurrent setups writing the same file share one handler, > and the file closes when the last one exits. - A context manager wraps `_setup_runtime_env`, releasing on every exit path > (success, failure, exception, timeout, cancellation). - The per-setup logger is a detached `logging.Logger` (not in the global > registry) with the same level/format/propagate as before. - Late writes from executor threads can't re-leak: release attaches a > `NullHandler` before detaching (no empty-handler-list window β†’ > `logging.lastResort`), and `_SetupFileHandler.emit` refuses to write once > closed (`FileHandler.emit` otherwise silently reopens the file in append > mode; `handle()`/`close()` share the handler lock, so the check is > race-free). Both races were reproduced with failing tests first. Log files on disk are untouched β€” retention stays decoupled from FD release. **Not a duplicate:** #54935 is open with no linked PR. The prior attempt #55613 (LRU cache) was auto-closed as stale; an LRU cap keeps up to `maxsize` FDs open and its eviction deleted log files out from under the log monitor, so this takes the setup-scoped-release approach instead. ## Related issues Closes #65451 Related to #55613", "url": "https://github.com/ray-project/ray/pull/65452", "createdAt": "2026-08-13T08:32:52Z", "updatedAt": "2026-08-13T13:33:17Z", "timestamp": "2026-08-13T13:33:17Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "core", "community-contribution" ], "author": "zzchun", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:9b54fb0238d595f22bdf", "signalId": "github:ray-project/ray:pull_request:65453", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65453", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[Data] Add support for writing ORC files", "text": "## Description Ray Data can read ORC files, but it can't write them. This adds `Dataset.write_orc()`, backed by a new `ORCDatasink` that writes each output block with `pyarrow.orc.write_table`. The API forwards PyArrow ORC writer options and supports `min_rows_per_file` through the existing block-based file datasink infrastructure. ## Related issues Follow-up to #64600 ## Additional information Mirrors the `write_csv` / `write_json` APIs. PyArrow ORC writer options can be passed directly to `write_orc()`. No new dependency (`pyarrow` already ships `pyarrow.orc`). Adds write tests in `test_orc.py` and API docs in `saving_data.rst`. ```python import ray ds = ray.data.range(100) ds.write_orc(\"s3://bucket/path/\", compression=\"zstd\")", "url": "https://github.com/ray-project/ray/pull/65453", "createdAt": "2026-08-13T09:53:45Z", "updatedAt": "2026-08-13T13:32:10Z", "timestamp": "2026-08-13T13:32:10Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "docs", "data", "community-contribution" ], "author": "WFY123wfy", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:cd9421721f59136329a9", "signalId": "github:ray-project/ray:pull_request:65455", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65455", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[Core][KubeRay] Route autoscaler INFO logs to stdout instead of stderr", "text": "## Description `_setup_logging()` in the KubeRay autoscaler entry point attached a single `logging._StderrHandler` to the root logger, so every record β€” INFO included β€” went to the container's **stderr**. Kubernetes' CRI log format tags each line with the stream it arrived on (`TIMESTAMP STREAM TAG CONTENT`), and log collectors derive severity from that field rather than parsing the `INFO` token out of the message. On a KubeRay cluster with log collection, that makes 100% of routine autoscaler output arrive as errors β€” measured 2392 `status:error` / 0 `status:info` over 20 minutes on an idle production cluster, where the container's only output was `skipping idle termination`. This PR splits the root logger's console output by severity: sub-WARNING records to stdout, WARNING and above to stderr. This restores the behaviour the function's docstring still describes (\"Also log to pod stdout\") and that the code's own comment described before #34182 (\"Logs will also be written to the container's stdout\"). ### Behaviour change **This changes the default.** Anyone parsing the autoscaler container's stderr for INFO-level lines will now find them on stdout instead. `kubectl logs` is unaffected, since it interleaves both streams. WARNING/ERROR/CRITICAL stay on stderr. `monitor.log` is untouched β€” it still receives every level at `RAY_LOGGER_LEVEL`, because it backs the Ray dashboard and the State API (`/api/v0/logs`). Neither its level nor its gating changed; there is a regression test for this. ### The stale comment The removed comment claimed: ```python # The stdout handler was set up in the Ray CLI entry point. # See ray.scripts.scripts::cli(). ``` That is not true on master, in two independent ways: 1. `cli()` calls `ray._private.ray_logging.setup_logger()`, which re-sets the level and re-formats handlers already on the `ray` logger. It never calls `addHandler`. 2. Even if it did, `setup_component_logger()` β€” called earlier in this same function β€” calls `ray._private.log.clear_logger(\"ray\")`, which empties that logger's handlers. Verified by running the real CLI path (`setup_logger()` then `_setup_logging()`) and inspecting the handler list: `handlers writing to sys.stdout = []`. The comment is removed. ### Design note: `_StderrHandler` and Rich compatibility #33652 deliberately chose `logging._StderrHandler` over a plain `StreamHandler` because it \"always uses whatever `sys.stderr` is currently set to rather than the value of `sys.stderr` at handler construction time,\" which was needed for Rich's `FileProxy` redirect to work when a logger is instantiated outside vs. inside `rich.live.Live()`. **This is the main design risk in the change and I did not want to quietly drop that property.** A naive swap to `logging.StreamHandler(sys.stdout)` would capture the stream at construction time and break under any post-setup redirect. Two things are worth noting: - #33652 modified only `python/ray/_private/ray_logging.py`. It never touched `run_autoscaler.py`; the `_StderrHandler` here arrived separately via #34182, as part of a formatting refactor rather than a stream-routing decision. - `FileProxy` no longer exists anywhere in the repo (`grep -rn \"FileProxy\" python/` returns nothing), and `rich` is not imported anywhere under `python/ray/autoscaler/`. So the original rationale arguably no longer applies here. Rather than rely on that, this PR keeps the lazy-stream property for **both** streams via a small `_StderrHandler`-equivalent that resolves its target at emit time: ```python class _LazyStreamHandler(logging.StreamHandler): def __init__(self, stream_name: str): logging.Handler.__init__(self) # skip StreamHandler.__init__; it assigns self.stream self._stream_name = stream_name @property def stream(self): return getattr(sys, self._stream_name) ``` This is strictly no worse than the status quo under stream redirection, and there is a test asserting the handler follows a `sys.stdout` reassignment made after construction. Happy to move this into `ray/_private/log.py` next to `PlainRayHandler` if reviewers would prefer it shared rather than local to the KubeRay entry point. The stderr handler's level is floored at `WARNING` rather than hardcoded to it, so a stricter `RAY_LOGGER_LEVEL=error` still suppresses warnings on stderr. The floor is computed by reading the level back off the handler after `setLevel()`, because `logging.getLevelName()` returns a *string* (`\"Level FOO\"`) for names it does not recognize, and `RAY_LOGGER_LEVEL` is read straight from the environment without validation. `setLevel()` is what rejects those, raising `ValueError` exactly as this function already did for an invalid `RAY_LOGGER_LEVEL`; there is a test pinning that. ### Unrelated pre-existing issue noticed while testing (not fixed here) `RAY_LOGGER_LEVEL=debug` does not actually produce DEBUG output from the autoscaler, before or after this PR. `_setup_logging` calls `logging.root.setLevel(level)`, but the `ray` logger has its own explicit `INFO` level set during `import ray`, and the autoscaler's logger is a descendant of it β€” so the effective level stays `INFO` and DEBUG records are never created. `clear_logger(\"ray\")` resets handlers and `propagate` but not the level. I left this alone to keep the PR focused. It does mean the level-related tests here assert handler *configuration* rather than emitted output for sub-INFO levels; that is called out in a docstring so the next reader does not mistake it for an oversight. Happy to file it separately if it is worth fixing. ## Related issues Closes #65454 Related to #48651 (directing logs to both stdout/stderr and log files) β€” that issue is broader and framework-wide; this is a targeted fix for the KubeRay autoscaler's own handler setup and does not conflict with it. Same bug class as #32339, fixed one component over. Regression from #34182. ## Additional information ### Tests Added `python/ray/tests/kuberay/test_run_autoscaler_logging.py` (14 tests, registered in `python/ray/tests/BUILD.bazel` in the existing small/`team:core` `py_test_module_list`): - `test_info_goes_to_stdout` β€” INFO on stdout, not stderr. - `test_warning_and_above_go_to_stderr` β€” WARNING/ERROR/CRITICAL on stderr, not stdout. - `test_monitor_log_still_receives_all_levels` β€” `monitor.log` keeps every level. - `test_default_level_filters_debug` β€” DEBUG dropped everywhere at the default level. - `test_handler_levels_track_logger_level` β€” parametrized over all five `LOGGER_LEVEL_CHOICES`, pinning the stderr WARNING floor and the stricter-level case. - `test_level_stricter_than_warning_suppresses_warnings` β€” `error`/`critical` keep warnings off both streams. - `test_split_is_numeric_not_name_based` β€” a custom level 25 between INFO and WARNING routes to stdout. - `test_invalid_logger_level_raises` β€” invalid `RAY_LOGGER_LEVEL` still raises `ValueError`. - `test_lazy_stream_handler_follows_stream_reassignment` β€” preserves the #33652 property under a post-construction `sys.stdout` swap. The tests snapshot and restore global logging state (root handlers/level, `ray` logger handlers/propagate) so they cannot leak into other tests in the same process, and close the handlers on teardown so `monitor.log`'s `RotatingFileHandler` does not leak an fd. Confirmed the tests actually catch the bug: reverting just the handler setup to the old single-`_StderrHandler` version fails 7 of the 14, including `test_info_goes_to_stdout` with `AssertionError: assert 'INFO_RECORD' in ''`. The `monitor.log` test passes both before and after, which is the check that the fix leaves `monitor.log` alone. ### Commands run ``` $ python -m pytest -q python/ray/tests/kuberay/test_run_autoscaler_logging.py 14 passed in 0.08s $ python -m pytest -q python/ray/tests/kuberay/test_autoscaling_config.py \\ python/ray/tests/kuberay/test_kuberay_node_provider.py \\ python/ray/tests/kuberay/test_run_autoscaler_logging.py 90 passed in 0.75s $ pre-commit run ruff --files # Passed $ pre-commit run black --files # Passed $ pre-commit run pydoclint --files # Passed $ pre-commit run buildifier --files python/ray/tests/BUILD.bazel # Passed $ pre-commit run buildifier-lint --files python/ray/tests/BUILD.bazel # Passed $ pre-commit run check-import-order docstyle python-no-log-warn ... # Passed ``` Environment: `3.0.0.dev0` nightly wheel (`eb5882b7ba`) with the local source tree symlinked in via `python/ray/setup-dev.py`, per the development guide. The `prettier` hook could not install in my environment (blocked npm registry); it only applies to `doc/` JS/CSS and does not cover any file in this PR. ### AI assistance AI assistance was used for this change. I reviewed every changed line, ran the tests and linters above locally, and can defend the change end-to-end. ### Duplicate check Searched `ray-project/ray` and `ray-project/kuberay` issues and PRs for this symptom before filing (`gh api search/issues` across several phrasings covering autoscaler + stderr/stdout + severity/log level). No existing issue or open PR covers it. The closest items are kuberay#4095 (closed by documenting `RAY_LOGGER_LEVEL`, which is not a fix here β€” it silences `monitor.log` as well, see the issue) and ray#48651 (broader, still open).", "url": "https://github.com/ray-project/ray/pull/65455", "createdAt": "2026-08-13T10:28:36Z", "updatedAt": "2026-08-13T13:31:22Z", "timestamp": "2026-08-13T13:31:22Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "kuberay", "core", "community-contribution", "kubernetes" ], "author": "vivekmahajan", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:78fd12f4a68c35db688b", "signalId": "github:ray-project/ray:pull_request:65370", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65370", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "serve: expose multiplexed model IDs in ReplicaDetails", "text": "## Summary Adds a backward-compatible `multiplexed_model_ids` field to `ReplicaDetails` and keeps it synchronized when the controller receives updated multiplexed model IDs. Extends the existing multiplexing test to verify the values through `ServeInstanceDetails` returned by `serve_instance.get_serve_details()`. Closes #65347 ## Duplicate check I checked open PRs for #65347 and for `multiplexed_model_ids ReplicaDetails`; no matching open PR existed. ## Testing - `python -m compileall` on all four changed Python files β€” passed. - Direct execution of `test_record_multiplexed_model_ids_updates_actor_details` β€” passed. - `git diff --check` β€” passed. - Ruff, pydoclint, Ray Serve mypy, Pyrefly, import-order, and Black checks β€” passed. - The full pytest integration test could not be collected locally because Windows `setup-dev.py` junctions caused duplicate `conftest.py` registration. Ray CI will run the added integration test on Linux. - `docstyle` could not run because `/bin/bash` is unavailable in this Windows environment; Semgrep's pre-commit environment had an invalid cached Python path. ## AI assistance I used AI assistance to help implement and validate this change. I reviewed and understand every changed line.", "url": "https://github.com/ray-project/ray/pull/65370", "createdAt": "2026-08-11T13:32:40Z", "updatedAt": "2026-08-13T13:31:16Z", "timestamp": "2026-08-13T13:31:16Z", "metrics": { "reactions": 1, "comments": 2 }, "labels": [ "serve", "community-contribution" ], "author": "upadhyay74aman", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:5b97807d257690852e68", "signalId": "github:ray-project/ray:pull_request:65456", "event": "discovered", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65456", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[doc] Pin the ReDoc bundle to v2.5.3 instead of tracking the CDN latest tag", "text": "## Why this change is needed `doc/source/conf.py` points `redoc_uri` at the ReDoc CDN's unpinned `latest` tag: ```python redoc_uri = \"https://cdn.redoc.ly/redoc/latest/bundles/redoc.standalone.js\" ``` `sphinxcontrib-redoc` injects that script tag alongside the spec; it doesn't bundle a renderer. So the published Jobs API page at `cluster/running-applications/job-submission/api.html` is rendered client-side, at page-view time, by whatever Redocly currently serves at `latest`. If a breaking change lands in that bundle, the live page degrades with no build-time signal at all, because the Sphinx build never executes the script. The build stays green while the page is broken. This pins the version so the renderer stops moving underneath the docs unobserved. ## Related changes None. This is deliberately scoped to the one line plus a comment. ## What changes - Pin `redoc_uri` to `v2.5.3`. - Add a comment recording why it's pinned and that bumping it requires a visual check on the Read the Docs preview, so a future dependency sweep doesn't helpfully restore `latest`. ## Why v2.5.3, and why this is a no-op today `v2.5.3` is what `latest` currently resolves to, so the pin doesn't silently downgrade the rendering. The two bundles are byte-identical, verified 2026-08-13: | URL | HTTP | Bytes | sha256 | | --- | --- | --- | --- | | `.../redoc/latest/bundles/redoc.standalone.js` | 200 | 1,097,271 | `1320f442151c57c447d3b70c7ffc6c4f86d08464020fe34c8cc5d3164e9944f0` | | `.../redoc/v2.5.3/bundles/redoc.standalone.js` | 200 | 1,097,271 | `1320f442151c57c447d3b70c7ffc6c4f86d08464020fe34c8cc5d3164e9944f0` | `2.5.3` is also the current `latest` dist-tag on npm. The next release line is `3.0.0-rc.0`, which is exactly the kind of bundle that would arrive through this URL unannounced and unverified today. For context on why the unpinned tag isn't hypothetical, `latest` has already moved well away from older 2.x builds: `v2.5.0` is 910,994 bytes against the current 1,097,271. ## Checks - [x] `doc/source/conf.py` parses; `pre-commit run` clean on the staged change. - [ ] Loaded `api.html` in the Read the Docs preview for this PR and confirmed the three-panel layout and the endpoint list render. Will confirm here once the preview build finishes β€” a green Sphinx build proves nothing for a client-rendered page. ## Out of scope Replacing the unmaintained `sphinxcontrib-redoc` extension, and the separate question of the spec's declared OpenAPI version. Both are tracked separately and neither blocks this pin, which is worth having regardless of how they land.", "url": "https://github.com/ray-project/ray/pull/65456", "createdAt": "2026-08-13T12:34:15Z", "updatedAt": "2026-08-13T13:30:14Z", "timestamp": "2026-08-13T13:30:14Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "core", "go", "docs-go" ], "author": "dstrodtman", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:538531ccd2fcea5a784b", "signalId": "github:ray-project/ray:pull_request:65457", "event": "discovered", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65457", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[doc] Remove invalid summary keys from Jobs API spec Response Objects", "text": "## Why this change is needed `summary` isn't a valid field on an OpenAPI Response Object, in either 3.0.x or 3.1. The valid fields are `description`, `headers`, `content`, and `links`. `doc/source/cluster/running-applications/job-submission/openapi.yml` carries 21 of them, and they make the document fail validation against the `3.1.0` version it declares: ``` $ openapi-spec-validator openapi.yml openapi.yml: Validation Error: Unevaluated properties are not allowed ('summary' was unexpected) Failed validating 'unevaluatedProperties' in schema['properties']['paths'][...]['responses'][...] $comment: https://spec.openapis.org/oas/v3.1.0#response-object On instance['paths']['/api/version']['get']['responses']['200'] ``` So the spec is invalid against its own declared version today. That matters because `rest.rst` explicitly invites readers to generate client libraries from this spec, and it blocks wiring a validation check into CI, since a validator that has to be run with known-failing output is one nobody will adopt. ## Related changes None. This is scoped to removing the invalid keys. ## What changes Removes the 21 `summary` keys from Response Objects. Nothing else β€” the diff is 21 deletions and zero insertions. The 8 operation-level `summary` fields are untouched. Those are valid on an Operation Object and they do render, as the endpoint titles in the sidebar and section headings. Only the response-level ones go. ## This does not change the published page ReDoc ignores unknown keys, so the rendered output is unaffected. Rather than assert that, I verified it against the live page by probing the rendered DOM after the ReDoc bundle ran: - All 21 summary values are **absent** from the rendered body text. The six that are unique to a response `summary` and appear nowhere else in the spec β€” `Job Submit Response`, `Job Submit Validation Error`, `Job Submit Internal Error`, `List of Job Details`, `Job Logs Internal Error`, `Wrong Job Type` β€” return no match. - They **are** present in the raw page HTML, because the spec is embedded in the page as JSON. The data reaches the browser; ReDoc simply doesn't render it. - The sibling `description` of those same Response Objects **does** render, which rules out the alternative explanation that those responses were never reached or stayed collapsed. For example, the rendered Responses block for `POST /api/jobs` reads: ``` Responses 200 The ID of the submitted job. 400 A TypeError or ValueError was raised when submitting the job. 500 An internal error occurred when submitting the job. ``` Status code and `description`, with the three `summary` values for those exact responses nowhere in the output. ## No documented content is lost Every one of the 21 affected responses already has a `description` that states the same thing in prose, and `description` is the required field on a Response Object. A few examples: | Path | Removed `summary` | Retained `description` | | --- | --- | --- | | `GET /api/version` 200 | `Version` | The Ray Jobs API version and the Ray version running on the cluster. | | `POST /api/jobs` 400 | `Job Submit Validation Error` | A TypeError or ValueError was raised when submitting the job. | | `DELETE /api/jobs/{submission_id}` 400 | `Wrong Job Type` | The job was not submitted via the Ray Jobs API, so it cannot be deleted. | Verified programmatically that all 21 responses retain a `description` after the edit, so none becomes invalid for the opposite reason. ## Checks - [x] Spec parses; the parsed document is identical to the previous one with exactly those 21 keys removed and no other difference. - [x] `openapi-spec-validator` now reports `OK` against the declared 3.1.0. It failed before this change. - [x] Rendered-DOM probe of the live page, described above. - [x] `pre-commit run` clean on the staged change. - [ ] Confirm `api.html` is unchanged in the Read the Docs preview for this PR. Will confirm here once the preview build finishes β€” this is a client-rendered page, so a green Sphinx build proves nothing about it. ## Note for reviewers The invalidity is pre-existing and isn't caused by any version change. It fails under 3.0.x too, so this cleanup is worth having independently of the separate open question about which OpenAPI version this document should declare. After this change the spec validates against both 3.1.0 and 3.0.3, which leaves that question free to be decided on its own merits.", "url": "https://github.com/ray-project/ray/pull/65457", "createdAt": "2026-08-13T12:43:16Z", "updatedAt": "2026-08-13T13:29:24Z", "timestamp": "2026-08-13T13:29:24Z", "metrics": { "reactions": 1, "comments": 1 }, "labels": [ "docs", "core", "go", "docs-go" ], "author": "dstrodtman", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:19402db54ebfe9fe60ee", "signalId": "github:ray-project/ray:pull_request:65458", "event": "discovered", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65458", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[doc] Declare OpenAPI 3.0.3 in the Jobs API spec instead of 3.1.0", "text": "> **Stacked PR.** This targets `doc-1513-remove-invalid-response-summary` (#65457), not `master`, because that PR makes the spec valid and this one shouldn't inherit a failing validator run. Review #65457 first; GitHub retargets this to `master` automatically when it merges. The two changes are independent in substance and can be split if you'd rather take them separately. ## Why this change is needed `doc/source/cluster/running-applications/job-submission/openapi.yml` declares `openapi: 3.1.0` and uses not one construct that requires 3.1. Audited the full spec for 3.1-only constructs. Every check came back empty: | 3.1-only construct | Present? | | --- | --- | | `jsonSchemaDialect` | no | | top-level `webhooks` | no | | `$schema` | no | | `const` | no | | `examples` (the plural, JSON-Schema-aligned form) | no | | type arrays (`type: [string, \"null\"]`) | no | | `null` type | no | | `info.summary` | no | It also uses no `nullable`, the 3.0-only keyword it would need if it had ever expressed nullability. The complete set of keywords in the file is `description`, `type`, `summary`, `schema`, `content`, `title`, `required`, `properties`, `parameters`, `in`, `$ref`, `allOf`, `additionalProperties`, `enum`, `deprecated`, `items`, `requestBody`, `responses`, `get`, `post`, `delete`, `components`, `schemas`, `info`, `paths`, and `version`. Every one is valid in 3.0.x with identical semantics. So the document is already a valid 3.0.3 document that happens to declare 3.1.0. **Declaring the oldest version the document actually conforms to widens the set of tools that can consume it.** 3.1 support across the OpenAPI ecosystem still lags 3.0 β€” client generators, validators, and renderers handle 3.0.x completely and 3.1 partially. `rest.rst` explicitly invites readers to generate client libraries from this spec, so the declared version directly affects whether that works for them. Nothing is gained by declaring 3.1 while using no 3.1 feature. There's a concrete in-repo case for this too. The doc build currently renders this spec with `sphinxcontrib-redoc`, unmaintained since 2020, and the maintained alternative from the same org, `sphinxcontrib-openapi`, has only partial 3.1 support: - The PR that added it, [sphinx-contrib/openapi#138](https://github.com/sphinx-contrib/openapi/pull/138) (merged January 2023), is titled \"Add **initial** OpenAPI 3.1 support\" and its description says \"we focus on the MVP here\" with \"a whole lot more to support here including things like webhooks.\" - [sphinx-contrib/openapi#160](https://github.com/sphinx-contrib/openapi/issues/160) is an open, uncommented 3.1 rendering bug filed October 2024 β€” an unhandled `TypeError` in the `openapi31.py` code path. Declaring 3.0.3 removes that whole risk class from any future evaluation of that extension, at the cost of one line and no lost expressiveness. ## Related changes Stacked on #65457, which removes 21 invalid `summary` keys from Response Objects. That's a separate, version-independent validity fix; this PR is about the declared version only. ## What changes One line: `openapi: 3.1.0` β†’ `openapi: 3.0.3`. **`info.version` deliberately stays at `4.0.0`.** It tracks the Ray Jobs API version, not the OpenAPI document version. The two are easy to conflate, so to be explicit: this PR changes only the OpenAPI specification version the document claims to conform to, and nothing about the API itself. ## The published page is unaffected, verified ReDoc handles both versions, so no visual change is expected β€” but this is a client-rendered page, and a green Sphinx build proves nothing about it. So I rendered the spec through the same pinned ReDoc v2.5.3 bundle the build uses, at both declared versions, and compared: | Check | Result | | --- | --- | | Rendered body text | **Byte-identical** (10,561 chars each) | | Full-page screenshot | **Pixel-identical** (same sha256) | | JavaScript errors or warnings | **None**, at either version | | Endpoint content actually rendered | Confirmed at both, so \"identical\" isn't two identical failures | ## Checks - [x] `openapi-spec-validator` reports `OK` against the declared 3.0.3. - [x] Rendered-output comparison above. - [x] `pre-commit run` clean on the staged change. - [ ] Confirm `api.html` is unchanged in the Read the Docs preview for this PR. ## If you'd rather not take this Reasonable outcomes here include declining it. If the intent is that this spec should eventually use 3.1 features, say so and I'll close this β€” the useful part is then the recorded finding that nothing in the document depends on 3.1 today.", "url": "https://github.com/ray-project/ray/pull/65458", "createdAt": "2026-08-13T12:57:00Z", "updatedAt": "2026-08-13T13:28:34Z", "timestamp": "2026-08-13T13:28:34Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "docs", "core", "go", "docs-go" ], "author": "dstrodtman", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:965b15bace83e86d5825", "signalId": "github:ray-project/ray:pull_request:65282", "event": "discovered", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65282", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[autoscaler] Precompute SerializeToString keys to eliminate redundant serialization in scheduling loop", "text": "## Why are these changes needed? After merging #64175, benchmark testing showed that scaling to 3000 nodes (15000 tasks Γ— 0.2 CPU + 30MB memory) still takes ~51 minutes. Profiling with py-spy revealed two independent hotspots in the scheduling loop: 1. **Pre-filter logic flaw** β€” fixed in #65171 (OR-across-dimensions β†’ AND-within-shape/OR-across-shapes) 2. **Redundant `SerializeToString` calls** β€” fixed in this PR The `UnschedulableRequestCache.contains()` serializes each request via `SerializeToString(deterministic=True)` once per candidate node inside `try_schedule`, resulting in **N_nodes Γ— M_requests** redundant serializations per scheduling round. For 15000 requests across 3000 nodes, this produces ~45M repeated protobuf serializations. ### Profiling evidence (py-spy, 30s sample during baseline run): - `contains()` in scheduler.py: **55% Own CPU** (7.77s / 14s) - `_audit_fork_safety` triggered by SerializeToString inside `contains()`: **18% Own CPU** (2.43s / 14s) - Combined: **73% of scheduler CPU** spent on redundant serialization \"image\" ![py-spy top](https://github.com/user-attachments/assets/8b5b56bb-0f87-445c-a5c9-9346bd3ce0d9) ### Fix Precompute a `{id(request): bytes}` dictionary **once** before entering the scheduling loop, and pass it through `_sched_best_node` β†’ `try_schedule`. The `UnschedulableRequestCache` interface is simplified to accept raw `bytes` keys directly. Includes a defensive `.get()` fallback: if a request is not in the precomputed dict, it falls back to on-the-fly serialization rather than raising a `KeyError`. ## Benchmark Cluster: 3000 max workers (1 CPU each), 15000 tasks Γ— 0.2 CPU. | Condition | Time to 3000 nodes | |-----------|-------------------| | After #64175 (baseline) | ~51 min | | + #65171 only (pre-filter AND/OR) | ~18 min | | + #65171 + this fix combined | ~14 min | ## Related PRs - #64175 β€” baseline optimization that this PR builds on (benchmark results from that PR revealed this hotspot) - #65171 β€” parallel fix for pre-filter logic (independent, can merge in either order) ## Checks - [x] I've signed all my commits with DCO sign-off. - [x] I've run the existing tests β€” all pass. The `try_schedule` interface change is backward-compatible via `Optional` parameter with on-the-fly fallback. - [x] Changes are confined to `python/ray/autoscaler/v2/scheduler.py` (single file, low blast radius). - [x] No duplicate PR exists for this optimization (searched open PRs for SerializeToString, precompute, UnschedulableRequestCache β€” zero matches). - [x] AI assistance was used. I (human submitter) have reviewed every changed line and understand the change end-to-end.", "url": "https://github.com/ray-project/ray/pull/65282", "createdAt": "2026-08-07T08:25:27Z", "updatedAt": "2026-08-13T13:19:08Z", "timestamp": "2026-08-13T13:19:08Z", "metrics": { "reactions": 1, "comments": 1 }, "labels": [ "core", "community-contribution" ], "author": "Jade07-1", "state": "open", "assignees": [ "rueian" ], "change": "new" } }, { "id": "event:44c093a9a75b83a41fbf", "signalId": "github:ray-project/ray:pull_request:65230", "event": "discovered", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65230", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[core] Add RAY_DISABLE_WORKER_LOG_PREFIX to control log prefix behavior", "text": "## Description As mentioned in the issue 64992. `log_to_driver=True` (the default) prepends a `(name pid=...)` prefix to every worker/actor stdout/stderr line forwarded to the driver. There was previously no way to disable *only* that prefix: `RAY_COLOR_PREFIX=0` only strips ANSI color, `log_to_driver=False` drops forwarding entirely (logs never reach the driver's stdout), and `logging_config=LoggingConfig(...)` strips the prefix but also seizes the root logger for the driver/actors/tasks, which isn't viable for applications with their own logging configuration. The `ignore_prefix` existed , but was only dependent on job_logging_config, this fix adds another env-var to decide the switch. ## Related issues Closes #64992. ## Additional information Testing performed: - Full `python/ray/tests/test_logging_2.py` suite: 12/12 passed, including the new `test_disable_worker_log_prefix_env_var` and the pre-existing prefix-related tests. - Manually verified default behavior is unchanged when the env var is unset . > === WITHOUT the flag (default behavior) === // Some warnings 2026-08-05 14:23:04,148 INFO worker.py:2024 -- Started a local Ray instance. Usage stats collection is enabled by default for nightly wheels. To disable this, run the following command: `ray disable-usage-stats` before starting Ray. See https://docs.ray.io/en/master/cluster/usage-stats.html for more details. (SegmentationActor pid=558) {\"event\": \"segmenting batch\", \"job_id\": \"abc123\", \"level\": \"info\"} > > === WITH RAY_DISABLE_WORKER_LOG_PREFIX=1 === > 2026-08-05 14:24:09,569 INFO worker.py:2024 -- Started a local Ray instance. > Usage stats collection is enabled by default for nightly wheels. To disable this, run the following command: `ray disable-usage-stats` before starting Ray. See https://docs.ray.io/en/master/cluster/usage-stats.html for more details. > {\"event\": \"segmenting batch\", \"job_id\": \"abc123\", \"level\": \"info\"} **AI assistance disclosure** : This change was tested with the help of AI (Claude). I have reviewed the tests and did manually as well to verify the same.", "url": "https://github.com/ray-project/ray/pull/65230", "createdAt": "2026-08-05T14:39:11Z", "updatedAt": "2026-08-13T13:18:05Z", "timestamp": "2026-08-13T13:18:05Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "core", "community-contribution" ], "author": "imtherealnaska", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:80c8e3f383cadd447796", "signalId": "github:ray-project/ray:pull_request:65077", "event": "discovered", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65077", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "fix json bug", "text": "## What happened + What you expected to happen Deploying a Ray Serve LLM (`ray.serve.llm.build_openai_app`) with a nested dataclass field in `engine_kwargs` β€” e.g. `structured_outputs_config` β€” crashes the vLLM engine on startup with: ``` AttributeError: 'dict' object has no attribute 'reasoning_parser' ``` or, depending on which field is accessed first: ``` AttributeError: 'dict' object has no attribute 'compute_hash' ``` **Expected:** the engine starts successfully and `structured_outputs_config` is applied to the model, exactly as it would with vLLM's own CLI (`--structured-outputs-config.enable_in_reasoning=True`). ### Why this matters This blocks the *documented* upstream vLLM fix for broken structured outputs on Qwen3 Coder + reasoning models ([vllm-project/vllm#18819](https://github.com/vllm-project/vllm/issues/18819)), which requires setting `structured_outputs_config.enable_in_reasoning=True`. Ray Serve LLM users deploying via YAML (`serveConfigV2` / `LLMConfig.engine_kwargs`) have no working way to set this flag β€” the only way to express a nested vLLM config field from YAML is as a plain dict, and that dict reaches `AsyncEngineArgs` unconverted. ## Root cause `_get_vllm_engine_config()` in `python/ray/llm/_internal/serve/engines/vllm/vllm_engine.py` builds the engine config like this: ```python async_engine_args = vllm.engine.arg_utils.AsyncEngineArgs( **engine_config.get_initialization_kwargs() ) vllm_engine_config = async_engine_args.create_engine_config( usage_context=UsageContext.OPENAI_API_SERVER ) ``` `engine_config.get_initialization_kwargs()` returns a plain `dict` for every `engine_kwargs` entry, including nested ones like `structured_outputs_config`. `vllm.engine.arg_utils.EngineArgs.__post_init__` only coerces a fixed allowlist of fields from dict to their dataclass type (`compilation_config`, `attention_config`, `mamba_config`, `kernel_config`, `eplb_config`, `weight_transfer_config`, `ir_op_priority`). `structured_outputs_config` and `reasoning_config` are **not** in that list, so they stay as plain `dict` objects on the `AsyncEngineArgs` instance. `create_engine_config()` then does (vLLM `engine/arg_utils.py`): ```python if self.reasoning_parser: self.structured_outputs_config.reasoning_parser = self.reasoning_parser ``` β€” which crashes immediately with `AttributeError: 'dict' object has no attribute 'reasoning_parser'` whenever a top-level `engine_kwargs.reasoning_parser` is also set (a very common config for Qwen3 reasoning models). Even without that top-level field, `VllmConfig.compute_hash()` later calls `self.structured_outputs_config.compute_hash()`, which crashes the same way. ### There already is a fix for a *similar* symptom β€” but it doesn't cover this path #60380 (`[LLM] Fix nested dict to Namespace conversion in vLLM engine initialization`, merged) introduced a helper, `_convert_config_dicts()`, that converts known `AsyncEngineArgs` dict fields into their proper dataclass instances using field type hints. It is called in `VLLMEngine.start()`: ```python merged = vllm_frontend_args.__dict__ | vllm_engine_args.__dict__ merged = _convert_config_dicts(merged) args = _dict_to_namespace(merged) self._vllm_args = args ``` This fixes `AttributeError` in the **serving/frontend layer** (`build_asgi_app()` / `init_app_state()`, which reads `args.structured_outputs_config.backend` etc. to configure the OpenAI-compatible endpoint). But `self._vllm_args` is only built *after* the engine itself has already been started (`self._start_async_llm_engine(vllm_engine_args, ...)`), using the args that came out of `_get_vllm_engine_config()`. That function builds `AsyncEngineArgs` and calls `create_engine_config()` **before** `_convert_config_dicts()` ever runs β€” so the crash described above still reproduces on top of #60380. ## Reproduction ```python from ray.llm._internal.serve.core.configs.llm_config import LLMConfig, ModelLoadingConfig from ray.llm._internal.serve.engines.vllm.vllm_engine import _get_vllm_engine_config llm_config = LLMConfig( model_loading_config=ModelLoadingConfig( model_id=\"qwen-test\", model_source=\"Qwen/Qwen3-4B\" ), engine_kwargs={ \"reasoning_parser\": \"qwen3\", \"structured_outputs_config\": {\"enable_in_reasoning\": True}, }, ) _get_vllm_engine_config(llm_config) # AttributeError: 'dict' object has no attribute 'reasoning_parser' ``` Or via a RayService `serveConfigV2`: ```yaml applications: - name: llms import_path: ray.serve.llm:build_openai_app route_prefix: \"/\" args: llm_configs: - model_loading_config: model_id: qwen3-4b-thinking model_source: /models/Qwen3-4B engine_kwargs: tensor_parallel_size: 2 reasoning_parser: qwen3 structured_outputs_config: backend: xgrammar enable_in_reasoning: true deployment_config: autoscaling_config: min_replicas: 1 max_replicas: 1 ``` The vLLM engine actor crashes on startup (`CrashLoopBackOff` under KubeRay). ## Related - vllm-project/vllm#18819 β€” the upstream fix this blocks (`enable_in_reasoning`) - ray-project/ray#60380 β€” fixes the same class of bug in the serving/frontend layer, but not this path - ray-project/ray#60430 β€” regression test for #60380", "url": "https://github.com/ray-project/ray/pull/65077", "createdAt": "2026-07-28T11:40:21Z", "updatedAt": "2026-08-13T12:31:43Z", "timestamp": "2026-08-13T12:31:43Z", "metrics": { "reactions": 1, "comments": 2 }, "labels": [ "serve", "stale", "community-contribution" ], "author": "MiXaiLL76", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:5a9bad95332705c8aac6", "signalId": "github:ray-project/ray:pull_request:64672", "event": "discovered", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:64672", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[serve] Build serve_haproxy_* metrics through MetricRegistry", "text": "## Description Replaces the seven raw `ray.util.metrics` constructions in `HAProxyMetricsCollector` with handles from a shared, module-level `MetricRegistry(namespace=\"serve_haproxy\")`. The registry get-or-creates by name, so re-constructing a collector in the same process (tests, actor restarts) reuses the existing Ray metric objects instead of re-creating them. **No behavior change**: exported metric names, descriptions, tag keys, histogram buckets, and default tags are byte-identical (the requests counter still gains `_total` from Ray on export), and the record paths are untouched β€” registry handles expose the same `inc`/`set`/`observe` verbs with Ray semantics. This is PR 3 of a 3-PR stack (stacked on #64670 and #64671; only the last commit is new here): 1. #64670 β€” `MetricRegistry` 2. #64671 β€” `PrometheusCollector` 3. **This PR** β€” serve HAProxy metrics on the registry ## Related issues N/A ## Additional information The existing pure parse/record unit tests pass unmodified; adds one regression test asserting handle dedup across collector constructions and the unchanged exported names. ``` python -m pytest python/ray/serve/tests/test_haproxy_metrics.py # 58 passed ``` (3 pre-existing `AF_UNIX path too long` failures on macOS reproduce identically on the base branch β€” environmental, unrelated.)", "url": "https://github.com/ray-project/ray/pull/64672", "createdAt": "2026-07-10T18:58:09Z", "updatedAt": "2026-08-13T12:31:24Z", "timestamp": "2026-08-13T12:31:24Z", "metrics": { "reactions": 1, "comments": 2 }, "labels": [ "serve", "stale" ], "author": "xyuzh", "state": "closed", "assignees": [], "change": "new" } }, { "id": "event:f3f0c8a44705c7d79b05", "signalId": "github:ray-project/ray:pull_request:64670", "event": "discovered", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:64670", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[core][metrics] Add MetricRegistry: idempotent facade over ray.util.metrics", "text": "## Description Adds `ray.util.metric_registry.MetricRegistry` (DeveloperAPI): a thread-safe, get-or-create factory over the three custom-metric primitives (`Counter`/`Gauge`/`Histogram`). It owns the quirks every caller currently re-solves by hand: - **Dedup by name** β€” the same name returns a handle to the same underlying Ray metric, so callers don't manage their own caches. - **Name sanitizing** β€” `:` β†’ `_` (illegal in Ray metric names), optional namespace prefix. - **The Counter `_total` quirk** β€” Ray's `Counter` re-appends `_total` on export; a trailing `_total` in the requested name is folded in rather than doubled. - **Fixed-`tag_keys` handling** β€” Ray fixes `tag_keys` at construction and validates strictly at record time. Handles pad declared-but-missing keys with `\"\"` (without clobbering default tags), and a label key first seen after creation warns once and is dropped instead of raising (Ray cannot widen `tag_keys`). Handles keep the kind-native verbs (`inc`/`set`/`observe`) with Ray semantics, plus a kind-agnostic `record()` for mirroring pipelines (counter `record()` no-ops on non-positive deltas) and a histogram `timer()` context manager. This is PR 1 of a 3-PR stack: 1. **This PR** β€” `MetricRegistry` 2. `PrometheusCollector` β€” mirror any Prometheus /metrics exposition into Ray metrics via the registry 3. Migrate serve's `haproxy_metrics.py` onto the registry ## Related issues N/A ## Additional information Tests are pure wrapper tests (no cluster) following the `metric_mock` pattern from `test_metrics_agent.py`: ``` python -m pytest python/ray/tests/test_metric_registry.py # 15 passed ```", "url": "https://github.com/ray-project/ray/pull/64670", "createdAt": "2026-07-10T18:57:37Z", "updatedAt": "2026-08-13T12:31:22Z", "timestamp": "2026-08-13T12:31:22Z", "metrics": { "reactions": 1, "comments": 2 }, "labels": [ "stale", "core", "go" ], "author": "xyuzh", "state": "closed", "assignees": [], "change": "new" } }, { "id": "event:d43ad8acc1bab6dc693e", "signalId": "github:ray-project/ray:pull_request:65423", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65423", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[docs] Retire the \"Ray Pod\" term from the Kubernetes docs", "text": "## Why this change Ray on Kubernetes docs use \"Ray Pod\" 89 times, and this PR removes the term. It's the concrete counterpart to the discussion in #65239, where a reviewer asked whether the Kubernetes casing rule should capitalize Pod in \"Ray pod\". Investigating that question produced a better answer than either casing. **A Pod isn't a Ray node, it's a host that contains one.** Kubernetes defines a Pod as an \"application-specific logical host\" holding one or more containers. KubeRay indexes exactly one Ray container per Pod through `utils.RayContainerIndex`, and routinely places non-Ray containers alongside it: the autoscaler sidecar, the history-server collector, the RayJob submitter in `SidecarMode`, and the Fluent Bit sidecars that `persist-kuberay-custom-resource-logs.md` documents. The Pod is a strict superset of the Ray node, so naming it after Ray overstates what it is. It's the same category error as \"head node pod\", one level up. **\"Pod\" is already taken in these docs, by something that isn't a Kubernetes object.** A TPU Pod is a group of interconnected TPU chips. `user-guides/tpu.md` writes \"The name of the TPU Pod or slice\" and \"The TPU pod type\" three lines apart, and `ray.util.tpu.get_current_pod_name()` is a public API whose docstring reads \"Return the name of the TPU pod that the worker is a part of\". Standardizing on a capitalized `Ray Pod` would put it beside a capitalized `TPU Pod` on the same pages, distinguished only by the preceding word. **The mapping is already documented.** `user-guides/k8s-autoscaler.md` says \"each Ray node is run as a Kubernetes Pod\", so two defined terms already cover what the compound names. **There was no convention to preserve.** Across the 55 prose lines using the term, casing split four ways: | Form | Occurrences | |---|---| | Ray Pods | 20 | | Ray pods | 16 | | Ray pod | 16 | | Ray Pod | 15 | Meanwhile \"head Pod\" and \"worker Pod\" already outnumber it 284 to 86 under `doc/source/cluster/kubernetes/`, so the replacement vocabulary is the one these docs mostly use. ## What changed 83 lines across 31 files. The term becomes \"Pod\" where context is unambiguous, which is most of them. Where a bare \"Pod\" would lose meaning I used an explicit form instead: - Headings, which have no surrounding context: \"Check logs of the head and worker Pods\", \"Verify the Ray cluster's Pods and PodGroup\", \"Configuring Kubernetes Pods for TPU usage\". - Pages where other kinds of Pod are in scope: \"the Ray cluster's Pods\" in `network-policy.md`, `prometheus-grafana.md`, `k8s-events.md`, and `scheduler-plugins.md`. - `rayservice-high-availability.md` listed \"Ray Pod: This Pod sends requests to the RayService\". That Pod is a client that isn't part of the RayCluster, and it's named `ray-pod` in the sample manifest, so it's now listed by its actual object name. ## What replaces it on the orientation pages Removing the compound takes away the handle a Ray-first reader was using for the node-and-Pod relationship, so the relationship now gets stated where readers form their mental model. It was only in `user-guides/k8s-autoscaler.md`, whose title begins \"(Advanced)\", and absent from `index.md`, `getting-started.md`, and `user-guides/config.md`. One sentence each on the two orientation pages: - `index.md`: \"KubeRay runs each Ray node as a Kubernetes Pod, so each Ray cluster consists of a head Pod and a collection of worker Pods.\" - `getting-started.md`: \"It runs each Ray node as a Kubernetes Pod, so a Ray cluster's head node is its head Pod and its worker nodes are its worker Pods.\" `getting-started/raycluster-quick-start.md` is the natural third location, since line 53 is where the two vocabularies genuinely first meet for a reader following the quickstart. It's left alone deliberately: that paragraph writes \"pods\" lowercase throughout, and adding a capitalized sentence into it would either read inconsistently or drag the casing sweep into this PR. It belongs to whoever runs that pass. The rules behind both changes are in #65239, which now carries a \"State the node-to-Pod mapping once, early\" section and a \"Keep Pods out of Ray-general pages\" section. The second one is why this PR touches `ray-core/scheduling/placement-group.rst` and `serve/production-guide/fault-tolerance.md`. ## What this deliberately doesn't touch - **`Ray head Pod` and `Ray worker Pod` stay.** Those qualify by role, which is the form these docs already prefer and the one Google's Ray on GKE docs use. Only the unqualified compound goes. - **No casing sweep.** Lowercase \"head pod\" occurrences are left alone. That's the Vale rule in #65239, not this PR. - **`python/` code comments and docstrings keep the term.** About 10 occurrences under `python/ray/autoscaler`, `python/ray/dashboard`, and the KubeRay tests identify which Pods the operator manages, which is reasonable for a code comment. This PR is scoped to documentation prose. - **The `kuberay` repo is out of scope.** Its authored prose has a handful of uses and its `docs/reference/api.md` is generated from Go doc comments. That's a maintainer call for that repo, and I'll raise it there separately rather than assume the same decision carries. ## Two fixes on lines this change already rewrites - `k8s-autoscaler.md` linked to `.../horizontal-Pod-autoscale/`, which returns **404**. The correct URL is all-lowercase and returns 200. An earlier Pod-casing sweep appears to have capitalized inside the URL. - `index.md` described a Ray cluster as \"a head node pod and a collection of worker node pods\", stacking both vocabularies in one phrase. Now \"a head Pod and a collection of worker Pods\". - `index.md` also wrote \"heterogenous compute nodes\" on that same line. Now \"heterogeneous\". ## One casing fix folded in `k8s-ecosystem/kai-scheduler.md` wrote \"KAI scheduler deployment\" in one place and \"KAI Scheduler\" everywhere else. The KubeRay Vale vocabulary proposed in #65239 surfaced it, and @win5923 confirmed the two-word title-case form on that PR. Folded in here rather than opened as a one-line PR, per the guidance in `AGENTS.md` about standalone trivial changes. The other hyphenated spellings in that file stay as they are. They sit inside links to `NVIDIA/KAI-Scheduler`, the Helm release name, and the `` placeholder, so none of them are prose. ## Testing ``` pre-commit run --files $(git diff --name-only) ``` Every hook reports \"no files to check\". Stating that plainly rather than implying coverage it doesn't give: no pre-commit hook applies to Markdown or reStructuredText under `doc/source/`, and the `vale` hook is scoped to `^doc/source/data/.*`, so none of these files are linted in CI today. What I verified by hand instead: - **No residual term.** `grep -rIniE '\\bray [*_`]* ?pods?\\b' .` over the whole repo returns only the `python/` code comments listed above as intentionally out of scope. This includes a markup-split case, \"Ray \\*\\*Pod quantities\\*\\*\", that a naive search misses. - **reStructuredText heading integrity.** `gpu.rst` has a section whose title shortened, so its underline was resized to match. Title and underline are both 30 characters. - **No broken anchors.** Four headings changed text. Searched for inbound references to their generated anchors and found none, and none carries an explicit MyST anchor. - **The URL fix.** `curl` returns 404 for the old casing and 200 for the new one. - **Config comments included.** `configs/static-ray-cluster.tls.yaml` carried the term in two YAML comments mirroring `config.md` prose, so the removal is complete rather than Markdown-only. Rendering is verified by the Read the Docs preview build on this PR. No code, no API, no behavior change. ## Not a duplicate ``` gh pr list --repo ray-project/ray --state open --search \"Ray Pod\" gh pr list --repo ray-project/ray --state open --search \"pod terminology\" ``` No open PR proposes this change. #65239 is the style-guide PR where the question came up and is the discussion surface for the rule; this PR is the cleanup and stands on its own against master. Two of my open PRs touch nearby files: #65287 rewrites lines in `jax-tpu-profiling.md`, which this PR doesn't touch, and #65263 is a draft adding `scheduling.md`, which introduces one new use of the term that I'll fix there rather than here. Per `AGENTS.md`, mechanical cleanups need maintainer coordination. That's the thread on #65239 with @win5923, who reviewed and approved the style-guide change and raised the question this PR answers. ## AI assistance Written with AI assistance (Claude). I've reviewed every changed line, generated and inspected the counts myself, and can defend the change end to end. --- ### Update: four casing defects folded in (f9fc8ea) Extending the KubeRay Vale vocabulary in #65239 turned four latent casing errors into alerts. They belong with the rest of the terminology cleanup rather than in a PR of their own. - `configuring-ippr.md`, two uses: \"Kubelet\" to \"kubelet\". Kubernetes writes the component lowercase in prose. - `rayservice-incremental-upgrade.md`: \"Install istio\" to \"Install Istio\" in a step label, where the word names the project. The lowercase `istio` elsewhere on that page is correct and stays, since it's a GatewayClass `metadata.name` and a `controllerName` value rather than the project name. - `resource-isolation-with-writable-cgroups.md`: the H1 read \"Resource Isolation with Writable Cgroups on Google Kubernetes Engine (GKE)\". Lowercasing only \"Cgroups\" would leave a title-case heading with one lowercase word, so the whole heading moves to sentence case, which the style guide requires anyway. **One coupling worth flagging for review.** The explicit MyST label above that H1 is unchanged, so every `{ref}` to the page still resolves, and the bare `{ref}` in `cluster/kubernetes/user-guides.md` picks up the new title automatically. But `ray-core/resource-isolation-with-cgroupv2.rst` passes custom link text that mirrored the old title case, so that text moves to sentence case too. That's the one file this PR touches outside `doc/source/cluster/kubernetes/`, and it's a one-line consistency fix caused directly by the heading change. Verified against the #65239 ruleset by diffing the alert set for these files before and after the change: the four alerts clear and no new alert appears. AI assistance was used for this commit and for that verification. ### Update: a bad replacement in this PR, plus three defects in the same files (89b0e49) **This PR had a bug.** The term replacement turned \"KubeRay pods\" into \"KubePods\" in `tpu.md`: the pattern matched \"Ray pods\" inside \"KubeRay pods\" and left the \"Kube\" prefix glued to the replacement. `git log -S` traces it to the retirement commit, so it isn't pre-existing. It's now \"Pods\", which is what the rule asks for when the qualifier only names the framework. I swept every added line in this PR for the same signature, a lowercase letter glued to \"Pod\" or \"Pods\", and this was the only instance. The other repo-wide matches are Kubernetes event reasons such as `CreatedHeadPod`, correct as literal strings. Three more defects, in files this PR already edits: - `tpu.md`: \"GKE nodepool\" to \"GKE node pool\" in prose. The `az aks nodepool` spelling elsewhere in the docs is a CLI subcommand and stays. - `prometheus-grafana.md`: \"spawned by replicaset controller\" to \"spawned by a ReplicaSet controller\", fixing the API object casing and a missing article. - `rayservice-incremental-upgrade.md`: \"utilizing\" to \"using\", \"like\" to \"such as\" for an example, and \"(e.g. `istio`)\" to \"for example `istio`\", plus the imperative in place of \"You will need to\". Verified against the #65239 ruleset with a column-independent diff of the alert set for these files: five alerts clear, no new finding. The two passive-voice alerts that change column position are pre-existing and unchanged in substance. **Deliberately not in this PR.** These sit in files this PR doesn't otherwise touch, so folding them in would widen its file scope for unrelated reasons: \"Deepseek\" for DeepSeek in `examples/rayserve-deepseek-example.md` (an H1) and `serve/llm/architecture/overview.md`; `gsutil` in `examples/distributed-checkpointing-with-gcsfuse.md`, where `gcloud storage` supersedes it and swapping a documented command needs someone who can verify the equivalent invocation; and \"nodepool\" in the prose of `user-guides/azure-aks-gpu-cluster.md`. AI assistance was used for this commit and for the verification above.", "url": "https://github.com/ray-project/ray/pull/65423", "createdAt": "2026-08-12T13:14:06Z", "updatedAt": "2026-08-13T12:25:43Z", "timestamp": "2026-08-13T12:25:43Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "docs", "core", "go", "docs-go" ], "author": "dstrodtman", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:3c7b07ea27e35712f81b", "signalId": "github:ray-project/ray:pull_request:65267", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65267", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[serve] Env-gated observe layer4 mark-down on HAProxy servers", "text": "## Summary When a node is drained (spot reclaim, compaction, scale-down), its displaced (soft-stopped) HAProxy processes keep serving established connections with a frozen config in which torn-down replicas remain nominally UP. Requests those processes accept are black-holed until `hard-stop-after` instead of failing over. This PR adds an env-gated `observe layer4 error-limit on-error mark-down` to backend replica servers so live traffic itself marks dead servers DOWN and the existing `backup` fallback takes over. Default off; the rendered config is byte-identical when unset. ## Chain of events 1. A node starts draining (2026-08-04 incident: spot reclaim; 2026-07-23: replica compaction). 2. The controller migrates the node's replicas; replacements are healthy elsewhere in ~2s. 3. Updated backend configs are pushed to every proxy. 4. Each HAProxy applies the new config by reloading: a new process starts, and the old one is soft-stopped (`-sf`) but keeps serving its established connections. 5. The old process's view is frozen: the stopped replicas stay UP in its map, and the replacements don't exist in it. 6. A request arrives on a connection held by the old process; the frontend accepts it. 7. Dispatch picks a stale server; the connect fails β€” the replica is gone. 8. Retries and redispatch only consider servers believed UP: the same dead ones. `nbsrv > 0`, so the `backup` fallback stays ineligible. 9. No error is ever returned; the client burns its full timeout (150s in our fleet). 10. Steps 6-9 repeat for every request landing on that process until `hard-stop-after` (400s) kills it: queued requests die as 502s (last client failures at drain+401s, to the second), and traffic then flows only to healthy processes. Impact: 8 failed client requests over ~6.5 minutes, exactly on the apps whose replicas all sat on the drained node (echo 1/1 replica, highscale 2/2). ## Fix `RAY_SERVE_HAPROXY_OBSERVE_MARK_DOWN_ENABLED` (default off) renders `observe layer4 error-limit on-error mark-down` (`RAY_SERVE_HAPROXY_OBSERVE_ERROR_LIMIT`, default 3) on HTTP and gRPC replica server lines. Connection-level failures observed by live traffic mark the server DOWN synchronously in the request path β€” verified in HAProxy source: `__health_adjust()` runs inline with no `stopping` guard, and soft-stop does not destroy check tasks, so this works inside displaced processes, the one place nothing else can. With all primaries DOWN, `option redispatch` + the existing `backup` fallback serve the request. Recovery is automatic: DOWN servers keep being health-probed and `rise` passes restore them in ~0.5s, so a false positive self-heals. Backup/fallback servers are never observed; router-path servers inherit state via `track`.", "url": "https://github.com/ray-project/ray/pull/65267", "createdAt": "2026-08-06T18:56:27Z", "updatedAt": "2026-08-13T12:08:26Z", "timestamp": "2026-08-13T12:08:26Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "serve", "go" ], "author": "harshit-anyscale", "state": "closed", "assignees": [ "harshit-anyscale" ], "change": "updated" } }, { "id": "event:d46115b48de7428347d6", "signalId": "github:ray-project/ray:pull_request:65228", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65228", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[Data] Deprecate `ray_remote_args` for Dataset transformations", "text": "### What changed This PR adds named worker options to: - `map` - `map_batches` - `flat_map` - `with_columns` - `with_column` - `filter` - `add_column` - `drop_columns` - `select_columns` - `rename_columns` The new parameters are: - `label_selector` - `fallback_strategy` - `max_calls` - `resources` - `accelerator_type` - `runtime_env` - `max_concurrency` Options passed through `**ray_remote_args` continue to work, but now emit a `RayDeprecationWarning`. `ray_remote_args` will be removed in Ray 2.64. `placement_group_bundles` and `placement_group_strategy` need additional execution support, so they will be added separately after #64090. Reads, writes, `random_shuffle`, and `join` are out of scope. --- Closes https://github.com/ray-project/ray/issues/65227", "url": "https://github.com/ray-project/ray/pull/65228", "createdAt": "2026-08-05T11:48:52Z", "updatedAt": "2026-08-13T12:02:15Z", "timestamp": "2026-08-13T12:02:15Z", "metrics": { "reactions": 1, "comments": 1 }, "labels": [ "data", "deprecation", "go", "community-contribution" ], "author": "yuhuan130", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:9ec975a4478cc21d0dbd", "signalId": "github:ray-project/ray:pull_request:61898", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:61898", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[Core] Mobilint Accelerator Support", "text": "## Description This PR adds first-class support for Mobilint MBLT NPUs to Ray's accelerator abstraction by introducing a new `MBLTAcceleratorManager` under `ray/_private/accelerators/mblt.py`. This integration is developed and contributed by ETRI (Electronics and Telecommunications Research Institute) in collaboration with Mobilint. Based on the SDKs provided by Mobilint, ETRI has implemented this software integration to align with Ray's official accelerator policies and architectural standards. The goal is to make MBLT resources natively schedulable in Ray (e.g., `resources={\"MBLT\": 1}`) and to provide per-worker/actor device isolation via a conventional \"visible devices\" environment variable, consistent with Ray's existing accelerator patterns (GPU, TPU, Neuron, etc.). **Key behaviors:** - Resource name: `MBLT` - Scheduling unit: Whole cards only (no fractional sharing). - Visibility isolation: Ray sets `MBLT_DEVICES` (comma-separated IDs) for worker/actor processes. - Runtime IDs: Users can access assigned IDs via `ray.get_runtime_context().get_accelerator_ids()`. ## Motivation Mobilint MBLT devices are increasingly utilized in heterogeneous clusters for specialized AI workloads. Currently, users must rely on custom resources and manual environment variable management to use these NPUs with Ray. This manual process is error-prone and breaks the seamless Ray-native experience. By introducing this PR, ETRI aims to provide a standardized interface for the MBLT lifecycle within the Ray ecosystem so that: - Ray's internal scheduler treats MBLT cards as first-class accelerators. - Cluster-wide resource management and isolation remains consistent across different hardware types. - Research and industrial users at ETRI and beyond can deploy MBLT-based workloads with minimal configuration. ## Usage example ```python import ray # Requesting 1 MBLT NPU card @ray.remote(resources={\"MBLT\": 1}) class MBLTWorker: def get_info(self): return ray.get_runtime_context().get_accelerator_ids() w = MBLTWorker.remote() # Expected output: {\"MBLT\": [\"0\"]} print(ray.get(w.get_info.remote())) ``` ## Contributors - Software Implementation: ETRI (Electronics and Telecommunications Research Institute) - Hardware/SDK Support: Mobilint", "url": "https://github.com/ray-project/ray/pull/61898", "createdAt": "2026-03-20T01:32:36Z", "updatedAt": "2026-08-13T11:45:13Z", "timestamp": "2026-08-13T11:45:13Z", "metrics": { "reactions": 3, "comments": 7 }, "labels": [ "core", "go", "community-contribution" ], "author": "dataminsu", "state": "open", "assignees": [ "Yicheng-Lu-llll" ], "change": "updated" } }, { "id": "event:c1ee734c3c14f2b132c5", "signalId": "github:ray-project/ray:pull_request:65390", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65390", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "Fix/tls server cert hot reload", "text": "## Description Fixes the **server-side** half of #65365. Ray's Python gRPC servers previously read TLS cert/key/CA files once at startup via `grpc.ssl_server_credentials()`. When cert-manager rotates files on disk (e.g. KubeRay mTLS), running servers kept presenting the old certificate until process restart. This PR switches `add_port_to_grpc_server()` to `grpc.dynamic_ssl_server_credentials()` with a `_ReloadableServerCertConfig` fetcher that: - Re-reads cert files when their mtimes change - Seeds the mtime cache after the initial load to avoid redundant reloads - Falls back to the last known-good configuration on stat/read/parse errors (raising from the fetcher would break new handshakes) **Affected Python gRPC servers:** Ray Client server, dashboard agent gRPC, Serve replica internal gRPC. ### Known gaps (out of scope) | Area | Status | Notes | |------|--------|-------| | **Client/channel side** (`init_grpc_channel` in `grpc_utils.py`) | Not in this PR | Still reads TLS material once via `grpc.ssl_channel_credentials()`. Blocked on [grpc/grpc#41779](https://github.com/grpc/grpc/issues/41779) β€” Python gRPC has no dynamic reload API for channels. | | **C++ core** (`src/ray/rpc/grpc_server.cc`) | Not in this PR | GCS, raylet, object manager, etc. load certs once at startup via `grpc::SslServerCredentials()`. Separate path, unaffected by this change. | ## Related issues Related to #65365 (server-side scope). ## Additional information ### Test plan - [x] `python -m pytest -v -s python/ray/_common/tests/test_tls_utils.py` β€” 7/7 passed (macOS) - [ ] `python -m pytest -v -s python/ray/tests/test_tls_auth.py::test_tls_server_cert_rotation_without_restart` β€” skipped locally on darwin; runs in CI on Linux - [x] `pre-commit run --from-ref master --to-ref HEAD` β€” passed ### Contribution notes - No duplicate open PR found for this change. - AI assistance was used; all changes reviewed by the submitter. - Every commit includes DCO `Signed-off-by`.", "url": "https://github.com/ray-project/ray/pull/65390", "createdAt": "2026-08-11T19:55:54Z", "updatedAt": "2026-08-13T11:21:33Z", "timestamp": "2026-08-13T11:21:33Z", "metrics": { "reactions": 1, "comments": 3 }, "labels": [ "core", "community-contribution" ], "author": "Divyam19", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:a30046d5cf41d44b31ad", "signalId": "github:ray-project/ray:pull_request:65441", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65441", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[doc][History server] Update doc for history server for `RAY_ROOT_DIR` -> `STORAGE_ROOT_DIR`", "text": "## Description https://github.com/ray-project/kuberay/pull/5121 updated the env var from `RAY_ROOT_DIR` to `STORAGE_ROOT_DIR`. The default storage root dir is now the bucket root itself, so updated example and file structure in the documentation. ## Related issues Related to https://github.com/ray-project/kuberay/pull/5121 ## Additional information", "url": "https://github.com/ray-project/ray/pull/65441", "createdAt": "2026-08-12T22:15:06Z", "updatedAt": "2026-08-13T11:18:33Z", "timestamp": "2026-08-13T11:18:33Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "core", "go", "community-contribution" ], "author": "chiayi", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:0b02053ecf3329aeff98", "signalId": "github:ray-project/ray:pull_request:65107", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65107", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[doc][KubeRay] Add mTLS for RayClusters user guide", "text": "## Description Adding user guide for mTLS configuration with RayClusters via KubeRay as per the new alpha feature in KubeRay 1.7 ## Related issues Relates to https://github.com/ray-project/kuberay/issues/5048 ## Additional information N/A", "url": "https://github.com/ray-project/ray/pull/65107", "createdAt": "2026-07-29T16:40:10Z", "updatedAt": "2026-08-13T11:15:12Z", "timestamp": "2026-08-13T11:15:12Z", "metrics": { "reactions": 1, "comments": 7 }, "labels": [ "docs", "core", "go", "community-contribution" ], "author": "chipspeak", "state": "open", "assignees": [ "machichima" ], "change": "updated" } }, { "id": "event:68d9c1a255bf2591fbd0", "signalId": "github:ray-project/ray:issue:64693", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:issue:64693", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "issue", "title": "[Core] Removing an in-flight placement group can leak prepared bundles and block later placement groups", "text": "## What happened + What you expected to happen Removing a placement group while bundle preparation is in flight can leave both prepared raylet resources and the GCS placement-group scheduling gate stuck if one prepare callback never arrives. The scheduler already documents that a `PrepareBundleResources` callback may never be called after node death. In this sequence: 1. A placement group starts preparing bundles on two nodes. 2. Node A reports successful preparation, so GCS knows that bundle is reserved. 3. Node B dies and its prepare callback is withheld. 4. The placement group is removed and `MarkScheduleCancelled` records only the canceled state. 5. Cleanup and manager completion remain dependent on the missing callback. Actual behavior: - no `RemovePlacementGroupBundles` RPC is sent for the known prepared bundle; - the manager's `scheduling_in_progress_id_` remains owned by the removed group; - a later pending placement group is not scheduled. Expected behavior: - cancellation immediately cleans up every prepared bundle already known to GCS; - a successful prepare callback that arrives after cancellation is cleaned up immediately; - removing the active group releases its manager scheduling token without waiting for unrelated callbacks that may never arrive. ## Versions / Dependencies - Ray: `3.0.0.dev0`, commit `3d026f5b65e52955cb8b3e21d88941bb0ff5392c` - OS: Ubuntu 24.04, Linux 6.17, x86-64 - Python: 3.12.3 - Bazel: 7.5.0 - GCC: 13.3.0 ## Reproduction script Apply the two complete regression-test patches included at the end, then run: ```bash bazel test //src/ray/gcs/tests:gcs_placement_group_scheduler_test \\ --test_filter=GcsPlacementGroupSchedulerTest.PlacementGroupCancelledAfterPartialPrepareReleasesPreparedBundle \\ --test_output=errors bazel test //src/ray/gcs/tests:gcs_placement_group_manager_test \\ --test_filter=GcsPlacementGroupManagerTest.TestRemovingLeasingPlacementGroupUnblocksPending \\ --test_output=errors ``` The first regression test uses the existing fake raylet clients: ```cpp auto node0 = GenNodeInfo(0); auto node1 = GenNodeInfo(1); AddNode(node0); AddNode(node1); auto pg = std::make_shared( GenCreatePlacementGroupRequest(), \"\", counter_, clock_); const auto pg_id = pg->GetPlacementGroupID(); scheduler_->ScheduleUnplacedBundles(SchedulePgRequest{ pg, [this](auto pg, bool) { absl::MutexLock lock(&placement_group_requests_mutex_); failure_placement_groups_.push_back(std::move(pg)); }, [this](auto pg) { absl::MutexLock lock(&placement_group_requests_mutex_); success_placement_groups_.push_back(std::move(pg)); }}); ASSERT_TRUE(raylet_clients_[0]->GrantPrepareBundleResources()); RemoveNode(node1); // Its prepare callback never arrives. scheduler_->MarkScheduleCancelled(pg_id); ASSERT_TRUE(raylet_clients_[0]->GrantRemovePlacementGroupBundles()); ASSERT_EQ(raylet_clients_[0]->num_bundles_removed, 1); ``` Current `master` fails the cleanup assertion: ```text Value of: raylet_clients_[0]->GrantRemovePlacementGroupBundles() Actual: false Expected: true ``` The manager regression independently fails because the second placement group is never handed to the scheduler: ```text Expected equality of these values: mock_placement_group_scheduler_->GetPlacementGroupCount() Which is: 0 1 ``` ## Relevant implementation `GcsPlacementGroupScheduler::MarkScheduleCancelled` currently marks the lease tracker canceled but does not clean its already-known prepared locations. `OnAllBundlePrepareRequestReturned` performs cleanup only after every callback has returned. `GcsPlacementGroupManager::RemovePlacementGroup` records cancellation but does not release the scheduling token for that group. A candidate fix that cleans known prepared locations on cancellation, cleans a late successful prepare immediately, and clears only the matching manager token makes both regression tests pass. It also passes the existing nearby placement- group cancellation/removal tests. Historical issue #17258 and PR #17320 concern a different race between removal and a placement-group storage callback after scheduling had already succeeded. This report concerns partial prepare progress plus a callback that may never arrive. ## Issue Severity High: one removed placement group can leave resources reserved and prevent subsequent placement groups from being scheduled indefinitely. ## Complete scheduler regression-test patch ```diff diff --git a/src/ray/gcs/tests/gcs_placement_group_scheduler_test.cc b/src/ray/gcs/tests/gcs_placement_group_scheduler_test.cc index 8c7b8a4..3938f23 100644 --- a/src/ray/gcs/tests/gcs_placement_group_scheduler_test.cc +++ b/src/ray/gcs/tests/gcs_placement_group_scheduler_test.cc @@ -627,6 +627,41 @@ TEST_F(GcsPlacementGroupSchedulerTest, DestroyCancelledPlacementGroup) { WaitPlacementGroupPendingDone(1, GcsPlacementGroupStatus::FAILURE); } +TEST_F(GcsPlacementGroupSchedulerTest, + PlacementGroupCancelledAfterPartialPrepareReleasesPreparedBundle) { + auto node0 = GenNodeInfo(0); + auto node1 = GenNodeInfo(1); + AddNode(node0); + AddNode(node1); + ASSERT_EQ(2, gcs_node_manager_->GetAllAliveNodes().size()); + + auto create_placement_group_request = GenCreatePlacementGroupRequest(); + auto placement_group = std::make_shared( + create_placement_group_request, \"\", counter_, clock_); + const auto &placement_group_id = placement_group->GetPlacementGroupID(); + + scheduler_->ScheduleUnplacedBundles(SchedulePgRequest{ + placement_group, + [this](std::shared_ptr placement_group, bool is_insfeasble) { + absl::MutexLock lock(&placement_group_requests_mutex_); + failure_placement_groups_.emplace_back(std::move(placement_group)); + }, + [this](std::shared_ptr placement_group) { + absl::MutexLock lock(&placement_group_requests_mutex_); + success_placement_groups_.emplace_back(std::move(placement_group)); + }}); + + ASSERT_TRUE(raylet_clients_[0]->GrantPrepareBundleResources()); + RemoveNode(node1); + + scheduler_->MarkScheduleCancelled(placement_group_id); + + ASSERT_TRUE(raylet_clients_[0]->GrantRemovePlacementGroupBundles()); + ASSERT_EQ(raylet_clients_[0]->num_bundles_removed, 1); + ASSERT_EQ(raylet_clients_[0]->commit_callbacks.size(), 0); + ASSERT_EQ(raylet_clients_[1]->commit_callbacks.size(), 0); +} + TEST_F(GcsPlacementGroupSchedulerTest, PlacementGroupCancelledDuringCommit) { auto node0 = GenNodeInfo(0); auto node1 = GenNodeInfo(1); ``` ## Complete manager regression-test patch ```diff diff --git a/src/ray/gcs/tests/gcs_placement_group_manager_test.cc b/src/ray/gcs/tests/gcs_placement_group_manager_test.cc index 09ece01..d4df34c 100644 --- a/src/ray/gcs/tests/gcs_placement_group_manager_test.cc +++ b/src/ray/gcs/tests/gcs_placement_group_manager_test.cc @@ -547,6 +547,40 @@ TEST_F(GcsPlacementGroupManagerTest, TestRemovingLeasingPlacementGroup) { ASSERT_EQ(counter_->Get(rpc::PlacementGroupTableData::REMOVED), 1); } +TEST_F(GcsPlacementGroupManagerTest, TestRemovingLeasingPlacementGroupUnblocksPending) { + auto request1 = GenCreatePlacementGroupRequest(); + std::atomic registered_placement_group_count(0); + RegisterPlacementGroup(request1, [®istered_placement_group_count](Status status) { + ++registered_placement_group_count; + }); + ASSERT_EQ(registered_placement_group_count, 1); + ASSERT_EQ(mock_placement_group_scheduler_->GetPlacementGroupCount(), 1); + auto removed_placement_group = + mock_placement_group_scheduler_->placement_groups_.back(); + mock_placement_group_scheduler_->placement_groups_.clear(); + + auto request2 = GenCreatePlacementGroupRequest(); + RegisterPlacementGroup(request2, [®istered_placement_group_count](Status status) { + ++registered_placement_group_count; + }); + ASSERT_EQ(registered_placement_group_count, 2); + ASSERT_EQ(mock_placement_group_scheduler_->GetPlacementGroupCount(), 0); + + const auto &removed_placement_group_id = + removed_placement_group->GetPlacementGroupID(); + EXPECT_CALL(*mock_placement_group_scheduler_, + MarkScheduleCancelled(removed_placement_group_id)) + .Times(1); + gcs_placement_group_manager_->RemovePlacementGroup( + removed_placement_group_id, [](const Status &status) {}); + RunIOService(); + + gcs_placement_group_manager_->SchedulePendingPlacementGroups(); + ASSERT_EQ(mock_placement_group_scheduler_->GetPlacementGroupCount(), 1); + ASSERT_NE(mock_placement_group_scheduler_->placement_groups_[0]->GetPlacementGroupID(), + removed_placement_group_id); +} + TEST_F(GcsPlacementGroupManagerTest, TestRemovingCreatedPlacementGroup) { auto request = GenCreatePlacementGroupRequest(); std::atomic registered_placement_group_count(0); ```", "url": "https://github.com/ray-project/ray/issues/64693", "createdAt": "2026-07-11T22:11:09Z", "updatedAt": "2026-08-13T11:03:52Z", "timestamp": "2026-08-13T11:03:52Z", "metrics": { "reactions": 0, "comments": 2 }, "labels": [ "core", "stability", "community-backlog" ], "author": "logical-misha", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:a4596ab8501d6157b241", "signalId": "github:ray-project/ray:pull_request:65450", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65450", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[Data] Dereference ObjectRefs passed through map UDF `fn_args`", "text": "## Description Ray Data currently captures UDF `fn_args` in the serialized UDF wrapper. As a result, direct `ObjectRef` values aren't passed as top-level Ray task arguments: - UDFs must manually call `ray.get()`. - Ray's scheduler can't wait for these dependencies before scheduling map work. - Map workers may reserve CPU or GPU resources and then block inside user code. Passing direct `ObjectRef` values as task arguments aligns this behavior with Ray Core and enables dependency-aware scheduling. However, changing the default immediately would break existing UDFs that expect to receive an `ObjectRef` and call `ray.get()` themselves. This PR therefore introduces the new behavior through a backward-compatible migration path. This PR adds `DataContext.enable_dereference_object_refs_in_fn_args`, which defaults to `False`. With the default legacy behavior: - Direct `ObjectRef` values continue to reach the UDF unchanged. - Ray Data emits a `RayDeprecationWarning` describing the migration. Users can opt in before creating the Dataset: ```python ray.data.DataContext.get_current().enable_dereference_object_refs_in_fn_args = True ``` or set: ```shell RAY_DATA_ENABLE_DEREFERENCE_OBJECT_REFS_IN_FN_ARGS=1 ``` When enabled: - Direct `ObjectRef` values in `fn_args` are passed as top-level task arguments. - Ray Core waits for and dereferences them before invoking the UDF. - Transform adapters reconstruct the positional argument tuple once per task. - Sync, async, async-generator, row, batch, flat-map, filter, and callable-class `__call__` paths use the resolved arguments consistently. - Failed dependencies can prevent user code from starting. Users who intentionally need reference semantics after the default changes can wrap the `ObjectRef` in a container. Only direct top-level `ObjectRef` values are automatically dereferenced. ## Related issues closes #65449 and related to #49207 (`fn_kwargs` and callable-class constructor arguments remain separate follow-up surfaces.) ## Additional information The download planner is updated because it also consumes the private `_get_udf` helper. The batch-transform helper retains its existing raw one-argument UDF calling convention when no argument resolver is supplied. Observable behavior changes only for direct `ObjectRef` values in `fn_args`: - The UDF now receives the resolved value. - Failed dependencies can stop a map task before user code begins. - Map-to-all-to-all execution remains unfused while task kwargs are present.", "url": "https://github.com/ray-project/ray/pull/65450", "createdAt": "2026-08-13T06:40:45Z", "updatedAt": "2026-08-13T10:51:14Z", "timestamp": "2026-08-13T10:51:14Z", "metrics": { "reactions": 1, "comments": 2 }, "labels": [ "data", "community-contribution" ], "author": "viiccwen", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:98704a07cdd96ed91f78", "signalId": "github:ray-project/ray:pull_request:65366", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65366", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[core][runtime_env] Support isolated and cached pip dependencies with image_uri containers", "text": "## Description Allow combining the `image_uri` runtime environment field with pip. The runtime env agent prepares one virtual environment per cache key (image digest, Python ABI, Ray version, normalized pip config, and pip install environment) using the image's own interpreter, publishes it atomically under a per-key file lock, and mounts it read-only into every equivalent worker container instead of reinstalling per worker. Highlights: - Node-resolved cache URIs (resolve_uris) with dynamic reference counting in the agent, including release of orphaned references when an env is deleted while its creation is still in flight. - Atomic publish: staging dir + fsynced manifest + rename; failed or cancelled installs are cleaned up and never published. - Cross-process flock protocol shared by create, eviction, and the startup staging sweep; eviction renames to trash and removes the tree off the event loop. - Workers with pip run the exact image the environment was built against; image-only envs keep tag references and podman auto-pull. - Python/Ray version match enforced when pip is used, warned otherwise. ## Related issues Closes #65322", "url": "https://github.com/ray-project/ray/pull/65366", "createdAt": "2026-08-11T11:39:54Z", "updatedAt": "2026-08-13T10:36:32Z", "timestamp": "2026-08-13T10:36:32Z", "metrics": { "reactions": 1, "comments": 1 }, "labels": [ "core", "community-contribution" ], "author": "zzchun", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:8463d188fdaec50f1a29", "signalId": "github:ray-project/ray:pull_request:65403", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65403", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[TPU] Add subslice_index to subslice_placement_group", "text": "## Description When requesting multiple subslices simultaneously, it can be helpful to allow selecting specific subslices to guarantee a spread of jobs. This change adds an optional subslice_index parameter to the subslice_placement_group builder which restricts placement group selection. ## Related issues Follow on to #64578 ## Additional information n/a", "url": "https://github.com/ray-project/ray/pull/65403", "createdAt": "2026-08-11T23:41:38Z", "updatedAt": "2026-08-13T10:15:49Z", "timestamp": "2026-08-13T10:15:49Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "train", "core", "community-contribution" ], "author": "spencer-p", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:3f97b20798c566b07927", "signalId": "github:ray-project/ray:pull_request:65448", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65448", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[Core] Randomize worker port allocation", "text": "## Description When multiple raylets share a network namespace and use the same worker port range, they currently allocate ports in the same ascending order. This causes deterministic collisions near the beginning of the range, even when most ports remain available. This PR builds and validates the worker port pool once during raylet startup, then shuffles it before constructing the free-port queue. It: - Shuffles both configured ranges and explicit `--worker-port-list` values. - Rejects out-of-range and duplicate explicit ports with errors that identify the offending port. - Preserves explicit-list precedence and the existing `max_worker_port=0` behavior. - Adds `worker_port_shuffle_enabled`, defaulting to `true`, with `false` restoring the previous allocation order. - Leaves `GetNextFreePort` and `MarkPortAsFree` unchanged. - Documents that randomization reduces collision probability but does not provide cross-raylet port reservation. Non-overlapping ranges or `--min-worker-port=0 --max-worker-port=0` remain the robust solutions. ## Related issues Closes #65444. ## Additional information The shuffle occurs only once, during `WorkerPool` construction. It does not change the configured set of ports or the existing occupied-port retry and port-return behavior.", "url": "https://github.com/ray-project/ray/pull/65448", "createdAt": "2026-08-13T06:11:56Z", "updatedAt": "2026-08-13T09:56:46Z", "timestamp": "2026-08-13T09:56:46Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "core", "community-contribution" ], "author": "zzchun", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:ae7fbe45a4f1818147c8", "signalId": "github:ray-project/ray:pull_request:64828", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:64828", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[Data][1/N] add external shuffle runtime library", "text": "Adds the file-transport runtime for a new external (on-disk) shuffle variant: TCP wire protocol, per-node ShuffleFileServer actor, connection / fetch primitives, and error classification. This PR lands only the runtime library and its testers required by external shuffle tasks. It's not wired into the plan yet and there's no user-visible behavior change. ## description - Reducers open one arrow flight connection per source ShuffleFileServer, authenticate with a per-shuffle token, and fetch data straight to disk. - ShuffleFileServer will be started by map task to serve reducer's file read request. Upon init, ShuffleFileServer will bind file serving function to a random available port. - Fault tolerance is driven by Ray's actor state (ActorDied / ActorUnavailable / ActorUnschedulable). On mid-fetch TCP failure, the reducer compares endpoints: if the manager was restarted (where the endpoint are likely to be different) we retry in place; if the endpoint is unchanged (which is Ray RPC still works but our TCP path doesn't) it's almost always a network-configuration problem (NetworkPolicy, firewall, routing) that retries to the same manager can't fix, so we surface a terminal ShuffleFileServerAnomalyError. ## Additional information The complete implementation of shuffle is in https://github.com/ray-project/ray/pull/64733, including runtime libraries, external shuffle task definition, plan wiring and a user-visible flag. I plan to do three PRs (1 is runtime libraries, 2 is task definition and wiring, and 3 is the user-visible flag) to gradually merge the complete implementation.", "url": "https://github.com/ray-project/ray/pull/64828", "createdAt": "2026-07-17T01:39:05Z", "updatedAt": "2026-08-13T09:38:04Z", "timestamp": "2026-08-13T09:38:04Z", "metrics": { "reactions": 1, "comments": 4 }, "labels": [ "data", "go" ], "author": "ShockYoungCHN", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:0b0d91e5b5b83f460b71", "signalId": "github:ray-project/ray:pull_request:64928", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:64928", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[Train] Add NCCL RAS health callback", "text": "## Description This PR adds an opt-in callback that detects hanging distributed training jobs by polling NCCL's RAS (Reliability/Availability/Serviceability) subsystem. NCCL β‰₯ 2.24 runs a monitoring thread inside every NCCL process that forms a peer mesh tracking per-rank health and collective op-counts. By querying the `ncclras` client and diffing collective-counts between successive polls, Ray Train can automatically detect and act on hangs that would otherwise stall a job until a torch process group timeout occurs. As RAS gives a snapshot of the current NCCL state, to determine if the mismatch is a temporary issue or a permanent one, the callback polls RAS every N seconds. If a communicator is mismatched *and* makes no progress at all across multiple sequential polls, then this most likely indicates that a deadlock / wedge / hang is in progress for that communicator. Its important to note that its possible for NCCL to continue (not hang) with incorrect data depending on the nccl operation, tensor size, nccl version, etc. In these cases, users will observe the collective-count continue to increase but often with a constant mismatch. In this version of the callback, we are not detecting those issues and focusing on purely NCCL collective mismatches which cause hangs. The callback is disabled by default and enabled with `RAY_TRAIN_ENABLE_NCCL_HANG_DETECTOR=1`, as the detection heuristics still need validation against real-world hangs (spot preemptions, pipeline-parallel send/recv comms that legitimately sit skewed for long periods) before it can be considered for default-on. ### Requirements - RAS itself landed in 2.24, but the poller parses `ncclras -f json`, and the `-f` flag requires NCCL 2.28+. If the binary rejects `-f`, or isn't on PATH, the callback logs a warning once and disables itself for the rest of the run rather than retrying forever. - `ci/docker/base.gpu.Dockerfile` pins and holds `libnccl2`/`libnccl-dev` at `2.28.9-1+cuda12.9` (previously whatever the CUDA base image shipped) so `ncclras` is present and JSON-capable in GPU CI. The build fails fast if the pin doesn't stick or the client binary is missing. This affects all GPU CI images, not just Train tests. - `py-spy` (and `SYS_PTRACE` in the container) for native stack traces at hang time. When unavailable, the callback falls back to a Python-only traceback of every thread. ### How it works - `NCCLRASCallback` lives on the controller and polls `ncclras` on a worker (ras forms a mesh-network between ranks requiring that only one of the workers to be polled and receive the whole network's state). Workers are tried in turn until one returns a usable report, and the query runs on a background thread so it never blocks the controller's poll loop. - A communicator's frozen-streak only advances when the RAS report shows a collective-count mismatch, every rank in that communicator is `RUNNING`, and *no* rank advanced *any* collective since the previous poll. Any progress on any op resets that communicator's streak; streaks are tracked per communicator so each one is confirmed on its own. - A confirmed hang requires the streak to persist for `RAY_TRAIN_NCCL_RAS_CONFIRM_DURATION_S` (default 600s), converted to a consecutive-poll count using `RAY_TRAIN_NCCL_RAS_MIN_POLL_INTERVAL_S` (default 15s), 40 polls by default, to ensure that a mismatch isn't a snapshot issue. - Before confirmation there is an escalation ladder: a `WARNING` when a communicator first crosses ~60s frozen, then a periodic `WARNING` every ~120s naming every still-frozen communicator, its stalled duration, and (in `fail` mode) the time remaining until `NCCLHangError`, alongside the human-readable `ncclras -f text` report. A communicator that resumes progress after being suspected logs an explicit recovery message. - On a confirmed hang, the callback logs the `ncclras -f text` report and captures py-spy stack traces from all workers (uploaded to `/nccl_ras_hang_stack_traces/` as `rank_.log`), then depending on the callback mode, in \"fail\" a `NCCLHangError` is raised and in \"observe\", a detailed log message is produced but no more. `observe` is the default; set `RAY_TRAIN_NCCL_RAS_ACTION=fail` to fail the run. - `NCCLHangError` (a `WorkerGroupError` subclass, `DeveloperAPI`, exported as `ray.train.NCCLHangError`) is treated as non-retryable by the `DefaultFailurePolicy` regardless of the `max_failures` budget, since a desync hang is usually deterministic and a restart would just hang again. ### Configuration | Env var | Default | Read on | | --- | --- | --- | | `NCCL_RAS_ADDR` | `localhost:28028` | worker | | `RAY_TRAIN_ENABLE_NCCL_HANG_DETECTOR` | `0` | driver | | `RAY_TRAIN_NCCL_RAS_ACTION` | `observe` | driver | | `RAY_TRAIN_NCCL_RAS_MIN_POLL_INTERVAL_S` | `15` | driver | | `RAY_TRAIN_NCCL_RAS_CONFIRM_DURATION_S` | `600` | driver | | `RAY_TRAIN_NCCLRAS_PATH` | `ncclras` (PATH lookup) | driver | The `RAY_TRAIN_*` knobs are read once when the callback is constructed on the driver validated eagerly (invalid values raise at construction); `NCCL_RAS_ADDR` is NCCL's variable and is read on the worker at query time. ### Known limitations - **Symmetric in-collective hangs are not detected.** RAS reports collective *launch* counts, so if every rank launches the same collective and the fabric wedges mid-op, all counts match and skew is zero. - **A rank that exited early is not detected.** A communicator is only considered for detection when all of its ranks report `RUNNING`, so a `FINALIZE`/`ABORT` rank alongside frozen peers is currently skipped. This could be considered in this implementation but I believe the Ray health checker should catch this first. - `missing_ranks` (unresponsive / considered-dead peers) is parsed around but not yet used as a signal. - No metrics are emitted yet, and there is no culprit-rank attribution β€” `NCCLHangError.worker_failures` is empty; the RAS text report and stack traces are the only diagnostics. ### Testing - `test_nccl_ras_callback.py` β€” CPU unit coverage of RAS JSON parsing (including the malformed `missing_ranks[]` comma NCCL 2.28.9 emits), `NCCL_RAS_ADDR` parsing, frozen-vs- advancing classification, per-communicator streak independence, communicators appearing and disappearing between polls, poll throttling, confirm-duration β†’ poll-count conversion, config validation, the escalation/observe messaging, and that a failed stack dump does not suppress the hang error. - `test_nccl_ras_hang_detection.py` β€” GPU end-to-end scenarios (`train_v2_gpu`) inducing real NCCL desyncs in a `TorchTrainer` with the detector configured for fast confirmation. Requires β‰₯ 2 visible GPUs and `ncclras` on PATH; the multi-communicator subset test requires 4 GPUs and skips otherwise. - `test_failure_policy.py` β€” added `test_nccl_hang_error_is_non_retryable`.", "url": "https://github.com/ray-project/ray/pull/64928", "createdAt": "2026-07-22T10:44:09Z", "updatedAt": "2026-08-13T09:28:47Z", "timestamp": "2026-08-13T09:28:47Z", "metrics": { "reactions": 1, "comments": 2 }, "labels": [ "train" ], "author": "pseudo-rnd-thoughts", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:c3a7406d9ae6d416061c", "signalId": "github:ray-project/ray:pull_request:65424", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65424", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[core] Enable TCP keepalive on GCS<->Redis connections", "text": "## Description Idle GCS↔Redis flows get silently removed by NAT, proxies, load balancers, and managed-service gateways. The GCS only finds out on its next command, which then hangs until a timeout instead of failing fast β€” and on a managed Redis behind an LB with a 60–350s idle timeout, this is a routine event, not an edge case. This PR enables TCP keepalive on every external Redis socket, applied from a single place in `ConnectWithoutRetries` so that sync, async, Sentinel, Cluster-redirect, and cleanup connections all get the same policy before TLS, AUTH, or any command runs. Two knobs, deliberately decoupled: | Config | Default | Purpose | | --- | --- | --- | | `redis_tcp_keepalive_interval_seconds` | `30` | Keeps an idle flow alive. Must stay **below** the shortest idle timeout on the network path. `0` disables keepalive entirely. | | `redis_tcp_keepalive_probes` | `9` | How many unanswered probes declare the connection dead, so detection β‰ˆ `interval * (1 + probes)`. | These two need opposite tuning β€” probes must be frequent enough to keep a flow alive, while declaring a connection dead must be slow enough to ride out transient congestion. hiredis ties them together (`TCP_KEEPINTVL = interval/3`, `TCP_KEEPCNT = 3`, so dead at `2 * interval` β‰ˆ 60s), so this PR overrides both on glibc Linux to get ~5 minutes instead. That margin matters because in-place reconnect has not landed yet: a connection declared dead escalates to a GCS crash once the request retry budget is exhausted (`RedisRequestContext::Run` β†’ `RAY_LOG(FATAL)`, and the periodic health check's `RAY_CHECK_OK` in `gcs_server.cc`). The docs carry an explicit warning about this, and the default is chosen conservatively for the same reason. **Platform support.** Only glibc Linux honors the probe count, because hiredis guards `TCP_KEEPIDLE`/`TCP_KEEPINTVL`/`TCP_KEEPCNT` on `__GLIBC__`. On musl only `SO_KEEPALIVE` is applied and kernel defaults govern the timing; macOS applies the interval alone; Windows uses a system-fixed probe count. The code, the runtime log, the config comments, the docs, and the test guards all agree on this β€” the log prints the *effective* values, not the configured ones, so a post-mortem shows which applied. ## Related issues Related to #65298 (`[core][2/N] Reconnect to Redis in place when the connection drops`). **This is not a duplicate.** I checked the open PR list before starting: - `gh pr list --repo ray-project/ray --state open --search \"redis keepalive\"` β†’ no results - `gh pr list --repo ray-project/ray --state open --search \"tcp keepalive in:title,body\"` β†’ only unrelated dependabot PRs - `gh pr diff 65298 | grep -iE \"keepalive|SO_KEEPALIVE|KEEPIDLE\"` β†’ **zero hits** #65298 is *recovery* (reconnect once a connection is known dead). This PR is *detection and prevention* (keep the flow alive; find out promptly when it isn't). They are complementary, and this PR does not implement or pre-empt any part of #65298. **Coordination note for reviewers:** #65298 is open and touches four of the same files (`redis_context.cc`, `redis_context.h`, `ray_config_def.h`, and `kuberay-gcs-ft.md`), so whichever lands second will need a rebase. Two specific follow-ups once #65298 merges: 1. The `{warning}` block added here (\"Ray does not yet reconnect to Redis in place… the GCS process exits\") becomes false and must be rewritten. 2. The `redis_tcp_keepalive_probes` default of 9 is conservative *because* a dead connection currently means a crash. With reconnect in place, a shorter detection window becomes safe and probably desirable. Happy to rebase on top of #65298 instead, or to hold this until it merges β€” reviewer's call. Builds on #64299 (`[core][1/N] Make RedisContext::Connect non-fatal on connection failure`), already merged: this PR relies on `Connect()` returning a `Status`, and extends the same contract by making configuration errors non-retryable.", "url": "https://github.com/ray-project/ray/pull/65424", "createdAt": "2026-08-12T13:52:19Z", "updatedAt": "2026-08-13T08:14:18Z", "timestamp": "2026-08-13T08:14:18Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "docs", "core", "community-contribution" ], "author": "zzchun", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:f49c60712d0fffe600d5", "signalId": "github:ray-project/ray:issue:65449", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:issue:65449", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "issue", "title": "[Data] ObjectRefs passed to map UDFs through `fn_args` are not dereferenced", "text": "### What happened + What you expected to happen A direct `ObjectRef` passed as an element of `fn_args` reaches the UDF as an `ObjectRef` instead of its resolved value. Because the reference is captured in the serialized UDF wrapper, Ray's scheduler also cannot wait for the dependency before scheduling the map task. Expected behavior: - Direct `ObjectRef` elements of `fn_args` are top-level map-task arguments. - Ray waits for those dependencies before running the task. - The UDF receives the resolved values. PR #49208 added the map-task kwargs plumbing needed to pass top-level task arguments, but did not connect Dataset UDF arguments to that path. ### Versions / Dependencies still can reproduce in `master` branch currently. ### Reproduction script ```python import ray def map_fn(row, arg): assert arg == 1 return row ray.data.range(1).map( map_fn, fn_args=(ray.put(1),), ).take_all() ``` Before the fix, the assertion fails because `arg` is an `ObjectRef`. ### Issue Severity Medium: It is a significant difficulty but I can work around it.", "url": "https://github.com/ray-project/ray/issues/65449", "createdAt": "2026-08-13T06:31:41Z", "updatedAt": "2026-08-13T08:06:48Z", "timestamp": "2026-08-13T08:06:48Z", "metrics": { "reactions": 0, "comments": 3 }, "labels": [ "bug", "data", "stability", "community-backlog" ], "author": "viiccwen", "state": "open", "assignees": [ "viiccwen" ], "change": "updated" } }, { "id": "event:ea8b4d722f8ee47fa239", "signalId": "github:ray-project/ray:pull_request:61053", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:61053", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[serve] [taskiq - 3/n] Implement full TaskiqTaskProcessorAdapter with E2E tests", "text": "## Summary - Implements all stubbed methods in `TaskiqTaskProcessorAdapter`, completing the adapter that bridges Ray Serve's sync `TaskProcessorAdapter` ABC to taskiq's async broker API - Adds 6 E2E tests covering both standalone adapter usage and full Ray Serve deployment integration ## Why is this change needed? PR #61008 (`[taskiq - 2/n]`) added the config and initialization but left all adapter methods as `raise NotImplementedError` stubs. This PR fills in the complete implementation so the taskiq adapter is actually usable end-to-end β€” enqueue tasks, consume them, query status, and gracefully shut down. ## Architecture ``` β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ RAY ACTOR THREAD β”‚ β”‚ β”‚ β”‚ initialize() β”‚ β”‚ └── creates broker, spawns loop ──────►┐ β”‚ β”‚ β”‚ β”‚ register_task_handle() β”‚ β”‚ β”‚ └── broker.register_task() β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ enqueue_task_sync() Β·Β·Β·BLOCKSΒ·Β·Β·β–Ί β”‚ β”‚ β”‚ get_task_status_sync() Β·BLOCKSΒ·Β·β–Ί β”‚ β”‚ β”‚ start_consumer() Β·Β·Β·Β·SCHEDULESΒ·Β·β–Ί β”‚ β”‚ β”‚ stop_consumer() β”‚ β”‚ β”‚ β”œβ”€β”€ finish_event.set() Β·Β·Β·Β·Β·Β·Β·Β·β–Ί β”‚ β”‚ β”‚ β”œβ”€β”€ wait for drain Β·Β·BLOCKSΒ·Β·β–Ί β”‚ β”‚ β”‚ └── broker.shutdown() Β·BLOCKSΒ·β–Ί β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ Β·Β·Β·β–Ί = run_coroutine_threadsafe β”‚ ───► = thread creation β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ BACKGROUND THREAD (daemon) β”‚β—„β”€β”€β”€β”˜ β”‚ β”‚ β”‚ asyncio event loop: β”‚ β”‚ β”‚ β”‚ β€’ broker.startup() β”‚ β”‚ β€’ broker.kick(message) β”‚ β”‚ β€’ result_backend.is_ready() β”‚ β”‚ β€’ receiver.listen(finish_event) β”‚ β”‚ β”œβ”€β”€ pull message from broker β”‚ β”‚ β”œβ”€β”€ execute handler β”‚ β”‚ β”œβ”€β”€ save result + ack β”‚ β”‚ └── loops until event is set β”‚ β”‚ β€’ broker.shutdown() β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` ## Key Design Decisions **Persistent background event loop** (not `asyncio.run()` per call): Keeps broker connections alive across method calls. Taskiq brokers create connection pools in `__init__` and manage them across the event loop lifetime. `_ensure_loop()` is protected by `threading.Lock` to prevent race conditions. **`broker.kick()` for enqueue** (not `task.kiq()`): The low-level send doesn't require `broker.startup()` (connection pool is created in `__init__`), avoids holding references to taskiq-decorated task objects on the producer side, and consumer-side middleware (retry, ack, result storage) still runs normally. **`Receiver.listen()` for consuming** (not `run_receiver_task()`): Accepts an external `asyncio.Event` for graceful shutdown β€” the receiver stops accepting new messages and drains in-flight tasks. `run_receiver_task()` creates its own internal event with no external access, making graceful shutdown impossible. We also pass `run_startup=False` since `broker.startup()` is not idempotent (re-runs middleware init, creates duplicate connections). **No in-flight cancellation on shutdown**: Taskiq defaults to `WHEN_SAVED` ack mode β€” messages are acked only after task executes and result is saved. **Bound method wrapping**: Taskiq's `broker.register_task()` sets `__name__` on the callable, but bound methods have a read-only `__name__` property. We wrap bound methods in plain functions when registered via `@task_consumer`. **Task cancellation**: Not supported by taskiq ([taskiq-python/taskiq#305](https://github.com/taskiq-python/taskiq/issues/305)). Raises `NotImplementedError`. ## What's in this PR - `register_task_handle()` β€” registers callables with bound-method wrapping - `enqueue_task_sync()` β€” low-level `broker.kick()` with `TaskiqMessage` construction - `get_task_status_sync()` β€” result backend polling via `AsyncTaskiqTask` - `start_consumer()` / `stop_consumer()` β€” `Receiver.listen()` lifecycle with graceful drain - `cancel_task_sync()` β€” `NotImplementedError` (taskiq limitation) - Background event loop helpers (`_ensure_loop`, `_run_async`, `_ensure_broker_started`) - 6 E2E tests (3 adapter-level + 3 full Serve deployment integration) ## Related PRs - Depends on: #60977 (`[taskiq - 1/n]` Enable async task handlers) β€” **merged** - Depends on: #61008 (`[taskiq - 2/n]` Taskiq adapter config and initialization) β€” **merged** - Follow-up: `[taskiq - 4/n]` Retry, DLQ middleware and reliability tests πŸ€– Generated with [Claude Code](https://claude.com/claude-code)", "url": "https://github.com/ray-project/ray/pull/61053", "createdAt": "2026-02-13T13:06:15Z", "updatedAt": "2026-08-13T07:50:23Z", "timestamp": "2026-08-13T07:50:23Z", "metrics": { "reactions": 0, "comments": 4 }, "labels": [ "serve", "go" ], "author": "harshit-anyscale", "state": "open", "assignees": [ "harshit-anyscale" ], "change": "updated" } }, { "id": "event:102bec58974cc059edbf", "signalId": "github:ray-project/ray:issue:65443", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:issue:65443", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "issue", "title": "[Train] Share PlacementGroupCleaner across concurrent Train v2 runs", "text": "### Description Ray Train v2 currently creates one detached `PlacementGroupCleaner` actor for each Train controller. The cleaner removes placement groups when its controller exits ungracefully. Since #64705, every cleaner is pinned to the head node so it does not prevent worker-node scale-down. However, this concentrates one Python cleaner process per concurrent Train run on the head node: ```text N concurrent Train runs -> N detached PlacementGroupCleaner processes ``` In our production workload, each `PlacementGroupCleaner` process consistently uses approximately 600 MB of memory. With 24 concurrent Train jobs, the cleaner processes alone consume approximately 14.4 GB on the head node. The cleaner's responsibility is cluster-level bookkeeping and does not require a dedicated process for every Train run. We propose replacing the per-controller cleaners with one cluster-wide shared cleaner: ```text 24 concurrent Train runs -> 1 shared PlacementGroupCleaner process ``` This makes the cleaner process and memory cost constant instead of growing linearly with the number of concurrent Train jobs. ### Proposed implementation Create one cluster-wide `PlacementGroupCleaner` with: - a fixed, versioned actor name; - a fixed internal namespace; - `get_if_exists=True`; - `lifetime=\"detached\"`; - head-node resource pinning; - `scheduling_strategy=\"DEFAULT\"`; - automatic actor restart. Every Train controller registers itself and its placement groups with the shared actor. The cleaner maintains controller-scoped records: ```python controller_actor_id -> placement_group_ids ``` The lifecycle is: - controller starts: register the controller with the shared cleaner; - worker group starts: register its placement group under that controller; - controller shuts down normally: unregister only that controller without stopping the shared cleaner; - controller dies unexpectedly: remove only the placement groups registered to that controller; - the registry becomes empty: keep the cleaner alive to avoid a race between actor shutdown and a new Train job registering. The actor name is versioned, for example: ```text train_v2_placement_group_cleaner_v1 ``` This prevents a newer Ray version from accidentally reusing an incompatible shared actor created by an older implementation. ### Reliability Sharing the cleaner must not weaken the cleanup guarantees of the current per-controller implementation. The implementation therefore handles the following cases: 1. **Concurrent creation across jobs** Controllers use the same actor name and namespace with `get_if_exists=True`. Concurrent Train jobs resolve to the same actor. 2. **Controller isolation** Placement groups are stored by controller ID. A failed controller causes only its own placement groups to be removed. 3. **Registration and controller-death races** A controller is atomically marked as being cleaned before its placement groups are processed. A delayed placement-group registration is rejected, and the controller callback removes that placement group directly. 4. **Cleaner restart recovery** Controller IDs, placement-group IDs, and cleanup state are checkpointed in GCS internal KV. After an unexpected cleaner restart, the actor restores the registry and resumes incomplete cleanup. This avoids losing cleanup responsibility for all active Train runs when the shared process restarts. 5. **Partial cleanup failures** Cleanup state is removed only after every associated placement group has been removed successfully. Failed removals remain persisted and are retried. 6. **Failure isolation** A State API failure or placement-group cleanup failure for one controller does not stop the cleaner from processing other controllers. ### Prototype We have implemented and tested a Python-only prototype based on the current Ray `master`. It does not require Core Worker or C++ changes. The prototype modifies: - `placement_group_callback.py` - `placement_group_cleaner.py` - `test_placement_group_cleaner.py` It preserves the existing scheduling behavior introduced by #64705: - detached lifetime; - head-node pinning; - `DEFAULT` scheduling to escape placement-group capture. ### Test coverage The tests cover: - concurrent singleton creation; - singleton reuse across independent Ray drivers/jobs; - multiple controllers and multiple placement groups; - cleanup isolation between controllers; - normal controller deregistration; - duplicate registration, deregistration, and monitoring startup; - placement-group registration versus controller-death races; - temporary State API failures; - individual placement-group cleanup failures; - cleaner restart with active registrations; - cleaner restart during incomplete cleanup; - fixed actor name and namespace; - detached lifetime; - head-node pinning; - `DEFAULT` scheduling; - existing controller callback behavior. Current local results against Ray `master`: ```text 13 placement-group cleaner tests passed 10 controller callback regression tests passed Ruff check passed Ruff format check passed git diff --check passed ``` ### Expected impact For concurrent Train workloads, this changes the number of cleaner processes from O(number of Train runs) to O(1). For our workload: ```text Before: 24 cleaner processes x approximately 600 MB = approximately 14.4 GB After: 1 shared cleaner process x approximately 600 MB = approximately 600 MB ``` This reduces the observed head-node cleaner memory footprint by approximately 13.8 GB while retaining controller-scoped placement-group cleanup and restart recovery. ### Related work - #58515 introduced `PlacementGroupCleaner`. - #61756 fixed cleaner lifecycle behavior. - #62754 fixed the placement-group registration/controller-death race. - #64705 pinned cleaners to the head node and noted that N concurrent Train runs result in N cleaner pollers on the head node. This proposal addresses the remaining per-run process and memory scaling cost.", "url": "https://github.com/ray-project/ray/issues/65443", "createdAt": "2026-08-13T01:52:13Z", "updatedAt": "2026-08-13T07:32:44Z", "timestamp": "2026-08-13T07:32:44Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "train", "performance", "community-backlog" ], "author": "OneSizeFitsQuorum", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:3620bf4e96b99967ccf4", "signalId": "github:ray-project/ray:issue:65444", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:issue:65444", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "issue", "title": "[Core] Randomize worker port allocation to reduce deterministic collisions between raylets", "text": "### What happened + What you expected to happen When multiple raylets run in the same host network namespace with the same worker port range, each `WorkerPool` builds its free-port pool in ascending order and allocates from the lower bound. As a result, the raylets deterministically attempt to use the same initial ports, even when most of the configured range is available. Repeated bind failures can delay or prevent workers from starting in host-network, multi-raylet deployments. We expect each raylet to shuffle its validated worker port list once during startup before placing the ports into the free-port pool. The behavior should preserve the following properties: - Ports remain unique within each raylet's pool. - All ports remain within the configured range. - An explicit `worker_port_list` is validated before it is shuffled. - A bind failure skips or retries the occupied port as it does today. - No port outside the configured range is selected. Randomization is intended to reduce deterministic collisions. It is not a cross-raylet port reservation protocol and cannot guarantee that two raylets never select the same port. ### Versions / Dependencies - Ray: community `master` - OS: Linux - Deployment: multiple raylets sharing one host network namespace ### Reproduction script 1. Start two raylets or Ray clusters in the same host network namespace. 2. Configure both with the same worker port range, for example `20000-20100`. 3. Assign distinct control-plane, dashboard, object-manager, node-manager, and Ray Client ports so those services do not collide. 4. Concurrently launch enough workers from both raylets to exercise the shared worker port range. 5. Observe the worker port selection order and bind failures. Current behavior: both raylets walk upward from port `20000`, producing deterministic collisions at the beginning of the range. Expected behavior: each raylet starts with an independently shuffled port order. Occupied ports are skipped or retried, and a raylet never returns duplicate or out-of-range ports from its own pool. A regression test could inject a fixed random seed and verify that: - the resulting pool is a permutation of the validated input ports; - allocation does not always begin at the lower bound; - duplicate and out-of-range ports are rejected; - occupied ports still follow the existing bind-failure retry path. ### Issue Severity Medium: It is a significant difficulty but I can work around it.", "url": "https://github.com/ray-project/ray/issues/65444", "createdAt": "2026-08-13T02:52:18Z", "updatedAt": "2026-08-13T07:32:08Z", "timestamp": "2026-08-13T07:32:08Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "bug", "triage", "core", "stability", "community-backlog" ], "author": "zzchun", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:18a0498a2e8140f5d7c2", "signalId": "github:ray-project/ray:pull_request:65445", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65445", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[docs][core] Document JVM signal conflicts with PyArrow HDFS", "text": "## Why are these changes needed? PyArrow HDFS loads `libhdfs`, which embeds a HotSpot JVM in the Python process. HotSpot uses some `SIGSEGV` events as recoverable VM control flow, while Ray's CoreWorker installs an Abseil failure-signal handler that can interpret the same signal as a fatal native crash. The conflict can be established during handler installation and surface later as an intermittent JVM crash. This PR documents: - the symptoms and root cause of the Ray/PyArrow HDFS signal-handler conflict; - HotSpot signal chaining with `libjsig.so` as the preferred mitigation because it keeps Ray's native crash diagnostics enabled; - `RAY_DISABLE_FAILURE_SIGNAL_HANDLER=1` as a fallback and its diagnostic trade-off; - the process-start timing requirement for `LD_PRELOAD` and a KubeRay configuration example; - discoverable entry points from the Ray Data and Ray Train HDFS documentation. Related to #36415. ## Duplicate-work check I reviewed #36415 and searched open PRs for the issue number, `libjsig`, and PyArrow/HDFS signal-handler documentation. I found no open PR covering this documentation work. #58984 is already merged and added `RAY_DISABLE_FAILURE_SIGNAL_HANDLER`, but it didn't document the option, its diagnostic trade-off, or HotSpot signal chaining. ## Validation Documentation checks: - `git diff --check` - `python -m pre_commit run --files doc/source/ray-observability/user-guides/debug-apps/debug-failures.rst doc/source/train/user-guides/persistent-storage.rst doc/source/data/loading-data.rst` - `sphinx-lint --disable trailing-whitespace doc/source/ray-observability/user-guides/debug-apps/debug-failures.rst doc/source/train/user-guides/persistent-storage.rst doc/source/data/loading-data.rst` - Verified the linked Oracle signal-handling pages and #36415 return HTTP 200. - Validated the documented `find`/`LD_PRELOAD` shell snippet with the local OpenJDK installation. Runtime validation behind the guidance: - Reproduced the unmitigated JVM termination and generated multiple `hs_err_pid*.log` files. - Completed 5,000- and 10,000-operation HDFS stress runs with `RAY_DISABLE_FAILURE_SIGNAL_HANDLER=1` without reproducing the crash. - Completed a 5,000-operation HDFS stress run with only `libjsig.so` preloaded and Ray's handler left enabled. - Completed a separate 1,000-operation `libjsig.so` run, followed by a clean `ray.shutdown()` and process exit code 0. The full Sphinx documentation site wasn't built locally because the existing Python environment doesn't include the Ray documentation dependency set. ## AI assistance AI assistance was used to investigate the failure, draft the documentation, and run the validation commands. I reviewed every changed line and the reported test results.", "url": "https://github.com/ray-project/ray/pull/65445", "createdAt": "2026-08-13T03:06:39Z", "updatedAt": "2026-08-13T07:31:01Z", "timestamp": "2026-08-13T07:31:01Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "docs", "core", "community-contribution" ], "author": "OneSizeFitsQuorum", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:605dca60fd2863eaff42", "signalId": "github:ray-project/ray:pull_request:65446", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65446", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[Data] Cluster autoscaler requests a bundle larger than the node it was copied from, because node memory is rounded up", "text": "# Issue #65237 β€” autoscaler requests a bundle larger than the node it was copied from ## 1. Root cause `DefaultClusterAutoscalerV2` builds its autoscaler request out of node *shapes*: `_get_node_resource_spec_and_count()` turns every live worker node's resource dict into a `_NodeResourceSpec`, and `try_trigger_scaling()` sends `spec.to_bundle()` back to the autoscaler as active bundles (one per existing node) and pending bundles (the scale-up delta). `_NodeResourceSpec.of()` quantized memory with a *nearest*-multiple rounding (`python/ray/data/_internal/cluster_autoscaler/default_cluster_autoscaler_v2.py:64` before the fix): ```python mem = int(round(mem / GiB, 1) * GiB) if mem > 0 else 0 ``` The quantization exists so that two nodes of the same instance type that report slightly different physical memory (Ray derives the `memory` resource from memory available at init time, so it jitters) collapse into one shape instead of two. Rounding to the *nearest* 0.1 GiB rounds **up** for anything in the upper half of a bucket. A node reporting 14.87 GiB becomes a 14.9 GiB bundle: strictly larger than the node it was copied from. That bundle does not fit on any node of that type, so the autoscaler treats the demand as infeasible rather than launching the node, and the cluster never scales up (CPU and GPU were already floored, so memory was the only resource with this defect). ## 2. The fix and why Quantize memory **down** instead of to the nearest multiple: ```python mem = math.floor(mem / _MEMORY_QUANTIZATION_BYTES) * _MEMORY_QUANTIZATION_BYTES ``` Rounding down makes `spec.mem <= node_memory` an invariant, which is the property the request actually needs: a bundle must be schedulable on the node shape it describes. The error is now asymmetric in the safe direction β€” under-requesting memory still lands on a node of that type, over-requesting can never be satisfied. The granularity is a named constant `_MEMORY_QUANTIZATION_BYTES = GiB`. It was widened from 0.1 GiB to 1 GiB because flooring at 0.1 GiB would have *lost* the grouping property the quantization was added for: 14.87 GiB and 14.93 GiB floor to 14.8 and 14.9, i.e. two shapes for one instance type (this is exactly what `test_nodes_with_similar_memory_grouped` covers). A 1 GiB bucket keeps both nodes on the same shape and gives 10x more slack against reported-memory jitter than the original rounding did, while never exceeding the source node. Matching CPU/GPU are still exact, so the coarser memory bucket does not realistically make the request match a different (smaller) instance type. ## 3. Files changed - `python/ray/data/_internal/cluster_autoscaler/default_cluster_autoscaler_v2.py` β€” added `_MEMORY_QUANTIZATION_BYTES`; `_NodeResourceSpec.of()` now floors memory to it. - `python/ray/data/tests/test_default_cluster_autoscaler_v2.py` β€” added `test_spec_never_exceeds_node_memory` (parametrized invariant check) and `test_bundles_fit_on_the_nodes_they_were_derived_from` (end-to-end through `_get_node_resource_spec_and_count`, asserting every resource in the emitted bundle fits the source node). ## 4. Risk / uncertainty - Memory demand per node is now under-reported by up to 1 GiB (previously by up to 0.05 GiB, or over-reported by up to 0.05 GiB). Under-reporting is safe for feasibility. It does slightly loosen `cap_resource_request_to_limits`: with a user-configured `resource_limits.memory`, pending bundles are measured with the floored value, so a scale-up can overshoot the configured memory limit by up to 1 GiB per pending bundle (bounded by `cluster_scaling_up_delta`, default 1, times the number of node shapes). The old 0.1 GiB rounding had the same failure mode, an order of magnitude smaller. - Nodes with less than 1 GiB of Ray `memory` now produce `memory: 0` bundles (the old code did this below 0.05 GiB). Such worker nodes are not realistic in an autoscaling Ray Data cluster, and CPU/GPU still carry the demand. - The choice of 1 GiB is a judgement call: it is the smallest round bucket that keeps the existing same-node-type grouping test passing under flooring. A maintainer may prefer a different constant; it is isolated in one named variable. - Head-node-group detection (`node_group_config` shape vs. the running head node's shape) also compares quantized specs. Coarser buckets make that comparison neither better nor worse in a way I could establish, since a configured shape and the reported shape can straddle a bucket boundary at any granularity. ## 5. How I verified it Ray is not installed in this environment (`import ray` fails), so the test suite could not be executed here. Verification done: - Extracted the real `_NodeResourceSpec` class from the patched source with `ast` and exec'd it against stubs, then checked, for `mem` in `{0, 1000 B, 0.5, 14.87, 14.93, 32, 8, 1.999, 2 GiB}`: the quantized value is always `<= mem`, is an `int` (the frozen dataclass asserts this in `__post_init__`), and `of(mem=14.87 GiB) == of(mem=14.93 GiB)` still holds. All passed; the pre-fix code returns 14.9 GiB for a 14.87 GiB node, reproducing the issue. - Hand-checked every existing memory value in `test_default_cluster_autoscaler_v2.py` against the new quantization: the small byte-valued fixtures (1000/2000/3000) already quantized to 0 under the old rounding and still do; the GiB-valued fixtures (2, 4, 8, 32 GiB) are exact multiples and are unchanged, including the `memory: 8.0GiB` log-message assertion. - `grep` confirmed `_NodeResourceSpec` / `_get_node_resource_spec_and_count` have no consumers outside this module and its test file. - `python -m py_compile` on both changed files; no line exceeds the configured ruff `line-length = 88`. Suggested command for a human to run before submitting: ```bash pytest -q python/ray/data/tests/test_default_cluster_autoscaler_v2.py ```", "url": "https://github.com/ray-project/ray/pull/65446", "createdAt": "2026-08-13T03:17:12Z", "updatedAt": "2026-08-13T07:30:28Z", "timestamp": "2026-08-13T07:30:28Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "data", "community-contribution" ], "author": "pjdurden", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:2583c0ed07c68efac352", "signalId": "github:ray-project/ray:pull_request:65437", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65437", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[Serve][2/2] Propagate TracingConfig to proxies via long poll", "text": "Follow-up to #63273 ([1/2]) completing the TracingConfig feature. - Added a `GLOBAL_TRACING_CONFIG` long-poll namespace. The controller broadcasts the global tracing config which is subscribed by the proxies, which runs `setup_tracing` on delivery. This wires proxy tracing without the init-time controller callback that cannot work (proxies can start before the controller). This broadcast mechanism also lets users change the tracing config at runtime through the existing serve-config API. - Controller: - checkpoint-ing the tracing config (So that it can be restored during the recovery). - Default it to an env-var-sourced TracingConfig() so it is never `None`, else it will cause attribute error. - Publishing the initial snapshot, and broadcasting changes from apply_config. get_tracing_config(). - reconfigure validates its input since the value is broadcasted cluster-wide. - `tracing_utils.setup_tracing`: - dropped the env-var fallback and the empty-path check, - `tracing_config` is now required (single source of truth). Its fields already default from the RAY_SERVE_TRACING_* env vars. - Proxy sets tracing up only from the long-poll callback, so `set_tracer_provider` is called once per process. - Replicas continue to read the config when they start. - Tests: assert proxy traces are produced via the imperative flow, and that a tracing config changed after startup survives controller recovery (so the checkpoint, not the constructor arg, is exercised). This PR used Claude Code Assistant for writing code.", "url": "https://github.com/ray-project/ray/pull/65437", "createdAt": "2026-08-12T20:09:27Z", "updatedAt": "2026-08-13T06:41:01Z", "timestamp": "2026-08-13T06:41:01Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "serve", "go", "community-contribution" ], "author": "suppagoddo", "state": "open", "assignees": [ "jeffreywang88" ], "change": "updated" } }, { "id": "event:9b459137d172996d10e7", "signalId": "github:ray-project/ray:pull_request:65351", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65351", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[llm][ci] Upgrade to vllm 0.27.0", "text": "## Description > Briefly describe what this PR accomplishes and why it's needed. ## Related issues > Link related issues: \"Fixes #1234\", \"Closes #1234\", or \"Related to #1234\". ## Additional information > Optional: Add implementation details, API changes, usage examples, screenshots, etc.", "url": "https://github.com/ray-project/ray/pull/65351", "createdAt": "2026-08-10T23:53:04Z", "updatedAt": "2026-08-13T06:39:35Z", "timestamp": "2026-08-13T06:39:35Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "go" ], "author": "jeffreywang88", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:51c7712b287b5b7bf686", "signalId": "github:ray-project/ray:pull_request:65363", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65363", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[doc] Update the number of index.rst document", "text": "## Description documents is five, the code is 2 the number is incorrect ## Related issues > Link related issues: \"Fixes #65364\" ## Additional information update the number", "url": "https://github.com/ray-project/ray/pull/65363", "createdAt": "2026-08-11T08:10:00Z", "updatedAt": "2026-08-13T06:33:03Z", "timestamp": "2026-08-13T06:33:03Z", "metrics": { "reactions": 1, "comments": 1 }, "labels": [ "core", "community-contribution" ], "author": "xubo245", "state": "open", "assignees": [ "Yicheng-Lu-llll" ], "change": "updated" } }, { "id": "event:5f62d1d5adcd5b9c4380", "signalId": "github:ray-project/ray:pull_request:65298", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65298", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[core][2/N] Reconnect to Redis in place when the connection drops", "text": "## Description When the async connection to Redis drops, hiredis frees the raw `redisAsyncContext` and nothing puts it back. Every subsequent GCS command fails with `Status::Disconnected` until `num_redis_request_retries` runs out (~3.5s), and gcs_server aborts with `Failed to run redis cmds`. A Redis restart, an idle timeout, a proxy dropping its backend, or a Sentinel failover is enough to take the head node down, which defeats the purpose of GCS fault tolerance. [#64299](https://github.com/ray-project/ray/pull/64299) (1/N) made the connection-failure paths return `Status` instead of crashing. This PR adds the reconnect itself, as a stack of three commits that each build and fix something on their own. The commit messages carry the design details; in short: **1/3: Rebind the async context in place.** In-flight `RedisRequestContext`s hold a raw `RedisAsyncContext*`, so the reconnect preserves object identity: `Reset()` swaps the raw hiredis handle while the wrapper stays put. Success is only declared from the hiredis connect callback, since `redisAsyncConnect` reports a healthy context even when nothing is listening. **2/3: Re-resolve the address on every reconnect attempt.** The recorded address survives a failover behind a proxy or Service, but for Sentinel it is the old primary's. Sentinel setups now re-query the Sentinel on each attempt and follow the promotion; everything else re-resolves through DNS. Redis Cluster addressed directly has no lookup point and stays out of scope (documented: front it with a Service or proxy). **3/3: Don't spend a command's retries on attempts that never reached Redis.** A Sentinel promotion takes ~4s, the command retry budget ~3.5s, so gcs_server aborted moments before the reconnect landed. Null replies (never reached Redis) are refunded within the new `redis_reconnect_grace_period_ms` (default 60s; `0` restores the old behaviour exactly). Error replies (`-READONLY`, …) drain the budget as before. `test_redis_with_sentinel_failureover` asserted the old behaviour and carried the TODO for this work in its docstring. It now asserts the same process survives the failover and that the old cold-restart coverage still holds. How the three commits interact, on the case that needs all of them (Sentinel direct, measured timings): ```mermaid sequenceDiagram participant REQ as RedisRequestContext participant RC as RedisContext participant S as Sentinel participant P1 as primary :7001 participant P2 as replica :7002 Note over REQ,P1: connected. command budget: 6 tries in ~3.5s P1--xRC: connection drops (t=0) Note over REQ: 3/3: null replies are refunded within the 60s grace,
so the command is not spending its 6 tries yet loop backoff 100ms..1s, up to redis_db_connect_retries=120 RC->>S: 2/3: SENTINEL MASTERS S-->>RC: primary is :7001 RC->>P1: async connect P1--xRC: refused end RC->>S: SENTINEL MASTERS S-->>RC: primary is :7002 (promoted at t=4s) RC->>P2: async connect P2-->>RC: connect callback OK Note over RC: 1/3: Reset() rebinds the same RedisAsyncContext,
so in-flight requests keep their raw pointer valid REQ->>P2: retry succeeds ``` Before this PR the timeline ends at t=3.5s with `Failed to run redis cmds: [PING] for 5 times`. ### Behaviour matrix | Topology | Before | After | |---|---|---| | Plain Redis, drop/restart | FATAL in ~3.5s / SIGSEGV | survives, reconnects in ~0ms | | Proxy / K8s Service, backend failover | dies before proxy recovers | survives (~1ms, same VIP) | | Sentinel direct, primary failover | dies ~0.5s before promotion | survives, follows the promotion | | Redis Cluster direct, failover | dies | still unsupported; documented, use a Service/proxy in front | | Redis permanently unreachable | FATAL ~3.5s | still exits (bounded by grace + retry drain), deliberately | ### Verification Everything below ran against real Redis processes and containers (killed, restarted, demoted; no mocks). Each scenario runs a task (`ray.get`) before and after the failure, so \"survives\" means the cluster still schedules work, not just that the process exists.
Plain Redis (single node) β€” 9 scenarios | Scenario | Before | After | |---|---|---| | Connection drop | FATAL in ~3.5s | survives, 0ms reconnect, tasks run | | Process restart (no persistence) | SIGSEGV | survives | | Restart while loading a 400k-key AOF | dies | survives, no `-LOADING` reply leaks | | Never comes back | FATAL ~3.5s | bounded FATAL after grace + drain, no hang | | Rapid flapping (drop every 200ms for 10s) | dies on first drop | survives, 88 reconnects, 0 crash markers | | Concurrent load across a drop | dies | 160,140 tasks, 0 errors | | Soak: 60 drops | β€” | RSS +0.7MB, fd +2, 120 reconnects | | Worker node attached across a drop | β€” | both nodes stay alive, tasks on both | | Detached actor across a drop | β€” | callable after reconnect |
HAProxy / Kubernetes Service + Sentinel-managed failover β€” 2 scenarios | Scenario | Before | After | |---|---|---| | Kill primary container β†’ Sentinel promotes β†’ proxy repoints | dies | survives, reconnects through the same VIP, tasks run | | Second backend churn 10s later | β€” | same recovery path |
Sentinel addressed directly β€” 8 scenarios | Scenario | Before | After | |---|---|---| | Primary killed β†’ replica promoted (~4s) | dies at ~3.5s | survives, logs `Sentinel now reports the primary at :7002, was :7001`, reconnects to the promoted node | | Two failovers in a row | β€” | follows both promotions, both directions | | Old primary rejoins as replica | β€” | stays on the new primary | | Sentinel dies while the primary is down | β€” | keeps retrying, no crash, no false \"Reconnected\" | | Sentinel reports a dead address, then a live one | β€” | adopts the good address once offered | | Password auth across the failover | β€” | 0 auth errors | | Promoted node rejects AUTH | β€” | keeps retrying, no false success | | GCS cold restart after the failover | covered by the old test | still covered; detached actor recovered from the promoted primary |
Redis Cluster / TLS β€” 4 scenarios | Scenario | Result | |---|---| | Single-shard cluster, drop | survives, reconnects, tasks run | | `CLUSTER FAILOVER`, GCS pointed at a node directly | does not recover; measured, documented with the workaround | | `rediss://` drop + reconnect | survives, reconnect re-initiates TLS | | TLS + Sentinel re-query | re-query initiates TLS like the startup path |
Config knobs β€” 6 scenarios | Knob | Verified | |---|---| | `redis_reconnect_grace_period_ms=0` | restores pre-change behaviour exactly (~3.5s FATAL) | | `redis_reconnect_grace_period_ms=5000` | bounded exit at grace + drain | | `redis_db_connect_retries=0` / `=1` | exits after exactly 0 / 1 attempts | | `redis_retry_base_ms=1, max_ms=10` | no hot spin, CPU flat | | `num_redis_request_retries=0` | idle drop survives, 0 wasted connects | | `retries=3` + `grace=60s` | the smaller bound wins |
Unit tests: 4 new cases in `redis_async_context_test` (in-place `Reset`, never-connected release, teardown during reconnect). All store-client tests pass at each commit of the stack independently. Not verified locally: Linux CI (this PR is its first run), ASAN/TSAN (macOS toolchain), IPv6. ### Known limitations (deliberately out of scope) - The reconnect handshake (Sentinel re-query, DNS resolve) runs synchronously on the io_service thread. Each step is bounded by the probe timeout, but a fully async handshake is follow-up work. - Error replies are not refunded: a Service that keeps routing writes to a demoted (read-only) primary for longer than the command budget still brings GCS down, exactly as before this PR. - Against a SYN-blackholed address, each async connect attempt settles only after the kernel TCP timeout, which stretches the `redis_db_connect_retries` budget in wall-clock terms. ## Related issues Closes #48781. Related: #53475, #64204 (SIGSEGV on failover, fixed earlier), #64299 (1/N prerequisite). ## Checks - [x] I've signed off every commit(by using the -s flag, i.e., `git commit -s`) in this PR. - [x] I've run `scripts/format.sh` to lint the changes in this PR. - [x] I've included any doc changes needed for https://docs.ray.io/en/master/. (`kuberay-gcs-ft.md`: new section on what happens when the Redis connection drops, plus the tuning knobs.) - [ ] I've added any new APIs to the API Reference. N/A: no public API change; one new internal config (`RAY_redis_reconnect_grace_period_ms`). - [x] I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/ - Testing Strategy - [x] Unit tests - [ ] Release tests - [ ] This PR is not tested :( ## AI assistance disclosure AI assistance (Claude) was used for the implementation, the verification harnesses, and this description. I (the human submitter) have reviewed every changed line and understand the change end-to-end. - **Not duplicating existing work:** the previous attempt (#48781) was auto-closed as stale in July 2025, and a search of open PRs on 2026-08-08 found none implementing Redis reconnection (#64847 and #62462 touch gRPC reconnect settings, not the Redis connection). The adjacent fixes #64204 and #64299 landed separately and this PR stacks on them. - **Test commands run locally** (macOS source build; results in the Verification section above): - `bazel test //src/ray/gcs/store_client/tests:redis_async_context_test //src/ray/gcs/store_client/tests:redis_store_client_test` β€” pass at each commit of the stack - `pytest python/ray/tests/test_gcs_fault_tolerance.py::test_redis_with_sentinel_failureover` β€” passed (16.4s), with `TEST_EXTERNAL_REDIS=1` - a 29-scenario topology harness against real Redis/Sentinel/HAProxy processes and containers, run before and after the change", "url": "https://github.com/ray-project/ray/pull/65298", "createdAt": "2026-08-08T14:08:57Z", "updatedAt": "2026-08-13T06:18:16Z", "timestamp": "2026-08-13T06:18:16Z", "metrics": { "reactions": 1, "comments": 3 }, "labels": [ "docs", "core", "community-contribution" ], "author": "nadongjun", "state": "open", "assignees": [ "rueian" ], "change": "updated" } }, { "id": "event:27413e5d43e71aeffd95", "signalId": "github:ray-project/ray:pull_request:64540", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:64540", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[data] Add orc datasource for V2", "text": "## Description Support read orc format ## Related issues > Link related issues: \"Fixes #1234\", \"Closes #1234\", or \"Related to #1234\". ## Additional information > Optional: Add implementation details, API changes, usage examples, screenshots, etc.", "url": "https://github.com/ray-project/ray/pull/64540", "createdAt": "2026-07-06T03:31:43Z", "updatedAt": "2026-08-13T04:07:18Z", "timestamp": "2026-08-13T04:07:18Z", "metrics": { "reactions": 1, "comments": 7 }, "labels": [ "data", "go", "community-contribution" ], "author": "laysfire", "state": "open", "assignees": [ "ayushk7102" ], "change": "updated" } }, { "id": "event:64159ee60324f1ced7bb", "signalId": "github:ray-project/ray:issue:36415", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:issue:36415", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "issue", "title": "[core] Ray session conflicts with PyArrow+HDFS", "text": "### What happened + What you expected to happen Using PyArrow fs with HDFS works fine outside a ray session: ``` file_sys, file_path = pyarrow.fs.FileSystem.from_uri(hdfs_folder) file_infos = file_sys.get_file_info(pyarrow.fs.FileSelector(file_path, recursive=False)) ``` However, after `ray.init()`, the same code results in a segmentation fault: ``` 2023-06-14 01:27:37,622 INFO worker.py:1614 -- Started a local Ray instance. View the dashboard at http://127.0.0.1:8265 *** SIGSEGV received at time=1686731258 on cpu 0 *** PC: @ 0x7f99d20c5822 (unknown) (unknown) @ 0x7f996fa6ec85 208 absl::lts_20220623::WriteFailureInfo() @ 0x7f996fa6e9c8 64 absl::lts_20220623::AbslFailureSignalHandler() @ 0x7f99e81c6420 3408 (unknown) @ 0x7f99d1c2782e 48 (unknown) @ 0x7f99d1c2cc0f 240 (unknown) @ 0x7f99d2267a5f 144 (unknown) @ 0x7f99d2267d53 128 (unknown) @ 0x7f99d21092a0 64 (unknown) @ 0x7f99e81ba609 (unknown) start_thread [2023-06-14 01:27:38,591 E 9716 9731] logging.cc:361: *** SIGSEGV received at time=1686731258 on cpu 0 *** [2023-06-14 01:27:38,591 E 9716 9731] logging.cc:361: PC: @ 0x7f99d20c5822 (unknown) (unknown) [2023-06-14 01:27:38,591 E 9716 9731] logging.cc:361: @ 0x7f996fa6ec85 208 absl::lts_20220623::WriteFailureInfo() [2023-06-14 01:27:38,592 E 9716 9731] logging.cc:361: @ 0x7f996fa6e9e1 64 absl::lts_20220623::AbslFailureSignalHandler() [2023-06-14 01:27:38,593 E 9716 9731] logging.cc:361: @ 0x7f99e81c6420 3408 (unknown) [2023-06-14 01:27:38,593 E 9716 9731] logging.cc:361: @ 0x7f99d1c2782e 48 (unknown) [2023-06-14 01:27:38,593 E 9716 9731] logging.cc:361: @ 0x7f99d1c2cc0f 240 (unknown) [2023-06-14 01:27:38,593 E 9716 9731] logging.cc:361: @ 0x7f99d2267a5f 144 (unknown) [2023-06-14 01:27:38,593 E 9716 9731] logging.cc:361: @ 0x7f99d2267d53 128 (unknown) [2023-06-14 01:27:38,593 E 9716 9731] logging.cc:361: @ 0x7f99d21092a0 64 (unknown) [2023-06-14 01:27:38,593 E 9716 9731] logging.cc:361: @ 0x7f99e81ba609 (unknown) start_thread Fatal Python error: Segmentation fault # # A fatal error has been detected by the Java Runtime Environment: # # SIGSEGV (0xb) at pc=0x00007f99e81c62ab, pid=9716, tid=0x00007f99baa56700 # # JRE version: OpenJDK Runtime Environment (8.0_362-b09) (build 1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09) # Java VM: OpenJDK 64-Bit Server VM (25.362-b09 mixed mode linux-amd64 compressed oops) # Problematic frame: # C [libpthread.so.0+0x142ab] raise+0xcb # # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try \"ulimit -c unlimited\" before starting Java again # # An error report file with more information is saved as: # /ray/hs_err_pid9716.log # # If you would like to submit a bug report, please visit: # http://bugreport.java.com/bugreport/crash.jsp # [failure_signal_handler.cc : 332] RAW: Signal 6 raised at PC=0x7f99e800300b while already in AbslFailureSignalHandler() *** SIGABRT received at time=1686731258 on cpu 0 *** PC: @ 0x7f99e800300b (unknown) raise @ 0x7f996fa6ec85 208 absl::lts_20220623::WriteFailureInfo() @ 0x7f996fa6e9c8 64 absl::lts_20220623::AbslFailureSignalHandler() @ 0x7f99e81c6420 3952 (unknown) @ 0x7f99d22c3843 240 (unknown) @ 0x7f99d211410e 352 JVM_handle_linux_signal @ 0x7f99d210731c 64 (unknown) @ 0x7f99e81c6420 10576 (unknown) @ 0x7f99d1c2782e 48 (unknown) @ 0x7f99d1c2cc0f 240 (unknown) @ 0x7f99d2267a5f 144 (unknown) @ 0x7f99d2267d53 128 (unknown) @ 0x7f99d21092a0 64 (unknown) @ 0x7f99e81ba609 (unknown) start_thread [2023-06-14 01:27:38,618 E 9716 9731] logging.cc:361: *** SIGABRT received at time=1686731258 on cpu 0 *** [2023-06-14 01:27:38,618 E 9716 9731] logging.cc:361: PC: @ 0x7f99e800300b (unknown) raise [2023-06-14 01:27:38,618 E 9716 9731] logging.cc:361: @ 0x7f996fa6ec85 208 absl::lts_20220623::WriteFailureInfo() [2023-06-14 01:27:38,618 E 9716 9731] logging.cc:361: @ 0x7f996fa6e9e1 64 absl::lts_20220623::AbslFailureSignalHandler() [2023-06-14 01:27:38,618 E 9716 9731] logging.cc:361: @ 0x7f99e81c6420 3952 (unknown) [2023-06-14 01:27:38,618 E 9716 9731] logging.cc:361: @ 0x7f99d22c3843 240 (unknown) [2023-06-14 01:27:38,618 E 9716 9731] logging.cc:361: @ 0x7f99d211410e 352 JVM_handle_linux_signal [2023-06-14 01:27:38,618 E 9716 9731] logging.cc:361: @ 0x7f99d210731c 64 (unknown) [2023-06-14 01:27:38,619 E 9716 9731] logging.cc:361: @ 0x7f99e81c6420 10576 (unknown) [2023-06-14 01:27:38,619 E 9716 9731] logging.cc:361: @ 0x7f99d1c2782e 48 (unknown) [2023-06-14 01:27:38,619 E 9716 9731] logging.cc:361: @ 0x7f99d1c2cc0f 240 (unknown) [2023-06-14 01:27:38,619 E 9716 9731] logging.cc:361: @ 0x7f99d2267a5f 144 (unknown) [2023-06-14 01:27:38,619 E 9716 9731] logging.cc:361: @ 0x7f99d2267d53 128 (unknown) [2023-06-14 01:27:38,619 E 9716 9731] logging.cc:361: @ 0x7f99d21092a0 64 (unknown) [2023-06-14 01:27:38,619 E 9716 9731] logging.cc:361: @ 0x7f99e81ba609 (unknown) start_thread Fatal Python error: Aborted ``` Here is the log dump from java: [hs_err_pid9716.log](https://github.com/ray-project/ray/files/11743507/h5PTZ.log) The segfault occurs almost every time, but not always. It never occurs when ray is not initialized. Thus there is probably some interference between the ray session/global state and the java/pyarrow/hdfs connection. ### Versions / Dependencies Ray latest master, hadoop 3.2.4, java openjdk version \"1.8.0_362\" ### Reproduction script - Install HDFS with `./ci/env-install-hdfs.sh` - Create some directory in HDFS e.g. with `/opt/hadoop-3.2.4/bin/hdfs dfs -put /tmp/somewhere hdfs://[host]:8020/somewhere` - Run this script ``` def setup_hdfs(): \"\"\"Set env vars required by pyarrow to talk to hdfs correctly. Returns hostname and port needed for the hdfs uri.\"\"\" # the following file is written in `install-hdfs.sh`. with open(\"/tmp/hdfs_env\", \"r\") as f: for line in f.readlines(): line = line.rstrip(\"\\n\") tokens = line.split(\"=\", maxsplit=1) os.environ[tokens[0]] = tokens[1] import sys sys.path.insert(0, os.path.join(os.environ[\"HADOOP_HOME\"], \"bin\")) hostname = os.getenv(\"CONTAINER_ID\") port = os.getenv(\"HDFS_PORT\") return hostname, port import os import pyarrow import pyarrow.fs hostname, port = setup_hdfs() workspace_dir = f'hdfs://{hostname}:{port}/somewhere' # from ray.air._internal.remote_storage import upload_to_uri # upload_to_uri(\"/tmp/content\", workspace_dir) def get_list_of_files_under_hdfs_folder(hdfs_folder): file_sys, file_path = pyarrow.fs.FileSystem.from_uri(hdfs_folder) file_infos = file_sys.get_file_info(pyarrow.fs.FileSelector(file_path, recursive=False)) return file_infos print(f\"Success!, number of files in {workspace_dir}: {len(get_list_of_files_under_hdfs_folder(workspace_dir))}\") print(f\"Success!, number of files in {workspace_dir}: {len(get_list_of_files_under_hdfs_folder(workspace_dir))}\") print(\"initializing ray, and get number of files again.\") import ray ray.is_initialized() ray.init() print(\"After ray init\", len(get_list_of_files_under_hdfs_folder(workspace_dir))) ``` ### Issue Severity High: It blocks me from completing my task.", "url": "https://github.com/ray-project/ray/issues/36415", "createdAt": "2023-06-14T08:34:15Z", "updatedAt": "2026-08-13T02:47:35Z", "timestamp": "2026-08-13T02:47:35Z", "metrics": { "reactions": 4, "comments": 27 }, "labels": [ "bug", "P1", "core", "stability", "core-fundamentals" ], "author": "krfricke", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:9e44cd3a1dafbee4a57e", "signalId": "github:ray-project/ray:issue:65259", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:issue:65259", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "issue", "title": "[serve][llm] Governance middleware layer for Ray Serve LLM β€” PII detection, cost budgets, policy enforcement, and audit trails", "text": "### Description Ray Serve LLM currently exposes an OpenAI-compatible API (chat completions, completions, embeddings) with no built-in mechanism for governance, guardrails, or policy enforcement at the serving layer. Teams deploying LLMs in production β€” especially in regulated industries (healthcare, finance, government) β€” need to add PII detection, cost budgets, tool authorization, and audit logging around every inference request. Today, the only way to achieve this is to write a custom Ray Serve deployment that wraps the engine directly (bypassing the standard `LLMConfig` + `build_openai_app` pipeline), losing all the benefits of `LLMRouter`, `LLMServer`, autoscaling, and multi-model management. **Proposed: A middleware/hook system for Ray Serve LLM** that allows governance logic to execute before and after inference β€” similar to ASGI middleware or Ray Serve's HTTP middleware, but specifically integrated into the LLM request pipeline. ```python from ray import serve from ray.serve.llm import LLMConfig, build_openai_app, LLMMiddleware class GovernanceMiddleware(LLMMiddleware): \"\"\"Deterministic governance β€” no LLM in the governance path, <2ms overhead.\"\"\" async def before_inference(self, request: ChatCompletionRequest, context: RequestContext) -> ChatCompletionRequest | BlockedResponse: # 1. Scan input for PII (regex-based, 40+ patterns) pii_findings = self.scan_pii(request.messages) if pii_findings and self.config.mode == \"ENFORCE\": return BlockedResponse(reason=\"PII detected\", findings=pii_findings) # 2. Check cost budget (per-user, per-session, per-model) if self.exceeds_budget(context.user_id, context.model): return BlockedResponse(reason=\"Budget exceeded\") # 3. Evaluate access policies (which users can use which models) if not self.policy_allows(context.user_id, request.model): return BlockedResponse(reason=\"Model access denied by policy\") return request # Allow request to proceed async def after_inference(self, request: ChatCompletionRequest, response: ChatCompletionResponse, context: RequestContext) -> ChatCompletionResponse: # 1. Scan output for PII/secrets leakage # 2. Record token usage against cost budget # 3. Emit structured audit receipt (JSON with correlation ID) self.record_usage(context, response.usage) self.emit_audit_receipt(context, request, response) return response # Usage: plug into standard LLMConfig pipeline llm_config = LLMConfig( model_loading_config={\"model_id\": \"gpt-serving\", \"model_source\": \"meta-llama/Llama-3-8B-Instruct\"}, accelerator_type=\"A10G\", ) app = build_openai_app({ \"llm_configs\": [llm_config], \"middleware\": [GovernanceMiddleware(config=governance_policy)], # <-- new }) serve.run(app, blocking=True) ``` **Key design principles:** - Middleware runs in the same process (no sidecar, no network hop) - Deterministic evaluation only (regex, policy rules) β€” no additional LLM call - <2ms overhead per request - Does not interfere with vLLM/SGLang engine internals - Composable: multiple middleware can chain (auth β†’ governance β†’ logging) ### Use case **1. Regulated industries deploying LLMs on Ray** Healthcare companies (HIPAA), financial services (SOX, PCI-DSS), and government agencies need to prove that every LLM interaction was governed. Today they either: - Build custom Ray Serve deployments from scratch (losing LLMRouter, autoscaling, multi-model support) - Add a separate proxy/gateway in front of Ray Serve (adding latency, operational complexity, and a failure point) A middleware layer inside Ray Serve LLM would let them use the standard `LLMConfig` pipeline with governance built in. **2. Multi-tenant LLM platforms** Companies serving multiple customers/teams from shared GPU infrastructure need: - Per-tenant cost budgets with hard enforcement (stop serving when budget is exhausted) - Per-tenant model access policies (tenant A gets Llama-70B, tenant B gets Llama-8B only) - PII isolation (scan inputs/outputs to prevent data leakage across tenants) The `LLMRouter` already handles routing, but there's no governance layer for access control or cost caps. **3. MCP tool governance in agentic deployments** As Ray Serve LLM adds tool-calling and agentic capabilities, there's a need to control which tools agents can invoke, validate tool arguments before execution, and scan tool outputs before they re-enter the context window. A middleware layer is the natural extension point for this. **4. Compliance evidence for SOC2/ISO 27001 audits** Enterprises need structured audit records proving that: - Every request was scanned for sensitive data - Access policies were evaluated deterministically - Cost budgets were enforced - Blocked requests have documented reasons Current Ray Serve metrics (Prometheus/Grafana) cover performance but not governance decisions. **Existing community demand:** The [Ray Discuss thread on preprocessing](https://discuss.ray.io/t/preprocessing-in-ray-serve-llm/23346) highlights that users already want more control over the request pipeline. A governance middleware would address this need specifically for security/compliance use cases. **I've built [TealTiger](https://github.com/agentguard-ai/tealtiger)** (Apache 2.0), a deterministic AI governance SDK already integrated with 15+ agent frameworks (LangChain, CrewAI, AG2, Haystack, n8n). Happy to contribute a Ray Serve LLM middleware integration that wraps TealTiger's governance engine. Key properties: - No LLM in the governance path β€” all evaluation is deterministic - <2ms latency overhead - 40+ PII detection patterns with confidence scoring - ENFORCE / MONITOR / REPORT_ONLY modes - Structured JSON decision receipts with OpenTelemetry-compatible trace IDs - Covers 7/10 OWASP AI Security Issues --- ## Labels `enhancement`, `triage`, `serve`", "url": "https://github.com/ray-project/ray/issues/65259", "createdAt": "2026-08-06T13:42:32Z", "updatedAt": "2026-08-13T02:09:08Z", "timestamp": "2026-08-13T02:09:08Z", "metrics": { "reactions": 0, "comments": 9 }, "labels": [ "enhancement", "serve", "security", "community-backlog" ], "author": "nagasatish007", "state": "open", "assignees": [ "richardliaw" ], "change": "updated" } }, { "id": "event:723f1ec0b0485b7dcb5e", "signalId": "github:ray-project/ray:pull_request:65410", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65410", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[serve] Enforce type checking on the scheduling/routing modules", "text": "Add deployment_scheduler.py, router.py, and request_router.py to the mypy + pyrefly serve allowlists and fix their type errors. - request_router: declare the members MultiplexMixin/FIFOMixin use from the composed RequestRouter (bare annotations, Callable for methods, so they don't shadow the real methods via MRO), fix the ReplicaID/str key confusion at its sources, assert-narrow where a None would already crash, ignore the asyncio.Task `.replica` monkey-patch. - router: add an asserting `_active_request_router` property (through the lazy-loading `request_router`) for the guaranteed-non-None accesses, keeping the Optional accessor for the None-checks, narrow the singleton `_asyncio_loop`, the async-router variant returns asyncio.Future by design (override), cast the LongPoll (namespace, DeploymentID) key dicts. - deployment_scheduler: fix the ReplicaID-keyed dicts (str -> ReplicaID), import IMPLICIT_RESOURCE_PREFIX/node_labels_match_selector with one ignore, annotate the scheduling-strategy union, give the launching/running loops distinct vars.", "url": "https://github.com/ray-project/ray/pull/65410", "createdAt": "2026-08-12T02:17:22Z", "updatedAt": "2026-08-13T01:59:22Z", "timestamp": "2026-08-13T01:59:22Z", "metrics": { "reactions": 1, "comments": 1 }, "labels": [ "serve", "go" ], "author": "johntaylor-cell", "state": "open", "assignees": [ "johntaylor-cell" ], "change": "updated" } }, { "id": "event:0639fdd8c5b8d68e5f44", "signalId": "github:ray-project/ray:pull_request:65173", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65173", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[dashboard] Stream job log responses", "text": "## Description `GET /api/jobs/{job_or_submission_id}/logs` currently materializes the full driver log at each dashboard hop. Large or concurrent requests can exhaust dashboard memory. This change: - reads driver logs in 64 KiB chunks - JSON-escapes chunks incrementally - streams the response through JobAgent, JobHead, and the parent dashboard proxy with backpressure - keeps stream duration unlimited while bounding connection setup and pool waits - preserves the existing `{\"logs\": string}` response and `JobSubmissionClient.get_job_logs()` behavior The SDK still returns one complete string, so client memory remains proportional to returned log size. ## Related issues Fixes #28336. No open PR implements request-time streaming. #61537 was closed and truncated large responses. #65006 rotates logs on disk, which is complementary. ## Additional information Testing: - `.venv/bin/pytest -q python/ray/dashboard/modules/job/tests/test_job_log_storage_client.py python/ray/dashboard/tests/test_utils.py`: 4 passed - `PATH=$PWD/.venv/bin:$PATH .venv/bin/pytest -q python/ray/dashboard/modules/job/tests/test_http_job_server.py::test_submit_job_with_exception_in_driver python/ray/dashboard/modules/job/tests/test_http_job_server.py::test_missing_resources`: 2 passed - `pre-commit run --files `: passed - `npm run build` in `python/ray/dashboard/client`: passed - Local Ray head and Jobs CLI/API smoke with an 8.39 MiB log: complete output, valid JSON, chunked transfer, healthy dashboard - Three concurrent throttled full-log requests: complete valid JSON and bounded server RSS growth AI assistance was used. I reviewed every changed line and ran the tests above locally.", "url": "https://github.com/ray-project/ray/pull/65173", "createdAt": "2026-08-03T16:29:57Z", "updatedAt": "2026-08-13T01:48:31Z", "timestamp": "2026-08-13T01:48:31Z", "metrics": { "reactions": 1, "comments": 2 }, "labels": [ "dashboard", "core", "go", "community-contribution" ], "author": "bvolpato", "state": "open", "assignees": [ "rueian" ], "change": "updated" } }, { "id": "event:6ac610350e7e68193cdb", "signalId": "github:ray-project/ray:issue:65433", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:issue:65433", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "issue", "title": "[data] Fair-share allocation overstates runnable task-pool demand", "text": "### What happened + What you expected to happen Ray Data's fair-share allocator uses a task-pool operator's configured maximum concurrency as its maximum resource demand. When no maximum is configured, the reported demand is unbounded even if the operator currently has only one input block and can only launch one task. This leaves resources assigned to an operator that cannot use them. In a local 8-CPU run, a single runnable `StreamingRepartition` task was allocated as many as 4 CPUs while runnable downstream work received 6 of the 7 CPUs it could use. I expected the task pool's fair-share allocation to reflect its currently runnable work and to be recalculated when more inputs arrive. This should not change configured concurrency limits or actor-pool prewarming. Ray tasks are not preemptive, so there is a trade-off: if downstream work borrows the capacity, newly runnable upstream work may wait for an already-running downstream task to finish. The allocator should stop admitting more downstream work once the upstream demand returns. ### Versions / Dependencies Ray master at `1d73f34d61ea292d1acee871c56b934951553393`. ### Reproduction script This reproduces the pipeline shape where one file-partitioning task returns many work items in one block: ```python from dataclasses import dataclass import ray @dataclass class FilePartitioningTask: files: list[str] def partition_files(_): return { \"item\": [ FilePartitioningTask(files=[f\"file-{i}.jsonl\"]) for i in range(20) ] } class ProcessFile: def __call__(self, batch): return batch ray.init(num_cpus=8) ds = ( ray.data.from_items([None]) .map_batches( partition_files, batch_size=1, compute=ray.data.TaskPoolStrategy(size=1), num_cpus=1, ) .repartition(target_num_rows_per_block=1) .map_batches( ProcessFile, batch_size=1, compute=ray.data.ActorPoolStrategy(size=7), num_cpus=1, ) ) assert ds.materialize().count() == 20 ``` The first map runs once and returns 20 dataclass rows in one block. `StreamingRepartition` receives that single block, so it can launch one task. That task emits 20 one-row blocks that the downstream actor pool can process concurrently. This is a fair-share accounting problem rather than a `StreamingRepartition` concurrency problem: additional CPUs cannot make the one runnable repartition task execute in parallel. ### Issue Severity Medium: the pipeline continues to make progress, but runnable downstream work can be under-allocated while another operator holds CPU or GPU budget it cannot currently use.", "url": "https://github.com/ray-project/ray/issues/65433", "createdAt": "2026-08-12T19:42:14Z", "updatedAt": "2026-08-13T01:34:47Z", "timestamp": "2026-08-13T01:34:47Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "performance", "data", "community-backlog" ], "author": "praateekmahajan", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:1cfb7e977a215b8b528f", "signalId": "github:ray-project/ray:issue:65434", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:issue:65434", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "issue", "title": "[data] BlockOutputBuffer rebuilds the remainder for every row-sized output", "text": "### What happened + What you expected to happen `StreamingRepartition(target_num_rows_per_block=1)` can spend a noticeable amount of time turning one large input block into many one-row output blocks. The row-sized path in `BlockOutputBuffer.next()` builds the buffered block, slices off the next output, and puts the remainder back into a new builder. It repeats that for every output block. The remainder is no longer copied, but the builder is still rebuilt once per emitted block. This is especially visible in fanout pipelines shaped like: ``` File partitioning -> StreamingRepartition(target_num_rows_per_block=1) -> map stages ``` On my machine with current master: - Arrow block, 100,000 rows: about 0.75 seconds (133k output blocks/sec) - Pandas block, 30,000 rows: about 2.29 seconds (13k output blocks/sec) I expected the row-only path to build the input block once and advance through it by row offset. It still needs to preserve ordering, partial tails across multiple input blocks, and the existing byte-based behavior. ### Versions / Dependencies Ray master at `1d73f34d61ea292d1acee871c56b934951553393`. ### Reproduction script A focused unit test can count calls to `DelegatingBlockBuilder.build()`: 1. Add a 100-row Arrow or Pandas block to a `BlockOutputBuffer` with `target_max_block_size_rows=1`. 2. Finalize the buffer and drain all 100 outputs. 3. Assert that the builder is built once rather than once per output block. The current implementation builds it 100 times. ### Issue Severity Medium: repartitioning still completes, but output delivery is slower and downstream stages take longer to receive enough work to scale up.", "url": "https://github.com/ray-project/ray/issues/65434", "createdAt": "2026-08-12T19:42:15Z", "updatedAt": "2026-08-13T01:34:15Z", "timestamp": "2026-08-13T01:34:15Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "performance", "data", "community-backlog" ], "author": "praateekmahajan", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:1496e138f6401532f60c", "signalId": "github:ray-project/ray:pull_request:65439", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65439", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "refactor(setup): modernize string formatting and ensure explicit file encoding", "text": "## Description This PR modernizes the `setup.py` build configuration script by refactoring legacy `.format()` string formatting to modern Python f-strings. Additionally, it explicitly enforces `encoding=\"utf-8\"` in `find_version()` to ensure cross-platform consistency and prevent encoding errors on non-UTF-8 default environments (e.g., Windows). ## Related issues Related to code cleanup and build script maintenance. ## Additional information - Refactored `.format()` calls to f-strings across `setup.py`. - Added explicit `encoding=\"utf-8\"` when reading version files. - Replaced slice-based extension checking (`[-3:] == \".py\"`) with `.endswith(\".py\")`. - No functional or breaking changes introduced to the build pipeline.", "url": "https://github.com/ray-project/ray/pull/65439", "createdAt": "2026-08-12T21:12:50Z", "updatedAt": "2026-08-13T01:33:13Z", "timestamp": "2026-08-13T01:33:13Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "core", "community-contribution" ], "author": "ardozcnn", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:8ca48c03b0fa45f01117", "signalId": "github:ray-project/ray:pull_request:65440", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65440", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[serve][llm] Surface engine errors on the direct-streaming ASGI app", "text": "## Description vLLM's **engine error handler** reads `app.state.server` to flip `should_exit` and stop the uvicorn server its launcher created. `VLLMEngine.build_asgi_app` calls only `build_app` and `init_app_state`, neither of which sets it. So on the direct-streaming path, every `EngineGenerateError` / `EngineDeadError` raises `AttributeError` inside the handler, and that replaces the engine error, and therefore clients get 'State' object has no attribute 'server' instead of the real failure. ## Solution Supply a dummy object that just tolerates an attribute write `types.SimpleNamespace()`. Nothing reads `should_exit`; `terminate_if_errored` only assigns it. With the read satisfied, the handler runs on to `create_error_response(exc)` and returns vLLM's real error. Discarding the write is safe: `should_exit` only stops vLLM's uvicorn loop, which Ray never runs β€” Serve restarts replicas via `LLMServer.check_health`. ## Reproduction Stand up a server with the following script and environment variables `RAY_SERVE_LLM_ENABLE_DIRECT_STREAMING=1` and `RAY_SERVE_ENABLE_HA_PROXY=1`. ``` # serve_app.py from ray import serve from ray.serve.llm import LLMConfig, LLMServingArgs, build_openai_app llm_config = LLMConfig( model_loading_config=dict( model_id=\"qwen\", model_source=\"Qwen/Qwen3-0.6B\", ), deployment_config=dict( autoscaling_config=dict(min_replicas=1, max_replicas=1), ), ) serve.run(build_openai_app(LLMServingArgs(llm_configs=[llm_config])), blocking=True) ``` - Before ``` { \"error\": { \"message\": \"'State' object has no attribute 'server'\", \"type\": \"InternalServerError\", \"param\": null, \"code\": 500 } } ``` - After ``` { \"error\": { \"message\": \"EngineCore encountered an issue. See stack trace (above) for the root cause.\", \"type\": \"InternalServerError\", \"param\": null, \"code\": 500 } } ``` ## Related issues > Link related issues: \"Fixes #1234\", \"Closes #1234\", or \"Related to #1234\". ## Additional information > Optional: Add implementation details, API changes, usage examples, screenshots, etc.", "url": "https://github.com/ray-project/ray/pull/65440", "createdAt": "2026-08-12T21:52:52Z", "updatedAt": "2026-08-13T01:32:35Z", "timestamp": "2026-08-13T01:32:35Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "serve", "llm", "go" ], "author": "jeffreywang88", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:5e9f8d84c6219b533c2d", "signalId": "github:ray-project/ray:pull_request:65442", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65442", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[TPU] Rename dispatch to run_on_slice", "text": "# Description We chose a terse name to be less redundant originally. However, when imported like `from ray.util.tpu import dispatch`, the function is taking a commonly used word away as an identifier and is also ambiguously named at call sites. For readability, rename dispatch to run_on_slice, and keep the former name as an alias for potential existing uses. ## Related issues Function introduced in #64493", "url": "https://github.com/ray-project/ray/pull/65442", "createdAt": "2026-08-12T23:50:50Z", "updatedAt": "2026-08-13T01:30:51Z", "timestamp": "2026-08-13T01:30:51Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "train", "core", "community-contribution" ], "author": "spencer-p", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:a7f40c4c9303ab083385", "signalId": "github:ray-project/ray:pull_request:65220", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65220", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[Data] [Core] Deferring gRPC Frees and OSS callbacks to Reduce OSM Usage Update Latency", "text": "## Description Move `free_object_on_nodes_async_` (gRPC free broadcast) and out-of-scope callbacks outside of `mutex_` in `ReferenceCounter::OnObjectOutOfScopeOrFreed`. Previously, when many `ObjectRefs` are freed simultaneously (e.g., LIMIT operators, UDF failures), the gRPC calls (~70% of `RemoveLocalReference` runtime) and callbacks (~7%) executed under `mutex_`, forcing 10,000 concurrent frees to serialize into ~1s of lock contention. This change: 1. Collects gRPC and callback work under the lock into a deferred work list; 2. Swaps it out before releasing the lock; 3. Fires callbacks first (so Data-side `BlockRefCounter` updates ASAP); 4. Posts gRPC free calls to a dedicated `object_free_rpc_service_` thread, keeping them off both `io_service_` (gRPC handlers, GCS) and `object_freed_callback_service_` (lightweight BRC callbacks). ## How the speedup works `FreeObjectOnNodesAsync` sends gRPC RPCs to remote nodes to free their copies of an object. Previously it ran under `mutex_` inside `OnObjectOutOfScopeOrFreed`, so each `RemoveLocalReference` held the lock for ~80us (18us ref counting + 56us gRPC). When Python frees 10k refs, each ref's `__dealloc__` calls `RemoveLocalReference` sequentially, and the next ref can't start until the current ref's gRPC finishes. With this change, the lock is released after ref counting (~18us) and the gRPC is posted to a dedicated thread, so the main thread immediately moves to the next ref while gRPC calls run in parallel. ## Benchmark results Compared against baseline (master, averaged across 4 builds). Each latency metric measures the time from ref drop to individual callback firing. p50, p95, p99, and max are all recorded by the benchmark, but p95 is selected here for horizontal comparison. | Metric | Scale | Baseline | This PR | Speedup | |--------|-------|----------|---------|---------| | Burst drain p95 | 100 | 7.80ms | 3.60ms | 2.2x | | Burst drain p95 | 1k | 76.98ms | 18.90ms | 4.1x | | Burst drain p95 | 5k | 574.63ms | 143.70ms | 4.0x | | Burst drain p95 | 10k | 986.28ms | 266.60ms | 3.7x | | BRC burst p95 | 100 | 7.83ms | 3.60ms | 2.2x | | BRC burst p95 | 1k | 77.45ms | 39.70ms | 2.0x | | BRC burst p95 | 5k | 470.73ms | 137.70ms | 3.4x | | BRC burst p95 | 10k | 977.90ms | 226.10ms | 4.3x | Pipeline p95 (steady-state, one ref at a time) is unchanged, as expected. Per-callback p95 (individual `del ref` to callback latency) regressed at large scales (0.7ms to 3.1ms at 10k) because callback posting now happens after mutex release + deferred work swap instead of immediately under mutex. This is a tradeoff: individual callback latency is slightly higher, but total burst drain time is 3-4x faster. ## Related issues Related to ray-project/ray#64521 (callback throughput benchmark). Related to ray-project/ray#64456 (BlockRefCounter-based memory tracking).", "url": "https://github.com/ray-project/ray/pull/65220", "createdAt": "2026-08-05T00:23:53Z", "updatedAt": "2026-08-13T00:53:18Z", "timestamp": "2026-08-13T00:53:18Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "core", "data", "go" ], "author": "rayhhome", "state": "open", "assignees": [ "Yicheng-Lu-llll", "rayhhome" ], "change": "updated" } }, { "id": "event:e5e19304655c00c61c65", "signalId": "github:ray-project/ray:pull_request:65071", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65071", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[train] set Neuron rendezvous env vars in the XLA backend", "text": "## Description `_TorchAwsNeuronXLABackend` cannot bring up a Neuron collective that spans instances. On two `trn1.32xlarge` hosts with 64 workers at one NeuronCore each, backend startup fails inside `_setup_xla_torch_process_group` with ``` Nrt::BuildGlobalComm failed on NeuronCores 0-1(2): nrt_status=1, message=\"Non specific failure\". ``` The Neuron runtime builds a cross-instance communicator by having every rank meet at the endpoint named in `NEURON_RT_ROOT_COMM_ID`, and nothing in Ray sets it. `on_start` already asks worker 0 for `get_address_and_port` and uses the answer for `MASTER_ADDR` and `MASTER_PORT`, so the address is already in hand; it is just never exported under the name the runtime reads. A single instance has nothing to rendezvous with, which is why this is invisible until the job crosses a host boundary. This patch takes a second free port from worker 0 and sets `NEURON_RT_ROOT_COMM_ID=:` in the existing `set_env_vars` closure, so every worker gets the same string. A second `get_address_and_port` call rather than `MASTER_PORT` plus an offset, because the runtime needs an endpoint of its own and `get_address_and_port` returns a port it has just verified free. Setting that alone was not sufficient in my environment, which is the part I did not expect. With the root comm id correct on every rank, the run then died in the bootstrap: ``` CCOM WARN Net : No interface found in the same subnet as remote address <48820> CCOM WARN NET/Socket : No usable listening interface found ``` CCOM selects its bootstrap socket by enumerating local interfaces and looking for one whose subnet contains the peer address. Under the AWS VPC CNI each pod holds a /32, so no local interface is ever in the same subnet as a peer and the search returns nothing. Naming the interface skips the search. I want to be clear that this half is deployment specific: the /32 comes from the CNI, and on bare EC2 instances where the host address sits in a real subnet I would expect the match to succeed and this failure never to appear. I have not tested that and cannot confirm it. The `NEURON_RT_ROOT_COMM_ID` half is not conditional on any of that. So the interface is an opt-in `socket_ifname` field on `TorchXLAConfig` defaulting to `None`, not a hardcoded `eth0`. `eth0` is correct for a VPC CNI pod and wrong in general, and #42808 already removed `DEFAULT_NCCL_SOCKET_IFNAME` from Ray Train because users prefer these left alone; defaulting to `None` keeps that promise while making the knob discoverable next to `neuron_parallel_compile`. When the field is set the backend uses `os.environ.setdefault`, so a value the user already exported wins, matching the `TORCH_NCCL_ASYNC_ERROR_HANDLING` guard in `python/ray/train/torch/config.py`. Open question for the reviewer, since I can see the argument both ways. The two interface variables can already be set through `runtime_env` `env_vars`, so the field buys discoverability rather than capability, and you may prefer to drop it and document the `runtime_env` route instead. `NEURON_RT_ROOT_COMM_ID` genuinely cannot be set that way, because its correct value is not known until the worker group exists. Happy to cut the field if you would rather keep the config surface small. `NEURON_RT_ROOT_COMM_ID` itself is set unconditionally, matching how `MASTER_ADDR` and `MASTER_PORT` are handled two lines above. If you would rather it also yield to a pre-existing value, say so and I will change it. ## Related issues Closes #65070 ## Additional information Tests: - Verified on 2 x `trn1.32xlarge` (32 NeuronCores each, 64 Ray Train workers at one core per worker) on Kubernetes with the AWS VPC CNI and EFA enabled, running ray 2.56.1, torch-neuronx 2.8.0.2.10.16998, torch-xla 2.8.1, neuronx-cc 2.21.33363.0, Python 3.11.11. Before the change, backend startup fails with `Nrt::BuildGlobalComm failed`. After it, the process group forms and a 64 rank job trains across both hosts: 24 steady state steps, no NaN, and throughput around 90 percent of twice the single instance rate measured on the same two hosts in the same session. - The equivalent of this change was carried as a `TorchXLAConfig` subclass overriding `on_start` before being written as a patch, and that is what the run above used. The in-tree version differs only in taking the interface name from the config instead of hardcoding it. - `black` and `ruff` clean on the changed file, at the repo's 88 column line length. - No unit test added. There is no existing test module for this backend, and the behaviour that matters here is not observable without Trainium hardware: the env vars are trivially assertable with a mocked worker group, but that only tests that the strings are spelled correctly. I did not find a Trainium multi-instance runner in Ray's CI, so this path cannot be covered there. If you want a mock-based unit test asserting the env vars land on every worker, say the word and I will add one. - Single instance behaviour is unchanged in substance. `NEURON_RT_ROOT_COMM_ID` is set but unused when there is nothing to rendezvous with, and `socket_ifname` defaults to `None` so no interface variable is touched unless asked for.", "url": "https://github.com/ray-project/ray/pull/65071", "createdAt": "2026-07-28T08:22:51Z", "updatedAt": "2026-08-13T00:39:12Z", "timestamp": "2026-08-13T00:39:12Z", "metrics": { "reactions": 1, "comments": 3 }, "labels": [ "stale", "train", "community-contribution" ], "author": "violivei", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:6210a3e1beab3bedf21b", "signalId": "github:ray-project/ray:pull_request:65031", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65031", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[Serve] Optimize RollingWindow metrics using monotonic deque in O(1)", "text": "### What is the problem this PR is trying to solve? This PR optimizes the metrics calculations in `RollingWindowMax` and `RollingWindowMin` by replacing the `O(num_buckets)` array scan with an amortized `O(1)` monotonic deque. It also fixes a correctness bug in the legacy implementation where the currently accumulating active bucket was ignored during `get_max()` / `get_min()` evaluations until it rolled over, which could blind autoscaler metrics to immediate traffic spikes. **Performance Improvements** Microbenchmarks measuring 2,000,000 concurrent writes (`add()`) and 500,000 reads (`get_max()`) across 10 threads yielded massive query speedups with no space complexity penalty (still strictly bounded to `O(num_buckets)`): - **Reads (`get_max`)**: 30x faster (0.58s vs 17.86s) - **Mixed Concurrent Workload**: 14x faster (0.77s vs 10.84s) - **Writes**: Marginally faster (0.74s vs 0.83s) ### AI Assistance AI assistance was used to analyze the codebase, implement the monotonic deque, and run performance validations. - **Non-duplication**: Verified via `gh pr list` and `gh issue list` that no existing PR or issue addresses this optimization or the uncommitted-bucket visibility bug. - **Tests run & results**: - `pytest python/ray/serve/tests/unit/test_rolling_window.py`: All 80 concurrent edge-case tests (including newly added parity tests for `RollingWindowMin`) passed successfully. - `pre-commit run --files python/ray/serve/_private/rolling_window.py python/ray/serve/tests/unit/test_rolling_window.py`: Formatting (ruff/black) and static analysis (mypy) passed.", "url": "https://github.com/ray-project/ray/pull/65031", "createdAt": "2026-07-27T09:48:46Z", "updatedAt": "2026-08-13T00:39:08Z", "timestamp": "2026-08-13T00:39:08Z", "metrics": { "reactions": 1, "comments": 2 }, "labels": [ "serve", "community-contribution", "unstale" ], "author": "pavan51", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:2e52bd72981c9d786c38", "signalId": "github:ray-project/ray:pull_request:64738", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:64738", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[Autoscaler][AWS] Retry key pair creation after duplicate", "text": "## Description Concurrent `ray up` processes can both observe the same generated EC2 key-pair name as available and then race in `CreateKeyPair`. The losing request receives `InvalidKeyPair.Duplicate`, which currently aborts AWS bootstrap even though the existing loop can safely try another generated name. Catch only `InvalidKeyPair.Duplicate` and continue with the next key-pair name. Other `ClientError` codes still propagate unchanged, and private key material is written only for the successful `CreateKeyPair` response. ## Related issues Fixes #64673 ## Additional information Added a botocore `Stubber` regression test that reproduces the `DescribeKeyPairs`/`CreateKeyPair` race. It verifies that bootstrap selects the next generated name, writes only its private key material, and applies that name to every node type. Validation performed: - Negative control on unmodified `master`: failed with `InvalidKeyPair.Duplicate` as expected. - Focused regression test: 20/20 consecutive passes. - `python -m pytest -q python/ray/tests/aws`: 61 passed. - Ruff, Black, Python AST, and `git diff --check`: passed. OpenAI Codex assisted with the investigation, implementation, and local validation. I reviewed the final change and test behavior and take responsibility for the contribution.", "url": "https://github.com/ray-project/ray/pull/64738", "createdAt": "2026-07-14T02:57:38Z", "updatedAt": "2026-08-13T00:38:46Z", "timestamp": "2026-08-13T00:38:46Z", "metrics": { "reactions": 1, "comments": 2 }, "labels": [ "core", "community-contribution", "unstale" ], "author": "hogeheer499-commits", "state": "open", "assignees": [ "rueian" ], "change": "updated" } }, { "id": "event:7a04a911defd78b53e58", "signalId": "github:ray-project/ray:pull_request:64451", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:64451", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[WIP][serve] Use `ObjectRefGenerator._get_next_ref_n` to avoid blocking on `_to_object_ref`", "text": "Integrates with https://github.com/ray-project/ray/pull/62741 to fix the long-standing issue in serve where `_to_object_ref` blocks until the underlying result has been produced.", "url": "https://github.com/ray-project/ray/pull/64451", "createdAt": "2026-06-30T18:52:38Z", "updatedAt": "2026-08-13T00:38:41Z", "timestamp": "2026-08-13T00:38:41Z", "metrics": { "reactions": 1, "comments": 2 }, "labels": [ "stale", "go" ], "author": "edoakes", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:e9a60652916f68581620", "signalId": "github:ray-project/ray:pull_request:65406", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65406", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[Data] [DO NOT MERGE] arrow-rs Parquet reader A/B β€” treatment arm (release-test trigger)", "text": "Draft PR whose only purpose is to trigger the release pipeline for the multi-node arrow-rs vs PyArrow A/B (the reader from #65117, ported onto #64985's planner). This branch is stacked on unmerged #64985, so its diff includes that PR's commits β€” do not review or merge. Baseline arm: see the companion draft PR for `arrow-rs-ab-baseline-64985`. πŸ€– Generated with [Claude Code](https://claude.com/claude-code)", "url": "https://github.com/ray-project/ray/pull/65406", "createdAt": "2026-08-12T00:47:40Z", "updatedAt": "2026-08-13T00:30:32Z", "timestamp": "2026-08-13T00:30:32Z", "metrics": { "reactions": 0, "comments": 2 }, "labels": [ "data", "release-test" ], "author": "AarryaSaraf", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:1aaed4fd85e4faf60cf6", "signalId": "github:ray-project/ray:issue:65352", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:issue:65352", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "issue", "title": "[Umbrella] Ray Sandboxing with gVisor", "text": "### Description This is an umbrella issue to track Ray sandboxing support using gVisor. PRs: - [x] https://github.com/ray-project/ray/pull/64964 TODO: - [ ] add support for referencing local tar files for images (specifically with tar files created with `docker save`) - [ ] add modal compatible API for sandboxing - [ ] root cause why --ignore-cgroups is required in CI - [ ] add install script for runsc tests (see https://github.com/ray-project/ray/pull/64964#discussion_r3754281842) - [ ] pin the gVIsor version in tests (see https://github.com/ray-project/ray/pull/64964#discussion_r3754232229) - [ ] explore using third party library to managing container images for sandboxes - [ ] explore managing Python dependencies in sandboxes https://github.com/ray-project/ray/issues/65322 ### Use case _No response_", "url": "https://github.com/ray-project/ray/issues/65352", "createdAt": "2026-08-11T00:28:50Z", "updatedAt": "2026-08-13T00:29:50Z", "timestamp": "2026-08-13T00:29:50Z", "metrics": { "reactions": 2, "comments": 7 }, "labels": [ "enhancement", "triage", "core", "security", "community-backlog" ], "author": "andrewsykim", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:633e7fae0fada9fbc31a", "signalId": "github:ray-project/ray:pull_request:65404", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65404", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[doc][TPU] Document TPU subslicing options with Ray", "text": "## Description This PR documents the three methods available for TPU subslicing with Ray: 1. Dynamic slicing with Kueue (provisions subslices via GKE) 2. RayClusters with a subslice annotation 3. SubslicePlacementGroup from the Ray TPU driver. ## Related issues - [Dynamic slicing docs](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/dynamic-slicing) - https://github.com/ai-on-gke/kuberay-tpu-webhook/pull/32 - #64578 ## Additional information May need updates pending https://github.com/ai-on-gke/kuberay-tpu-webhook/pull/32 Also included some minor code changes to make SubslicePlacementGroup compatible with tpu.dispatch that were previously missed.", "url": "https://github.com/ray-project/ray/pull/65404", "createdAt": "2026-08-11T23:49:34Z", "updatedAt": "2026-08-12T23:42:21Z", "timestamp": "2026-08-12T23:42:21Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "docs", "core", "community-contribution" ], "author": "spencer-p", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:06ffb261b9d3c34bc231", "signalId": "github:ray-project/ray:issue:56431", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:issue:56431", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "issue", "title": "[Data/LLM] Non stop CPU autoscaling with vLLM", "text": "### What happened + What you expected to happen I'm doing batch LLM inference using Ray Data, and I've noticed for large batches of prompts, the CPU autoscaler will continuously autoscale until it reaches maximum specified amount of CPU worker nodes as defined in my configuration file, and then it will start launching extra GPU instances even though I specified that GPU instances only have 1 CPU under resources (need at least 1 or else the code for LLM inference breaks). It reaches a point where I have over 300 non active CPUs in the cluster and Ray is still trying to launch more. I see {\"CPU\":1} requested over 200 times in bursts which seemingly causes this, but this resource constraint is gone within a few seconds and I'm not sure what is causing this burst of single CPU requests. \"Image\" In the image below, each cpu worker has 32 CPUs, and we scaled it to 10 nodes which is the maximum in my config, and when it runs out it begins trying to start GPU instances (p5en, which are very expensive) to try to scale CPUs further. This ends up being a very expensive bug because of the price of these GPU instances. \"Image\" ### Versions / Dependencies Ray 2.48.0 vLLM 0.10.0 ### Reproduction script ```python start = time.time() # 1) Resolve I/O paths in_s3 = cfg[\"s3_prompts_path\"] out_s3 = cfg[\"s3_output_path\"] # 2) Load prompts οƒ  Ray Dataset instances = int(cfg[\"instances\"]) data_parallel_size = 2 partitions = max(instances * 8, 1) model = cfg[\"model\"] engine_kwargs = cfg[\"engine_kwargs\"] if model.startswith(\"s3://\"): engine_kwargs[\"load_format\"] = \"runai_streamer\" engine_cfg = vLLMEngineProcessorConfig( model_source=model, engine_kwargs=engine_kwargs, concurrency=instances, apply_chat_template=bool(cfg[\"apply_chat_template\"]), batch_size=int(cfg[\"batch_size\"]), max_concurrent_batches=4, # accelerator_type=cfg[\"accelerator_type\"], # Add runtime_env here if you need HF_TOKEN, etc. ) ds = ray.data.read_parquet(in_s3) ds = ds.repartition(partitions) processor = build_llm_processor( engine_cfg, preprocess=lambda row: dict( messages=[ {\"role\": \"system\", \"content\": row[\"system\"]}, {\"role\": \"user\", \"content\": row[\"prompt\"]}, ], sampling_params=cfg[\"sampling_params\"], ), postprocess=lambda row: dict(**{k: v for k, v in row.items() if k not in (\"prompt\", \"system\")}), ) # 4) Execute & write results print(\"⏳ Running batch inference …\") processed = processor(ds) processed.write_parquet(out_s3) duration = time.time() - start print(f\"βœ… Completed in {duration:,.1f} s β†’ {out_s3}\") ``` ### Issue Severity Medium: It is a significant difficulty but I can work around it.", "url": "https://github.com/ray-project/ray/issues/56431", "createdAt": "2025-09-10T23:10:51Z", "updatedAt": "2026-08-12T21:26:25Z", "timestamp": "2026-08-12T21:26:25Z", "metrics": { "reactions": 0, "comments": 11 }, "labels": [ "bug", "P1", "performance", "data", "llm", "stability", "community-backlog" ], "author": "jiangwu300", "state": "open", "assignees": [ "bveeramani" ], "change": "updated" } }, { "id": "event:edc577d4a6aacca36905", "signalId": "github:ray-project/ray:pull_request:63793", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:63793", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[core] Add opt-in swap accounting to memory monitor and scheduler", "text": "## Why Ray's OOM monitor kills tasks based on RAM alone, while the Linux OOM killer only fires when **both** RAM and swap are exhausted. On nodes provisioned with swap as overflow, Ray kills workers long before the kernel would. The scheduler's `memory` resource and the dashboard's Node Memory graph have the same RAM-only blind spot, and under `--enable-resource-isolation` the user-slice threshold ignores swap too β€” so scheduling, reporting, and enforcement all disagree. ## What Adds opt-in `RAY_count_swap_in_memory_monitor` (default `false`). When on, swap is folded into: 1. **C++ memory monitor** β€” cgroup v1 `memory.memsw.*` or cgroup v2 `memory.swap.{max,current}`; `/proc/meminfo` swap on bare metal. 2. **Scheduler `memory` resource** and **dashboard Node Memory** β€” via shared `get_cgroup_aware_swap_memory()` so containers see cgroup-scoped swap, not host swap. 3. **Resource-isolation user-slice OOM monitor** β€” `TakeUserSliceMemoryUsageSnapshot` now reads `user.memory.swap.{max,current}` and folds them into both used and total. Without this, `ray status` advertises RAM + swap while the monitor enforces RAM only. `memory.high` (RAM-only kernel constraint) is **not** inflated by swap β€” `GetCGroupMemoryBytes` takes `include_swap` so the cgroup-manager path stays RAM-only. Flag-off is the existing behavior. ## Test plan - C++ gtests: cgroup v2 swap, cgroup v1 memsw, `\"max\"` sentinel, missing files, user-slice swap fold, flag-off pinning. - Python unit tests: scheduler swap inclusion, explicit `--memory` bypass, dashboard `_get_mem_usage` clamps `used ≀ total`. - Manual: `RAY_count_swap_in_memory_monitor=1 ray start --head --enable-resource-isolation` β€” `ray status`, dashboard, and OOM threshold all reflect RAM + cgroup swap. ## Benchmark Verified on an Azure VM with 125.8 GB RAM + 18 GB swap (Ubuntu 24.04, cgroup v2). A Ray task allocates memory in 256 MB chunks. Same task and parameters across each pair β€” only `RAY_count_swap_in_memory_monitor` differs. ### Default OOM monitor (no resource isolation) Threshold = 70% of `total`. Flag off makes `total = RAM` (88 GB); flag on makes `total = RAM + swap` (100 GB). Target = 78 GB. | flag | completed | killed by Ray | elapsed | |---|:-:|:-:|---| | off | no | yes | 23 s | | on | yes | no | 28 s | ### Resource isolation (`--enable-resource-isolation`, cgroup v2 in a privileged container) Threshold = user-slice `memory.high`. Flag off keeps it RAM-only; flag on adds host swap. With `user_memory_proportion_high=0.5`, `memory.high` β‰ˆ 62.5 GB. Target = 69 GB. | flag | completed | killed by Ray | elapsed | |---|:-:|:-:|---| | off | no | yes | 20 s | | on | yes | no | 92 s | On-cell elapsed is 4Γ— longer than the no-isolation case because the task actually pages out to swap β€” kernel reclaim doing real I/O. ### Reproduction Script: `release/nightly_tests/stress_tests/test_count_swap_in_memory_monitor.py` (single file, deps: `ray`, `numpy`, `psutil`). Resource-isolation case requires running inside a privileged container so Ray's cgroup-v2 manager doesn't conflict with the host's systemd cgroup hierarchy.", "url": "https://github.com/ray-project/ray/pull/63793", "createdAt": "2026-06-02T09:14:52Z", "updatedAt": "2026-08-12T21:20:37Z", "timestamp": "2026-08-12T21:20:37Z", "metrics": { "reactions": 1, "comments": 10 }, "labels": [ "core", "observability", "go", "community-contribution" ], "author": "preneond", "state": "open", "assignees": [ "Kunchd" ], "change": "updated" } }, { "id": "event:d8b521d6f8ca97badcad", "signalId": "github:ray-project/ray:pull_request:65202", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65202", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[Data] Throttle OutputBackpressureGuard releases with a per-op interval", "text": "## Description **Summary:** Add an opt-in, per-operator minimum interval between `OutputBackpressureGuard` releases so the liveness escape hatch can't defeat backpressure on large-block workloads. Disabled by default; no behavior change unless configured. `OutputBackpressureGuard` is a liveness escape hatch: when backpressure policies clamp an operator's output budget to 0 bytes, the guard flips it to 1 byte so the executor emits one more block and the pipeline can't deadlock. This check runs on every scheduling iteration, so whenever the guard keeps voting to release, an operator effectively emits a block per iteration. On workloads with very large blocks (e.g. rows carrying raw audio buffers), that per-iteration release rate can outpace how fast downstream drains. At that point the escape hatch effectively defeats the backpressure it's meant to be a narrow exception to: the policies say \"0 bytes\", but the operator keeps emitting a large block every iteration anyway. Object store usage then grows unbounded even though the pipeline is nominally under backpressure, pushing the store toward spilling / OOM. This PR adds an opt-in, per-operator **minimum interval between successive guard releases**. With a positive interval, the guard releases an operator at most once per interval, giving downstream time to drain between unblocks. The first release for each operator is never throttled, and idle-detection remains the liveness fallback, so pipeline liveness is preserved. The feature is disabled by default (`None`), so existing behavior is unchanged. ## Related issues ## Additional information **New configuration** - `DataContext.output_backpressure_guard_release_interval_s: Optional[float]` - Env var: `RAY_DATA_OUTPUT_BACKPRESSURE_GUARD_RELEASE_INTERVAL_S` - Default: `None` (no throttling β€” legacy behavior). Non-positive values also disable it. **Usage** ```python import ray ctx = ray.data.DataContext.get_current() # Release a fully-throttled operator at most once every 300s. ctx.output_backpressure_guard_release_interval_s = 300 ```", "url": "https://github.com/ray-project/ray/pull/65202", "createdAt": "2026-08-04T14:18:54Z", "updatedAt": "2026-08-12T21:14:02Z", "timestamp": "2026-08-12T21:14:02Z", "metrics": { "reactions": 1, "comments": 3 }, "labels": [ "data", "go", "community-contribution" ], "author": "dragongu", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:39841622118cc1be0329", "signalId": "github:ray-project/ray:pull_request:65438", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65438", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[doc] Add a KubeRay nav section and generalize the section sidebar", "text": "## Why The KubeRay guides are one of the most-visited parts of the Ray docs, but they're buried two levels down under Ray Clusters and share the global sidebar with the entire site. A reader working through RayService configuration gets every top-level Ray section in the left nav and no KubeRay-specific orientation. This promotes KubeRay to a top-level nav tab with a sidebar scoped to just KubeRay, reusing the mechanism the APIs tab already ships. ## What changed **Navigation (`doc/source/navbar.yml`)** `Use Cases` is replaced by a `KubeRay` tab in the #2 position. Tab order is now Get Started, KubeRay, Example Gallery, Library, APIs, Resources. `ray-overview/use-cases` still exists and stays in the root toctree and global sidebar; it only loses its tab. KubeRay also stays in the global sidebar under Ray Clusters, so `cluster/getting-started` is unchanged and readers browsing Ray Clusters still find it. No page moved, so no URLs change and no redirects are needed. **Section sidebars (`doc/source/_ext/api_sidebar.py`)** The APIs tab's client-side sidebar mechanism is generalized from one hardcoded section to an N-section table. Adding a section is now a `SECTIONS` entry (root docname, fragment filename, path prefixes, label, maxdepth) plus a `navbar.yml` tab β€” no template or loader change. That also covers the \"document the extension's config contract\" follow-up from the original APIs work. Two changes were needed to make it general: - **Subtree isolation.** The old code assumed the section root was a top-level `li.toctree-l1` marked `current`. `cluster/kubernetes/index` is nested two levels deep, so it isn't. It now walks to the *deepest* `li.current`, which is the section root by construction since the toctree was resolved for it. This reduces to the identical result for APIs. - **Href rebasing.** `_root_relative` stripped exactly one leading `../`, hardcoding `apis/index`'s depth-1 position. It now rebases onto `dirname(root_docname)` via `posixpath.normpath`, correct at any depth and byte-identical for APIs. `_templates/api-sidebar.html` β†’ `section-sidebar.html` and `_static/api-nav-loader.js` β†’ `section-nav-loader.js`; the fragment is per-section (`api-nav.html`, `kuberay-nav.html`). **Tab highlighting (`doc/source/custom_directives.py`)** `render_header_nodes` compared `item[\"file\"] == pagename`, so a tab highlighted only on its exact landing page β€” the APIs tab went dark as soon as you clicked into `data/api/…`. It now reads `context.get(\"navbar_active_file\") or pagename`, which the extension sets to the matched section's root. Both the APIs and KubeRay tabs now stay highlighted across their whole section. Tabs outside a section keep the previous exact-match behavior. **No-JS and accessibility fallback** Previously the section sidebar rendered `Loading API navigation…` and populated entirely from JavaScript, so no-JS readers got no navigation at all. The mount now server-renders the section's top-level pages as a real list of links (from `env.toctree_includes`, via `pathto`). The loader upgrades it to the full tree on success and, on failure, leaves it in place rather than replacing it with an error β€” a failed fetch now degrades to a shallower nav instead of none. `aria-busy` flips to `false` on both paths. This is a meaningful improvement, not full parity: the fallback is the section's top level only, not the whole tree. ## Design note: the sidebar swap is intentional Worth stating up front, because it's the obvious review question. KubeRay keeps its existing position in the global sidebar under Ray Clusters, so nothing moves for readers browsing that section. But once you're on a `cluster/kubernetes/**` page, the sidebar is *replaced* by the KubeRay-only tree rather than the KubeRay subtree being highlighted inside the full site nav. Key Concepts, Deploying on VMs, the Applications Guide, and the rest of `cluster/` drop out of the left nav until you navigate back out. That's deliberate. KubeRay is the recommended path for running Ray on Kubernetes and it's a large body of content β€” around 80 pages across getting-started, user guides, examples, ecosystem integrations, benchmarks, and troubleshooting β€” currently reachable only by drilling two levels down through Ray Clusters. Surfacing it at the top level and giving it undivided navigation is the point of the change; keeping the whole-site tree alongside it would preserve the discoverability problem this is meant to fix. Cross-links out of the section (roughly 32 of them, mostly into `serve/`, `ray-observability/`, and `ray-core/`) still work normally and land the reader back on the global sidebar. ## Notes for reviewers - **`_ext/api_sidebar.py` keeps its name even though the mechanism is no longer APIs-specific.** It's referenced by `.buildkite/test.rules.txt`, `.buildkite/test.rules.test.txt`, `doc/BUILD.bazel`, `ci/pipeline/BUILD.bazel`, and `ci/pipeline/test_doc_api_rules_sync.py`. Renaming it would pull the shared CI tag-rules surface into this PR. `API_PATH_PREFIXES` is preserved as a module-level literal tuple so that sync test still parses it out of the AST. There's a docstring note; happy to do the rename as a separate CI-rules PR. - **Two unrelated `black` hunks in `custom_directives.py`.** Pre-existing formatting drift in `collect_example_orphans`. The file isn't in black's exclude list, so pre-commit reformats the whole file on any commit touching it. - **The fallback nav's titles come from `env.titles`**, i.e. each target page's own title, not any custom label the section root's toctree gives the entry. So the APIs fallback reads \"Ray Data API\" where the hydrated tree reads \"Ray Data\". Recovering custom labels needs `env.get_doctree(root)` per page, too expensive for a fallback. KubeRay uses no custom labels and is unaffected. - The `no toctree children` warning is suppressed when the section root isn't in `env.all_docs`, so scoped `DOC_LIB` / `build_one_lib` builds don't fail under `fail_on_warning` on an out-of-scope section. ## Testing Opened as a draft because **this has not been through a Sphinx build yet** β€” I'm looking to the RtD preview for that. What's verified locally: - `black` clean at the repo's pinned 22.10.0; modules compile; `node --check` on the loader. - `ci/pipeline/test_doc_api_rules_sync.py` passes (2/2), confirming `API_PATH_PREFIXES` still parses and stays in sync with the `doc_api` tag rule. - Subtree isolation and href rebasing checked against synthetic toctree HTML for both the depth-1 (APIs) and depth-2 (KubeRay) cases: APIs output unchanged, KubeRay correctly excludes Key Concepts, VMs, and the Ray Clusters parent. - `section-sidebar.html` rendered at three page depths, confirming correct relative fragment URLs, correct relative link targets, and current-page marking. The main thing the preview needs to confirm is the runtime assumption that `global_toctree_for_doc` marks `current` down the nested KubeRay path the way the fixture assumes. Worth a look on the preview: - KubeRay tab present at position 2, sidebar scoped to KubeRay on `cluster/kubernetes/**`. - APIs sidebar unchanged from today. - Both tabs stay highlighted on section sub-pages. - With JavaScript disabled, both sections still show a top-level nav.", "url": "https://github.com/ray-project/ray/pull/65438", "createdAt": "2026-08-12T20:41:51Z", "updatedAt": "2026-08-12T20:50:16Z", "timestamp": "2026-08-12T20:50:16Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [], "author": "dstrodtman", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:c07833be0ea3aa8326b6", "signalId": "github:ray-project/ray:pull_request:65436", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65436", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[data] Avoid rebuilding row-sized output buffers", "text": "## Description When only `target_num_rows_per_block` is set, `BlockOutputBuffer` currently builds the buffered block, emits one slice, and puts the remainder into a new builder. A StreamingRepartition with a one-row target repeats that work for every output block. This PR keeps the built block and advances through it by row offset. A builder is only needed again when a partial tail must be combined with later input. Byte-based sizing, combined row-and-byte sizing, disabled block shaping, ordering, and final partial blocks retain their existing behavior. I searched open issues and PRs for the same change. This complements the remainder-copying fix in #48266: that change avoided copying the remainder, while this change avoids rebuilding it for every emitted block. #62232 addresses user-configured StreamingRepartition concurrency and is independent of this slicing path. AI assistance was used to help investigate, implement, and test this change. ## Related issues Fixes #65434 ## Additional information I tested the change with a standalone version of the pipeline that motivated it, without importing Curator: ``` one seed -> MapBatches(generate_files_to_process) -> StreamingRepartition(target_num_rows_per_block=1) -> downstream map ``` `generate_files_to_process` returns a list of `FilePartitioningTask(files: list[str], metadata: dict)` objects from one task. With the default `DataContext`, Ray stores these rows in an `ArrowPythonObjectType` column. For 30,000 tasks, draining the row-sized output buffer changed from 30,000 builder builds in 0.281s to one build in 0.105s (2.7x). A full 1,000-task pipeline improved from a median 18.34s to 18.05s, while median time to the first downstream task was 1.383s versus 1.374s and time to the 100th was 2.825s in both cases. This shows the repeated build is removed for Python object rows, but it is not by itself an explanation for a large downstream scale-up delay. Tests run: - `.venv/bin/python -m pytest -q python/ray/data/tests/test_map_operator.py` β€” 68 passed - `.venv/bin/python -m pytest -q python/ray/data/tests/test_repartition_e2e.py -k 'target_num_rows_per_block or guarantee_row_num or streaming_repartition or partial_last_block'` β€” 21 passed, 6 deselected - Ray pre-commit hooks on all changed files β€” passed", "url": "https://github.com/ray-project/ray/pull/65436", "createdAt": "2026-08-12T19:53:17Z", "updatedAt": "2026-08-12T20:42:43Z", "timestamp": "2026-08-12T20:42:43Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [], "author": "praateekmahajan", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:8871f1d6f6d17832fa48", "signalId": "github:ray-project/ray:pull_request:65428", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65428", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[docs] vendor the KubeRay CRD API reference into the Ray docs", "text": "## Why are these changes needed? The KubeRay CRD API reference β€” the field-level reference for `RayCluster`, `RayJob`, `RayService`, and `RayCronJob` β€” is published only on the MkDocs site at `ray-project.github.io/kuberay`. That site has two problems that compound each other: - Its `mkdocs.yml` `nav:` block contains only `index.md`. The API reference builds and serves but appears nowhere in site navigation, so it's reachable only by direct URL, site search, or a search engine. - Its landing page states that documentation has moved to `ray-project/ray`. Meanwhile `doc/source/cluster/kubernetes/references.md` here is a ~14-line stub whose entire substance is an outbound link to that page. So Ray's documented answer for \"what fields does a RayCluster support?\" is a link off-site to a nav-hidden page on a site that says it holds no documentation. It's also the most actively updated file in the KubeRay `docs/` tree β€” recent CRD features (`AuthOptions`, `GcsEmbeddedStorage`, `NetworkPolicyRules`, `IngressOptions`, `HistoryServerOptions`, `RayCronJob`) all extend it. This vendors the generated reference into the Ray docs and repoints the inbound links at it. ## What changed | File | Change | | --- | --- | | `doc/source/cluster/kubernetes/references/api.md` | **New.** Vendored generated reference, `ray.io/v1` only, from the `v1.6.2` release tag. | | `doc/source/cluster/kubernetes/references.md` | Toctree entry; the outbound link becomes a `{ref}`; added a note that v1alpha1 is deprecated. | | `doc/source/cluster/kubernetes/getting-started/rayjob-quick-start.md` | Outbound link β†’ `{ref}`. | | `doc/source/cluster/kubernetes/user-guides/gcp-gke-tpu-cluster.md` | Outbound link β†’ the operator installation page (see below). | Content-only: every changed file is under `doc/`, so the `docs-go` label's scope guard (`ci/lint/validate_docs_go_scope.sh`) passes. An earlier revision of this PR also added a `.github/CODEOWNERS` entry for the generated file; it's dropped, both because it would have failed that guard and because the existing `/doc/source/cluster/kubernetes/` rule already assigns `@andrewsykim @ray-project/ray-core @ray-project/ray-docs` β€” a more specific rule would have *narrowed* ownership by dropping ray-core. The generated-file header is what deters hand-edits. ### How the source artifact is produced `docs/reference/api.md` in `ray-project/kuberay` is a committed build artifact, not a hand-written page. `make api-docs` runs `elastic/crd-ref-docs` (pinned v0.2.0) over the Go CRD types in `ray-operator/apis/ray/`. The `ray-operator-verify-api-docs` job in `.github/workflows/consistency-check.yaml` regenerates and diffs it on every push and PR to `master` **and `release-*`**, failing on drift. That CI job is what makes a vendored copy viable: freshness is already enforced upstream, so the copy has a single well-defined source of truth. Because the job covers `release-*` too, a release tag's `api.md` is equally guaranteed consistent with that branch's CRD types. ### Why the `v1.6.2` release tag, not `master` Per @andrewsykim's review, this now vendors from the latest KubeRay release rather than from `master`. `master` carries CRD surface that no release ships β€” diffing `docs/reference/api.md` at `v1.6.2` against `master` is **+247 lines**, adding twelve types that exist in no release: `CollectorOptions`, `GcsEmbeddedStorage`, `GCSStorageDeletionPolicy`, `GcsFaultToleranceBackend`, `HistoryServerOptions`, `IngressOptions`, `IngressPathType`, `NetworkPolicyConfig`, `NetworkPolicyMode`, `NetworkPolicyRules`, `TLSOptions`, `WorkerGroupNetworkPolicyRules` plus field-level additions such as `priority`, `suspend`, `timeZone`, and `command`/`args`. The source ref is the **tag**, not the release branch head: `release-1.6` is currently one commit ahead of `v1.6.2`. The page body now states the KubeRay version, so the fields a reader sees are attributable to a version they can install; the generated-file comment records the tag and its commit SHA. ### Why `ray.io/v1` only The upstream artifact documents both `ray.io/v1` and `ray.io/v1alpha1`. This copy carries only v1, for two reasons. **v1alpha1 is deprecated and on a removal runway.** It hasn't been the storage version since v1.0 (October 2023), receives no new fields, and ray-project/kuberay#5124 tracks its removal. Every CRD feature added since December 2023 is v1-only. **Carrying it would import a silent correctness bug.** `crd-ref-docs` derives anchors from the type name alone, and eleven type names are defined in both API versions (`RayClusterSpec`, `WorkerGroupSpec`, `RayJob`, `RayService`, `AutoscalerOptions`, ...). Only the v1 heading owns the slug, so every v1alpha1 cross-link resolves to the v1 definition β€” with no warning at generation time. The definitions differ substantially (`RayClusterSpec` has 14 fields in v1, 7 in v1alpha1), so a reader following a v1alpha1 link lands on a table advertising fields that don't exist in that version. Filed upstream as ray-project/kuberay#5091. Dropping the section resolves the collision rather than papering over it. The stripping step is written to become a silent no-op once upstream drops v1alpha1 from generation. ### The gcp-gke-tpu-cluster.md link This one pointed at the MkDocs `/deploy/helm/` page, not the API reference. That page documents Helm *chart values*, while the sentence introducing it is about *installing* KubeRay. It's repointed at `kuberay-operator-installation.md`, which is the maintained page for what the sentence actually promises. The `helm repo add kuberay https://ray-project.github.io/kuberay-helm/` commands elsewhere in the docs are untouched β€” that's the Helm chart repository, a different host from the docs site. ## Checks **Anchor resolution.** The vendored page has 34 h1–h4 headings producing 34 distinct slugs (zero collisions, confirming the v1-only decision eliminates that class) and 67 in-page anchor links across 31 distinct targets, all resolving. **Build.** Built the page in a minimal Sphinx project against the pinned docs versions β€” sphinx 8.2.3, myst-parser 5.1.0 β€” with `myst_heading_anchors = 4`, `nitpicky = True`, and `-W --keep-going`: ``` build succeeded. exit 0 ``` Zero warnings. Inspected the rendered HTML: all 134 internal `href`s have a matching `id`, and none resolves to a docutils fallback `idN`. This page depends on #65240 (h4 heading anchors), which has merged. `crd-ref-docs` renders every CRD type as an h4 and cross-links types by anchor, so the page could not build under the previous `myst_heading_anchors = 3`. A full `make rtd` build was not run locally; the docs CI build on this PR is the check for that. ## Follow-ups (not in this PR) **The sync job.** It diffs the processed upstream output against the copy committed here β€” not against the previous upstream version β€” so it's idempotent and self-healing: an unmerged sync PR gets updated rather than duplicated, and a hand-edit here gets corrected on the next run. Two open design points, discussed in the review thread: - *What triggers it.* The semantically correct signal is the KubeRay `release` event with `types: [released]`, which fires only for non-prerelease publishes (every `-rc.N` tag is flagged `prerelease: true`). A `push: tags` trigger would fire at Step 3 of the release process, before images, charts, and validation. Until the workflow can live in a repo that receives KubeRay events, the equivalent is a scheduled poll of `GET /repos/ray-project/kuberay/releases/latest`, which returns the maintainer-designated latest stable and so isn't fooled by interleaved release lines (`v1.5.2` and `v1.6.1` published 22 seconds apart). - *Whether Ray master should also carry a dev reference* built from KubeRay `master`, alongside the stable one. Open question for the maintainers. **Remaining MkDocs-only content.** Several other pages on the MkDocs site have no equivalent in the Ray docs β€” the Helm chart values tables, additional `rayStartParams` coverage (`--no-monitor` and the autoscaler READONLY rationale, `--block`, `--address` FQDN construction), and the image tagging and registry scheme. Being handled separately. ## Related issue number Depends on #65240 (merged). Related upstream: ray-project/kuberay#5091, ray-project/kuberay#5124. ## AI assistance AI assistance (Claude Code) was used for the anchor-collision analysis, the release-mechanics analysis, the build verification, the post-processing script, and drafting this description. I reviewed the vendored content and the link changes and can defend both.", "url": "https://github.com/ray-project/ray/pull/65428", "createdAt": "2026-08-12T14:54:47Z", "updatedAt": "2026-08-12T20:40:21Z", "timestamp": "2026-08-12T20:40:21Z", "metrics": { "reactions": 1, "comments": 6 }, "labels": [ "docs", "go", "docs-go" ], "author": "dstrodtman", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:29b75dfe9932c59b958e", "signalId": "github:ray-project/ray:pull_request:65435", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65435", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[data] Cap task pool resource demand by runnable inputs", "text": "## Description `TaskPoolMapOperator.min_max_resource_requirements()` reports infinite maximum demand when `max_concurrency` is unset. That is a useful capacity bound, but it is not always a useful fair-share bound: an operator with one active or queued input can run at most one task. This matters for pipelines where one task fans out into many rows. The following is a complete Ray Data example matching that shape: ```python from dataclasses import dataclass import ray @dataclass class FilePartitioningTask: files: list[str] def partition_files(_): return { \"item\": [ FilePartitioningTask(files=[f\"file-{i}.jsonl\"]) for i in range(20) ] } class ProcessFile: def __call__(self, batch): return batch ray.init(num_cpus=8) ds = ( ray.data.from_items([None]) .map_batches( partition_files, batch_size=1, compute=ray.data.TaskPoolStrategy(size=1), num_cpus=1, ) .repartition(target_num_rows_per_block=1) .map_batches( ProcessFile, batch_size=1, compute=ray.data.ActorPoolStrategy(size=7), num_cpus=1, ) ) assert ds.materialize().count() == 20 ``` The first map runs once and returns 20 dataclass rows in one block. `StreamingRepartition` receives one input block, so it can launch one task, then emits 20 one-row blocks for the actor pool. On an 8-CPU local cluster, master allocated as many as 4 CPUs to that single runnable repartition task while the downstream actor pool received 6. With this change, repartition stayed at 1 CPU and the downstream pool received 7 once it had queued work. This PR caps a task-pool operator's current maximum demand at its active tasks plus queued input bundles. The configured `max_concurrency` remains the upper bound. Resource budgets are recomputed as inputs arrive, so the cap grows with runnable work. Actor pools continue to report their configured pool bounds. Their actors may need to be created before input arrives, and applying the same input-based cap would change `min_size` and `initial_size` prewarming behavior. The change applies to task-pool operators generally; it does not inspect operator names or special-case `StreamingRepartition`. Capacity left by an operator's current-demand cap is offered to downstream operators that can use it. ## Trade-offs Benefits: - Fair-share allocation reflects work the task pool can run now. - Otherwise-unused CPU or GPU capacity can serve runnable downstream work. - Allocation expands on later scheduler updates when more input becomes available. - Actor-pool startup and autoscaling behavior is unchanged. Costs: - Ray tasks are not preemptive. If downstream borrows the capacity and more upstream input later arrives, the allocator stops admitting additional downstream work, but already-running downstream tasks keep their resources until they finish. - The allocator does not reserve speculative headroom for inputs that an upstream operator has not produced yet. - This changes fair-share accounting for every task-pool map operator, not only streaming repartition. The first two costs are the work-conserving trade-off: idle capacity is used now in exchange for reacquisition waiting on already-running work if demand returns. I searched open issues and PRs for the same change. #62232 is related to StreamingRepartition concurrency, but it enforces an explicit user-configured ceiling. This PR instead changes fair-share accounting for currently runnable work. Short local wall-clock results varied with actor startup and autoscaling, so the local validation above is evidence for the allocation change, not a throughput claim. AI assistance was used to help investigate, implement, and test this change. ## Related issues Fixes #65433 ## Tests - `.venv/bin/python -m pytest -q python/ray/data/tests/test_reservation_based_resource_allocator.py` β€” 18 passed - `.venv/bin/python -m pytest -q python/ray/data/tests/test_task_pool_map_operator.py::test_min_max_resource_requirements` β€” 1 passed - `.venv/bin/python -m pytest -q python/ray/data/tests/test_actor_pool_map_operator.py -k 'test_min_max_resource_requirements or test_min_max_resource_requirements_unbounded'` β€” 2 passed - Standalone example above β€” passed with 20 output rows - Ray pre-commit hooks on all changed files β€” passed", "url": "https://github.com/ray-project/ray/pull/65435", "createdAt": "2026-08-12T19:53:17Z", "updatedAt": "2026-08-12T23:38:55Z", "timestamp": "2026-08-12T23:38:55Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [], "author": "praateekmahajan", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:9721cc63f19872da0c73", "signalId": "github:ray-project/ray:issue:65350", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:issue:65350", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "issue", "title": "[Data] read_webdataset emits one DataFrame per sample, causing per-sample size_bytes overhead in the output buffer", "text": "### Problem `WebDatasetDatasource._read_stream` yields one single-row pandas DataFrame per sample. Ray's output buffer estimates block memory on every emitted block: ``` BlockOutputBuffer.has_next -> _exceeded_buffer_size_limit -> get_estimated_memory_usage -> PandasBlockAccessor.size_bytes ``` So with N samples per shard, this deep `size_bytes` pass runs ~N times per shard. PandasBlockAccessor.size_bytes runs once per sample, so it accounts for ~63% of the ReadWebDataset operator's CPU time (~2115s of 3349s per epoch), spent purely in size estimation rather than reading, according to our repro. So with N samples per shard, this deep `size_bytes` pass runs ~N times per shard. ### Reproduction Read a WebDataset shard through `read_webdataset(decoder=None)` at scale and instrument the read worker (counting `PandasBlockAccessor.size_bytes` calls and CPU), or read `ds.stats()`. On a 2000-sample shard, current master runs ~2000 `size_bytes` calls, ~3s of worker CPU, purely in size estimation rather than reading. ### Environment - **Node**: single node, g6e.4xlarge (1x NVIDIA L40S 48 GB, 13 CPU, 64 GB RAM) - **Ray**: 2.55 - **Data**: 641 train + 25 val WebDataset shards, 2000 samples/shard (1.28M / 50k), 1000 classes, `jpg` + `cls`, on AWS S3 - **Read**: `read_webdataset(decoder=None, concurrency=11, override_num_blocks=641)` - **Preprocess**: `map_batches` (PIL decode + RandomResizedCrop(224) + normalize), batch 256 - **Train**: `TorchTrainer` resnet18, global batch 512, one epoch with validation - **Compared**: current master vs a chunked reader that emits one DataFrame per 512 samples ### Benchmark | metric | current master | chunked (512) | | --- | --- | --- | | `size_bytes` share of ReadWebDataset CPU | ~63% (2115s / 3349s) | negligible | | read remote CPU per epoch | 3349 s | 1234 s (-63%) | | read throughput | 1170 rows/s | 1444 rows/s | | whole epoch (train + val) | 1238 s | 998 s | Corroboration: on an earlier cluster run steady throughput went 1287 -> 1604 img/s and time-to-first-batch 16.1 s -> 9.2 s. The reporting user independently reproduced this and measured read CPU ~4 s -> ~1 s per shard. ### Proposed fix Accumulate ~512 samples per emitted DataFrame in `_read_stream`, so the output buffer runs the memory-size estimation ~512x less. Output is byte-identical (verified via SHA256, ordered and multiset, with default and raw decoders). PR to follow.", "url": "https://github.com/ray-project/ray/issues/65350", "createdAt": "2026-08-10T23:20:05Z", "updatedAt": "2026-08-12T20:34:51Z", "timestamp": "2026-08-12T20:34:51Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "performance", "data", "community-backlog" ], "author": "Hyunoh-Yeo", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:c5e7731c75d92ff79baa", "signalId": "github:ray-project/ray:pull_request:65394", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65394", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[Data] Chunk DataFrame emission in WebDatasetDatasource._read_stream", "text": "## Description `WebDatasetDatasource._read_stream` emits one single-row DataFrame per sample, which means the output buffer runs `PandasBlockAccessor.size_bytes` once per sample. On large shards this dominates read-side CPU (~63% of the `ReadWebDataset` operator's CPU). This changes the behavior of readwebdataset and let it accumulates 512 samples per emitted DataFrame. ## Related issues Closes #65350 ## Additional information Benchmark specs are specified in #65350 - It was cross-validated with a reporter Added a parametrized test (`test_read_webdataset_chunked_samples`) that verifies the output is identical to the original per-sample behavior (row count, values, and order) - test with different sample sizes (1 / 511 / 512 / 513 / 1000 samples).", "url": "https://github.com/ray-project/ray/pull/65394", "createdAt": "2026-08-11T20:35:01Z", "updatedAt": "2026-08-12T20:34:49Z", "timestamp": "2026-08-12T20:34:49Z", "metrics": { "reactions": 1, "comments": 1 }, "labels": [ "data", "go", "community-contribution" ], "author": "Hyunoh-Yeo", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:9ddd265a2b50c3d1f205", "signalId": "github:ray-project/ray:pull_request:65420", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65420", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[core] Tombstone lease ids on CancelWorkerLease", "text": "Following up on #56195 and making CancelWorkerLease truly idempotent by tombstoning lease ids. There's currently a race with RequestWorkerLease where if the RequestWorkerLease RPC arrives AFTER CancelWorkerLease RPC then we just repeatedly trigger additional CancelWorkerLease RPC requests. We've seen a couple cases where there's a large amount of CancelWorkerLease RPC requests and this PR roughly bounds the number of CancelWorkerLease RPCs to O(number of leases), and this should allow us to ignore the message reordering case completely. There is still one potential race on the other side where if for some reason the lease is granted from the perspective of the raylet, but the reply takes a while to reach the core worker for some reason, the core worker could potentially spam additional CancelWorkerLease requests for the same lease id. Leaving this as a follow up to verify if it actually matters.", "url": "https://github.com/ray-project/ray/pull/65420", "createdAt": "2026-08-12T08:56:34Z", "updatedAt": "2026-08-12T20:04:18Z", "timestamp": "2026-08-12T20:04:18Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "core", "go" ], "author": "Sparks0219", "state": "open", "assignees": [ "Yicheng-Lu-llll" ], "change": "updated" } }, { "id": "event:074a2fb645d95089ac64", "signalId": "github:ray-project/ray:issue:65300", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:issue:65300", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "issue", "title": "The task_id/put_index contract in `GetGeneratorReturnId` is unenforced (the RAY_CHECK is a tautology)", "text": "`WorkerContext::GetGeneratorReturnId` (`src/ray/core_worker/context.cc`) documents that the caller supplies both `task_id` and `put_index` or neither, because deducing only one of them keys the ObjectID to one task while drawing the index from another. The `RAY_CHECK` meant to enforce that is a tautology: ```cpp RAY_CHECK((task_id.IsNil() && !put_index.has_value()) || (!task_id.IsNil() || put_index.has_value())); ``` With A = `task_id.IsNil()` and B = `put_index.has_value()`, the second term `!A || B` is false only when `A && !B`, and in exactly that case the first term `A && !B` is true. The disjunction holds for all four inputs, so the check has never fired since it was added in #35584. The unguarded combination that matters is a real `task_id` with no `put_index`: it takes the index from this thread's put counter, which belongs to whatever task this thread is running, so the resulting ObjectID can collide with one that task mints. `CoreWorker::AllocateDynamicReturnId` (`src/ray/core_worker/core_worker.h`) has a related problem in the same contract. Its `put_index` default is `-1`, and `ObjectIDIndexType` is `uint32_t`, so that default is an optional holding `UINT32_MAX`, not the `std::nullopt` its docstring describes. A caller using the default would abort: the two defaults together are the illegal Nil-plus-index combination, and even past the check, 4294967295 exceeds the 1e8 generator-return limit. Neither is reachable today. `libcoreworker.pxd` declares all three parameters without defaults so Cython cannot omit one, there is no C++ caller, and all four call sites in `_raylet.pyx` pass a legal combination. So this is about making the contract hold for the next caller, not about a bug users are hitting. I have a fix and tests ready.", "url": "https://github.com/ray-project/ray/issues/65300", "createdAt": "2026-08-08T18:38:41Z", "updatedAt": "2026-08-12T20:02:55Z", "timestamp": "2026-08-12T20:02:55Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "core", "tech-debt", "stability", "community-backlog" ], "author": "LuciferYang", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:eb642232b7838d76edaf", "signalId": "github:ray-project/ray:pull_request:65301", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65301", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[Core] Enforce the task_id/put_index contract in GetGeneratorReturnId", "text": "## Description `WorkerContext::GetGeneratorReturnId` documents that the caller supplies both `task_id` and `put_index` or neither, because deducing only one of them keys the ObjectID to one task while drawing the index from another. The `RAY_CHECK` meant to enforce that is a tautology: ```cpp RAY_CHECK((task_id.IsNil() && !put_index.has_value()) || (!task_id.IsNil() || put_index.has_value())); ``` With A = `task_id.IsNil()` and B = `put_index.has_value()`, the second term `!A || B` is false only when `A && !B`, and in exactly that case the first term `A && !B` is true. The disjunction holds for all four inputs, so the check has never fired since it was added in #35584. The unguarded combination that matters is a real `task_id` with no `put_index`: it takes the index from this thread's put counter, which belongs to whatever task this thread is running, so the resulting ObjectID can collide with one that task mints. Using `&&` for the second term makes the check the both-or-neither predicate the doc describes. `AllocateDynamicReturnId`'s `put_index` default was `-1`. `ObjectIDIndexType` is `uint32_t`, so that default is an optional holding `UINT32_MAX`, not the `std::nullopt` its docstring describes. A caller using the default would abort: the two defaults together are the illegal Nil-plus-index combination, and even past the check, 4294967295 exceeds the 1e8 generator-return limit. Defaulting it to `std::nullopt` makes the two defaults together mean \"deduce both\", which is what the docstring promises. This also compares the unwrapped `current_put_index` against the generator-return limit instead of the optional. `std::optional`'s mixed comparison yields false when empty, so an empty optional would skip that check rather than trip it. The optional is engaged at that point today, so this is only to keep the guard from silently going dead if the surrounding branches are ever merged. Neither defect is reachable today: `libcoreworker.pxd` declares all three parameters without defaults so Cython cannot omit one, there is no C++ caller, and all four call sites in `_raylet.pyx` pass a legal combination. So this makes the contract hold for the next caller rather than fixing something users hit. ## Related issues Fixes #65300 ## Additional information Adds `context_test.cc`; `WorkerContext` had no test file. Two death tests cover the illegal combinations and two cases cover the legal ones, including the Nil-plus-nullopt form that both production Python call sites use. The death tests match the check message rather than `\"\"` or `\".*\"` deliberately. Before the fix, Nil with an index slipped past the tautology and segfaulted dereferencing a null current task, and since a segfault also counts as death, a permissive matcher would have accepted it as the expected failure. I verified both directions by reverting only the predicate and re-running: the two death tests then fail in two different ways (one never dies, the other dies with the segfault instead of the check), and both pass with the fix. `bazel test //src/ray/core_worker/tests:context_test` passes, and `//src/ray/core_worker:core_worker_lib` builds. I used AI assistance to investigate and draft this change. I reviewed every changed line and ran the build and tests myself.", "url": "https://github.com/ray-project/ray/pull/65301", "createdAt": "2026-08-08T18:39:02Z", "updatedAt": "2026-08-12T20:02:54Z", "timestamp": "2026-08-12T20:02:54Z", "metrics": { "reactions": 1, "comments": 1 }, "labels": [ "core", "go", "community-contribution" ], "author": "LuciferYang", "state": "closed", "assignees": [ "martinlhw" ], "change": "updated" } }, { "id": "event:3eab64a6407d37fa23a1", "signalId": "github:ray-project/ray:pull_request:65245", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65245", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[core] Add local:// runtime_env URIs for code already in node", "text": "## Description - Adds a `local://` runtime_env URI: `working_dir: local:///app` - Used in place with no uploading or unpacking ## Changes - `Protocol.LOCAL`: `parse_uri` returns the path itself, rejects non absolute paths. - `working_dir`/`py_modules`: no upload or download, 0 bytes cached, never deleted. - new section in the docs.", "url": "https://github.com/ray-project/ray/pull/65245", "createdAt": "2026-08-06T00:50:03Z", "updatedAt": "2026-08-12T19:35:14Z", "timestamp": "2026-08-12T19:35:14Z", "metrics": { "reactions": 1, "comments": 1 }, "labels": [ "docs", "core", "go" ], "author": "YashwanthRanjanSingaravel", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:1b9e07dd8c2abf70c6aa", "signalId": "github:ray-project/ray:pull_request:65215", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65215", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[serve] remove deprecated Serve APIs", "text": "## Changes - Remove deploy_mode from ServeInstanceDetails, plus the single member ServeDeployMode enum and doc example - Remove the RAY_AGENT_ADDRESS deprecation warning and its three CLI call sites - Remove use_new_handle_api from DeploymentHandle.options - Raise ValueError instead of warning on '#' in deployment names, since '#' is the replica ID delimiter - Remove _deprecation_warning, marked for removal in 3.0.0", "url": "https://github.com/ray-project/ray/pull/65215", "createdAt": "2026-08-04T21:44:37Z", "updatedAt": "2026-08-12T19:34:19Z", "timestamp": "2026-08-12T19:34:19Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "serve", "go" ], "author": "YashwanthRanjanSingaravel", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:8acd1ceabe591fb3347e", "signalId": "github:ray-project/ray:pull_request:65263", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65263", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[docs] Add a Kubernetes and Ray scheduling orientation guide", "text": "## Description Running Ray on Kubernetes involves two schedulers. Kubernetes places pods on machines, and Ray places tasks and actors on pods. The docs cover each layer well on its own, but nothing explains how they relate or which one to investigate when a workload doesn't start. The closest existing page is `k8s-autoscaler.md`, but it's scoped to autoscaling and marked advanced, so a user debugging a pending actor is unlikely to find it. This PR adds a user guide that: - Names the two layers and what each one places. - Documents how KubeRay derives Ray's logical resource capacity from the container spec, which is where the two layers most often disagree. - Maps pod state to the layer worth investigating. - Defers all mechanism to the existing Ray Core scheduling, RayCluster config, and autoscaler pages rather than restating them. It also adds a pointer to the new page from the autoscaler guide. The KubeRay resource-derivation details are drawn from `user-guides/config.md` (container limits, the CPU-request fallback as of KubeRay 1.3.0, integer rounding, and ignored memory and GPU requests). ## Related issues None. This came out of a documentation gap audit, not a filed issue. ## Additional information ### Not a duplicate Searched open PRs against `ray-project/ray` for \"kubernetes scheduling\", \"scheduling overview\", and related area keywords. No open PR covers this. No existing page in `doc/source` joins the two scheduling layers. ### Testing **Source verification against the KubeRay implementation.** Every claim about how KubeRay derives Ray's logical capacity was checked against `ray-operator/controllers/ray/common/pod.go` in `ray-project/kuberay`, not just against the existing docs: - `generateRayStartCommand` reads `resource.Limits[ResourceCPU]` for `num-cpus`, and falls back to `resource.Requests[ResourceCPU]` when the limit is zero. - The CPU-request fallback landed in kuberay commit `ba50bfa8` (#2365). It's absent from `v1.2.1` and present in `v1.3.0`, which confirms the \"starting with KubeRay 1.3.0\" attribution. - Integer rounding follows from `Quantity.Value()`, which rounds up away from zero, so a `500m` limit becomes one logical CPU. - Memory reads `Limits[ResourceMemory]` only, with no request fallback, and accelerators go through `addWellKnownAcceleratorResources(rayStartParams, resource.Limits)`. Both confirm that requests are ignored for memory and GPU. **Other checks:** - `vale doc/source/cluster/kubernetes/user-guides/scheduling.md` β€” remaining errors are `Vale.Spelling` on \"tolerations\" and \"autoscalers\" (standard terms used throughout the existing KubeRay docs, absent from the Vale vocabulary) and one `Google.OxfordComma` false positive on a two-item disjunction. The existing `k8s-autoscaler.md` reports a comparable baseline. - `pre-commit run` on the staged files β€” every hook reports \"no files to check\". No pre-commit hook currently covers Markdown under `doc/source/`. - Every `{ref}` target used on the page was verified to exist by grepping for its label definition. - **A full Sphinx docs build was not run locally.** Link resolution and toctree wiring rely on CI. Flagging this explicitly rather than implying broader verification than was performed. ### Self-review findings already fixed A critical pass before requesting review caught that the page claimed Ray integrates with four batch schedulers. There are five: the `kubernetes-sigs/scheduler-plugins` integration provides gang scheduling for RayCluster through the PodGroup API as of KubeRay v1.4.0. Corrected, with the fifth link added. ### AI assistance AI assistance was used to draft this page. The content is grounded in the repository sources cited above rather than generated from prior knowledge. It still needs a final human review pass before it should be considered ready, which is why this is opened as a draft. --- ### Update: review feedback addressed (983186c) No longer a draft, so the \"final human review pass\" caveat above is satisfied. @andrewsykim asked for the Ray node to Kubernetes Pod mapping stated in the intro, since it isn't obvious to readers new to Ray or KubeRay. It was present but buried as a subordinate clause. It now opens the page in its own paragraph, with the head and worker corollary and a note on why the page switches between \"Pod\" and \"Ray node\". The resource table also said a Ray node \"is\" a Pod; it runs as one, and the Pod is a superset that can hold the autoscaler sidecar, a log shipper, or a RayJob submitter. Both bot comments applied. One added a technical claim, verified the same way as the rest: - **The limits come from the main Ray container, which must be first in the `containers` list.** `utils.RayContainerIndex` is `0` in `ray-operator/controllers/ray/utils/constant.go`, and `BuildPod` passes `pod.Spec.Containers[utils.RayContainerIndex].Resources` into `generateRayStartCommand`. So limits set on a sidecar don't contribute to the Pod's logical capacity, which is the reason the detail is worth stating. - Added a `` placeholder to the `kubectl describe pod` command. `pre-commit run` again reports \"no files to check\" for every hook, consistent with the note above that nothing covers Markdown under `doc/source/`. Still no local Sphinx build.", "url": "https://github.com/ray-project/ray/pull/65263", "createdAt": "2026-08-06T16:54:19Z", "updatedAt": "2026-08-12T19:27:42Z", "timestamp": "2026-08-12T19:27:42Z", "metrics": { "reactions": 1, "comments": 1 }, "labels": [ "docs", "core", "go", "docs-go" ], "author": "dstrodtman", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:83485c11d7877a47df54", "signalId": "github:ray-project/ray:pull_request:65264", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65264", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[docs] Add a scheduling overview with defaults to the Ray Core scheduling page", "text": "## Description `doc/source/ray-core/scheduling/index.rst` carried a standing note that the page lacked an overview of the features and configs that affect scheduling, along with their default values. Most of those defaults are already documented, but they're spread across the resources, labels, and placement group pages, so a reader has no single place to see what Ray does before any configuration. This PR adds a \"Scheduling at a glance\" section that: - Describes the two-step placement model: narrow to feasible nodes, then pick one. - Tabulates each control with its default, linking to the page that documents it rather than restating the mechanism. - Tabulates the three environment variables that tune the `\"DEFAULT\"` strategy. - Calls out that the non-zero actor scheduling default means an actor needs a node with a free CPU to start, which is a recurring source of confusion. It removes the standing note, since this addresses it. Happy to restore it if maintainers consider the gap only partly closed. One genuine addition: `RAY_scheduler_top_k_absolute` was named on the page without its value. It defaults to `1` per `src/ray/common/ray_config_def.h`. ## Related issues None. The work is driven by the in-file note. ## Additional information ### Not a duplicate Searched open PRs against `ray-project/ray` for \"scheduling overview\" and related area keywords. Nothing open touches this page. ### Testing **Every default in the new tables was verified against source, not against other docs pages:** - The three scheduler environment variables come from `src/ray/common/ray_config_def.h`: `scheduler_spread_threshold` 0.5, `scheduler_top_k_fraction` 0.2, `scheduler_top_k_absolute` 1. - The actor defaults were checked in `python/ray/actor.py`. A bare actor takes the simple branch, giving it 0 lifetime CPU and an `actor_method_cpu` of 1. Because method CPU is 1, `actor_placement_resources` becomes the lifetime resources with CPU incremented by 1. That's what makes \"1 CPU for scheduling, 0 for running\" true, and it's why the constants alone read as if they say the opposite. - Node auto-detection and the task default come from `scheduling/resources.rst`; the default labels come from `scheduling/labels.md`. - Every `:ref:` and `:doc:` target in the new section was verified to exist by grepping for its label definition. **Other checks:** - `vale doc/source/ray-core/scheduling/index.rst` β€” the added content introduces no new class of Vale error. The message-set diff against the pre-change file is empty. - `pre-commit run` on the staged file β€” every hook reports \"no files to check\". - **A full Sphinx docs build was not run locally.** Cross-reference resolution relies on CI. ### Self-review findings already fixed A critical pass before requesting review caught three problems in the first draft: - Data locality was described as breaking ties for tasks. The page's own `\"DEFAULT\"` section says locality takes precedence over utilization scoring, so Ray prefers the locality node regardless of how utilized it is. \"Tie-breaking\" understated it. - The gang placement row referred to bundles being scheduled independently outside a placement group, but bundles only exist inside one, so it described something that can't happen. - A sentence said \"both defaults above are non-zero\" when the two actor defaults are 1 and 0, contradicting the row directly above it. ### AI assistance AI assistance was used to draft this section. Every default stated was read out of the source tree rather than recalled. It still needs a final human review pass, which is why this is opened as a draft.", "url": "https://github.com/ray-project/ray/pull/65264", "createdAt": "2026-08-06T16:54:31Z", "updatedAt": "2026-08-12T19:26:51Z", "timestamp": "2026-08-12T19:26:51Z", "metrics": { "reactions": 1, "comments": 1 }, "labels": [ "docs", "core" ], "author": "dstrodtman", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:0ec2775b85074417e6f3", "signalId": "github:ray-project/ray:pull_request:65429", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65429", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[doc] Pin the RocksDB GCS example to the release that introduced it", "text": "## Why The embedded RocksDB GCS backend page told readers the feature wasn't in a Ray release yet and pointed its manifest at `rayproject/ray:nightly`. The Ray half of that is no longer true: the backend shipped in **2.57.0** (`_is_rocksdb_gcs` is present in `ray-2.57.0`, absent in `ray-2.56.1`), and `rayproject/ray:2.57.0` resolves on Docker Hub. Beyond the expired claim, a floating tag isn't the right way to flag a preview feature. Docs land on `master` immediately but aren't published until they ship to `latest`, which tracks a Ray release, so by the time a reader sees this page the feature is in a release by construction. Naming the release that introduced it is both accurate and more useful than a nightly build. ## What changed - Pin both container images to `rayproject/ray:2.57.0` instead of `nightly`. - Replace the \"isn't in a stable Ray release yet\" prerequisite with the concrete floor: Ray 2.57.0 or later. - State the KubeRay prerequisite as KubeRay v1.7 or later, per @andrewsykim's review. `GCSFaultToleranceEmbeddedStorage` is on KubeRay master and absent from `v1.6.2`, the current release, so v1.7 is the release that ships the gate. The alpha admonition stays. The feature is alpha and gated off by default, which is independent of which release contains it. ## Verification - `_is_rocksdb_gcs` present in `git show ray-2.57.0:python/ray/_private/node.py`, absent in `ray-2.56.1`. - `rayproject/ray:2.57.0` returns 200 from the Docker Hub tag API. - `GCSFaultToleranceEmbeddedStorage` present in `ray-operator/pkg/features/features.go` on kuberay master, absent at tag `v1.6.2`.", "url": "https://github.com/ray-project/ray/pull/65429", "createdAt": "2026-08-12T15:37:58Z", "updatedAt": "2026-08-12T19:24:14Z", "timestamp": "2026-08-12T19:24:14Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "docs", "core", "go", "docs-go" ], "author": "dstrodtman", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:b996a4d8f20ee4fff0aa", "signalId": "github:ray-project/ray:pull_request:65430", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65430", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[doc] Correct the fake multi-node docker image guidance", "text": "## Why Three claims in this paragraph are stale or wrong: 1. **The default image is misstated.** The doc says `rayproject/ray:nightly`. The code says `DEFAULT_DOCKER_IMAGE = \"rayproject/ray:nightly-py{major}{minor}-cpu\"` (`python/ray/autoscaler/_private/fake_multi_node/test_utils.py:26`), formatted with the interpreter's own version at `test_utils.py:195`. 2. **`rayproject/ray:nightly-py38-cpu` is frozen.** Docker Hub last pushed it on 2024-03-13, when py38 support was dropped. It still returns 200, so a contributor who copies it gets a two-year-old image with no error. 3. **The Buildkite image it points at doesn't exist.** `rayproject/ray:multinode-py38` 404s on Docker Hub, and no step in `.buildkite/`, `ci/`, or `docker/` builds it. The paragraph tells contributors to set `RAY_DOCKER_IMAGE` to an image nothing produces. ## What changed - State the actual default, `rayproject/ray:nightly-py-cpu`, and why nightly is the right default here: this provider tests the autoscaler on master, not on a release. - Drop the dead Buildkite/`multinode-py38` instructions and keep the reusable advice they were wrapping: bring an image with `openssh-server` already installed and set `RAY_HAS_SSH=1`, otherwise every node installs the package on startup (`node_provider.py:148-155`). `RAY_DOCKER_IMAGE`, `RAY_HAS_SSH`, `RAY_TEMPDIR`, and `RAY_TESTHOST` are all still read by the provider, so the surrounding guidance stands as written.", "url": "https://github.com/ray-project/ray/pull/65430", "createdAt": "2026-08-12T15:39:02Z", "updatedAt": "2026-08-12T19:23:24Z", "timestamp": "2026-08-12T19:23:24Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "docs", "core", "go", "docs-go" ], "author": "dstrodtman", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:76acf765168a965f4286", "signalId": "github:ray-project/ray:pull_request:65431", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65431", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[ci] Scope the doc_readme lint check to the files it actually checks", "text": "## Why this change is needed `lint: doc_readme` carries the `always` tag, so it runs on every pull request, including documentation-prose-only ones. Its inputs are the repo-root `README.rst` and `python/setup.py`. Nothing under `doc/` feeds it, so a prose change cannot change its result. What it actually checks, from `ci/lint/lint.sh`: ```bash doc_readme() { /usr/bin/python -m pip install -c python/requirements_compiled.txt docutils cd python && /usr/bin/python setup.py check --restructuredtext --strict --metadata } ``` `--restructuredtext` validates `long_description`, which `python/setup.py` reads from the repo-root `README.rst`. `--metadata` validates `setup.py`'s own metadata fields. So the check guards how Ray's PyPI long description renders. It's a useful gate and this PR doesn't touch what it does; it's pointed at the wrong trigger. It ended up in the `lint-small-prose` matrix, which exists for the opposite reason: that matrix holds the checks a Markdown or reStructuredText change *can* fail, so they survive the prose filter that strips the code-oriented `lint-small` matrix. The name `doc_readme` reads like documentation. The other three steps in that matrix belong there; this one is packaging. ## What this PR does 1. Moves `doc_readme` into its own step tagged `pypi_readme` alone, with no `lint` and no `always`. 2. Adds a rule in `.buildkite/always.rules.txt` emitting `pypi_readme` for `README.rst` and `python/setup.py`. Both files keep `always lint`, which the catch-all would otherwise have given them, so no other step's trigger narrows. 3. Adds test cases to `.buildkite/always.rules.test.txt` for both inputs and for neighbors that must not emit the tag. 4. Rewrites the matrix comment to state the rule rather than imply it: adding a step to the prose matrix asserts a prose-only diff can fail it. Prose pull requests and code pull requests that leave both files alone both stop selecting the step. The second half of that is broader than strictly necessary to fix the prose case, and it's the point: a check that cannot fail from your change shouldn't be in your PR's status list. ## Two consequences worth flagging **Microcheck eligibility.** Dropping `lint` from the step means the microcheck pipeline's hardcoded `tag:lint` selector no longer makes it eligible there, so it runs on premerge only. Selects and rule-emitted tags are ANDed in rayci's step filter, so this is the expected outcome rather than a surprise. Microcheck is non-blocking; premerge is the gate this check has to hold. Noted in the step comment. **Step IDs shift within the lint group.** Step IDs are positional (`fmt.Sprintf(\"g%d_s%d\", i, j)`), so inserting a step moves `api-param-coverage` from `g11_s3` to `g11_s4` and `docs-go-scope` from `g11_s4` to `g11_s5`. `Test.gen_microcheck_step_ids` derives step IDs from recent Bazel test results, and group `g11` contains no Bazel test steps, so nothing keyed on step ID is affected. ## Not a duplicate No open PR touches `doc_readme` (`gh pr list --repo ray-project/ray --state open --search doc_readme` returns nothing). #65368 edits the same matrix to remove `banned_words`, so these two will conflict textually. They're independent changes and either merge order works; whichever lands second needs a trivial rebase. ## Testing Rules layer, from the branch: ``` $ rayci test-rules $ echo $? 0 ``` 125 cases pass. Negative control: reverting the `README.rst` expectation to `always lint` fails with `+pypi_readme (unexpected)`, confirming the test is reading the new rule rather than passing vacuously. Step selection, generating the pipeline that the pinned rayci binary (v0.46.0, from `.rayciversion`) actually emits for a probe diff, rather than reasoning about tags: | Probe diff | Selected tags | `lint: doc_readme` | | --- | --- | --- | | `doc/source/index.rst`, at `master` | `always` | runs, inside the `g11_s0` matrix | | `doc/source/index.rst`, on this branch | `always` | not emitted | | `python/ray/actor.py`, on this branch | `always lint …` | not emitted | | `README.rst`, on this branch | `always lint pypi_readme` | emitted, `g11_s1` | | `python/setup.py`, on this branch | `… pypi_readme …` | emitted, `g11_s1` | The `master` row is the before state: on a prose-only diff the emitted prose matrix contains `banned_words`, `doc_readme`, `documentation_style`, `doc_no_new_rst`. On this branch that matrix has three entries and no separate `doc_readme` step appears. Not verifiable outside Buildkite: `if:` expression evaluation, which this PR doesn't touch. ## AI assistance AI assistance was used for this change. Every changed line was reviewed by a human before this PR was opened, and the commands above were run locally.", "url": "https://github.com/ray-project/ray/pull/65431", "createdAt": "2026-08-12T15:44:20Z", "updatedAt": "2026-08-12T19:22:17Z", "timestamp": "2026-08-12T19:22:17Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "core", "devprod", "go" ], "author": "dstrodtman", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:f3241afae4e3883daea1", "signalId": "github:ray-project/ray:issue:65280", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:issue:65280", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "issue", "title": "[Data] OpTask._cancel never passes force=True", "text": "## Description `OpTask._cancel` never passes `force=True` to `ray.cancel`, on any path, so a blocked UDF keeps holding its CPU slot after execution fails. The actor-task check is the cause (`python/ray/data/_internal/execution/interfaces/physical_operator.py`): ```python is_actor_task = not self.get_task_id().actor_id().is_nil() ... force=force and not is_actor_task, ``` `TaskID::ForNormalTask` fills the actor-id slot with `ActorID::NilFromJob(job_id)` β€” nil unique bytes plus the job id β€” while `ActorID::Nil()` is all `0xff`. A job-scoped nil never equals it, so `is_nil()` is `False` for normal tasks too. `is_actor_task` is unconditionally `True` and `force and not is_actor_task` is always `False`. Ray Core classifies correctly β€” `CoreWorker::CancelTask` reads the task spec and calls `IsActorTask()`. This is a Ray Data–side issue only. **Expected:** `force=True` reaches `ray.cancel` for normal tasks and the CPU is released. **Actual:** every task is treated as an actor task and the blocked task holds its CPU indefinitely.", "url": "https://github.com/ray-project/ray/issues/65280", "createdAt": "2026-08-07T06:55:08Z", "updatedAt": "2026-08-12T19:09:21Z", "timestamp": "2026-08-12T19:09:21Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "data", "stability", "community-backlog" ], "author": "Hyunoh-Yeo", "state": "open", "assignees": [ "Hyunoh-Yeo" ], "change": "updated" } }, { "id": "event:f31b8aac5c3f3234e507", "signalId": "github:ray-project/ray:pull_request:65239", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65239", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[docs] Add Kubernetes and KubeRay conventions to the style guide", "text": "## Why this change Ray on Kubernetes docs describe Kubernetes API objects alongside Ray concepts, and the two vocabularies collide. Deployment, Job, and Service each name both a Kubernetes API object and a distinct Ray concept, so capitalization is the only thing telling them apart, and the style guide didn't say which form to use where. A contributor asking \"RayCluster or Ray cluster?\" had to derive the answer from two guides and adjudicate between them. This adds a `Writing Ray on Kubernetes and KubeRay docs` section to the style guide, scoped to the user-facing pages under `doc/source/cluster/kubernetes/` and the contributor-facing pages in [ray-project/kuberay](https://github.com/ray-project/kuberay). The approach is to extend this guide with a Kubernetes deference section rather than stand up a separate KubeRay style guide. The two guides already overlap heavily, and a forked guide would drift within a release or two. ## What the casing scan found The term list isn't invented. It comes from a prose-only scan of three corpora: 79 user-facing pages in this repo, 12 contributor pages in `ray-project/kuberay`, and 128 pages of `kubernetes/website` (`content/en/docs/concepts/`, commit `bb38117c0`). The scan strips front matter, fenced code blocks, inline code, MyST roles, and link targets before counting, so YAML field names and CLI output don't inflate the figures. Headline: **these docs already follow the Kubernetes casing rule.** Upstream Kubernetes capitalizes \"Pod\" in 71% of prose uses; the Ray-side KubeRay docs do so in 70%. So this PR documents a rule the docs mostly already follow, and does not propose a capitalization sweep. | Term | Ray-side KubeRay docs | Upstream Kubernetes | Read | |---|---|---|---| | Pod | 371/527 (70%) | 3712/5195 (71%) | Matches upstream almost exactly. | | ConfigMap | 20/21 (95%) | 119/130 (91%) | Healthy. | | Ingress | 8/15 (53%) | 148/242 (61%) | Roughly aligned. | | Service | 18/114 (15%) | 515/946 (54%) | Diverges by design. | | Job | 17/143 (11%) | 522/610 (85%) | Diverges by design. | | Deployment | 6/53 (11%) | 355/461 (77%) | Diverges by design. | The three \"diverges by design\" rows are the reason this section exists. Sampling shows the lowercase uses are Ray Serve deployments, Ray jobs, and Ray services, not Kubernetes API objects. That's a naming collision, not drift, so the fix is a disambiguation rule rather than a sweep. Kubernetes's own rule expects a mix (UpperCamelCase for the API object, sentence case for the general concept), which is why the upstream column isn't 100% either. The one genuine inconsistency the scan found is **\"Ray Autoscaler\" (44 uses) against \"Ray autoscaler\" (35)**, spread across at least eight files. This guide already says to lowercase generic nouns even when they name a Ray concept, so lowercase wins. Recording the rule here is the prerequisite for a mechanical follow-up cleanup. I've left that out of this PR to keep the rule change reviewable on its own. ## Open questions for reviewers I've deliberately written only the rules I can support from the two published guides plus the scan. These five are genuinely open, and I'd rather resolve them in review than assert them: 1. ~~**Confirm the custom-resource split.**~~ **Resolved** by @win5923: the split is intentional and consistent with Kubernetes terminology. Original question: The section documents `RayCluster` (the resource) versus \"Ray cluster\" (the running thing it produces), currently 335 against 106 uses in this repo. I read it as intentional and matching the Kubernetes API-object rule. Worth confirming rather than normalizing to one form. 1. ~~**\"head Pod\" versus \"head node.\"**~~ **Resolved** by @win5923: use \"head Pod\" consistently when a Ray cluster runs on Kubernetes. Original question: The section prefers \"head Pod\" for the Kubernetes-hosted case, since a node is a machine on Kubernetes. `doc/source/cluster/kubernetes/index.md` currently says \"a head node pod and a collection of worker node pods,\" which stacks both vocabularies. Reasonable, or is \"head node\" worth keeping for continuity with Ray's core architecture docs? 1. **Feature graduation phases.** Kubernetes uses start case (\"DRA is Beta\"); Ray docs lowercase feature stages. This bites on pages discussing a Kubernetes feature's stage and a Ray feature's stage in one paragraph. I left it out of the section rather than pick. Which wins? 1. **Field values.** Kubernetes wants unquoted, unstyled values (\"set `imagePullPolicy` to Always\"); this guide says to use backticks for configuration values and gives `num_replicas: 1`. These are closer than they look, since one is a bare value and the other a key-value pair, but the guidance should say which form applies where. 1. **\"Ray dashboard\" versus \"Ray Dashboard\"** (43 title-case against 18 lowercase). This guide implies lowercase, but the dashboard is arguably a named UI surface. Not addressed in the section pending a call. Two further items are out of scope here but worth flagging: - **Line wrapping in the `kuberay` repo.** This guide requires soft wrap. The Kubernetes guide requires manual wrap, justified by localization teams diffing per line, which KubeRay doesn't have. Nothing in this PR changes that, since this guide already governs `doc/source/`. But `kuberay`'s `.markdownlint.yaml` sets `MD013` to 120 characters via pre-commit, which is a latent gate: 266 lines in its `docs/` already exceed it, it isn't enforced in CI, and it isn't auto-fixable, so contributors hit it unpredictably on whichever file they touch. Disabling it there would align the two repos, but that's a call for KubeRay maintainers in their own repo. - **Turning Vale enforcement on.** This PR now adds the ruleset but doesn't enforce it. See the section below. ## Staged Vale ruleset A later commit adds a Vale ruleset for these pages, following the pattern already used for the Ray Data docs. **Nothing is enforced yet.** The Vale hook in `.pre-commit-config.yaml` stays scoped to `^doc/source/data/.*`, so CI checks exactly what it checked before. - `.vale/styles/config/vocabularies/KubeRay/accept.txt` β€” the KubeRay custom resources, the Kubernetes and ecosystem API objects these pages reference, and project names. Skips what the `General` vocabulary already accepts (`Kubernetes`, `kubectl`, `CRD(s)?`, `namespace`). - `.vale/styles/KubeRay/Terms.yml` β€” a `substitution` rule at `warning` level covering only the casing forms the new style-guide section settles, with `vocab: false` (see below). The open questions above are deliberately absent. - `.vale.ini` β€” registers the vocabulary and scopes the style to `doc/source/cluster/kubernetes/`. Registering a vocabulary that the hook doesn't exercise matches existing practice here: `RLlib` and `Train` are already in `Vocab` while the hook only runs on `doc/source/data/`. Contributors can run it by hand today: ``` vale doc/source/cluster/kubernetes/ ``` To enforce later, widen the `files` pattern on the `vale` hook. I'd still hold that until the open questions land. ### What I verified Vale 3.12.0 locally. **Path scoping works.** The KubeRay rules fire on a file under `doc/source/cluster/kubernetes/` and stay silent on identical content at the repo root. **Current state of these pages** β€” 146 `KubeRay.Terms` alerts, which lines up with the casing scan: | Alert | Count | |---|---| | Use 'Ray autoscaler' instead of 'Ray Autoscaler' | 42 | | Use 'head Pod' instead of 'head pod' | 20 | | Use 'worker Pod' instead of 'worker pod' | 7 | | Use 'Ray job' instead of 'ray job' | 2 | | Use 'Ray Serve' instead of 'Ray serve' | 1 | | Use 'Ray cluster' instead of 'ray cluster' | 1 | | Use 'KubeRay' instead of 'Kuberay' | 1 | | Use 'GCS fault tolerance' instead of 'GCS FT' | 1 | The 42 matches the scan's count for these pages exactly, which is a useful cross-check on both. None of it is enforced, so nothing here has to be fixed in this PR. **No regression on the gated path.** Alert count on `doc/source/data/` goes 174 β†’ 173, deterministic across repeated runs in both states, with no new alerts. A `Vocab` list is global, so it can only widen what's accepted. The single alert that disappears is a `Google.Parens` hit in `concurrent-dataset-execution.md`; no term in the new vocabulary appears on that line, so I'm reporting that as observed rather than claiming I understand the mechanism. It's a reduction either way, so it can't break the gate. ### The vocabulary shadowing trap Worth knowing if you write Vale rules here: a `Vocab` accept entry shadows its token for **every** rule, case-insensitively. A substitution whose target appears in an accept list silently never fires, and nothing warns you. This bit the `Ray Autoscaler` swap, because `autoscaler` is accepted in `config/vocabularies/Data/accept.txt`. The fix is `vocab: false` on the rule, which opts out of the filtering without touching the Data vocabulary. The rule carries a comment saying so. Credit to Bugbot for catching it. I'd first shipped that swap commented out, having concluded the only way to enable it was dropping `autoscaler` from the Data vocabulary and accepting new spelling alerts on the gated path. That was wrong, and `vocab: false` avoids the tradeoff entirely. ## Added after review Two rules that review surfaced, both in the style-guide section plus the Vale ruleset. **Node stays lowercase.** Node is a Kubernetes API object, so \"capitalize Kubernetes API objects\" read literally tells contributors to write \"Kubernetes Node.\" These pages almost always mean the machine, and all 24 uses are correctly lowercase today, so a literal reading of the rule would have someone \"fix\" working prose into wrongness. Kubernetes applies the same two-part rule it applies elsewhere, capitalizing Node only for the resource, and the section now says so. **Don't write \"Ray Pod.\"** This answers the casing question raised in review, and the answer turned out to be neither casing. A Pod is a Kubernetes host that contains a Ray node plus any sidecars, so naming it after Ray overstates what it is. \"TPU Pod\" also already occupies the qualified-Pod form in these pages and means a group of TPU chips rather than a Kubernetes object. The term had no settled form to preserve either: across the 55 prose lines using it, casing split four ways at 20, 16, 16, and 15. Stating that alone would be a trap, because a bare \"Pod\" isn't always an adequate replacement, so the section gives the test that separates the cases: - Qualify when the qualifier distinguishes two referents. That's why \"Ray node\" against \"Kubernetes node\" is correct and should stay. - Don't qualify when it relabels one thing. - Keep the scope explicit where a page also describes Pods the RayCluster doesn't own, such as the operator, Redis, or a `curl` Pod, and where a nearby command filters by label. \"List all Pods\" and \"List the Ray cluster's Pods\" describe different commands and shouldn't read the same way. The cleanup is #65423, which removes all 89 uses from the docs and is reviewable on its own. This PR is the rule; that one is the sweep. Vale picks up four swaps for the retired forms. Plurals get their own entries, for the word-boundary reason @win5923 identified on the `head pod` swaps. Verified that they don't fire on \"Ray head Pod\" or \"Ray worker Pods\", and that the gated `doc/source/data/` path holds at 173 alerts. ## Not a duplicate Checked open PRs for this area before starting: ``` gh pr list --repo ray-project/ray --state open --search \"writing-style\" gh pr list --repo ray-project/ray --state open --search \"style guide in:title\" gh pr list --repo ray-project/ray --state open --search \"kuberay style\" ``` No open PR touches `doc/source/ray-contribute/writing-style.md` or proposes KubeRay style guidance. The `writing-style` matches were all false positives on `write_delta`. This extends the guide added in #64518. ## Testing ``` pre-commit run --files doc/source/ray-contribute/writing-style.md ``` Passes, but as a no-op on the Markdown: every hook reports \"no files to check,\" because no pre-commit hook applies to Markdown under `doc/source/` and Vale is scoped to `doc/source/data/`. Stating that plainly instead of claiming coverage it doesn't give. The Vale config files do have hook coverage, and those checks ran for real: ``` pre-commit run --files .vale.ini .vale/styles/KubeRay/Terms.yml .vale/styles/config/vocabularies/KubeRay/accept.txt ``` `trim trailing whitespace`, `fix end of files`, `check for added large files`, and `semgrep` all pass. Vale verification for the ruleset is in the section above: path scoping, the 31-alert current state, and the 174 β†’ 173 no-regression check on the gated path. Verified by hand: the new `(kubernetes-docs-style)=` anchor doesn't collide with any existing target in `doc/source/`, and the section uses the same MyST anchor form and Markdown pipe-table style already used elsewhere in this file. Rendering is verified by the Read the Docs preview build on this PR. No code, no API, no behavior change. The prose is a contributor guide; the Vale files add configuration without widening what CI enforces. ## AI assistance Written with AI assistance (Claude). I've reviewed every changed line, generated and inspected the casing counts myself, and can defend the change end to end. The scan scripts that produced the numbers live outside this repo; happy to share the methodology or re-run against a different corpus if a reviewer wants to check a figure. --- ### Update: the vocabulary now covers the ecosystem terms these pages use (555ebdad) Running `vale doc/source/cluster/kubernetes/`, which this PR's `.vale.ini` comment tells contributors to do, produced 215 `Vale.Spelling` alerts on correctly spelled terms. The accept list covered API objects, the custom resources, and project names, but not the wider ecosystem vocabulary the pages actually contain. Left as is, a contributor following the instruction learns to ignore Vale on exactly the paths this PR wants gated. Added: the ecosystem projects (Istio, Kiali, gVisor, Kata, Krew, Kustomize, Bottlerocket, Aliyun, GCSFuse, Promtail, Fluentd, Filebeat, Alertmanager), the Gateway API `HTTPRoute`, and the Kubernetes and infrastructure terms in use (cgroup, finalizer, kubeconfig, kubelet, namespaced, namespaces, toleration, upscaling, downscaling, vCPU, FQDN, AMI, subnet, syscall, hostname, cron, CA, autoscalers). **Excluded on purpose, since an accept list shouldn't launder a defect.** `CR` and `CRs`, 12 uses, because this PR's own guidance says to write \"custom resource\" and to define CustomResourceDefinition then use CRD. Real misspellings: `heterogenous`, `replicaset` for ReplicaSet, `Deepseek` for DeepSeek, `nodepool` for node pool. `gsutil`, which `gcloud storage` supersedes. Unbackticked code identifiers, which are a formatting defect rather than a vocabulary gap: `podSpec`, `podSelector`, `nodeSelector`, `jsonPointers`, `intervalSeconds`, `workerGroup`, `ignoreDifferences`, `HTTPMode`, `args`, `yaml`. And generic English that Vale's dictionary lacks, which belongs in the General vocabulary rather than a KubeRay-specific one. Three entries are case-tolerant (`[Tt]oleration(s)?`, `[Uu]pscaling`, `[Dd]ownscaling`). A lowercase-only entry makes `Vale.Terms` flag the legitimate sentence-start capitalization. **Verification**, deterministic across repeated runs: - `doc/source/cluster/kubernetes/`: 1329 alerts to 1178. `Vale.Spelling` drops from 215 to 72. `KubeRay.Terms` is unchanged at 160, confirming no swap got shadowed by a new accept entry, which is the trap the `Ray Autoscaler` comment in `Terms.yml` documents. - `doc/source/data/`, the only path the pre-commit hook gates: 173 to 170, with **no new alerts**. A `Vocab` is global, so it can only widen what's accepted. - One candidate entry was dropped for exactly that reason. A `runtime(s)?` entry made `Vale.Terms` flag \"Runtime\" in `doc/source/data/monitoring-your-workload.rst`, a new alert on the gated path. Four new `Vale.Terms` alerts do appear on the KubeRay paths, and all four are genuine casing defects in existing pages rather than noise: \"Kubelet\" twice in `configuring-ippr.md`, lowercase \"istio\" naming the project in `rayservice-incremental-upgrade.md`, and \"Cgroups\" in the `resource-isolation-with-writable-cgroups.md` H1, which is also title case. Those are content fixes for a separate change, not this one. AI assistance was used for this commit, including the Vale sweep and the before-and-after verification above.", "url": "https://github.com/ray-project/ray/pull/65239", "createdAt": "2026-08-05T19:50:41Z", "updatedAt": "2026-08-12T19:01:06Z", "timestamp": "2026-08-12T19:01:06Z", "metrics": { "reactions": 1, "comments": 2 }, "labels": [ "docs", "core", "go" ], "author": "dstrodtman", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:d38f908a75427091ba7e", "signalId": "github:ray-project/ray:pull_request:65015", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65015", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[core][dashboard] Return 4xx from node and actor detail APIs", "text": "## Description Makes three node/actor dashboard endpoints report client errors with 4xx status codes instead of 200/500. [#51417](https://github.com/ray-project/ray/pull/51417) added the `HTTPStatusCode` enum and the `rest_response(status_code=...)` plumbing, but adoption stalled after one endpoint. Across non-test dashboard source there are currently 28 `OK` and 13 `INTERNAL_ERROR` usages versus only 4 non-500 error codes, all confined to `state_head.py` / `state_api_utils.py`. Backend (`python/ray/dashboard/modules/node/node_head.py`): - `GET /nodes/{node_id}` returned **200 with a near-empty payload** for an unknown node ID, because `DataOrganizer.get_node_info` falls back to `.get(node_id, {})`. Now returns **404**. - `GET /logical/actors/{actor_id}` returned **200 with `detail: null`** for an unknown actor ID, because `get_actor_infos` maps unknown IDs to `None`. Now returns **404**. - `GET /nodes` with an unsupported `view` returned **500**. An unsupported query parameter is a client error, so it now returns **400**. ## Related issues Related to [#51442](https://github.com/ray-project/ray/issues/51442) (umbrella: revisit Ray dashboard API status codes). Since that issue is an umbrella, this PR covers `node_head.py` only, so each module can be reviewed independently. ## Additional information ### Frontend change is required, not incidental The node and actor detail pages surfaced these cases by reading `result === false` out of the **200 response body**. axios rejects on 4xx, so those branches go dead the moment the status code changes, and the pages would sit on \"Loading the node infos...\" indefinitely with `msg` never updated. The SWR fetchers in `useNodeDetail.ts` and `useActorDetail.ts` now catch the 404 and set the same message they set before, preserving existing UI behavior. Consumers of `useFetchActor` (the Serve pages) already use optional chaining on the result, so returning `undefined` there is unchanged behavior. ### Tests Added `test_node_api_status_codes` (single cluster, looping over all three error paths plus the three success paths β€” not parametrized, since `ray_start_with_dashboard` starts a fresh cluster per invocation) and `test_actor_not_found_status_code`. Test environment: because Ray's core is compiled, these were run against the Ray Windows nightly wheel built from `3fb63d965f` β€” the exact base commit of this branch β€” with the patched `node_head.py` overlaid, so the patch was the only delta. ``` pytest -v python/ray/dashboard/modules/node/tests/test_node.py -k test_node_api_status_codes 6 passed in 122.16s pytest -v python/ray/dashboard/modules/node/tests/test_actor.py -k test_actor_not_found_status_code 1 passed in 19.74s pytest -q python/ray/dashboard/modules/node/tests/test_actor.py 5 passed in 142.51s pytest -q python/ray/dashboard/modules/node/tests/test_node.py 7 passed, 4 skipped, 2 failed in 286.73s ``` The two failures in the full `test_node.py` run are `test_node_info` and `test_worker_pids_reported`. Both **fail identically against unpatched `node_head.py`** and are pre-existing Windows-environment issues, not regressions: worker cmdlines don't match `\"ray::Actor\"` on Windows, and runtime_env's uv virtualenv setup hits `WinError 206` (path length). Control run confirming the new tests are not vacuous β€” against unpatched code: - `/nodes/{unknown}` β†’ `assert 200 == 404` (fails) - `/nodes?view=unknown_view` β†’ `assert 500 == 400` (fails) - `/nodes` β†’ `assert 500 == 400` (fails) - the three success paths β†’ still pass - actor test β†’ `assert 200 == 404`, body `{\"result\": true, \"msg\": \"Actor details fetched.\", \"data\": {\"detail\": null}}` (fails) Frontend checks: ``` npx tsc --noEmit # clean, exit 0 npx eslint # clean npx prettier -c # clean ``` Python lint: `ruff check` clean; `black` leaves the changed files unmodified. Caveats worth reviewer attention: all of the above ran on Windows rather than Ray's Linux CI, and against a nightly wheel plus overlaid file rather than a from-source build. Linux CI remains the authoritative check. ### Not a duplicate No open PR references [#51442](https://github.com/ray-project/ray/issues/51442), and the issue timeline has no cross-referenced PRs. The issue has been assigned since 2025-03-18 with no linked work; I commented on it before starting ([comment](https://github.com/ray-project/ray/issues/51442#issuecomment-5084257594)). ### AI assistance AI assistance (Claude Code) was used to produce this change.", "url": "https://github.com/ray-project/ray/pull/65015", "createdAt": "2026-07-26T17:41:02Z", "updatedAt": "2026-08-12T18:52:12Z", "timestamp": "2026-08-12T18:52:12Z", "metrics": { "reactions": 1, "comments": 3 }, "labels": [ "dashboard", "core", "go", "community-contribution" ], "author": "chenyuan99", "state": "open", "assignees": [ "MortalHappiness" ], "change": "updated" } }, { "id": "event:72ba1ac17b9df14556a5", "signalId": "github:ray-project/ray:pull_request:65398", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65398", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[serve] Add separate fast path for unary gRPC direct ingress", "text": "## What Currently, a unary request returns its single response through an interposed async generator instead of returning directly. Instead we split the handler into three pieces to maintain the speed of the unary-unary hot path. 1. `_prepare_direct_ingress_grpc_request` does the setup every request needs regardless of shape. 2. `_direct_ingress_grpc_unary_response` handles calls that return exactly one response. Returns the response directly. 3. `_direct_ingress_grpc_streaming_response` stays a generator and handles calls that yield a stream. Setup and teardown still are shared. The only difference is how they consume the response. ## Why Unary gRPC throughput on `pytest_serve_throughput_optimized_microbenchmarks` dropped when the handlers were unified, and this recovers most of it. Numbers are average rps from the release test. | metric | before | now | with this change | |---|---:|---:|---:| | `grpc_avg_rps` | 4710.6 | 4601.1 | 4684.1 | | `grpc_100_max_ongoing_requests_avg_rps` | 4726.2 | 4641.3 | 4678.3 | | `grpc_800_max_ongoing_requests_avg_rps` | 4759.7 | 4675.7 | 4731.0 |", "url": "https://github.com/ray-project/ray/pull/65398", "createdAt": "2026-08-11T21:14:00Z", "updatedAt": "2026-08-12T18:24:46Z", "timestamp": "2026-08-12T18:24:46Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "serve", "go" ], "author": "YashwanthRanjanSingaravel", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:ee0832d361e583ebe013", "signalId": "github:ray-project/ray:pull_request:65419", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65419", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "fix(autoscaler): deduplicate cloud instances during termination", "text": "Fixes #65416 ## Summary Multiple instance-manager records can refer to the same cloud instance after a Raylet restarts in place. The cloud-instance subscriber previously forwarded every record's ID to the provider, so KubeRay could decrement the same worker more than once while building a scale request and fail with an assertion before submitting the deletion. This change de-duplicates cloud instance IDs at the existing subscriber/provider boundary while preserving first-seen order. Each cloud instance is therefore requested for termination at most once per update batch. ## Tests - Added a regression test covering duplicate termination events for one cloud instance and a distinct instance. - `python -m py_compile python/ray/autoscaler/v2/instance_manager/subscribers/cloud_instance_updater.py python/ray/autoscaler/v2/tests/test_subscribers.py` - `ruff check` on the two changed files - `black --check` on the two changed files - `git diff --check` - Executed a lightweight stub regression against the changed module; duplicate IDs produced exactly `[c1, c2]`. The focused pytest module could not run in this Windows environment because Ray has no installable Windows wheel and the source tree requires generated/runtime Ray modules. The KubeRay kind/Kubernetes end-to-end test was not run because no cluster tooling is available here. AI assistance was used for issue screening, investigation, implementation, and test drafting. Per the repository contribution policy, the human submitter must review and be able to defend every changed line.", "url": "https://github.com/ray-project/ray/pull/65419", "createdAt": "2026-08-12T08:37:09Z", "updatedAt": "2026-08-12T18:22:57Z", "timestamp": "2026-08-12T18:22:57Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "core", "community-contribution" ], "author": "mikemikimike", "state": "open", "assignees": [ "martinlhw" ], "change": "updated" } }, { "id": "event:c8aa237310e247bd439d", "signalId": "github:ray-project/ray:pull_request:65287", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65287", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[doc] Style pass on the JAX TPU profiling user guide", "text": "## Description A documentation style and grammar pass over the JAX TPU profiling user guide added in #64735, plus the TPU profiling section that PR added to `profiling.md`. #64735 merged before a docs-team review landed (@rueian had asked for one on the thread). This follow-up applies [the Ray documentation style guide](https://docs.ray.io/en/master/ray-contribute/writing-style.html) to the new page. The first commit is style and grammar only. The second commit carries two factual corrections that came out of review on this PR, both verified against the source and described below. ### Corrections (commit 2) - **Port discovery.** The page implied that passing a custom port to `init_jax_profiler()` means you must also pass `port=` to the dashboard endpoint. `init_jax_profiler()` registers whatever port it binds to under `jax_profiler_port::` in the GCS internal KV store (`python/ray/util/tpu.py`), and the `/worker/jax_profile` handler reads that key only when the request omits `port=`, returning a 400 when the lookup is empty (`python/ray/dashboard/modules/reporter/reporter_head.py`). So `port=` bypasses the lookup rather than being required for custom ports. The rewrite also documents `JAX_PROFILER_PORT`, which the same function reads and which the page didn't mention. - **Profile plugin name.** `tensorboard-plugin-profile` publishes on PyPI as the XProf Profiler Plugin ([openxla/xprof](https://github.com/openxla/xprof)), covering TPU, GPU, and CPU. Dropped the inaccurate \"Google TPU\" qualifier. ### Style changes (commit 1) - **Added the required `myst.html_meta` `description` front matter.** The guide asks for one on every page; the new page had none. - **Tagged the untagged fence.** The expected `ls` output block had no language; it's now `text`. - **Active voice where there's a nameable actor.** \"`tensorflow` is required by the Ray Dashboard `ReporterAgent`\" becomes the agent using it. \"The JAX trace file was captured and saved\" becomes the profiler capturing and saving it. - **Cut filler and hedging.** Dropped \"Note that.\" `ensure X is set to \"1\"` becomes `set X to \"1\"`. \"Ensure you have TensorBoard installed\" becomes \"Install TensorBoard.\" - **`Ray 2.57+` spelled out as `Ray 2.57 or later`**, per the rule against symbols standing in for words in prose. - **Consistent product naming.** \"the dashboard head,\" \"the dashboard profiling endpoints,\" and \"the head node dashboard port\" now all say **Ray Dashboard**. - **Split one long multi-clause sentence** in the profiler-initialization section, which carried a mid-sentence \"which defaults to port 9999\" clause. - **Added the missing blank line between each H3 and its lead-in prose.** The H2s in the file already had them. - **`profiling.md`:** rewrote the new TPU profiling paragraph, which used the \"Ray provides integration for\" framing the guide steers away from. Four H3 headings changed wording slightly, so their generated anchors change. The page is a day old and nothing links to those anchors yet. ### Left alone on purpose The `` placeholders. The style guide's example uses lowercase-hyphenated names inside angle brackets, but a survey of `doc/source` shows uppercase-in-brackets is the established convention across Ray docs. Not worth churning this page against the rest. ## Questions for the author and reviewers Two technical points I noticed but didn't touch, since they're outside a style reviewer's scope: 1. **The `ls` command and its expected output don't line up.** The guide runs: ```bash kubectl exec -it -c ray-worker -- ls -la /tmp/ray/session_latest/logs/profiles ``` and shows the output as: ```text -rw-r--r-- 1 ray users 79526813 Jun 2 15:26 /tmp/ray/session_latest/logs/profiles/plugins/profile/2026_06_02_15_26_15/localhost_9999.xplane.pb ``` A plain `ls -la` on that directory would list its immediate children (`plugins`), not a nested file with its full path. Was the sample output captured with `ls -laR`, a `find`, or an `ls` against the full nested path? Happy to send the fix once you confirm which. 2. **`-it` on that `exec` looks unnecessary** for a non-interactive `ls`. It also fails in contexts without a TTY, such as a CI script. Worth dropping to just `kubectl exec -c ray-worker -- ls ...`? ## Not a duplicate No open PR or issue targets `doc/source/ray-observability/user-guides/jax-tpu-profiling.md`. Checked with `gh pr list --repo ray-project/ray --state open --search \"jax-tpu-profiling in:title,body\"` and `gh search prs --repo ray-project/ray --state open \"jax tpu profiling\"`; both empty. All of the review feedback on #64735 (the hardcoded personal paths and registry names from the Gemini review, @andrewsykim's questions on the custom image, the sample RayJob, and pod scoping) was addressed by the author before merge, so nothing here reopens a settled thread. ## Verification - `vale --config=.vale.ini --output=line doc/source/ray-observability/user-guides/jax-tpu-profiling.md` β€” clean apart from `Google.Acronyms` on \"JAX\" and the two `Google.Passive` hits this PR fixes. Advisory only: per `.vale.ini`, Vale is scoped to the Ray Data docs and the example gallery, so it doesn't gate `ray-observability`. - `pre-commit run --files ` β€” no hook covers Markdown under `doc/source/`, so every check reports \"no files to check.\" - The two corrections in commit 2 were read out of `python/ray/util/tpu.py` and `python/ray/dashboard/modules/reporter/reporter_head.py`, and the plugin's identity from the PyPI metadata for `tensorboard-plugin-profile`. No behavior claim here comes from the page's own prose. - No code changed, so no unit tests apply. The Sphinx build is the real gate. Relying on the Read the Docs preview build on this PR to confirm the new front matter parses and the `{ref}` targets still resolve. AI assistance (Claude Code) was used to draft this change. I reviewed every changed line and I stand behind it.", "url": "https://github.com/ray-project/ray/pull/65287", "createdAt": "2026-08-07T16:02:21Z", "updatedAt": "2026-08-12T18:07:35Z", "timestamp": "2026-08-12T18:07:35Z", "metrics": { "reactions": 1, "comments": 2 }, "labels": [ "docs", "core", "go", "docs-go" ], "author": "dstrodtman", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:963aa67f826feda2521d", "signalId": "github:ray-project/ray:pull_request:65388", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65388", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[doc] State that the ray-ml images are deprecated, and stop recommending them", "text": "## Why Ray announced the `rayproject/ray-ml` deprecation in #46378 and, in a follow-up comment on that issue, stopped publishing the images entirely as of Ray 2.50 β€” \"not even with `-deprecated` tags.\" Nothing a reader can see says so. That issue went stale and was closed, and the deprecation never reached the docs. The gap costs the reader real time, because the frozen tags don't fail loudly. Verified against Docker Hub on 2026-08-11: | Tag | Status | |---|---| | `ray-ml:latest`, `ray-ml:latest-gpu` | 200, frozen at the `2.30.0` digest, last pushed 2024-06-20 | | `ray-ml:2.31.0` … `2.57.0` | 404 | | `ray-ml` newest push of any kind | `nightly-*` on 2025-10-01 | A 404 teaches the reader something. A 200 serving a two-year-old Ray teaches them nothing. `ray-overview/installation.rst`, the canonical image reference, documents `rayproject/ray` only and has never mentioned `ray-ml`, so a reader arriving with an existing `ray-ml` reference finds no explanation for its absence. ## What changed **`ray-overview/installation.rst`** β€” adds the deprecation statement to the `docker-images` section, where a reader looking for image guidance lands. States what replaces it: build on a `rayproject/ray` base, or declare the packages in a runtime environment. Documents the constraint file every `rayproject/ray` image ships at `/home/ray/requirements_compiled.txt`, which installs the exact library version Ray tested against for that release. Links #46378 for the rationale rather than restating it. **`cluster/kubernetes/user-guides/gpu.rst`** β€” recommended `ray-ml:2.6.3-gpu` as \"ideal for running GPU-based ML workloads,\" in both prose and a config snippet. Now recommends `ray:2.57.0-gpu` and says plainly that the image doesn't include TensorFlow or PyTorch. The page already pointed at custom images and runtime environments as the way to add dependencies; that guidance just contradicted the recommendation above it. **`cluster/vms/configs/xgboost-benchmark.yaml`** β€” ran the XGBoostTrainer benchmark on `ray-ml:2.0.0`. Now uses `ray:2.57.0`, plus a `setup_commands` entry that installs `xgboost` against the image's constraint file. `xgboost` appears nowhere in `python/setup.py`, so `ray[all]` doesn't include it and a bare image swap would have broken the benchmark. ## Deliberately not in this PR Four other `ray-ml` references remain in `doc/source`, each blocked on something outside this change: - `cluster/vms/references/ray-cluster-configuration.rst` and `cluster/kubernetes/user-guides/config.md` β€” already covered by #65339 and #65336 respectively. Excluded to avoid conflicts. - `cluster/kubernetes/examples/mobilenet-rayservice.md`, `stable-diffusion-rayservice.md`, and `rayjob-batch-inference-example.md` β€” these describe sample manifests that live in `ray-project/kuberay`, and the prose accurately reports what those manifests do. Rewriting the prose alone would put the docs at odds with the file the reader applies. That last group needs a heads-up, because those three examples are broken today rather than merely stale. On both kuberay `v1.6.0` and `master`, `ray-service.mobilenet.yaml`, `ray-service.stable-diffusion.yaml`, and `ray-job.batch-inference.yaml` all pin `rayproject/ray-ml:2.46.0.0e19ea-py39-{cpu,gpu}`, which 404s. Following any of those three examples ends in `ImagePullBackOff`. Fixing them isn't a tag bump either: `mobilenet` needs `tensorflow` and the other two need `torch`, all of which came from the `ray-ml` image, so each manifest needs `runtime_env` additions. Happy to take that on in kuberay if maintainers agree on the approach, or to defer to whoever owns those samples. Separately, `ray-job.batch-inference.yaml`'s image tag is already out of sync with the snippet quoted in `rayjob-batch-inference-example.md` (`2.6.3-gpu` in the docs vs. `2.46.0.0e19ea-py39-gpu` in the manifest), independent of the deprecation. ## One thing outside docs scope The frozen `ray-ml:latest` and `latest-gpu` tags are the actual trap. A deprecation notice helps readers who come to the docs; removing or re-tagging those two on Docker Hub would help everyone with an existing Dockerfile or manifest that references them. Not something I can do from a docs PR, but worth a decision from whoever owns the registry. ## Testing Docs-only prose, RST, and cluster-config YAML; no code paths touched. - `pre-commit run --files ` β€” every hook reports \"no files to check.\" No pre-commit hook currently covers prose under `doc/source`. - Every replacement tag verified live against the Docker Hub registry API: `rayproject/ray` `2.57.0`, `2.57.0-gpu`, `2.57.0-cpu`, `2.57.0-py311-gpu`, `2.57.0-py311-cpu` all return 200, pushed 2026-08-11. - `xgboost-benchmark.yaml` parses under `yaml.safe_load`. - `xgboost==2.1.0` confirmed pinned in `python/requirements_compiled.txt`, so the constraint comment in that file is accurate. - Confirmed `setup_commands` run inside the container when `docker.image` is set: `DockerCommandRunner.run` with the default `run_env=\"auto\"` routes any command not beginning with `docker` through `docker exec`. - Confirmed the constraint file's in-image path: `docker/base-deps/Dockerfile` copies it to `/home/ray/requirements_compiled.txt`. - Cross-reference targets confirmed present: `runtime-environments` (`ray-core/handling-dependencies.rst:52`) and `docker-images` (`ray-overview/installation.rst:313`). - `installation.rst` parsed with docutils; the only diagnostics in the changed range are the expected Sphinx-only `:ref:` role. I did not run the 100Gi XGBoost benchmark or apply the KubeRay snippet, which need cloud resources I don't have here. ## AI assistance AI assistance (Claude Code) was used to inventory the `ray-ml` references and draft these edits. I reviewed every changed line and ran the verification above.", "url": "https://github.com/ray-project/ray/pull/65388", "createdAt": "2026-08-11T19:27:45Z", "updatedAt": "2026-08-12T18:07:21Z", "timestamp": "2026-08-12T18:07:21Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "docs", "core", "go" ], "author": "dstrodtman", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:50c1139ffd4a8db2f34a", "signalId": "github:ray-project/ray:pull_request:65132", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65132", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[core][gcs][Actice-Passive] Phase 2.1 Add Active-Passive leader-election interface (protocol, status, client cache)", "text": "## Description This is the foundational, interface-only PR for the Active-Passive GCS feature (split from the larger #64422). It introduces the wire-protocol field, status code, config flag, and client-side leadership-caching plumbing β€” all feature-gated behind `LEADER_ELECT` / `RAY_LEADER_ELECT`, which **defaults to OFF**. No server-side passive behavior is included here; this PR is pure additive scaffolding that later PRs build on. ## Issue https://github.com/ray-project/ray/issues/63643 ## High-level changes | Layer | Change | | :--- | :--- | | **Protocol** | Add `optional bool is_leader = 4` to `CheckAliveReply`. Marked `optional` so a new client can distinguish \"field absent\" (an old GCS during a rolling upgrade) from an explicit `false`. | | **Status** | New `StatusCode::GcsPassive` (38) with `Status::GcsPassive()` and `IsGcsPassive()`, registered in the code↔string maps. No code path returns it yet. | | **Config** | New `RAY_CONFIG(bool, LEADER_ELECT, false)` (C++) and matching `ray_constants.RAY_LEADER_ELECT` (Python), kept in sync via the `RAY_LEADER_ELECT` env var. | | **C++ client** | `NodeInfoAccessor` caches an `atomic is_gcs_leader_`, updated from each `CheckAlive` reply (`has_is_leader() ? is_leader() : true` β€” absence is treated as the legacy leader). Exposes `IsGcsLeader()`. When `LEADER_ELECT` is off, it initializes to `true` and never changes (legacy behavior). The test-only default constructor sets `client_impl_ = nullptr`. | | **Python bindings** | `GcsClient.is_gcs_leader_local()` (reads the cache) and `is_gcs_leader()` (triggers a `CheckAlive`, then reads the cache). Both return `True` when `RAY_LEADER_ELECT` is off. `check_status` maps `GcsPassive β†’ RpcError(UNAVAILABLE)`. | ## Behavior change / backward compatibility - **With `LEADER_ELECT` off (the default): zero observable change.** `is_gcs_leader()` / `is_gcs_leader_local()` always return `True`; the `GcsPassive` status code and the `is_leader` field are never produced. - **Wire compatibility:** the new proto field is `optional` and defaults to absent, so old/new clients and servers interoperate during rolling upgrades (absence = legacy leader). - **No public API signatures change**; all additions are new symbols. ## Test plan **Unit (C++)** β€” accessor_test.cc - `TestIsGcsLeaderCaching` β€” cache defaults to `true`; `store`/`load` behave correctly. - `TestLeaderElectionDisabledByDefault` β€” with `LEADER_ELECT` off, the accessor always reports itself as leader regardless of any `is_leader` value (rolling-upgrade / legacy guard). - Command: `bazel test //src/ray/gcs_rpc_client/tests:accessor_test` β†’ **PASS** **Integration (Python)** β€” test_gcs_utils.py - `test_is_gcs_leader_defaults` β€” default path: both methods return `True`; and in a spawned subprocess with `RAY_LEADER_ELECT=true`, `is_gcs_leader_local()` starts as `False` (client-side cache init). The cross-layer server assertion is intentionally deferred to a later PR. - Command: `python -m pytest python/ray/tests/test_gcs_utils.py::test_is_gcs_leader_defaults` β†’ **PASS**", "url": "https://github.com/ray-project/ray/pull/65132", "createdAt": "2026-07-30T17:19:02Z", "updatedAt": "2026-08-12T16:47:45Z", "timestamp": "2026-08-12T16:47:45Z", "metrics": { "reactions": 1, "comments": 2 }, "labels": [ "core", "go", "community-contribution" ], "author": "YoyinZyc", "state": "closed", "assignees": [ "rueian" ], "change": "updated" } }, { "id": "event:706de33a18e758c2676d", "signalId": "github:ray-project/ray:pull_request:65432", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65432", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[serve] test_cli deflake by adding explicit wait timeouts and stopping tracing config leak", "text": "Two independent cleanups to `python/ray/serve/tests/test_cli.py`. **1. Explicit wait timeouts.** 17 of the file's 41 `wait_for_condition` calls had an effective timeout of 10s, and 12 of those were bare calls inheriting the default rather than stating one. Every call now carries an explicit timeout, and existing values are doubled, matching the convention already used elsewhere in this file. **2. Stop global tracing config leaking across tests.** `test_deploy_config_tracing_config_declarative_flow` deploys a config with `tracing_config: {enabled: true, sampling_ratio: 1.0}`. `ServeController.deploy_config` only ever *sets* `global_tracing_config` β€” there is no reset path β€” and the `serve_instance` fixture is session-scoped, its teardown only deleting apps. Every subsequent test in the file therefore ran with tracing enabled at ratio 1.0, exporting a span per request. The test now resets tracing to disabled in its teardown. ## Testing - 114 unloaded full-file runs, all green. - Under induced CPU contention, the timeout change fixed 3 previously-failing tests (`test_deploy_multi_app_basic`, `test_deploy_multi_app_builder_with_args`, `test_status_multi_app`); tests that wait for a terminal `DEPLOY_FAILED` state are unaffected, as that duration is governed by replica retry exhaustion rather than the wait timeout. - Tracing leak fix verified directly: a probe test in the same session now observes `enabled=False` where it previously saw `enabled=True`.", "url": "https://github.com/ray-project/ray/pull/65432", "createdAt": "2026-08-12T16:34:25Z", "updatedAt": "2026-08-12T16:39:56Z", "timestamp": "2026-08-12T16:39:56Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "serve", "go" ], "author": "johntaylor-cell", "state": "open", "assignees": [ "johntaylor-cell" ], "change": "updated" } }, { "id": "event:157eef1c93436ff59b68", "signalId": "github:ray-project/ray:pull_request:65368", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65368", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[ci] Remove the banned-words lint check in favor of Vale", "text": "## Summary Removes the `banned_words` lint check and its Buildkite matrix entry. `ci/lint/check-banned-words.sh` is a `git grep` for three fixed strings across `*.py`, `*.rst`, `*.md`, and `*.ipynb`. Vale already flags all three, at `error` severity, with better messages. This PR deletes the redundant check. ## The three terms, and what catches them today The check enforces exactly three case-sensitive strings. Every one is already covered by a Vale rule in this repository, with no configuration change required by this PR: | Term removed from the grep | Correct form | Vale rule that catches it | Source | | --- | --- | --- | --- | | `Kuberay` | `KubeRay` | `Google.WordList` (error) | `\"[Kk]ube[Rr]ay\": KubeRay` in `.vale/styles/Google/WordList.yml` | | `RLLib` | `RLlib` | `Vale.Terms` (error) | `(RL\\|rl)lib` in `.vale/styles/config/vocabularies/RLlib/accept.txt` | | `Rllib` | `RLlib` | `Vale.Terms` (error) | same entry | Verified against a fixture page containing both the correct and incorrect forms: ``` t.md:5:18:Google.WordList:Use 'KubeRay' instead of 'Kuberay'. t.md:5:27:Vale.Terms:Use 'RLlib' instead of 'RLLib'. t.md:5:38:Vale.Terms:Use 'RLlib' instead of 'Rllib'. ``` The correct spellings `KubeRay` and `RLlib` produce no alerts, so the rules discriminate on capitalization the same way the case-sensitive grep did. Vale's messages name the correct form; the grep only says a misspelled word was found and asks the author to go fix it. The `Vale.Terms` message improves further on Vale 3.17.1 (#65375), which renders the vocabulary entry as `RLlib` rather than echoing the raw `(RL|rl)lib` pattern. Because coverage already exists, this PR deliberately adds nothing to the `reject.txt` vocabulary. A reject entry would fire `Vale.Avoid` *in addition to* the rule above, double-reporting the same token with a strictly less useful message (\"Avoid using 'Kuberay'\" versus \"Use 'KubeRay' instead of 'Kuberay'\"). ## What the check costs The step carries the `always` tag in `.buildkite/lint.rayci.yml`, so it runs on every pull request, including documentation-only ones. Measured across 89 `lint: banned_words` jobs from recent passing `premerge` builds: | Measure | p50 | p90 | Range | | --- | --- | --- | --- | | Job run time | 111s | 116s | 108s – 642s | | Queue wait before start | 42s | 51s | 8s – 146s | | Wall clock, queue plus run | 153s | β€” | β€” | Nearly all of that is fixed overhead rather than work. Taking build 71819 as a representative sample, the job started at 10:59:01.6 and finished at 11:00:52.8, and `Checking for common mis-spellings...` printed at 11:00:52. So roughly 110 seconds went to the repository clone and the forge image pull, and the `git grep` itself accounted for under a second of it. At 391 `premerge` builds over the last seven days, the step consumes on the order of 12 agent-hours per week to run three sub-second greps. ## Why not keep it anyway Beyond the cost, the grep can't express the rule properly: - **No scoping.** Vale rules apply per path glob. The grep is repo-wide, with no way to hold a stricter bar in one tree than another. - **No exceptions.** Vale has `TokenIgnores`, `BlockIgnores`, and vocabularies. The grep has none, so a page that needs to *name* an incorrect spelling can't. That's not hypothetical: #65239 adds a Kubernetes and KubeRay terminology table to the style guide, and the row telling contributors to write `KubeRay` rather than `Kuberay` fails the check. There is no way to satisfy both. - **No severity.** Vale has `suggestion`, `warning`, and `error`. The grep is a hard CI failure or nothing. - **No extensibility.** Adding a term means editing a shell string that ships its own bespoke failure output. Adding a term to Vale means one line in a vocabulary or `swap` map, and it inherits the shared message, link, and reporting. ## The remaining gap, stated plainly The gap this PR opens is **scope, not rule coverage**. Vale runs today only against `doc/source/data/` and `doc/source/ray-overview/examples`, both in CI and in the pre-commit hook. Within those trees the three spellings are enforced today and stay enforced after this PR. Outside them, the repo-wide grep was the only thing checking, and nothing replaces it until Vale's scope widens. Adding the terms to a Vale vocabulary would not close that gap, because the limit is which paths Vale is pointed at, not which rules it has. The docs team is widening that scope over the coming weeks. ## Related changes - Drops the `banned_words` entry from the always-on lint matrix in `.buildkite/lint.rayci.yml`. - Removes the local-run instructions from `doc/source/ray-contribute/getting-involved.md`. - Updates the prose-only lint step count in `doc/source/ray-contribute/ci.md` from four to three. ## Checks - [x] I've signed off every commit (`git commit -s`). - [x] I've made sure the tests are passing. - [x] Release tests and any other tests I've added and changed pass. - [x] This PR is not tested :(", "url": "https://github.com/ray-project/ray/pull/65368", "createdAt": "2026-08-11T12:30:49Z", "updatedAt": "2026-08-12T16:33:29Z", "timestamp": "2026-08-12T16:33:29Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "docs", "core", "devprod", "go" ], "author": "dstrodtman", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:c19f78ebfbb18423db3e", "signalId": "github:ray-project/ray:pull_request:65389", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65389", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[Data] OpTask._cancel never passes force=True", "text": "## Description Current behavior of `OpTask._cancel` never passes `force=True` to `ray.cancel`. It is because whether the task is an actor task is determined by checking if the actor id is nil, which is false for both normal tasks and actor tasks (refer to the issue). Slicing the hex method was withdrawn through discussions with maintainers. Instead, removed the check and lets Ray Core classify the task, with a fallback to force=False when Core rejects it for an actor task. ## Related issues Closes #65280 ## Additional information Two tests added in `TestOpTaskCancel` (`python/ray/data/tests/test_streaming_executor.py`) - One test for a normal task - One test for an actor task", "url": "https://github.com/ray-project/ray/pull/65389", "createdAt": "2026-08-11T19:46:54Z", "updatedAt": "2026-08-12T16:02:40Z", "timestamp": "2026-08-12T16:02:40Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "data", "community-contribution" ], "author": "Hyunoh-Yeo", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:78a6fcd447c12919427b", "signalId": "github:ray-project/ray:pull_request:65426", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65426", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[docs] vendor the KubeRay CRD API reference into the Ray docs", "text": "## Why are these changes needed? The KubeRay CRD API reference β€” the field-level reference for `RayCluster`, `RayJob`, `RayService`, and `RayCronJob` β€” is published only on the MkDocs site at `ray-project.github.io/kuberay`. That site has two problems that compound each other: - Its `mkdocs.yml` `nav:` block contains only `index.md`. The API reference builds and serves but appears nowhere in site navigation, so it's reachable only by direct URL, site search, or a search engine. - Its landing page states that documentation has moved to `ray-project/ray`. Meanwhile `doc/source/cluster/kubernetes/references.md` here is a ~14-line stub whose entire substance is an outbound link to that page. So Ray's documented answer for \"what fields does a RayCluster support?\" is a link off-site to a nav-hidden page on a site that says it holds no documentation. It's also the most actively updated file in the KubeRay `docs/` tree β€” recent CRD features (`AuthOptions`, `GcsEmbeddedStorage`, `NetworkPolicyRules`, `IngressOptions`, `HistoryServerOptions`, `RayCronJob`) all extend it. This vendors the generated reference into the Ray docs and repoints the inbound links at it. ## What changed | File | Change | | --- | --- | | `doc/source/cluster/kubernetes/references/api.md` | **New.** Vendored generated reference, `ray.io/v1` only. | | `doc/source/cluster/kubernetes/references.md` | Toctree entry; the outbound link becomes a `{ref}`; added a note that v1alpha1 is deprecated. | | `doc/source/cluster/kubernetes/getting-started/rayjob-quick-start.md` | Outbound link β†’ `{ref}`. | | `doc/source/cluster/kubernetes/user-guides/gcp-gke-tpu-cluster.md` | Outbound link β†’ the operator installation page (see below). | | `.github/CODEOWNERS` | Entry for the generated file. | ### How the source artifact is produced `docs/reference/api.md` in `ray-project/kuberay` is a committed build artifact, not a hand-written page. `make api-docs` runs `elastic/crd-ref-docs` (pinned v0.2.0) over the Go CRD types in `ray-operator/apis/ray/`. The `ray-operator-verify-api-docs` job in `.github/workflows/consistency-check.yaml` regenerates and diffs it on every push and PR to `master` and `release-*`, failing on drift. That CI job is what makes a vendored copy viable: freshness is already enforced upstream, so the copy has a single well-defined source of truth. The file carries a `GENERATED FILE -- DO NOT EDIT` header naming the upstream commit it came from, with a permalink. ### Why `ray.io/v1` only The upstream artifact documents both `ray.io/v1` and `ray.io/v1alpha1`. This copy carries only v1, for two reasons. **v1alpha1 is deprecated and on a removal runway.** It hasn't been the storage version since v1.0 (October 2023), receives no new fields, and ray-project/kuberay#5124 tracks its removal. Every CRD feature added since December 2023 is v1-only. **Carrying it would import a silent correctness bug.** `crd-ref-docs` derives anchors from the type name alone, and eleven type names are defined in both API versions (`RayClusterSpec`, `WorkerGroupSpec`, `RayJob`, `RayService`, `AutoscalerOptions`, ...). Only the v1 heading owns the slug, so every v1alpha1 cross-link resolves to the v1 definition β€” with no warning at generation time. The definitions differ substantially (`RayClusterSpec` has 14 fields in v1, 7 in v1alpha1), so a reader following a v1alpha1 link lands on a table advertising fields that don't exist in that version. Filed upstream as ray-project/kuberay#5091. Dropping the section resolves the collision rather than papering over it. The stripping step is written to become a silent no-op once upstream drops v1alpha1 from generation. ### The gcp-gke-tpu-cluster.md link This one pointed at the MkDocs `/deploy/helm/` page, not the API reference. That page documents Helm *chart values*, while the sentence introducing it is about *installing* KubeRay. It's repointed at `kuberay-operator-installation.md`, which is the maintained page for what the sentence actually promises. The `helm repo add kuberay https://ray-project.github.io/kuberay-helm/` commands elsewhere in the docs are untouched β€” that's the Helm chart repository, a different host from the docs site. ## Checks **Anchor resolution.** The vendored page has 46 h1–h4 headings producing 46 distinct slugs (zero collisions, confirming the v1-only decision eliminates that class) and 93 in-page anchor links across 43 distinct targets, all resolving. **Build.** Built the page in a minimal Sphinx project against the pinned docs versions β€” sphinx 8.2.3, myst-parser 5.1.0 β€” with `myst_heading_anchors = 4`, `nitpicky = True`, and `-W --keep-going`: ``` build succeeded. exit 0 ``` Zero warnings. Inspected the rendered HTML: all 46 internal `href`s have a matching `id`, and no link resolves to a docutils fallback `idN`. This page depends on #65240 (h4 heading anchors), which has merged. `crd-ref-docs` renders every CRD type as an h4 and cross-links types by anchor, so the page could not build under the previous `myst_heading_anchors = 3`. A full `make rtd` build was not run locally; the docs CI build on this PR is the check for that. ## Follow-ups (not in this PR) - A sync job to keep the vendored copy current. It diffs the processed upstream output against the copy committed here β€” not against the previous upstream version β€” so it's idempotent and self-healing: an unmerged sync PR gets updated rather than duplicated, and a hand-edit here gets corrected on the next run. - Several other pages on the MkDocs site have no equivalent in the Ray docs β€” the Helm chart values tables, additional `rayStartParams` coverage (`--no-monitor` and the autoscaler READONLY rationale, `--block`, `--address` FQDN construction), and the image tagging and registry scheme. Being handled separately. ## Related issue number Depends on #65240 (merged). Related upstream: ray-project/kuberay#5091, ray-project/kuberay#5124. ## AI assistance AI assistance (Claude Code) was used for the anchor-collision analysis, the build verification, the post-processing script, and drafting this description. I reviewed the vendored content and the link changes and can defend both.", "url": "https://github.com/ray-project/ray/pull/65426", "createdAt": "2026-08-12T14:46:49Z", "updatedAt": "2026-08-12T14:55:01Z", "timestamp": "2026-08-12T14:55:01Z", "metrics": { "reactions": 1, "comments": 1 }, "labels": [], "author": "dstrodtman", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:7be2f315e6ab84b2611f", "signalId": "github:ray-project/ray:issue:65427", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:issue:65427", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "issue", "title": "[Docs] Run pre-commit hooks on doc/source: a per-directory ratchet", "text": "## Why reopen this [#42587](https://github.com/ray-project/ray/issues/42587) asked for pre-commit hooks to run on `doc/`. It was closed as completed, but the configuration change it asked for was never made. I closed it, and that was my error. What actually happened: six PRs in Aug–Sep 2024 ([#47004](https://github.com/ray-project/ray/pull/47004), [#47005](https://github.com/ray-project/ray/pull/47005), [#47107](https://github.com/ray-project/ray/pull/47107), [#47108](https://github.com/ray-project/ray/pull/47108), [#47341](https://github.com/ray-project/ray/pull/47341), [#47342](https://github.com/ray-project/ray/pull/47342)) ran one-shot bulk reformats over `ray-core/`, `rllib/`, `train/`, `tune/`, `data/`, and `cluster/` by temporarily lifting the global exclude. None of them touched `.pre-commit-config.yaml`. So the trees were cleaned once and then left ungated, and they've drifted since. I'd like agreement on an approach before sending more configuration PRs, rather than landing them one at a time and discovering on the fifth that the direction is wrong. ## Current state `.pre-commit-config.yaml` has a top-level `exclude` covering `doc/source/`. pre-commit ANDs that global `exclude` with each hook's own `files` pattern, so any hook scoped inside `doc/source/` matches nothing at all. That has produced silently dead hooks rather than merely absent ones, which is the part worth flagging: - The `vale` hook declared `files: ^doc/source/data/.*\\.(md|rst)$` and matched zero files for its entire life. Fix open for review in [#65375](https://github.com/ray-project/ray/pull/65375). - The `semgrep` hook's `code-block-python` rule names five `doc/source` paths in `paths.include` and has never run on any of them. There are 31 occurrences across 16 files that the rule was written to prevent. Not yet fixed. - `rst-directive-colons` and `rst-inline-touching-normal` from `pygrep-hooks` are configured, apply to `.rst` by default, and are skipped for all 362 `.rst` files under `doc/source`. A dead hook is worse than a missing one: `pre-commit run --all-files` comes back green and reads as coverage. Coverage today, counting `.md` and `.rst` under `doc/source`: | | Files | | --- | --- | | `.rst` | 362 | | `.md` | 229 | | Total | 591 | | Gated by CI Vale today (`doc/source/data` + `ray-overview/examples`) | 49 | | Covered by the pre-commit Vale hook, once #65375 lands | 48 | Note those are two separate narrow scopes that already disagree with each other, which is its own drift problem and the first item in the proposal below. ## Evidence the gate is worth having `doc/source/data` is the one directory Vale has gated in CI. Running Vale over all of `doc/source` on the repo's own configuration, error-level alerts per file: | Directory | Files | Errors | Errors/file | | --- | --- | --- | --- | | `data` (gated) | 27 | 11 | **0.4** | | `train` | 37 | 330 | 8.9 | | `cluster` | 104 | 1278 | 12.3 | | `ray-core` | 81 | 1212 | 15.0 | | `serve` | 79 | 1128 | 14.3 | | `tune` | 42 | 730 | 17.4 | | `ray-contribute` | 18 | 347 | 19.3 | The gated directory is roughly thirty times cleaner than comparable ungated ones. The gate works; it just isn't pointed at much. (Measured with a newer local Vale than CI's pin, so treat these as magnitudes for planning rather than targets.) ## Proposal Ratchet per directory, and never add a path to the gate that isn't already clean: 1. **Fix the invocation drift first.** `ci/lint/check-documentation-style.sh` lints `doc/source/data` plus `doc/source/ray-overview/examples`; the pre-commit hook covers only the first. Two hardcoded lists that already disagree. Give them one shared definition so every later step edits one place. 2. **Clear the vocabulary backlog.** `Vale.Spelling` is the largest single bucket and is mostly accept-list entries rather than prose defects. Repo-wide, no gate change, no rewriting. 3. **Extend directory by directory**, clearing that directory's errors in the same PR or an immediately preceding one, so `master` is never red. 4. **Order by ownership.** The docs-team-owned directories are small and need no other team's sign-off: `ray-contribute`, `ray-overview`, `ray-security`, `ray-governance`, `ray-air`, `ray-more-libs`, `_templates`, `templates`. Roughly 54 files. The large library directories need the owning team's buy-in on a prose sweep and should be separate per-team PRs. The already-configured hooks (whitespace, end-of-file, the two rST checks) can ride along per directory at no marginal cost, since they're in the config already. ## Questions for maintainers 1. **Ratchet or baseline?** Widening the invocation directory by directory keeps `master` green by construction and needs no new tooling. The alternative is opening all of `doc/source` at once behind a baseline file that grandfathers existing alerts, which gets new-file coverage immediately but adds a file that rots. I lean ratchet. Happy to go the other way if you'd rather have coverage sooner. 2. **Is the `code-block-python` semgrep rule still wanted for docs?** It was written deliberately and points at a real contributor guide, but it's been dormant long enough that the answer isn't obvious. Converting the 31 occurrences to `testcode` makes them execute in CI, so some will need fixtures. Worth sizing per path if yes. 3. **Notebook policy.** `doc/source` ships 59 executed `.ipynb` files with embedded outputs. That complicates any strip-on-commit hook. Is there an existing preference here? 4. **PR granularity.** The 2024 thread preferred small subtree-scoped PRs to limit CODEOWNERS blast radius. Assuming that still holds unless you say otherwise. ## Related, all currently open None of these are blocked on this issue, and I'm not asking anyone to hold them for it. They're the first increment of the ratchet above, and reviewing them doesn't commit anyone to the wider plan. - [#65375](https://github.com/ray-project/ray/pull/65375) β€” makes the Vale hook actually run, and upgrades Vale 3.4.1 β†’ 3.17.1. CI green, awaiting review. This is effectively step 1 of the proposal, scoped to one directory. - [#65368](https://github.com/ray-project/ray/pull/65368) β€” removes the `banned_words` grep in favor of Vale. Awaiting review. - [#65425](https://github.com/ray-project/ray/pull/65425) β€” whitespace cleanup on `doc/source/data`, the fallout from #65375. Draft, stacked on #65375. If the answer to question 1 is \"baseline, not ratchet,\" #65375 and #65368 still stand on their own and the change would only affect what comes after them.", "url": "https://github.com/ray-project/ray/issues/65427", "createdAt": "2026-08-12T14:48:15Z", "updatedAt": "2026-08-12T14:48:15Z", "timestamp": "2026-08-12T14:48:15Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "docs" ], "author": "dstrodtman", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:7247bd5b249f8a9ad685", "signalId": "github:ray-project/ray:pull_request:65375", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65375", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[CI] Make the Vale pre-commit hook run, and upgrade Vale to 3.17.1", "text": "## Description Two related fixes to the Vale setup. ### 1. The `vale` pre-commit hook has never run The top-level `exclude` in `.pre-commit-config.yaml` lists `doc/source/`, and pre-commit ANDs that global `exclude` with each hook's own `files` pattern. The Vale hook's pattern, `^doc/source/data/.*\\.(md|rst)$`, lives entirely inside `doc/source/`, so the two can never both be satisfied and the hook matches zero files: ``` $ pre-commit run vale --all-files vale.................................................(no files to check)Skipped ``` Committing a change under `doc/source/data` skips it silently too. Vale still ran in CI through `ci/lint/check-documentation-style.sh`, so the gate existed, but contributors got no local signal, and a green pre-commit run said nothing at all about Vale. This narrows the `doc/source/` entry to `doc/source/(?!data/.*[.](md|rst)$)`, so it keeps excluding the tree except the prose the hook targets. Narrowing it also exposes those 48 files to hooks that had never seen them. Three needed exemptions, all recorded in the config with comments: * `trailing-whitespace` and `end-of-file-fixer` would rewrite 17 unrelated files. I've exempted those two hooks from `doc/source/data` so that enabling Vale stays a config-only change. **The whitespace cleanup is worth doing, and I'm happy to send it as its own PR** rather than bury it here, partly because three of those 17 files are also touched by the open #65372. * `semgrep` would fail. Caught by Cursor Bugbot, and it was right. `semgrep.yml`'s `code-block-python` rule lists `doc/source/data/**/*.rst` in its `paths.include` at `severity: ERROR`, there are 8 pre-existing violations across 5 files, and `ci/lint/lint.sh` runs `pre-commit run semgrep --all-files` as the `semgrep_lint` step. Converting those code blocks to `testcode` is real work that belongs in its own PR, not in a Vale change. The two rST hooks that also become eligible pass as-is. If you'd rather just take the whitespace churn here, say so and I'll fold it in. **A bigger finding, which I'm raising rather than fixing here.** That `semgrep` exemption is a second instance of the same bug, not a new one. Because the top-level `exclude` has covered `doc/source/`, the `code-block-python` rule has never run on **any** of the five doc paths it targets: 31 occurrences across 16 files in `doc/source/data`, `ray-core`, `ray-observability`, and `train`. Every one is an untested documentation snippet the rule was written to catch. Happy to take that on as a follow-up, but converting snippets means they start executing in CI, so it wants a maintainer opinion on scope and on whether the rule is still wanted for docs at all. ### 2. Vale was pinned to 3.4.1, from June 2024 This upgrades both the hook `rev` and the CI script to 3.17.1. The bump isn't inert, so three supporting changes come with it. `Vale.Spelling` produces **no alerts at all** under 3.4.1, but 11 error-severity alerts under 3.17.1, because the newer engine word-boundary-anchors vocabulary patterns. Existing entries such as `[Bb]ackpressure`, `dtype`, and `namespace` therefore stopped covering `backpressured`, `dtypes`, and `namespaces`. This widens those three and adds terms that were never listed at all: `Databricks`, `Datasink`, `Multimodality`, `preprocess`, `resample`, `vCPU`, `Vectorizer`. Two Google rules also became error-severity on content where they're wrong, so they're set to `warning` rather than editing correct prose: * `Google.HeadingPunctuation` reads the period in a numbered heading such as `### 1. Label your worker nodes` as a heading-ending period. The match it reports is literally `1.`. * `Google.OxfordComma` fires on \"X, A and B\" where A and B modify X rather than continuing a list, as in \"resources used by all operators, active and requested\". Both follow the existing precedent of tuning Google rules in `.vale.ini` alongside `Google.Colons` and `Google.Headings`, and both preserve 3.4.1's effective behavior, where neither rule blocked. ## Related issues None. Cross-referencing two of my own open PRs that touch adjacent surfaces, neither of which shares a file with this one: * #65368 removes the banned-words lint in favor of Vale. That increases how much Ray leans on Vale, which makes a dead local hook and a two-year-old pin more worth fixing. Touches `ci/lint/check-banned-words.sh`, `ci/lint/lint.sh`, and `.buildkite/lint.rayci.yml`. * #65342 makes the docs-go scope guard count Vale configuration as documentation content. Worth noting that after it merges, a Vale-config-only PR like this one would classify as documentation. Touches `ci/lint/validate_docs_go_scope.sh`. ## Additional information **AI assistance was used.** I used Claude Code to investigate and draft this. I reviewed every changed line, ran the measurements below myself, and can defend the change end to end. **Not a duplicate.** `gh pr list --repo ray-project/ray --state open --search \"vale in:title\"` and the same for `pre-commit` return only #65368 and #65342 above, which change different files and have different goals. **Measurements**, on the exact set `ci/lint/check-documentation-style.sh` lints (`doc/source/data` and `doc/source/ray-overview/examples`): | Vale | Total alerts | Error-severity | Exit code | Lint step | |---|---|---|---|---| | 3.4.1 (current pin) | 162 | 0 | 0 | green | | 3.17.1, bump alone | 208 | 14 | 1 | **red** | | 3.17.1, with this PR | 196 | 0 | 0 | green | That middle row is the reason the vocabulary and `.vale.ini` changes are in the same PR as the version bump: without them the bump turns `lint: documentation_style` red. **Checks run locally:** ``` # the CI script's exact target set, both versions vale doc/source/data doc/source/ray-overview/examples # 3.4.1 -> exit 0; 3.17.1 pre-fix -> exit 1; post-fix -> exit 0 # the hook now actually matches files and passes pre-commit run vale --files doc/source/data/data-internals.rst # vale.....................................................................Passed # and genuinely fails, rather than passing vacuously: appended a line # containing \"This sentance has a deliberate spelling mistake.\" pre-commit run vale --files doc/source/data/data-internals.rst # 349:6 error Did you really mean 'sentance'? Vale.Spelling # 1 error, 0 warnings and 17 suggestions in 1 file. # the two hooks I exempted stay exempt, so no file churn pre-commit run trailing-whitespace --files $(git ls-files 'doc/source/data' | grep -E '\\.(md|rst)$') pre-commit run end-of-file-fixer --files $(git ls-files 'doc/source/data' | grep -E '\\.(md|rst)$') # both: (no files to check)Skipped # the two rST hooks that newly become eligible pre-commit run rst-directive-colons --files ... # Passed pre-commit run rst-inline-touching-normal --files ... # Passed ``` I also confirmed the 3.17.1 release publishes the two asset filenames the CI script downloads, `vale_3.17.1_Linux_64-bit.tar.gz` and `vale_3.17.1_macOS_arm64.tar.gz`. **Net effect:** contributors get Vale locally, CI runs a Vale two years newer, and `Vale.Spelling` becomes an enforced check instead of a silent one.", "url": "https://github.com/ray-project/ray/pull/65375", "createdAt": "2026-08-11T16:20:57Z", "updatedAt": "2026-08-12T14:45:09Z", "timestamp": "2026-08-12T14:45:09Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "docs", "core", "devprod", "go" ], "author": "dstrodtman", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:c578a23fc5565989f9c0", "signalId": "github:ray-project/ray:pull_request:65425", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65425", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[CI] Clear whitespace debt in doc/source/data and drop the pre-commit carve-out", "text": "## Why Stacked on #65375, and should merge after it. #65375 fixes a pre-commit `exclude` bug that had kept the `vale` hook from matching any files. Narrowing that exclude so Vale can see `doc/source/data` prose also exposed those files to `trailing-whitespace` and `end-of-file-fixer` for the first time. Rather than make a config-only change rewrite 17 unrelated prose files, #65375 exempted the two hooks from the directory and deferred the cleanup. This is that cleanup. ## What 1. Removes both `exclude: ^doc/source/data/` lines that #65375 added, along with the comments explaining them. 2. Runs `trailing-whitespace` and `end-of-file-fixer` over the 48 Markdown and rST files under `doc/source/data`. That rewrites 17 of them: trailing spaces trimmed, missing final newlines added. No content changes. `git diff --ignore-all-space --ignore-blank-lines -- doc/source/data` is empty, so the whole prose diff is whitespace. Both hooks now pass `--all-files`. ## Not in scope The `semgrep` exemption on the same directory stays. It covers 8 pre-existing `code-block-python` violations, which want a real fix rather than a whitespace pass. ## Sequencing Draft until #65375 merges. **Reviewers: the diff currently includes #65375's two commits.** This branch is built on top of that PR's branch, which lives on a fork, so GitHub can't take it as the base ref and the shared history shows up here. Once #65375 lands, a rebase drops those commits and this becomes the 18-file whitespace change described above. Until then, the commit to review here is the last one. Two open PRs touch files in this diff and will need a rebase here once they land. Resolution is mechanical: rebase, re-run the two hooks, amend. - #65372 touches `aggregating-data.rst`, `saving-data.rst`, and `data-internals.rst`. - #65374 touches `data-internals.rst`. ## Reviewing this A pure-whitespace diff is only reviewable when nothing else is in it, which is the point of keeping it as its own PR. `git diff -w` showing nothing is the check that matters.", "url": "https://github.com/ray-project/ray/pull/65425", "createdAt": "2026-08-12T14:21:20Z", "updatedAt": "2026-08-12T14:21:20Z", "timestamp": "2026-08-12T14:21:20Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [], "author": "dstrodtman", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:6bde6ec110c0a6656156", "signalId": "github:ray-project/ray:pull_request:65461", "event": "discovered", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65461", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[doc] llms.txt: tell agents the page links also serve Markdown", "text": "## Summary Follow-up to #64458. Adds one line to `llms.txt` telling agents that the page links also serve Markdown. Read the Docs serves a Markdown rendering of any page **from that page's own `.html` URL**, under an `Accept: text/markdown` request. There is no separate `.md` file to link to β€” `page.md`, `page.html.md`, `/md/.md`, and `?format=md` all 404, and no `Link: rel=alternate` header advertises one. So the `.html` links `llms.txt` already emits *are* the Markdown links. Nothing in the file says so, and an agent has no way to discover it. ``` # Ray > Ray is an open-source unified compute framework for scaling AI and Python workloads… Full page text, grouped by section, is in [llms-full.txt](https://docs.ray.io/en/master/llms-full.txt). Request the links that follow with the HTTP header `Accept: text/markdown` to retrieve a Markdown rendering of the page. ## Overview … ``` ## The header has to be explicit Measured against `docs.ray.io/en/master/data/data.html`: | `Accept:` sent | returns | |---|---| | `*/*` β€” the default for curl, `requests`, `httpx` | `text/html` | | `text/html,application/xhtml+xml,*/*;q=0.8` β€” browser default | `text/html` | | `text/plain` | `text/html` | | `text/markdown` | **`text/markdown`** | | `text/markdown, */*` | **`text/markdown`** | | `text/markdown;q=1.0, text/html;q=0.9` | **`text/markdown`** | A wildcard isn't enough; the type has to be named. A client gets Markdown only by asking deliberately, so the pointer has a real audience. ## Config-gated New `llms_txt_markdown_hint`, **default `False`**. Content negotiation is a property of the host, not of the generator β€” a build served from somewhere that doesn't negotiate Markdown would otherwise advertise a representation that doesn't exist. `conf.py` sets it `True` for Ray. ## Wording note It says \"a Markdown rendering of **the page**\", deliberately not \"the source\". What comes back is the built page converted to Markdown β€” front matter, then site chrome (`Skip to main content`, theme switcher, version picker), then the content. That is not what `_sources/..txt` returns, and `_sources` genuinely exists, so calling it \"the source\" would point agents at the wrong artifact. ## Why not link `_sources/..txt` instead Considered and rejected: - **It leaks the source format to the consumer.** master is **363 `.rst` to 232 `.md`**, so most pages would hand back reStructuredText, not Markdown β€” noisier for an agent than the HTMLβ†’Markdown conversion. - **The URL isn't derivable from the page URL** β€” it depends on each page's source suffix (`index.md.txt` vs `installation.rst.txt`). - **It isn't human-clickable.** - **It would churn.** As pages migrate to MyST, each link would change from `.rst.txt` to `.md.txt`. Negotiation happens on the rendered HTML, so it's identical for both source formats and survives the migration. - **It's redundant** β€” the verbatim source is already in the `llms-full.txt` shards. ## Testing ``` $ python doc/source/_ext/test_llms_txt.py PASS: llms_txt extension produced a correct index, Optional section, llms-full shards, excludes notebooks by source type, gates the Markdown pointer, and honors the llms_txt_build gate. $ ruff check doc/source/_ext/llms_txt.py doc/source/_ext/test_llms_txt.py All checks passed! ``` New `test_markdown_hint` covers both directions: the pointer appears in the header (asserted to sit before the first `##` section) when the config is on, and is absent when it's off. The header table above was produced by probing the live `docs.ray.io`. All three changed files route to `@ doc doc_api` under the current `.buildkite/test.rules.txt`, so `doc: test llms.txt extension` runs on this PR. Not a duplicate: `gh pr list --state open` finds no other open PR touching `llms.txt` or Markdown content negotiation. ## AI assistance Drafted with Claude Code. The behavioural claims here (which `Accept` values return what, which `.md` URL forms 404, the negotiated output including page chrome) were each verified against the live site rather than assumed.", "url": "https://github.com/ray-project/ray/pull/65461", "createdAt": "2026-08-13T15:06:22Z", "updatedAt": "2026-08-13T15:31:51Z", "timestamp": "2026-08-13T15:31:51Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "go", "docs-go" ], "author": "ronny-anyscale", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:87932c4c0d2be83bb71c", "signalId": "github:ray-project/ray:pull_request:65370", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "updatedAt", "metrics" ], "signal": { "id": "github:ray-project/ray:pull_request:65370", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "serve: expose multiplexed model IDs in ReplicaDetails", "text": "## Summary Adds a backward-compatible `multiplexed_model_ids` field to `ReplicaDetails` and keeps it synchronized when the controller receives updated multiplexed model IDs. Extends the existing multiplexing test to verify the values through `ServeInstanceDetails` returned by `serve_instance.get_serve_details()`. Closes #65347 ## Duplicate check I checked open PRs for #65347 and for `multiplexed_model_ids ReplicaDetails`; no matching open PR existed. ## Testing - `python -m compileall` on all four changed Python files β€” passed. - Direct execution of `test_record_multiplexed_model_ids_updates_actor_details` β€” passed. - `git diff --check` β€” passed. - Ruff, pydoclint, Ray Serve mypy, Pyrefly, import-order, and Black checks β€” passed. - The full pytest integration test could not be collected locally because Windows `setup-dev.py` junctions caused duplicate `conftest.py` registration. Ray CI will run the added integration test on Linux. - `docstyle` could not run because `/bin/bash` is unavailable in this Windows environment; Semgrep's pre-commit environment had an invalid cached Python path. ## AI assistance I used AI assistance to help implement and validate this change. I reviewed and understand every changed line.", "url": "https://github.com/ray-project/ray/pull/65370", "createdAt": "2026-08-11T13:32:40Z", "updatedAt": "2026-08-13T14:46:49Z", "timestamp": "2026-08-13T14:46:49Z", "metrics": { "reactions": 1, "comments": 3 }, "labels": [ "serve", "community-contribution" ], "author": "upadhyay74aman", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:5be54980a3eaaf1e01bd", "signalId": "github:ray-project/ray:pull_request:65460", "event": "discovered", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65460", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[doc] Render the Jobs API spec with sphinxcontrib-openapi instead of ReDoc", "text": "> **Stacked PR.** Targets `doc-1513-declare-openapi-303` (#65458), which targets #65457. Review order is #65457 β†’ #65458 β†’ this. GitHub retargets automatically as each merges. > > **Conflicts with #65456 by design.** That PR pins `redoc_uri` to a specific ReDoc version; this one deletes `redoc_uri` entirely. If #65456 lands first I'll resolve the conflict here. If this lands, #65456 becomes moot and should be closed. Either order works β€” #65456 is worth keeping as the safe fix in case this one stalls in review. ## Why this change is needed `sphinxcontrib-redoc` is unmaintained. 1.6.0 is the latest release and dates from 2020; the `2.0.0a1` alpha still imports `pkg_resources`. Keeping it costs this repo three things: | Cost | Location | | --- | --- | | A `setuptools<82` ceiling, because the extension imports `pkg_resources` at module load and setuptools 82 removed it | `doc/requirements-doc.txt` | | A `misc.copy_overwrite` entry in `suppress_warnings`, because the extension redundantly copies its own bundled asset | `doc/source/conf.py` | | The renderer fetched from a third-party CDN at page-view time | `doc/source/conf.py` | All of that buys exactly one page. The Jobs API spec is the only consumer: `conf.py` had one `redoc` entry and there's no other OpenAPI document in the repo. There's a second, larger reason. **The current page is a client-rendered JavaScript island.** ReDoc has no source file at all β€” the extension synthesizes `api.html` from `conf.py`. So today the page: - is in **no toctree**, and - is reachable only through a hardcoded `` `... `_ `` link at `rest.rst:94`, and - has **none of its endpoint content in the built HTML**, so it can't participate in Sphinx search or cross-referencing. ## Related changes Fourth in the chain above. Supersedes the \"record why we stay on ReDoc\" PR (#65459), which I'll close β€” its conclusion is obsolete now that this works. ## What changes - Swap `sphinxcontrib.redoc` β†’ `sphinxcontrib.openapi` (0.9.0) in `conf.py` and `doc/requirements-doc.txt`, and regenerate the deplock. - Add a real `api.rst`, put it in the `job-submission` toctree, and replace the raw `` link with a `:ref:`. - Drop the now-unneeded `misc.copy_overwrite` suppression. - Change two operation `summary` fields from YAML block scalars to plain scalars (see below). - Update the `setuptools` comment: the `<82` ceiling's cause is gone. The pin itself is left in place, since unpinning is a separate resolve worth verifying on its own. `sphinxcontrib-openapi` is pure Python, from the same `sphinx-contrib` org, and needs no `pkg_resources`. The deplock diff is `-sphinxcontrib-redoc` and `+sphinxcontrib-openapi` plus its five dependencies (`sphinxcontrib-httpdomain`, `sphinx-mdinclude`, `deepmerge`, `mistune`, `picobox`). ## Verification Built with `make -C doc rtd-build`, which runs the RtD-faithful clean build with `-W`, matching `fail_on_warning: true`: ``` build succeeded. ``` Exit 0, and **zero `WARNING:`/`ERROR:` lines in the whole log**. `raydepsets --check` on `docbuild_depset_311` reports \"Lock files are up to date.\" Checked against the built `api.html` rather than inferred: | Check | Result | | --- | --- | | Request/response schemas rendered | Yes β€” `Request JSON Object` and `Response JSON Object` field lists, with `entrypoint`, `runtime_env`, `metadata`, and the rest, including types and required flags | | `$ref` and `allOf` resolution | Resolved, including through `JobDetails` | | Literal `**` on the page | None β€” zero docutils `problematic` nodes | | `http-routingtable` index | Generated | | Page in the sidebar, with prev/next pagination | Yes | ## Two non-obvious configuration details Both are commented at their sites in the diff, because both fail quietly. **The renderer is selected via `openapi_default_renderer`, not by the directive spelling.** Writing `.. openapi:httpdomain::` in the page adds an `unknown directive name` warning, which is fatal under `fail_on_warning`. Selecting it in `conf.py` and using a plain `.. openapi::` avoids that. **The default renderer emits no request or response schemas at all** β€” just paths, parameters, and status codes. So leaving `openapi_default_renderer` unset would silently drop most of the reference content while still building green. That's the trap worth knowing about if anyone touches this later. ## Why the summary fields change `sphinxcontrib-openapi`'s renderer interpolates an operation's `summary` into `**...**` without stripping it. A YAML block scalar (`summary: |`) leaves a trailing newline in the value, so the markup opens on one line and closes on the next, docutils can't pair it, and readers see literal asterisks. Filed upstream as [sphinx-contrib/openapi#173](https://github.com/sphinx-contrib/openapi/issues/173). Two operations used block scalars (`GET /api/version` and `POST /api/jobs`); the other six already used plain scalars. Converting those two is the whole fix and changes no rendered text. ## What this gives up, honestly **Request and response samples.** ReDoc rendered example payloads; this doesn't. It isn't a configuration choice β€” both routes to examples are blocked by upstream bugs, and I've confirmed each against this spec: - `:generate-examples-from-schemas:` crashes with `ValueError: dictionary update sequence element #0 has length 1; 2 is required` ([#166](https://github.com/sphinx-contrib/openapi/issues/166)). Example generation can't handle an `allOf` that targets a non-object schema, which is exactly how `JobDetails` attaches descriptions to the `JobType` and `JobStatus` string enums. - `:response-examples-for:` is rejected as an unknown option ([#165](https://github.com/sphinx-contrib/openapi/issues/165)). `api.rst` carries a comment recording this so nobody re-derives it, and I've added the reproduction detail to both upstream issues. Weighed against gaining search, cross-referencing, navigation, a dependency that's actually maintained, and no third-party CDN at page-view time, losing sample payloads seemed like the right trade β€” but it is a real loss, and it's reasonable to want the upstream fixes first instead. I've also filed [#171](https://github.com/sphinx-contrib/openapi/issues/171), [#172](https://github.com/sphinx-contrib/openapi/issues/172), and [#173](https://github.com/sphinx-contrib/openapi/issues/173), each with a minimal reproducer, and am planning PRs for them. ## Checks - [x] `make -C doc rtd-build`: build succeeded, zero warnings. - [x] `raydepsets --check`: lock files up to date. - [x] Built page inspected for schema content, `allOf`/`$ref` resolution, and stray markup. - [x] `pre-commit run` clean. - [ ] Read the Docs preview reviewed for the appearance change.", "url": "https://github.com/ray-project/ray/pull/65460", "createdAt": "2026-08-13T14:03:32Z", "updatedAt": "2026-08-13T14:05:12Z", "timestamp": "2026-08-13T14:05:12Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [], "author": "dstrodtman", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:ba675ad165954d34263f", "signalId": "github:ray-project/ray:pull_request:65459", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "updatedAt", "metrics", "state" ], "signal": { "id": "github:ray-project/ray:pull_request:65459", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[doc] Record what blocks lifting the setuptools ceiling and replacing sphinxcontrib-redoc", "text": "> **Stacked PR.** Fourth in a chain: #65456 (pin ReDoc) β†’ #65457 (spec validity) β†’ #65458 (declare 3.0.3) β†’ this. It's independent in substance and touches only `doc/requirements-doc.txt`, so it can be retargeted to `master` if you'd rather take it on its own. ## Why this change is needed The `setuptools==80.9.0` pin in `doc/requirements-doc.txt` exists because `sphinxcontrib-redoc` imports `pkg_resources` at module load, and setuptools 82 removed it. The comment ends with \"Lifting this ceiling means replacing or vendoring that extension,\" which leaves two questions open: 1. Is `sphinxcontrib-redoc` actually the *only* thing in the doc environment that needs `pkg_resources`? If not, removing it wouldn't lift the ceiling anyway. 2. Is there a usable replacement? I evaluated both. This records the answers so the next person hitting this pin doesn't re-derive them. ## Related changes Fourth in the chain above. No functional dependency on the three below it. ## What changes A comment in `doc/requirements-doc.txt`. No behavior change, no dependency change, no build change. ## Finding 1: `sphinxcontrib-redoc` is the sole blocker Installed every extension `conf.py` loads into an environment with no setuptools and no importable `pkg_resources`, then imported each one: | Result | Extensions | | --- | --- | | Import cleanly | `sphinx.ext.autodoc`, `viewcode`, `napoleon`, `doctest`, `coverage`, `autosummary`, `intersphinx`, `sphinx_click.ext`, `sphinxemoji`, `sphinx_copybutton`, `sphinx_sitemap`, `myst_nb`, `sphinxcontrib.autodoc_pydantic`, `sphinx_remove_toctrees`, `sphinx_design`, `sphinx_docsearch`, `sphinx_collections`, `sphinxext.opengraph` | | **Fails** | `sphinxcontrib.redoc` β€” `ModuleNotFoundError: No module named 'pkg_resources'` | That failure is the control: it reproduces the exact documented breakage, which confirms the test has power rather than passing vacuously. So dropping that one extension is sufficient to lift the ceiling. ## Finding 2: the maintained replacement works, in exactly one configuration `sphinxcontrib-openapi` 0.9.0 (released 2026-02-10, `requires_python >=3.10`) is the actively maintained alternative from the same `sphinx-contrib` org. It needs no setuptools, and it **does** render this spec with full request and response schemas and **zero warnings under `-W`** β€” but only with both of these: - `openapi_default_renderer = \"httpdomain\"` in `conf.py`, then a plain `.. openapi::` directive. - Operation `summary` fields as plain scalars rather than YAML block scalars. Neither is obvious, and each failure mode is silent or fatal rather than self-explaining: | Configuration | Schema content | Warnings under `-W` | | --- | --- | --- | | default renderer (`httpdomain:old`) | **none** β€” paths, parameters, status codes only | 0 | | `httpdomain:old` + `:request:` | request bodies only, as a raw JSON blob | 1 β€” [#171](https://github.com/sphinx-contrib/openapi/issues/171) | | `httpdomain:old` + `:examples:` | full | 1 β€” [#172](https://github.com/sphinx-contrib/openapi/issues/172) | | `httpdomain`, block-scalar summaries | full | 2 β€” [#173](https://github.com/sphinx-contrib/openapi/issues/173), plus literal `**` on the page | | **`httpdomain`, plain-scalar summaries** | **full** | **0** | The literal-asterisk defect is worth calling out because it's triggered by this repo's own spec: two operations use `summary: |`, which leaves a trailing newline in the value that the renderer interpolates straight into `**...**`. Changing those two lines to plain scalars is sufficient. The three configurations that genuinely don't work are filed upstream with minimal reproducers. ## So what actually gates migrating Not dependency hygiene, and not warnings. It's that ReDoc's standalone three-panel page would be replaced by output rendered inline in the site theme. That's a visible change to a published reference page, so it's a maintainer call rather than a docs-tooling cleanup, and this PR doesn't make it. Two things worth weighing whenever that call happens. In favor: the current page is a client-rendered JavaScript island. It's in no toctree, it's reachable only through a hardcoded `` link at `rest.rst:94`, and its endpoint content isn't in the built HTML, so it doesn't participate in Sphinx search or cross-referencing. The trial build produced real Sphinx nodes and an `http-routingtable` index. Migrating would also let the `setuptools` ceiling lift, per finding 1. Against: it's a published reference page that readers may have bookmarked in its current form, and the inline rendering is a different reading experience rather than a strictly better one. My trial builds used the `basic` theme, so I can't show what it looks like under this site's actual theme without a full doc build. ## Checks - [x] Comment-only change; `pre-commit run` clean. - [x] Every claim in the comment verified by the trial builds described above.", "url": "https://github.com/ray-project/ray/pull/65459", "createdAt": "2026-08-13T13:30:11Z", "updatedAt": "2026-08-13T14:03:44Z", "timestamp": "2026-08-13T14:03:44Z", "metrics": { "reactions": 1, "comments": 1 }, "labels": [], "author": "dstrodtman", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:93e682bfbbf2f52c86d9", "signalId": "github:ray-project/ray:pull_request:65453", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "updatedAt", "metrics" ], "signal": { "id": "github:ray-project/ray:pull_request:65453", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[Data] Add support for writing ORC files", "text": "## Description Ray Data can read ORC files, but it can't write them. This adds `Dataset.write_orc()`, backed by a new `ORCDatasink` that writes each output block with `pyarrow.orc.write_table`. The API forwards PyArrow ORC writer options and supports `min_rows_per_file` through the existing block-based file datasink infrastructure. ## Related issues Follow-up to #64600 ## Additional information Mirrors the `write_csv` / `write_json` APIs. PyArrow ORC writer options can be passed directly to `write_orc()`. No new dependency (`pyarrow` already ships `pyarrow.orc`). Adds write tests in `test_orc.py` and API docs in `saving_data.rst`. ```python import ray ds = ray.data.range(100) ds.write_orc(\"s3://bucket/path/\", compression=\"zstd\")", "url": "https://github.com/ray-project/ray/pull/65453", "createdAt": "2026-08-13T09:53:45Z", "updatedAt": "2026-08-13T13:59:52Z", "timestamp": "2026-08-13T13:59:52Z", "metrics": { "reactions": 1, "comments": 1 }, "labels": [ "docs", "data", "community-contribution" ], "author": "WFY123wfy", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:84f189ad349a47489e6b", "signalId": "github:ray-project/ray:pull_request:65239", "event": "changed", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [ "updatedAt", "metrics" ], "signal": { "id": "github:ray-project/ray:pull_request:65239", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[docs] Add Kubernetes and KubeRay conventions to the style guide", "text": "## Why this change Ray on Kubernetes docs describe Kubernetes API objects alongside Ray concepts, and the two vocabularies collide. Deployment, Job, and Service each name both a Kubernetes API object and a distinct Ray concept, so capitalization is the only thing telling them apart, and the style guide didn't say which form to use where. A contributor asking \"RayCluster or Ray cluster?\" had to derive the answer from two guides and adjudicate between them. This adds a `Writing Ray on Kubernetes and KubeRay docs` section to the style guide, scoped to the user-facing pages under `doc/source/cluster/kubernetes/` and the contributor-facing pages in [ray-project/kuberay](https://github.com/ray-project/kuberay). The approach is to extend this guide with a Kubernetes deference section rather than stand up a separate KubeRay style guide. The two guides already overlap heavily, and a forked guide would drift within a release or two. ## What the casing scan found The term list isn't invented. It comes from a prose-only scan of three corpora: 79 user-facing pages in this repo, 12 contributor pages in `ray-project/kuberay`, and 128 pages of `kubernetes/website` (`content/en/docs/concepts/`, commit `bb38117c0`). The scan strips front matter, fenced code blocks, inline code, MyST roles, and link targets before counting, so YAML field names and CLI output don't inflate the figures. Headline: **these docs already follow the Kubernetes casing rule.** Upstream Kubernetes capitalizes \"Pod\" in 71% of prose uses; the Ray-side KubeRay docs do so in 70%. So this PR documents a rule the docs mostly already follow, and does not propose a capitalization sweep. | Term | Ray-side KubeRay docs | Upstream Kubernetes | Read | |---|---|---|---| | Pod | 371/527 (70%) | 3712/5195 (71%) | Matches upstream almost exactly. | | ConfigMap | 20/21 (95%) | 119/130 (91%) | Healthy. | | Ingress | 8/15 (53%) | 148/242 (61%) | Roughly aligned. | | Service | 18/114 (15%) | 515/946 (54%) | Diverges by design. | | Job | 17/143 (11%) | 522/610 (85%) | Diverges by design. | | Deployment | 6/53 (11%) | 355/461 (77%) | Diverges by design. | The three \"diverges by design\" rows are the reason this section exists. Sampling shows the lowercase uses are Ray Serve deployments, Ray jobs, and Ray services, not Kubernetes API objects. That's a naming collision, not drift, so the fix is a disambiguation rule rather than a sweep. Kubernetes's own rule expects a mix (UpperCamelCase for the API object, sentence case for the general concept), which is why the upstream column isn't 100% either. The one genuine inconsistency the scan found is **\"Ray Autoscaler\" (44 uses) against \"Ray autoscaler\" (35)**, spread across at least eight files. This guide already says to lowercase generic nouns even when they name a Ray concept, so lowercase wins. Recording the rule here is the prerequisite for a mechanical follow-up cleanup. I've left that out of this PR to keep the rule change reviewable on its own. ## Open questions for reviewers I've deliberately written only the rules I can support from the two published guides plus the scan. These five are genuinely open, and I'd rather resolve them in review than assert them: 1. ~~**Confirm the custom-resource split.**~~ **Resolved** by @win5923: the split is intentional and consistent with Kubernetes terminology. Original question: The section documents `RayCluster` (the resource) versus \"Ray cluster\" (the running thing it produces), currently 335 against 106 uses in this repo. I read it as intentional and matching the Kubernetes API-object rule. Worth confirming rather than normalizing to one form. 1. ~~**\"head Pod\" versus \"head node.\"**~~ **Resolved** by @win5923: use \"head Pod\" consistently when a Ray cluster runs on Kubernetes. Original question: The section prefers \"head Pod\" for the Kubernetes-hosted case, since a node is a machine on Kubernetes. `doc/source/cluster/kubernetes/index.md` currently says \"a head node pod and a collection of worker node pods,\" which stacks both vocabularies. Reasonable, or is \"head node\" worth keeping for continuity with Ray's core architecture docs? 1. **Feature graduation phases.** Kubernetes uses start case (\"DRA is Beta\"); Ray docs lowercase feature stages. This bites on pages discussing a Kubernetes feature's stage and a Ray feature's stage in one paragraph. I left it out of the section rather than pick. Which wins? 1. **Field values.** Kubernetes wants unquoted, unstyled values (\"set `imagePullPolicy` to Always\"); this guide says to use backticks for configuration values and gives `num_replicas: 1`. These are closer than they look, since one is a bare value and the other a key-value pair, but the guidance should say which form applies where. 1. **\"Ray dashboard\" versus \"Ray Dashboard\"** (43 title-case against 18 lowercase). This guide implies lowercase, but the dashboard is arguably a named UI surface. Not addressed in the section pending a call. Two further items are out of scope here but worth flagging: - **Line wrapping in the `kuberay` repo.** This guide requires soft wrap. The Kubernetes guide requires manual wrap, justified by localization teams diffing per line, which KubeRay doesn't have. Nothing in this PR changes that, since this guide already governs `doc/source/`. But `kuberay`'s `.markdownlint.yaml` sets `MD013` to 120 characters via pre-commit, which is a latent gate: 266 lines in its `docs/` already exceed it, it isn't enforced in CI, and it isn't auto-fixable, so contributors hit it unpredictably on whichever file they touch. Disabling it there would align the two repos, but that's a call for KubeRay maintainers in their own repo. - **Turning Vale enforcement on.** This PR now adds the ruleset but doesn't enforce it. See the section below. ## Staged Vale ruleset A later commit adds a Vale ruleset for these pages, following the pattern already used for the Ray Data docs. **Nothing is enforced yet.** The Vale hook in `.pre-commit-config.yaml` stays scoped to `^doc/source/data/.*`, so CI checks exactly what it checked before. - `.vale/styles/config/vocabularies/KubeRay/accept.txt` β€” the KubeRay custom resources, the Kubernetes and ecosystem API objects these pages reference, and project names. Skips what the `General` vocabulary already accepts (`Kubernetes`, `kubectl`, `CRD(s)?`, `namespace`). - `.vale/styles/KubeRay/Terms.yml` β€” a `substitution` rule at `warning` level covering only the casing forms the new style-guide section settles, with `vocab: false` (see below). The open questions above are deliberately absent. - `.vale.ini` β€” registers the vocabulary and scopes the style to `doc/source/cluster/kubernetes/`. Registering a vocabulary that the hook doesn't exercise matches existing practice here: `RLlib` and `Train` are already in `Vocab` while the hook only runs on `doc/source/data/`. Contributors can run it by hand today: ``` vale doc/source/cluster/kubernetes/ ``` To enforce later, widen the `files` pattern on the `vale` hook. I'd still hold that until the open questions land. ### What I verified Vale 3.12.0 locally. **Path scoping works.** The KubeRay rules fire on a file under `doc/source/cluster/kubernetes/` and stay silent on identical content at the repo root. **Current state of these pages** β€” 146 `KubeRay.Terms` alerts, which lines up with the casing scan: | Alert | Count | |---|---| | Use 'Ray autoscaler' instead of 'Ray Autoscaler' | 42 | | Use 'head Pod' instead of 'head pod' | 20 | | Use 'worker Pod' instead of 'worker pod' | 7 | | Use 'Ray job' instead of 'ray job' | 2 | | Use 'Ray Serve' instead of 'Ray serve' | 1 | | Use 'Ray cluster' instead of 'ray cluster' | 1 | | Use 'KubeRay' instead of 'Kuberay' | 1 | | Use 'GCS fault tolerance' instead of 'GCS FT' | 1 | The 42 matches the scan's count for these pages exactly, which is a useful cross-check on both. None of it is enforced, so nothing here has to be fixed in this PR. **No regression on the gated path.** Alert count on `doc/source/data/` goes 174 β†’ 173, deterministic across repeated runs in both states, with no new alerts. A `Vocab` list is global, so it can only widen what's accepted. The single alert that disappears is a `Google.Parens` hit in `concurrent-dataset-execution.md`; no term in the new vocabulary appears on that line, so I'm reporting that as observed rather than claiming I understand the mechanism. It's a reduction either way, so it can't break the gate. ### The vocabulary shadowing trap Worth knowing if you write Vale rules here: a `Vocab` accept entry shadows its token for **every** rule, case-insensitively. A substitution whose target appears in an accept list silently never fires, and nothing warns you. This bit the `Ray Autoscaler` swap, because `autoscaler` is accepted in `config/vocabularies/Data/accept.txt`. The fix is `vocab: false` on the rule, which opts out of the filtering without touching the Data vocabulary. The rule carries a comment saying so. Credit to Bugbot for catching it. I'd first shipped that swap commented out, having concluded the only way to enable it was dropping `autoscaler` from the Data vocabulary and accepting new spelling alerts on the gated path. That was wrong, and `vocab: false` avoids the tradeoff entirely. ## Added after review Two rules that review surfaced, both in the style-guide section plus the Vale ruleset. **Node stays lowercase.** Node is a Kubernetes API object, so \"capitalize Kubernetes API objects\" read literally tells contributors to write \"Kubernetes Node.\" These pages almost always mean the machine, and all 24 uses are correctly lowercase today, so a literal reading of the rule would have someone \"fix\" working prose into wrongness. Kubernetes applies the same two-part rule it applies elsewhere, capitalizing Node only for the resource, and the section now says so. **Don't write \"Ray Pod.\"** This answers the casing question raised in review, and the answer turned out to be neither casing. A Pod is a Kubernetes host that contains a Ray node plus any sidecars, so naming it after Ray overstates what it is. \"TPU Pod\" also already occupies the qualified-Pod form in these pages and means a group of TPU chips rather than a Kubernetes object. The term had no settled form to preserve either: across the 55 prose lines using it, casing split four ways at 20, 16, 16, and 15. Stating that alone would be a trap, because a bare \"Pod\" isn't always an adequate replacement, so the section gives the test that separates the cases: - Qualify when the qualifier distinguishes two referents. That's why \"Ray node\" against \"Kubernetes node\" is correct and should stay. - Don't qualify when it relabels one thing. - Keep the scope explicit where a page also describes Pods the RayCluster doesn't own, such as the operator, Redis, or a `curl` Pod, and where a nearby command filters by label. \"List all Pods\" and \"List the Ray cluster's Pods\" describe different commands and shouldn't read the same way. The cleanup is #65423, which removes all 89 uses from the docs and is reviewable on its own. This PR is the rule; that one is the sweep. Vale picks up four swaps for the retired forms. Plurals get their own entries, for the word-boundary reason @win5923 identified on the `head pod` swaps. Verified that they don't fire on \"Ray head Pod\" or \"Ray worker Pods\", and that the gated `doc/source/data/` path holds at 173 alerts. ## Not a duplicate Checked open PRs for this area before starting: ``` gh pr list --repo ray-project/ray --state open --search \"writing-style\" gh pr list --repo ray-project/ray --state open --search \"style guide in:title\" gh pr list --repo ray-project/ray --state open --search \"kuberay style\" ``` No open PR touches `doc/source/ray-contribute/writing-style.md` or proposes KubeRay style guidance. The `writing-style` matches were all false positives on `write_delta`. This extends the guide added in #64518. ## Testing ``` pre-commit run --files doc/source/ray-contribute/writing-style.md ``` Passes, but as a no-op on the Markdown: every hook reports \"no files to check,\" because no pre-commit hook applies to Markdown under `doc/source/` and Vale is scoped to `doc/source/data/`. Stating that plainly instead of claiming coverage it doesn't give. The Vale config files do have hook coverage, and those checks ran for real: ``` pre-commit run --files .vale.ini .vale/styles/KubeRay/Terms.yml .vale/styles/config/vocabularies/KubeRay/accept.txt ``` `trim trailing whitespace`, `fix end of files`, `check for added large files`, and `semgrep` all pass. Vale verification for the ruleset is in the section above: path scoping, the 31-alert current state, and the 174 β†’ 173 no-regression check on the gated path. Verified by hand: the new `(kubernetes-docs-style)=` anchor doesn't collide with any existing target in `doc/source/`, and the section uses the same MyST anchor form and Markdown pipe-table style already used elsewhere in this file. Rendering is verified by the Read the Docs preview build on this PR. No code, no API, no behavior change. The prose is a contributor guide; the Vale files add configuration without widening what CI enforces. ## AI assistance Written with AI assistance (Claude). I've reviewed every changed line, generated and inspected the casing counts myself, and can defend the change end to end. The scan scripts that produced the numbers live outside this repo; happy to share the methodology or re-run against a different corpus if a reviewer wants to check a figure. --- ### Update: the vocabulary now covers the ecosystem terms these pages use (555ebdad) Running `vale doc/source/cluster/kubernetes/`, which this PR's `.vale.ini` comment tells contributors to do, produced 215 `Vale.Spelling` alerts on correctly spelled terms. The accept list covered API objects, the custom resources, and project names, but not the wider ecosystem vocabulary the pages actually contain. Left as is, a contributor following the instruction learns to ignore Vale on exactly the paths this PR wants gated. Added: the ecosystem projects (Istio, Kiali, gVisor, Kata, Krew, Kustomize, Bottlerocket, Aliyun, GCSFuse, Promtail, Fluentd, Filebeat, Alertmanager), the Gateway API `HTTPRoute`, and the Kubernetes and infrastructure terms in use (cgroup, finalizer, kubeconfig, kubelet, namespaced, namespaces, toleration, upscaling, downscaling, vCPU, FQDN, AMI, subnet, syscall, hostname, cron, CA, autoscalers). **Excluded on purpose, since an accept list shouldn't launder a defect.** `CR` and `CRs`, 12 uses, because this PR's own guidance says to write \"custom resource\" and to define CustomResourceDefinition then use CRD. Real misspellings: `heterogenous`, `replicaset` for ReplicaSet, `Deepseek` for DeepSeek, `nodepool` for node pool. `gsutil`, which `gcloud storage` supersedes. Unbackticked code identifiers, which are a formatting defect rather than a vocabulary gap: `podSpec`, `podSelector`, `nodeSelector`, `jsonPointers`, `intervalSeconds`, `workerGroup`, `ignoreDifferences`, `HTTPMode`, `args`, `yaml`. And generic English that Vale's dictionary lacks, which belongs in the General vocabulary rather than a KubeRay-specific one. Three entries are case-tolerant (`[Tt]oleration(s)?`, `[Uu]pscaling`, `[Dd]ownscaling`). A lowercase-only entry makes `Vale.Terms` flag the legitimate sentence-start capitalization. **Verification**, deterministic across repeated runs: - `doc/source/cluster/kubernetes/`: 1329 alerts to 1178. `Vale.Spelling` drops from 215 to 72. `KubeRay.Terms` is unchanged at 160, confirming no swap got shadowed by a new accept entry, which is the trap the `Ray Autoscaler` comment in `Terms.yml` documents. - `doc/source/data/`, the only path the pre-commit hook gates: 173 to 170, with **no new alerts**. A `Vocab` is global, so it can only widen what's accepted. - One candidate entry was dropped for exactly that reason. A `runtime(s)?` entry made `Vale.Terms` flag \"Runtime\" in `doc/source/data/monitoring-your-workload.rst`, a new alert on the gated path. Four new `Vale.Terms` alerts do appear on the KubeRay paths, and all four are genuine casing defects in existing pages rather than noise: \"Kubelet\" twice in `configuring-ippr.md`, lowercase \"istio\" naming the project in `rayservice-incremental-upgrade.md`, and \"Cgroups\" in the `resource-isolation-with-writable-cgroups.md` H1, which is also title case. Those are content fixes for a separate change, not this one. AI assistance was used for this commit, including the Vale sweep and the before-and-after verification above.", "url": "https://github.com/ray-project/ray/pull/65239", "createdAt": "2026-08-05T19:50:41Z", "updatedAt": "2026-08-13T17:41:41Z", "timestamp": "2026-08-13T17:41:41Z", "metrics": { "reactions": 1, "comments": 3 }, "labels": [ "docs", "core", "go" ], "author": "dstrodtman", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:b6329021146f05fc21c5", "signalId": "github:ray-project/ray:pull_request:65463", "event": "discovered", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65463", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "core: release runtime env per-job loggers", "url": "https://github.com/ray-project/ray/pull/65463", "createdAt": "2026-08-13T17:27:57Z", "updatedAt": "2026-08-13T17:32:13Z", "timestamp": "2026-08-13T17:32:13Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [], "author": "rohitsalla", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:beba4006abb71fc95a18", "signalId": "github:ray-project/ray:pull_request:64928", "event": "changed", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [ "updatedAt" ], "signal": { "id": "github:ray-project/ray:pull_request:64928", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[Train] Add NCCL RAS health callback", "text": "## Description This PR adds an opt-in callback that detects hanging distributed training jobs by polling NCCL's RAS (Reliability/Availability/Serviceability) subsystem. NCCL β‰₯ 2.24 runs a monitoring thread inside every NCCL process that forms a peer mesh tracking per-rank health and collective op-counts. By querying the `ncclras` client and diffing collective-counts between successive polls, Ray Train can automatically detect and act on hangs that would otherwise stall a job until a torch process group timeout occurs. As RAS gives a snapshot of the current NCCL state, to determine if the mismatch is a temporary issue or a permanent one, the callback polls RAS every N seconds. If a communicator is mismatched *and* makes no progress at all across multiple sequential polls, then this most likely indicates that a deadlock / wedge / hang is in progress for that communicator. Its important to note that its possible for NCCL to continue (not hang) with incorrect data depending on the nccl operation, tensor size, nccl version, etc. In these cases, users will observe the collective-count continue to increase but often with a constant mismatch. In this version of the callback, we are not detecting those issues and focusing on purely NCCL collective mismatches which cause hangs. The callback is disabled by default and enabled with `RAY_TRAIN_ENABLE_NCCL_HANG_DETECTOR=1`, as the detection heuristics still need validation against real-world hangs (spot preemptions, pipeline-parallel send/recv comms that legitimately sit skewed for long periods) before it can be considered for default-on. ### Requirements - RAS itself landed in 2.24, but the poller parses `ncclras -f json`, and the `-f` flag requires NCCL 2.28+. If the binary rejects `-f`, or isn't on PATH, the callback logs a warning once and disables itself for the rest of the run rather than retrying forever. - `ci/docker/base.gpu.Dockerfile` pins and holds `libnccl2`/`libnccl-dev` at `2.28.9-1+cuda12.9` (previously whatever the CUDA base image shipped) so `ncclras` is present and JSON-capable in GPU CI. The build fails fast if the pin doesn't stick or the client binary is missing. This affects all GPU CI images, not just Train tests. - `py-spy` (and `SYS_PTRACE` in the container) for native stack traces at hang time. When unavailable, the callback falls back to a Python-only traceback of every thread. ### How it works - `NCCLRASCallback` lives on the controller and polls `ncclras` on a worker (ras forms a mesh-network between ranks requiring that only one of the workers to be polled and receive the whole network's state). Workers are tried in turn until one returns a usable report, and the query runs on a background thread so it never blocks the controller's poll loop. - A communicator's frozen-streak only advances when the RAS report shows a collective-count mismatch, every rank in that communicator is `RUNNING`, and *no* rank advanced *any* collective since the previous poll. Any progress on any op resets that communicator's streak; streaks are tracked per communicator so each one is confirmed on its own. - A confirmed hang requires the streak to persist for `RAY_TRAIN_NCCL_RAS_CONFIRM_DURATION_S` (default 600s), converted to a consecutive-poll count using `RAY_TRAIN_NCCL_RAS_MIN_POLL_INTERVAL_S` (default 15s), 40 polls by default, to ensure that a mismatch isn't a snapshot issue. - Before confirmation there is an escalation ladder: a `WARNING` when a communicator first crosses ~60s frozen, then a periodic `WARNING` every ~120s naming every still-frozen communicator, its stalled duration, and (in `fail` mode) the time remaining until `NCCLHangError`, alongside the human-readable `ncclras -f text` report. A communicator that resumes progress after being suspected logs an explicit recovery message. - On a confirmed hang, the callback logs the `ncclras -f text` report and captures py-spy stack traces from all workers (uploaded to `/nccl_ras_hang_stack_traces/` as `rank_.log`), then depending on the callback mode, in \"fail\" a `NCCLHangError` is raised and in \"observe\", a detailed log message is produced but no more. `observe` is the default; set `RAY_TRAIN_NCCL_RAS_ACTION=fail` to fail the run. - `NCCLHangError` (a `WorkerGroupError` subclass, `DeveloperAPI`, exported as `ray.train.NCCLHangError`) is treated as non-retryable by the `DefaultFailurePolicy` regardless of the `max_failures` budget, since a desync hang is usually deterministic and a restart would just hang again. ### Configuration | Env var | Default | Read on | | --- | --- | --- | | `NCCL_RAS_ADDR` | `localhost:28028` | worker | | `RAY_TRAIN_ENABLE_NCCL_HANG_DETECTOR` | `0` | driver | | `RAY_TRAIN_NCCL_RAS_ACTION` | `observe` | driver | | `RAY_TRAIN_NCCL_RAS_MIN_POLL_INTERVAL_S` | `15` | driver | | `RAY_TRAIN_NCCL_RAS_CONFIRM_DURATION_S` | `600` | driver | | `RAY_TRAIN_NCCLRAS_PATH` | `ncclras` (PATH lookup) | driver | The `RAY_TRAIN_*` knobs are read once when the callback is constructed on the driver validated eagerly (invalid values raise at construction); `NCCL_RAS_ADDR` is NCCL's variable and is read on the worker at query time. ### Known limitations - **Symmetric in-collective hangs are not detected.** RAS reports collective *launch* counts, so if every rank launches the same collective and the fabric wedges mid-op, all counts match and skew is zero. - **A rank that exited early is not detected.** A communicator is only considered for detection when all of its ranks report `RUNNING`, so a `FINALIZE`/`ABORT` rank alongside frozen peers is currently skipped. This could be considered in this implementation but I believe the Ray health checker should catch this first. - `missing_ranks` (unresponsive / considered-dead peers) is parsed around but not yet used as a signal. - No metrics are emitted yet, and there is no culprit-rank attribution β€” `NCCLHangError.worker_failures` is empty; the RAS text report and stack traces are the only diagnostics. ### Testing - `test_nccl_ras_callback.py` β€” CPU unit coverage of RAS JSON parsing (including the malformed `missing_ranks[]` comma NCCL 2.28.9 emits), `NCCL_RAS_ADDR` parsing, frozen-vs- advancing classification, per-communicator streak independence, communicators appearing and disappearing between polls, poll throttling, confirm-duration β†’ poll-count conversion, config validation, the escalation/observe messaging, and that a failed stack dump does not suppress the hang error. - `test_nccl_ras_hang_detection.py` β€” GPU end-to-end scenarios (`train_v2_gpu`) inducing real NCCL desyncs in a `TorchTrainer` with the detector configured for fast confirmation. Requires β‰₯ 2 visible GPUs and `ncclras` on PATH; the multi-communicator subset test requires 4 GPUs and skips otherwise. - `test_failure_policy.py` β€” added `test_nccl_hang_error_is_non_retryable`.", "url": "https://github.com/ray-project/ray/pull/64928", "createdAt": "2026-07-22T10:44:09Z", "updatedAt": "2026-08-13T17:20:01Z", "timestamp": "2026-08-13T17:20:01Z", "metrics": { "reactions": 1, "comments": 2 }, "labels": [ "train" ], "author": "pseudo-rnd-thoughts", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:b34aeeb653209f369330", "signalId": "github:ray-project/ray:pull_request:65441", "event": "changed", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [ "updatedAt", "metrics" ], "signal": { "id": "github:ray-project/ray:pull_request:65441", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[doc][History server] Update doc for history server for `RAY_ROOT_DIR` -> `STORAGE_ROOT_DIR`", "text": "## Description https://github.com/ray-project/kuberay/pull/5121 updated the env var from `RAY_ROOT_DIR` to `STORAGE_ROOT_DIR`. The default storage root dir is now the bucket root itself, so updated example and file structure in the documentation. ## Related issues Related to https://github.com/ray-project/kuberay/pull/5121 ## Additional information", "url": "https://github.com/ray-project/ray/pull/65441", "createdAt": "2026-08-12T22:15:06Z", "updatedAt": "2026-08-13T17:19:26Z", "timestamp": "2026-08-13T17:19:26Z", "metrics": { "reactions": 1, "comments": 1 }, "labels": [ "core", "go", "community-contribution" ], "author": "chiayi", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:e382f866a1758851c75a", "signalId": "github:ray-project/ray:pull_request:65072", "event": "discovered", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65072", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[core] feat(rdt): enable driver-side ray.put with NIXL tensor transport", "text": "Allow ray.put(value, _tensor_transport='nixl') from a driver process. Previously this failed because get_runtime_context().current_actor raises RuntimeError in a non-actor context. Changes: - Make RDTMeta.src_actor and TransferMetadata.src_actor Optional - Catch RuntimeError in put_object and set src_actor=None for driver - Reject two-sided transports (NCCL/GLOO) from driver with clear error - Guard src_actor dereferences in _abort_transport and transfer paths - Handle free_object_primary_copy locally for driver-owned objects - Raise ValueError for _use_object_store=True on driver-owned RDT objects - Align worker.py error to RuntimeError with 'one-sided transport' message Fixes ray-project/ray#64714 ## Tests Three new tests in `python/ray/tests/rdt/test_rdt_nixl.py`: ### `test_driver_put_nixl` End-to-end driver-side NIXL flow: 1. `ray.put(tensor, _tensor_transport=\"nixl\")` from the driver 2. Verify `src_actor is None` in RDT metadata 3. Verify tensor registered in NIXL transport 4. Actor consumes the driver-created ref via `actor.sum.remote(ref)` 5. Driver-local `ray.get(ref)` returns the original tensor 6. After `del ref` + `gc.collect()`, verify cleanup: tensor freed, RDT metadata removed, NIXL metadata removed, tensor descriptor cache cleared ### `test_driver_owned_rdt_rejects_object_store_fallback` Verify `_use_object_store=True` on a driver-owned RDT object raises `ValueError` matching `_use_object_store=True` instead of crashing with `AttributeError` on `None.__ray_call__`. ### `test_driver_rejects_two_sided_transport` Verify `ray.put(tensor, _tensor_transport=\"nccl\")` from the driver raises `RuntimeError` matching `one-sided transport`. Two-sided transports require an actor context to coordinate the send. ### Running ```bash # New driver tests pytest -sv python/ray/tests/rdt/test_rdt_nixl.py \\ -k \"test_driver_put_nixl or test_driver_rejects_two_sided_transport or test_driver_owned_rdt_rejects_object_store_fallback\" ``` ### Results | Test | Result | |------|--------| | `test_driver_put_nixl` | PASSED | | `test_driver_owned_rdt_rejects_object_store_fallback` | PASSED | | `test_driver_rejects_two_sided_transport` | PASSED |", "url": "https://github.com/ray-project/ray/pull/65072", "createdAt": "2026-07-28T10:28:32Z", "updatedAt": "2026-08-13T17:02:53Z", "timestamp": "2026-08-13T17:02:53Z", "metrics": { "reactions": 1, "comments": 2 }, "labels": [ "core", "community-contribution" ], "author": "n3sfan", "state": "open", "assignees": [ "stephanie-wang", "Sparks0219" ], "change": "new" } }, { "id": "event:8de8c752c439cd7c4dc3", "signalId": "github:ray-project/ray:issue:64992", "event": "discovered", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:issue:64992", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "issue", "title": "[Core] Provide a way to disable the worker-log `(pid=…)` prefix without disabling driver forwarding or overriding the root logger", "text": "### What happens With `log_to_driver=True` (the default), Ray prepends a prefix to every worker/actor stdout/stderr line it forwards to the driver: ``` (SegmentationActor pid=18654) {\"event\": \"segmenting batch\", \"job_id\": \"…\", \"level\": \"info\"} ``` Our workers emit **structured logs β€” one JSON object per line** β€” which a stdout-based collector ships to Loki/Grafana. The prepended `(… pid=…)` text means the line is **no longer valid JSON**, so structured parsing drops or mangles it. The logs are present in raw `kubectl logs` but effectively **invisible in the log backend** β€” a silent observability hole that is very expensive to notice (the pipeline \"goes quiet\" while thousands of lines land on pod stdout). The underlying issue: **Ray rewrites the bytes of a user's stdout/stderr lines by default.** Prepending text to a line silently breaks any line-oriented structured format. And there is no way to turn *only that* off. ### Why every existing option is unusable | Option | Result | |---|---| | `RAY_COLOR_PREFIX=0` | Removes only the ANSI color codes. The `(… pid=…)` text remains. | | `log_to_driver=False` | Removes the prefix β€” but also stops forwarding worker logs to the driver's stdout. They go to session files under `/tmp/ray/session_*/logs/`, which a stdout-based collector (the standard Kubernetes pattern) never sees. **The logs disappear entirely.** | | `logging_config=LoggingConfig(encoding=\"JSON\")` | Ray's own documented structured-logging recipe **pairs this with `log_to_driver=False`** (\"the redirected logs to the driver will include prefixes that made the logs not JSON parsable\") β€” i.e. it does **not** strip the prefix from forwarded logs either. Worse, it \"configures the **root loggers** for the driver process, Ray actors, and Ray tasks\" β€” it seizes the root logger, overrides the application's own logging configuration, and imposes Ray's JSON schema. For an app with an established structured-logging stack this is a non-starter β€” especially given Ray already tears down pre-configured logging handlers on import (#48846, #52371). | | `RAY_LOG_TO_STDERR=1` | Sends worker logs to the real (inherited) stderr with no prefix β€” but also dumps the raylet/GCS **C++ logs** (non-JSON) onto stderr, re-breaking the JSON stream, with known flooding problems (#47506). | So: **there is no way to keep worker logs on the driver's stdout, in the application's own format, without the prefix.** Every switch that removes the prefix either drops the logs or takes over the root logger. ### The request A first-class, **orthogonal** switch that keeps forwarding worker logs to the driver's stdout but leaves the **line content untouched** β€” for example: ```python ray.init(log_to_driver=True, worker_log_prefix=False) # or an env var, set before ray.init: # RAY_DISABLE_WORKER_LOG_PREFIX=1 ``` Log **forwarding** (a transport concern) and worker **attribution** (pid/actor identity) should not be entangled with the **byte content** of the user's log line. Attribution is already available structurally via `ray.get_runtime_context()` (`get_actor_id()` / `get_worker_id()` / `get_task_id()`); applications that want it can add it as a field or label. It should not be force-injected into the text of every line. Minimally: provide an opt-out for the prefix that does **not** require adopting `LoggingConfig` or disabling driver forwarding. More generally β€” a reasonable default would be to **not mutate the content of user log lines**; if the prefix stays the default, make it trivially disable-able in isolation. ### The switch already exists internally β€” it's just not exposed The prefix is added by `print_worker_logs()`, which already takes an `ignore_prefix` argument that skips it entirely (`python/ray/_private/worker.py`, v2.49.0): ```python def print_worker_logs(data, print_file, ignore_prefix: bool = False): # :2189 ... if ignore_prefix: # :2272 ... # emit the line unmodified, no \"(name pid=…)\" wrapper ``` But the only thing that ever sets it `True` is the presence of a job logging config β€” in both places that stream worker logs to the driver: ```python ignore_prefix = global_worker.job_logging_config is not None # :2632 ... ignore_prefix = worker.job_logging_config is not None # :2683 ``` So `ignore_prefix` is `True` **iff** a `LoggingConfig` is set β€” i.e. the capability we want is already implemented and working, but it can only be reached through the one option that also seizes the root logger. **Decoupling these two solves the whole problem with essentially no new machinery**: let `ignore_prefix` be set directly β€” a `ray.init(..., worker_log_prefix=False)` parameter, or an env var checked at lines 2632/2683 β€” instead of deriving it solely from `job_logging_config`. ### Environment - Ray 2.49.0, Python 3.13, Linux (Kubernetes / AKS) - Workers emit newline-delimited JSON to stdout; Grafana Alloy ships pod stdout to Loki - Current workaround: a regex `stage.replace` in the Alloy pipeline to strip the prefix before ingest β€” i.e. we are paying to undo, downstream, a mutation Ray made with no way to disable it. ### Related - #48846 β€” importing `ray.data` closes custom logging handlers - #52371 β€” import overwrites the ray logger; handlers deleted on init when `configure_logging=False` - #47506 β€” `RAY_LOG_TO_STDERR` floods the job API with head-node C++ logs - #21707 β€” support for directing *all* logs to stdout - #17467 β€” stdout logging for Ray head and worker nodes", "url": "https://github.com/ray-project/ray/issues/64992", "createdAt": "2026-07-24T14:07:54Z", "updatedAt": "2026-08-13T17:00:26Z", "timestamp": "2026-08-13T17:00:26Z", "metrics": { "reactions": 0, "comments": 6 }, "labels": [ "usability", "core", "community-backlog" ], "author": "Liquidmasl", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:6ffc750d3cb5f184f69f", "signalId": "github:ray-project/ray:issue:45751", "event": "discovered", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:issue:45751", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "issue", "title": "Ray Dashboard is susceptible to a Local File Inclusion bug with default settings", "text": "### What happened + What you expected to happen **Description** Just as the title suggests, when initially starting up the Ray dashboard I can parse my local file system by supplying the `folder` HTTP parameter in GET requests to the `/logs` API endpoint in conjunction with relative file paths. **Expected Behavior** As a best practice, relative file paths should not be permitted, or should be ignored. Basic access controls should be implemented to prevent unauthorized users from being able to parse the underlying file system being used by Ray. **Useful Information** 1) Taking a cursory glance at the Ray GH Issues board, it appears that this issue was discussed (and addressed) [here](https://github.com/ray-project/ray/issues/39701) and [here](https://www.anyscale.com/blog/update-on-ray-cves-cve-2023-6019-cve-2023-6020-cve-2023-6021-cve-2023-48022-cve-2023-48023) 2) As seen in the provided script, the `nodeId` HTTP parameter must be included in GET requests. ![image](https://github.com/ray-project/ray/assets/45905457/2eeed262-caa0-4e0e-9f32-5a9604be16f6) ### Versions / Dependencies 2.9.1 and 2.23.0 ### Reproduction script from requests import get url = 'http://``:8265/#/logs' headers = {'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:126.0) Gecko/20100101 Firefox/126.0', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8', 'Accept-Encoding': 'gzip, deflate, br, zstd', } params = {'folder': '../../../../home', 'nodeId': '``' } try: resp = get(url=url, params=params, headers=headers) except Exception as e: print(f'GET request failed: {str(e)}') if resp.status_code != 200: print('ERROR server responded with: {resp.status_code}, {resp.content}') else: print(f'Response\\n==========\\n{resp.content}') ### Issue Severity Medium: It is a significant difficulty but I can work around it.", "url": "https://github.com/ray-project/ray/issues/45751", "createdAt": "2024-06-05T12:04:49Z", "updatedAt": "2026-08-13T16:46:16Z", "timestamp": "2026-08-13T16:46:16Z", "metrics": { "reactions": 0, "comments": 9 }, "labels": [ "bug", "P2", "dashboard", "observability", "security" ], "author": "rmhowe425", "state": "closed", "assignees": [ "rmhowe425" ], "change": "new" } }, { "id": "event:9acef0b32fdf0d0dfaa0", "signalId": "github:ray-project/ray:pull_request:65453", "event": "changed", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [ "updatedAt" ], "signal": { "id": "github:ray-project/ray:pull_request:65453", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[Data] Add support for writing ORC files", "text": "## Description Ray Data can read ORC files, but it can't write them. This adds `Dataset.write_orc()`, backed by a new `ORCDatasink` that writes each output block with `pyarrow.orc.write_table`. The API forwards PyArrow ORC writer options and supports `min_rows_per_file` through the existing block-based file datasink infrastructure. ## Related issues Follow-up to #64600 ## Additional information Mirrors the `write_csv` / `write_json` APIs. PyArrow ORC writer options can be passed directly to `write_orc()`. No new dependency (`pyarrow` already ships `pyarrow.orc`). Adds write tests in `test_orc.py` and API docs in `saving_data.rst`. ```python import ray ds = ray.data.range(100) ds.write_orc(\"s3://bucket/path/\", compression=\"zstd\")", "url": "https://github.com/ray-project/ray/pull/65453", "createdAt": "2026-08-13T09:53:45Z", "updatedAt": "2026-08-13T16:44:39Z", "timestamp": "2026-08-13T16:44:39Z", "metrics": { "reactions": 1, "comments": 1 }, "labels": [ "docs", "data", "community-contribution" ], "author": "WFY123wfy", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:f6d18e01d48f53848222", "signalId": "github:ray-project/ray:pull_request:65460", "event": "changed", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [ "text", "updatedAt" ], "signal": { "id": "github:ray-project/ray:pull_request:65460", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[doc] Render the Jobs API spec with sphinxcontrib-openapi instead of ReDoc", "text": "> **Stacked PR.** Targets `doc-1513-declare-openapi-303` (#65458), which targets #65457. Review order is #65457 β†’ #65458 β†’ this. GitHub retargets automatically as each merges. > > **Conflicts with #65456 by design.** That PR pins `redoc_uri` to a specific ReDoc version; this one deletes `redoc_uri` entirely. If #65456 lands first I'll resolve the conflict here. If this lands, #65456 becomes moot and should be closed. Either order works β€” #65456 is worth keeping as the safe fix in case this one stalls in review. ## Why this change is needed `sphinxcontrib-redoc` is unmaintained. 1.6.0 is the latest release and dates from 2020; the `2.0.0a1` alpha still imports `pkg_resources`. Keeping it costs this repo three things: | Cost | Location | | --- | --- | | A `setuptools<82` ceiling, because the extension imports `pkg_resources` at module load and setuptools 82 removed it | `doc/requirements-doc.txt` | | A `misc.copy_overwrite` entry in `suppress_warnings`, because the extension redundantly copies its own bundled asset | `doc/source/conf.py` | | The renderer fetched from a third-party CDN at page-view time | `doc/source/conf.py` | All of that buys exactly one page. The Jobs API spec is the only consumer: `conf.py` had one `redoc` entry and there's no other OpenAPI document in the repo. There's a second, larger reason. **The current page is a client-rendered JavaScript island.** ReDoc has no source file at all β€” the extension synthesizes `api.html` from `conf.py`. So today the page: - is in **no toctree**, and - is reachable only through a hardcoded `` `... `_ `` link at `rest.rst:94`, and - has **none of its endpoint content in the built HTML**, so it can't participate in Sphinx search or cross-referencing. ## Related changes Fourth in the chain above. Supersedes the \"record why we stay on ReDoc\" PR (#65459), which I'll close β€” its conclusion is obsolete now that this works. ## What changes - Swap `sphinxcontrib.redoc` β†’ `sphinxcontrib.openapi` (0.9.0) in `conf.py` and `doc/requirements-doc.txt`, and regenerate the deplock. - Add a real `api.md`, put it in the `job-submission` toctree, and replace the raw `` link with a `:ref:`. - Drop the now-unneeded `misc.copy_overwrite` suppression. - Change two operation `summary` fields from YAML block scalars to plain scalars (see below). - Update the `setuptools` comment: the `<82` ceiling's cause is gone. The pin itself is left in place, since unpinning is a separate resolve worth verifying on its own. `sphinxcontrib-openapi` is pure Python, from the same `sphinx-contrib` org, and needs no `pkg_resources`. The deplock diff is `-sphinxcontrib-redoc` and `+sphinxcontrib-openapi` plus its five dependencies (`sphinxcontrib-httpdomain`, `sphinx-mdinclude`, `deepmerge`, `mistune`, `picobox`). ## Verification Built with `make -C doc rtd-build`, which runs the RtD-faithful clean build with `-W`, matching `fail_on_warning: true`: ``` build succeeded. ``` Exit 0, and **zero `WARNING:`/`ERROR:` lines in the whole log**. `raydepsets --check` on `docbuild_depset_311` reports \"Lock files are up to date.\" Checked against the built `api.html` rather than inferred: | Check | Result | | --- | --- | | Request/response schemas rendered | Yes β€” `Request JSON Object` and `Response JSON Object` field lists, with `entrypoint`, `runtime_env`, `metadata`, and the rest, including types and required flags | | `$ref` and `allOf` resolution | Resolved, including through `JobDetails` | | Literal `**` on the page | None β€” zero docutils `problematic` nodes | | `http-routingtable` index | Generated | | Page in the sidebar, with prev/next pagination | Yes | | MyST page equivalent to the `.rst` draft it replaced | Yes β€” 8 httpdomain operation objects, identical `Parameters` / `Request JSON Object` / `Response JSON Object` / `Status Codes` field lists, no leaked rST, and both `:ref:` directions resolving with the right link text | ## Three non-obvious configuration details All three are commented at their sites in the diff, because all three fail quietly. **The renderer is selected via `openapi_default_renderer`, not by the directive spelling.** Writing `.. openapi:httpdomain::` in the page adds an `unknown directive name` warning, which is fatal under `fail_on_warning`. Selecting it in `conf.py` and using a plain `.. openapi::` avoids that. **The default renderer emits no request or response schemas at all** β€” just paths, parameters, and status codes. So leaving `openapi_default_renderer` unset would silently drop most of the reference content while still building green. That's the trap worth knowing about if anyone touches this later. **The `openapi` directive can't be a bare MyST fence; it has to be wrapped in `eval-rst`.** `sphinxcontrib-openapi` emits reStructuredText into a `ViewList` and hands it to `nested_parse`, but MyST's `nested_parse` renders that content as Markdown. A ```` ```{openapi} ```` fence therefore builds green with zero warnings while silently degrading the entire page: every `.. http:get::` and `:resjson:` line renders as literal text, so nothing becomes an httpdomain object and the page has no field lists, no anchors, and no search or cross-referencing. That's a worse failure than the one above, and just as quiet. ## Why the summary fields change `sphinxcontrib-openapi`'s renderer interpolates an operation's `summary` into `**...**` without stripping it. A YAML block scalar (`summary: |`) leaves a trailing newline in the value, so the markup opens on one line and closes on the next, docutils can't pair it, and readers see literal asterisks. Filed upstream as [sphinx-contrib/openapi#173](https://github.com/sphinx-contrib/openapi/issues/173). Two operations used block scalars (`GET /api/version` and `POST /api/jobs`); the other six already used plain scalars. Converting those two is the whole fix and changes no rendered text. ## What this gives up, honestly **Request and response samples.** ReDoc rendered example payloads; this doesn't. It isn't a configuration choice β€” both routes to examples are blocked by upstream bugs, and I've confirmed each against this spec: - `:generate-examples-from-schemas:` crashes with `ValueError: dictionary update sequence element #0 has length 1; 2 is required` ([#166](https://github.com/sphinx-contrib/openapi/issues/166)). Example generation can't handle an `allOf` that targets a non-object schema, which is exactly how `JobDetails` attaches descriptions to the `JobType` and `JobStatus` string enums. - `:response-examples-for:` is rejected as an unknown option ([#165](https://github.com/sphinx-contrib/openapi/issues/165)). `api.md` carries a comment recording this so nobody re-derives it, and I've added the reproduction detail to both upstream issues. Weighed against gaining search, cross-referencing, navigation, a dependency that's actually maintained, and no third-party CDN at page-view time, losing sample payloads seemed like the right trade β€” but it is a real loss, and it's reasonable to want the upstream fixes first instead. I've also filed [#171](https://github.com/sphinx-contrib/openapi/issues/171), [#172](https://github.com/sphinx-contrib/openapi/issues/172), and [#173](https://github.com/sphinx-contrib/openapi/issues/173), each with a minimal reproducer, and am planning PRs for them. ## Checks - [x] `make -C doc rtd-build`: build succeeded, zero warnings. - [x] `raydepsets --check`: lock files up to date. - [x] Built page inspected for schema content, `allOf`/`$ref` resolution, and stray markup. - [x] `pre-commit run` clean. - [ ] Read the Docs preview reviewed for the appearance change.", "url": "https://github.com/ray-project/ray/pull/65460", "createdAt": "2026-08-13T14:03:32Z", "updatedAt": "2026-08-13T16:25:52Z", "timestamp": "2026-08-13T16:25:52Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [], "author": "dstrodtman", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:b91b4feb01e696b0891b", "signalId": "github:ray-project/ray:pull_request:65462", "event": "discovered", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:pull_request:65462", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[docs] Sync the vendored KubeRay CRD API reference", "text": "## Why are these changes needed? Vendors the generated KubeRay CRD API reference at `doc/source/cluster/kubernetes/references/api.md` from `ray-project/kuberay` release tag v1.6.2, pinned at commit [`0a9eabd`](https://github.com/ray-project/kuberay/blob/0a9eabde561ab59ab4bbe626072c359da9214484/docs/reference/api.md). Opened automatically. The upstream file is a build artifact that `elastic/crd-ref-docs` generates from the Go CRD types in `ray-operator/apis/ray/`, and KubeRay's `ray-operator-verify-api-docs` CI job fails on drift between the two, so the content is verified upstream before it reaches here. That job covers `release-*` branches as well as `master`, so a release tag's artifact carries the same guarantee. This syncs from the latest KubeRay *release*, not from `master`. Master carries CRD surface that no release ships, and documenting it here would advertise fields a reader can't use. The `ray.io/v1alpha1` section is stripped on the way in. That API version is deprecated and on a removal runway (ray-project/kuberay#5124), and its cross-links collide with the v1 anchors (ray-project/kuberay#5091). With one package left, the `Packages` index and the `ray.io/v1` heading restate the page's own subject, so both are dropped and the headings under them move up to h2. Sphinx's right-hand page nav lists h2 only, so this is what puts the CRD types in it. Heading text is untouched, so the anchors cross-links resolve against don't move. A page intro replaces the generated package blurb those headings carried, naming the resource types read off the page's own index. ## Checks Content-only: the single changed file is documentation under `doc/`, so the `docs-go` scope guard applies cleanly. The Read the Docs preview build on this PR is the gate β€” a green build means the regenerated reference still renders and all of its in-page anchors resolve. Nothing else in this PR. If it contains any other change, something is wrong with the sync job.", "url": "https://github.com/ray-project/ray/pull/65462", "createdAt": "2026-08-13T16:20:13Z", "updatedAt": "2026-08-13T16:23:14Z", "timestamp": "2026-08-13T16:23:14Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "docs", "docs-go" ], "author": "dstrodtman", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:5a59445acd02eb85db01", "signalId": "github:ray-project/ray:issue:64643", "event": "discovered", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [], "signal": { "id": "github:ray-project/ray:issue:64643", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "issue", "title": "[serve] Enable mypy type checking on ray/serve to catch real production bugs (e.g. the rank-corruption bug in #64181)", "text": "### What happened + What you expected to happen ### Description Ray Serve's controller code is fully type-annotated but never type-checked. The mypy pre-commit hook covers an allowlist of exactly 7 files repo-wide β€” from all of Serve, only `python/ray/serve/handle.py` and a handle-typing fixture: # .pre-commit-config.yaml - id: mypy args: ['--follow-imports=skip', '--ignore-missing-imports'] files: | (?x)^( python/ray/autoscaler/node_provider.py| ... python/ray/serve/handle.py| python/ray/serve/tests/typing_files/check_handle_typing.py ) Additionally, `--follow-imports=skip` degrades every cross-module type to `Any`, so even allowlisted files get almost no protection from wrong-type arguments: any parameter annotated with an imported class effectively becomes untyped. ### Motivating case: a production bug mypy flags exactly The rank-corruption bug behind #63862 (fixed in ) was a one-liner in `deployment_state.py` β€” the lightweight-reconfigure path passed `rank=current_rank.rank` (a bare `int`) where the callee declares `rank: ReplicaRank`. The int round-tripped through the replica and broke rank recovery on every controller restart, permanently corrupting the rank system (\"Rank system is in an invalid state\" loop). Running mypy 1.7.0 (the version already pinned in pre-commit) with import following enabled flags the bug precisely: deployment_state.py:3749: error: Argument \"rank\" to \"reconfigure\" of \"DeploymentReplica\" has incompatible type \"int\"; expected \"ReplicaRank\" [arg-type] With Ray's current flags (`--follow-imports=skip`), the same run reports 57 unrelated errors in the file and does NOT flag this line β€” `ReplicaRank` collapses to `Any`. So this isn't just \"add the file to the list\"; the flags matter too. The same run surfaces more latent inconsistencies in the rank system alone: deployment_state.py: error: Argument \"assign_rank_callback\" to \"start\" of \"DeploymentReplica\" has incompatible type \"Callable[[str, str], ReplicaRank]\"; expected \"Callable[[ReplicaID], ReplicaRank]\" [arg-type] (2 call sites) deployment_state.py: error: Argument 3 to \"recover_rank\" of \"DeploymentRankManager\" has incompatible type \"ReplicaRank | None\"; expected \"ReplicaRank\" [arg-type] This bug class β€” a raw value passed where a wrapper object is expected, then serialized across the controller/replica RPC boundary and only exploding much later during recovery β€” is essentially invisible to unit tests that mock the boundary, and exactly what static typing is for. ### Current scope of the debt (measured on master) `mypy --follow-imports=silent --ignore-missing-imports ray/serve/_private/`: - 1028 errors in 65 of 86 files - 237 of them are `[arg-type]` (the category that catches bugs like the above) ### Proposal (incremental, no big-bang) 1. Add a Serve-scoped mypy CI step (or extend the pre-commit hook) with `--follow-imports=silent` so imported types resolve, scoped initially to an error-free subset of `ray/serve/_private/` (per-file allowlist, same mechanism as today). 2. Burn down `[arg-type]` errors first in controller-critical modules (`deployment_state.py`, `replica.py`, `controller.py`, `application_state.py`) β€” these guard the state-checkpoint/recovery paths where type confusion becomes persistent state corruption. 3. Ratchet: once a file is clean, add it to the allowlist so it can't regress. ### Use case Prevent recurrence of 64181-style state-corruption bugs in Serve's controller, where type annotations already encode the contract but nothing enforces it. ### Versions / Dependencies ray[serve] 2.56.0 ### Reproduction script mypy --follow-imports=silent --ignore-missing-imports ray/serve/_private/ ### Issue Severity None", "url": "https://github.com/ray-project/ray/issues/64643", "createdAt": "2026-07-09T19:33:09Z", "updatedAt": "2026-08-13T16:22:59Z", "timestamp": "2026-08-13T16:22:59Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "bug", "triage", "serve", "tech-debt" ], "author": "johntaylor-cell", "state": "open", "assignees": [ "johntaylor-cell" ], "change": "new" } }, { "id": "event:4c629863ce1f16a8cf35", "signalId": "github:ray-project/ray:pull_request:65447", "event": "changed", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [ "metrics" ], "signal": { "id": "github:ray-project/ray:pull_request:65447", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[train] Share PlacementGroupCleaner across Train runs", "text": "## Description Ray Train v2 currently creates one detached `PlacementGroupCleaner` actor per Train controller. Since cleaners are pinned to the head node, their memory use grows linearly with concurrent Train runs. In the workload described in #65443, 24 cleaners use approximately 14.4 GB on the head node. This PR replaces the per-controller actors with one cluster-wide shared cleaner. The cleaner tracks controllers independently, so a failed controller only triggers cleanup for its own placement groups. The shared cleaner: - uses a fixed, versioned name and internal namespace with `get_if_exists=True`; - remains detached, pinned to the head node, and scheduled outside captured placement groups; - persists controller and placement-group registrations in GCS internal KV; - restores active and incomplete cleanup records after actor restart; - rejects late placement-group registration after controller death; - isolates State API and placement-group cleanup failures between controllers; - scopes State API queries to its own cluster when multiple clusters are active. ## Related issues Fixes #65443. ## Testing ```text PYTHONPATH=\"$PWD/python\" python -m pytest -q \\ python/ray/train/v2/tests/test_placement_group_cleaner.py # 15 passed PYTHONPATH=\"$PWD/python\" python -m pytest -q \\ python/ray/train/v2/tests/test_controller_callback_behaviour.py # 10 passed pre-commit run --files \\ python/ray/train/v2/_internal/callbacks/placement_group_callback.py \\ python/ray/train/v2/_internal/execution/controller/placement_group_cleaner.py \\ python/ray/train/v2/_internal/state/util.py \\ python/ray/train/v2/tests/test_placement_group_cleaner.py # Passed git diff --check origin/master...HEAD # Passed ``` A local end-to-end multi-job scenario used two independent drivers, each running a real Train v2 `DataParallelTrainer` with a controller, worker, and placement group. Both runs shared one cleaner. After one driver was killed, only its placement group was removed; the other run remained active and completed normally. The cleaner stayed alive throughout. ## Contribution notes No other open PR referencing #65443 or matching this PlacementGroupCleaner change was found before requesting review. AI assistance was used for implementation, testing, and self-review of this change.", "url": "https://github.com/ray-project/ray/pull/65447", "createdAt": "2026-08-13T05:34:32Z", "updatedAt": "2026-08-13T13:34:06Z", "timestamp": "2026-08-13T13:34:06Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "train", "community-contribution" ], "author": "OneSizeFitsQuorum", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:1d06d3ef12c1e56c8f0f", "signalId": "github:ray-project/ray:issue:65443", "event": "changed", "observedAt": "2026-08-13T17:47:07.884300Z", "changedFields": [ "updatedAt", "metrics" ], "signal": { "id": "github:ray-project/ray:issue:65443", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "issue", "title": "[Train] Share PlacementGroupCleaner across concurrent Train v2 runs", "text": "### Description Ray Train v2 currently creates one detached `PlacementGroupCleaner` actor for each Train controller. The cleaner removes placement groups when its controller exits ungracefully. Since #64705, every cleaner is pinned to the head node so it does not prevent worker-node scale-down. However, this concentrates one Python cleaner process per concurrent Train run on the head node: ```text N concurrent Train runs -> N detached PlacementGroupCleaner processes ``` In our production workload, each `PlacementGroupCleaner` process consistently uses approximately 600 MB of memory. With 24 concurrent Train jobs, the cleaner processes alone consume approximately 14.4 GB on the head node. The cleaner's responsibility is cluster-level bookkeeping and does not require a dedicated process for every Train run. We propose replacing the per-controller cleaners with one cluster-wide shared cleaner: ```text 24 concurrent Train runs -> 1 shared PlacementGroupCleaner process ``` This makes the cleaner process and memory cost constant instead of growing linearly with the number of concurrent Train jobs. ### Proposed implementation Create one cluster-wide `PlacementGroupCleaner` with: - a fixed, versioned actor name; - a fixed internal namespace; - `get_if_exists=True`; - `lifetime=\"detached\"`; - head-node resource pinning; - `scheduling_strategy=\"DEFAULT\"`; - automatic actor restart. Every Train controller registers itself and its placement groups with the shared actor. The cleaner maintains controller-scoped records: ```python controller_actor_id -> placement_group_ids ``` The lifecycle is: - controller starts: register the controller with the shared cleaner; - worker group starts: register its placement group under that controller; - controller shuts down normally: unregister only that controller without stopping the shared cleaner; - controller dies unexpectedly: remove only the placement groups registered to that controller; - the registry becomes empty: keep the cleaner alive to avoid a race between actor shutdown and a new Train job registering. The actor name is versioned, for example: ```text train_v2_placement_group_cleaner_v1 ``` This prevents a newer Ray version from accidentally reusing an incompatible shared actor created by an older implementation. ### Reliability Sharing the cleaner must not weaken the cleanup guarantees of the current per-controller implementation. The implementation therefore handles the following cases: 1. **Concurrent creation across jobs** Controllers use the same actor name and namespace with `get_if_exists=True`. Concurrent Train jobs resolve to the same actor. 2. **Controller isolation** Placement groups are stored by controller ID. A failed controller causes only its own placement groups to be removed. 3. **Registration and controller-death races** A controller is atomically marked as being cleaned before its placement groups are processed. A delayed placement-group registration is rejected, and the controller callback removes that placement group directly. 4. **Cleaner restart recovery** Controller IDs, placement-group IDs, and cleanup state are checkpointed in GCS internal KV. After an unexpected cleaner restart, the actor restores the registry and resumes incomplete cleanup. This avoids losing cleanup responsibility for all active Train runs when the shared process restarts. 5. **Partial cleanup failures** Cleanup state is removed only after every associated placement group has been removed successfully. Failed removals remain persisted and are retried. 6. **Failure isolation** A State API failure or placement-group cleanup failure for one controller does not stop the cleaner from processing other controllers. ### Prototype We have implemented and tested a Python-only prototype based on the current Ray `master`. It does not require Core Worker or C++ changes. The prototype modifies: - `placement_group_callback.py` - `placement_group_cleaner.py` - `test_placement_group_cleaner.py` It preserves the existing scheduling behavior introduced by #64705: - detached lifetime; - head-node pinning; - `DEFAULT` scheduling to escape placement-group capture. ### Test coverage The tests cover: - concurrent singleton creation; - singleton reuse across independent Ray drivers/jobs; - multiple controllers and multiple placement groups; - cleanup isolation between controllers; - normal controller deregistration; - duplicate registration, deregistration, and monitoring startup; - placement-group registration versus controller-death races; - temporary State API failures; - individual placement-group cleanup failures; - cleaner restart with active registrations; - cleaner restart during incomplete cleanup; - fixed actor name and namespace; - detached lifetime; - head-node pinning; - `DEFAULT` scheduling; - existing controller callback behavior. Current local results against Ray `master`: ```text 13 placement-group cleaner tests passed 10 controller callback regression tests passed Ruff check passed Ruff format check passed git diff --check passed ``` ### Expected impact For concurrent Train workloads, this changes the number of cleaner processes from O(number of Train runs) to O(1). For our workload: ```text Before: 24 cleaner processes x approximately 600 MB = approximately 14.4 GB After: 1 shared cleaner process x approximately 600 MB = approximately 600 MB ``` This reduces the observed head-node cleaner memory footprint by approximately 13.8 GB while retaining controller-scoped placement-group cleanup and restart recovery. ### Related work - #58515 introduced `PlacementGroupCleaner`. - #61756 fixed cleaner lifecycle behavior. - #62754 fixed the placement-group registration/controller-death race. - #64705 pinned cleaners to the head node and noted that N concurrent Train runs result in N cleaner pollers on the head node. This proposal addresses the remaining per-run process and memory scaling cost.", "url": "https://github.com/ray-project/ray/issues/65443", "createdAt": "2026-08-13T01:52:13Z", "updatedAt": "2026-08-13T17:44:36Z", "timestamp": "2026-08-13T17:44:36Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "train", "performance", "community-backlog" ], "author": "OneSizeFitsQuorum", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:d63f93808bb0fac86a79", "signalId": "github:ray-project/ray:pull_request:65239", "event": "changed", "observedAt": "2026-08-13T17:47:07.884300Z", "changedFields": [ "updatedAt" ], "signal": { "id": "github:ray-project/ray:pull_request:65239", "source": "github", "group": "ai-infrastructure", "project": "ray-project/ray", "kind": "pull_request", "title": "[docs] Add Kubernetes and KubeRay conventions to the style guide", "text": "## Why this change Ray on Kubernetes docs describe Kubernetes API objects alongside Ray concepts, and the two vocabularies collide. Deployment, Job, and Service each name both a Kubernetes API object and a distinct Ray concept, so capitalization is the only thing telling them apart, and the style guide didn't say which form to use where. A contributor asking \"RayCluster or Ray cluster?\" had to derive the answer from two guides and adjudicate between them. This adds a `Writing Ray on Kubernetes and KubeRay docs` section to the style guide, scoped to the user-facing pages under `doc/source/cluster/kubernetes/` and the contributor-facing pages in [ray-project/kuberay](https://github.com/ray-project/kuberay). The approach is to extend this guide with a Kubernetes deference section rather than stand up a separate KubeRay style guide. The two guides already overlap heavily, and a forked guide would drift within a release or two. ## What the casing scan found The term list isn't invented. It comes from a prose-only scan of three corpora: 79 user-facing pages in this repo, 12 contributor pages in `ray-project/kuberay`, and 128 pages of `kubernetes/website` (`content/en/docs/concepts/`, commit `bb38117c0`). The scan strips front matter, fenced code blocks, inline code, MyST roles, and link targets before counting, so YAML field names and CLI output don't inflate the figures. Headline: **these docs already follow the Kubernetes casing rule.** Upstream Kubernetes capitalizes \"Pod\" in 71% of prose uses; the Ray-side KubeRay docs do so in 70%. So this PR documents a rule the docs mostly already follow, and does not propose a capitalization sweep. | Term | Ray-side KubeRay docs | Upstream Kubernetes | Read | |---|---|---|---| | Pod | 371/527 (70%) | 3712/5195 (71%) | Matches upstream almost exactly. | | ConfigMap | 20/21 (95%) | 119/130 (91%) | Healthy. | | Ingress | 8/15 (53%) | 148/242 (61%) | Roughly aligned. | | Service | 18/114 (15%) | 515/946 (54%) | Diverges by design. | | Job | 17/143 (11%) | 522/610 (85%) | Diverges by design. | | Deployment | 6/53 (11%) | 355/461 (77%) | Diverges by design. | The three \"diverges by design\" rows are the reason this section exists. Sampling shows the lowercase uses are Ray Serve deployments, Ray jobs, and Ray services, not Kubernetes API objects. That's a naming collision, not drift, so the fix is a disambiguation rule rather than a sweep. Kubernetes's own rule expects a mix (UpperCamelCase for the API object, sentence case for the general concept), which is why the upstream column isn't 100% either. The one genuine inconsistency the scan found is **\"Ray Autoscaler\" (44 uses) against \"Ray autoscaler\" (35)**, spread across at least eight files. This guide already says to lowercase generic nouns even when they name a Ray concept, so lowercase wins. Recording the rule here is the prerequisite for a mechanical follow-up cleanup. I've left that out of this PR to keep the rule change reviewable on its own. ## Open questions for reviewers I've deliberately written only the rules I can support from the two published guides plus the scan. These five are genuinely open, and I'd rather resolve them in review than assert them: 1. ~~**Confirm the custom-resource split.**~~ **Resolved** by @win5923: the split is intentional and consistent with Kubernetes terminology. Original question: The section documents `RayCluster` (the resource) versus \"Ray cluster\" (the running thing it produces), currently 335 against 106 uses in this repo. I read it as intentional and matching the Kubernetes API-object rule. Worth confirming rather than normalizing to one form. 1. ~~**\"head Pod\" versus \"head node.\"**~~ **Resolved** by @win5923: use \"head Pod\" consistently when a Ray cluster runs on Kubernetes. Original question: The section prefers \"head Pod\" for the Kubernetes-hosted case, since a node is a machine on Kubernetes. `doc/source/cluster/kubernetes/index.md` currently says \"a head node pod and a collection of worker node pods,\" which stacks both vocabularies. Reasonable, or is \"head node\" worth keeping for continuity with Ray's core architecture docs? 1. **Feature graduation phases.** Kubernetes uses start case (\"DRA is Beta\"); Ray docs lowercase feature stages. This bites on pages discussing a Kubernetes feature's stage and a Ray feature's stage in one paragraph. I left it out of the section rather than pick. Which wins? 1. **Field values.** Kubernetes wants unquoted, unstyled values (\"set `imagePullPolicy` to Always\"); this guide says to use backticks for configuration values and gives `num_replicas: 1`. These are closer than they look, since one is a bare value and the other a key-value pair, but the guidance should say which form applies where. 1. **\"Ray dashboard\" versus \"Ray Dashboard\"** (43 title-case against 18 lowercase). This guide implies lowercase, but the dashboard is arguably a named UI surface. Not addressed in the section pending a call. Two further items are out of scope here but worth flagging: - **Line wrapping in the `kuberay` repo.** This guide requires soft wrap. The Kubernetes guide requires manual wrap, justified by localization teams diffing per line, which KubeRay doesn't have. Nothing in this PR changes that, since this guide already governs `doc/source/`. But `kuberay`'s `.markdownlint.yaml` sets `MD013` to 120 characters via pre-commit, which is a latent gate: 266 lines in its `docs/` already exceed it, it isn't enforced in CI, and it isn't auto-fixable, so contributors hit it unpredictably on whichever file they touch. Disabling it there would align the two repos, but that's a call for KubeRay maintainers in their own repo. - **Turning Vale enforcement on.** This PR now adds the ruleset but doesn't enforce it. See the section below. ## Staged Vale ruleset A later commit adds a Vale ruleset for these pages, following the pattern already used for the Ray Data docs. **Nothing is enforced yet.** The Vale hook in `.pre-commit-config.yaml` stays scoped to `^doc/source/data/.*`, so CI checks exactly what it checked before. - `.vale/styles/config/vocabularies/KubeRay/accept.txt` β€” the KubeRay custom resources, the Kubernetes and ecosystem API objects these pages reference, and project names. Skips what the `General` vocabulary already accepts (`Kubernetes`, `kubectl`, `CRD(s)?`, `namespace`). - `.vale/styles/KubeRay/Terms.yml` β€” a `substitution` rule at `warning` level covering only the casing forms the new style-guide section settles, with `vocab: false` (see below). The open questions above are deliberately absent. - `.vale.ini` β€” registers the vocabulary and scopes the style to `doc/source/cluster/kubernetes/`. Registering a vocabulary that the hook doesn't exercise matches existing practice here: `RLlib` and `Train` are already in `Vocab` while the hook only runs on `doc/source/data/`. Contributors can run it by hand today: ``` vale doc/source/cluster/kubernetes/ ``` To enforce later, widen the `files` pattern on the `vale` hook. I'd still hold that until the open questions land. ### What I verified Vale 3.12.0 locally. **Path scoping works.** The KubeRay rules fire on a file under `doc/source/cluster/kubernetes/` and stay silent on identical content at the repo root. **Current state of these pages** β€” 146 `KubeRay.Terms` alerts, which lines up with the casing scan: | Alert | Count | |---|---| | Use 'Ray autoscaler' instead of 'Ray Autoscaler' | 42 | | Use 'head Pod' instead of 'head pod' | 20 | | Use 'worker Pod' instead of 'worker pod' | 7 | | Use 'Ray job' instead of 'ray job' | 2 | | Use 'Ray Serve' instead of 'Ray serve' | 1 | | Use 'Ray cluster' instead of 'ray cluster' | 1 | | Use 'KubeRay' instead of 'Kuberay' | 1 | | Use 'GCS fault tolerance' instead of 'GCS FT' | 1 | The 42 matches the scan's count for these pages exactly, which is a useful cross-check on both. None of it is enforced, so nothing here has to be fixed in this PR. **No regression on the gated path.** Alert count on `doc/source/data/` goes 174 β†’ 173, deterministic across repeated runs in both states, with no new alerts. A `Vocab` list is global, so it can only widen what's accepted. The single alert that disappears is a `Google.Parens` hit in `concurrent-dataset-execution.md`; no term in the new vocabulary appears on that line, so I'm reporting that as observed rather than claiming I understand the mechanism. It's a reduction either way, so it can't break the gate. ### The vocabulary shadowing trap Worth knowing if you write Vale rules here: a `Vocab` accept entry shadows its token for **every** rule, case-insensitively. A substitution whose target appears in an accept list silently never fires, and nothing warns you. This bit the `Ray Autoscaler` swap, because `autoscaler` is accepted in `config/vocabularies/Data/accept.txt`. The fix is `vocab: false` on the rule, which opts out of the filtering without touching the Data vocabulary. The rule carries a comment saying so. Credit to Bugbot for catching it. I'd first shipped that swap commented out, having concluded the only way to enable it was dropping `autoscaler` from the Data vocabulary and accepting new spelling alerts on the gated path. That was wrong, and `vocab: false` avoids the tradeoff entirely. ## Added after review Two rules that review surfaced, both in the style-guide section plus the Vale ruleset. **Node stays lowercase.** Node is a Kubernetes API object, so \"capitalize Kubernetes API objects\" read literally tells contributors to write \"Kubernetes Node.\" These pages almost always mean the machine, and all 24 uses are correctly lowercase today, so a literal reading of the rule would have someone \"fix\" working prose into wrongness. Kubernetes applies the same two-part rule it applies elsewhere, capitalizing Node only for the resource, and the section now says so. **Don't write \"Ray Pod.\"** This answers the casing question raised in review, and the answer turned out to be neither casing. A Pod is a Kubernetes host that contains a Ray node plus any sidecars, so naming it after Ray overstates what it is. \"TPU Pod\" also already occupies the qualified-Pod form in these pages and means a group of TPU chips rather than a Kubernetes object. The term had no settled form to preserve either: across the 55 prose lines using it, casing split four ways at 20, 16, 16, and 15. Stating that alone would be a trap, because a bare \"Pod\" isn't always an adequate replacement, so the section gives the test that separates the cases: - Qualify when the qualifier distinguishes two referents. That's why \"Ray node\" against \"Kubernetes node\" is correct and should stay. - Don't qualify when it relabels one thing. - Keep the scope explicit where a page also describes Pods the RayCluster doesn't own, such as the operator, Redis, or a `curl` Pod, and where a nearby command filters by label. \"List all Pods\" and \"List the Ray cluster's Pods\" describe different commands and shouldn't read the same way. The cleanup is #65423, which removes all 89 uses from the docs and is reviewable on its own. This PR is the rule; that one is the sweep. Vale picks up four swaps for the retired forms. Plurals get their own entries, for the word-boundary reason @win5923 identified on the `head pod` swaps. Verified that they don't fire on \"Ray head Pod\" or \"Ray worker Pods\", and that the gated `doc/source/data/` path holds at 173 alerts. ## Not a duplicate Checked open PRs for this area before starting: ``` gh pr list --repo ray-project/ray --state open --search \"writing-style\" gh pr list --repo ray-project/ray --state open --search \"style guide in:title\" gh pr list --repo ray-project/ray --state open --search \"kuberay style\" ``` No open PR touches `doc/source/ray-contribute/writing-style.md` or proposes KubeRay style guidance. The `writing-style` matches were all false positives on `write_delta`. This extends the guide added in #64518. ## Testing ``` pre-commit run --files doc/source/ray-contribute/writing-style.md ``` Passes, but as a no-op on the Markdown: every hook reports \"no files to check,\" because no pre-commit hook applies to Markdown under `doc/source/` and Vale is scoped to `doc/source/data/`. Stating that plainly instead of claiming coverage it doesn't give. The Vale config files do have hook coverage, and those checks ran for real: ``` pre-commit run --files .vale.ini .vale/styles/KubeRay/Terms.yml .vale/styles/config/vocabularies/KubeRay/accept.txt ``` `trim trailing whitespace`, `fix end of files`, `check for added large files`, and `semgrep` all pass. Vale verification for the ruleset is in the section above: path scoping, the 31-alert current state, and the 174 β†’ 173 no-regression check on the gated path. Verified by hand: the new `(kubernetes-docs-style)=` anchor doesn't collide with any existing target in `doc/source/`, and the section uses the same MyST anchor form and Markdown pipe-table style already used elsewhere in this file. Rendering is verified by the Read the Docs preview build on this PR. No code, no API, no behavior change. The prose is a contributor guide; the Vale files add configuration without widening what CI enforces. ## AI assistance Written with AI assistance (Claude). I've reviewed every changed line, generated and inspected the casing counts myself, and can defend the change end to end. The scan scripts that produced the numbers live outside this repo; happy to share the methodology or re-run against a different corpus if a reviewer wants to check a figure. --- ### Update: the vocabulary now covers the ecosystem terms these pages use (555ebdad) Running `vale doc/source/cluster/kubernetes/`, which this PR's `.vale.ini` comment tells contributors to do, produced 215 `Vale.Spelling` alerts on correctly spelled terms. The accept list covered API objects, the custom resources, and project names, but not the wider ecosystem vocabulary the pages actually contain. Left as is, a contributor following the instruction learns to ignore Vale on exactly the paths this PR wants gated. Added: the ecosystem projects (Istio, Kiali, gVisor, Kata, Krew, Kustomize, Bottlerocket, Aliyun, GCSFuse, Promtail, Fluentd, Filebeat, Alertmanager), the Gateway API `HTTPRoute`, and the Kubernetes and infrastructure terms in use (cgroup, finalizer, kubeconfig, kubelet, namespaced, namespaces, toleration, upscaling, downscaling, vCPU, FQDN, AMI, subnet, syscall, hostname, cron, CA, autoscalers). **Excluded on purpose, since an accept list shouldn't launder a defect.** `CR` and `CRs`, 12 uses, because this PR's own guidance says to write \"custom resource\" and to define CustomResourceDefinition then use CRD. Real misspellings: `heterogenous`, `replicaset` for ReplicaSet, `Deepseek` for DeepSeek, `nodepool` for node pool. `gsutil`, which `gcloud storage` supersedes. Unbackticked code identifiers, which are a formatting defect rather than a vocabulary gap: `podSpec`, `podSelector`, `nodeSelector`, `jsonPointers`, `intervalSeconds`, `workerGroup`, `ignoreDifferences`, `HTTPMode`, `args`, `yaml`. And generic English that Vale's dictionary lacks, which belongs in the General vocabulary rather than a KubeRay-specific one. Three entries are case-tolerant (`[Tt]oleration(s)?`, `[Uu]pscaling`, `[Dd]ownscaling`). A lowercase-only entry makes `Vale.Terms` flag the legitimate sentence-start capitalization. **Verification**, deterministic across repeated runs: - `doc/source/cluster/kubernetes/`: 1329 alerts to 1178. `Vale.Spelling` drops from 215 to 72. `KubeRay.Terms` is unchanged at 160, confirming no swap got shadowed by a new accept entry, which is the trap the `Ray Autoscaler` comment in `Terms.yml` documents. - `doc/source/data/`, the only path the pre-commit hook gates: 173 to 170, with **no new alerts**. A `Vocab` is global, so it can only widen what's accepted. - One candidate entry was dropped for exactly that reason. A `runtime(s)?` entry made `Vale.Terms` flag \"Runtime\" in `doc/source/data/monitoring-your-workload.rst`, a new alert on the gated path. Four new `Vale.Terms` alerts do appear on the KubeRay paths, and all four are genuine casing defects in existing pages rather than noise: \"Kubelet\" twice in `configuring-ippr.md`, lowercase \"istio\" naming the project in `rayservice-incremental-upgrade.md`, and \"Cgroups\" in the `resource-isolation-with-writable-cgroups.md` H1, which is also title case. Those are content fixes for a separate change, not this one. AI assistance was used for this commit, including the Vale sweep and the before-and-after verification above.", "url": "https://github.com/ray-project/ray/pull/65239", "createdAt": "2026-08-05T19:50:41Z", "updatedAt": "2026-08-13T17:43:42Z", "timestamp": "2026-08-13T17:43:42Z", "metrics": { "reactions": 1, "comments": 3 }, "labels": [ "docs", "core", "go" ], "author": "dstrodtman", "state": "open", "assignees": [], "change": "updated" } } ] } }