{
  "schemaVersion": 3,
  "dataset": {
    "version": 3,
    "date": "2026-08-13",
    "group": {
      "id": "data-infrastructure",
      "name": "Data / Messaging / Storage Infrastructure"
    },
    "repository": {
      "id": "nats-server",
      "repo": "nats-io/nats-server",
      "name": "NATS Server",
      "keywords": [
        "NATS server",
        "NATS.io"
      ]
    },
    "context": {
      "repository": "nats-io/nats-server",
      "url": "https://github.com/nats-io/nats-server",
      "description": "High-Performance server for NATS.io, the cloud and edge native messaging system.",
      "homepage": "https://nats.io",
      "language": "Go",
      "topics": [
        "cloud",
        "cloud-computing",
        "cloud-native",
        "connected-vehicle",
        "distributed-systems",
        "edge",
        "edge-ai",
        "edge-computing",
        "go",
        "golang",
        "message-bus",
        "message-queue",
        "messaging",
        "microservices-architecture",
        "nats-server"
      ],
      "license": "Apache-2.0",
      "defaultBranch": "main",
      "stars": 20490,
      "forks": 1904,
      "openIssues": 541,
      "archived": false,
      "collectedAt": "2026-08-13T18:02:04.571778+00:00"
    },
    "news": {
      "repository": "nats-io/nats-server",
      "collectedAt": "2026-08-13T18:02:04.571778+00:00",
      "latestRelease": {
        "repository": "nats-io/nats-server",
        "tag": "v2.14.5",
        "title": "Release v2.14.5",
        "url": "https://github.com/nats-io/nats-server/releases/tag/v2.14.5",
        "publishedAt": "2026-08-12T15:31:27Z",
        "notes": "## Changelog\r\n\r\nRefer to the [2.14 Upgrade Guide](https://docs.nats.io/release-notes/whats_new/whats_new_214) for backwards compatibility notes with 2.12.x. Please note that the 2.13.x version was skipped.\r\n\r\n### Go Version\r\n\r\n- 1.26.5\r\n\r\n### Dependencies\r\n\r\n- github.com/klauspost/compress v1.19.2 (#8459)\r\n- golang.org/x/crypto v0.55.0 (#8459)\r\n\r\n### Added\r\n\r\nLeafnodes\r\n\r\n- New `dial_timeout` option can be specified in the `leafnode` config block or for specific remotes in the configuration, allowing it to be increased above the default 1 second for high-latency links (#8427)\r\n\r\n### Fixed\r\n\r\nGeneral\r\n\r\n- Fixed a deadlock that could occur in the logger when a close error occurs (#8430)\r\n\r\nJetStream\r\n\r\n- Fix a bug that could result in potential data loss when handling idempotent stream creates when an offline node catches up from a metalayer snapshot, caused by an incorrect update to the create time in the stream assignment (#8449)\r\n\r\n### Complete Changes\r\n \r\nhttps://github.com/nats-io/nats-server/compare/v2.14.4...v2.14.5",
        "highlights": [
          "Changelog",
          "Go Version",
          "1.26.5",
          "Dependencies",
          "github.com/klauspost/compress v1.19.2 (#8459)",
          "golang.org/x/crypto v0.55.0 (#8459)"
        ],
        "prerelease": false
      },
      "upcoming": [],
      "communityDiscussions": []
    },
    "runs": [
      {
        "collectedAt": "2026-08-13T12:26:38.318Z",
        "since": "2026-08-12T12:26:38.318Z",
        "observedCount": 19,
        "changedCount": 19
      },
      {
        "collectedAt": "2026-08-13T13:48:00.446149Z",
        "since": "2026-08-12T13:48:00.446149Z",
        "observedCount": 15,
        "changedCount": 15
      },
      {
        "collectedAt": "2026-08-13T16:19:22.035158Z",
        "since": "2026-08-12T16:19:22.035158Z",
        "observedCount": 13,
        "changedCount": 4
      },
      {
        "collectedAt": "2026-08-13T17:43:20.785491Z",
        "since": "2026-08-12T17:43:20.785491Z",
        "observedCount": 13,
        "changedCount": 3
      },
      {
        "collectedAt": "2026-08-13T17:47:07.884300Z",
        "since": "2026-08-12T17:47:07.884300Z",
        "observedCount": 13,
        "changedCount": 0
      },
      {
        "collectedAt": "2026-08-13T18:01:55.420671Z",
        "since": "2026-08-12T18:01:55.420671Z",
        "observedCount": 13,
        "changedCount": 0
      }
    ],
    "signals": [
      {
        "id": "github:nats-io/nats-server:issue:5811",
        "source": "github",
        "group": "data-infrastructure",
        "project": "nats-io/nats-server",
        "kind": "issue",
        "title": "Always retain last message for each subject in a stream",
        "text": "### Proposed change Add a `--keep-last-msg-per-subject` flag to the nats server configuration i.e. `nats stream add --keep-last-msg-per-subject <more-options> <stream>` that works with all limits-based message retention policies (i.e time-based, size-based, count-based). ### Use case Stream limits options (e.g. `max-age`, `max-bytes`, `max-msgs` etc.) mostly seem to operate on a per stream basis. The exception is the `max-msgs-per-subject` which allows for a per subject based limit. It would be nice to have the capability to always retain the last message on all given subjects in a stream while setting a time-based message retention policy i.e. delete all messages past an age of 10 mins except the last. A use case of this can be as follows: - We want to record changes to a configuration (stored in a nats subject) and keep a history for a certain period of time e.g. 5 days (specified by the --max-age flag) - We always want to be able to fetch the last configuration to initialize our long-running application (which may run for weeks on end) whenever it restarts ### Contribution Yes, I can contribute to this feature",
        "url": "https://github.com/nats-io/nats-server/issues/5811",
        "createdAt": "2024-08-20T19:33:25Z",
        "updatedAt": "2026-08-13T08:57:16Z",
        "timestamp": "2026-08-13T08:57:16Z",
        "metrics": {
          "reactions": 0,
          "comments": 2
        },
        "labels": [
          "stale",
          "proposal"
        ],
        "author": "SomeshDaga",
        "state": "open",
        "assignees": []
      },
      {
        "id": "github:nats-io/nats-server:issue:6206",
        "source": "github",
        "group": "data-infrastructure",
        "project": "nats-io/nats-server",
        "kind": "issue",
        "title": "JetStream stream source stalls/is out of sync after sourced stream is re-created for unrelated reason [v2.10.20]",
        "text": "### Observed behavior Setup: basic hub spoke with leafnodes. hub is a single jetstream enabled server in its own domain, so are the leaf nodes. On the hub, there is a stream named match_events that sources from the match_events stream in each leaf node. Symptoms: Sourcing from match_events on one leafnode stopped working. Despite new messages being added to it, no data is sent to match_events in the hub. Cause: I believe we ran into a crash during recovery that sounds similar to one fixed in a release candidate of 2.10.23. In order to recover from this issue, we renamed the data directory, which resulted in data loss but allowed the system to become functional again. On the leaf node with the issue, sequence numbers in the stream are less than 80. Create time is right after the time when the jetstream data directory got renamed and everything started to seemingly function again. On the hub node, stream info showed the edge connected with low lag. But when subscribing to the consumer create message of that leaf, we noticed that the requested start seq no is 995 and hence way bigger than 80. Meaning 995 was the sequence number the previous incarnation of match_events had on that leaf and 79 is the value of the current incarnation This would explain why there seems to be no data flow ... yet. Failed workaround: We edited the match_events stream in the hub to remove the source corresponding to the leaf. Then we re-added it. This did not seem to work as we recorded the consumer create messages and the sequence number remained at the old value it had before the source got removed & re-added. I then edited the source to set the op_start_seq to 1. This didn't work either, as presumably a higher sequence number was still in memory. A few hours later we restarted the hub server. After that the sequence number was reset to one. ---------------------- Logs: Hub consumer create messages: ``` [#13] Received on \"$JS.edge-652efb2da425e854a1e8d203.API.CONSUMER.CREATE.match_events\" with reply \"$JSC.R.FEkNK9Is\" {\"stream_name\":\"match_events\",\"config\":{\"deliver_policy\":\"by_start_sequence\",\"opt_start_seq\":995,\"ack_policy\":\"none\",\"ack_wait\":79200000000000,\"max_deliver\":1,\"replay_policy\":\"instant\",\"idle_heartbeat\":1000000000,\"flow_control\":true,\"deliver_subject\":\"deliver.edge-652efb2da425e854a1e8d203.match_events.S.l2lkXkVm\",\"inactive_threshold\":10000000000,\"num_replicas\":0,\"direct\":true},\"action\":\"\"} [#21] Received on \"$JS.edge-652efb2da425e854a1e8d203.API.CONSUMER.CREATE.match_events\" with reply \"$JSC.R.3L70SiTW\" {\"stream_name\":\"match_events\",\"config\":{\"deliver_policy\":\"by_start_sequence\",\"opt_start_seq\":995,\"ack_policy\":\"none\",\"ack_wait\":79200000000000,\"max_deliver\":1,\"replay_policy\":\"instant\",\"idle_heartbeat\":1000000000,\"flow_control\":true,\"deliver_subject\":\"deliver.edge-652efb2da425e854a1e8d203.match_events.S.qyIUgE5D\",\"inactive_threshold\":10000000000,\"num_replicas\":0,\"direct\":true},\"action\":\"\"} ``` Leafnode match_events stream: ``` > nats stream ls -a ╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ Streams │ ├─────────────────────┬────────────────────────────────────────────┬─────────────────────┬───────────┬─────────┬──────────────┤ │ Name │ Description │ Created │ Messages │ Size │ Last Message │ ├─────────────────────┼────────────────────────────────────────────┼─────────────────────┼───────────┼─────────┼──────────────┤ ... │ match_events │ │ 2024-11-27 17:53:49 │ 79 │ 15 MiB │ 25m10s │ ... ╰─────────────────────┴────────────────────────────────────────────┴─────────────────────┴───────────┴─────────┴──────────────╯ > nats stream info match_events Information for Stream match_events created 2024-11-27 17:53:49 Subjects: faceio.match.> Replicas: 1 Storage: File Options: Retention: Limits Acknowledgments: true Discard Policy: Old Duplicate Window: 2m0s Allows Msg Delete: true Allows Purge: true Allows Rollups: false Limits: Maximum Messages: unlimited Maximum Per Subject: unlimited Maximum Bytes: unlimited Maximum Age: 7d0h0m0s Maximum Message Size: unlimited Maximum Consumers: unlimited Cluster Information: Name: edge-652efb2da425e854a1e8d203 Leader: edge-652efb2da425e854a1e8d203 State: Messages: 79 Bytes: 15 MiB First Sequence: 1 @ 2024-11-27 18:13:14 Last Sequence: 79 @ 2024-12-02 17:57:52 Active Consumers: 1 Number of Subjects: 79 > ``` Hub consumer create messages for failed workaround: The first message was received after re-creating the source. The second message was received after editing opt_start_seq of the source. In both cases, opt_start_seq in the request remains at the last old value. ``` >nats sub \"$JS.edge-652efb2da425e854a1e8d203.API.CONSUMER.CREATE.match_events\" 10:44:28 Subscribing on $JS.edge-652efb2da425e854a1e8d203.API.CONSUMER.CREATE.match_events [#1] Received on \"$JS.edge-652efb2da425e854a1e8d203.API.CONSUMER.CREATE.match_events\" with reply \"$JSC.R.XoBsIZ61\" {\"stream_name\":\"match_events\",\"config\":{\"deliver_policy\":\"by_start_sequence\",\"opt_start_seq\":995,\"ack_policy\":\"none\",\"ack_wait\":79200000000000,\"max_deliver\":1,\"replay_policy\":\"instant\",\"idle_heartbeat\":1000000000,\"flow_control\":true,\"deliver_subject\":\"deliver.edge-652efb2da425e854a1e8d203.match_events.S.58t1cta4\",\"inactive_threshold\":10000000000,\"num_replicas\":0,\"direct\":true},\"action\":\"\"} [#2] Received on \"$JS.edge-652efb2da425e854a1e8d203.API.CONSUMER.CREATE.match_events\" with reply \"$JSC.R.h5dA0pYu\" {\"stream_name\":\"match_events\",\"config\":{\"deliver_policy\":\"by_start_sequence\",\"opt_start_seq\":995,\"ack_policy\":\"none\",\"ack_wait\":79200000000000,\"max_deliver\":1,\"replay_policy\":\"instant\",\"idle_heartbeat\":1000000000,\"flow_control\":true,\"deliver_subject\":\"deliver.edge-652efb2da425e854a1e8d203.match_events.S.j4aQssSi\",\"inactive_threshold\":10000000000,\"num_replicas\":0,\"direct\":true},\"action\":\"\"} ``` Hub consumer create messages after hub nats server restart show opt_start_seq=1. ``` [#7] Received on \"$JS.edge-652efb2da425e854a1e8d203.API.CONSUMER.CREATE.match_events\" with reply \"$JSC.R.snChCWwx\" {\"stream_name\":\"match_events\",\"config\":{\"deliver_policy\":\"by_start_sequence\",\"opt_start_seq\":995,\"ack_policy\":\"none\",\"ack_wait\":79200000000000,\"max_deliver\":1,\"replay_policy\":\"instant\",\"idle_heartbeat\":1000000000,\"flow_control\":true,\"deliver_subject\":\"deliver.edge-652efb2da425e854a1e8d203.match_events.S.6U6FCNWD\",\"inactive_threshold\":10000000000,\"num_replicas\":0,\"direct\":true},\"action\":\"\"} 15:19:43 Disconnected due to: read tcp 127.0.0.1:32592->127.0.0.1:4222: wsarecv: An existing connection was forcibly closed by the remote host., will attempt reconnect 15:20:09 Reconnected [nats://ff-queue-user:LtJQaah4Y2PUGAUyx8_oLZ@127.0.0.1:4222] [#8] Received on \"$JS.edge-652efb2da425e854a1e8d203.API.CONSUMER.CREATE.match_events\" with reply \"$JSC.R.hwd0hutm\" {\"stream_name\":\"match_events\",\"config\":{\"deliver_policy\":\"by_start_sequence\",\"opt_start_seq\":1,\"ack_policy\":\"none\",\"ack_wait\":79200000000000,\"max_deliver\":1,\"replay_policy\":\"instant\",\"idle_heartbeat\":1000000000,\"flow_control\":true,\"deliver_subject\":\"deliver.edge-652efb2da425e854a1e8d203.match_events.S.ZoqTXdlB\",\"inactive_threshold\":10000000000,\"num_replicas\":0,\"direct\":true},\"action\":\"\"} ``` Hub match_events stream taken right after consumer create [#2] from above, showing start seqno 1 ``` Information for Stream match_events created 2024-11-15 10:45:14 Subjects: faceio.match.> Replicas: 1 Storage: File Options: Retention: Limits Acknowledgments: true Discard Policy: Old Duplicate Window: 2m0s Allows Msg Delete: true Allows Purge: true Allows Rollups: false Limits: Maximum Messages: unlimited Maximum Per Subject: unlimited Maximum Bytes: unlimited Maximum Age: 7d0h0m0s Maximum Message Size: unlimited Maximum Consumers: unlimited Replication: Sources: ... match_events, Start Seq: 1, API Prefix: $JS.edge-652efb2da425e854a1e8d203.API, Delivery Prefix: deliver.edge-652efb2da425e854a1e8d203.match_events ... Source Information: ... Stream Name: match_events Lag: 0 Last Seen: 934ms Ext. API Prefix: $JS.edge-652efb2da425e854a1e8d203.API Ext. Delivery Prefix: deliver.edge-652efb2da425e854a1e8d203.match_events ... State: Messages: 40,303 Bytes: 7.4 GiB First Sequence: 505,520 @ 2024-11-26 11:02:59 Last Sequence: 545,822 @ 2024-12-03 11:02:56 Active Consumers: 1 Number of Subjects: 37,538 ``` Leafnode nats-server logs showing restart loop: nats server basically starts and exits right away. (within same second. different from being killed because it's taking too long) ``` [15052] 2024/11/27 17:43:00.113089 [INF] Starting nats-server [15052] 2024/11/27 17:43:00.113621 [INF] Version: 2.10.20 [15052] 2024/11/27 17:43:00.113621 [INF] Git: [7140387] [15052] 2024/11/27 17:43:00.113621 [INF] Cluster: edge-652efb2da425e854a1e8d203 [15052] 2024/11/27 17:43:00.113621 [INF] Name: edge-652efb2da425e854a1e8d203 [15052] 2024/11/27 17:43:00.113621 [INF] Node: cSXjN80e [15052] 2024/11/27 17:43:00.113621 [INF] ID: NCPMUJYVKW6V742BE4GN4RH2V6LCIFTB5MESCYDKJKURJMXCVQWJQXOQ [15052] 2024/11/27 17:43:00.113671 [INF] Using configuration file: C:\\Program Files\\FaceFirst\\FaceFirst-MessageBroker-Edge\\conf\\edge.conf [15052] 2024/11/27 17:43:00.114322 [INF] Starting http monitor on 127.0.0.1:8223 [15052] 2024/11/27 17:43:00.114322 [INF] Starting JetStream [15052] 2024/11/27 17:43:00.114827 [INF] _ ___ _____ ___ _____ ___ ___ _ __ __ [15052] 2024/11/27 17:43:00.114884 [INF] _ | | __|_ _/ __|_ _| _ \\ __| /_\\ | \\/ | [15052] 2024/11/27 17:43:00.114884 [INF] | || | _| | | \\__ \\ | | | / _| / _ \\| |\\/| | [15052] 2024/11/27 17:43:00.114884 [INF] \\__/|___| |_| |___/ |_| |_|_\\___/_/ \\_\\_| |_| [15052] 2024/11/27 17:43:00.114884 [INF] [15052] 2024/11/27 17:43:00.114884 [INF] https://docs.nats.io/jetstream [15052] 2024/11/27 17:43:00.114884 [INF] [15052] 2024/11/27 17:43:00.114884 [INF] ---------------- JETSTREAM ---------------- [15052] 2024/11/27 17:43:00.114884 [INF] Max Memory: 23.94 GB [15052] 2024/11/27 17:43:00.114884 [INF] Max Storage: 1.00 TB [15052] 2024/11/27 17:43:00.114884 [INF] Store Directory: \"D:\\ProgramData\\FaceFirst\\FaceFirst-MessageBroker-Edge\\data-store\\jetstream\" [15052] 2024/11/27 17:43:00.114884 [INF] Domain: edge-652efb2da425e854a1e8d203 [15052] 2024/11/27 17:43:00.114884 [INF] ------------------------------------------- [15052] 2024/11/27 17:43:00.115438 [INF] Starting restore for stream 'FF-QUEUE-ACC > KV_enrollments' [15052] 2024/11/27 17:43:00.166680 [INF] Restored 69,478 messages for stream 'FF-QUEUE-ACC > KV_enrollments' in 51ms [15052] 2024/11/27 17:43:00.166680 [INF] Starting restore for stream 'FF-QUEUE-ACC > KV_media' [15052] 2024/11/27 17:43:00.208948 [WRN] Filestore [KV_media] loadBlock error: message block data missing [15052] 2024/11/27 17:43:00.211651 [WRN] Filestore [KV_media] Detected skew in subject-based total (64) vs raw total (58), rebuilding [15052] 2024/11/27 17:43:00.238941 [INF] Restored 58 messages for stream 'FF-QUEUE-ACC > KV_media' in 72ms [15052] 2024/11/27 17:43:00.239938 [INF] Starting restore for stream 'FF-QUEUE-ACC > KV_settings' [15052] 2024/11/27 17:43:00.239938 [INF] Restored 42 messages for stream 'FF-QUEUE-ACC > KV_settings' in 0s [15052] 2024/11/27 17:43:00.240936 [INF] Starting restore for stream 'FF-QUEUE-ACC > KV_visitors' [17012] 2024/11/27 17:45:03.817424 [INF] Starting nats-server [17012] 2024/11/27 17:45:03.818469 [INF] Version: 2.10.20 [17012] 2024/11/27 17:45:03.818469 [INF] Git: [7140387] [17012] 2024/11/27 17:45:03.818469 [INF] Cluster: edge-652efb2da425e854a1e8d203 [17012] 2024/11/27 17:45:03.818469 [INF] Name: edge-652efb2da425e854a1e8d203 [17012] 2024/11/27 17:45:03.818469 [INF] Node: cSXjN80e [17012] 2024/11/27 17:45:03.818469 [INF] ID: NBZL4XFOC64CKXUWPM7ZCS7NNDI5X25LSMHXWZOCCB7XVM4UTOP6TD66 [17012] 2024/11/27 17:45:03.818469 [INF] Using configuration file: C:\\Program Files\\FaceFirst\\FaceFirst-MessageBroker-Edge\\conf\\edge.conf [17012] 2024/11/27 17:45:03.819454 [INF] Starting http monitor on 127.0.0.1:8223 [17012] 2024/11/27 17:45:03.819454 [INF] Starting JetStream [17012] 2024/11/27 17:45:03.819977 [INF] _ ___ _____ ___ _____ ___ ___ _ __ __ [17012] 2024/11/27 17:45:03.819977 [INF] _ | | __|_ _/ __|_ _| _ \\ __| /_\\ | \\/ | [17012] 2024/11/27 17:45:03.819977 [INF] | || | _| | | \\__ \\ | | | / _| / _ \\| |\\/| | [17012] 2024/11/27 17:45:03.819977 [INF] \\__/|___| |_| |___/ |_| |_|_\\___/_/ \\_\\_| |_| [17012] 2024/11/27 17:45:03.819977 [INF] [17012] 2024/11/27 17:45:03.819977 [INF] https://docs.nats.io/jetstream [17012] 2024/11/27 17:45:03.819977 [INF] [17012] 2024/11/27 17:45:03.819977 [INF] ---------------- JETSTREAM ---------------- [17012] 2024/11/27 17:45:03.819977 [INF] Max Memory: 23.94 GB [17012] 2024/11/27 17:45:03.819977 [INF] Max Storage: 1.00 TB [17012] 2024/11/27 17:45:03.819977 [INF] Store Directory: \"D:\\ProgramData\\FaceFirst\\FaceFirst-MessageBroker-Edge\\data-store\\jetstream\" [17012] 2024/11/27 17:45:03.819977 [INF] Domain: edge-652efb2da425e854a1e8d203 [17012] 2024/11/27 17:45:03.819977 [INF] ------------------------------------------- [17012] 2024/11/27 17:45:03.821034 [INF] Starting restore for stream 'FF-QUEUE-ACC > KV_enrollments' [17012] 2024/11/27 17:45:03.878169 [INF] Restored 69,478 messages for stream 'FF-QUEUE-ACC > KV_enrollments' in 57ms [17012] 2024/11/27 17:45:03.878169 [INF] Starting restore for stream 'FF-QUEUE-ACC > KV_media' [17012] 2024/11/27 17:45:03.879175 [WRN] Filestore [KV_media] loadBlock error: message block data missing [17012] 2024/11/27 17:45:03.882158 [WRN] Filestore [KV_media] Detected skew in subject-based total (64) vs raw total (58), rebuilding [17012] 2024/11/27 17:45:03.911169 [INF] Restored 58 messages for stream 'FF-QUEUE-ACC > KV_media' in 33ms [17012] 2024/11/27 17:45:03.911169 [INF] Starting restore for stream 'FF-QUEUE-ACC > KV_settings' [17012] 2024/11/27 17:45:03.911169 [INF] Restored 42 messages for stream 'FF-QUEUE-ACC > KV_settings' in 0s [17012] 2024/11/27 17:45:03.912166 [INF] Starting restore for stream 'FF-QUEUE-ACC > KV_visitors' [14928] 2024/11/27 17:47:06.923509 [INF] Starting nats-server [14928] 2024/11/27 17:47:06.924012 [INF] Version: 2.10.20 [14928] 2024/11/27 17:47:06.924068 [INF] Git: [7140387] [14928] 2024/11/27 17:47:06.924068 [INF] Cluster: edge-652efb2da425e854a1e8d203 [14928] 2024/11/27 17:47:06.924068 [INF] Name: edge-652efb2da425e854a1e8d203 [14928] 2024/11/27 17:47:06.924068 [INF] Node: cSXjN80e [14928] 2024/11/27 17:47:06.924068 [INF] ID: NBNOJN5SXYFVUKV73CCNICSKU22K2TBCLHQW2GG5ATAQAHIWI2GMWYDA [14928] 2024/11/27 17:47:06.924109 [INF] Using configuration file: C:\\Program Files\\FaceFirst\\FaceFirst-MessageBroker-Edge\\conf\\edge.conf [14928] 2024/11/27 17:47:06.924644 [INF] Starting http monitor on 127.0.0.1:8223 [14928] 2024/11/27 17:47:06.924644 [INF] Starting JetStream [14928] 2024/11/27 17:47:06.925157 [INF] _ ___ _____ ___ _____ ___ ___ _ __ __ [14928] 2024/11/27 17:47:06.925157 [INF] _ | | __|_ _/ __|_ _| _ \\ __| /_\\ | \\/ | [14928] 2024/11/27 17:47:06.925157 [INF] | || | _| | | \\__ \\ | | | / _| / _ \\| |\\/| | [14928] 2024/11/27 17:47:06.925157 [INF] \\__/|___| |_| |___/ |_| |_|_\\___/_/ \\_\\_| |_| [14928] 2024/11/27 17:47:06.925157 [INF] [14928] 2024/11/27 17:47:06.925157 [INF] https://docs.nats.io/jetstream [14928] 2024/11/27 17:47:06.925157 [INF] [14928] 2024/11/27 17:47:06.925157 [INF] ---------------- JETSTREAM ---------------- [14928] 2024/11/27 17:47:06.925157 [INF] Max Memory: 23.94 GB [14928] 2024/11/27 17:47:06.925157 [INF] Max Storage: 1.00 TB [14928] 2024/11/27 17:47:06.925157 [INF] Store Directory: \"D:\\ProgramData\\FaceFirst\\FaceFirst-MessageBroker-Edge\\data-store\\jetstream\" [14928] 2024/11/27 17:47:06.925157 [INF] Domain: edge-652efb2da425e854a1e8d203 [14928] 2024/11/27 17:47:06.925157 [INF] ------------------------------------------- [14928] 2024/11/27 17:47:06.926284 [INF] Starting restore for stream 'FF-QUEUE-ACC > KV_enrollments' [14928] 2024/11/27 17:47:06.977355 [INF] Restored 69,478 messages for stream 'FF-QUEUE-ACC > KV_enrollments' in 51ms [14928] 2024/11/27 17:47:06.977355 [INF] Starting restore for stream 'FF-QUEUE-ACC > KV_media' [14928] 2024/11/27 17:47:06.978352 [WRN] Filestore [KV_media] loadBlock error: message block data missing [14928] 2024/11/27 17:47:06.981374 [WRN] Filestore [KV_media] Detected skew in subject-based total (64) vs raw total (58), rebuilding [14928] 2024/11/27 17:47:07.008302 [INF] Restored 58 messages for stream 'FF-QUEUE-ACC > KV_media' in 31ms [14928] 2024/11/27 17:47:07.009270 [INF] Starting restore for stream 'FF-QUEUE-ACC > KV_settings' [14928] 2024/11/27 17:47:07.009270 [INF] Restored 42 messages for stream 'FF-QUEUE-ACC > KV_settings' in 0s [14928] 2024/11/27 17:47:07.009270 [INF] Starting restore for stream 'FF-QUEUE-ACC > KV_visitors' [11864] 2024/11/27 17:49:10.535479 [INF] Starting nats-server [11864] 2024/11/27 17:49:10.536034 [INF] Version: 2.10.20 [11864] 2024/11/27 17:49:10.536034 [INF] Git: [7140387] [11864] 2024/11/27 17:49:10.536034 [INF] Cluster: edge-652efb2da425e854a1e8d203 [11864] 2024/11/27 17:49:10.536034 [INF] Name: edge-652efb2da425e854a1e8d203 [11864] 2024/11/27 17:49:10.536034 [INF] Node: cSXjN80e [11864] 2024/11/27 17:49:10.536034 [INF] ID: NCP5BHETIHWEQ6UR7G6GKEVMF23YRHRMDP56QJ5HHQX7ROZ7MB6PBXD3 [11864] 2024/11/27 17:49:10.536034 [INF] Using configuration file: C:\\Program Files\\FaceFirst\\FaceFirst-MessageBroker-Edge\\conf\\edge.conf [11864] 2024/11/27 17:49:10.537076 [INF] Starting http monitor on 127.0.0.1:8223 [11864] 2024/11/27 17:49:10.537076 [INF] Starting JetStream [11864] 2024/11/27 17:49:10.537076 [INF] _ ___ _____ ___ _____ ___ ___ _ __ __ [11864] 2024/11/27 17:49:10.537076 [INF] _ | | __|_ _/ __|_ _| _ \\ __| /_\\ | \\/ | [11864] 2024/11/27 17:49:10.537076 [INF] | || | _| | | \\__ \\ | | | / _| / _ \\| |\\/| | [11864] 2024/11/27 17:49:10.537076 [INF] \\__/|___| |_| |___/ |_| |_|_\\___/_/ \\_\\_| |_| [11864] 2024/11/27 17:49:10.537076 [INF] [11864] 2024/11/27 17:49:10.537076 [INF] https://docs.nats.io/jetstream [11864] 2024/11/27 17:49:10.537076 [INF] [11864] 2024/11/27 17:49:10.537076 [INF] ---------------- JETSTREAM ---------------- [11864] 2024/11/27 17:49:10.537585 [INF] Max Memory: 23.94 GB [11864] 2024/11/27 17:49:10.537585 [INF] Max Storage: 1.00 TB [11864] 2024/11/27 17:49:10.537585 [INF] Store Directory: \"D:\\ProgramData\\FaceFirst\\FaceFirst-MessageBroker-Edge\\data-store\\jetstream\" [11864] 2024/11/27 17:49:10.537612 [INF] Domain: edge-652efb2da425e854a1e8d203 [11864] 2024/11/27 17:49:10.537612 [INF] ------------------------------------------- [11864] 2024/11/27 17:49:10.538210 [INF] Starting restore for stream 'FF-QUEUE-ACC > KV_enrollments' [11864] 2024/11/27 17:49:10.592603 [INF] Restored 69,478 messages for stream 'FF-QUEUE-ACC > KV_enrollments' in 54ms [11864] 2024/11/27 17:49:10.592603 [INF] Starting restore for stream 'FF-QUEUE-ACC > KV_media' [11864] 2024/11/27 17:49:10.593601 [WRN] Filestore [KV_media] loadBlock error: message block data missing [11864] 2024/11/27 17:49:10.596593 [WRN] Filestore [KV_media] Detected skew in subject-based total (64) vs raw total (58), rebuilding [11864] 2024/11/27 17:49:10.623551 [INF] Restored 58 messages for stream 'FF-QUEUE-ACC > KV_media' in 31ms [11864] 2024/11/27 17:49:10.624519 [INF] Starting restore for stream 'FF-QUEUE-ACC > KV_settings' [11864] 2024/11/27 17:49:10.624519 [INF] Restored 42 messages for stream 'FF-QUEUE-ACC > KV_settings' in 0s [11864] 2024/11/27 17:49:10.625523 [INF] Starting restore for stream 'FF-QUEUE-ACC > KV_visitors' [7624] 2024/11/27 17:51:13.625652 [INF] Starting nats-server [7624] 2024/11/27 17:51:13.625652 [INF] Version: 2.10.20 [7624] 2024/11/27 17:51:13.625652 [INF] Git: [7140387] [7624] 2024/11/27 17:51:13.625652 [INF] Cluster: edge-652efb2da425e854a1e8d203 [7624] 2024/11/27 17:51:13.625652 [INF] Name: edge-652efb2da425e854a1e8d203 [7624] 2024/11/27 17:51:13.625652 [INF] Node: cSXjN80e [7624] 2024/11/27 17:51:13.626155 [INF] ID: NAI7ED2WRL3F2QVFHDTH7VKDAMONEN5ONZHVLMFLMDV4GVT3VGRSHPKP [7624] 2024/11/27 17:51:13.626160 [INF] Using configuration file: C:\\Program Files\\FaceFirst\\FaceFirst-MessageBroker-Edge\\conf\\edge.conf [7624] 2024/11/27 17:51:13.626671 [INF] Starting http monitor on 127.0.0.1:8223 [7624] 2024/11/27 17:51:13.626671 [INF] Starting JetStream [7624] 2024/11/27 17:51:13.627179 [INF] _ ___ _____ ___ _____ ___ ___ _ __ __ [7624] 2024/11/27 17:51:13.627179 [INF] _ | | __|_ _/ __|_ _| _ \\ __| /_\\ | \\/ | [7624] 2024/11/27 17:51:13.627179 [INF] | || | _| | | \\__ \\ | | | / _| / _ \\| |\\/| | [7624] 2024/11/27 17:51:13.627179 [INF] \\__/|___| |_| |___/ |_| |_|_\\___/_/ \\_\\_| |_| [7624] 2024/11/27 17:51:13.627179 [INF] [7624] 2024/11/27 17:51:13.627179 [INF] https://docs.nats.io/jetstream [7624] 2024/11/27 17:51:13.627179 [INF] [7624] 2024/11/27 17:51:13.627179 [INF] ---------------- JETSTREAM ---------------- [7624] 2024/11/27 17:51:13.627179 [INF] Max Memory: 23.94 GB [7624] 2024/11/27 17:51:13.627179 [INF] Max Storage: 1.00 TB [7624] 2024/11/27 17:51:13.627179 [INF] Store Directory: \"D:\\ProgramData\\FaceFirst\\FaceFirst-MessageBroker-Edge\\data-store\\jetstream\" [7624] 2024/11/27 17:51:13.627179 [INF] Domain: edge-652efb2da425e854a1e8d203 [7624] 2024/11/27 17:51:13.627179 [INF] ------------------------------------------- [7624] 2024/11/27 17:51:13.628405 [INF] Starting restore for stream 'FF-QUEUE-ACC > KV_enrollments' [7624] 2024/11/27 17:51:13.678519 [INF] Restored 69,478 messages for stream 'FF-QUEUE-ACC > KV_enrollments' in 50ms [7624] 2024/11/27 17:51:13.678519 [INF] Starting restore for stream 'FF-QUEUE-ACC > KV_media' [7624] 2024/11/27 17:51:13.679484 [WRN] Filestore [KV_media] loadBlock error: message block data missing [7624] 2024/11/27 17:51:13.682459 [WRN] Filestore [KV_media] Detected skew in subject-based total (64) vs raw total (58), rebuilding [7624] 2024/11/27 17:51:13.710384 [INF] Restored 58 messages for stream 'FF-QUEUE-ACC > KV_media' in 32ms [7624] 2024/11/27 17:51:13.711381 [INF] Starting restore for stream 'FF-QUEUE-ACC > KV_settings' [7624] 2024/11/27 17:51:13.711381 [INF] Restored 42 messages for stream 'FF-QUEUE-ACC > KV_settings' in 0s [7624] 2024/11/27 17:51:13.711381 [INF] Starting restore for stream 'FF-QUEUE-ACC > KV_visitors' ``` ### Expected behavior The problem is that jetstream at the core does not differentiate between the old and new version of the stream match_events. If the source stream where to receive and keep track of the remote stream's create time and included that time in the consumer create, the remote server can check if the time submitted differs from the create time of the stream and return an error instead. This error could be the signal for the hub jetstream to start over by resetting the sequence number, obtain and update the new timestamp, create the new consumer again. This probably has implications for mirroring as well. There a differing timestamp should probably result in deleting the current content prior to starting at 0 again. ### Server and client version 2.10.20 & 2.10.23 ### Host environment _No response_ ### Steps to reproduce _No response_",
        "url": "https://github.com/nats-io/nats-server/issues/6206",
        "createdAt": "2024-12-03T21:02:39Z",
        "updatedAt": "2026-08-13T14:19:34Z",
        "timestamp": "2026-08-13T14:19:34Z",
        "metrics": {
          "reactions": 0,
          "comments": 2
        },
        "labels": [
          "defect"
        ],
        "author": "facefirstMH",
        "state": "closed",
        "assignees": [
          "neilalexander",
          "jnmoyne"
        ]
      },
      {
        "id": "github:nats-io/nats-server:issue:8346",
        "source": "github",
        "group": "data-infrastructure",
        "project": "nats-io/nats-server",
        "kind": "issue",
        "title": "JetStream sourced stream reports lag=0 after source stream recreation but no longer ingests new messages",
        "text": "### Observed behavior We have a JetStream deployment where a destination stream sources data from many remote JetStream domains using external stream sources. For one source, after the source stream was recreated, the destination stream stopped ingesting new messages from that source. However, stream info continues to report: lag: 0 no error non-zero active making the source appear healthy even though no new messages are arriving. Simplified topology <img width=\"413\" height=\"296\" alt=\"Image\" src=\"https://github.com/user-attachments/assets/eb91a0d6-6d8a-4257-94f5-4be0da4ab5e7\" /> ORDERS sources from many remote JetStream domains. All sources continue functioning normally except H1314. Configuration The destination stream contains a source similar to: ```json { \"name\": \"H1314\", \"external\": { \"api\": \"$JS.H1314.API\" }, \"opt_start_time\": \"2026-06-30T00:00:00Z\" } ``` What happened The source stream H1314 was recreated. The recreated source stream continues receiving new messages. For example: nats sub EXAMPLE.SUBJECT --last --js-domain H1314 Source stream: H1314 Sequence: 1029746 Timestamp: 2026-06-30 17:47:54 UTC Subject: EXAMPLE.SUBJECT However, the most recent message for the same subject in the destination stream remains from the previous day: nats sub EXAMPLE.SUBJECT --stream ORDERS --last Destination stream: ORDERS Timestamp: 2026-06-29 21:24:36 UTC Subject: EXAMPLE.SUBJECT The copied message still contains a header similar to: Nats-Stream-Source: H1314:<consumer> 12002532 > > EXAMPLE.SUBJECT which references the previous incarnation of the source stream. Observed behavior nats stream info ORDERS -j reports: ```json { \"name\": \"H1314\", \"lag\": 0, \"active\": 265738727 } ``` There is no reported error. Despite this, no new messages from H1314 are arriving in ORDERS. I tested creating a new stream TEST1314 with a single source [\"H1314\"] and it replicated into the new stream just fine. ### Expected behavior The destination stream reconnects to the recreated source stream and resumes sourcing automatically. or JetStream reports that sourcing has failed (non-zero lag or an error condition). Reporting lag=0 while the destination stream is no longer ingesting messages makes the problem difficult to diagnose. ### Server and client version nats-server: v2.14.0 (all nodes) CLI version v0.4.0 ### Host environment _No response_ ### Steps to reproduce _No response_",
        "url": "https://github.com/nats-io/nats-server/issues/8346",
        "createdAt": "2026-07-01T14:12:57Z",
        "updatedAt": "2026-08-13T14:19:34Z",
        "timestamp": "2026-08-13T14:19:34Z",
        "metrics": {
          "reactions": 0,
          "comments": 4
        },
        "labels": [
          "defect"
        ],
        "author": "ordeneaux",
        "state": "closed",
        "assignees": []
      },
      {
        "id": "github:nats-io/nats-server:pull_request:8133",
        "source": "github",
        "group": "data-infrastructure",
        "project": "nats-io/nats-server",
        "kind": "pull_request",
        "title": "[ADDED] Support PROXY protocol for websocket listeners",
        "text": "Add websocket `proxy_protocol` support by parsing optional PROXY headers on the raw listener before TLS and HTTP upgrade processing. Non-proxied websocket clients continue to work because pre-read bytes are replayed into the next protocol layer, and the setting is exposed in `/varz`. Validation: - `go test -v ./server -run '^(TestWSParseOptions|TestWSProxyProtocol|TestWSProxyProtocolNonProxiedConnection|TestMonitorWebsocket)$' -count=1 -vet=off`\\n- `git diff --check`\\n\\nResolves https://github.com/nats-io/nats-server/issues/7924",
        "url": "https://github.com/nats-io/nats-server/pull/8133",
        "createdAt": "2026-05-12T05:12:22Z",
        "updatedAt": "2026-08-13T09:48:25Z",
        "timestamp": "2026-08-13T09:48:25Z",
        "metrics": {
          "reactions": 2,
          "comments": 2
        },
        "labels": [],
        "author": "officialasishkumar",
        "state": "open",
        "assignees": []
      },
      {
        "id": "github:nats-io/nats-server:pull_request:8384",
        "source": "github",
        "group": "data-infrastructure",
        "project": "nats-io/nats-server",
        "kind": "pull_request",
        "title": "(2.15) [IMPROVED] Source stream recreation detection",
        "text": "Previously if you started sourcing from a stream that was recreated, it would wait for the recreated stream's sequence to move back up to the last sourced sequence before it accepted new messages. This PR detects a stream being recreated by noticing a changed `Nats-Stream-Identity`, which is the hash of the stream's created time. There exists no unique identifier for a stream/consumer lifetime apart from the created time, so that's used and is the most practical since collisions are unlikely enough. - This hash is appended to the source header to know what \"identity\" of the stream was associated with the highest sequence for that source. (source header before: `TEST 1 > > foo`, after: `TEST 1 > > foo sLkPMPK6`) - This identity is passed in the consumer create request to prevent a consumer from being created based on sequences derived from the previous stream. An error is returned (`JSConsumerStreamIdentityMismatch`) if a mismatch is detected, the request is then retried with reset sequences to allow sourcing from the beginning. - The consumer reset request does not allow for the identity to be passed in, and instead relies on the caller to notice a different identity and subsequent in-memory state reset. This prevents unnecessary churn of requests since the reset will always happen based on a consumer that already exists and needs to be reset anyway. - Both the consumer create and reset responses contain this identity to get informed of the latest identity of the stream, which is then passed into the source header. This PR additionally requires that the starting sequence for a consumer is respected, even for sourcing consumers that were previously (incorrectly) clamped within the sourced stream's first/last sequence range. Upgrade note: the servers on both ends will need to be updated for this mechanism to kick in. A downgrade should ignore the extra appended identity from the header and fall back to the previous behavior. Resolves https://github.com/nats-io/nats-server/issues/8346, https://github.com/nats-io/nats-server/issues/6206, https://github.com/nats-io/nats-server/issues/5847",
        "url": "https://github.com/nats-io/nats-server/pull/8384",
        "createdAt": "2026-07-15T15:39:20Z",
        "updatedAt": "2026-08-13T14:19:35Z",
        "timestamp": "2026-08-13T14:19:35Z",
        "metrics": {
          "reactions": 2,
          "comments": 9
        },
        "labels": [],
        "author": "MauriceVanVeen",
        "state": "closed",
        "assignees": []
      },
      {
        "id": "github:nats-io/nats-server:pull_request:8432",
        "source": "github",
        "group": "data-infrastructure",
        "project": "nats-io/nats-server",
        "kind": "pull_request",
        "title": "(2.15) [ADDED] Desired state reconcilitation for scaling and moves",
        "text": "This PR adds initial support for desired state reconciliation of stream and consumer updates that change the peer set. These, and follow-up, changes intend to fix loads of related issues surrounding peer set management of assets. For example, including but not limited to: a scale down not being guaranteed to select the right leader or servers with all data to prevent data loss, peers re-appearing in the stream/consumer Raft group after peer-removing a node, moving and scaling a stream concurrently may brick it such that it's stuck halfway into a move or is reverted back to the previous cluster with the stream config still referencing where it was meant to move to, and a consumer reverting to stale state after a consumer move. This initial PR tries to do the minimal amount of work to have passing CI, while transitioning to a desired state design that allows for guaranteed and safe scaling and moving of assets. Importantly: - The meta leader MUST never overwrite a stream/consumer peer set in-place as it does today. The meta leader may dictate where a stream/consumer needs to move, but the stream/consumer is responsible for making those changes be guaranteed safe. - The meta leader always dictates the peer set, unless the stream/consumer is scaled down, since only the group itself knows which servers would be best to be preserved (i.e. the leader and most up-to-date servers), to guarantee no data loss. Previously the meta leader did a `sysRequest` stream/consumer info to figure out the leader (and for consumer moves the consumer state). Both could result in a stale leader/state being picked. And worse, if the leader doesn't respond in time (or at all), the meta leader would randomly pick a leader (or lose consumer state). This PR allows the removal of `sysRequest` entirely. - Peer additions and removals are managed by the group's leader, no automatic additions of peers are allowed (like for the meta group). Before the group's leader is allowed to make any change, it first needs to communicate with the meta leader to assert that it's up-to-date with the meta state (linearizable and no stale local assignment), and the leader's Raft `term` is also wired through, so the meta leader also knows to ignore requests from stale leaders. This PR allows removing `ProposeKnownPeers` and `UpdateKnownPeers` from `raft.go`, as well as the calls to them in `js.createRaftGroup`, since those were unsafe and could result in removed peers re-appearing, desyncing the peer set in the meta assignment with that of the group. - The stream/consumer `ClusterInfo` now returns the desired state information. Allowing tooling (like the CLI) to inspect the state as it changes. This PR is split in 3 commits, to ease reviewing but also ease backporting to 2.14. Put simply: the first commit adds the models, the second commit adds meta layer reconciliation awareness, and the last commit actually starts populating the desired state and triggering the stream/consumer migration paths. Likely 2.14 will require a compatibility commit to ease downgrading, and upgrading/downgrading as a whole should be tackled in a follow-up PR. However, the design of the desired state always guarantees: the stream/consumer config is always what the user asked it to be, the group's peer set is always the one that equals the underlying Raft group (or if the Raft group contains less peers, they get automatically added under the old behavior). So, worst-case given a downgrade to an incompatible version, the desired state configuration in the JSON will \"fall off\" and the assignment will remain \"stuck\" on which peers it was at that time. That does not violate correctness or safety when it comes to preserving data, it does however prevent the stream/consumer from moving to the desired config. Having passing CI and the mentioned upgrade/downgrade focus isn't enough, and at least these follow-ups will be required in separate PRs: - The system \"move\" and \"move cancel\" APIs should be made more reliable. A \"move\" can simply trigger a peer set change through desired state, and a \"move cancel\" can look at the desired state's origin to see where the stream needs to be moved back to. Both just need time to be wired appropriately. - Server and asset-level peer-remove should be made desired state aware. Additionally, an \"evacuate\" endpoint should be added to enable the user to more easily move streams and consumers off a server while it's still online. Not only easing the decommissioning of a node, but also enabling to move data of R1 assets without losing data when a peer-remove of the node would've been used instead and without requiring to move all R1 assets in all accounts manually. - Add stream move/scale guards. This PR removes the (admittedly brittle) guards to prevent concurrent move and scale of a stream, which allows to scale and move an R1 stream from cluster A to a R3 stream in cluster B in one go and safely. However, this technically opens up being able to constantly cycle moves between clusters until the stream/consumer peer set contain the total number of peers. This should likely get an upper bound, for example 10 peers maximum (since replication is capped at 5), to ensure such moves are still allowed from the API, but the system prevents the peers from growing semi-unbounded. - Optimizations in how and when peers are added, ensuring a group doesn't go out of quorum when nodes are offline. - An additional optimization can be made when moving a stream between clusters, to first scale up one stream replica in the new cluster, making it the leader and then doing local catchup for the other replicas in that cluster. This would allow both faster and less expensive (bandwidth-wise) moves of assets. But, although this is enabled by the desired state approach as well, it might need to be done after 2.15.0 lands considering timing and the other required follow-ups (this isn't a required optimization). These follow-ups are also roughly marked with `FIXME` comments that need to be addressed prior to releasing 2.15.",
        "url": "https://github.com/nats-io/nats-server/pull/8432",
        "createdAt": "2026-07-29T13:15:47Z",
        "updatedAt": "2026-08-13T09:30:59Z",
        "timestamp": "2026-08-13T09:30:59Z",
        "metrics": {
          "reactions": 0,
          "comments": 6
        },
        "labels": [],
        "author": "MauriceVanVeen",
        "state": "open",
        "assignees": []
      },
      {
        "id": "github:nats-io/nats-server:pull_request:8437",
        "source": "github",
        "group": "data-infrastructure",
        "project": "nats-io/nats-server",
        "kind": "pull_request",
        "title": "(2.15) [ADDED] Drive stream move and cancel move via desired state",
        "text": "The move and cancel move endpoints are now wired fully through desired state. The cancel move endpoint can now be reliably used; the first move (either through stream updates or the move endpoint) initialize the \"origin\" which is the peer set and config (replicas, placement and retention) at that time, multiple moves or scales could technically be run even before the full desired state change completes, a cancel will then always roll back to the origin. However, concurrent stream move or scale requests are rejected to keep the peer set bounded, as otherwise cycling moves between clusters or cycling between replication counts could result in all peers being admitted larger than `2*replicas`. This restriction does not apply to consumers, they can be freely scaled within the stream's peer set. This PR additionally fixes the JS lock being released and re-acquired before calling `jsa.configUpdateCheck`, and ensures legacy moves are translated to desired state properly and can be rolled back through the improved cancel move endpoint as well. Follow-up of https://github.com/nats-io/nats-server/pull/8432",
        "url": "https://github.com/nats-io/nats-server/pull/8437",
        "createdAt": "2026-07-30T18:47:50Z",
        "updatedAt": "2026-08-13T09:48:16Z",
        "timestamp": "2026-08-13T09:48:16Z",
        "metrics": {
          "reactions": 0,
          "comments": 5
        },
        "labels": [],
        "author": "MauriceVanVeen",
        "state": "open",
        "assignees": []
      },
      {
        "id": "github:nats-io/nats-server:pull_request:8439",
        "source": "github",
        "group": "data-infrastructure",
        "project": "nats-io/nats-server",
        "kind": "pull_request",
        "title": "(2.15) [ADDED] Report requested stream config while running at desired origin",
        "text": "The retention and placement from the stream's config must always return the latest state the user requested, and the desired state contains the 'origin' if needed. Before this PR the stream would already keep running under Limits if it was updated to Interest retention, since this allows the consumers to be scaled up to have parity with the stream's replication first, and only after safely switch the retention to Interest. But this would also result in the stream config in stream info and update responses as well as JSZ showing up as the 'origin' retention and placement instead of the latest values. This PR fixes that by always returning the latest state, and only the desired origin contains the previous value as expected. Additionally, it ensures the desired origin is always set, even when it's registered through a retention change, such that it can be canceled through the 'cancel move' endpoint. Follow-up of https://github.com/nats-io/nats-server/pull/8432",
        "url": "https://github.com/nats-io/nats-server/pull/8439",
        "createdAt": "2026-07-31T11:37:26Z",
        "updatedAt": "2026-08-13T09:52:28Z",
        "timestamp": "2026-08-13T09:52:28Z",
        "metrics": {
          "reactions": 1,
          "comments": 5
        },
        "labels": [],
        "author": "MauriceVanVeen",
        "state": "open",
        "assignees": []
      },
      {
        "id": "github:nats-io/nats-server:pull_request:8443",
        "source": "github",
        "group": "data-infrastructure",
        "project": "nats-io/nats-server",
        "kind": "pull_request",
        "title": "(2.15) [ADDED] Peer evacuation and reconciliation of assignments",
        "text": "This PR wires peer removals and additions into desired state. - `reconcilePeerAssignments` runs on a meta leader change, so proposals that were missed or only partially applied can be retried by the next leader. - Peer additions through `processAddPeer` now also obey the rules of `cc.selectPeerGroup`, which requires waiting for STATSZ info if it isn't yet available for the newly added node. - A new set of \"peer evacuate\" endpoints (`$JS.API.STREAM.PEER.EVACUATE.<stream>` and, for the system account, `$JS.API.SERVER.EVACUATE`) offers the same semantics as peer-remove while not immediately removing the peer, so it can cooperate in catching up the new peer. This also allows evacuating all R1 streams of a server that is to be decommissioned ahead of time, whereas a peer-remove of a server hosting an R1 stream does get a new server assigned but would otherwise lose all data. `reconcilePeerAssignments` scans every stream and consumer assignment on each meta leader change and peer add, benchmarked locally at ~100-290 ms for 100k streams/500k consumers, under `js.mu`. That is purely a no-op scan; actually reconciling is significantly more expensive, since each affected assignment's proposal needs to be retried. A large contributor is `selectPeerGroup`, which loops over all stream assignments per call to determine HA asset counts. This is not addressed in this PR, but noted for clarity, and `selectPeerGroup` can only be made faster once all `meta.ForwardProposal` calls are gone since we can then purely rely on atomics. Follow-up of https://github.com/nats-io/nats-server/pull/8432",
        "url": "https://github.com/nats-io/nats-server/pull/8443",
        "createdAt": "2026-08-07T18:06:36Z",
        "updatedAt": "2026-08-13T17:32:19Z",
        "timestamp": "2026-08-13T17:32:19Z",
        "metrics": {
          "reactions": 0,
          "comments": 3
        },
        "labels": [],
        "author": "MauriceVanVeen",
        "state": "open",
        "assignees": []
      },
      {
        "id": "github:nats-io/nats-server:pull_request:8452",
        "source": "github",
        "group": "data-infrastructure",
        "project": "nats-io/nats-server",
        "kind": "pull_request",
        "title": "(2.15) [FIXED] Evict peer-removed peers for group below quorum",
        "text": "If a stream/consumer Raft group was below quorum, it wasn't possible to safely peer-remove nodes to get into quorum again. This PR fixes that by allowing nodes to be evicted when below quorum, but only those nodes that were explicitly peer-removed. Given a R3 stream without quorum due to 2 nodes being offline, it's now possible to peer-remove those two offline nodes and the stream will become operational again, albeit it at R1. If new peers were assigned to the stream, the stream would automatically scale up to R3, but on the new peers. The desired state gets a new `rg.Desired.Removed` field that contains the set of explicitly peer-removed nodes, which are eligible for eviction if below quorum. Eviction is only allowed if this node thinks the stream/consumer Raft group is leaderless, it is not leaderless on the meta layer, it has desired state that specifies removed peers, and the node itself is still part of the group's peer set. A safety check also prevents eviction if the remaining peers could form a disjoint majority. Follow-up of https://github.com/nats-io/nats-server/pull/8432",
        "url": "https://github.com/nats-io/nats-server/pull/8452",
        "createdAt": "2026-08-11T14:10:44Z",
        "updatedAt": "2026-08-13T17:30:14Z",
        "timestamp": "2026-08-13T17:30:14Z",
        "metrics": {
          "reactions": 0,
          "comments": 3
        },
        "labels": [],
        "author": "MauriceVanVeen",
        "state": "open",
        "assignees": []
      },
      {
        "id": "github:nats-io/nats-server:pull_request:8453",
        "source": "github",
        "group": "data-infrastructure",
        "project": "nats-io/nats-server",
        "kind": "pull_request",
        "title": "Cherry-picks for 2.12.15",
        "text": "Includes the following: - https://github.com/nats-io/nats-server/pull/8430 - https://github.com/nats-io/nats-server/pull/8449 - https://github.com/nats-io/nats-server/pull/8454",
        "url": "https://github.com/nats-io/nats-server/pull/8453",
        "timestamp": "2026-08-12T13:35:39Z",
        "metrics": {
          "reactions": 1,
          "comments": 1
        },
        "labels": [],
        "author": "neilalexander",
        "assignees": [],
        "change": "new"
      },
      {
        "id": "github:nats-io/nats-server:pull_request:8455",
        "source": "github",
        "group": "data-infrastructure",
        "project": "nats-io/nats-server",
        "kind": "pull_request",
        "title": "[IMPROVED] Expire MaxAge block prefixes after v4 recovery",
        "text": "Recovered v4 file-store blocks now reach the range validator. It learns timestamp order before it selects an eligible MaxAge prefix. Cold upgraded stores can remove complete expired blocks with one range update instead of one removal per message. The repair also reuses the existing `selectNextFirst` owner after prefix removal. This removes leading empty blocks and makes the stream `FirstSeq` and removal callback use the first live sequence. Tests and benchmarks remain in `server/filestore_test.go`. No new proof file or second cleanup path is added. **Workload:** `cold restarted v4-index file-store MaxAge expiry of 16,384 timestamp-ordered messages across complete blocks` | Metric | Before | After | Change | | --- | --- | --- | --- | | `ns/op` | 11582922 | 3967472 | 65.7% lower | **Workload:** `end-to-end file-store MaxAge expiry of 16,384 messages with 128 pending and redelivered consumer records` | Metric | Before | After | Change | | --- | --- | --- | --- | | `ns/op` | 15349524 | 3909440 | 74.5% lower | | `p95-ns/op` | 16824277 | 4498940 | 73.3% lower | **Workload:** `end-to-end file-store MaxAge expiry of 16,384 messages with 4,096 pending and redelivered consumer records` | Metric | Before | After | Change | | --- | --- | --- | --- | | `ns/op` | 28702905 | 16201338 | 43.6% lower | | `p95-ns/op` | 30625625 | 17492837 | 42.9% lower | The cleanup repair showed no material change in five sequential 100-iteration benchmark samples against the exact measured Candidate. Proof includes the focused file-store, consumer, and cluster tests; the regression repeated 10 times; its race run; `go vet ./server`; and Linux, Linux/386, and Windows builds. The full native store suite had one existing timing flake. It also reproduces on the Candidate and the untouched upstream base. --- Generated by [Perfloop](https://app.perfloop.ai). Human sponsor: [Tomas Senart](https://github.com/tsenart). [Measurements and checks](https://app.perfloop.ai/t/oss/case_yng2yjnryc).",
        "url": "https://github.com/nats-io/nats-server/pull/8455",
        "timestamp": "2026-08-12T12:49:02Z",
        "metrics": {
          "reactions": 1,
          "comments": 4
        },
        "labels": [],
        "author": "perfloop-agent",
        "assignees": [],
        "change": "new"
      },
      {
        "id": "github:nats-io/nats-server:pull_request:8456",
        "source": "github",
        "group": "data-infrastructure",
        "project": "nats-io/nats-server",
        "kind": "pull_request",
        "title": "[FIXED] `syncBlocks` may skip sync on compacted blocks",
        "text": "`syncBlocks` should leave all message blocks synced after it returns. This was not always the case when a block needed compaction during the same pass. Specifically, if the block was initially marked as not needing sync, i.e. `mb.needSync = false`, then `syncBlocks` would miss the fact that `compactWithFloor` could later set `mb.needSync = true` on that same block. Keep track of blocks compacted and sync both the new compacted file and its directory before returning.",
        "url": "https://github.com/nats-io/nats-server/pull/8456",
        "timestamp": "2026-08-12T12:27:12Z",
        "metrics": {
          "reactions": 1,
          "comments": 2
        },
        "labels": [],
        "author": "sciascid",
        "assignees": [],
        "change": "new"
      },
      {
        "id": "github:nats-io/nats-server:pull_request:8457",
        "source": "github",
        "group": "data-infrastructure",
        "project": "nats-io/nats-server",
        "kind": "pull_request",
        "title": "De-flake TestJetStreamSuperClusterStreamDirectGetMirrorQueueGroup",
        "url": "https://github.com/nats-io/nats-server/pull/8457",
        "createdAt": "2026-08-12T12:17:38Z",
        "updatedAt": "2026-08-12T15:45:09Z",
        "timestamp": "2026-08-12T15:45:09Z",
        "metrics": {
          "reactions": 1,
          "comments": 2
        },
        "labels": [],
        "author": "sciascid",
        "state": "closed",
        "assignees": [],
        "change": "updated"
      },
      {
        "id": "github:nats-io/nats-server:pull_request:8458",
        "source": "github",
        "group": "data-infrastructure",
        "project": "nats-io/nats-server",
        "kind": "pull_request",
        "title": "Cherry-picks for 2.14.5",
        "text": "Includes the following: - https://github.com/nats-io/nats-server/pull/8454",
        "url": "https://github.com/nats-io/nats-server/pull/8458",
        "timestamp": "2026-08-12T13:42:24Z",
        "metrics": {
          "reactions": 1,
          "comments": 1
        },
        "labels": [],
        "author": "neilalexander",
        "assignees": [],
        "change": "new"
      },
      {
        "id": "github:nats-io/nats-server:pull_request:8459",
        "source": "github",
        "group": "data-infrastructure",
        "project": "nats-io/nats-server",
        "kind": "pull_request",
        "title": "Update dependencies",
        "url": "https://github.com/nats-io/nats-server/pull/8459",
        "createdAt": "2026-08-12T14:16:34Z",
        "updatedAt": "2026-08-12T15:32:11Z",
        "timestamp": "2026-08-12T15:32:11Z",
        "metrics": {
          "reactions": 1,
          "comments": 1
        },
        "labels": [],
        "author": "neilalexander",
        "state": "closed",
        "assignees": [],
        "change": "updated"
      },
      {
        "id": "github:nats-io/nats-server:pull_request:8460",
        "source": "github",
        "group": "data-infrastructure",
        "project": "nats-io/nats-server",
        "kind": "pull_request",
        "title": "(2.15) [FIXED] Migration added offline peers & removed peers before catchup",
        "text": "This PR fixes two remaining issues, and removes the final two FIXMEs: - Migrations would add peers as fast as possible, without checking whether the peer being added was even online. Adding an offline peer grows the quorum size, which could drop the group out of quorum until enough nodes came online. `selectPeerToAdd` now prefers the candidate we've heard from most recently, and only adds an unheard peer if the live members still reach quorum in the grown group. - Migrations also didn't wait for upper-layer stream catchup to complete before removing old peers. An old peer is now only removed if the group left after the removal retains a store-current majority, based on `Current`/`Lag` and the stream's active `catchupPeers`. Follow-up of https://github.com/nats-io/nats-server/pull/8432",
        "url": "https://github.com/nats-io/nats-server/pull/8460",
        "createdAt": "2026-08-12T16:02:04Z",
        "updatedAt": "2026-08-13T17:27:16Z",
        "timestamp": "2026-08-13T17:27:16Z",
        "metrics": {
          "reactions": 0,
          "comments": 3
        },
        "labels": [],
        "author": "MauriceVanVeen",
        "state": "open",
        "assignees": []
      },
      {
        "id": "github:nats-io/nats-server:pull_request:8461",
        "source": "github",
        "group": "data-infrastructure",
        "project": "nats-io/nats-server",
        "kind": "pull_request",
        "title": "[IMPROVED] Sync the block directory once in syncBlocks",
        "text": "Do single directory fsync after all blocks have been processed, instead of syncing the dir multiple times if more than one block was compacted.",
        "url": "https://github.com/nats-io/nats-server/pull/8461",
        "createdAt": "2026-08-13T09:14:37Z",
        "updatedAt": "2026-08-13T12:59:47Z",
        "timestamp": "2026-08-13T12:59:47Z",
        "metrics": {
          "reactions": 1,
          "comments": 2
        },
        "labels": [],
        "author": "sciascid",
        "state": "closed",
        "assignees": []
      },
      {
        "id": "github:nats-io/nats-server:pull_request:8462",
        "source": "github",
        "group": "data-infrastructure",
        "project": "nats-io/nats-server",
        "kind": "pull_request",
        "title": "[FIXED] Extend dios to cover filestore block syncs",
        "url": "https://github.com/nats-io/nats-server/pull/8462",
        "createdAt": "2026-08-13T10:35:36Z",
        "updatedAt": "2026-08-13T13:43:43Z",
        "timestamp": "2026-08-13T13:43:43Z",
        "metrics": {
          "reactions": 1,
          "comments": 0
        },
        "labels": [],
        "author": "sciascid",
        "state": "closed",
        "assignees": []
      },
      {
        "id": "github:nats-io/nats-server:release:369333995",
        "source": "github",
        "group": "data-infrastructure",
        "project": "nats-io/nats-server",
        "kind": "release",
        "title": "Release v2.14.5",
        "text": "## Changelog Refer to the [2.14 Upgrade Guide](https://docs.nats.io/release-notes/whats_new/whats_new_214) for backwards compatibility notes with 2.12.x. Please note that the 2.13.x version was skipped. ### Go Version - 1.26.5 ### Dependencies - github.com/klauspost/compress v1.19.2 (#8459) - golang.org/x/crypto v0.55.0 (#8459) ### Added Leafnodes - New `dial_timeout` option can be specified in the `leafnode` config block or for specific remotes in the configuration, allowing it to be increased above the default 1 second for high-latency links (#8427) ### Fixed General - Fixed a deadlock that could occur in the logger when a close error occurs (#8430) JetStream - Fix a bug that could result in potential data loss when handling idempotent stream creates when an offline node catches up from a metalayer snapshot, caused by an incorrect update to the create time in the stream assignment (#8449) ### Complete Changes https://github.com/nats-io/nats-server/compare/v2.14.4...v2.14.5",
        "url": "https://github.com/nats-io/nats-server/releases/tag/v2.14.5",
        "createdAt": "2026-08-12T15:31:27Z",
        "updatedAt": "2026-08-12T15:31:27Z",
        "timestamp": "2026-08-12T15:31:27Z",
        "labels": [],
        "author": "github-actions[bot]",
        "assignees": [],
        "change": "new"
      },
      {
        "id": "github:nats-io/nats-server:release:369334266",
        "source": "github",
        "group": "data-infrastructure",
        "project": "nats-io/nats-server",
        "kind": "release",
        "title": "Release v2.12.15",
        "text": "## Changelog ### Go Version - 1.25.12 ### Dependencies - github.com/klauspost/compress v1.19.2 (#8459) - golang.org/x/crypto v0.55.0 (#8459) ### Fixed General - Fixed a deadlock that could occur in the logger when a close error occurs (#8430) JetStream - Fix a bug that could result in potential data loss when handling idempotent stream creates when an offline node catches up from a metalayer snapshot, caused by an incorrect update to the create time in the stream assignment (#8449) ### Complete Changes https://github.com/nats-io/nats-server/compare/v2.12.14...v2.12.15",
        "url": "https://github.com/nats-io/nats-server/releases/tag/v2.12.15",
        "createdAt": "2026-08-12T15:30:42Z",
        "updatedAt": "2026-08-12T15:30:42Z",
        "timestamp": "2026-08-12T15:30:42Z",
        "labels": [],
        "author": "github-actions[bot]",
        "assignees": [],
        "change": "new"
      },
      {
        "id": "github:nats-io/nats-server:release:https://github.com/nats-io/nats-server/releases/tag/v2.12.15",
        "source": "github",
        "group": "data-infrastructure",
        "project": "nats-io/nats-server",
        "kind": "release",
        "title": "Release v2.12.15",
        "text": "## Changelog ### Go Version - 1.25.12 ### Dependencies - github.com/klauspost/compress v1.19.2 (#8459) - golang.org/x/crypto v0.55.0 (#8459) ### Fixed General - Fixed a deadlock that could occur in the logger when a close error occurs (#8430) JetStream - Fix a bug that could result in potential data loss when handling idempotent stream creates when an offline node catches up from a metalayer snapshot, caused by an incorrect update to the create time in the stream assignment (#8449) ### Complete Changes https://github.com/nats-io/nats-server/compare/v2.12.14...v2.12.15",
        "url": "https://github.com/nats-io/nats-server/releases/tag/v2.12.15",
        "timestamp": "2026-08-12T15:30:42Z",
        "labels": [],
        "author": "github-actions[bot]",
        "assignees": [],
        "change": "new"
      },
      {
        "id": "github:nats-io/nats-server:release:https://github.com/nats-io/nats-server/releases/tag/v2.14.5",
        "source": "github",
        "group": "data-infrastructure",
        "project": "nats-io/nats-server",
        "kind": "release",
        "title": "Release v2.14.5",
        "text": "## Changelog Refer to the [2.14 Upgrade Guide](https://docs.nats.io/release-notes/whats_new/whats_new_214) for backwards compatibility notes with 2.12.x. Please note that the 2.13.x version was skipped. ### Go Version - 1.26.5 ### Dependencies - github.com/klauspost/compress v1.19.2 (#8459) - golang.org/x/crypto v0.55.0 (#8459) ### Added Leafnodes - New `dial_timeout` option can be specified in the `leafnode` config block or for specific remotes in the configuration, allowing it to be increased above the default 1 second for high-latency links (#8427) ### Fixed General - Fixed a deadlock that could occur in the logger when a close error occurs (#8430) JetStream - Fix a bug that could result in potential data loss when handling idempotent stream creates when an offline node catches up from a metalayer snapshot, caused by an incorrect update to the create time in the stream assignment (#8449) ### Complete Changes https://github.com/nats-io/nats-server/compare/v2.14.4...v2.14.5",
        "url": "https://github.com/nats-io/nats-server/releases/tag/v2.14.5",
        "timestamp": "2026-08-12T15:31:27Z",
        "labels": [],
        "author": "github-actions[bot]",
        "assignees": [],
        "change": "new"
      }
    ],
    "events": [
      {
        "id": "event:bb1999ba830c4db13a31",
        "signalId": "github:nats-io/nats-server:pull_request:8462",
        "event": "changed",
        "observedAt": "2026-08-13T13:48:00.446149Z",
        "changedFields": [],
        "signal": {
          "id": "github:nats-io/nats-server:pull_request:8462",
          "source": "github",
          "group": "data-infrastructure",
          "project": "nats-io/nats-server",
          "kind": "pull_request",
          "title": "[FIXED] Extend dios to cover filestore block syncs",
          "url": "https://github.com/nats-io/nats-server/pull/8462",
          "createdAt": "2026-08-13T10:35:36Z",
          "updatedAt": "2026-08-13T13:43:43Z",
          "timestamp": "2026-08-13T13:43:43Z",
          "metrics": {
            "reactions": 1,
            "comments": 0
          },
          "labels": [],
          "author": "sciascid",
          "state": "closed",
          "assignees": [],
          "change": "updated"
        }
      },
      {
        "id": "event:5970178750feca30d7b7",
        "signalId": "github:nats-io/nats-server:pull_request:8461",
        "event": "changed",
        "observedAt": "2026-08-13T13:48:00.446149Z",
        "changedFields": [],
        "signal": {
          "id": "github:nats-io/nats-server:pull_request:8461",
          "source": "github",
          "group": "data-infrastructure",
          "project": "nats-io/nats-server",
          "kind": "pull_request",
          "title": "[IMPROVED] Sync the block directory once in syncBlocks",
          "text": "Do single directory fsync after all blocks have been processed, instead of syncing the dir multiple times if more than one block was compacted.",
          "url": "https://github.com/nats-io/nats-server/pull/8461",
          "createdAt": "2026-08-13T09:14:37Z",
          "updatedAt": "2026-08-13T12:59:47Z",
          "timestamp": "2026-08-13T12:59:47Z",
          "metrics": {
            "reactions": 1,
            "comments": 2
          },
          "labels": [],
          "author": "sciascid",
          "state": "closed",
          "assignees": [],
          "change": "updated"
        }
      },
      {
        "id": "event:1a28698f69b7ea6c834b",
        "signalId": "github:nats-io/nats-server:pull_request:8443",
        "event": "changed",
        "observedAt": "2026-08-13T13:48:00.446149Z",
        "changedFields": [],
        "signal": {
          "id": "github:nats-io/nats-server:pull_request:8443",
          "source": "github",
          "group": "data-infrastructure",
          "project": "nats-io/nats-server",
          "kind": "pull_request",
          "title": "(2.15) [ADDED] Peer evacuation and reconciliation of assignments",
          "text": "This PR wires peer removals and additions into desired state. - `reconcilePeerAssignments` runs on a meta leader change, so proposals that were missed or only partially applied can be retried by the next leader. - Peer additions through `processAddPeer` now also obey the rules of `cc.selectPeerGroup`, which requires waiting for STATSZ info if it isn't yet available for the newly added node. - A new set of \"peer evacuate\" endpoints (`$JS.API.STREAM.PEER.EVACUATE.<stream>` and, for the system account, `$JS.API.SERVER.EVACUATE`) offers the same semantics as peer-remove while not immediately removing the peer, so it can cooperate in catching up the new peer. This also allows evacuating all R1 streams of a server that is to be decommissioned ahead of time, whereas a peer-remove of a server hosting an R1 stream does get a new server assigned but would otherwise lose all data. `reconcilePeerAssignments` scans every stream and consumer assignment on each meta leader change and peer add, benchmarked locally at ~100-290 ms for 100k streams/500k consumers, under `js.mu`. That is purely a no-op scan; actually reconciling is significantly more expensive, since each affected assignment's proposal needs to be retried. A large contributor is `selectPeerGroup`, which loops over all stream assignments per call to determine HA asset counts. This is not addressed in this PR, but noted for clarity, and `selectPeerGroup` can only be made faster once all `meta.ForwardProposal` calls are gone since we can then purely rely on atomics. Follow-up of https://github.com/nats-io/nats-server/pull/8432",
          "url": "https://github.com/nats-io/nats-server/pull/8443",
          "createdAt": "2026-08-07T18:06:36Z",
          "updatedAt": "2026-08-13T12:07:45Z",
          "timestamp": "2026-08-13T12:07:45Z",
          "metrics": {
            "reactions": 0,
            "comments": 3
          },
          "labels": [],
          "author": "MauriceVanVeen",
          "state": "open",
          "assignees": [],
          "change": "updated"
        }
      },
      {
        "id": "event:676e73e2f986d6544246",
        "signalId": "github:nats-io/nats-server:pull_request:8452",
        "event": "changed",
        "observedAt": "2026-08-13T13:48:00.446149Z",
        "changedFields": [],
        "signal": {
          "id": "github:nats-io/nats-server:pull_request:8452",
          "source": "github",
          "group": "data-infrastructure",
          "project": "nats-io/nats-server",
          "kind": "pull_request",
          "title": "(2.15) [FIXED] Evict peer-removed peers for group below quorum",
          "text": "If a stream/consumer Raft group was below quorum, it wasn't possible to safely peer-remove nodes to get into quorum again. This PR fixes that by allowing nodes to be evicted when below quorum, but only those nodes that were explicitly peer-removed. Given a R3 stream without quorum due to 2 nodes being offline, it's now possible to peer-remove those two offline nodes and the stream will become operational again, albeit it at R1. If new peers were assigned to the stream, the stream would automatically scale up to R3, but on the new peers. The desired state gets a new `rg.Desired.Removed` field that contains the set of explicitly peer-removed nodes, which are eligible for eviction if below quorum. Eviction is only allowed if this node thinks the stream/consumer Raft group is leaderless, it is not leaderless on the meta layer, it has desired state that specifies removed peers, and the node itself is still part of the group's peer set. A safety check also prevents eviction if the remaining peers could form a disjoint majority. Follow-up of https://github.com/nats-io/nats-server/pull/8432",
          "url": "https://github.com/nats-io/nats-server/pull/8452",
          "createdAt": "2026-08-11T14:10:44Z",
          "updatedAt": "2026-08-13T12:03:41Z",
          "timestamp": "2026-08-13T12:03:41Z",
          "metrics": {
            "reactions": 1,
            "comments": 3
          },
          "labels": [],
          "author": "MauriceVanVeen",
          "state": "open",
          "assignees": [],
          "change": "updated"
        }
      },
      {
        "id": "event:d58650191d82bd256c2d",
        "signalId": "github:nats-io/nats-server:pull_request:8460",
        "event": "changed",
        "observedAt": "2026-08-13T13:48:00.446149Z",
        "changedFields": [],
        "signal": {
          "id": "github:nats-io/nats-server:pull_request:8460",
          "source": "github",
          "group": "data-infrastructure",
          "project": "nats-io/nats-server",
          "kind": "pull_request",
          "title": "(2.15) [FIXED] Migration added offline peers & removed peers before catchup",
          "text": "This PR fixes two remaining issues, and removes the final two FIXMEs: - Migrations would add peers as fast as possible, without checking whether the peer being added was even online. Adding an offline peer grows the quorum size, which could drop the group out of quorum until enough nodes came online. `selectPeerToAdd` now prefers the candidate we've heard from most recently, and only adds an unheard peer if the live members still reach quorum in the grown group. - Migrations also didn't wait for upper-layer stream catchup to complete before removing old peers. An old peer is now only removed if the group left after the removal retains a store-current majority, based on `Current`/`Lag` and the stream's active `catchupPeers`. Follow-up of https://github.com/nats-io/nats-server/pull/8432",
          "url": "https://github.com/nats-io/nats-server/pull/8460",
          "createdAt": "2026-08-12T16:02:04Z",
          "updatedAt": "2026-08-13T11:56:54Z",
          "timestamp": "2026-08-13T11:56:54Z",
          "metrics": {
            "reactions": 1,
            "comments": 3
          },
          "labels": [],
          "author": "MauriceVanVeen",
          "state": "open",
          "assignees": [],
          "change": "updated"
        }
      },
      {
        "id": "event:69a103cefe26d06c6aff",
        "signalId": "github:nats-io/nats-server:pull_request:8384",
        "event": "changed",
        "observedAt": "2026-08-13T13:48:00.446149Z",
        "changedFields": [],
        "signal": {
          "id": "github:nats-io/nats-server:pull_request:8384",
          "source": "github",
          "group": "data-infrastructure",
          "project": "nats-io/nats-server",
          "kind": "pull_request",
          "title": "(2.15) [IMPROVED] Source stream recreation detection",
          "text": "Previously if you started sourcing from a stream that was recreated, it would wait for the recreated stream's sequence to move back up to the last sourced sequence before it accepted new messages. This PR detects a stream being recreated by noticing a changed `Nats-Stream-Identity`, which is the hash of the stream's created time. There exists no unique identifier for a stream/consumer lifetime apart from the created time, so that's used and is the most practical since collisions are unlikely enough. - This hash is appended to the source header to know what \"identity\" of the stream was associated with the highest sequence for that source. (source header before: `TEST 1 > > foo`, after: `TEST 1 > > foo sLkPMPK6`) - This identity is passed in the consumer create request to prevent a consumer from being created based on sequences derived from the previous stream. An error is returned (`JSConsumerStreamIdentityMismatch`) if a mismatch is detected, the request is then retried with reset sequences to allow sourcing from the beginning. - The consumer reset request does not allow for the identity to be passed in, and instead relies on the caller to notice a different identity and subsequent in-memory state reset. This prevents unnecessary churn of requests since the reset will always happen based on a consumer that already exists and needs to be reset anyway. - Both the consumer create and reset responses contain this identity to get informed of the latest identity of the stream, which is then passed into the source header. This PR additionally requires that the starting sequence for a consumer is respected, even for sourcing consumers that were previously (incorrectly) clamped within the sourced stream's first/last sequence range. Upgrade note: the servers on both ends will need to be updated for this mechanism to kick in. A downgrade should ignore the extra appended identity from the header and fall back to the previous behavior. Resolves https://github.com/nats-io/nats-server/issues/8346, https://github.com/nats-io/nats-server/issues/6206, https://github.com/nats-io/nats-server/issues/5847",
          "url": "https://github.com/nats-io/nats-server/pull/8384",
          "createdAt": "2026-07-15T15:39:20Z",
          "updatedAt": "2026-08-13T11:26:27Z",
          "timestamp": "2026-08-13T11:26:27Z",
          "metrics": {
            "reactions": 2,
            "comments": 9
          },
          "labels": [],
          "author": "MauriceVanVeen",
          "state": "open",
          "assignees": [],
          "change": "updated"
        }
      },
      {
        "id": "event:c878b3006200d23b6ed0",
        "signalId": "github:nats-io/nats-server:pull_request:8439",
        "event": "changed",
        "observedAt": "2026-08-13T13:48:00.446149Z",
        "changedFields": [],
        "signal": {
          "id": "github:nats-io/nats-server:pull_request:8439",
          "source": "github",
          "group": "data-infrastructure",
          "project": "nats-io/nats-server",
          "kind": "pull_request",
          "title": "(2.15) [ADDED] Report requested stream config while running at desired origin",
          "text": "The retention and placement from the stream's config must always return the latest state the user requested, and the desired state contains the 'origin' if needed. Before this PR the stream would already keep running under Limits if it was updated to Interest retention, since this allows the consumers to be scaled up to have parity with the stream's replication first, and only after safely switch the retention to Interest. But this would also result in the stream config in stream info and update responses as well as JSZ showing up as the 'origin' retention and placement instead of the latest values. This PR fixes that by always returning the latest state, and only the desired origin contains the previous value as expected. Additionally, it ensures the desired origin is always set, even when it's registered through a retention change, such that it can be canceled through the 'cancel move' endpoint. Follow-up of https://github.com/nats-io/nats-server/pull/8432",
          "url": "https://github.com/nats-io/nats-server/pull/8439",
          "createdAt": "2026-07-31T11:37:26Z",
          "updatedAt": "2026-08-13T09:52:28Z",
          "timestamp": "2026-08-13T09:52:28Z",
          "metrics": {
            "reactions": 1,
            "comments": 5
          },
          "labels": [],
          "author": "MauriceVanVeen",
          "state": "open",
          "assignees": [],
          "change": "updated"
        }
      },
      {
        "id": "event:6e4f18e53090bb18fb01",
        "signalId": "github:nats-io/nats-server:pull_request:8133",
        "event": "changed",
        "observedAt": "2026-08-13T13:48:00.446149Z",
        "changedFields": [],
        "signal": {
          "id": "github:nats-io/nats-server:pull_request:8133",
          "source": "github",
          "group": "data-infrastructure",
          "project": "nats-io/nats-server",
          "kind": "pull_request",
          "title": "[ADDED] Support PROXY protocol for websocket listeners",
          "text": "Add websocket `proxy_protocol` support by parsing optional PROXY headers on the raw listener before TLS and HTTP upgrade processing. Non-proxied websocket clients continue to work because pre-read bytes are replayed into the next protocol layer, and the setting is exposed in `/varz`. Validation: - `go test -v ./server -run '^(TestWSParseOptions|TestWSProxyProtocol|TestWSProxyProtocolNonProxiedConnection|TestMonitorWebsocket)$' -count=1 -vet=off`\\n- `git diff --check`\\n\\nResolves https://github.com/nats-io/nats-server/issues/7924",
          "url": "https://github.com/nats-io/nats-server/pull/8133",
          "createdAt": "2026-05-12T05:12:22Z",
          "updatedAt": "2026-08-13T09:48:25Z",
          "timestamp": "2026-08-13T09:48:25Z",
          "metrics": {
            "reactions": 2,
            "comments": 2
          },
          "labels": [],
          "author": "officialasishkumar",
          "state": "open",
          "assignees": [],
          "change": "updated"
        }
      },
      {
        "id": "event:cbec04251b4d8462014e",
        "signalId": "github:nats-io/nats-server:pull_request:8437",
        "event": "changed",
        "observedAt": "2026-08-13T13:48:00.446149Z",
        "changedFields": [],
        "signal": {
          "id": "github:nats-io/nats-server:pull_request:8437",
          "source": "github",
          "group": "data-infrastructure",
          "project": "nats-io/nats-server",
          "kind": "pull_request",
          "title": "(2.15) [ADDED] Drive stream move and cancel move via desired state",
          "text": "The move and cancel move endpoints are now wired fully through desired state. The cancel move endpoint can now be reliably used; the first move (either through stream updates or the move endpoint) initialize the \"origin\" which is the peer set and config (replicas, placement and retention) at that time, multiple moves or scales could technically be run even before the full desired state change completes, a cancel will then always roll back to the origin. However, concurrent stream move or scale requests are rejected to keep the peer set bounded, as otherwise cycling moves between clusters or cycling between replication counts could result in all peers being admitted larger than `2*replicas`. This restriction does not apply to consumers, they can be freely scaled within the stream's peer set. This PR additionally fixes the JS lock being released and re-acquired before calling `jsa.configUpdateCheck`, and ensures legacy moves are translated to desired state properly and can be rolled back through the improved cancel move endpoint as well. Follow-up of https://github.com/nats-io/nats-server/pull/8432",
          "url": "https://github.com/nats-io/nats-server/pull/8437",
          "createdAt": "2026-07-30T18:47:50Z",
          "updatedAt": "2026-08-13T09:48:16Z",
          "timestamp": "2026-08-13T09:48:16Z",
          "metrics": {
            "reactions": 0,
            "comments": 5
          },
          "labels": [],
          "author": "MauriceVanVeen",
          "state": "open",
          "assignees": [],
          "change": "updated"
        }
      },
      {
        "id": "event:6eb63f3594fad2063ecb",
        "signalId": "github:nats-io/nats-server:pull_request:8432",
        "event": "changed",
        "observedAt": "2026-08-13T13:48:00.446149Z",
        "changedFields": [],
        "signal": {
          "id": "github:nats-io/nats-server:pull_request:8432",
          "source": "github",
          "group": "data-infrastructure",
          "project": "nats-io/nats-server",
          "kind": "pull_request",
          "title": "(2.15) [ADDED] Desired state reconcilitation for scaling and moves",
          "text": "This PR adds initial support for desired state reconciliation of stream and consumer updates that change the peer set. These, and follow-up, changes intend to fix loads of related issues surrounding peer set management of assets. For example, including but not limited to: a scale down not being guaranteed to select the right leader or servers with all data to prevent data loss, peers re-appearing in the stream/consumer Raft group after peer-removing a node, moving and scaling a stream concurrently may brick it such that it's stuck halfway into a move or is reverted back to the previous cluster with the stream config still referencing where it was meant to move to, and a consumer reverting to stale state after a consumer move. This initial PR tries to do the minimal amount of work to have passing CI, while transitioning to a desired state design that allows for guaranteed and safe scaling and moving of assets. Importantly: - The meta leader MUST never overwrite a stream/consumer peer set in-place as it does today. The meta leader may dictate where a stream/consumer needs to move, but the stream/consumer is responsible for making those changes be guaranteed safe. - The meta leader always dictates the peer set, unless the stream/consumer is scaled down, since only the group itself knows which servers would be best to be preserved (i.e. the leader and most up-to-date servers), to guarantee no data loss. Previously the meta leader did a `sysRequest` stream/consumer info to figure out the leader (and for consumer moves the consumer state). Both could result in a stale leader/state being picked. And worse, if the leader doesn't respond in time (or at all), the meta leader would randomly pick a leader (or lose consumer state). This PR allows the removal of `sysRequest` entirely. - Peer additions and removals are managed by the group's leader, no automatic additions of peers are allowed (like for the meta group). Before the group's leader is allowed to make any change, it first needs to communicate with the meta leader to assert that it's up-to-date with the meta state (linearizable and no stale local assignment), and the leader's Raft `term` is also wired through, so the meta leader also knows to ignore requests from stale leaders. This PR allows removing `ProposeKnownPeers` and `UpdateKnownPeers` from `raft.go`, as well as the calls to them in `js.createRaftGroup`, since those were unsafe and could result in removed peers re-appearing, desyncing the peer set in the meta assignment with that of the group. - The stream/consumer `ClusterInfo` now returns the desired state information. Allowing tooling (like the CLI) to inspect the state as it changes. This PR is split in 3 commits, to ease reviewing but also ease backporting to 2.14. Put simply: the first commit adds the models, the second commit adds meta layer reconciliation awareness, and the last commit actually starts populating the desired state and triggering the stream/consumer migration paths. Likely 2.14 will require a compatibility commit to ease downgrading, and upgrading/downgrading as a whole should be tackled in a follow-up PR. However, the design of the desired state always guarantees: the stream/consumer config is always what the user asked it to be, the group's peer set is always the one that equals the underlying Raft group (or if the Raft group contains less peers, they get automatically added under the old behavior). So, worst-case given a downgrade to an incompatible version, the desired state configuration in the JSON will \"fall off\" and the assignment will remain \"stuck\" on which peers it was at that time. That does not violate correctness or safety when it comes to preserving data, it does however prevent the stream/consumer from moving to the desired config. Having passing CI and the mentioned upgrade/downgrade focus isn't enough, and at least these follow-ups will be required in separate PRs: - The system \"move\" and \"move cancel\" APIs should be made more reliable. A \"move\" can simply trigger a peer set change through desired state, and a \"move cancel\" can look at the desired state's origin to see where the stream needs to be moved back to. Both just need time to be wired appropriately. - Server and asset-level peer-remove should be made desired state aware. Additionally, an \"evacuate\" endpoint should be added to enable the user to more easily move streams and consumers off a server while it's still online. Not only easing the decommissioning of a node, but also enabling to move data of R1 assets without losing data when a peer-remove of the node would've been used instead and without requiring to move all R1 assets in all accounts manually. - Add stream move/scale guards. This PR removes the (admittedly brittle) guards to prevent concurrent move and scale of a stream, which allows to scale and move an R1 stream from cluster A to a R3 stream in cluster B in one go and safely. However, this technically opens up being able to constantly cycle moves between clusters until the stream/consumer peer set contain the total number of peers. This should likely get an upper bound, for example 10 peers maximum (since replication is capped at 5), to ensure such moves are still allowed from the API, but the system prevents the peers from growing semi-unbounded. - Optimizations in how and when peers are added, ensuring a group doesn't go out of quorum when nodes are offline. - An additional optimization can be made when moving a stream between clusters, to first scale up one stream replica in the new cluster, making it the leader and then doing local catchup for the other replicas in that cluster. This would allow both faster and less expensive (bandwidth-wise) moves of assets. But, although this is enabled by the desired state approach as well, it might need to be done after 2.15.0 lands considering timing and the other required follow-ups (this isn't a required optimization). These follow-ups are also roughly marked with `FIXME` comments that need to be addressed prior to releasing 2.15.",
          "url": "https://github.com/nats-io/nats-server/pull/8432",
          "createdAt": "2026-07-29T13:15:47Z",
          "updatedAt": "2026-08-13T09:30:59Z",
          "timestamp": "2026-08-13T09:30:59Z",
          "metrics": {
            "reactions": 0,
            "comments": 6
          },
          "labels": [],
          "author": "MauriceVanVeen",
          "state": "open",
          "assignees": [],
          "change": "updated"
        }
      },
      {
        "id": "event:28bd7ed3a9d5bfe78a80",
        "signalId": "github:nats-io/nats-server:issue:5811",
        "event": "changed",
        "observedAt": "2026-08-13T13:48:00.446149Z",
        "changedFields": [],
        "signal": {
          "id": "github:nats-io/nats-server:issue:5811",
          "source": "github",
          "group": "data-infrastructure",
          "project": "nats-io/nats-server",
          "kind": "issue",
          "title": "Always retain last message for each subject in a stream",
          "text": "### Proposed change Add a `--keep-last-msg-per-subject` flag to the nats server configuration i.e. `nats stream add --keep-last-msg-per-subject <more-options> <stream>` that works with all limits-based message retention policies (i.e time-based, size-based, count-based). ### Use case Stream limits options (e.g. `max-age`, `max-bytes`, `max-msgs` etc.) mostly seem to operate on a per stream basis. The exception is the `max-msgs-per-subject` which allows for a per subject based limit. It would be nice to have the capability to always retain the last message on all given subjects in a stream while setting a time-based message retention policy i.e. delete all messages past an age of 10 mins except the last. A use case of this can be as follows: - We want to record changes to a configuration (stored in a nats subject) and keep a history for a certain period of time e.g. 5 days (specified by the --max-age flag) - We always want to be able to fetch the last configuration to initialize our long-running application (which may run for weeks on end) whenever it restarts ### Contribution Yes, I can contribute to this feature",
          "url": "https://github.com/nats-io/nats-server/issues/5811",
          "createdAt": "2024-08-20T19:33:25Z",
          "updatedAt": "2026-08-13T08:57:16Z",
          "timestamp": "2026-08-13T08:57:16Z",
          "metrics": {
            "reactions": 0,
            "comments": 2
          },
          "labels": [
            "stale",
            "proposal"
          ],
          "author": "SomeshDaga",
          "state": "open",
          "assignees": [],
          "change": "updated"
        }
      },
      {
        "id": "event:c7cc782508214e058402",
        "signalId": "github:nats-io/nats-server:pull_request:8457",
        "event": "changed",
        "observedAt": "2026-08-13T13:48:00.446149Z",
        "changedFields": [],
        "signal": {
          "id": "github:nats-io/nats-server:pull_request:8457",
          "source": "github",
          "group": "data-infrastructure",
          "project": "nats-io/nats-server",
          "kind": "pull_request",
          "title": "De-flake TestJetStreamSuperClusterStreamDirectGetMirrorQueueGroup",
          "url": "https://github.com/nats-io/nats-server/pull/8457",
          "createdAt": "2026-08-12T12:17:38Z",
          "updatedAt": "2026-08-12T15:45:09Z",
          "timestamp": "2026-08-12T15:45:09Z",
          "metrics": {
            "reactions": 1,
            "comments": 2
          },
          "labels": [],
          "author": "sciascid",
          "state": "closed",
          "assignees": [],
          "change": "updated"
        }
      },
      {
        "id": "event:bd31273d80a4b4eb525a",
        "signalId": "github:nats-io/nats-server:pull_request:8459",
        "event": "changed",
        "observedAt": "2026-08-13T13:48:00.446149Z",
        "changedFields": [],
        "signal": {
          "id": "github:nats-io/nats-server:pull_request:8459",
          "source": "github",
          "group": "data-infrastructure",
          "project": "nats-io/nats-server",
          "kind": "pull_request",
          "title": "Update dependencies",
          "url": "https://github.com/nats-io/nats-server/pull/8459",
          "createdAt": "2026-08-12T14:16:34Z",
          "updatedAt": "2026-08-12T15:32:11Z",
          "timestamp": "2026-08-12T15:32:11Z",
          "metrics": {
            "reactions": 1,
            "comments": 1
          },
          "labels": [],
          "author": "neilalexander",
          "state": "closed",
          "assignees": [],
          "change": "updated"
        }
      },
      {
        "id": "event:04f6baa9f47484f37d8c",
        "signalId": "github:nats-io/nats-server:release:369333995",
        "event": "discovered",
        "observedAt": "2026-08-13T13:48:00.446149Z",
        "changedFields": [],
        "signal": {
          "id": "github:nats-io/nats-server:release:369333995",
          "source": "github",
          "group": "data-infrastructure",
          "project": "nats-io/nats-server",
          "kind": "release",
          "title": "Release v2.14.5",
          "text": "## Changelog Refer to the [2.14 Upgrade Guide](https://docs.nats.io/release-notes/whats_new/whats_new_214) for backwards compatibility notes with 2.12.x. Please note that the 2.13.x version was skipped. ### Go Version - 1.26.5 ### Dependencies - github.com/klauspost/compress v1.19.2 (#8459) - golang.org/x/crypto v0.55.0 (#8459) ### Added Leafnodes - New `dial_timeout` option can be specified in the `leafnode` config block or for specific remotes in the configuration, allowing it to be increased above the default 1 second for high-latency links (#8427) ### Fixed General - Fixed a deadlock that could occur in the logger when a close error occurs (#8430) JetStream - Fix a bug that could result in potential data loss when handling idempotent stream creates when an offline node catches up from a metalayer snapshot, caused by an incorrect update to the create time in the stream assignment (#8449) ### Complete Changes https://github.com/nats-io/nats-server/compare/v2.14.4...v2.14.5",
          "url": "https://github.com/nats-io/nats-server/releases/tag/v2.14.5",
          "createdAt": "2026-08-12T15:31:27Z",
          "updatedAt": "2026-08-12T15:31:27Z",
          "timestamp": "2026-08-12T15:31:27Z",
          "labels": [],
          "author": "github-actions[bot]",
          "assignees": [],
          "change": "new"
        }
      },
      {
        "id": "event:8beeb4b8e3024b66263d",
        "signalId": "github:nats-io/nats-server:release:369334266",
        "event": "discovered",
        "observedAt": "2026-08-13T13:48:00.446149Z",
        "changedFields": [],
        "signal": {
          "id": "github:nats-io/nats-server:release:369334266",
          "source": "github",
          "group": "data-infrastructure",
          "project": "nats-io/nats-server",
          "kind": "release",
          "title": "Release v2.12.15",
          "text": "## Changelog ### Go Version - 1.25.12 ### Dependencies - github.com/klauspost/compress v1.19.2 (#8459) - golang.org/x/crypto v0.55.0 (#8459) ### Fixed General - Fixed a deadlock that could occur in the logger when a close error occurs (#8430) JetStream - Fix a bug that could result in potential data loss when handling idempotent stream creates when an offline node catches up from a metalayer snapshot, caused by an incorrect update to the create time in the stream assignment (#8449) ### Complete Changes https://github.com/nats-io/nats-server/compare/v2.12.14...v2.12.15",
          "url": "https://github.com/nats-io/nats-server/releases/tag/v2.12.15",
          "createdAt": "2026-08-12T15:30:42Z",
          "updatedAt": "2026-08-12T15:30:42Z",
          "timestamp": "2026-08-12T15:30:42Z",
          "labels": [],
          "author": "github-actions[bot]",
          "assignees": [],
          "change": "new"
        }
      },
      {
        "id": "event:d6fc0db29c448ff572ec",
        "signalId": "github:nats-io/nats-server:pull_request:8460",
        "event": "changed",
        "observedAt": "2026-08-13T16:19:22.035158Z",
        "changedFields": [
          "updatedAt"
        ],
        "signal": {
          "id": "github:nats-io/nats-server:pull_request:8460",
          "source": "github",
          "group": "data-infrastructure",
          "project": "nats-io/nats-server",
          "kind": "pull_request",
          "title": "(2.15) [FIXED] Migration added offline peers & removed peers before catchup",
          "text": "This PR fixes two remaining issues, and removes the final two FIXMEs: - Migrations would add peers as fast as possible, without checking whether the peer being added was even online. Adding an offline peer grows the quorum size, which could drop the group out of quorum until enough nodes came online. `selectPeerToAdd` now prefers the candidate we've heard from most recently, and only adds an unheard peer if the live members still reach quorum in the grown group. - Migrations also didn't wait for upper-layer stream catchup to complete before removing old peers. An old peer is now only removed if the group left after the removal retains a store-current majority, based on `Current`/`Lag` and the stream's active `catchupPeers`. Follow-up of https://github.com/nats-io/nats-server/pull/8432",
          "url": "https://github.com/nats-io/nats-server/pull/8460",
          "createdAt": "2026-08-12T16:02:04Z",
          "updatedAt": "2026-08-13T14:28:17Z",
          "timestamp": "2026-08-13T14:28:17Z",
          "metrics": {
            "reactions": 1,
            "comments": 3
          },
          "labels": [],
          "author": "MauriceVanVeen",
          "state": "open",
          "assignees": [],
          "change": "updated"
        }
      },
      {
        "id": "event:0965a649e3387ef85af2",
        "signalId": "github:nats-io/nats-server:pull_request:8384",
        "event": "changed",
        "observedAt": "2026-08-13T16:19:22.035158Z",
        "changedFields": [
          "updatedAt",
          "state"
        ],
        "signal": {
          "id": "github:nats-io/nats-server:pull_request:8384",
          "source": "github",
          "group": "data-infrastructure",
          "project": "nats-io/nats-server",
          "kind": "pull_request",
          "title": "(2.15) [IMPROVED] Source stream recreation detection",
          "text": "Previously if you started sourcing from a stream that was recreated, it would wait for the recreated stream's sequence to move back up to the last sourced sequence before it accepted new messages. This PR detects a stream being recreated by noticing a changed `Nats-Stream-Identity`, which is the hash of the stream's created time. There exists no unique identifier for a stream/consumer lifetime apart from the created time, so that's used and is the most practical since collisions are unlikely enough. - This hash is appended to the source header to know what \"identity\" of the stream was associated with the highest sequence for that source. (source header before: `TEST 1 > > foo`, after: `TEST 1 > > foo sLkPMPK6`) - This identity is passed in the consumer create request to prevent a consumer from being created based on sequences derived from the previous stream. An error is returned (`JSConsumerStreamIdentityMismatch`) if a mismatch is detected, the request is then retried with reset sequences to allow sourcing from the beginning. - The consumer reset request does not allow for the identity to be passed in, and instead relies on the caller to notice a different identity and subsequent in-memory state reset. This prevents unnecessary churn of requests since the reset will always happen based on a consumer that already exists and needs to be reset anyway. - Both the consumer create and reset responses contain this identity to get informed of the latest identity of the stream, which is then passed into the source header. This PR additionally requires that the starting sequence for a consumer is respected, even for sourcing consumers that were previously (incorrectly) clamped within the sourced stream's first/last sequence range. Upgrade note: the servers on both ends will need to be updated for this mechanism to kick in. A downgrade should ignore the extra appended identity from the header and fall back to the previous behavior. Resolves https://github.com/nats-io/nats-server/issues/8346, https://github.com/nats-io/nats-server/issues/6206, https://github.com/nats-io/nats-server/issues/5847",
          "url": "https://github.com/nats-io/nats-server/pull/8384",
          "createdAt": "2026-07-15T15:39:20Z",
          "updatedAt": "2026-08-13T14:19:35Z",
          "timestamp": "2026-08-13T14:19:35Z",
          "metrics": {
            "reactions": 2,
            "comments": 9
          },
          "labels": [],
          "author": "MauriceVanVeen",
          "state": "closed",
          "assignees": [],
          "change": "updated"
        }
      },
      {
        "id": "event:6f01dc171abd14b96859",
        "signalId": "github:nats-io/nats-server:issue:8346",
        "event": "discovered",
        "observedAt": "2026-08-13T16:19:22.035158Z",
        "changedFields": [],
        "signal": {
          "id": "github:nats-io/nats-server:issue:8346",
          "source": "github",
          "group": "data-infrastructure",
          "project": "nats-io/nats-server",
          "kind": "issue",
          "title": "JetStream sourced stream reports lag=0 after source stream recreation but no longer ingests new messages",
          "text": "### Observed behavior We have a JetStream deployment where a destination stream sources data from many remote JetStream domains using external stream sources. For one source, after the source stream was recreated, the destination stream stopped ingesting new messages from that source. However, stream info continues to report: lag: 0 no error non-zero active making the source appear healthy even though no new messages are arriving. Simplified topology <img width=\"413\" height=\"296\" alt=\"Image\" src=\"https://github.com/user-attachments/assets/eb91a0d6-6d8a-4257-94f5-4be0da4ab5e7\" /> ORDERS sources from many remote JetStream domains. All sources continue functioning normally except H1314. Configuration The destination stream contains a source similar to: ```json { \"name\": \"H1314\", \"external\": { \"api\": \"$JS.H1314.API\" }, \"opt_start_time\": \"2026-06-30T00:00:00Z\" } ``` What happened The source stream H1314 was recreated. The recreated source stream continues receiving new messages. For example: nats sub EXAMPLE.SUBJECT --last --js-domain H1314 Source stream: H1314 Sequence: 1029746 Timestamp: 2026-06-30 17:47:54 UTC Subject: EXAMPLE.SUBJECT However, the most recent message for the same subject in the destination stream remains from the previous day: nats sub EXAMPLE.SUBJECT --stream ORDERS --last Destination stream: ORDERS Timestamp: 2026-06-29 21:24:36 UTC Subject: EXAMPLE.SUBJECT The copied message still contains a header similar to: Nats-Stream-Source: H1314:<consumer> 12002532 > > EXAMPLE.SUBJECT which references the previous incarnation of the source stream. Observed behavior nats stream info ORDERS -j reports: ```json { \"name\": \"H1314\", \"lag\": 0, \"active\": 265738727 } ``` There is no reported error. Despite this, no new messages from H1314 are arriving in ORDERS. I tested creating a new stream TEST1314 with a single source [\"H1314\"] and it replicated into the new stream just fine. ### Expected behavior The destination stream reconnects to the recreated source stream and resumes sourcing automatically. or JetStream reports that sourcing has failed (non-zero lag or an error condition). Reporting lag=0 while the destination stream is no longer ingesting messages makes the problem difficult to diagnose. ### Server and client version nats-server: v2.14.0 (all nodes) CLI version v0.4.0 ### Host environment _No response_ ### Steps to reproduce _No response_",
          "url": "https://github.com/nats-io/nats-server/issues/8346",
          "createdAt": "2026-07-01T14:12:57Z",
          "updatedAt": "2026-08-13T14:19:34Z",
          "timestamp": "2026-08-13T14:19:34Z",
          "metrics": {
            "reactions": 0,
            "comments": 4
          },
          "labels": [
            "defect"
          ],
          "author": "ordeneaux",
          "state": "closed",
          "assignees": [],
          "change": "new"
        }
      },
      {
        "id": "event:8f20c3e3774c525ef340",
        "signalId": "github:nats-io/nats-server:issue:6206",
        "event": "discovered",
        "observedAt": "2026-08-13T16:19:22.035158Z",
        "changedFields": [],
        "signal": {
          "id": "github:nats-io/nats-server:issue:6206",
          "source": "github",
          "group": "data-infrastructure",
          "project": "nats-io/nats-server",
          "kind": "issue",
          "title": "JetStream stream source stalls/is out of sync after sourced stream is re-created for unrelated reason [v2.10.20]",
          "text": "### Observed behavior Setup: basic hub spoke with leafnodes. hub is a single jetstream enabled server in its own domain, so are the leaf nodes. On the hub, there is a stream named match_events that sources from the match_events stream in each leaf node. Symptoms: Sourcing from match_events on one leafnode stopped working. Despite new messages being added to it, no data is sent to match_events in the hub. Cause: I believe we ran into a crash during recovery that sounds similar to one fixed in a release candidate of 2.10.23. In order to recover from this issue, we renamed the data directory, which resulted in data loss but allowed the system to become functional again. On the leaf node with the issue, sequence numbers in the stream are less than 80. Create time is right after the time when the jetstream data directory got renamed and everything started to seemingly function again. On the hub node, stream info showed the edge connected with low lag. But when subscribing to the consumer create message of that leaf, we noticed that the requested start seq no is 995 and hence way bigger than 80. Meaning 995 was the sequence number the previous incarnation of match_events had on that leaf and 79 is the value of the current incarnation This would explain why there seems to be no data flow ... yet. Failed workaround: We edited the match_events stream in the hub to remove the source corresponding to the leaf. Then we re-added it. This did not seem to work as we recorded the consumer create messages and the sequence number remained at the old value it had before the source got removed & re-added. I then edited the source to set the op_start_seq to 1. This didn't work either, as presumably a higher sequence number was still in memory. A few hours later we restarted the hub server. After that the sequence number was reset to one. ---------------------- Logs: Hub consumer create messages: ``` [#13] Received on \"$JS.edge-652efb2da425e854a1e8d203.API.CONSUMER.CREATE.match_events\" with reply \"$JSC.R.FEkNK9Is\" {\"stream_name\":\"match_events\",\"config\":{\"deliver_policy\":\"by_start_sequence\",\"opt_start_seq\":995,\"ack_policy\":\"none\",\"ack_wait\":79200000000000,\"max_deliver\":1,\"replay_policy\":\"instant\",\"idle_heartbeat\":1000000000,\"flow_control\":true,\"deliver_subject\":\"deliver.edge-652efb2da425e854a1e8d203.match_events.S.l2lkXkVm\",\"inactive_threshold\":10000000000,\"num_replicas\":0,\"direct\":true},\"action\":\"\"} [#21] Received on \"$JS.edge-652efb2da425e854a1e8d203.API.CONSUMER.CREATE.match_events\" with reply \"$JSC.R.3L70SiTW\" {\"stream_name\":\"match_events\",\"config\":{\"deliver_policy\":\"by_start_sequence\",\"opt_start_seq\":995,\"ack_policy\":\"none\",\"ack_wait\":79200000000000,\"max_deliver\":1,\"replay_policy\":\"instant\",\"idle_heartbeat\":1000000000,\"flow_control\":true,\"deliver_subject\":\"deliver.edge-652efb2da425e854a1e8d203.match_events.S.qyIUgE5D\",\"inactive_threshold\":10000000000,\"num_replicas\":0,\"direct\":true},\"action\":\"\"} ``` Leafnode match_events stream: ``` > nats stream ls -a ╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ Streams │ ├─────────────────────┬────────────────────────────────────────────┬─────────────────────┬───────────┬─────────┬──────────────┤ │ Name │ Description │ Created │ Messages │ Size │ Last Message │ ├─────────────────────┼────────────────────────────────────────────┼─────────────────────┼───────────┼─────────┼──────────────┤ ... │ match_events │ │ 2024-11-27 17:53:49 │ 79 │ 15 MiB │ 25m10s │ ... ╰─────────────────────┴────────────────────────────────────────────┴─────────────────────┴───────────┴─────────┴──────────────╯ > nats stream info match_events Information for Stream match_events created 2024-11-27 17:53:49 Subjects: faceio.match.> Replicas: 1 Storage: File Options: Retention: Limits Acknowledgments: true Discard Policy: Old Duplicate Window: 2m0s Allows Msg Delete: true Allows Purge: true Allows Rollups: false Limits: Maximum Messages: unlimited Maximum Per Subject: unlimited Maximum Bytes: unlimited Maximum Age: 7d0h0m0s Maximum Message Size: unlimited Maximum Consumers: unlimited Cluster Information: Name: edge-652efb2da425e854a1e8d203 Leader: edge-652efb2da425e854a1e8d203 State: Messages: 79 Bytes: 15 MiB First Sequence: 1 @ 2024-11-27 18:13:14 Last Sequence: 79 @ 2024-12-02 17:57:52 Active Consumers: 1 Number of Subjects: 79 > ``` Hub consumer create messages for failed workaround: The first message was received after re-creating the source. The second message was received after editing opt_start_seq of the source. In both cases, opt_start_seq in the request remains at the last old value. ``` >nats sub \"$JS.edge-652efb2da425e854a1e8d203.API.CONSUMER.CREATE.match_events\" 10:44:28 Subscribing on $JS.edge-652efb2da425e854a1e8d203.API.CONSUMER.CREATE.match_events [#1] Received on \"$JS.edge-652efb2da425e854a1e8d203.API.CONSUMER.CREATE.match_events\" with reply \"$JSC.R.XoBsIZ61\" {\"stream_name\":\"match_events\",\"config\":{\"deliver_policy\":\"by_start_sequence\",\"opt_start_seq\":995,\"ack_policy\":\"none\",\"ack_wait\":79200000000000,\"max_deliver\":1,\"replay_policy\":\"instant\",\"idle_heartbeat\":1000000000,\"flow_control\":true,\"deliver_subject\":\"deliver.edge-652efb2da425e854a1e8d203.match_events.S.58t1cta4\",\"inactive_threshold\":10000000000,\"num_replicas\":0,\"direct\":true},\"action\":\"\"} [#2] Received on \"$JS.edge-652efb2da425e854a1e8d203.API.CONSUMER.CREATE.match_events\" with reply \"$JSC.R.h5dA0pYu\" {\"stream_name\":\"match_events\",\"config\":{\"deliver_policy\":\"by_start_sequence\",\"opt_start_seq\":995,\"ack_policy\":\"none\",\"ack_wait\":79200000000000,\"max_deliver\":1,\"replay_policy\":\"instant\",\"idle_heartbeat\":1000000000,\"flow_control\":true,\"deliver_subject\":\"deliver.edge-652efb2da425e854a1e8d203.match_events.S.j4aQssSi\",\"inactive_threshold\":10000000000,\"num_replicas\":0,\"direct\":true},\"action\":\"\"} ``` Hub consumer create messages after hub nats server restart show opt_start_seq=1. ``` [#7] Received on \"$JS.edge-652efb2da425e854a1e8d203.API.CONSUMER.CREATE.match_events\" with reply \"$JSC.R.snChCWwx\" {\"stream_name\":\"match_events\",\"config\":{\"deliver_policy\":\"by_start_sequence\",\"opt_start_seq\":995,\"ack_policy\":\"none\",\"ack_wait\":79200000000000,\"max_deliver\":1,\"replay_policy\":\"instant\",\"idle_heartbeat\":1000000000,\"flow_control\":true,\"deliver_subject\":\"deliver.edge-652efb2da425e854a1e8d203.match_events.S.6U6FCNWD\",\"inactive_threshold\":10000000000,\"num_replicas\":0,\"direct\":true},\"action\":\"\"} 15:19:43 Disconnected due to: read tcp 127.0.0.1:32592->127.0.0.1:4222: wsarecv: An existing connection was forcibly closed by the remote host., will attempt reconnect 15:20:09 Reconnected [nats://ff-queue-user:LtJQaah4Y2PUGAUyx8_oLZ@127.0.0.1:4222] [#8] Received on \"$JS.edge-652efb2da425e854a1e8d203.API.CONSUMER.CREATE.match_events\" with reply \"$JSC.R.hwd0hutm\" {\"stream_name\":\"match_events\",\"config\":{\"deliver_policy\":\"by_start_sequence\",\"opt_start_seq\":1,\"ack_policy\":\"none\",\"ack_wait\":79200000000000,\"max_deliver\":1,\"replay_policy\":\"instant\",\"idle_heartbeat\":1000000000,\"flow_control\":true,\"deliver_subject\":\"deliver.edge-652efb2da425e854a1e8d203.match_events.S.ZoqTXdlB\",\"inactive_threshold\":10000000000,\"num_replicas\":0,\"direct\":true},\"action\":\"\"} ``` Hub match_events stream taken right after consumer create [#2] from above, showing start seqno 1 ``` Information for Stream match_events created 2024-11-15 10:45:14 Subjects: faceio.match.> Replicas: 1 Storage: File Options: Retention: Limits Acknowledgments: true Discard Policy: Old Duplicate Window: 2m0s Allows Msg Delete: true Allows Purge: true Allows Rollups: false Limits: Maximum Messages: unlimited Maximum Per Subject: unlimited Maximum Bytes: unlimited Maximum Age: 7d0h0m0s Maximum Message Size: unlimited Maximum Consumers: unlimited Replication: Sources: ... match_events, Start Seq: 1, API Prefix: $JS.edge-652efb2da425e854a1e8d203.API, Delivery Prefix: deliver.edge-652efb2da425e854a1e8d203.match_events ... Source Information: ... Stream Name: match_events Lag: 0 Last Seen: 934ms Ext. API Prefix: $JS.edge-652efb2da425e854a1e8d203.API Ext. Delivery Prefix: deliver.edge-652efb2da425e854a1e8d203.match_events ... State: Messages: 40,303 Bytes: 7.4 GiB First Sequence: 505,520 @ 2024-11-26 11:02:59 Last Sequence: 545,822 @ 2024-12-03 11:02:56 Active Consumers: 1 Number of Subjects: 37,538 ``` Leafnode nats-server logs showing restart loop: nats server basically starts and exits right away. (within same second. different from being killed because it's taking too long) ``` [15052] 2024/11/27 17:43:00.113089 [INF] Starting nats-server [15052] 2024/11/27 17:43:00.113621 [INF] Version: 2.10.20 [15052] 2024/11/27 17:43:00.113621 [INF] Git: [7140387] [15052] 2024/11/27 17:43:00.113621 [INF] Cluster: edge-652efb2da425e854a1e8d203 [15052] 2024/11/27 17:43:00.113621 [INF] Name: edge-652efb2da425e854a1e8d203 [15052] 2024/11/27 17:43:00.113621 [INF] Node: cSXjN80e [15052] 2024/11/27 17:43:00.113621 [INF] ID: NCPMUJYVKW6V742BE4GN4RH2V6LCIFTB5MESCYDKJKURJMXCVQWJQXOQ [15052] 2024/11/27 17:43:00.113671 [INF] Using configuration file: C:\\Program Files\\FaceFirst\\FaceFirst-MessageBroker-Edge\\conf\\edge.conf [15052] 2024/11/27 17:43:00.114322 [INF] Starting http monitor on 127.0.0.1:8223 [15052] 2024/11/27 17:43:00.114322 [INF] Starting JetStream [15052] 2024/11/27 17:43:00.114827 [INF] _ ___ _____ ___ _____ ___ ___ _ __ __ [15052] 2024/11/27 17:43:00.114884 [INF] _ | | __|_ _/ __|_ _| _ \\ __| /_\\ | \\/ | [15052] 2024/11/27 17:43:00.114884 [INF] | || | _| | | \\__ \\ | | | / _| / _ \\| |\\/| | [15052] 2024/11/27 17:43:00.114884 [INF] \\__/|___| |_| |___/ |_| |_|_\\___/_/ \\_\\_| |_| [15052] 2024/11/27 17:43:00.114884 [INF] [15052] 2024/11/27 17:43:00.114884 [INF] https://docs.nats.io/jetstream [15052] 2024/11/27 17:43:00.114884 [INF] [15052] 2024/11/27 17:43:00.114884 [INF] ---------------- JETSTREAM ---------------- [15052] 2024/11/27 17:43:00.114884 [INF] Max Memory: 23.94 GB [15052] 2024/11/27 17:43:00.114884 [INF] Max Storage: 1.00 TB [15052] 2024/11/27 17:43:00.114884 [INF] Store Directory: \"D:\\ProgramData\\FaceFirst\\FaceFirst-MessageBroker-Edge\\data-store\\jetstream\" [15052] 2024/11/27 17:43:00.114884 [INF] Domain: edge-652efb2da425e854a1e8d203 [15052] 2024/11/27 17:43:00.114884 [INF] ------------------------------------------- [15052] 2024/11/27 17:43:00.115438 [INF] Starting restore for stream 'FF-QUEUE-ACC > KV_enrollments' [15052] 2024/11/27 17:43:00.166680 [INF] Restored 69,478 messages for stream 'FF-QUEUE-ACC > KV_enrollments' in 51ms [15052] 2024/11/27 17:43:00.166680 [INF] Starting restore for stream 'FF-QUEUE-ACC > KV_media' [15052] 2024/11/27 17:43:00.208948 [WRN] Filestore [KV_media] loadBlock error: message block data missing [15052] 2024/11/27 17:43:00.211651 [WRN] Filestore [KV_media] Detected skew in subject-based total (64) vs raw total (58), rebuilding [15052] 2024/11/27 17:43:00.238941 [INF] Restored 58 messages for stream 'FF-QUEUE-ACC > KV_media' in 72ms [15052] 2024/11/27 17:43:00.239938 [INF] Starting restore for stream 'FF-QUEUE-ACC > KV_settings' [15052] 2024/11/27 17:43:00.239938 [INF] Restored 42 messages for stream 'FF-QUEUE-ACC > KV_settings' in 0s [15052] 2024/11/27 17:43:00.240936 [INF] Starting restore for stream 'FF-QUEUE-ACC > KV_visitors' [17012] 2024/11/27 17:45:03.817424 [INF] Starting nats-server [17012] 2024/11/27 17:45:03.818469 [INF] Version: 2.10.20 [17012] 2024/11/27 17:45:03.818469 [INF] Git: [7140387] [17012] 2024/11/27 17:45:03.818469 [INF] Cluster: edge-652efb2da425e854a1e8d203 [17012] 2024/11/27 17:45:03.818469 [INF] Name: edge-652efb2da425e854a1e8d203 [17012] 2024/11/27 17:45:03.818469 [INF] Node: cSXjN80e [17012] 2024/11/27 17:45:03.818469 [INF] ID: NBZL4XFOC64CKXUWPM7ZCS7NNDI5X25LSMHXWZOCCB7XVM4UTOP6TD66 [17012] 2024/11/27 17:45:03.818469 [INF] Using configuration file: C:\\Program Files\\FaceFirst\\FaceFirst-MessageBroker-Edge\\conf\\edge.conf [17012] 2024/11/27 17:45:03.819454 [INF] Starting http monitor on 127.0.0.1:8223 [17012] 2024/11/27 17:45:03.819454 [INF] Starting JetStream [17012] 2024/11/27 17:45:03.819977 [INF] _ ___ _____ ___ _____ ___ ___ _ __ __ [17012] 2024/11/27 17:45:03.819977 [INF] _ | | __|_ _/ __|_ _| _ \\ __| /_\\ | \\/ | [17012] 2024/11/27 17:45:03.819977 [INF] | || | _| | | \\__ \\ | | | / _| / _ \\| |\\/| | [17012] 2024/11/27 17:45:03.819977 [INF] \\__/|___| |_| |___/ |_| |_|_\\___/_/ \\_\\_| |_| [17012] 2024/11/27 17:45:03.819977 [INF] [17012] 2024/11/27 17:45:03.819977 [INF] https://docs.nats.io/jetstream [17012] 2024/11/27 17:45:03.819977 [INF] [17012] 2024/11/27 17:45:03.819977 [INF] ---------------- JETSTREAM ---------------- [17012] 2024/11/27 17:45:03.819977 [INF] Max Memory: 23.94 GB [17012] 2024/11/27 17:45:03.819977 [INF] Max Storage: 1.00 TB [17012] 2024/11/27 17:45:03.819977 [INF] Store Directory: \"D:\\ProgramData\\FaceFirst\\FaceFirst-MessageBroker-Edge\\data-store\\jetstream\" [17012] 2024/11/27 17:45:03.819977 [INF] Domain: edge-652efb2da425e854a1e8d203 [17012] 2024/11/27 17:45:03.819977 [INF] ------------------------------------------- [17012] 2024/11/27 17:45:03.821034 [INF] Starting restore for stream 'FF-QUEUE-ACC > KV_enrollments' [17012] 2024/11/27 17:45:03.878169 [INF] Restored 69,478 messages for stream 'FF-QUEUE-ACC > KV_enrollments' in 57ms [17012] 2024/11/27 17:45:03.878169 [INF] Starting restore for stream 'FF-QUEUE-ACC > KV_media' [17012] 2024/11/27 17:45:03.879175 [WRN] Filestore [KV_media] loadBlock error: message block data missing [17012] 2024/11/27 17:45:03.882158 [WRN] Filestore [KV_media] Detected skew in subject-based total (64) vs raw total (58), rebuilding [17012] 2024/11/27 17:45:03.911169 [INF] Restored 58 messages for stream 'FF-QUEUE-ACC > KV_media' in 33ms [17012] 2024/11/27 17:45:03.911169 [INF] Starting restore for stream 'FF-QUEUE-ACC > KV_settings' [17012] 2024/11/27 17:45:03.911169 [INF] Restored 42 messages for stream 'FF-QUEUE-ACC > KV_settings' in 0s [17012] 2024/11/27 17:45:03.912166 [INF] Starting restore for stream 'FF-QUEUE-ACC > KV_visitors' [14928] 2024/11/27 17:47:06.923509 [INF] Starting nats-server [14928] 2024/11/27 17:47:06.924012 [INF] Version: 2.10.20 [14928] 2024/11/27 17:47:06.924068 [INF] Git: [7140387] [14928] 2024/11/27 17:47:06.924068 [INF] Cluster: edge-652efb2da425e854a1e8d203 [14928] 2024/11/27 17:47:06.924068 [INF] Name: edge-652efb2da425e854a1e8d203 [14928] 2024/11/27 17:47:06.924068 [INF] Node: cSXjN80e [14928] 2024/11/27 17:47:06.924068 [INF] ID: NBNOJN5SXYFVUKV73CCNICSKU22K2TBCLHQW2GG5ATAQAHIWI2GMWYDA [14928] 2024/11/27 17:47:06.924109 [INF] Using configuration file: C:\\Program Files\\FaceFirst\\FaceFirst-MessageBroker-Edge\\conf\\edge.conf [14928] 2024/11/27 17:47:06.924644 [INF] Starting http monitor on 127.0.0.1:8223 [14928] 2024/11/27 17:47:06.924644 [INF] Starting JetStream [14928] 2024/11/27 17:47:06.925157 [INF] _ ___ _____ ___ _____ ___ ___ _ __ __ [14928] 2024/11/27 17:47:06.925157 [INF] _ | | __|_ _/ __|_ _| _ \\ __| /_\\ | \\/ | [14928] 2024/11/27 17:47:06.925157 [INF] | || | _| | | \\__ \\ | | | / _| / _ \\| |\\/| | [14928] 2024/11/27 17:47:06.925157 [INF] \\__/|___| |_| |___/ |_| |_|_\\___/_/ \\_\\_| |_| [14928] 2024/11/27 17:47:06.925157 [INF] [14928] 2024/11/27 17:47:06.925157 [INF] https://docs.nats.io/jetstream [14928] 2024/11/27 17:47:06.925157 [INF] [14928] 2024/11/27 17:47:06.925157 [INF] ---------------- JETSTREAM ---------------- [14928] 2024/11/27 17:47:06.925157 [INF] Max Memory: 23.94 GB [14928] 2024/11/27 17:47:06.925157 [INF] Max Storage: 1.00 TB [14928] 2024/11/27 17:47:06.925157 [INF] Store Directory: \"D:\\ProgramData\\FaceFirst\\FaceFirst-MessageBroker-Edge\\data-store\\jetstream\" [14928] 2024/11/27 17:47:06.925157 [INF] Domain: edge-652efb2da425e854a1e8d203 [14928] 2024/11/27 17:47:06.925157 [INF] ------------------------------------------- [14928] 2024/11/27 17:47:06.926284 [INF] Starting restore for stream 'FF-QUEUE-ACC > KV_enrollments' [14928] 2024/11/27 17:47:06.977355 [INF] Restored 69,478 messages for stream 'FF-QUEUE-ACC > KV_enrollments' in 51ms [14928] 2024/11/27 17:47:06.977355 [INF] Starting restore for stream 'FF-QUEUE-ACC > KV_media' [14928] 2024/11/27 17:47:06.978352 [WRN] Filestore [KV_media] loadBlock error: message block data missing [14928] 2024/11/27 17:47:06.981374 [WRN] Filestore [KV_media] Detected skew in subject-based total (64) vs raw total (58), rebuilding [14928] 2024/11/27 17:47:07.008302 [INF] Restored 58 messages for stream 'FF-QUEUE-ACC > KV_media' in 31ms [14928] 2024/11/27 17:47:07.009270 [INF] Starting restore for stream 'FF-QUEUE-ACC > KV_settings' [14928] 2024/11/27 17:47:07.009270 [INF] Restored 42 messages for stream 'FF-QUEUE-ACC > KV_settings' in 0s [14928] 2024/11/27 17:47:07.009270 [INF] Starting restore for stream 'FF-QUEUE-ACC > KV_visitors' [11864] 2024/11/27 17:49:10.535479 [INF] Starting nats-server [11864] 2024/11/27 17:49:10.536034 [INF] Version: 2.10.20 [11864] 2024/11/27 17:49:10.536034 [INF] Git: [7140387] [11864] 2024/11/27 17:49:10.536034 [INF] Cluster: edge-652efb2da425e854a1e8d203 [11864] 2024/11/27 17:49:10.536034 [INF] Name: edge-652efb2da425e854a1e8d203 [11864] 2024/11/27 17:49:10.536034 [INF] Node: cSXjN80e [11864] 2024/11/27 17:49:10.536034 [INF] ID: NCP5BHETIHWEQ6UR7G6GKEVMF23YRHRMDP56QJ5HHQX7ROZ7MB6PBXD3 [11864] 2024/11/27 17:49:10.536034 [INF] Using configuration file: C:\\Program Files\\FaceFirst\\FaceFirst-MessageBroker-Edge\\conf\\edge.conf [11864] 2024/11/27 17:49:10.537076 [INF] Starting http monitor on 127.0.0.1:8223 [11864] 2024/11/27 17:49:10.537076 [INF] Starting JetStream [11864] 2024/11/27 17:49:10.537076 [INF] _ ___ _____ ___ _____ ___ ___ _ __ __ [11864] 2024/11/27 17:49:10.537076 [INF] _ | | __|_ _/ __|_ _| _ \\ __| /_\\ | \\/ | [11864] 2024/11/27 17:49:10.537076 [INF] | || | _| | | \\__ \\ | | | / _| / _ \\| |\\/| | [11864] 2024/11/27 17:49:10.537076 [INF] \\__/|___| |_| |___/ |_| |_|_\\___/_/ \\_\\_| |_| [11864] 2024/11/27 17:49:10.537076 [INF] [11864] 2024/11/27 17:49:10.537076 [INF] https://docs.nats.io/jetstream [11864] 2024/11/27 17:49:10.537076 [INF] [11864] 2024/11/27 17:49:10.537076 [INF] ---------------- JETSTREAM ---------------- [11864] 2024/11/27 17:49:10.537585 [INF] Max Memory: 23.94 GB [11864] 2024/11/27 17:49:10.537585 [INF] Max Storage: 1.00 TB [11864] 2024/11/27 17:49:10.537585 [INF] Store Directory: \"D:\\ProgramData\\FaceFirst\\FaceFirst-MessageBroker-Edge\\data-store\\jetstream\" [11864] 2024/11/27 17:49:10.537612 [INF] Domain: edge-652efb2da425e854a1e8d203 [11864] 2024/11/27 17:49:10.537612 [INF] ------------------------------------------- [11864] 2024/11/27 17:49:10.538210 [INF] Starting restore for stream 'FF-QUEUE-ACC > KV_enrollments' [11864] 2024/11/27 17:49:10.592603 [INF] Restored 69,478 messages for stream 'FF-QUEUE-ACC > KV_enrollments' in 54ms [11864] 2024/11/27 17:49:10.592603 [INF] Starting restore for stream 'FF-QUEUE-ACC > KV_media' [11864] 2024/11/27 17:49:10.593601 [WRN] Filestore [KV_media] loadBlock error: message block data missing [11864] 2024/11/27 17:49:10.596593 [WRN] Filestore [KV_media] Detected skew in subject-based total (64) vs raw total (58), rebuilding [11864] 2024/11/27 17:49:10.623551 [INF] Restored 58 messages for stream 'FF-QUEUE-ACC > KV_media' in 31ms [11864] 2024/11/27 17:49:10.624519 [INF] Starting restore for stream 'FF-QUEUE-ACC > KV_settings' [11864] 2024/11/27 17:49:10.624519 [INF] Restored 42 messages for stream 'FF-QUEUE-ACC > KV_settings' in 0s [11864] 2024/11/27 17:49:10.625523 [INF] Starting restore for stream 'FF-QUEUE-ACC > KV_visitors' [7624] 2024/11/27 17:51:13.625652 [INF] Starting nats-server [7624] 2024/11/27 17:51:13.625652 [INF] Version: 2.10.20 [7624] 2024/11/27 17:51:13.625652 [INF] Git: [7140387] [7624] 2024/11/27 17:51:13.625652 [INF] Cluster: edge-652efb2da425e854a1e8d203 [7624] 2024/11/27 17:51:13.625652 [INF] Name: edge-652efb2da425e854a1e8d203 [7624] 2024/11/27 17:51:13.625652 [INF] Node: cSXjN80e [7624] 2024/11/27 17:51:13.626155 [INF] ID: NAI7ED2WRL3F2QVFHDTH7VKDAMONEN5ONZHVLMFLMDV4GVT3VGRSHPKP [7624] 2024/11/27 17:51:13.626160 [INF] Using configuration file: C:\\Program Files\\FaceFirst\\FaceFirst-MessageBroker-Edge\\conf\\edge.conf [7624] 2024/11/27 17:51:13.626671 [INF] Starting http monitor on 127.0.0.1:8223 [7624] 2024/11/27 17:51:13.626671 [INF] Starting JetStream [7624] 2024/11/27 17:51:13.627179 [INF] _ ___ _____ ___ _____ ___ ___ _ __ __ [7624] 2024/11/27 17:51:13.627179 [INF] _ | | __|_ _/ __|_ _| _ \\ __| /_\\ | \\/ | [7624] 2024/11/27 17:51:13.627179 [INF] | || | _| | | \\__ \\ | | | / _| / _ \\| |\\/| | [7624] 2024/11/27 17:51:13.627179 [INF] \\__/|___| |_| |___/ |_| |_|_\\___/_/ \\_\\_| |_| [7624] 2024/11/27 17:51:13.627179 [INF] [7624] 2024/11/27 17:51:13.627179 [INF] https://docs.nats.io/jetstream [7624] 2024/11/27 17:51:13.627179 [INF] [7624] 2024/11/27 17:51:13.627179 [INF] ---------------- JETSTREAM ---------------- [7624] 2024/11/27 17:51:13.627179 [INF] Max Memory: 23.94 GB [7624] 2024/11/27 17:51:13.627179 [INF] Max Storage: 1.00 TB [7624] 2024/11/27 17:51:13.627179 [INF] Store Directory: \"D:\\ProgramData\\FaceFirst\\FaceFirst-MessageBroker-Edge\\data-store\\jetstream\" [7624] 2024/11/27 17:51:13.627179 [INF] Domain: edge-652efb2da425e854a1e8d203 [7624] 2024/11/27 17:51:13.627179 [INF] ------------------------------------------- [7624] 2024/11/27 17:51:13.628405 [INF] Starting restore for stream 'FF-QUEUE-ACC > KV_enrollments' [7624] 2024/11/27 17:51:13.678519 [INF] Restored 69,478 messages for stream 'FF-QUEUE-ACC > KV_enrollments' in 50ms [7624] 2024/11/27 17:51:13.678519 [INF] Starting restore for stream 'FF-QUEUE-ACC > KV_media' [7624] 2024/11/27 17:51:13.679484 [WRN] Filestore [KV_media] loadBlock error: message block data missing [7624] 2024/11/27 17:51:13.682459 [WRN] Filestore [KV_media] Detected skew in subject-based total (64) vs raw total (58), rebuilding [7624] 2024/11/27 17:51:13.710384 [INF] Restored 58 messages for stream 'FF-QUEUE-ACC > KV_media' in 32ms [7624] 2024/11/27 17:51:13.711381 [INF] Starting restore for stream 'FF-QUEUE-ACC > KV_settings' [7624] 2024/11/27 17:51:13.711381 [INF] Restored 42 messages for stream 'FF-QUEUE-ACC > KV_settings' in 0s [7624] 2024/11/27 17:51:13.711381 [INF] Starting restore for stream 'FF-QUEUE-ACC > KV_visitors' ``` ### Expected behavior The problem is that jetstream at the core does not differentiate between the old and new version of the stream match_events. If the source stream where to receive and keep track of the remote stream's create time and included that time in the consumer create, the remote server can check if the time submitted differs from the create time of the stream and return an error instead. This error could be the signal for the hub jetstream to start over by resetting the sequence number, obtain and update the new timestamp, create the new consumer again. This probably has implications for mirroring as well. There a differing timestamp should probably result in deleting the current content prior to starting at 0 again. ### Server and client version 2.10.20 & 2.10.23 ### Host environment _No response_ ### Steps to reproduce _No response_",
          "url": "https://github.com/nats-io/nats-server/issues/6206",
          "createdAt": "2024-12-03T21:02:39Z",
          "updatedAt": "2026-08-13T14:19:34Z",
          "timestamp": "2026-08-13T14:19:34Z",
          "metrics": {
            "reactions": 0,
            "comments": 2
          },
          "labels": [
            "defect"
          ],
          "author": "facefirstMH",
          "state": "closed",
          "assignees": [
            "neilalexander",
            "jnmoyne"
          ],
          "change": "new"
        }
      },
      {
        "id": "event:ef3f5a4719329d48d58f",
        "signalId": "github:nats-io/nats-server:pull_request:8443",
        "event": "changed",
        "observedAt": "2026-08-13T17:43:20.785491Z",
        "changedFields": [
          "updatedAt"
        ],
        "signal": {
          "id": "github:nats-io/nats-server:pull_request:8443",
          "source": "github",
          "group": "data-infrastructure",
          "project": "nats-io/nats-server",
          "kind": "pull_request",
          "title": "(2.15) [ADDED] Peer evacuation and reconciliation of assignments",
          "text": "This PR wires peer removals and additions into desired state. - `reconcilePeerAssignments` runs on a meta leader change, so proposals that were missed or only partially applied can be retried by the next leader. - Peer additions through `processAddPeer` now also obey the rules of `cc.selectPeerGroup`, which requires waiting for STATSZ info if it isn't yet available for the newly added node. - A new set of \"peer evacuate\" endpoints (`$JS.API.STREAM.PEER.EVACUATE.<stream>` and, for the system account, `$JS.API.SERVER.EVACUATE`) offers the same semantics as peer-remove while not immediately removing the peer, so it can cooperate in catching up the new peer. This also allows evacuating all R1 streams of a server that is to be decommissioned ahead of time, whereas a peer-remove of a server hosting an R1 stream does get a new server assigned but would otherwise lose all data. `reconcilePeerAssignments` scans every stream and consumer assignment on each meta leader change and peer add, benchmarked locally at ~100-290 ms for 100k streams/500k consumers, under `js.mu`. That is purely a no-op scan; actually reconciling is significantly more expensive, since each affected assignment's proposal needs to be retried. A large contributor is `selectPeerGroup`, which loops over all stream assignments per call to determine HA asset counts. This is not addressed in this PR, but noted for clarity, and `selectPeerGroup` can only be made faster once all `meta.ForwardProposal` calls are gone since we can then purely rely on atomics. Follow-up of https://github.com/nats-io/nats-server/pull/8432",
          "url": "https://github.com/nats-io/nats-server/pull/8443",
          "createdAt": "2026-08-07T18:06:36Z",
          "updatedAt": "2026-08-13T17:32:19Z",
          "timestamp": "2026-08-13T17:32:19Z",
          "metrics": {
            "reactions": 0,
            "comments": 3
          },
          "labels": [],
          "author": "MauriceVanVeen",
          "state": "open",
          "assignees": [],
          "change": "updated"
        }
      },
      {
        "id": "event:32cb212cb1eee39bc869",
        "signalId": "github:nats-io/nats-server:pull_request:8452",
        "event": "changed",
        "observedAt": "2026-08-13T17:43:20.785491Z",
        "changedFields": [
          "updatedAt",
          "metrics"
        ],
        "signal": {
          "id": "github:nats-io/nats-server:pull_request:8452",
          "source": "github",
          "group": "data-infrastructure",
          "project": "nats-io/nats-server",
          "kind": "pull_request",
          "title": "(2.15) [FIXED] Evict peer-removed peers for group below quorum",
          "text": "If a stream/consumer Raft group was below quorum, it wasn't possible to safely peer-remove nodes to get into quorum again. This PR fixes that by allowing nodes to be evicted when below quorum, but only those nodes that were explicitly peer-removed. Given a R3 stream without quorum due to 2 nodes being offline, it's now possible to peer-remove those two offline nodes and the stream will become operational again, albeit it at R1. If new peers were assigned to the stream, the stream would automatically scale up to R3, but on the new peers. The desired state gets a new `rg.Desired.Removed` field that contains the set of explicitly peer-removed nodes, which are eligible for eviction if below quorum. Eviction is only allowed if this node thinks the stream/consumer Raft group is leaderless, it is not leaderless on the meta layer, it has desired state that specifies removed peers, and the node itself is still part of the group's peer set. A safety check also prevents eviction if the remaining peers could form a disjoint majority. Follow-up of https://github.com/nats-io/nats-server/pull/8432",
          "url": "https://github.com/nats-io/nats-server/pull/8452",
          "createdAt": "2026-08-11T14:10:44Z",
          "updatedAt": "2026-08-13T17:30:14Z",
          "timestamp": "2026-08-13T17:30:14Z",
          "metrics": {
            "reactions": 0,
            "comments": 3
          },
          "labels": [],
          "author": "MauriceVanVeen",
          "state": "open",
          "assignees": [],
          "change": "updated"
        }
      },
      {
        "id": "event:46406dd424430a683aa9",
        "signalId": "github:nats-io/nats-server:pull_request:8460",
        "event": "changed",
        "observedAt": "2026-08-13T17:43:20.785491Z",
        "changedFields": [
          "updatedAt",
          "metrics"
        ],
        "signal": {
          "id": "github:nats-io/nats-server:pull_request:8460",
          "source": "github",
          "group": "data-infrastructure",
          "project": "nats-io/nats-server",
          "kind": "pull_request",
          "title": "(2.15) [FIXED] Migration added offline peers & removed peers before catchup",
          "text": "This PR fixes two remaining issues, and removes the final two FIXMEs: - Migrations would add peers as fast as possible, without checking whether the peer being added was even online. Adding an offline peer grows the quorum size, which could drop the group out of quorum until enough nodes came online. `selectPeerToAdd` now prefers the candidate we've heard from most recently, and only adds an unheard peer if the live members still reach quorum in the grown group. - Migrations also didn't wait for upper-layer stream catchup to complete before removing old peers. An old peer is now only removed if the group left after the removal retains a store-current majority, based on `Current`/`Lag` and the stream's active `catchupPeers`. Follow-up of https://github.com/nats-io/nats-server/pull/8432",
          "url": "https://github.com/nats-io/nats-server/pull/8460",
          "createdAt": "2026-08-12T16:02:04Z",
          "updatedAt": "2026-08-13T17:27:16Z",
          "timestamp": "2026-08-13T17:27:16Z",
          "metrics": {
            "reactions": 0,
            "comments": 3
          },
          "labels": [],
          "author": "MauriceVanVeen",
          "state": "open",
          "assignees": [],
          "change": "updated"
        }
      }
    ]
  }
}