Skip to content

Support retaining subscription progress after unsubscribe - #18718

Open
Caideyipi wants to merge 3 commits into
apache:masterfrom
Caideyipi:feature/subscription-retain-progress-after-unsubscribe
Open

Caideyipi wants to merge 3 commits into
apache:masterfrom
Caideyipi:feature/subscription-retain-progress-after-unsubscribe

Conversation

@Caideyipi

Copy link
Copy Markdown
Collaborator

Description

Add the incremental topic attribute retain.progress.after.unsubscribe to preserve a consumer group's committed progress and the WAL files required to resume from that progress after the group unsubscribes from a topic.

Configuration and validation

  • The attribute defaults to false, preserving the existing unsubscribe behavior.
  • Attribute names are matched case-insensitively. Values are trimmed and must be true or false, also case-insensitively.
  • The attribute is supported only by incremental topics, together with the existing retention attributes.
  • The attribute can be updated online. Changing it from true to false removes retained state for groups that are no longer subscribed. The existing retention.bytes and retention.ms attributes remain immutable after topic creation.

Progress and WAL lifecycle

  • When the final consumer in a group unsubscribes, ConfigNode retains that group's committed topic progress when the attribute is enabled.
  • DataNode atomically replaces the active prefetching queue with detached WAL retention. Detached retention does not receive live subscription requests and only prevents deletion of WAL files needed by the committed progress.
  • Resubscribing atomically replaces detached retention with an active queue and resumes from the retained progress.
  • During DataNode recovery, IoTConsensus installs a conservative retention guard before the recovered peer starts. Subscription metadata recovery then replaces that guard with the active queue, the final detached retention policy, or removes it when the retained state is no longer valid.
  • Dropping a topic, consumer group, or region always removes its progress and WAL retention state. Setup rollback restores the startup retention guard.

Tests

  • TopicConfigTest: 9 tests
  • CommitProgressKeeperTest: 14 tests
  • SubscriptionQueueRegistryTest: 4 tests
  • Targeted DataNode subscription tests: 32 tests
  • Targeted ConfigNode subscription tests: 43 tests
  • mvn test-compile -DskipTests: 53-module reactor, BUILD SUCCESS
  • mvn test-compile -DskipTests -P with-zh-locale: 53-module reactor, BUILD SUCCESS

This PR has:

  • been self-reviewed.
    • concurrent read
    • concurrent write
    • concurrent read and write
  • added comments explaining the "why" and the intent of the code wherever it would not be obvious to an unfamiliar reader.
  • added or modified unit tests to cover the new code paths.

Key changed/added classes (or packages if there are too many classes) in this PR
  • TopicConfig and TopicConstant: expose the new topic attribute and its default.
  • SubscriptionInfo and CommitProgressKeeper: validate the attribute and retain or clean ConfigNode progress.
  • ConsensusSubscriptionSetupHandler and ConsensusSubscriptionWalRetention: switch between active queues and detached WAL retention, including recovery and rollback.
  • IoTConsensusServerImpl and SubscriptionQueueRegistry: manage WAL retention policies and their atomic lifecycle.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant