feat: detect dependent resource API version changes (#3536)

Add an opt-in, experimental detectApiVersionChange option on
@KubernetesDependent that records the API version the operator applies
in the javaoperatorsdk.io/last-applied-api-version annotation. The
regular matcher then detects a mismatch when that marker differs from
(or is missing relative to) the currently applied API version, causing
a one-time update without triggering repeated reconciliations once the
resource is up-to-date. Disabled by default, so existing behavior and
matching are unaffected unless explicitly enabled.

Guard against a null or immutable annotations map (e.g. Map.of(...))
on the desired resource when detectApiVersionChange is enabled, since
writing the last-applied-api-version marker (and the pre-existing
previous-annotation bookkeeping that runs alongside it) requires a
mutable map. Also rename a misleadingly-named test helper and replace
a no-op assertion on a primitive boolean with a concrete expectation.

Addresses Copilot review feedback on PR #3536.

Add DetectApiVersionChangeIT covering the end-to-end scenario: a
ConfigMap dependent resource configured with detectApiVersionChange
is marked with the current API version on creation (without
triggering an update), and a stale marker annotation left on the
actual resource (simulating an older operator/CRD version) is
detected and corrected with exactly one update, after which no
further reconciliation loop occurs.

Signed-off-by: hej090224 <fc49854985@gmail.com>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

Author: hej090224

docs/content/en/docs/documentation/dependent-resource-and-workflows/dependent-resources.md

@@ -289,6 +289,41 @@ If you encounter this issue on an older Kubernetes version, consider changing yo
 that resource, or even upgrading your Kubernetes version. If you encounter it on a newer Kubernetes version, please log
 an issue with the JOSDK and with upstream Kubernetes.
 
+### Detecting dependent resource API version changes (experimental)
+
+When a dependent resource's CRD gains a new API version and the operator is upgraded to target it,
+comparing `actualResource.getApiVersion()` with the desired resource's API version is not a
+reliable way to detect resources that still need to be updated: the Kubernetes API server serves a
+resource using the requested, served API version regardless of which version it is actually stored
+as, so this comparison would always trivially match.
+
+`KubernetesDependentResource` therefore ignores `apiVersion` when matching. To still force a
+one-time update of dependent resources after such an upgrade, without triggering an update on every
+reconciliation, `KubernetesDependent` provides the opt-in, experimental
+`detectApiVersionChange` flag:
+
+```java
+@KubernetesDependent(detectApiVersionChange = true)
+public class MyDependentResource extends CRUDKubernetesDependentResource<MyResource, MyPrimary> {
+  // ...
+}
+```
+
+When enabled, JOSDK records the API version it applies in the `javaoperatorsdk.io/last-applied-api-version`
+annotation. On subsequent reconciliations, the resource is considered mismatched (and thus updated)
+if that recorded marker differs from the API version the operator currently uses - this also
+covers resources that predate this feature and therefore have no marker at all. Once the resource
+has been updated, the marker matches the current API version again, so no further update is
+requested until the API version changes again.
+
+This is disabled by default: existing behavior, including for resources created before this
+feature existed, is unaffected unless you opt in. It does not read or infer the actual storage
+version of the resource from the Kubernetes API, since that information is not reliably exposed;
+it only tracks what the operator itself last applied. It is also not a replacement for
+Kubernetes' [StorageVersionMigration](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/storage-version-migration/),
+which addresses migrating the stored representation of resources, a concern orthogonal to this
+feature.
+
 ## Telling JOSDK how to find which secondary resources are associated with a given primary resource
 
 [`KubernetesDependentResource`](https://github.com/java-operator-sdk/java-operator-sdk/blob/main/operator-framework-core/src/main/java/io/javaoperatorsdk/operator/processing/dependent/kubernetes/KubernetesDependentResource.java)

operator-framework-core/src/main/java/io/javaoperatorsdk/operator/processing/dependent/kubernetes/KubernetesDependent.java

@@ -21,6 +21,9 @@
 import java.lang.annotation.Target;
 
 import io.javaoperatorsdk.operator.api.config.informer.Informer;
+import io.javaoperatorsdk.operator.api.reconciler.Experimental;
+
+import static io.javaoperatorsdk.operator.api.reconciler.Experimental.API_MIGHT_CHANGE;
 
 @Retention(RetentionPolicy.RUNTIME)
 @Target({ElementType.TYPE})
@@ -62,4 +65,32 @@ boolean createResourceOnlyIfNotExistingWithSSA() default
    */
   Class<? extends SSABasedGenericKubernetesResourceMatcher> matcher() default
       SSABasedGenericKubernetesResourceMatcher.class;
+
+  /**
+   * Whether JOSDK should detect that the API version of this dependent resource's desired state has
+   * changed since it was last applied by the operator (for example after the operator was upgraded
+   * to target a new CRD version) and, in that case, request a one-time update of the actual
+   * resource.
+   *
+   * <p>When enabled, JOSDK records the API version it applies in the {@value
+   * KubernetesDependentResource#LAST_APPLIED_API_VERSION_ANNOTATION_KEY} annotation. On subsequent
+   * reconciliations, the resource is considered mismatched (and thus updated) if that recorded
+   * marker differs from the API version the operator currently uses, including when the marker is
+   * missing entirely (for example on resources created before this feature was enabled). Once the
+   * resource has been updated, the marker matches the current API version again, so no further
+   * update is requested until the API version changes again.
+   *
+   * <p>This is opt-in and disabled by default: when disabled, no marker annotation is ever added or
+   * read, and matching behavior is unchanged. It does not read or infer the actual storage version
+   * of the resource in Kubernetes, since that information is not reliably exposed by the API
+   * server; it only tracks what the operator itself last applied. It is not a replacement for
+   * Kubernetes' <a
+   * href="https://kubernetes.io/docs/tasks/manage-kubernetes-objects/storage-version-migration/">StorageVersionMigration</a>.
+   *
+   * @return {@code true} if API version change detection is enabled, {@code false} otherwise
+   * @since 5.6
+   */
+  @Experimental(API_MIGHT_CHANGE)
+  boolean detectApiVersionChange() default
+      KubernetesDependentResourceConfig.DEFAULT_DETECT_API_VERSION_CHANGE;
 }

operator-framework-core/src/main/java/io/javaoperatorsdk/operator/processing/dependent/kubernetes/KubernetesDependentConverter.java

@@ -35,6 +35,8 @@ public KubernetesDependentResourceConfig<R> configFrom(
       ControllerConfiguration<?> controllerConfig) {
     var createResourceOnlyIfNotExistingWithSSA =
         DEFAULT_CREATE_RESOURCE_ONLY_IF_NOT_EXISTING_WITH_SSA;
+    var detectApiVersionChange =
+        KubernetesDependentResourceConfig.DEFAULT_DETECT_API_VERSION_CHANGE;
 
     Boolean useSSA = null;
     SSABasedGenericKubernetesResourceMatcher<R> matcher =
@@ -43,6 +45,7 @@ public KubernetesDependentResourceConfig<R> configFrom(
       createResourceOnlyIfNotExistingWithSSA =
           configAnnotation.createResourceOnlyIfNotExistingWithSSA();
       useSSA = configAnnotation.useSSA().asBoolean();
+      detectApiVersionChange = configAnnotation.detectApiVersionChange();
 
       // check if we have a specific matcher
       Class<? extends KubernetesDependentResource<?, ?>> dependentResourceClass =
@@ -62,7 +65,11 @@ public KubernetesDependentResourceConfig<R> configFrom(
             controllerConfig);
 
     return new KubernetesDependentResourceConfig<>(
-        useSSA, createResourceOnlyIfNotExistingWithSSA, informerConfiguration, matcher);
+        useSSA,
+        createResourceOnlyIfNotExistingWithSSA,
+        informerConfiguration,
+        matcher,
+        detectApiVersionChange);
   }
 
   @SuppressWarnings({"unchecked"})

operator-framework-core/src/main/java/io/javaoperatorsdk/operator/processing/dependent/kubernetes/KubernetesDependentResource.java

@@ -15,6 +15,7 @@
  */
 package io.javaoperatorsdk.operator.processing.dependent.kubernetes;
 
+import java.util.LinkedHashMap;
 import java.util.Map;
 import java.util.Objects;
 import java.util.Optional;
@@ -52,6 +53,15 @@ public abstract class KubernetesDependentResource<R extends HasMetadata, P exten
 
   private static final Logger log = LoggerFactory.getLogger(KubernetesDependentResource.class);
 
+  /**
+   * Annotation used to record the API version the operator applied to this resource, when {@link
+   * KubernetesDependentResourceConfig#detectApiVersionChange()} is enabled.
+   *
+   * @see KubernetesDependent#detectApiVersionChange()
+   */
+  public static final String LAST_APPLIED_API_VERSION_ANNOTATION_KEY =
+      "javaoperatorsdk.io/last-applied-api-version";
+
   private final boolean garbageCollected = this instanceof GarbageCollected;
   private KubernetesDependentResourceConfig<R> kubernetesDependentResourceConfig;
   private volatile Boolean useSSA;
@@ -160,6 +170,12 @@ public Result<R> match(R actualResource, R desired, P primary, Context<P> contex
 
   protected void addMetadata(
       boolean forMatch, R actualResource, final R target, P primary, Context<P> context) {
+    if (kubernetesDependentResourceConfig != null
+        && kubernetesDependentResourceConfig.detectApiVersionChange()) {
+      // desired resources might expose a null or immutable annotations map (e.g. Map.of(...));
+      // make sure it's a mutable one before this method or its callees write to it
+      ensureMutableAnnotations(target);
+    }
     if (forMatch) { // keep the current previous annotation
       String actual =
           actualResource
@@ -173,9 +189,36 @@ protected void addMetadata(
         annotations.remove(InformerEventSource.PREVIOUS_ANNOTATION_KEY);
       }
     }
+    addLastAppliedApiVersion(target);
     addReferenceHandlingMetadata(target, primary);
   }
 
+  private static void ensureMutableAnnotations(HasMetadata target) {
+    var metadata = target.getMetadata();
+    metadata.setAnnotations(
+        new LinkedHashMap<>(Optional.ofNullable(metadata.getAnnotations()).orElseGet(Map::of)));
+  }
+
+  /**
+   * When {@link KubernetesDependentResourceConfig#detectApiVersionChange()} is enabled, marks the
+   * target resource with the API version the operator is currently applying. Comparing this marker
+   * with the one recorded on the actual resource lets the regular matching logic detect a mismatch,
+   * without ever inspecting the actual, potentially unreliable, stored API version.
+   */
+  private void addLastAppliedApiVersion(R target) {
+    if (kubernetesDependentResourceConfig == null
+        || !kubernetesDependentResourceConfig.detectApiVersionChange()) {
+      return;
+    }
+    var apiVersion = target.getApiVersion();
+    if (apiVersion != null) {
+      target
+          .getMetadata()
+          .getAnnotations()
+          .put(LAST_APPLIED_API_VERSION_ANNOTATION_KEY, apiVersion);
+    }
+  }
+
   protected boolean useSSA(Context<P> context) {
     if (useSSA == null) {
       useSSA =

operator-framework-core/src/main/java/io/javaoperatorsdk/operator/processing/dependent/kubernetes/KubernetesDependentResourceConfig.java

@@ -21,11 +21,13 @@
 public class KubernetesDependentResourceConfig<R extends HasMetadata> {
 
   public static final boolean DEFAULT_CREATE_RESOURCE_ONLY_IF_NOT_EXISTING_WITH_SSA = true;
+  public static final boolean DEFAULT_DETECT_API_VERSION_CHANGE = false;
 
   private final Boolean useSSA;
   private final boolean createResourceOnlyIfNotExistingWithSSA;
   private final InformerConfiguration<R> informerConfig;
   private final SSABasedGenericKubernetesResourceMatcher<R> matcher;
+  private final boolean detectApiVersionChange;
 
   public KubernetesDependentResourceConfig(
       Boolean useSSA,
@@ -39,11 +41,26 @@ public KubernetesDependentResourceConfig(
       boolean createResourceOnlyIfNotExistingWithSSA,
       InformerConfiguration<R> informerConfig,
       SSABasedGenericKubernetesResourceMatcher<R> matcher) {
+    this(
+        useSSA,
+        createResourceOnlyIfNotExistingWithSSA,
+        informerConfig,
+        matcher,
+        DEFAULT_DETECT_API_VERSION_CHANGE);
+  }
+
+  public KubernetesDependentResourceConfig(
+      Boolean useSSA,
+      boolean createResourceOnlyIfNotExistingWithSSA,
+      InformerConfiguration<R> informerConfig,
+      SSABasedGenericKubernetesResourceMatcher<R> matcher,
+      boolean detectApiVersionChange) {
     this.useSSA = useSSA;
     this.createResourceOnlyIfNotExistingWithSSA = createResourceOnlyIfNotExistingWithSSA;
     this.informerConfig = informerConfig;
     this.matcher =
         matcher != null ? matcher : SSABasedGenericKubernetesResourceMatcher.getInstance();
+    this.detectApiVersionChange = detectApiVersionChange;
   }
 
   public boolean createResourceOnlyIfNotExistingWithSSA() {
@@ -61,4 +78,16 @@ public InformerConfiguration<R> informerConfig() {
   public SSABasedGenericKubernetesResourceMatcher<R> matcher() {
     return matcher;
   }
+
+  /**
+   * Whether JOSDK should detect when the API version of this dependent resource's desired state has
+   * changed since it was last applied by the operator and, in that case, request a one-time update
+   * of the actual resource.
+   *
+   * @return {@code true} if API version change detection is enabled, {@code false} otherwise
+   * @since 5.6
+   */
+  public boolean detectApiVersionChange() {
+    return detectApiVersionChange;
+  }
 }

operator-framework-core/src/main/java/io/javaoperatorsdk/operator/processing/dependent/kubernetes/KubernetesDependentResourceConfigBuilder.java

@@ -24,6 +24,8 @@ public final class KubernetesDependentResourceConfigBuilder<R extends HasMetadat
   private Boolean useSSA = null;
   private InformerConfiguration<R> informerConfiguration;
   private SSABasedGenericKubernetesResourceMatcher<R> matcher;
+  private boolean detectApiVersionChange =
+      KubernetesDependentResourceConfig.DEFAULT_DETECT_API_VERSION_CHANGE;
 
   public KubernetesDependentResourceConfigBuilder() {}
 
@@ -51,8 +53,18 @@ public KubernetesDependentResourceConfigBuilder<R> withSSAMatcher(
     return this;
   }
 
+  public KubernetesDependentResourceConfigBuilder<R> withDetectApiVersionChange(
+      boolean detectApiVersionChange) {
+    this.detectApiVersionChange = detectApiVersionChange;
+    return this;
+  }
+
   public KubernetesDependentResourceConfig<R> build() {
     return new KubernetesDependentResourceConfig<>(
-        useSSA, createResourceOnlyIfNotExistingWithSSA, informerConfiguration, matcher);
+        useSSA,
+        createResourceOnlyIfNotExistingWithSSA,
+        informerConfiguration,
+        matcher,
+        detectApiVersionChange);
   }
 }

operator-framework-core/src/test/java/io/javaoperatorsdk/operator/processing/dependent/kubernetes/KubernetesDependentConverterTest.java

@@ -0,0 +1,113 @@
+/*
+ * Copyright Java Operator SDK Authors
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *         http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.javaoperatorsdk.operator.processing.dependent.kubernetes;
+
+import java.util.Set;
+
+import org.junit.jupiter.api.Test;
+
+import io.fabric8.kubernetes.api.model.ConfigMap;
+import io.fabric8.kubernetes.api.model.GenericKubernetesResource;
+import io.javaoperatorsdk.operator.api.config.ConfigurationService;
+import io.javaoperatorsdk.operator.api.config.ControllerConfiguration;
+import io.javaoperatorsdk.operator.api.config.dependent.DependentResourceSpec;
+import io.javaoperatorsdk.operator.api.reconciler.dependent.DependentResourceFactory;
+import io.javaoperatorsdk.operator.api.reconciler.dependent.GarbageCollected;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Focused unit test for the {@code detectApiVersionChange} wiring performed by {@link
+ * KubernetesDependentConverter}, independent of the shared, process-wide {@link
+ * io.javaoperatorsdk.operator.api.config.dependent.DependentResourceConfigurationResolver} state
+ * that other tests in this module mutate.
+ */
+class KubernetesDependentConverterTest {
+
+  private final KubernetesDependentConverter<GenericKubernetesResource, ConfigMap> converter =
+      new KubernetesDependentConverter<>();
+
+  @Test
+  void detectApiVersionChangeDefaultsToFalseWhenAnnotationAbsent() {
+    var config =
+        converter.configFrom(null, spec(PlainWidgetDependentResource.class), controllerConfig());
+
+    assertThat(config.detectApiVersionChange()).isFalse();
+  }
+
+  @Test
+  void detectApiVersionChangeDefaultsToFalseWhenNotSetOnAnnotation() {
+    var annotation = PlainWidgetDependentResource.class.getAnnotation(KubernetesDependent.class);
+    var config =
+        converter.configFrom(
+            annotation, spec(PlainWidgetDependentResource.class), controllerConfig());
+
+    assertThat(config.detectApiVersionChange()).isFalse();
+  }
+
+  @Test
+  void detectApiVersionChangeCanBeEnabledViaAnnotation() {
+    var annotation =
+        ApiVersionAwareWidgetDependentResource.class.getAnnotation(KubernetesDependent.class);
+    var config =
+        converter.configFrom(
+            annotation, spec(ApiVersionAwareWidgetDependentResource.class), controllerConfig());
+
+    assertThat(config.detectApiVersionChange()).isTrue();
+  }
+
+  @SuppressWarnings({"unchecked", "rawtypes"})
+  private static DependentResourceSpec<
+          GenericKubernetesResource,
+          ConfigMap,
+          KubernetesDependentResourceConfig<GenericKubernetesResource>>
+      spec(
+          Class<? extends KubernetesDependentResource<GenericKubernetesResource, ConfigMap>>
+              dependentResourceClass) {
+    return new DependentResourceSpec(
+        dependentResourceClass, "test", Set.of(), null, null, null, null, null);
+  }
+
+  private static ControllerConfiguration<ConfigMap> controllerConfig() {
+    ControllerConfiguration<ConfigMap> controllerConfig = mock();
+    when(controllerConfig.getName()).thenReturn("test-reconciler");
+    ConfigurationService configurationService = mock();
+    when(configurationService.dependentResourceFactory())
+        .thenReturn(DependentResourceFactory.DEFAULT);
+    when(controllerConfig.getConfigurationService()).thenReturn(configurationService);
+    return controllerConfig;
+  }
+
+  @KubernetesDependent
+  static class PlainWidgetDependentResource
+      extends KubernetesDependentResource<GenericKubernetesResource, ConfigMap>
+      implements GarbageCollected<ConfigMap> {
+    public PlainWidgetDependentResource() {
+      super(GenericKubernetesResource.class, null);
+    }
+  }
+
+  @KubernetesDependent(detectApiVersionChange = true)
+  static class ApiVersionAwareWidgetDependentResource
+      extends KubernetesDependentResource<GenericKubernetesResource, ConfigMap>
+      implements GarbageCollected<ConfigMap> {
+    public ApiVersionAwareWidgetDependentResource() {
+      super(GenericKubernetesResource.class, null);
+    }
+  }
+}