kubernetes - Jenkins helm chart not updating plugins

Question

I am having a problem with jenkins kubernetes pod that stopped working after the last pod restart (done by kubernetes).

So, I am having errors like this in my log:

2021-02-05 11:00:55.856+0000 [id=27]    INFO    jenkins.InitReactorRunner$1#onAttained: Listed all plugins
2021-02-05 11:00:56.883+0000 [id=30]    SEVERE  jenkins.InitReactorRunner$1#onTaskFailed: Failed Loading plugin Pipeline: Multibranch v2.22 (workflow-multibranch)
java.io.IOException: Failed to load: Pipeline: Multibranch (2.22)
 - Update required: Pipeline: Job (2.36) to be updated to 2.39 or higher
    at hudson.PluginWrapper.resolvePluginDependencies(PluginWrapper.java:952)
    at hudson.PluginManager$2$1$1.run(PluginManager.java:549)
    at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
    at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
    at jenkins.model.Jenkins$5.runTask(Jenkins.java:1131)
    at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
    at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
    at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

I can see that I should upgrade the version of workflow-job plugin to 2.39.

Jenkins is managed with helm. If I download the latest helm chart from S3, I can see it has a folder jenkins in it, where I can see jenkins/values.yaml looking like this:

# Default values for jenkins.
# This is a YAML-formatted file.
# Declare name/value pairs to be passed into your templates.
# name: value

## Overrides for generated resource names
# See templates/_helpers.tpl
# nameOverride:
# fullnameOverride:

Master:
  Name: jenkins-master
  Image: "jenkins/jenkins"
  ImageTag: "lts"
  ImagePullPolicy: "Always"
# ImagePullSecret: jenkins
  Component: "jenkins-master"
  NumExecutors: 0
  # configAutoReload requires UseSecurity is set to true:
  UseSecurity: true
  # SecurityRealm:
  # Optionally configure a different AuthorizationStrategy using Jenkins XML
  # AuthorizationStrategy: |-
  #    <authorizationStrategy class="hudson.security.FullControlOnceLoggedInAuthorizationStrategy">
  #      <denyAnonymousReadAccess>true</denyAnonymousReadAccess>
  #    </authorizationStrategy>
  HostNetworking: false
  # When enabling LDAP or another non-Jenkins identity source, the built-in admin account will no longer exist.
  # Since the AdminUser is used by configAutoReload, in order to use configAutoReload you must change the
  # .Master.AdminUser to a valid username on your LDAP (or other) server.  This user does not need
  # to have administrator rights in Jenkins (the default Overall:Read is sufficient) nor will it be granted any
  # additional rights.  Failure to do this will cause the sidecar container to fail to authenticate via SSH and enter
  # a restart loop.  Likewise if you disable the non-Jenkins identity store and instead use the Jenkins internal one,
  # you should revert Master.AdminUser to your preferred admin user:
  AdminUser: admin
  # AdminPassword: <defaults to random>
  OwnSshKey: false
  # If CasC auto-reload is enabled, an SSH (RSA) keypair is needed.  Can either provide your own, or leave unconfiguredfalse to allow a random key to be auto-generated.
  # If you choose to use your own, you must upload your decrypted RSA private key (not the public key above) to a Kubernetes secret using the following command:
  # kubectl -n <namespace> create secret generic <helm_release_name> --dry-run --from-file=jenkins-admin-private-key=~/.ssh/id_rsa -o yaml |kubectl -n <namespace> apply -f -
  # Replace ~/.ssh/id_rsa in the above command with the path to your private key file and the <helm_release_name> and <namespace> placeholders to suit.
  RollingUpdate: {}
  # Ignored if Persistence is enabled
  # maxSurge: 1
  # maxUnavailable: 25%
  resources:
    requests:
      cpu: "50m"
      memory: "256Mi"
    limits:
      cpu: "2000m"
      memory: "4096Mi"
  # Environment variables that get added to the init container (useful for e.g. http_proxy)
  # InitContainerEnv:
  #   - name: http_proxy
  #     value: "http://192.168.64.1:3128"
  # ContainerEnv:
  #   - name: http_proxy
  #     value: "http://192.168.64.1:3128"
  # Set min/max heap here if needed with:
  # JavaOpts: "-Xms512m -Xmx512m"
  # JenkinsOpts: ""
  # JenkinsUrl: ""
  # If you set this prefix and use ingress controller then you might want to set the ingress path below
  # JenkinsUriPrefix: "/jenkins"
  # Enable pod security context (must be `true` if RunAsUser or FsGroup are set)
  UsePodSecurityContext: true
  # Set RunAsUser to 1000 to let Jenkins run as non-root user 'jenkins' which exists in 'jenkins/jenkins' docker image.
  # When setting RunAsUser to a different value than 0 also set FsGroup to the same value:
  # RunAsUser: <defaults to 0>
  # FsGroup: <will be omitted in deployment if RunAsUser is 0>
  ServicePort: 8080
  # For minikube, set this to NodePort, elsewhere use LoadBalancer
  # Use ClusterIP if your setup includes ingress controller
  ServiceType: LoadBalancer
  # Master Service annotations
  ServiceAnnotations: {}
  # Master Custom Labels
  DeploymentLabels: {}
  #   foo: bar
  #   bar: foo
  # Master Service Labels
  ServiceLabels: {}
  #   service.beta.kubernetes.io/aws-load-balancer-backend-protocol: https
  # Put labels on jeknins-master pod
  PodLabels: {}
  # Used to create Ingress record (should used with ServiceType: ClusterIP)
  # HostName: jenkins.cluster.local
  # NodePort: <to set explicitly, choose port between 30000-32767
  # Enable Kubernetes Liveness and Readiness Probes
  # ~ 2 minutes to allow Jenkins to restart when upgrading plugins. Set ReadinessTimeout to be shorter than LivenessTimeout.
  HealthProbes: true
  HealthProbesLivenessTimeout: 90
  HealthProbesReadinessTimeout: 60
  HealthProbeReadinessPeriodSeconds: 10
  HealthProbeLivenessFailureThreshold: 12
  SlaveListenerPort: 50000
#  SlaveHostPort: 50000
  DisabledAgentProtocols:
    - JNLP-connect
    - JNLP2-connect
  CSRF:
    DefaultCrumbIssuer:
      Enabled: true
      ProxyCompatability: true
  CLI: false
  # Kubernetes service type for the JNLP slave service
  # SlaveListenerServiceType is the Kubernetes Service type for the JNLP slave service,
  # either 'LoadBalancer', 'NodePort', or 'ClusterIP'
  # Note if you set this to 'LoadBalancer', you *must* define annotations to secure it. By default
  # this will be an external load balancer and allowing inbound 0.0.0.0/0, a HUGE
  # security risk:  https://github.com/kubernetes/charts/issues/1341
  SlaveListenerServiceType: ClusterIP
  SlaveListenerServiceAnnotations: {}

  # Example of 'LoadBalancer' type of slave listener with annotations securing it
  # SlaveListenerServiceType: LoadBalancer
  # SlaveListenerServiceAnnotations:
  #   service.beta.kubernetes.io/aws-load-balancer-internal: "True"
  #   service.beta.kubernetes.io/load-balancer-source-ranges: "172.0.0.0/8, 10.0.0.0/8"

  # LoadBalancerSourcesRange is a list of allowed CIDR values, which are combined with ServicePort to
  # set allowed inbound rules on the security group assigned to the master load balancer
  LoadBalancerSourceRanges:
  - 0.0.0.0/0
  # Optionally assign a known public LB IP
  # LoadBalancerIP: 1.2.3.4
  # Optionally configure a JMX port
  # requires additional JavaOpts, ie
  # JavaOpts: >
  #   -Dcom.sun.management.jmxremote.port=4000
  #   -Dcom.sun.management.jmxremote.authenticate=false
  #   -Dcom.sun.management.jmxremote.ssl=false
  # JMXPort: 4000
  # Optionally configure other ports to expose in the Master container
  ExtraPorts:
  # - name: BuildInfoProxy
  #   port: 9000

  # List of plugins to be install during Jenkins master start
  OverwritePlugins: true
  InstallPlugins:
    - kubernetes:1.16.1
    - workflow-job:2.39
    - workflow-aggregator:2.6
    - workflow-basic-steps:2.18
    - credentials-binding:1.23
    - job-dsl:1.76
    - git:4.2.2
    - parameterized-trigger:2.35.2
    - slack:2.34
    - global-slack-notifier:1.5
    - ansicolor:0.6.2
    - simple-theme-plugin:0.5.1
    - aws-bucket-credentials:1.0.0
    - aws-credentials:1.28
    - ssh-agent:1.17
    #      - blueocean:1.21.0
    - basic-branch-build-strategies:1.3.2
    - buildtriggerbadge:2.10
    - rebuild:1.31
    - ghprb:1.42.0
    - antisamy-markup-formatter:1.5
    - github-oauth:0.31
    - role-strategy:2.15

  # Enable to always override the installed plugins with the values of 'Master.InstallPlugins' on upgrade or redeployment.
  # OverwritePlugins: true
  # Enable HTML parsing using OWASP Markup Formatter Plugin (antisamy-markup-formatter), useful with ghprb plugin.
  # The plugin is not installed by default, please update Master.InstallPlugins.
  # EnableRawHtmlMarkupFormatter: true
  # Used to approve a list of groovy functions in pipelines used the script-security plugin. Can be viewed under /scriptApproval
  # ScriptApproval:
  #   - "method groovy.json.JsonSlurperClassic parseText java.lang.String"
  #   - "new groovy.json.JsonSlurperClassic"
  # List of groovy init scripts to be executed during Jenkins master start
  InitScripts:
  #  - |
  #    print 'adding global pipeline libraries, register properties, bootstrap jobs...'
  # Kubernetes secret that contains a 'credentials.xml' for Jenkins
  # CredentialsXmlSecret: jenkins-credentials
  # Kubernetes secret that contains files to be put in the Jenkins 'secrets' directory,
  # useful to manage encryption keys used for credentials.xml for instance (such as
  # master.key and hudson.util.Secret)
  # SecretsFilesSecret: jenkins-secrets
  # Jenkins XML job configs to provision
  # Jobs:
  #   test: |-
  #     <<xml here>>

  # Below is the implementation of Jenkins Configuration as Code.  Add a key under ConfigScripts for each configuration area,
  # where each corresponds to a plugin or section of the UI.  Each key (prior to | character) is just a label, and can be any value.
  # Keys are only used to give the section a meaningful name.  The only restriction is they may only contain RFC 1123  DNS label
  # characters: lowercase letters, numbers, and hyphens.  The keys become the name of a configuration yaml file on the master in
  # /var/jenkins_home/casc_configs (by default) and will be processed by the Configuration as Code Plugin.  The lines after each |
  # become the content of the configuration yaml file.  The first line after this is a JCasC root element, eg jenkins, credentials,
  # etc.  Best reference is https://<jenkins_url>/configuration-as-code/reference.  The example below creates a welcome message:
  JCasC:
    enabled: false
    PluginVersion: 1.5
    SupportPluginVersion: 1.5
    ConfigScripts:
      welcome-message: |
        jenkins:
          systemMessage: Welcome to our CICD server.  This Jenkins is configured and managed 'as code'.

  Sidecars:
    configAutoReload:
      # If enabled: true, Jenkins Configuration as Code will be reloaded on-the-fly without a reboot.  If false or not-specified,
      # jcasc changes will cause a reboot and will only be applied at the subsequent start-up.  Auto-reload uses the Jenkins CLI
      # over SSH to reapply config when chan

Answer 1

Waitting for answers