[PCF-Dev v1.3.1]: Deployment of Spring Cloud Services (SCS) fails again - Service broker error: Could not find an available leader

[jeksopra]

Hi @aemengo ,
sorry to bother you anew with a formerly reported issue, but unfortunatly I get the following error again, when trying to deploy the Spring Cloud Services into PCF Dev v1.3.1:
Service broker error: Could not find an available leader from [https://10.144.0.13:8443]

The problem occurs when the deployment process tries to bind the spring-cloud-broker to the just created mysql service instance named spring-cloud-broker-db (Command: cf bind-service spring-cloud-broker spring-cloud-broker-db).

I also get the error when I try to bind a formerly created MySQL service instance to the Spring Music app:

PS C:\Users\jekirch\Documents\NetBeansProjects\spring-music> cf s
Getting services in org cfdev-org / space cfdev-space as admin...

name              service   plan       bound apps   last operation     broker                   upgrade available
p-mysql-service   p.mysql   db-small                create succeeded   dedicated-mysql-broker

TIP: Please upgrade to CC API v2.138.0 or higher for individual service upgrades
PS C:\Users\jekirch\Documents\NetBeansProjects\spring-music> cf push
Pushing from manifest to org cfdev-org / space cfdev-space as admin...
Using manifest file C:\Users\jekirch\Documents\NetBeansProjects\spring-music\manifest.yml
Getting app info...
Updating app with these attributes...
  name:                spring-music
  path:                C:\Users\jekirch\Documents\NetBeansProjects\spring-music\build\libs\spring-music-1.0.jar
  buildpacks:
    java_buildpack_offline
  disk quota:          1G
  health check type:   port
  instances:           1
  memory:              1G
  stack:               cflinuxfs3
  services:
+   p-mysql-service
  env:
    JBP_CONFIG_SPRING_AUTO_RECONFIGURATION
  routes:
    spring-music.dev.cfdev.sh

Updating app spring-music...
Mapping routes...
Binding services...
Service broker error: Could not find an available leader from [https://10.144.0.14:8443]
FAILED

Some weeks ago you have fixed this issue in PCF Dev Version 1.2.0 and released a new version (v1.3.0/v1.3.1), which worked fine in the beginning... Since yesterday I got the same error message again.

Last time you identified an issue concerning the internal SSL certificates used in the Spring Cloud Services execution path that have expired:

# /var/vcap/sys/log/broker/broker.log
[on-demand-service-broker] [f347182b-3834-4f1d-9648-d92ebbe8fb51] 2020/06/30 22:47:58.061686 service adapter will delete binding with ID b3756eda-c00f-42c2-b9fa-dbb78fd06ddd for instance 0b252b61-0831-4cf3-a920-bd68dc012a20
[on-demand-service-broker] [f347182b-3834-4f1d-9648-d92ebbe8fb51] 2020/06/30 22:47:58.088705 external service adapter exited with 1 at /var/vcap/packages/odb-service-adapter/bin/service-adapter:
stdout:
    'Could not find an available leader from [https://10.144.0.13:8443]',
stderr: '
    [odb-sdk] handling delete-binding
    [dedicated-mysql-adapter] 2020/06/30 22:47:58 Using agent deleter
    [dedicated-mysql-adapter] 2020/06/30 22:47:58 Couldn't get read only information about URL https://10.144.0.13:8443, error: &errors.errorString{s:"GET https://10.144.0.13:8443/status failed: Get https://10.144.0.13:8443/status: x509: certificate has expired or is not yet valid"}
    [odb-sdk] Could not find an available leader from [https://10.144.0.13:8443]

Perhaps there is a similiar issue again?
I would be very thankful if you could remedy this issue.

Please find the whole deploy-spring cloud services.log as an attached file to this issue report.
deploy-spring cloud services.log

Thanks in advance and best regards
JK

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this. Unfortunately, the Pivotal Tracker project is private so you may be unable to view the contents of the story.

The labels on this github issue will be updated when the story is started.

[jeksopra]

After some investigations into the bosh tool in conjunction with PCF Dev I was finally able to extract the broker logs of the pivotal-mysql-service-broker by issuing the follwing command:
bosh -e lite logs -d pivotal-mysql-ae8328f772d941ee036b

Here is the relevant log file:
broker.log

And indeed it seems to be a certificate issue again. Here is some excerpt:

[dedicated-mysql-adapter] 2020/07/17 14:12:16 Using agent creator
[dedicated-mysql-adapter] 2020/07/17 14:12:16 Couldn't get read only information about URL https://10.144.0.8:8443, error: &errors.errorString{s:"GET https://10.144.0.8:8443/status failed: Get https://10.144.0.8:8443/status: **x509: certificate has expired or is not yet valid"}**
[odb-sdk] Could not find an available leader from [https://10.144.0.8:8443]
'
[on-demand-service-broker] [a31c097b-9a93-4d4b-87c6-3dcea5b604e9] 2020/07/17 14:12:16.600862 broker.resolve_secrets_at_bind was: false 
[on-demand-service-broker] [a31c097b-9a93-4d4b-87c6-3dcea5b604e9] 2020/07/17 14:12:16.600868 creating binding: Could not find an available leader from [https://10.144.0.8:8443]
[on-demand-service-broker] [a31c097b-9a93-4d4b-87c6-3dcea5b604e9] 2020/07/17 14:12:16.600872 Could not find an available leader from [https://10.144.0.8:8443]
[on-demand-service-broker] 2020/07/17 14:12:16.600932 Request PUT /v2/service_instances/5f18bb3b-852d-4456-9991-1ccda830abef/service_bindings/5142ba87-b134-45fa-a2ad-6c0b1f9036ab Completed 500 in 1.334313723s | Start Time: 2020/07/17 14:12:15.266605
[on-demand-service-broker] [01fe20b3-bd17-4f5b-856d-714bc27d3219] 2020/07/17 14:12:17.097612 getting manifest from bosh for deployment service-instance_5f18bb3b-852d-4456-9991-1ccda830abef
[on-demand-service-broker] [01fe20b3-bd17-4f5b-856d-714bc27d3219] 2020/07/17 14:12:17.146403 retrieving VMs for deployment service-instance_5f18bb3b-852d-4456-9991-1ccda830abef from bosh
[on-demand-service-broker] [01fe20b3-bd17-4f5b-856d-714bc27d3219] 2020/07/17 14:12:18.338040 service adapter will delete binding with ID 5142ba87-b134-45fa-a2ad-6c0b1f9036ab for instance 5f18bb3b-852d-4456-9991-1ccda830abef
[on-demand-service-broker] [01fe20b3-bd17-4f5b-856d-714bc27d3219] 2020/07/17 14:12:18.358657 external service adapter exited with 1 at /var/vcap/packages/odb-service-adapter/bin/service-adapter: stdout: 'Could not find an available leader from [https://10.144.0.8:8443]', stderr: '[odb-sdk] handling delete-binding
[dedicated-mysql-adapter] 2020/07/17 14:12:18 Using agent deleter
[dedicated-mysql-adapter] 2020/07/17 14:12:18 Couldn't get read only information about URL https://10.144.0.8:8443, error: &errors.errorString{s:"GET https://10.144.0.8:8443/status failed: Get https://10.144.0.8:8443/status: **x509: certificate has expired or is not yet valid"}**
[odb-sdk] Could not find an available leader from [https://10.144.0.8:8443]

Thanks a lot!