Starting from Chef server version 12, SSL verification is enabled by default for all the communication that are initiated from any of the chef utilities (knife, chef-client).
For this, Chef server will generate a self-signed SSL certificate during the installation process.
From your Chef workstation when you execute knife command, you should download the SSL certificate before you can get started. If you don’t do this, knife command will not connect to the Chef server.
Why Chef SSL Certificate?
Starting from Chef server version 12, when you perform a simple command like knife client list, it will display the SSL validation failure error message, as the workstation don’t have the server’s SSL certificate for verification.
# knife client list ERROR: SSL Validation failure connecting to host: datadb - SSL_connect returned=1 errno=0 state=error: certificate verify failed ERROR: Could not establish a secure connection to the server. Use `knife ssl check` to troubleshoot your SSL configuration. If your Chef Server uses a self-signed certificate, you can use `knife ssl fetch` to make knife trust the server's certificates. Original Exception: OpenSSL::SSL::SSLError: SSL Error connecting to https://datadb/organizations/thegeekstuff/clients - SSL_connect returned=1 errno=0 state=error: certificate verify failed
Even the chef-client that runs on the individual nodes will need the SSL certificate.
To avoid this SSL error, we should download the self-signed SSL certificate that is generated by the Chef server and store it in the trusted store on the chef client machines including the workstations and individual nodes.
On the Chef workstation when the SSL certificate is installed properly, it will not display the SSL error message when executing knife command.
# knife client list devdb proddb devapp prodapp
On a high-level, you have to do the following. All of these are explained in this tutorial along with possible error messages that you might get and how to resolve those:
- On your chef workstation, download the chef server’s SSL certificate
- On all your individual nodes, make sure a valid chef server’s certificate is installed as part of the bootstrap.
- Also, verify the check-sum of the downloaded chef server’s SSL certificate.
Note: If you are new to knife command, this might help: 12 Chef Knife Cookbook Command Examples
Knife SSL Check – Error 1 (Invalid chef-repo Directory)
You can use the knife ssl check command to verify whether the Chef SSL certificate is installed properly on your local machine.
You should execute ssl check command inside the chef repository directory. If not, you’ll get the “No knife configuration file found” error message as shown below.
At this stage, this has nothing to do with unable to verify SSL certificate. The following error message is because you are inside a wrong directory when executing the knife ssl check command.
# knife ssl check WARNING: No knife configuration file found Connecting to host localhost:443 ERROR: The SSL cert is signed by a trusted authority but is not valid for the given hostname ERROR: You are attempting to connect to: 'localhost' ERROR: The server's certificate belongs to 'datadb' TO FIX THIS ERROR: The solution for this issue depends on your networking configuration. If you are able to connect to this server using the hostname datadb instead of localhost, then you can resolve this issue by updating chef_server_url in your configuration file. If you are not able to connect to the server using the hostname datadb you will have to update the certificate on the server to use the correct hostname.
So, to fix this particular error message, simply cd to chef-repo and execute the knife ssl check command as shown below.
cd ~/chef-repo knife ssl check
Knife SSL Check – Error 2 (No SSL Certificate Found)
In the following knife ssl check command example, the output indicates that on this Chef workstation, the Chef server’s SSL certificate is not installed.
# cd ~/chef-repo # knife ssl check Connecting to host datadb:443 ERROR: The SSL certificate of datadb could not be verified Certificate issuer data: /C=US/O=YouCorp/OU=Operations/CN=datadb Configuration Info: OpenSSL Configuration: * Version: OpenSSL 1.0.1s 1 Mar 2016 * Certificate file: /opt/chefdk/embedded/ssl/cert.pem * Certificate directory: /opt/chefdk/embedded/ssl/certs Chef SSL Configuration: * ssl_ca_path: nil * ssl_ca_file: nil * trusted_certs_dir: "/root/chef-repo/.chef/trusted_certs" TO FIX THIS ERROR: If the server you are connecting to uses a self-signed certificate, you must configure chef to trust that server's certificate. By default, the certificate is stored in the following location on the host where your chef-server runs: /var/opt/opscode/nginx/ca/SERVER_HOSTNAME.crt Copy that file to your trusted_certs_dir (currently: /root/chef-repo/.chef/trusted_certs) using SSH/SCP or some other secure method, then re-run this command to confirm that the server's certificate is now trusted.
When you execute the ssl check command, it will look for the certificate under .chef/trusted_certs directory. If it finds it there, it will verify to make sure it is a valid X.590 SSL certificate. If it can’t find a valid certificate, it will throw the above error message.
Knife SSL Fetch – Error 1 (Network Error – Connection Refused)
Knife ssl fetch command will get the certificate from the Chef server and install it on the individual workstation or node where you are executing the fetch command.
During knife ssl fetch, you might get the following Network error connection refused message as shown below.
# knife ssl fetch WARNING: Certificates from datadb will be fetched and placed in your trusted_cert directory (/root/chef-repo/.chef/trusted_certs). Knife has no means to verify these are the correct certificates. You should verify the authenticity of these certificates after downloading. ERROR: Network Error: Connection refused - connect(2) for "datadb" port 443 Check your knife configuration and network settings
The same network error connection refused error message for “knife client list” also. So, this error has to do with some unable to resolve/reach using proper name “datadb”.
When you get the above connection refused error message, the same thing will happen for other knife commands also as shown below.
# knife client list ERROR: Connection refused connecting to https://datadb/organizations/thegeekstuff/clients, retry 1/5 ERROR: Connection refused connecting to https://datadb/organizations/thegeekstuff/clients, retry 2/5
This means that your Chef workstation or individual node is unable to initiate network communication to the Chef server.
To fix this error, you have to make sure your server is using a fully qualified domain name. In the following example, we don’t see the FQDN setup properly.
# hostname -f datadb
Also, to fix this error message, in my example, my /etc/hosts file looked like the following. Basically you have to make sure both your hostname and FQDN properly setup.
In the following example, the ip-address in the last line the ip-address of the datadb server.
# cat /etc/hosts 127.0.0.1 datadb.thegeekstuff.com datadb 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.100.2 datadb.thegeekstuff.com datadb
Now, when I execute the hostname -f command, I’ll get the FQDN as shown below.
# hostname -f datadb.thegeekstuff.com
After the above changes to the /etc/hosts file, now you shouldn’t get the Network Error connection refused message anymore from the knife command.
Knife Fetch SSL certificate and Store it
Finally, use the knife ssl fetch command as shown below to retrieve the SSL certificate from the Chef server and store it to the .chef/trusted_certs directory on the workstation or the individual nodes by either the knife or chef-client command.
The location of where the certificate will be stored when you execute the knife ssl fetch command will be determined by the trusted_certs_dir parameter in the /etc/chef/client.rb file
The following is the output of the knife ssl fetch command when everything is properly working. As shown in the following output, it will download the certificate and store it in the trusted_certs directory.
# knife ssl fetch WARNING: Certificates from datadb will be fetched and placed in your trusted_cert directory (/root/chef-repo/.chef/trusted_certs). Knife has no means to verify these are the correct certificates. You should verify the authenticity of these certificates after downloading. Adding certificate for datadb in /root/chef-repo/.chef/trusted_certs/datadb.crt
As we see here, on the workstation, the certificate is now downloaded and stored. This certificate will be used by knife command to communicate to the Chef server on SSL.
# ls -l .chef/trusted_certs/ -rw-r--r--. 1 root root 1330 Jun 24 09:44 datadb.crt
As you see, this is a standard SSL certificate. This is a self-signed certificated generated by Chef server.
# cat .chef/trusted_certs/datadb.crt -----BEGIN CERTIFICATE----- ANBgkqhkiG9w0BAQsFADBFMQswCQYDVQQGEwJVUzEQ ETMBEGA1UECwwKT3BlcmF0aW9uczEPMA0GA1UEAwwG .. .. IwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAN3l WO7bpxbQ1qqeuq9aLhNTIM1alwHfvlckmzSJsHRI+V TaWuk7xncRs4VCUw== -----END CERTIFICATE-----
Final Knife SSL Check Verification
After installing the SSL certificate properly as shown above, now when you execute the knife ssl check, it will not report any error. The following verifies that the Chef Server’s self-signed SSL certificate is properly installed on this machine and knife or chef-client command can use it without any issue.
# knife ssl check Connecting to host datadb:443 Successfully verified certificates from `datadb'
Knife SSL Check for a different Chef Server URL
By default, when you execute knife ssl check (or knife ssl fetch), how does it know which Chef server to connect to?
For this, it will use the chef_server_url from the knife.rb file as shown below.
# cat .chef/knife.rb
current_dir = File.dirname(__FILE__)
log_level :info
log_location STDOUT
node_name "ramesh"
client_key "#{current_dir}/ramesh.pem"
validation_client_name "thegeekstuff-validator"
validation_key "#{current_dir}/thegeekstuff-validator.pem"
chef_server_url "https://datadb/organizations/thegeekstuff"
cookbook_path ["#{current_dir}/../cookbooks"]
But, using knife ssl check, you can verify the SSL certificate for a different Chef server by passing the chef server url as a parameter. In the following example, it will connect to datadb-prod Chef server to verify the SSL certificate.
# knife ssl check https://datadb-prod/organizations/thegeekstuff Connecting to host datadb-prod:443 Successfully verified certificates from `datadb-prod'
Knife SSL Check on Individual Node (On Chef Client)
If you want to verify whether your client machine where the chef-client command is execute has the properly valid Chef server’s SSL certificate, then execute the knife ssl check command and pass the –config parameter as shown below.
# knife ssl check --config /etc/chef/client.rb Connecting to host datadb:443 Successfully verified certificates from `datadb'
In the above example, the knife ssl check command will use the chef_server_url parameter’s value from the following client.rb file as the Chef Server to connect and verify the SSL certificate.
# cat /etc/chef/client.rb log_location STDOUT chef_server_url "https://datadb/organizations/thegeekstuff" validation_client_name "thegeekstuff-validator" # Using default node name (fqdn) trusted_certs_dir "/etc/chef/trusted_certs"
Knife SSL Fetch from a different Chef Server URL
You can also get the SSL certificate from a different chef server (than the default one from the knife.rb) by specifying the chef server’s URL as a parameter to the knife ssl fetch command as shown below.
# knife ssl fetch https://datadb-prod/organizations/thegeekstuff WARNING: Certificates from datadb-prod will be fetched and placed in your trusted_cert directory (/root/chef-repo/.chef/trusted_certs). Knife has no means to verify these are the correct certificates. You should verify the authenticity of these certificates after downloading. Adding certificate for datadb-prod in /root/chef-repo/.chef/trusted_certs/datadb-prod.crt
Validate the Downloaded Chef Server Certificate
When you have the chef server and the chef workstation running on different machine and when you are doing the knife ssl fetch, it is a good idea to verify the check sum to make sure you have downloaded the correct certificate.
On the Chef Server, the SSL certificate is located under /var/opt/opscode/nginx/ca directory. Use the sha256sum command to get the check-sum as shown below.
# sha256sum /var/opt/opscode/nginx/ca/datadb.crt bd32b7f8ddc2e4e405f990dd6 /var/opt/opscode/nginx/ca/datadb.crt
On your Chef workstation, where you did “knife ssl fetch” command to download the certificate, execute the following sha256sum command and make sure the following check-sum value matches the above value.
# sha256sum .chef/trusted_certs/datadb.crt bd32b7f8ddc2e4e405f990dd6 .chef/trusted_certs/datadb.crt
My name is Ramesh Natarajan. I will be posting instruction guides, how-to, troubleshooting tips and tricks on Linux, database, hardware, security and web. My focus is to write articles that will either teach you or help you resolve a problem. Read more about