4.12.15

OracleVM Serial Console Not Working - serial_console (serial_console) main process ended, respawning

 

After upgrading to the latest OracleVM 3.3.3 and updating all Oracle Linux 6 guests to the latest versions I started to see problems with OL6 serial console interface. First of all, the serial console didn’t seem to work at all, I could not connect to the console via OracleVM Manager serial console. Secondly there started to be continuous error messages in the OL6 /var/log/messages file:

Dec  4 06:30:34 atlassian init: serial_console (serial_console) main process (4704) terminated with status 1
Dec  4 06:30:34 atlassian init: serial_console (serial_console) main process ended, respawning

At the same time all Oracle Linux 5 servers worked fine and also few OL6 servers as well, but most of the OL6 servers suffered from this.

Those OL6 servers that worked fine had symbolic link in /dev/ where “serial_console” pointed to hvc0 device. Those OL6 servers that had problems this symbolic link points to ttyS0.

Don’t know exactly what is the root cause of this problem, but looks like there are two symlink definitions in kernel device udev rules and in some servers they point to hvc0 and some servers to ttyS0.

My fix to this problem was to edit the udev rules

vi /etc/udev/rules.d/50-udev.rules

Original content:

KERNEL=="ttyS0",                SYMLINK+="serial_console"
KERNEL=="hvc0",                 SYMLINK+="serial_console"

Remove the ttyS0 line so that the content looks like:

KERNEL=="hvc0",                 SYMLINK+="serial_console"

Save the file and reboot the server. After this serial console should work ok and no extra error messages should be appearing in the messages log file.

27.10.15

Changing OracleVM Manager 3.3.3 Server Certificate

 

OracleVM 3.x Manager server uses self signed certificates by default. OracleVM Manager certificate can be changed to use custom server certificates but unfortunately every time OVM Manager is upgraded, the server certificates are reset back to self signed certificates.

The latest OVM Manager versions use certificates to connect between OVM client and the server. This means that in order to get OVM Manager working properly also the ovmclient certificate keystore needs to be updated to include the needed root CAs. The problem with updating the ovmclient keystore is that the password for the keystore is generated and stored in the JPS keystore.

Use abbreviations in this blog:

  • CA = Certificate Authority
  • CSR = Certificate signing request

In this article I’ve used imagenary company ACME (A company that makes everything).

Working directory for the certificates is:

/u01/app/oracle/ovm-manager-3/domains/ovm_domain/security

Please execute all these steps as “oracle” user, with the exception of restarting the OVM Manager.

Step 0: Backup

Before your start doing anything, you should back up all needed files. Start with backing up the domain security directory and it’s contents:

/u01/app/oracle/ovm-manager-3/domains/ovm_domain/security

Secondly back up domain config directory and it’s contents:

/u01/app/oracle/ovm-manager-3/domains/ovm_domain/config

Step 1: Create new OVM Manager identity keystore

For example:

/u01/app/oracle/java/bin/keytool -keystore vmm3.jks -genkey -alias vmm3 -dname "EMAILADDRESS=admin@acme.com, CN=vmm3.acme.com, OU=ACME Ltd, O=IT, L=Espoo, ST=Uusimaa, C=FI" -keyalg rsa -keysize 2048
You don’t need to change the trust keystore.

Step2: Create certificate request

/u01/app/oracle/java/bin/keytool -keystore vmm3.jks -certreq -alias vmm3 -keyalg rsa -keysize 2048 -file vmm3.csr

Step 3: Use your chosen CA to sign the CSR


Send the contents of vmm3.csr to your CA and generate the service certificate.


Step 4: Import your CA


If you are using custom CA or public CA you need to import the CA public certificate into your keystore. In my example cacert.crt file is the public certificate for ACME CA.

/u01/app/oracle/java/bin/keytool -import -keystore vmm3.jks -file cacert.crt -alias ACMECA

Step 5: Import OVM CA


In my installation I also imported the OVM CA that was generated  by default for the installation. This step might not be needed since the OVM CA is already included in the trust jks. But anyways, importing this CA would not make any harm either.

/u01/app/oracle/java/bin/keytool -keystore vmm3.jks -import -file ovmca.pem -trustcacerts -alias ovmca

Step 6: Import your server certificate


After your CA has generated the server certificate, save it to e.g. vmm3.crt file and import it to the new keystore:


/u01/app/oracle/java/bin/keytool -import -keystore vmm3.jks -file vmm3.crt -alias vmm3


Step 7: Update WebLogic managed server keystore and SSL


At this point change the default ovmssl.jks to vmm3.jks and change the password to reflect the pwd you used when creating the vmm3.jks keystore.


When you restart the OVM Manager server you can see that https should now be working but you are still not able to connect to OVM Manager application. Instead you might see following error messages in the AdminServer.log file:

Caused By: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:385)
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292)
at sun.security.validator.Validator.validate(Validator.java:260)
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:326)
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:231)
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:126)
at com.oracle.ovm.appfw.ws.client.SSLContextFactory$X509ExtendedTrustManagerWrapper.checkServerTrusted(SSLContextFactory.java:307)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1428)

Step8: Update ovmclient.jks to include the used CA


ovmclient.jks is self generated keystore that also needs to have the CA certificate that OVM Manager server is using. In my example case the ACME CE certificate needs to be imported into this JKS keystore. The problem is that we need to dig out the keystore password first. This can be done using little piece of Jython code.


Create file e.g. called getpwd.py:

import sys
from oracle.security.jps.mas.mgmt.jmx.credstore import PortableCredential
connect('weblogic','password,'t3://localhost:7001')
domainRuntime()

on = ObjectName("com.oracle.jps:type=JpsCredentialStore")
sign = ["java.lang.String","java.lang.String"]
params = ["ovm_console","client.keystore"]
pwd = mbs.invoke(on, "getPortableCredential", params, sign)
credObject = PortableCredential.from(pwd)
print "PASSWORD:" + String.valueOf(credObject.getPassword())
Execute the script using wlst.sh
cd /u01/app/oracle/Middleware/oracle_common/common/bin
./wlst.sh getpwd.py
You should now have the password for the ovmclient.jks and ready for the final step.

Step 9: Import your CA into ovmclient.jks

/u01/app/oracle/java/bin/keytool -keystore ovmclient.jks -import -trustcacerts -file cacert.crt -alias ACMECA

Step 10: Restart the OVM server


As root:


service ovmm stop


service ovmm start


After these steps you should have your OVM Manager up and running using your chosen server certificate and your chosen CA.

 

25.6.15

OSB 12c Deployment Issues with BPM Enabled Domain

 

Another gray hair on my head today. It took a long time to find out the problem on my OSB 12.1.3 deployment problem. Created relatively simple OSB project with XSLT transformation. Started getting weird problems stating that I had problems with my WSDLs related to the XSLT. Removed the XSLT, still getting deployment errors “java.net.MalformedURLException”:

[05:28:43 PM] ----  Deployment incomplete  ----.
[05:28:43 PM] Conflicts found during publish.
The WSDL is not semantically valid: Failed to read wsdl file from url due to -- java.net.MalformedURLException: Unknown protocol: servicebus.
[OSB-398016]Error loading the WSDL from the repository: Failed to read wsdl file from url due to -- java.net.MalformedURLException: Unknown protocol: servicebus
[05:28:43 PM] Conflicts found during publish.

This is bug 18856204 that occurs when OSB is installed on the same domain as BPM.

The fix is to add:
felix.service.urlhandlers=false

in "bac-svnserver-osgi-framework" Init Properties –setup and then restart admin server.

Fix was originally found in this article:

https://middlewarebylink.wordpress.com/2014/07/17/soa-12c-end-to-end-e2e-tutorial-error-deploying-validatepayment/

OSB 12c Test Console Not Working

 

You might face an issue with 12.1.3 OSB web based test console not working when you press the “Play” button from OSB console. You might see following error:

Error Accessing Test Configuration
"Test Console" service is not running. Contact administrator to start this service.

Manual instructs you to set the Admin Server listen address to something else than empty (which means admin server listens for all IP addresses). An example of “localhost” could be valid if you are running your installation e.g. on your own workstation.

In my case I had to also set the listen addres for the osb server as well in order to get the test console working properly. After setting the listen address you need to bounce the servers.

28.4.15

Changing Environment Warning Color on WLS Console

 

One of the rare mistakes in system configuring is when you actually thought you were in development or testing environment, but the browser actually pointed to production. You might even have the same credentials between the environment (e.g. linked to AD or LDAP directory), making it difficult to recognize the fatal error until it’s too late.

When the consoles look alike there is nothing to differentiate from the other environments but you just have to be sharp to acknowledge were you are.

One of the easiest changes to prevent these errors happening could be to change the production (or some other) environment WLS console coloring so that it differs from other environments to alarm that this is not for testing or development.

How about if the WLS console looked like this:

image

This is easily done.

WebLogic 12c

cd $WLHOME/wlserver/server/lib/consoleapp/webapp/css

WebLogic 11g

cd $WLHOME/wlserver_10.3/server/lib/consoleapp/webapp/css

 

Make backup of the content.css.

Edit the toolbar background color:

/*
    Toolbar Area
*/
.toolbar {
    background-color: #D2E5F9;
    overflow: hidden;
    width: 100%;
    padding: 1px;
}

 

change it to something else, like orange/red:

/*
    Toolbar Area
*/
.toolbar {
    background-color: #FF6666;
    overflow: hidden;
    width: 100%;
    padding: 1px;
}

After this change the WLS console immediately shows “warning color” in the toolbar as a marker for a production environment.

You might need to make sure the change is redone after system upgrades if the default content.css is for some reason overridden.

19.3.15

The trustAnchors parameter must be non-empty -error

If you ever configured AD or some other LDAP authentication provider for you WebCenter, SOA or BPM Suite and decided to follow the best practice on securing the LDAP traffice with SSL you will most propably end up with problems connecting to LDAPS at some point.

You might see these errors in your log file:

Caused By: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty

The problem is that even if you imported your LDAP or AD server root and intermediate CAs into either JKS (11G) or KSS (12c) you need to perform one additional step on creating keystore for libOVD and is used with identity provider virtualization (once you set up “virtualize=true” in the domain security provider setup.

To get the problem solved you need to perform two steps:

1. Run libovdconfig.sh that creates the adapters.jks for libOVD

2. Import the needed CA certificates into this adapters.jks

 

Detailed steps are described below.

For the step 1 (12c installation being here as a sample, the same applies with 11g but a bit different subdirectory names for Oracle Home and WL Home):

cd /u01/app/oracle/product/fmw12c/oracle_common/bin/

export ORACLE_HOME=/u01/app/oracle/product/fmw12c/soa
export WL_HOME=/u01/app/oracle/product/fmw12c/wlserver
export JAVA_HOME=/usr/java/latest

./libovdconfig.sh -host myhost.com -port 7001-userName weblogic -domainPath /u01/app/oracle/admin/BPMDEV_Domain/mserver/BPMDEV_Domain -createKeystore

Once the keystore is created, proceed to step 2:

cd /u01/app/oracle/admin/BPMDEV_Domain/mserver/BPMDEV_Domain/config/fmwconfig/ovd/default/keystores
/usr/java/latest/bin/keytool -import -file mycacertificate.der -keystore adapters.jks -trustcacerts -alias ldap.myhost.com

Enter keystore password: ********
Certificate already exists in system-wide CA keystore under alias <ldap.myhost.com>
Do you still want to add it to your own keystore? [no]: yes
Certificate was added to keystore

After these steps restart the managed servers and you are good to go.

28.2.15

Best Practice on Pointing to Latest JDK in FMW Installation

 

One of the best practices in Oracle Middleware installations is to point to logical “latest” JDK directory in the configuration files. When you install e.g. WebLogic server 11g on Linux you need to have the JDK installed before starting up the WebLogic installation.

You might have installed e.g. JDK 1.7. rpm which installs the JDK into /usr/java and by default has the best practice directories “default” that points to “latest” and “latest” symbolic link that points to the latest/wanted JDK installation.

/usr/java might look like this:

drwxr-xr-x. 8 root root 4096 Feb 28 08:09 jdk1.7.0_75
lrwxrwxrwx. 1 root root   21 Feb 28 08:09 latest -> /usr/java/jdk1.7.0_75
lrwxrwxrwx. 1 root root   16 Feb 28 08:09 default -> /usr/java/latest

If you didn’t have the “latest” symbolic link there, you might be running older versions like JRockit or JDK6.

Despite of having the /usr/java/latest symbolic link, you will get the full real path suggested in the installer:

image

Despite of pressing “Browse…” and choosing the /usr/java/latest, the JDK path will not change in the installer. You will get “Please select appropriate JDK” error.

image

There are two options to go:

1. Install the software pointing to the real path and then later change all shell scripts etc. pointing to the real path, to point to /usr/java/latest

or

2. Exit the installer and change the real JDK directory name temporarily to /usr/java/latest, restart the installer and the installer will find the JDK in the path /usr/java/latest, configure the environment variables. After the installer has finished, change the real JDK path back to what it was and point the /usr/java/latest to point to the wanted JKD version.

Here are the steps for the option 2)

cd /usr/java

mv latest latest.old

mv jdk1.7.0_75 latest

Install the Oracle FMW software. When installing e.g. Weblogic server you should point the JDK to /usr/java/latest.

image

After the installation has finished, reverse the changes:

mv latest jdk1.7.0_75

mv latest.old latest.

 

You can veriry that the shell scripts point to this logical directory by looking at:

$MW_HOME/wlserver_10.3/common/bin/commEnv.sh.