18.12.19

OracleVM Serial Console Not Working - serial_console (serial_console) main process ended, respawning - Part 2

Back in 2015 I wrote an article that explained the workaround to the serial console problem (continuous errors in /var/log/messages) and serial console not working.

http://kaukovuo.blogspot.com/2015/12/oraclevm-serial-console-not-working.html

The issue resurfaced at some point and I continued having Linux servers that printed continuous error messages  like this:

Dec 18 20:33:49 lb01 init: serial_console (serial_console) main process (30384) terminated with status 1
Dec 18 20:33:49 lb01 init: serial_console (serial_console) main process ended, respawning
Dec 18 20:33:59 lb01 init: serial_console (serial_console) main process (30399) terminated with status 1
Dec 18 20:33:59 lb01 init: serial_console (serial_console) main process ended, respawning
Dec 18 20:34:09 lb01 init: serial_console (serial_console) main process (30411) terminated with status 1
Dec 18 20:34:09 lb01 init: serial_console (serial_console) main process ended, respawning
 
The fix for this modifying the /etc/udev/rules.d/50-udev.rules
 
Change hvc0 to ttyS0, like this:
KERNEL=="hvc0",                 SYMLINK+="serial_console"
to
KERNEL=="ttyS0",                 SYMLINK+="serial_console"
 
Save the file and reboot the server.

10.11.17

OracleVM 3.3.4 and Data Corruption When Cloning VM from Template

We faced quite serious issues once trying to create a virtual server from a template that resided in NFS mount, target repository was on iSCSI storage.

The problem was that for some reason OracleVM 3.3.4 kernel 3.8.13 started corrupting the image while cloning the virtual server from template. Symptoms were that after the clone operation everything looks good from OracleVM Manager point ot view but when trying to startup the server, it fails with error stating there is no bootable operating system.

During the cloning operations there were huge amount of following errors in the /var/log/messages. The errors were the same despite I changed the utilility server to be different, so this is not hardware issue:

Nov  9 18:35:17 vs9 kernel: sd 5:0:0:0: [sdd] CDB:
Nov  9 18:35:17 vs9 kernel: Write(10): 2a 00 26 bf f0 00 00 0a 00 00
Nov  9 18:35:17 vs9 kernel: sd 5:0:0:0: [sdd] Invalid command failure
Nov  9 18:35:17 vs9 kernel: sd 5:0:0:0: [sdd]
Nov  9 18:35:17 vs9 kernel: Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Nov  9 18:35:17 vs9 kernel: sd 5:0:0:0: [sdd]
Nov  9 18:35:17 vs9 kernel: Sense Key : Illegal Request [current]
Nov  9 18:35:17 vs9 kernel: sd 5:0:0:0: [sdd]
Nov  9 18:35:17 vs9 kernel: Add. Sense: Invalid field in cdb
Nov  9 18:35:17 vs9 kernel: sd 5:0:0:0: [sdd] CDB:
Nov  9 18:35:17 vs9 kernel: Write(10): 2a 00 26 bf fa 00 00 0a 00 00
Nov  9 18:35:17 vs9 kernel: JBD2: Detected IO errors while flushing file data on dm-3-617
When searching for the explanation, looks like this is an issue with using iSCSI and jumbo frames with certain 3.9 kernel versions. Could be that this is an issue with OVM 3.8 kernel as well.What makes this particularly nasty is that we’ve made several copies of virtual servers for backup purposes and there is no guarantee that those copies are valid and functional any more.

Possible Solution

To troubleshoot the fix I decided to upgrade the whole OracleVM park to the latest OVM 3.4.4 that uses Kernel 4.1.

After upgrading the all the pools and OVM Manager to 3.4.4 looks like we got rid of this nasty behaviour.

Tried exactly the same way ot cloning, using the same servers, no errors and the cloned virtual server works just fine.

Recommendation

I strongly recommend to upgrade to OracleVM 3.4.x as soon as possible if you are using 3.3.x, iSCSI and jumbo frames AND you are seeing these errors.

Check your OracleVM servers, if you see any of these errors in /var/log/messages, you might have data corruption issues in the images.

6.7.17

RedHat 6.9 Update Breaks Oracle Reports

For those that have Oracle Reports running, be aware. Updating the latest RedHat 6 or 7 updates will break the Oracle Reports execution.

For example upgrading to the latest RedHat 6.9 caused all the reports to be failing with signal 11 or signal 6.

This is a known issue (referring to Oracle technical note 2280616.1).


For fixing edit the reports.sh file

Reports 11.1.2.x:

File: INSTANCE_HOME/config/reports/bin/reports.sh

Add in the last line:
REPORTS_JVM_OPTIONS="-Xss2M"; export REPORTS_JVM_OPTIONS

Reports 12.2.1.x:

File: DOMAIN_HOME/reports/bin/reports.sh

Add in the last line:
REPORTS_JVM_OPTIONS="-Xss2M"; export REPORTS_JVM_OPTIONS

After modifying, restart the Reports server.

3.7.17

OVM Guest Linux LVM2 Disk Mounting

There might be need some time to mount OVM guest server disk images directly from OVM Server and modify some settings e.g. that prevent the server from booting or change passwords.

Typically doing this is quite straight forward by setting up a loop device and mounting the wanted partition. If the target partition is LVM2 partition, this becomes a bit more complex in OracleVM environment.

The problem is that by default OracleVM server /etc/lvm/lvm.conf has filtering enabled to prevent discovering loop device LVM2 devices.

This article describes the steps to perform to get a LVM2 volume mounted and data there changed off-line.

If you are unsure what you are doing, please make a backup of the virtual disk and lvm.conf you are going to change, before you proceed with following actions.

Instructions below expect that an experience Linux/OVM administrator knows what she/he is doing. I’m not going into details what a root will do after issuing the chroot command. ANY CHANGES ARE AT YOUR OWN RISK. TAKE GOOD BACKUPS ANYWAYS.

Preparation: modify /etc/lvm/lvm.conf

In order for OVM server to able to scan the loop devices /etc/lvm.conf needs to be modified.

# 30.6.2017 Harri Kaukovuo, modify the preferred_names
#preferred_names = [ "^/dev/mpath/", "^/dev/mapper/mpath", "^/dev/[hs]d" ]
preferred_names = [ ]

# 30.6.2017 Harri Kaukovuo, uncomment
filter = [ "a|.*/|" ]

# 30.6.2017 Harri Kaukovuo, comment out this line
#global_filter = [ "r|.*/|" ]

Mount the Virtual LVM2 Disk

Find the next free loop device:

losetup –f

By default OVM Server has max 10 loop devices. You might be running out of loop devices, which you can work around by either shutting down all the guest VM servers or increasing the loop devices by adding following line in /etc/rc.local and rebooting the OVM server:

# 28.6.2017 Harri Kaukovuo
/sbin/MAKEDEV -m 32 /dev/loop

Please note that above step is only needed if you ran out of loop devices.

Anyways, when you have a free loop device, in my example I have /dev/loop9 as the loop device, you can proceed with following step which is setting up the loop device to point to the virtual disk. In the example below I have retrieved the disk image file name from OVM Manager console:

losetup  /dev/loop9 /OVS/Repositories/0004fb000003000088c2307002d1b442/VirtualDisks/0004fb0000120000ba5aa22cb02675b7.img

Read partition tables from the loop device and create device maps with kpartx:

kpartx -av /dev/loop9

The output is something like:

[root@myovm01 etc]# kpartx -av /dev/loop9
add map loop9p1 (249:1): 0 401562 linear /dev/loop9 63
add map loop9p2 (249:2): 0 142898175 linear /dev/loop9 401625


Perform volume group scan: by issuing command:

vgscan

Output is like:

[root@myovm01 etc]# vgscan
   Reading all physical volumes.  This may take a while...
   Found volume group "VolGroup00" using metadata type lvm2

Activate volume groups by issuing command:

vgchange –ay

Output is like:

[root@myovm01 etc]# vgchange -ay
   4 logical volume(s) in volume group "VolGroup00" now active


Show logical volumes:

[root@myovm01 etc]# lvscan
   ACTIVE            '/dev/VolGroup00/LogVol03' [42.03 GiB] inherit
   ACTIVE            '/dev/VolGroup00/LogVol02' [6.06 GiB] inherit
   ACTIVE            '/dev/VolGroup00/LogVol00' [4.00 GiB] inherit
   ACTIVE            '/dev/VolGroup00/LogVol01' [16.00 GiB] inherit


Mount the wanted  logical volume. In this example case I know already that LogVol03 is the root partition and I want to change something there.


[root@myovm01 etc]# mount /dev/VolGroup00/LogVol03 /mnt/virtualdisk
[root@myovm01 etc]# df -H
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda2              53G  2.8G   48G   6% /
tmpfs                 3.2G     0  3.2G   0% /dev/shm
/dev/sda1             500M  145M  325M  31% /boot
none                  3.2G  213k  3.2G   1% /var/lib/xenstored
/dev/mapper/361866da0905943002027220113a0a9c8
                       8.4T  3.0T  5.5T  36% /OVS/Repositories/0004fb000003000088c2307002d1b442
/dev/mapper/VolGroup00-LogVol03
                        45G   33G  9.4G  78% /mnt/virtualdisk


Use chroot to change the root directory, if you need to change something there:

chroot /mnt/virtualdisk

(do your stuff here)

Get out of chroot by issuing “exit”

Unmount the disk after use:

umount /mnt/virtualdisk

Deactivate the volume group:

[root@myovm01 ~]# vgchange --activate n VolGroup00
   0 logical volume(s) in volume group "VolGroup00" now active

Delete the partition device mappings:

kpartx -dv /dev/loop9

Output is something like:

[root@myovm01 etc]# kpartx -dv /dev/loop9
del devmap : loop9p2
del devmap : loop9p1


Delete the loop device mapping:

losetup -d /dev/loop9

Now there should not be any LVM2 mappings found, also the loop device should be free:

[root@myovm01 ~]# pvscan
   No matching physical volumes found
[root@myovm01 ~]# vgscan
   Reading all physical volumes.  This may take a while...
[root@myovm01 ~]# losetup -f
/dev/loop9

After unmounting the disk, startup the guest linux and enjoy the changes.

13.6.17

OracleVM Server Update Disabled

One of the tasks to setup the OracleVM platform is to setup the YUM repository for updating the OracleVM servers from OracleVM Manager console.

If your OVM server sits behind the firewall and cannot connect the Oracle public YUM server without proxy setup you might have a situation where your OracleVM server context menu shows up like:

image

Update menu option is disabled even after you’ve setup the YUM server and enabled it.

Problem might be that your OVM Server needs to connect the public YUM server using company proxy server.

To fix this:

1. Login to OracleVM server as root

2. Edit /etc/yum.conf, configure to setup your company http proxy. Add following lines:

enableProxy=1
httpProxy=http://proxy.acme.com:8080
proxy=http://proxy.acme.com:8080

3. Disable and Enable the YUM repository setup in OracleVM Manager.

image


After this you should have the “Update” menu item enabled:

image

4.2.16

OracleVM Manager Console Failing with ERR_SSL_VERSION_OR_CIPHER_MISMATCH

Google Chrome version 48 dropped out the support for RC4 algorithm. This causes problems with OracleVM Manager 3.3 that uses RC4 as one of the default cipher suites.

The error is occuring once you try to access OVM Manager console. You will get
“ERR_SSL_VERSION_OR_CIPHER_MISMATCH”

To fix this, you need to add a new cipher suite to the OVM Manager weblogic configuration file.

Steps:
1. Login as oracle –user
2. cd /u01/app/oracle/ovm-manager-3/domains/ovm_domain/config
3. Back up the config.xml (e.g. copy it to config.xml.2016-02-04 or something)
4. Edit config.xml, add “<ciphersuite>TLS_RSA_WITH_AES_128_CBC_SHA</ciphersuite>” to the end of the AdminServer ciphersuite listing.

Should look something like this:









5. Restart the OVM Manager server as root:
service ovmm restart

After this you should be able to connect to OVM Manager console.

If you tried to use AES256 ciphersuite instead of AES128 you will get:

"java.lang.IllegalArgumentException: Cannot support TLS_RSA_WITH_AES_256_CBC_SHA with currently installed providers"
This is due to export restrictions, so you should use AES128 if you haven't updated the needed jars to support AES256.

4.12.15

OracleVM Serial Console Not Working - serial_console (serial_console) main process ended, respawning

 

After upgrading to the latest OracleVM 3.3.3 and updating all Oracle Linux 6 guests to the latest versions I started to see problems with OL6 serial console interface. First of all, the serial console didn’t seem to work at all, I could not connect to the console via OracleVM Manager serial console. Secondly there started to be continuous error messages in the OL6 /var/log/messages file:

Dec  4 06:30:34 atlassian init: serial_console (serial_console) main process (4704) terminated with status 1
Dec  4 06:30:34 atlassian init: serial_console (serial_console) main process ended, respawning

At the same time all Oracle Linux 5 servers worked fine and also few OL6 servers as well, but most of the OL6 servers suffered from this.

Those OL6 servers that worked fine had symbolic link in /dev/ where “serial_console” pointed to hvc0 device. Those OL6 servers that had problems this symbolic link points to ttyS0.

Don’t know exactly what is the root cause of this problem, but looks like there are two symlink definitions in kernel device udev rules and in some servers they point to hvc0 and some servers to ttyS0.

My fix to this problem was to edit the udev rules

vi /etc/udev/rules.d/50-udev.rules

Original content:

KERNEL=="ttyS0",                SYMLINK+="serial_console"
KERNEL=="hvc0",                 SYMLINK+="serial_console"

Remove the ttyS0 line so that the content looks like:

KERNEL=="hvc0",                 SYMLINK+="serial_console"

Save the file and reboot the server. After this serial console should work ok and no extra error messages should be appearing in the messages log file.

27.10.15

Changing OracleVM Manager 3.3.3 Server Certificate

 

OracleVM 3.x Manager server uses self signed certificates by default. OracleVM Manager certificate can be changed to use custom server certificates but unfortunately every time OVM Manager is upgraded, the server certificates are reset back to self signed certificates.

The latest OVM Manager versions use certificates to connect between OVM client and the server. This means that in order to get OVM Manager working properly also the ovmclient certificate keystore needs to be updated to include the needed root CAs. The problem with updating the ovmclient keystore is that the password for the keystore is generated and stored in the JPS keystore.

Use abbreviations in this blog:

  • CA = Certificate Authority
  • CSR = Certificate signing request

In this article I’ve used imagenary company ACME (A company that makes everything).

Working directory for the certificates is:

/u01/app/oracle/ovm-manager-3/domains/ovm_domain/security

Please execute all these steps as “oracle” user, with the exception of restarting the OVM Manager.

Step 0: Backup

Before your start doing anything, you should back up all needed files. Start with backing up the domain security directory and it’s contents:

/u01/app/oracle/ovm-manager-3/domains/ovm_domain/security

Secondly back up domain config directory and it’s contents:

/u01/app/oracle/ovm-manager-3/domains/ovm_domain/config

Step 1: Create new OVM Manager identity keystore

For example:

/u01/app/oracle/java/bin/keytool -keystore vmm3.jks -genkey -alias vmm3 -dname "EMAILADDRESS=admin@acme.com, CN=vmm3.acme.com, OU=ACME Ltd, O=IT, L=Espoo, ST=Uusimaa, C=FI" -keyalg rsa -keysize 2048
You don’t need to change the trust keystore.

Step2: Create certificate request

/u01/app/oracle/java/bin/keytool -keystore vmm3.jks -certreq -alias vmm3 -keyalg rsa -keysize 2048 -file vmm3.csr

Step 3: Use your chosen CA to sign the CSR


Send the contents of vmm3.csr to your CA and generate the service certificate.


Step 4: Import your CA


If you are using custom CA or public CA you need to import the CA public certificate into your keystore. In my example cacert.crt file is the public certificate for ACME CA.

/u01/app/oracle/java/bin/keytool -import -keystore vmm3.jks -file cacert.crt -alias ACMECA

Step 5: Import OVM CA


In my installation I also imported the OVM CA that was generated  by default for the installation. This step might not be needed since the OVM CA is already included in the trust jks. But anyways, importing this CA would not make any harm either.

/u01/app/oracle/java/bin/keytool -keystore vmm3.jks -import -file ovmca.pem -trustcacerts -alias ovmca

Step 6: Import your server certificate


After your CA has generated the server certificate, save it to e.g. vmm3.crt file and import it to the new keystore:


/u01/app/oracle/java/bin/keytool -import -keystore vmm3.jks -file vmm3.crt -alias vmm3


Step 7: Update WebLogic managed server keystore and SSL


At this point change the default ovmssl.jks to vmm3.jks and change the password to reflect the pwd you used when creating the vmm3.jks keystore.


When you restart the OVM Manager server you can see that https should now be working but you are still not able to connect to OVM Manager application. Instead you might see following error messages in the AdminServer.log file:

Caused By: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:385)
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292)
at sun.security.validator.Validator.validate(Validator.java:260)
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:326)
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:231)
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:126)
at com.oracle.ovm.appfw.ws.client.SSLContextFactory$X509ExtendedTrustManagerWrapper.checkServerTrusted(SSLContextFactory.java:307)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1428)

Step8: Update ovmclient.jks to include the used CA


ovmclient.jks is self generated keystore that also needs to have the CA certificate that OVM Manager server is using. In my example case the ACME CE certificate needs to be imported into this JKS keystore. The problem is that we need to dig out the keystore password first. This can be done using little piece of Jython code.


Create file e.g. called getpwd.py:

import sys
from oracle.security.jps.mas.mgmt.jmx.credstore import PortableCredential
connect('weblogic','password,'t3://localhost:7001')
domainRuntime()

on = ObjectName("com.oracle.jps:type=JpsCredentialStore")
sign = ["java.lang.String","java.lang.String"]
params = ["ovm_console","client.keystore"]
pwd = mbs.invoke(on, "getPortableCredential", params, sign)
credObject = PortableCredential.from(pwd)
print "PASSWORD:" + String.valueOf(credObject.getPassword())
Execute the script using wlst.sh
cd /u01/app/oracle/Middleware/oracle_common/common/bin
./wlst.sh getpwd.py
You should now have the password for the ovmclient.jks and ready for the final step.

Step 9: Import your CA into ovmclient.jks

/u01/app/oracle/java/bin/keytool -keystore ovmclient.jks -import -trustcacerts -file cacert.crt -alias ACMECA

Step 10: Restart the OVM server


As root:


service ovmm stop


service ovmm start


After these steps you should have your OVM Manager up and running using your chosen server certificate and your chosen CA.

 

25.6.15

OSB 12c Deployment Issues with BPM Enabled Domain

 

Another gray hair on my head today. It took a long time to find out the problem on my OSB 12.1.3 deployment problem. Created relatively simple OSB project with XSLT transformation. Started getting weird problems stating that I had problems with my WSDLs related to the XSLT. Removed the XSLT, still getting deployment errors “java.net.MalformedURLException”:

[05:28:43 PM] ----  Deployment incomplete  ----.
[05:28:43 PM] Conflicts found during publish.
The WSDL is not semantically valid: Failed to read wsdl file from url due to -- java.net.MalformedURLException: Unknown protocol: servicebus.
[OSB-398016]Error loading the WSDL from the repository: Failed to read wsdl file from url due to -- java.net.MalformedURLException: Unknown protocol: servicebus
[05:28:43 PM] Conflicts found during publish.

This is bug 18856204 that occurs when OSB is installed on the same domain as BPM.

The fix is to add:
felix.service.urlhandlers=false

in "bac-svnserver-osgi-framework" Init Properties –setup and then restart admin server.

Fix was originally found in this article:

https://middlewarebylink.wordpress.com/2014/07/17/soa-12c-end-to-end-e2e-tutorial-error-deploying-validatepayment/

OSB 12c Test Console Not Working

 

You might face an issue with 12.1.3 OSB web based test console not working when you press the “Play” button from OSB console. You might see following error:

Error Accessing Test Configuration
"Test Console" service is not running. Contact administrator to start this service.

Manual instructs you to set the Admin Server listen address to something else than empty (which means admin server listens for all IP addresses). An example of “localhost” could be valid if you are running your installation e.g. on your own workstation.

In my case I had to also set the listen addres for the osb server as well in order to get the test console working properly. After setting the listen address you need to bounce the servers.

28.4.15

Changing Environment Warning Color on WLS Console

 

One of the rare mistakes in system configuring is when you actually thought you were in development or testing environment, but the browser actually pointed to production. You might even have the same credentials between the environment (e.g. linked to AD or LDAP directory), making it difficult to recognize the fatal error until it’s too late.

When the consoles look alike there is nothing to differentiate from the other environments but you just have to be sharp to acknowledge were you are.

One of the easiest changes to prevent these errors happening could be to change the production (or some other) environment WLS console coloring so that it differs from other environments to alarm that this is not for testing or development.

How about if the WLS console looked like this:

image

This is easily done.

WebLogic 12c

cd $WLHOME/wlserver/server/lib/consoleapp/webapp/css

WebLogic 11g

cd $WLHOME/wlserver_10.3/server/lib/consoleapp/webapp/css

 

Make backup of the content.css.

Edit the toolbar background color:

/*
    Toolbar Area
*/
.toolbar {
    background-color: #D2E5F9;
    overflow: hidden;
    width: 100%;
    padding: 1px;
}

 

change it to something else, like orange/red:

/*
    Toolbar Area
*/
.toolbar {
    background-color: #FF6666;
    overflow: hidden;
    width: 100%;
    padding: 1px;
}

After this change the WLS console immediately shows “warning color” in the toolbar as a marker for a production environment.

You might need to make sure the change is redone after system upgrades if the default content.css is for some reason overridden.

19.3.15

The trustAnchors parameter must be non-empty -error

If you ever configured AD or some other LDAP authentication provider for you WebCenter, SOA or BPM Suite and decided to follow the best practice on securing the LDAP traffice with SSL you will most propably end up with problems connecting to LDAPS at some point.

You might see these errors in your log file:

Caused By: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty

The problem is that even if you imported your LDAP or AD server root and intermediate CAs into either JKS (11G) or KSS (12c) you need to perform one additional step on creating keystore for libOVD and is used with identity provider virtualization (once you set up “virtualize=true” in the domain security provider setup.

To get the problem solved you need to perform two steps:

1. Run libovdconfig.sh that creates the adapters.jks for libOVD

2. Import the needed CA certificates into this adapters.jks

 

Detailed steps are described below.

For the step 1 (12c installation being here as a sample, the same applies with 11g but a bit different subdirectory names for Oracle Home and WL Home):

cd /u01/app/oracle/product/fmw12c/oracle_common/bin/

export ORACLE_HOME=/u01/app/oracle/product/fmw12c/soa
export WL_HOME=/u01/app/oracle/product/fmw12c/wlserver
export JAVA_HOME=/usr/java/latest

./libovdconfig.sh -host myhost.com -port 7001-userName weblogic -domainPath /u01/app/oracle/admin/BPMDEV_Domain/mserver/BPMDEV_Domain -createKeystore

Once the keystore is created, proceed to step 2:

cd /u01/app/oracle/admin/BPMDEV_Domain/mserver/BPMDEV_Domain/config/fmwconfig/ovd/default/keystores
/usr/java/latest/bin/keytool -import -file mycacertificate.der -keystore adapters.jks -trustcacerts -alias ldap.myhost.com

Enter keystore password: ********
Certificate already exists in system-wide CA keystore under alias <ldap.myhost.com>
Do you still want to add it to your own keystore? [no]: yes
Certificate was added to keystore

After these steps restart the managed servers and you are good to go.

28.2.15

Best Practice on Pointing to Latest JDK in FMW Installation

 

One of the best practices in Oracle Middleware installations is to point to logical “latest” JDK directory in the configuration files. When you install e.g. WebLogic server 11g on Linux you need to have the JDK installed before starting up the WebLogic installation.

You might have installed e.g. JDK 1.7. rpm which installs the JDK into /usr/java and by default has the best practice directories “default” that points to “latest” and “latest” symbolic link that points to the latest/wanted JDK installation.

/usr/java might look like this:

drwxr-xr-x. 8 root root 4096 Feb 28 08:09 jdk1.7.0_75
lrwxrwxrwx. 1 root root   21 Feb 28 08:09 latest -> /usr/java/jdk1.7.0_75
lrwxrwxrwx. 1 root root   16 Feb 28 08:09 default -> /usr/java/latest

If you didn’t have the “latest” symbolic link there, you might be running older versions like JRockit or JDK6.

Despite of having the /usr/java/latest symbolic link, you will get the full real path suggested in the installer:

image

Despite of pressing “Browse…” and choosing the /usr/java/latest, the JDK path will not change in the installer. You will get “Please select appropriate JDK” error.

image

There are two options to go:

1. Install the software pointing to the real path and then later change all shell scripts etc. pointing to the real path, to point to /usr/java/latest

or

2. Exit the installer and change the real JDK directory name temporarily to /usr/java/latest, restart the installer and the installer will find the JDK in the path /usr/java/latest, configure the environment variables. After the installer has finished, change the real JDK path back to what it was and point the /usr/java/latest to point to the wanted JKD version.

Here are the steps for the option 2)

cd /usr/java

mv latest latest.old

mv jdk1.7.0_75 latest

Install the Oracle FMW software. When installing e.g. Weblogic server you should point the JDK to /usr/java/latest.

image

After the installation has finished, reverse the changes:

mv latest jdk1.7.0_75

mv latest.old latest.

 

You can veriry that the shell scripts point to this logical directory by looking at:

$MW_HOME/wlserver_10.3/common/bin/commEnv.sh.

22.11.14

OracleVM 3.3.1 and External Authentication

Some of the customers (including my company Ratioware) has defined OracleVM Manager to use external authentication (like LDAP or Active Directory) to be used to authenticate the users when logging into OracleVM Manager console.

This used to work well with OracleVM 3.2.x.

When we upgraded OracleVM 3.2.8 to OracleVM 3.3.1 we noticed that all those custom authentication settings were lost. Well, that is “kind” of acceptable assuming Oracle perhaps doesn’t like us to tweak the underlying WebLogic.

After changing the authentication provider back to utilize our external authentication provider I noticed that I wasn’t any more able to login to OracleVM console. I was able to login to WebLogic Console but not into OracleVM Manager console. I got “Unexpected error during login”.

Error messages in the weblogic log files stated:

<2014-11-11T20:19:25.066+0200> <Error> <com.oracle.ovm.appfw.coreinterface.ConnectionManager> <BEA-000000> <AppFw session 1: Failed to connect to Web Service API.
com.oracle.ovm.mgr.ws.model.WsException: AUTH_000002:Connection to manager failed: AUTH_000002:Connection to manager failed: Certificate authentication failed: certificate unrecognized (CN=admin, OU=Oracle VM Manager, O=Oracle Corporation, C=US).
Tue Nov 11 20:19:25 EET 2014 (AUTH_000002)

Changing the order of authentication providers or the requried –attributes didn’t help in the issue.

There is Oracle Doc ID 1942473.1 related to this with solution suggestion:

Steps to fix this:

  1. cd /u01/app/oracle/ovm-manager-3/bin/
  2. ./configure_client_cert_login.sh

The configure_client_cert_login.sh will ask you the username and the password for the OVM manager. In my case I did have “admin” user but for some reason the password was not upgraded correctly and I needed to reset the “admin” user password in OVM WLS console before I could execute this script correctly.

You could potentially also use “weblogic” user to run the script.

After running this client certification script the OracleVM Manager login started working correctly and we were able to use our external authenticator with our OracleVM Manager console.

9.11.14

Connection to VirtualBox BPM 12c Project Access Manager (PAM) from Local JDeveloper

I attended Oracle BPM Suite 12c partner training in Finland and the labs had Oracle Virtual Box image that we used in the training.

Virtual Box had JDeveloper embedded but I wanted to use my local copy of BPM Quick Start JDeveloper running on my bare metal laptop.

When trying to set up Project Access Manager (PAM) and connect to the repository I would get errors like: Could not connect to repository endpoint: localhost:7323

The reason for this was that when the internal SVN server starts up it looks up the hostname (soabpm-vm in this case) and queries the IP address for this. In pre-built virtual box the /etc/hosts file looks like:

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
soabpm-vm soabpm-vm.site

When BPM Suite 12c is starting up all other components start listening for 0.0.0.0 address (all IP addresses) but SVN server starts listening for 127.0.0.1 address. This means we cannot access this SVN address outside the Virtual Box, even when using port forwarding.

I tried to find a way to configure the SVN server to listed for 0.0.0.0 addresses but I could not find that. That was somewhere deep inside the code that wasn’t easily discovered.

To workaround this issue it requires reconfiguring the network interfaces:

Step 1

Add second network interface to virtual machine that is attached to “Host-Only Adapter”.

image

This enables the connectivity to 192.168.56.0 network.

In my setup Host Only network details looks like this in the main Virtual Box preferences window:

SNAGHTML7043576

As we can see the built-in DHCP server starts delivering the DHCP addresses starting from 192.168.56.101. It is safe to assign fixed addresses below 192.168.56.100, in this case I decided to assign my Virtual Box BPM 12c installation a fixed IP address of 192.168.56.50.

I still wanted to keep the NAT network interface in the virtual box because that enables me to use network resources outside the virtual box (like external YUM repositories etc).

Step 2

Configure the ethernet interface (either eth1 or eth2 depending on how you defined your virtual networks in VM settings):

cd /etc/sysconfig/network-scripts

ifcfg-eth2 or (ifcfg-eth1 if you changed the NAT’ed to Host-Only Adapter)
# Please read /usr/share/doc/initscripts-*/sysconfig.txt
# for the documentation of these parameters.
DEVICE=eth1
BOOTPROTO=static
TYPE=Ethernet
HWADDR=08:00:27:41:19:1a
NM_CONTROLLED=no
PEERDNS=yes
IPADDR=192.168.56.50
SUBNET=255.255.25.0

Remember to change the HWADDR to point to your virtual NIC Hardware address.

After changing the network adapter settings you can reset the network settings my issuing command “service network restart” as root.

Step 3

Changed the IP address in the /etc/hosts file like this:

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.56.50 soabpm-vm soabpm-vm.site

Step 4

After network settings has been changed restart the BPM Suite 12c installation. If running pre-build BPM 12c image, just bounce the AdminServer.

Step 5

Edit your workstartion hosts file to point to the fixed IP address. As Windows Administrator edit the file

C:\Windows\System32\drivers\etc\hosts

Add line:

192.168.56.50 soabpm-vm.site soabpm-vm

Step 6

Configure your BPM Studio PAM connection and test that it works:

image

22.4.14

OpenSSL Heartbleed Oracle Fixes

Some of the Oracle software is also affected by the Heartbleed vulnerability. Specifically if you are using Oracle Linux 6, you should update your operating system using ULN or Oracle Public YUM repository.

For more information please see:

http://www.oracle.com/technetwork/topics/security/opensslheartbleedcve-2014-0160-2188454.html

3.4.14

OracleVM 3.2.8 Downloadable

OracleVM 3.2.8 just got published in Oracle Support and Public Yum repository.

Use Patch 16410417 to download the OracleVM Manager 3.2.8 zipped ISO image. You can use Oracle yum repository to upgrade your OracleVM Servers.

7.2.14

Oracle BPM Suite 11.1.1.7.2 WebForms entryId is null

Ran into Oracle BPM Suite 11.1.1.7 Web Form issue after upgrading from 11.1.1.6 to 11.1.1.7. When trying to create web form in Oracle BPM composer, I get “entryId is null” error.

When looking at the log files I can see

[2014-02-06T13:54:03.233-08:00] [AdminServer] [ERROR] [] [com.frevvo.forms.web.ConfigurationResource] [tid: [ACTIVE].ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: weblogic] [ecid: 11d1def534ea1be0:-4d925ede:14409288963:-8000-00000000000002a3,0] [APP: frevvo] Configuration error: Users database schema - ORA-00942: table or view does not exist[[]]

… 
[2014-02-06T13:54:03.245-08:00] [AdminServer] [WARNING] [] [com.frevvo.billing.database.UserDBUtil] [tid: [ACTIVE].ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: weblogic] [ecid: 11d1def534ea1be0:-4d925ede:14409288963:-8000-00000000000002a3,0] [APP: frevvo] Could not get user ids for tenant d[[
java.sql.SQLSyntaxErrorException: ORA-00942: table or view does not exist      

Solution

Even though I had executed psa against the SOA infrastructure, looks like still some tables were not upgraded properly. Might have been because I extended the domain with Frevvo extensions later on.

Solution is to execute psa (once more) for soainfra. This will upgrade soa-infra to 11.1.1.7. so that those Frevvo tables are created properly and you can design your Web forms.

11.11.13

Extending OracleVM Guest Disk

These instructions has been working at least with OracleVM 2.2 disk images.

Every now and then you might end up with situation where you run out of disk space on your guest VM.

1. Shut down your guest VM.

In order for the guest VM to recognize the extended disk size you need to have the system restarted.

2. Navigate to the guest VM directory in your OracleVM Server.

cd /OVS/running_pool/myhost

In this example the guest VM has ran out of disk space on yumrepo.img.

We want to extend this disk from 100GB –> 200 GB.

3. Create an additional 100GB empty temporary file:

dd if=/dev/zero of=tempfile bs=1024 count=10000000

or:

dd if=/dev/zero of=tempfile bs=100G count=1

This might take some time depending on your storage connection.

4. Make a backup out of extendable disk image file

cp yumrepo.img yourepo-old.img

5. Concatenate the empty disk image to the end of the existing disk image.

cat tempfile >> yumrepo.img &

Depending on your storage system, this step could also be taking some time. If you experience network timeouts on your ssh terminal session, it’s good practice to submit your concatenation to background using “&”.

6. Start the guest VM again and login as root.

After restart the guest VM should internally recognize the extended device, althought you cannot see that bigger size before you have resized the disk.

7. Unmount the disk if that has been in use.

8. You have to repartition the disk image with fdisk.

fdisk /dev/xvdb

Print out the partition layout with “p” option to see how the disk has been formatted. If you see that image is constructed out of one primary partition, please go ahead with these instructions. If you see that disk is constructed such way that e.g. the swap space is the latest and the partition you wanted to extend cannot be extended without deleting the swap partition, please do not go ahead with these instructions. This latter example is e.g. when you try to extend a system disk.

Remove the partition with “d” option and create again (n –> primary –> 1). When recreating the partition again you will see larger sector end number.

9. Check the filesystem:

fsck -f /dev/xvdb1

10. Resize the filesystem:

resize2fs /dev/xvdb1

11. Mount the file system and it’s ready to use.

30.9.12

SOA Suite 10g–> 11g migration and Issue with composite.xml Attributes

Here is a heads up for those that might still be working on 10g –> 11g migrations. There is one unfortunate bug (8980875) in the migration process that seems innocent in the beginning but has critical effects in the end.

I logged a service request for continuous XML parse errors in the managed server log files:

org.xml.sax.SAXParseException: <Line 5, Column 92>: XML-20129: (Error) Namespace prefix 'ui' used but not declared.

This occurs at deployment time. In most cases the deployed composite worked just fine.

Error comes from the composite.xml content where the migrated 10g SOA projects have all the XML root element “composite” attributes in a single line. The deployment parser seems to be some sort of home grown XML parser that assumes that all the attributes are physically separated in their own lines, like this:

<composite name="Archive"
           revision="1.0"
           label="2012-08-16_09-51-28_557"
           mode="active"
           state="on"
           xmlns="
http://xmlns.oracle.com/sca/1.0"
           xmlns:xs="http://www.w3.org/2001/XMLSchema"
           xmlns:wsp="http://schemas.xmlsoap.org/ws/2004/09/policy"
           xmlns:orawsp="http://schemas.oracle.com/ws/2006/01/policy"
           xmlns:ui="http://xmlns.oracle.com/soa/designer/">
</composite>

and when the migrated projects have something like this:

<composite name="HelloWorld" revision="1.0" mode="active" state="on" xmlns:ui="http://xmlns.oracle.com/soa/designer/

in a single line, the deployment will give errors in this part. You might have some references in composite.xml pointing to external servers using “ui.wsdlLocation” like this:

<reference ui:wsdlLocation=http://mydevhost1.mydomain.com:8001/soa-infra/services/CommonExceptionHandling/CommonExceptionHandling.wsdl name="CommonException...

By default these should not affect the runtime environment since this is used at development time. Unfortunately combined with this bug 8980875, the pointer seems to stay in the runtime environment causing gray hairs when e.g. all of the sudden production environment has pointers to development environment. Changing this ui.wsdlLocation seems very hard if not possible using the built-in deployment scripts and default search/replace element descriptors.

In the end, the advice from me is to change all the migrated 10g –> 11g SOA projects manually to correct the composite.xml “composite” element attributes to their own lines. This will help greatly in the dev->tst->prd deployments avoiding those unwanted cross environment pointers.