6.7.17

RedHat 6.9 Update Breaks Oracle Reports

For those that have Oracle Reports running, be aware. Updating the latest RedHat 6 or 7 updates will break the Oracle Reports execution.

For example upgrading to the latest RedHat 6.9 caused all the reports to be failing with signal 11 or signal 6.

This is a known issue (referring to Oracle technical note 2280616.1).


For fixing edit the reports.sh file

Reports 11.1.2.x:

File: INSTANCE_HOME/config/reports/bin/reports.sh

Add in the last line:
REPORTS_JVM_OPTIONS="-Xss2M"; export REPORTS_JVM_OPTIONS

Reports 12.2.1.x:

File: DOMAIN_HOME/reports/bin/reports.sh

Add in the last line:
REPORTS_JVM_OPTIONS="-Xss2M"; export REPORTS_JVM_OPTIONS

After modifying, restart the Reports server.

3.7.17

OVM Guest Linux LVM2 Disk Mounting

There might be need some time to mount OVM guest server disk images directly from OVM Server and modify some settings e.g. that prevent the server from booting or change passwords.

Typically doing this is quite straight forward by setting up a loop device and mounting the wanted partition. If the target partition is LVM2 partition, this becomes a bit more complex in OracleVM environment.

The problem is that by default OracleVM server /etc/lvm/lvm.conf has filtering enabled to prevent discovering loop device LVM2 devices.

This article describes the steps to perform to get a LVM2 volume mounted and data there changed off-line.

If you are unsure what you are doing, please make a backup of the virtual disk and lvm.conf you are going to change, before you proceed with following actions.

Instructions below expect that an experience Linux/OVM administrator knows what she/he is doing. I’m not going into details what a root will do after issuing the chroot command. ANY CHANGES ARE AT YOUR OWN RISK. TAKE GOOD BACKUPS ANYWAYS.

Preparation: modify /etc/lvm/lvm.conf

In order for OVM server to able to scan the loop devices /etc/lvm.conf needs to be modified.

# 30.6.2017 Harri Kaukovuo, modify the preferred_names
#preferred_names = [ "^/dev/mpath/", "^/dev/mapper/mpath", "^/dev/[hs]d" ]
preferred_names = [ ]

# 30.6.2017 Harri Kaukovuo, uncomment
filter = [ "a|.*/|" ]

# 30.6.2017 Harri Kaukovuo, comment out this line
#global_filter = [ "r|.*/|" ]

Mount the Virtual LVM2 Disk

Find the next free loop device:

losetup –f

By default OVM Server has max 10 loop devices. You might be running out of loop devices, which you can work around by either shutting down all the guest VM servers or increasing the loop devices by adding following line in /etc/rc.local and rebooting the OVM server:

# 28.6.2017 Harri Kaukovuo
/sbin/MAKEDEV -m 32 /dev/loop

Please note that above step is only needed if you ran out of loop devices.

Anyways, when you have a free loop device, in my example I have /dev/loop9 as the loop device, you can proceed with following step which is setting up the loop device to point to the virtual disk. In the example below I have retrieved the disk image file name from OVM Manager console:

losetup  /dev/loop9 /OVS/Repositories/0004fb000003000088c2307002d1b442/VirtualDisks/0004fb0000120000ba5aa22cb02675b7.img

Read partition tables from the loop device and create device maps with kpartx:

kpartx -av /dev/loop9

The output is something like:

[root@myovm01 etc]# kpartx -av /dev/loop9
add map loop9p1 (249:1): 0 401562 linear /dev/loop9 63
add map loop9p2 (249:2): 0 142898175 linear /dev/loop9 401625


Perform volume group scan: by issuing command:

vgscan

Output is like:

[root@myovm01 etc]# vgscan
   Reading all physical volumes.  This may take a while...
   Found volume group "VolGroup00" using metadata type lvm2

Activate volume groups by issuing command:

vgchange –ay

Output is like:

[root@myovm01 etc]# vgchange -ay
   4 logical volume(s) in volume group "VolGroup00" now active


Show logical volumes:

[root@myovm01 etc]# lvscan
   ACTIVE            '/dev/VolGroup00/LogVol03' [42.03 GiB] inherit
   ACTIVE            '/dev/VolGroup00/LogVol02' [6.06 GiB] inherit
   ACTIVE            '/dev/VolGroup00/LogVol00' [4.00 GiB] inherit
   ACTIVE            '/dev/VolGroup00/LogVol01' [16.00 GiB] inherit


Mount the wanted  logical volume. In this example case I know already that LogVol03 is the root partition and I want to change something there.


[root@myovm01 etc]# mount /dev/VolGroup00/LogVol03 /mnt/virtualdisk
[root@myovm01 etc]# df -H
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda2              53G  2.8G   48G   6% /
tmpfs                 3.2G     0  3.2G   0% /dev/shm
/dev/sda1             500M  145M  325M  31% /boot
none                  3.2G  213k  3.2G   1% /var/lib/xenstored
/dev/mapper/361866da0905943002027220113a0a9c8
                       8.4T  3.0T  5.5T  36% /OVS/Repositories/0004fb000003000088c2307002d1b442
/dev/mapper/VolGroup00-LogVol03
                        45G   33G  9.4G  78% /mnt/virtualdisk


Use chroot to change the root directory, if you need to change something there:

chroot /mnt/virtualdisk

(do your stuff here)

Get out of chroot by issuing “exit”

Unmount the disk after use:

umount /mnt/virtualdisk

Deactivate the volume group:

[root@myovm01 ~]# vgchange --activate n VolGroup00
   0 logical volume(s) in volume group "VolGroup00" now active

Delete the partition device mappings:

kpartx -dv /dev/loop9

Output is something like:

[root@myovm01 etc]# kpartx -dv /dev/loop9
del devmap : loop9p2
del devmap : loop9p1


Delete the loop device mapping:

losetup -d /dev/loop9

Now there should not be any LVM2 mappings found, also the loop device should be free:

[root@myovm01 ~]# pvscan
   No matching physical volumes found
[root@myovm01 ~]# vgscan
   Reading all physical volumes.  This may take a while...
[root@myovm01 ~]# losetup -f
/dev/loop9

After unmounting the disk, startup the guest linux and enjoy the changes.

13.6.17

OracleVM Server Update Disabled

One of the tasks to setup the OracleVM platform is to setup the YUM repository for updating the OracleVM servers from OracleVM Manager console.

If your OVM server sits behind the firewall and cannot connect the Oracle public YUM server without proxy setup you might have a situation where your OracleVM server context menu shows up like:

image

Update menu option is disabled even after you’ve setup the YUM server and enabled it.

Problem might be that your OVM Server needs to connect the public YUM server using company proxy server.

To fix this:

1. Login to OracleVM server as root

2. Edit /etc/yum.conf, configure to setup your company http proxy. Add following lines:

enableProxy=1
httpProxy=http://proxy.acme.com:8080
proxy=http://proxy.acme.com:8080

3. Disable and Enable the YUM repository setup in OracleVM Manager.

image


After this you should have the “Update” menu item enabled:

image

4.2.16

OracleVM Manager Console Failing with ERR_SSL_VERSION_OR_CIPHER_MISMATCH

Google Chrome version 48 dropped out the support for RC4 algorithm. This causes problems with OracleVM Manager 3.3 that uses RC4 as one of the default cipher suites.

The error is occuring once you try to access OVM Manager console. You will get
“ERR_SSL_VERSION_OR_CIPHER_MISMATCH”

To fix this, you need to add a new cipher suite to the OVM Manager weblogic configuration file.

Steps:
1. Login as oracle –user
2. cd /u01/app/oracle/ovm-manager-3/domains/ovm_domain/config
3. Back up the config.xml (e.g. copy it to config.xml.2016-02-04 or something)
4. Edit config.xml, add “<ciphersuite>TLS_RSA_WITH_AES_128_CBC_SHA</ciphersuite>” to the end of the AdminServer ciphersuite listing.

Should look something like this:









5. Restart the OVM Manager server as root:
service ovmm restart

After this you should be able to connect to OVM Manager console.

If you tried to use AES256 ciphersuite instead of AES128 you will get:

"java.lang.IllegalArgumentException: Cannot support TLS_RSA_WITH_AES_256_CBC_SHA with currently installed providers"
This is due to export restrictions, so you should use AES128 if you haven't updated the needed jars to support AES256.

4.12.15

OracleVM Serial Console Not Working - serial_console (serial_console) main process ended, respawning

 

After upgrading to the latest OracleVM 3.3.3 and updating all Oracle Linux 6 guests to the latest versions I started to see problems with OL6 serial console interface. First of all, the serial console didn’t seem to work at all, I could not connect to the console via OracleVM Manager serial console. Secondly there started to be continuous error messages in the OL6 /var/log/messages file:

Dec  4 06:30:34 atlassian init: serial_console (serial_console) main process (4704) terminated with status 1
Dec  4 06:30:34 atlassian init: serial_console (serial_console) main process ended, respawning

At the same time all Oracle Linux 5 servers worked fine and also few OL6 servers as well, but most of the OL6 servers suffered from this.

Those OL6 servers that worked fine had symbolic link in /dev/ where “serial_console” pointed to hvc0 device. Those OL6 servers that had problems this symbolic link points to ttyS0.

Don’t know exactly what is the root cause of this problem, but looks like there are two symlink definitions in kernel device udev rules and in some servers they point to hvc0 and some servers to ttyS0.

My fix to this problem was to edit the udev rules

vi /etc/udev/rules.d/50-udev.rules

Original content:

KERNEL=="ttyS0",                SYMLINK+="serial_console"
KERNEL=="hvc0",                 SYMLINK+="serial_console"

Remove the ttyS0 line so that the content looks like:

KERNEL=="hvc0",                 SYMLINK+="serial_console"

Save the file and reboot the server. After this serial console should work ok and no extra error messages should be appearing in the messages log file.

27.10.15

Changing OracleVM Manager 3.3.3 Server Certificate

 

OracleVM 3.x Manager server uses self signed certificates by default. OracleVM Manager certificate can be changed to use custom server certificates but unfortunately every time OVM Manager is upgraded, the server certificates are reset back to self signed certificates.

The latest OVM Manager versions use certificates to connect between OVM client and the server. This means that in order to get OVM Manager working properly also the ovmclient certificate keystore needs to be updated to include the needed root CAs. The problem with updating the ovmclient keystore is that the password for the keystore is generated and stored in the JPS keystore.

Use abbreviations in this blog:

  • CA = Certificate Authority
  • CSR = Certificate signing request

In this article I’ve used imagenary company ACME (A company that makes everything).

Working directory for the certificates is:

/u01/app/oracle/ovm-manager-3/domains/ovm_domain/security

Please execute all these steps as “oracle” user, with the exception of restarting the OVM Manager.

Step 0: Backup

Before your start doing anything, you should back up all needed files. Start with backing up the domain security directory and it’s contents:

/u01/app/oracle/ovm-manager-3/domains/ovm_domain/security

Secondly back up domain config directory and it’s contents:

/u01/app/oracle/ovm-manager-3/domains/ovm_domain/config

Step 1: Create new OVM Manager identity keystore

For example:

/u01/app/oracle/java/bin/keytool -keystore vmm3.jks -genkey -alias vmm3 -dname "EMAILADDRESS=admin@acme.com, CN=vmm3.acme.com, OU=ACME Ltd, O=IT, L=Espoo, ST=Uusimaa, C=FI" -keyalg rsa -keysize 2048
You don’t need to change the trust keystore.

Step2: Create certificate request

/u01/app/oracle/java/bin/keytool -keystore vmm3.jks -certreq -alias vmm3 -keyalg rsa -keysize 2048 -file vmm3.csr

Step 3: Use your chosen CA to sign the CSR


Send the contents of vmm3.csr to your CA and generate the service certificate.


Step 4: Import your CA


If you are using custom CA or public CA you need to import the CA public certificate into your keystore. In my example cacert.crt file is the public certificate for ACME CA.

/u01/app/oracle/java/bin/keytool -import -keystore vmm3.jks -file cacert.crt -alias ACMECA

Step 5: Import OVM CA


In my installation I also imported the OVM CA that was generated  by default for the installation. This step might not be needed since the OVM CA is already included in the trust jks. But anyways, importing this CA would not make any harm either.

/u01/app/oracle/java/bin/keytool -keystore vmm3.jks -import -file ovmca.pem -trustcacerts -alias ovmca

Step 6: Import your server certificate


After your CA has generated the server certificate, save it to e.g. vmm3.crt file and import it to the new keystore:


/u01/app/oracle/java/bin/keytool -import -keystore vmm3.jks -file vmm3.crt -alias vmm3


Step 7: Update WebLogic managed server keystore and SSL


At this point change the default ovmssl.jks to vmm3.jks and change the password to reflect the pwd you used when creating the vmm3.jks keystore.


When you restart the OVM Manager server you can see that https should now be working but you are still not able to connect to OVM Manager application. Instead you might see following error messages in the AdminServer.log file:

Caused By: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:385)
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292)
at sun.security.validator.Validator.validate(Validator.java:260)
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:326)
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:231)
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:126)
at com.oracle.ovm.appfw.ws.client.SSLContextFactory$X509ExtendedTrustManagerWrapper.checkServerTrusted(SSLContextFactory.java:307)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1428)

Step8: Update ovmclient.jks to include the used CA


ovmclient.jks is self generated keystore that also needs to have the CA certificate that OVM Manager server is using. In my example case the ACME CE certificate needs to be imported into this JKS keystore. The problem is that we need to dig out the keystore password first. This can be done using little piece of Jython code.


Create file e.g. called getpwd.py:

import sys
from oracle.security.jps.mas.mgmt.jmx.credstore import PortableCredential
connect('weblogic','password,'t3://localhost:7001')
domainRuntime()

on = ObjectName("com.oracle.jps:type=JpsCredentialStore")
sign = ["java.lang.String","java.lang.String"]
params = ["ovm_console","client.keystore"]
pwd = mbs.invoke(on, "getPortableCredential", params, sign)
credObject = PortableCredential.from(pwd)
print "PASSWORD:" + String.valueOf(credObject.getPassword())
Execute the script using wlst.sh
cd /u01/app/oracle/Middleware/oracle_common/common/bin
./wlst.sh getpwd.py
You should now have the password for the ovmclient.jks and ready for the final step.

Step 9: Import your CA into ovmclient.jks

/u01/app/oracle/java/bin/keytool -keystore ovmclient.jks -import -trustcacerts -file cacert.crt -alias ACMECA

Step 10: Restart the OVM server


As root:


service ovmm stop


service ovmm start


After these steps you should have your OVM Manager up and running using your chosen server certificate and your chosen CA.

 

25.6.15

OSB 12c Deployment Issues with BPM Enabled Domain

 

Another gray hair on my head today. It took a long time to find out the problem on my OSB 12.1.3 deployment problem. Created relatively simple OSB project with XSLT transformation. Started getting weird problems stating that I had problems with my WSDLs related to the XSLT. Removed the XSLT, still getting deployment errors “java.net.MalformedURLException”:

[05:28:43 PM] ----  Deployment incomplete  ----.
[05:28:43 PM] Conflicts found during publish.
The WSDL is not semantically valid: Failed to read wsdl file from url due to -- java.net.MalformedURLException: Unknown protocol: servicebus.
[OSB-398016]Error loading the WSDL from the repository: Failed to read wsdl file from url due to -- java.net.MalformedURLException: Unknown protocol: servicebus
[05:28:43 PM] Conflicts found during publish.

This is bug 18856204 that occurs when OSB is installed on the same domain as BPM.

The fix is to add:
felix.service.urlhandlers=false

in "bac-svnserver-osgi-framework" Init Properties –setup and then restart admin server.

Fix was originally found in this article:

https://middlewarebylink.wordpress.com/2014/07/17/soa-12c-end-to-end-e2e-tutorial-error-deploying-validatepayment/

OSB 12c Test Console Not Working

 

You might face an issue with 12.1.3 OSB web based test console not working when you press the “Play” button from OSB console. You might see following error:

Error Accessing Test Configuration
"Test Console" service is not running. Contact administrator to start this service.

Manual instructs you to set the Admin Server listen address to something else than empty (which means admin server listens for all IP addresses). An example of “localhost” could be valid if you are running your installation e.g. on your own workstation.

In my case I had to also set the listen addres for the osb server as well in order to get the test console working properly. After setting the listen address you need to bounce the servers.