ISSUE:
cutover failed and came out of adop sessions
findings:
But adop status shows cutover is running
Tried to rerun the adop cutover but failed , as cutover is already in running phase got below error
error
Session status : RUNNING
[ERROR] Unable to continue as an existing adop session was found
Node Name Node Type Phase Status Started Finished Elapsed
--------------- ---------- --------------- --------------- -------------------- -------------------- ------------
hosterld01vn36 master PREPARE COMPLETED 2016/08/31 19:55:03 2016/08/31 20:17:27 0:22:24
APPLY COMPLETED 2016/08/31 20:47:08 2016/08/31 22:56:07 2:08:59
FINALIZE COMPLETED 2016/08/31 23:19:42 2016/08/31 23:20:02 0:00:20
CUTOVER RUNNING 2016/09/01 00:18:57
CLEANUP NOT STARTED
hosterld01vn37 slave PREPARE COMPLETED 2016/08/31 20:05:02 2016/08/31 20:33:22 0:28:20
APPLY COMPLETED 2016/08/31 21:16:28 2016/08/31 23:06:54 1:50:26
FINALIZE COMPLETED 2016/08/31 23:19:42 2016/08/31 23:20:02 0:00:20
CUTOVER RUNNING 2016/09/01 00:29:00
CLEANUP NOT STARTED
hosterld01vn38 slave PREPARE COMPLETED 2016/08/31 20:04:18 2016/08/31 20:31:45 0:27:27
APPLY COMPLETED 2016/08/31 21:15:45 2016/08/31 23:06:09 1:50:24
FINALIZE COMPLETED 2016/08/31 23:19:42 2016/08/31 23:20:02 0:00:20
CUTOVER RUNNING 2016/09/01 00:28:15
CLEANUP NOT STARTED
hosterld01vn39 slave PREPARE COMPLETED 2016/08/31 20:04:44 2016/08/31 20:33:56 0:29:12
APPLY COMPLETED 2016/08/31 21:16:10 2016/08/31 23:06:35 1:50:25
FINALIZE COMPLETED 2016/08/31 23:19:42 2016/08/31 23:20:02 0:00:20
CUTOVER RUNNING 2016/09/01 00:28:41
CLEANUP NOT STARTED
Workaround
===========
SQL> select ADOP_SESSION_ID,PREPARE_STATUS,APPLY_STATUS,FINALIZE_STATUS,CUTOVER_STATUS,CLEANUP_STATUS,ABORT_STATUS,STATUS,ABANDON_FLAG,NODE_NAME from AD_ADOP_SESSIONS where ADOP_SESSION_ID='102';
ADOP_SESSION_ID P A F C C A S ABANDON_FLAG NODE_NAME
--------------- - - - - - - - ------------ -------------------------------
102 Y Y Y 4 N X R 102 hosterld01vn39
102 Y Y Y 1 N N R 102 hosterld01vn37
102 Y Y Y 4 N X R 102 hosterl001vn36
102 Y Y Y 4 N X R 102 hosterl001vn38
Took backup of AD_ADOP_SESSIONS, and tried to update and made CUTOVER_STATUS to 'F' (fail)
SQL> update AD_ADOP_SESSIONS set CUTOVER_STATUS='F' where ADOP_SESSION_ID='102';
4 rows updated.
reran cutover
this time cutover failed on oneadmin node hosterld01vn37
+ Cutover completed on allother nodes except pwerld01vn37
+ Started adminserver on patch file system
+ updated AD_ADOP_SESSIONS for node 37 ( updated CUTOVER_STATUS and STATUS to N ( not started)
SQL> update AD_ADOP_SESSIONS set CUTOVER_STATUS='N'WHERE node_name = 'pwerld01vn37' AND ADOP_SESSION_ID='102';
1 row updated.
SQL> update AD_ADOP_SESSIONS set STATUS='N'WHERE node_name = 'pwerld01vn37' AND ADOP_SESSION_ID='102';
1 row updated.
+ Reran Cutover .. this time fs swithed and cutover completed
********************************************************
ISSUE fs_clone failing
error from log
/tti/applmgr/fs_ne/EBSapps/log/adop/134/fs_clone_20151017_010354/TTi_eu504vm0142/TXK_SYNC_create_Sat_Oct_17_01_05_20_2015/fsclone_apply/FSCloneApplyAppsTier_10170220.log
+++++++++++++++++++++++++
configProperty id = Server24
Count for NodeIterator nextNode = 3
ERROR: Managed Server's are not in sync between file system context and DB context
ERROR: Update Moveplan Fail
START: Inside exitClone....
Updating status INCOMPLETE for ApplywlsTechStack
START: Updating status INCOMPLETE for action ApplywlsTechStack
END: Updated status INCOMPLETE for action ApplywlsTechStack
Findings/observation
======================
SQL> select name,created from v$database;
NAME CREATED
--------------------------- ---------------
TTi 23-SEP-15
SQL>
+ context files have been registered in the database
select path,status,serial_number,version from fnd_oam_context_files where name not in ('TEMPLATE','METADATA') and (status is null or status !='VALID');
PATH STA SERIAL_NUMBER VERSION
-------------------------------------------------------------------------------- --- ------------- ----------------------------------------
/tti/inst/fs1/inst/apps/TTi_eu504vm0145/appl/admin/TTi_eu504vm0145.xml S 228 120.401.12020000.45
/tti/inst/fs2/inst/apps/TTi_eu504vm0147/appl/admin/TTi_eu504vm0147.xml S 120.401.12020000.45
/tti/oracle/product/11.2.0.4/appsutil/TTi1_eu01dbadm01.xml S 120.52.12020000.3
/tti/inst/fs1/inst/apps/TTi_eu504vm0142/appl/admin/TTi_eu504vm0142.xml S 120.401.12020000.45
/tti/inst/fs1/inst/apps/TTi_eu504vm0143/appl/admin/TTi_eu504vm0143.xml S 228 120.401.12020000.45
/tti/inst/fs1/inst/apps/TTi_eu504vm0146/appl/admin/TTi_eu504vm0146.xml S 228 120.401.12020000.45
/tti/oracle/product/112/appsutil/TTi2_eu01dbadm02.xml H 120.52.12020000.3
/tti/inst/fs2/inst/apps/TTi_eu504vm0143/appl/admin/TTi_eu504vm0143.xml S 120.401.12020000.45
/tti/inst/fs1/inst/apps/TTi_eu504vm0144/appl/admin/TTi_eu504vm0144.xml S 228 120.401.12020000.45
/tti/inst/fs2/inst/apps/TTi_eu504vm0146/appl/admin/TTi_eu504vm0146.xml S 120.401.12020000.45
/tti/inst/fs2/inst/apps/TTi_eu504vm0144/appl/admin/TTi_eu504vm0144.xml S 120.401.12020000.45
/tti/inst/fs1/inst/apps/TTi_eu504vm0147/appl/admin/TTi_eu504vm0147.xml S 228 120.401.12020000.45
/tti/oracle/product/11.2.0.4/appsutil/TTi2_eu01dbadm02.xml S 120.52.12020000.3
/tti/inst/fs2/inst/apps/TTi_eu504vm0145/appl/admin/TTi_eu504vm0145.xml S 120.401.12020000.45
/tti/oracle/product/112/appsutil/TTi1_eu01dbadm01.xml H 120.52.12020000.3
/tti/inst/fs2/inst/apps/TTi_eu504vm0142/appl/admin/TTi_eu504vm0142.xml S 120.401.12020000.45
16 rows selected.
+ number/types of managed servers registered in the run file systems
RUN FILE SYSTEM ENTRIES IN DATABASE
====================================
SELECT extractValue(XMLType(TEXT),'//host[@oa_var="s_hostname"]'),
extractValue(XMLType(TEXT),'//oacore_server_ports'),
extractValue(XMLType(TEXT),'//forms_server_ports'),
extractValue(XMLType(TEXT),'//oafm_server_ports'),
extractValue(XMLType(TEXT),'//forms-c4ws_server_ports'),
extractValue(XMLType(TEXT),'//oaea_server_ports')
from fnd_oam_context_files
where name not in ('TEMPLATE','METADATA')
and (status is null or status !='H')
and EXTRACTVALUE(XMLType(TEXT),'//file_edition_type')='run'
and CTX_TYPE = 'A';
eu504vm0147
oacore_server6:7202,oacore_server63:7206,oacore_server62:7204,oacore_server64:7208
forms_server6:7402
oafm_server6:7602
forms-c4ws_server6:7802
eu504vm0143
oacore_server2:7202,oacore_server23:7206,oacore_server22:7204,oacore_server24:7208
forms_server2:7402
oafm_server2:7602
forms-c4ws_server2:7802
eu504vm0146
oacore_server5:7202,oacore_server53:7206,oacore_server52:7204,oacore_server54:7208
forms_server5:7402
oafm_server5:7602
forms-c4ws_server5:7802
eu504vm0144
oacore_server3:7202,oacore_server33:7206,oacore_server32:7204,oacore_server34:7208
forms_server3:7402
oafm_server3:7602
forms-c4ws_server3:7802
eu504vm0145
oacore_server4:7202,oacore_server43:7206,oacore_server42:7204,oacore_server44:7208
forms_server4:7402
oafm_server4:7602
forms-c4ws_server4:7802
eu504vm0142
oacore_server1:7202,oacore_server12:7204,oacore_server13:7206,oacore_server14:7208
forms_server1:7402
oafm_server1:7602
forms-c4ws_server1:7802
6 rows selected.
+++++++++++++++++++
+ number/types of managed servers registered in the patch file systems
PATCH FILE SYSTEM ENTRIES IN DATABASE
====================================
SELECT extractValue(XMLType(TEXT),'//host[@oa_var="s_hostname"]'),
extractValue(XMLType(TEXT),'//oacore_server_ports'),
extractValue(XMLType(TEXT),'//forms_server_ports'),
extractValue(XMLType(TEXT),'//oafm_server_ports'),
extractValue(XMLType(TEXT),'//forms-c4ws_server_ports'),
extractValue(XMLType(TEXT),'//oaea_server_ports')
from fnd_oam_context_files
where name not in ('TEMPLATE','METADATA')
and (status is null or status !='H')
and EXTRACTVALUE(XMLType(TEXT),'//file_edition_type')='patch'
and CTX_TYPE = 'A';
eu504vm0145
oacore_server42:7205,oacore_server44:7209
forms_server4:7401
oafm_server4:7601
forms-c4ws_server4:7801
eu504vm0142
oacore_server1:7201,oacore_server12:7203,oacore_server13:7205,oacore_server14:7207
forms_server1:7401
oafm_server1:7601
forms-c4ws_server1:7801
eu504vm0143
oacore_server22:7205
forms_server2:7401
oafm_server2:7601
forms-c4ws_server2:7801
eu504vm0146
oacore_server54:7209,oacore_server53:7207
forms_server5:7401
oafm_server5:7601
forms-c4ws_server5:7801
eu504vm0144
oacore_server33:7207
forms_server3:7401
oafm_server3:7601
forms-c4ws_server3:7801
eu504vm0147
oacore_server64:7209
forms_server6:7401
oafm_server6:7601
forms-c4ws_server6:7801
6 rows selected.
++ run and patch filesystem managed servers registered in the database are not in sync ( mismatch)
ACTION PLAN
=============
from Doc ID 1582473.1
If the patch file system data (managed servers or context file) is missing from the database then run the following after sourcing the run file system environment.
This command is executed on the RUN file system to upload the PATCH context file:
$ADJVAPRG oracle.apps.ad.autoconfig.oam.CtxSynchronizer \
action=upload \
contextfile=<full path to patch context file> \
logfile=/tmp/patchctxupload.log
Review the log file to confirm the process ran without error.
Confirm that the fnd_oam_context_files table now holds the correct data.
Thanks,
&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&
ISSUE FS_CLONE FAILING
ADOPLOG
========
====================================
Inside _validateETCHosts()...
====================================
Started execution : ADOPValidations.java
Validating entries in /etc/hosts
WARNING: /etc/hosts file does not contain correct entry for localhost.
Add the following entry to the /etc/hosts file:
127.0.0.1 localhost.localdomain localhost
Completed execution : ADOPValidations.java
cmd : java weblogic.WLST /pti/applmgr/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkValidateDomainInRC.py --adminuser=weblogic --verify=domainEditModeEnabled --adminurl=t3://eu504vm0202.tp.tpnet.intra:7001>/pti/inst/fs1/inst/apps/PTi_eu504vm0202/logs/appl/rgf/TXK/txkValidateDomainInRCOut.txt
Domain is not in edit mode.
[appti@eu504vm0202 <RUN> ]$ ls -lrt /pti/applmgr/fs2/FMW_Home/user_projects/domains/EBS_domain_PTi/edit.lok
-rw-r----- 1 appti aapti 33 Aug 22 04:50 /pti/applmgr/fs2/FMW_Home/user_projects/domains/EBS_domain_PTi/edit.lok
[appti@eu504vm0202 <RUN> ]$
[appti@eu504vm0202 <RUN> ]$ ls -lrt /pti/applmgr/fs1/FMW_Home/user_projects/domains/EBS_domain_PTi/edit.lok
-rw-r----- 1 appti aapti 33 Aug 21 06:03 /pti/applmgr/fs1/FMW_Home/user_projects/domains/EBS_domain_PTi/edit.lok
[appti@eu504vm0202 <RUN> ]$
Action Plan
=============
1) On PATCH fs
rename /pti/applmgr/fs2/FMW_Home/user_projects/domains/EBS_domain_PTi/edit.lok edit.lok_old
2) Source the patch file system environment
3) Execute :
cd $ADMIN_SCRIPTS_HOME
./adadminsrvctl.sh stop forcepatchfs
Thanks,
.
&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&
FINDINGS:
adop phase=fs_clone fails
ValidateInventory:
[ERROR]: At least one Oracle inventory check has failed.
ERROR: /dalq3i/applmgr/fs2/EBSapps/10.1.2 is not registered in the inventory
Corrective Action: Provide the location of a valid inventory file.
If you believe the inventory is valid, you may want to attach the /dalq3i/applmgr/fs2/EBSapps/10.1.2.
Checking well formed-ness of file /dalq3i/applmgr/oraInventory/ContentsXML/inventory.xml
/dalq3i/applmgr/oraInventory/ContentsXML/inventory.xml
SOLUTION:
=========
How to Create a Clean oraInventory in Release 12.2 (Doc ID 1967205.1)
cd $ORACLE_HOME/oui/bin
./runInstaller -silent -ignoreSysPrereqs -attachHome ORACLE_HOME="/dalq3i/applmgr/fs2/EBSapps/10.1.2" ORACLE_HOME_NAME="dalq3i_TOOLS__dalq3i_applmgr_fs2_EBSapps_10_1_2"
&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&
.5523TVL TTi cutover failed Unable to acquire lock on sessions table 1903316.1
cutover failed
/tti/applmgr/fs_ne/EBSapps/log/adop/129/cutover_20150815_101941/TTi_eu504vm0142/log/cutover.lo
g
>>>
...
AutoPatch error:
XBOL_TOP value does not match value from topfile.
Expected: /pti/applmgr/fs1/EBSapps/appl/xbol/12.0.0
Found : /tti/applmgr/fs1/EBSapps/appl/xbol/12.0.0
AutoPatch error:
Error reconciling product top settings with environment in aimini().
You should check the file
/tti/applmgr/fs_ne/EBSapps/log/adop/129/cutover_20150815_101941/TTi_eu504vm0142/log/cutover.lo
g
...
<<<
[aptti@eu504vm0142 <RUN> ]$ set | grep XBOL
XBOL_TOP=/tti/applmgr/fs2/EBSapps/appl/xbol/12.0.0
[aptti@eu504vm0142 <RUN> ]$
[aptti@eu504vm0142 <RUN> ]$ grep xbol /tti/applmgr/fs2/EBSapps/appl/admin/topfile.txt
xbol /tti/applmgr/fs2/EBSapps/appl
[aptti@eu504vm0142 <RUN> ]$
[aptti@eu504vm0142 <RUN> ]$ ls -ltr /tti/applmgr/fs2/EBSapps/appl/admin/topfile.txt*
-rw-r--r-- 1 aptti aatti 6751 Aug 5 11:36
/tti/applmgr/fs2/EBSapps/appl/admin/topfile.txt_bkp_05AUG2015
-rw-r--r-- 1 aptti aatti 6751 Aug 9 04:53
/tti/applmgr/fs2/EBSapps/appl/admin/topfile.txt_bkp_09AUG2015
-rw-r--r-- 1 aptti aatti 6751 Aug 13 15:01
/tti/applmgr/fs2/EBSapps/appl/admin/topfile.txt_bkp13aug15
-rw-r--r-- 1 aptti aatti 6751 Aug 14 02:37
/tti/applmgr/fs2/EBSapps/appl/admin/topfile.txt_bkp_14aug2015
-rw-r--r-- 1 aptti aatti 6751 Aug 14 02:38
/tti/applmgr/fs2/EBSapps/appl/admin/topfile.txt
[aptti@eu504vm0142 <RUN> ]$ diff
/tti/applmgr/fs2/EBSapps/appl/admin/topfile.txt_bkp_14aug2015
/tti/applmgr/fs2/EBSapps/appl/admin/topfile.txt
170c170
< xbol /pti/applmgr/fs2/EBSapps/appl
---
> xbol /tti/applmgr/fs2/EBSapps/appl
[aptti@eu504vm0142 <RUN> ]$
Similar issue in:
Bug 18697209 : OMCS:ADOP PREPARE PHASE FAILS WHEN RECONCILING CUSTOM PRODUCT TOP
SQL> l
1* select OA_VAR, NAME, DEFAULT_VALUE from FND_OAM_CONTEXT_CUSTOM
SQL> /
OA_VAR NAME DEFAULT_VALUE
------------------------------ ------------------------------
----------------------------------------------------------------------
c_xbol c_xbol
/pti/applmgr/fs2/EBSapps/appl/xbol/12.0.0
c_xx_itesoft c_xx_itesoft
/pti/applmgr/fs1/EBSapps/appl/xx_itesoft/12.0.0
c_xx c_xx
/data/applgeos12d9/geos12d9/fs1/EBSapps/appl/xx/12.0.0
c_cle c_cle
/data/applgeos12d9/geos12d9/fs1/EBSapps/appl/cle/12.0.0
c_xxmig c_xxmig
/pti/applmgr/fs2/EBSapps/appl/xxmig/12.0.0
> XBOL_TOP value is not reflected correctly (looks like post clonning issue)
=== Action Plan ===
1. abort ADOP
2. run autoconfig
3. re-try applying patch hrglobal.drv (last action plan with adop only)
[aptti@eu504vm0142 <RUN> ]$ adop phase=cutover
.
.
.
adop exiting with status = 0 (Success)
[aptti@eu504vm0142 <RUN> ]$
&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&
.53G76XB pti "Fs_clone failing txkValidateDomainInRCOut.txt
Domain is not in edit mode" 1920804.1
&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&
RFC .5DQ9WAL
ISSUE : cutover failing
from node mohpc015
=============================================
ValidateOHSConfigFilesForCrossRef ...
=============================================
ERROR: File - /tp1i/applmgr/fs1/FMW_Home/webtier/instances/EBS_web_Tp1I_OHS2/bin/opmnctl contains cross reference in it.
ERROR: File - /tp1i/applmgr/fs1/FMW_Home/webtier/instances/EBS_web_Tp1I_OHS2/bin/opmnctl contains cross reference in it.
ERROR: File - /tp1i/applmgr/fs1/FMW_Home/webtier/instances/EBS_web_Tp1I_OHS2/bin/opmnctl contains cross reference in it.
Corrective Action: Contact Oracle Support to identify the best course of action.
from node mohpc016
/tp1i/applmgr/fs_ne/EBSapps/log/adop/58/cutover_20151118_201829/Tp1I_mohpc016/ADOPValidations_detailed.log
ERROR: File - /tp1i/applmgr/fs1/FMW_Home/webtier/instances/EBS_web_Tp1I_OHS3/bin/opmnctl contains cross reference in it.
ERROR: File - /tp1i/applmgr/fs1/FMW_Home/webtier/instances/EBS_web_Tp1I_OHS3/bin/opmnctl contains cross reference in it.
ERROR: File - /tp1i/applmgr/fs1/FMW_Home/webtier/instances/EBS_web_Tp1I_OHS3/bin/opmnctl contains cross reference in it
Solution:
-========
Corrected files /tp1i/applmgr/fs1/FMW_Home/webtier/instances/EBS_web_Tp1I_OHS2/bin/opmnctl and
/tp1i/applmgr/fs1/FMW_Home/webtier/instances/EBS_web_Tp1I_OHS3/bin/opmnctl
Which were referencing wrong FS
cd /tp1i/applmgr/fs1/FMW_Home/webtier/instances/EBS_web_Tp1I_OHS2
[aptp1i@mohpc017 <RUN> ]$ diff opmnctl_abkp19nov opmnctl
1c1
< #!/tp1i/applmgr/fs2/FMW_Home/webtier/perl/bin/perl
---
> #!/tp1i/applmgr/fs1/FMW_Home/webtier/perl/bin/perl
25,26c25,26
< $OracleInstance = '/tp1i/applmgr/fs2/FMW_Home/webtier/instances/EBS_web_Tp1I_OHS3';
< $OracleHome = '/tp1i/applmgr/fs2/FMW_Home/webtier';
---
> $OracleInstance = '/tp1i/applmgr/fs1/FMW_Home/webtier/instances/EBS_web_Tp1I_OHS3';
> $OracleHome = '/tp1i/applmgr/fs1/FMW_Home/webtier';
[aptp1i@mohpc017 <RUN> ]$
+ reran cutover
+ FS switch completed , but failed to start services
Summary report for current adop session -
Node mohpc017: FAILED
- Cutover status: FS_CUTOVER_COMPLETED
[UNEXPECTED] Error occurred running "sh /tp1i/inst/fs1/inst/apps/Tp1I_mohpc
017/admin/scripts/adadminsrvctl.sh start -nopromptmsg -silent forcepa
tchfs"
[UNEXPECTED] Error occurred starting Admin Server
[UNEXPECTED] Cutover phase has failed.
Node mohpc016: HALTED
- Cutover status: FS_CUTOVER_COMPLETED
[UNEXPECTED] Detected a failed session on another node(s): mohpc015 and mohgn
pc017
[UNEXPECTED] Execution halted.
Node mohpc015: HALTED
- Cutover status: FS_CUTOVER_COMPLETED
[UNEXPECTED] Detected a failed session on another node(s): mohpc017
[UNEXPECTED] Execution halted.
For more details, run the command: adop -status -detail
[STATEMENT] Please run adopscanlog utility, using the command
"adopscanlog -latest=yes"
to get the list of the log files along with snippet of the error message corresponding to each log file.
adop exiting with status = 2 (Fail)
+ started the services manually
+ restarting cutover
Thanks,
.
+ cutover completed Successfully
Summary report for current adop session -
Node mohpc017: COMPLETED
- Cutover status: COMPLETED
Node mohpc016: COMPLETED
- Cutover status: COMPLETED
Node mohpc015: COMPLETED
- Cutover status: COMPLETED
For more details, run the command: adop -status -detail
adop exiting with status = 0 (Success)
[aptp1i@mohpc017 <RUN> ]$
&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&
.5DQ9WAL tp1i
fs_clone failed
------------------
-----------------------------
ERROR: The following required ports are in use:
-----------------------------
10000 : OHS Administration Proxy Port
6201 : ONS Remote Port
Corrective Action: Free the listed ports and retry the adop operation.
-----------------------------
ERROR: The following required ports are in use:
-----------------------------
10000 : OHS Administration Proxy Port
6201 : ONS Remote Port
Corrective Action: Free the listed ports and retry the adop operation.
Completed execution : ADOPValidations.java
[root@mohpc016 ~]# netstat -anp | grep 10000
tcp 0 0 0.0.0.0:10000 0.0.0.0:* LISTEN 26384/httpd.worker
[root@mohpc016 ~]# ps -ef | grep 26384
aptp1i 26384 27568 0 07:44 ? 00:00:02 /tp1i/applmgr/fs1/FMW_Home/webtier/ohs/bin/httpd.worker -DSSL
[root@mohpc016 ~]# netstat -anp | grep 6201
tcp 0 0 0.0.0.0:6201 0.0.0.0:* LISTEN 27568/opmn
[root@mohpc016 ~]# ps -ef | grep 27568
aptp1i 26384 27568 0 07:44 ? 00:00:02 /tp1i/applmgr/fs1/FMW_Home/webtier/ohs/bin/httpd.worker -DSSL
MT mohpc015
===============
[aptp1i@mohpc017 <RUN> ]$ grep -i ohs_adminport /tp1i/inst/fs1/inst/apps/Tp1I_mohpc015/appl/admin/Tp1I_mohpc015.xml
<ohs_adminport oa_var="s_ohs_adminport" oa_type="PORT" base="9999" step="1" range="-1" label="OHS Administration Proxy Port">9999</ohs_adminport>
[aptp1i@mohpc017 <RUN> ]$ grep -i ohs_adminport /tp1i/inst/fs2/inst/apps/Tp1I_mohpc015/appl/admin/Tp1I_mohpc015.xml
<ohs_adminport oa_var="s_ohs_adminport" oa_type="PORT" base="9999" step="1" range="-1" label="OHS Administration Proxy Port">10000</ohs_adminport>
[aptp1i@mohpc017 <RUN> ]$
[aptp1i@mohpc017 <RUN> ]$ grep -i s_ons_remoteport /tp1i/inst/fs1/inst/apps/Tp1I_mohpc015/appl/admin/Tp1I_mohpc015.xml
<ons_remoteport oa_var="s_ons_remoteport" oa_type="PORT" base="6200" step="1" range="-1" label="ONS Remote Port">6200</ons_remoteport>
[aptp1i@mohpc017 <RUN> ]$ grep -i s_ons_remoteport /tp1i/inst/fs2/inst/apps/Tp1I_mohpc015/appl/admin/Tp1I_mohpc015.xml
<ons_remoteport oa_var="s_ons_remoteport" oa_type="PORT" base="6200" step="1" range="-1" label="ONS Remote Port">6201</ons_remoteport>
MT mohpc016
===============
[aptp1i@mohpc017 <RUN> ]$ grep -i ohs_adminport /tp1i/inst/fs1/inst/apps/Tp1I_mohpc016/appl/admin/Tp1I_mohpc016.xml
<ohs_adminport oa_var="s_ohs_adminport" oa_type="PORT" base="9999" step="1" range="-1" label="OHS Administration Proxy Port">9999</ohs_adminport>
[aptp1i@mohpc017 <RUN> ]$ grep -i ohs_adminport /tp1i/inst/fs2/inst/apps/Tp1I_mohpc016/appl/admin/Tp1I_mohpc016.xml
<ohs_adminport oa_var="s_ohs_adminport" oa_type="PORT" base="9999" step="1" range="-1" label="OHS Administration Proxy Port">10000</ohs_adminport>
[aptp1i@mohpc017 <RUN> ]$
[aptp1i@mohpc017 <RUN> ]$ grep -i s_ons_remoteport /tp1i/inst/fs1/inst/apps/Tp1I_mohpc016/appl/admin/Tp1I_mohpc016.xml
<ons_remoteport oa_var="s_ons_remoteport" oa_type="PORT" base="6200" step="1" range="-1" label="ONS Remote Port">6200</ons_remoteport>
[aptp1i@mohpc017 <RUN> ]$ grep -i s_ons_remoteport /tp1i/inst/fs2/inst/apps/Tp1I_mohpc016/appl/admin/Tp1I_mohpc016.xml
<ons_remoteport oa_var="s_ons_remoteport" oa_type="PORT" base="6200" step="1" range="-1" label="ONS Remote Port">6201</ons_remoteport>
[aptp1i@mohpc017 <RUN> ]$
Findings
=========
on mohpc017
[aptp1i@mohpc017 <RUN> ]$ adopmnctl.sh status -l
You are running adopmnctl.sh version 120.0.12020000.2
Checking status of OPMN managed processes...
Processes in Instance: EBS_web_Tp1I_OHS1
---------------------------------+--------------------+---------+----------+------------+----------+-----------+------
ias-component | process-type | pid | status | uid | memused | uptime | ports
---------------------------------+--------------------+---------+----------+------------+----------+-----------+------
EBS_web_Tp1I | OHS | 54378 | Alive | 321651577 | 4421204 | 24:25:51 | https:4443,https:9999,http:8000
adopmnctl.sh: exiting with status 0
adopmnctl.sh: check the logfile /tp1i/inst/fs1/inst/apps/Tp1I_mohpc017/logs/appl/admin/log/adopmnctl.txt for more information ...
mohpc015
==============
[aptp1i@mohpc017 <RUN> ]$ adopmnctl.sh status -l
You are running adopmnctl.sh version 120.0.12020000.2
Checking status of OPMN managed processes...
Processes in Instance: EBS_web_Tp1I_OHS2
---------------------------------+--------------------+---------+----------+------------+----------+-----------+------
ias-component | process-type | pid | status | uid | memused | uptime | ports
---------------------------------+--------------------+---------+----------+------------+----------+-----------+------
EBS_web_Tp1I | OHS | 46776 | Alive | 1285967872 | 2122628 | 14:03:18 | https:4443,https:10000,http:8000
adopmnctl.sh: exiting with status 0
adopmnctl.sh: check the logfile /tp1i/inst/fs1/inst/apps/Tp1I_mohpc015/logs/appl/admin/log/adopmnctl.txt for more information ..
mohpc016
============
[aptp1i@mohpc017 <RUN> ]$ adopmnctl.sh status -l
You are running adopmnctl.sh version 120.0.12020000.2
Checking status of OPMN managed processes...
Processes in Instance: EBS_web_Tp1I_OHS3
---------------------------------+--------------------+---------+----------+------------+----------+-----------+------
ias-component | process-type | pid | status | uid | memused | uptime | ports
---------------------------------+--------------------+---------+----------+------------+----------+-----------+------
EBS_web_Tp1I | OHS | 26384 | Alive | 1286021582 | 2122620 | 13:28:10 | https:4443,https:10000,http:8000
adopmnctl.sh: exiting with status 0
adopmnctl.sh: check the logfile /tp1i/inst/fs1/inst/apps/Tp1I_mohpc016/logs/appl/admin/log/adopmnctl.txt for more information ...
+ OHS services on node mohpc015 and mohpc016 are started on patch FS ports 10000 and 6201
Action Done:
------------
Made below changes to /tp1i/applmgr/fs1/FMW_Home/user_projects/domains/EBS_domain_Tp1I/opmn/topology.xml
[aptp1i@mohpc017 <RUN> ]$ pwd
/tp1i/applmgr/fs1/FMW_Home/user_projects/domains/EBS_domain_Tp1I/opmn
[aptp1i@mohpc017 <RUN> ]$ diff topology.xml_abkp20nov topology.xml
3,4c3,4
< <ias-instance id="EBS_web_Tp1I_OHS2" oracle-home="/tp1i/applmgr/fs1/FMW_Home/webtier" instance-home="/tp1i/applmgr/fs1/FMW_Home/webtier/instances/EBS_web_Tp1I_OHS2" host="mohpc015.orclsourcing.com" port="6201">
< <ias-component id="EBS_web_Tp1I" type="OHS" mbean-class-name="oracle.ohs.OHSGlobalConfig" mbean-interface-name="oracle.ohs.OHSGlobalConfigMXBean" port="10000">
---
> <ias-instance id="EBS_web_Tp1I_OHS2" oracle-home="/tp1i/applmgr/fs1/FMW_Home/webtier" instance-home="/tp1i/applmgr/fs1/FMW_Home/webtier/instances/EBS_web_Tp1I_OHS2" host="mohpc015.orclsourcing.com" port="6200">
> <ias-component id="EBS_web_Tp1I" type="OHS" mbean-class-name="oracle.ohs.OHSGlobalConfig" mbean-interface-name="oracle.ohs.OHSGlobalConfigMXBean" port="9999">
17,18c17,18
< <ias-instance id="EBS_web_Tp1I_OHS3" oracle-home="/tp1i/applmgr/fs1/FMW_Home/webtier" instance-home="/tp1i/applmgr/fs1/FMW_Home/webtier/instances/EBS_web_Tp1I_OHS3" host="mohpc016.orclsourcing.com" port="6201">
< <ias-component id="EBS_web_Tp1I" type="OHS" mbean-class-name="oracle.ohs.OHSGlobalConfig" mbean-interface-name="oracle.ohs.OHSGlobalConfigMXBean" port="10000">
---
> <ias-instance id="EBS_web_Tp1I_OHS3" oracle-home="/tp1i/applmgr/fs1/FMW_Home/webtier" instance-home="/tp1i/applmgr/fs1/FMW_Home/webtier/instances/EBS_web_Tp1I_OHS3" host="mohpc016.orclsourcing.com" port="6200">
> <ias-component id="EBS_web_Tp1I" type="OHS" mbean-class-name="oracle.ohs.OHSGlobalConfig" mbean-interface-name="oracle.ohs.OHSGlobalConfigMXBean" port="9999">
45c45
< </topology>
\ No newline at end of file
---
> </topology>
[aptp1i@mohpc017 <RUN> ]$
+ Request customer and bounce servers mohpc016 and mohpc015
+ make sure no services are running on ports 10000 and 6201
netstat -anp | grep 10000 -> nothing shd return
netstat -anp | grep 6201 -> nothing shd return
Thanks,
.
&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&
.5F6HGXJ
adop prepare failing
logs
======
-----------------------------
ERROR: The following required ports are in use:
-----------------------------
6803 : Managed Server Port 2
6803 : WLS OAEA Application Port
Corrective Action: Free the listed ports and retry the adop operation.
Completed execution : ADOPValidations.java
RUn FS
[aptiso2i@mohisos010 <RUN> ]$ netstat -an | grep 6803
tcp 0 0 141.143.130.5:6803 0.0.0.0:* LISTEN
tcp 0 0 141.143.130.5:6803 141.143.130.5:10530 ESTABLISHED
tcp 0 0 141.143.130.5:10530 141.143.130.5:6803 ESTABLISHED
[aptiso2i@mohisos010 <RUN> ]$ grep -i 6803 $CONTEXT_FILE
<wls_oaeaport oa_var="s_wls_oaeaport" oa_type="PORT" base="6801" step="1" range="-1" label="WLS OAEA Application Port">6803</wls_oaeaport>
<oaea_server_ports oa_var="s_oaea_server_ports">oaea_server1:6803</oaea_server_ports>
<oaea_nodes oa_var="s_oaea_nodes">mohisos010.orclsourcing.com:6803</oaea_nodes>
[aptiso2i@mohisos010 <RUN> ]$ grep -i 6803 /tiso2i/applmgr/fs2/FMW_Home/user_projects/domains/EBS_domain_TISO2I/config/config.xml
<listen-port>6803</listen-port>
[aptiso2i@mohisos010 <RUN> ]$
+ Patch FS
[aptiso2i@mohisos010 <RUN> ]$ grep -i 6803 /tiso2i/inst/fs1/inst/apps/TISO2I_mohisos010/appl/admin/TISO2I_mohisos010.xml
<wls_oaeaport oa_var="s_wls_oaeaport" oa_type="PORT" base="6801" step="1" range="-1" label="WLS OAEA Application Port">6803</wls_oaeaport>
<oaea_server_ports oa_var="s_oaea_server_ports">oaea_server1:6803</oaea_server_ports>
<oaea_nodes oa_var="s_oaea_nodes">mohisos010.orclsourcing.com:6803</oaea_nodes>
[aptiso2i@mohisos010 <RUN> ]$ grep -i 6803 /tiso2i/applmgr/fs1/FMW_Home/user_projects/domains/EBS_domain_TISO2I/config/config.xml
<listen-port>6803</listen-port>
[aptiso2i@mohisos010 <RUN> ]$
+ Stopped process temporerly on port 6803
+ restart adop prepare
Thanks,
.
&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&
.5FE1J6L TCZB1I
adop prepare failing
error
======
adalnctl.sh: check the logfile /tczb1i/inst/fs2/inst/apps/TCZB1I_mohczbk007/logs/appl/admin/log/adalnctl.txt for more information ...
[EVENT] [END 2015/12/08 00:39:58] Start Apps Listner
[EVENT] [START 2015/12/08 00:39:58] Run the ADZDPATCH concurrent program
[EVENT] [START 2015/12/08 00:40:02] Checking status of the ADZDPATCH concurrent program
[EVENT] Status of ADZDPATCH:
[EVENT] This request is waiting to be processed by the Conflict Resolution
Manager. This request cannot yet begin execution because other requests
may conflict with it. The Conflict Resolution Manager will determine when
this request may begin execution. User SYSADMIN submitted this request on
08-DEC-2015 00:40:01. No action required. This is a normal condition.
[EVENT] [END 2015/12/08 00:40:04] Checking status of the ADZDPATCH concurrent program
[UNEXPECTED]Error while trying to find the status oF ICM
[UNEXPECTED]Error while runConcurrentProgram sub-routine is called.
[ERROR] Prepare phase completed with errors or warnings. Please check logfiles.
Log file: /tczb1i/applmgr/fs_ne/EBSapps/log/adop/16/adop_mohczbk007_20151208_003931.log
Action Plan
============
+ Bringdown CM services
+ Run the adadmin to relink the FND product executables.
+ restart the ICM and retest the issue
ie : run adop phase=prepare
Thanks,
.
Prepare failed again
error from
/tczb1i/applmgr/fs_ne/EBSapps/log/adop/16/prepare_20151208_021005/TCZB1I_mohczbk007/fndsvcrg.log
Cannot complete applications logon. You may have entered an invalid applications password, or there may have been a database connect error.
Internal Concurrent Manager status could not be determined.
SQL> select FND_WEB_SEC.VALIDATE_LOGIN('GUEST','ORACLE') from dual;
FND_WEB_SEC.VALIDATE_LOGIN('GUEST','ORACLE')
--------------------------------------------------------------------------------
Y
SQL> select FND_WEB_SEC.VALIDATE_LOGIN('APPS','D1xMF9pc') from dual;
FND_WEB_SEC.VALIDATE_LOGIN('APPS','W9AOG1LB')
--------------------------------------------------------------------------------
N
+ changed the APPS password
+ changed the SYSADMIN password
+ rerun adop phase=prepare
completed successfully
Thanks,
.
&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&
RFC .5F3PZMD (APPLY DATA-FIX 17983453:R12.AP.C FOR INVOICE_ID = 38471 IN PpCI)
sme update
error log
===========
Validating system setup.
Node registry is valid.
Log: /ppci/inst/fs1/inst/apps/PpCI_mohpc032/logs/appl/rgf/TXK/verifyssh.log
Output: /ppci/inst/fs1/inst/apps/PpCI_mohpc032/logs/appl/rgf/TXK/out.xml
ssh is not enabled for the following nodes
- - - - - - - - - - - - - - - - - - - - - -
mohpc029,mohpc031
[UNEXPECTED]ssh is not enabled.
[UNEXPECTED]Available options are as follows.
[UNEXPECTED]- To enable ssh, run the following command:
perl /ppci/applmgr/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkRunSSHSetup.pl enablessh
-contextfile=/ppci/inst/fs1/inst/apps/PpCI_mohpc032/appl/admin/PpCI_mohpc032.xml
-hosts=mohpc029,mohpc031,mohpc032
Reading /ppci/inst/fs1/inst/apps/PpCI_mohpc032/logs/appl/rgf/TXK/out.xml file
mohpc029=>failed
mohpc031=>failed
Total number of invalid node is 2
[apppci@mohpc031 <RUN> ]$
[apppci@mohpc031 <RUN> ]$ hostname -f
mohpc032.orclsourcing.com
[apppci@mohpc031 <RUN> ]$ ssh mohpc032 date
Thu Dec 3 19:19:07 MST 2015
[apppci@mohpc031 <RUN> ]$ ssh mohpc031 date
Thu Dec 3 19:19:12 MST 2015
[apppci@mohpc031 <RUN> ]$ ssh mohpc029 date
Thu Dec 3 19:19:20 MST 2015
[apppci@mohpc031 <RUN> ]
[apppci@mohpc031 <RUN> ]$ hostname
mohpc029
[apppci@mohpc031 <RUN> ]$ ssh mohpc032 date
Thu Dec 3 19:18:47 MST 2015
[apppci@mohpc031 <RUN> ]$ ssh mohpc031 date
Thu Dec 3 19:18:52 MST 2015
[apppci@mohpc031 <RUN> ]$ ssh mohpc029 date
Thu Dec 3 19:18:58 MST 2015
[apppci@mohpc031 <RUN> ]$ hostname -f
mohpc031.orclsourcing.com
[apppci@mohpc031 <RUN> ]$ ssh mohpc032 date
Thu Dec 3 19:18:16 MST 2015
[apppci@mohpc031 <RUN> ]$ ssh mohpc031 date
Thu Dec 3 19:18:24 MST 2015
[apppci@mohpc031 <RUN> ]$ ssh mohpc029 date
Thu Dec 3 19:18:36 MST 2015
+ need to enable ssh for all MTs
Request SA to enable ssh
perl /ppci/applmgr/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkRunSSHSetup.pl enablessh
-contextfile=/ppci/inst/fs1/inst/apps/PpCI_mohpc032/appl/admin/PpCI_mohpc032.xml
-hosts=mohpc029,mohpc031,mohpc032
Thanks,
.
Customer faced other issue after configuring manually
&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&
decrypt weblogic password
1. Create a script decrypt_password.py in $DOMAIN_HOME/security directory and paste the following code into it:
========================================================================
from weblogic.security.internal import *
from weblogic.security.internal.encryption import *
encryptionService = SerializedSystemIni.getEncryptionService(".")
clearOrEncryptService = ClearOrEncryptedService(encryptionService)
# Take encrypt password from user
pwd = raw_input("Paste encrypted password ({AES}fk9EK...): ")
# Delete unnecessary escape characters
preppwd = pwd.replace("\\", "")
# Display password
print "Decrypted string is: " + clearOrEncryptService.decrypt(preppwd)
========================================================================
.11619910049
2. Set domain environment variables
source $DOMAIN_HOME/bin/setDomainEnv.sh
3. Get encrypted password, in this example from boot.properties file of AdminServer
4. Navigate to $DOMAIN_HOME/security directory and run the following command to start decryption:
cd $DOMAIN_HOME/security
java weblogic.WLST decrypt_password.py
Initializing WebLogic Scripting Tool (WLST) ...
Welcome to WebLogic Server Administration Scripting Shell
Type help() for help on available commands
Please enter encrypted password (Eg. {AES}fk9EK...): {AES}jkIkkdh693dsyLt+DrKUfNcXryuHKLJD76*SXnPqnl5oo\=
Decrypted string is: welcome01
example:
Bfv2rtQs
[apdmsa1c@mohmsan004 <RUN> ]$ pwd
/dmsa1c/applmgr/fs1/FMW_Home/user_projects/domains/EBS_domain_dmsa1c
[apdmsa1c@mohmsan004 <RUN> ]$ cd security/
[apdmsa1c@mohmsan004 <RUN> ]$ vi decrypt_password.py
[apdmsa1c@mohmsan004 <RUN> ]$ java weblogic.WLST decrypt_password.py
Initializing WebLogic Scripting Tool (WLST) ...
Welcome to WebLogic Server Administration Scripting Shell
Type help() for help on available commands
Paste encrypted password ({AES}fk9EK...): {AES}nRwZgIecHhpFdol/YtnExJNbySzalAI+Zm+Hg3maeR8=
Decrypted string is: Bfv2rtQs
[apdmsa1c@mohmsan004 <RUN> ]$
FOR APPS
========
[apdmsa1c@mohmsan004 <RUN> ]$ . ./setDomainEnv.sh
[apdmsa1c@mohmsan004 <RUN> ]$ pwd
/dmsa1c/applmgr/fs1/FMW_Home/user_projects/domains/EBS_domain_dmsa1c
[apdmsa1c@mohmsan004 <RUN> ]$ cd security/
[apdmsa1c@mohmsan004 <RUN> ]$ vi decrypt_password.py
[apdmsa1c@mohmsan004 <RUN> ]$ java weblogic.WLST decrypt_password.py
Initializing WebLogic Scripting Tool (WLST) ...
Welcome to WebLogic Server Administration Scripting Shell
Type help() for help on available commands
Paste encrypted password ({AES}fk9EK...): {AES}nRwZgIecHhpFdol/YtnExJNbySzalAI+Zm+Hg3maeR8=
Decrypted string is: Bfv2rtQs
[apdmsa1c@mohmsan004 <RUN> ]$
&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&
RFC: .59HUODD
ISSUE:
=====
Unable to start CM( fndlibr) processes on secondary pcp node
SOlution:
=========
$ diff sqlnet.ora sqlnet.ora.bak
13c13
< tcp.invited_nodes = (RMOHSAVCR99.orclsourcing.com, mohAVCR598.orclsourcing.com,
mohAVCR596, mohAVCR599.orclsourcing.com)
---
> tcp.invited_nodes = (RMOHSAVCR99.orclsourcing.com, mohAVCR598.orclsourcing.com,
mohAVCR596)
16c16
-> node 599 was not allowed to Net8 connect to node 598
=> adding it to tcp.invited_nodes fixed the issue
&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&
No comments:
Post a Comment