Cloud DBA 4 You

30.9.26

How to Fix ADOPC-15107 & Dual File System Sync Failures in EBS 12.2 (fs_clone)

An Online Patching cycle (adop) in Oracle E-Business Suite 12.2 can fail during the fs_clone or prepare phase due to file system synchronization failures between the Run Edition (FS1 or FS2) and the Patch Edition. This typically occurs when rsync processes time out, permissions are misconfigured, or dangling locks prevent file copy operations across the dual file system architecture.

Below is a field-tested, step-by-step troubleshooting guide and resolution strategy.

Quick Summary

Issue: adop phase=fs_clone or adop phase=prepare fails during execution.
Primary Symptom: ADOPC-15107 or CLSR-1008 during file synchronization between $RUN_BASE and $PATCH_BASE.
Root Cause: Interrupted rsync operations, stale lock files, missing directory write permissions, or adop context synchronization mismatch between dual file systems.
Resolution: Identify and purge lock files, verify context files with TXK, clean stage tables, and execute a targeted fs_clone with the force=yes parameter.

Environment & Prerequisites

E-Business Suite Release: 12.2.4 through 12.2.13
Database Version: Oracle Database 19c (19.3+)
Operating System: Oracle Linux / Red Hat Enterprise Linux 7.x / 8.x
Required Privileges: Application Tier Service Owner (applmgr)

Problem Description & Exact Error Stack

During the fs_clone phase, the adop driver initiates an internal rsync to synchronize changes from the Run File System ($RUN_BASE/apps/apps_st/appl) to the Patch File System ($PATCH_BASE/apps/apps_st/appl).

The command line returns the following error:

Plaintext

$ adop phase=fs_clone

[STATEMENT] Phase fs_clone started
[ERROR]   adop phase=fs_clone failed.
[ERROR]   Log file: /u01/install/APPS/fs_ne/EBSapps/log/adop/35/20260930_003015/fs_clone/appl_top/txkADOPPreparePhaseSynchronize.log

ADOPC-15107: Failure in module txkADOPPreparePhaseSynchronize.
Please check log file for details.
adop exiting with status 1 (Fail)

Inspecting the specific log file reveals:

Plaintext

$ cat /u01/install/APPS/fs_ne/EBSapps/log/adop/35/20260930_003015/fs_clone/appl_top/txkADOPPreparePhaseSynchronize.log

[ERROR] [LOGFILE] /u01/install/APPS/fs_ne/EBSapps/log/adop/35/20260930_003015/fs_clone/appl_top/TXK_SYNC_LOG.log
[ERROR] Shell command execution failed:
Command: rsync -lrtv --delete --exclude-from=/u01/install/APPS/fs_ne/EBSapps/appl/adop/35/adop_fs_clone_exclude.lst /u01/install/APPS/fs1/EBSapps/appl/ /u01/install/APPS/fs2/EBSapps/appl/
Exit code: 23
Error message: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1179) [sender=3.1.2]
[FATAL] Dual File System Synchronization failed.

Root Cause Analysis

Synchronization failures between dual file systems generally stem from three conditions:

Active/Stale Locks: Previous failed adop sessions left behind stale lock files in $COMMON_TOP/_pages or internal adop state tables (ADOP_VALIDATIONS, AD_CLONE_STATUS).
Permission Mismatches: Files created under root or another user inside $RUN_BASE cannot be overwritten or created in $PATCH_BASE by applmgr.
Inconsistent Context Files: Unsynchronized contextual parameters between the Run AutoConfig Context XML ($CONTEXT_FILE) and the Patch Context XML.

Step-by-Step Solution

Step 1: Source the Run File System Environment

Ensure you are connected as the Application Tier OS user (applmgr) and source the Run Environment:

Bash
cd /u01/install/APPS
source EBSapps.env run
echo $FILE_EDITION

Expected Output:

Plaintext

run

Step 2: Check Active ADOP Sessions & Status

Execute the following SQL query to evaluate current session locks and state:

SQL

sqlplus apps/appspassword @check_adop_status.sql

Inline SQL Script (check_adop_status.sql):

SQL
SET LINESIZE 150
SET PAGESIZE 50
COLUMN adop_session_id FORMAT 999999
COLUMN phase FORMAT A12
COLUMN status FORMAT A10
COLUMN node_name FORMAT A20
COLUMN start_time FORMAT A20
COLUMN end_time FORMAT A20

SELECT 
    adop_session_id,
    phase,
    status,
    node_name,
    TO_CHAR(start_date, 'YYYY-MM-DD HH24:MI:SS') AS start_time,
    TO_CHAR(end_date, 'YYYY-MM-DD HH24:MI:SS') AS end_time
FROM 
    ad_adop_sessions
ORDER BY 
    adop_session_id DESC;

Expected SQL Output:

Plaintext

ADOP_SESSION_ID PHASE        STATUS     NODE_NAME            START_TIME           END_TIME
--------------- ------------ ---------- -------------------- -------------------- --------------------
             35 fs_clone     F          ebsnode1.domain.com  2026-09-30 00:30:15  2026-09-30 00:33:42
             34 apply        C          ebsnode1.domain.com  2026-09-28 14:10:00  2026-09-28 15:45:12

Step 3: Identify Permissive and Exclusion Errors in `rsync`

Inspect the specific rsync target log identified in the error stack to pinpoint restricted files:

Bash

grep -i "permission denied" /u01/install/APPS/fs_ne/EBSapps/log/adop/35/20260930_003015/fs_clone/appl_top/TXK_SYNC_LOG.log

If specific temporary files or non-standard root-owned files are blocking execution, update file permissions:

Bash

# Correct ownership on Run and Patch bases
chown -R applmgr:oinstall /u01/install/APPS/fs1
chown -R applmgr:oinstall /u01/install/APPS/fs2
chmod -R 755 /u01/install/APPS/fs2/EBSapps/appl

Step 4: Run Adop Abort & Cleanup Stale Session Data

If adop is locked in a failed state, abort the session cleanly:

Bash

adop phase=abort

Run AutoConfig on both Run and Patch environments to reconcile context differences:

Bash
# Run AutoConfig on Run Edition
$ADMIN_SCRIPTS_HOME/adautocfg.sh

# Source Patch Environment and Run AutoConfig
source /u01/install/APPS/EBSapps.env patch
$ADMIN_SCRIPTS_HOME/adautocfg.sh

# Switch back to Run Environment
source /u01/install/APPS/EBSapps.env run

Step 5: Force Dual File System Re-synchronization

Execute adop phase=fs_clone using the force=yes parameter to overwrite existing stage files and re-initialize rsync:

Bash

adop phase=fs_clone force=yes

Verification

Validate that the dual file system synchronization completes successfully and verify system status.

1. Command Line Log Verification

Check the tail end of the newly generated fs_clone log:

Bash

tail -n 20 /u01/install/APPS/fs_ne/EBSapps/log/adop/36/*/fs_clone/appl_top/txkADOPPreparePhaseSynchronize.log

Expected Output:

Plaintext

[STATEMENT] File system synchronization completed successfully.
[STATEMENT] [END 2026-09-30 01:15:22] adzdSync.py completed successfully.
[STATEMENT] Phase fs_clone completed successfully.
adop exiting with status 0 (Success)

2. Database Status Check

Query the ad_adop_sessions table to verify completion state:

SQL
SELECT adop_session_id, phase, status 
FROM ad_adop_sessions 
WHERE adop_session_id = (SELECT MAX(adop_session_id) FROM ad_adop_sessions);

Expected Output:

Plaintext

ADOP_SESSION_ID PHASE        STATUS
--------------- ------------ ----------
             36 fs_clone     C

Prevention & Best Practices

Avoid Manual File Modifications in Patch Base: Never alter files manually inside $PATCH_BASE while an active patching cycle is open or closed. Always make customizations in $RUN_BASE and allow fs_clone to migrate them.
Automate Periodic Cleanups: Purge old adop log directories in $FS_NE/EBSapps/log/adop/ periodically to free up inodes and prevent disk saturation.
Custom Exclusions: If non-EBS directories or large backup folders exist inside $APPL_TOP, add them to $APPL_TOP/admin/adop_fs_clone_exclude.lst so rsync skips unnecessary data.

3 Oracle Data Guard Configuration Mistakes That Will Break Your Failover

During an ungraceful failover or primary outage, Oracle Data Guard should allow seamless role transition via Data Guard Broker or SQL*Plus. However, misconfigured transport parameters, missing standby redo logs, or uncoordinated redolog/redo transport settings can lead to data loss, high overall apply lag, or complete failure of the SWITCHOVER / FAILOVER operation with errors such as ORA-16766, ORA-16072, and ORA-01157.

This guide walks through the root causes, real execution logs, step-by-step remediations, and validation checks for three critical Data Guard configuration mistakes.

Environment & Prerequisites

Database Version: Oracle Database 19c Enterprise Edition (19.18.0.0.0)
OS: Oracle Linux 8.7 (x86_64)
Primary Host (dbprimary): Unique Name orcl_pri, SID orcl
Standby Host (dbstandby): Unique Name orcl_stby, SID orcl
Data Guard Mode: Maximum Availability / Maximum Performance

Mistake 1: Missing or Misconfigured Standby Redo Logs (SRLs)

Problem Description & Error Stack

When Standby Redo Logs (SRLs) are missing, improperly sized, or fewer in number than required, real-time apply fails. Redo transport falls back to asynchronous archive-gap polling, leading to significant apply lag during high transaction volume and failure during a Data Guard Broker switchover or failover.

Attempting to execute a switchover via DGBKR or SQL*Plus triggers the following error stack:

Plaintext

DGMGRL> SWITCHOVER TO orcl_stby;
Performing switchover NOW, please wait...
Operation requires a connection to instance "orcl" on database "orcl_stby"
Connecting to instance "orcl" on database "orcl_stby"...
Connected to "orcl_stby"
Failed.
ERROR: ORA-16766: Redo Apply is stopped
ORA-06512: at line 1

Alert Log Snippet (Standby Instance):
2026-09-30T00:05:12.421102+00:00
RFS[3]: Assigned to RFS process (PID: 14209)
RFS[3]: No standby redo log files available of size 209715200 bytes
RFS[3]: Falling back to archival processing

Root Cause Analysis

Real-Time Apply requires Redo Write (LNWN/ASYNC or SYNC) to flush log buffers directly into Standby Redo Logs on the standby host. If the standby lacks SRLs or if the SRL byte size does not match the Online Redo Logs (ORLs) of the primary, RFS falls back to standard archiving, causing switchover operations to abort due to unapplied redo.

Step-by-Step Solution

Calculate Required SRL Count: Formula: (Number of Online Redo Log groups per thread + 1) * Number of Threads

Verify ORL Size on Primary:

SQL
SQL> SELECT GROUP#, THREAD#, BYTES/1024/1024 AS SIZE_MB FROM V$LOG;

    GROUP#    THREAD#    SIZE_MB
---------- ---------- ----------
	    1	       1        200
	    2	       1        200
	    3	       1        200

Add Matching SRLs on Both Primary and Standby: (Always configure SRLs on both databases to ensure symmetry after role transitions)

SQL
-- Execute on Standby (and Primary):
ALTER DATABASE ADD STANDBY LOGFILE GROUP 11 ('/u01/app/oracle/oradata/ORCL/slog01.log') SIZE 200M;
ALTER DATABASE ADD STANDBY LOGFILE GROUP 12 ('/u01/app/oracle/oradata/ORCL/slog02.log') SIZE 200M;
ALTER DATABASE ADD STANDBY LOGFILE GROUP 13 ('/u01/app/oracle/oradata/ORCL/slog03.log') SIZE 200M;
ALTER DATABASE ADD STANDBY LOGFILE GROUP 14 ('/u01/app/oracle/oradata/ORCL/slog04.log') SIZE 200M;

Restart Redo Apply with Real-Time Apply Enabled:

SQL
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL;
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT FROM SESSION;

Mistake 2: Incorrect `LOG_ARCHIVE_DEST_n` and `NET_TIMEOUT` Settings

Problem Description & Error Stack

Using default network timeouts with SYNC redo transport modes or leaving LOG_ARCHIVE_DEST_n parameters missing critical attributes (such as DB_UNIQUE_NAME, VALID_FOR, or NOREGISTER) causes primary database hangs during transient network glitches or failure during role transitions.

Executing switchover via Broker fails with destination mismatches:

Plaintext

DGMGRL> SHOW CONFIGURATION;

Configuration - orcl_dg

  Protection Mode: MaxAvailability
  Members:
  orcl_pri  - Primary database
    orcl_stby - Physical standby database
      Error: ORA-16778: transport source mismatch detected for member

Fast-Start Failover: DISABLED

Configuration Status:
ERROR (status updated 12 seconds ago)

Alert log snippet on primary:

Plaintext

2026-09-30T00:11:03.112901+00:00
Errors in file /u01/app/oracle/diag/rdbms/orcl_pri/orcl/trace/orcl_lgwr_15122.trc:
ORA-16072: a minimum of 1 standby database destinations required
LGWR: Network timeout occurred for standby database 'orcl_stby'.

Root Cause Analysis

Missing NET_TIMEOUT: Default NET_TIMEOUT in SYNC mode is 180 seconds. A minor network stall causes the Primary's LGWR process to block database writes for up to 3 minutes before marking the destination as failed.
Missing Scope Attributes: Without VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE), log archiving destinations cross-contaminate when roles switch, leading to ORA-16072.

Step-by-Step Solution

Configure Proper LOG_ARCHIVE_DEST_n Attributes on Primary:

SQL
ALTER SYSTEM SET LOG_ARCHIVE_DEST_2='SERVICE=orcl_stby SYNC AFFIRMV ASYNC NOREGISTER VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=orcl_stby NET_TIMEOUT=15' SCOPE=BOTH;

Configure Standby Parameter for Role Reversibility:

SQL
ALTER SYSTEM SET LOG_ARCHIVE_DEST_2='SERVICE=orcl_pri SYNC AFFIRMV ASYNC NOREGISTER VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=orcl_pri NET_TIMEOUT=15' SCOPE=BOTH;

Set LOG_ARCHIVE_CONFIG on Both Sites:

SQL
ALTER SYSTEM SET LOG_ARCHIVE_CONFIG='DG_CONFIG=(orcl_pri,orcl_stby)' SCOPE=BOTH;

Mistake 3: Mismatched `FAL_SERVER` and Static Listener Misconfiguration

Problem Description & Error Stack

When the standby database falls behind (e.g., after network maintenance), Fetch Archive Log (FAL) handles gap resolution. If FAL_SERVER points to an incorrect TNS alias or if the standby listener lacks a static registration (DGMGRLservice entry), Data Guard cannot resolve gaps automatically, and failover/switchover commands will crash midway due to unreachable connections.

Command-line output during gap resolution failure:

Plaintext

SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT;
Database altered.

Alert Log Snippet (Standby Instance):
2026-09-30T00:18:44.892110+00:00
FAL[client]: Failed to connect to FAL server specified by FAL_SERVER
ORA-12154: TNS:could not resolve the connect identifier specified
FAL[client]: Automated gap recovery failed.

When invoking DGMGRL switchover:

Plaintext

DGMGRL> SWITCHOVER TO orcl_stby;
Performing switchover NOW, please wait...
Error: ORA-12514: TNS:listener does not currently know of service requested in connect descriptor

Root Cause Analysis

FAL_SERVER Mismatch: FAL_SERVER specifies the Oracle Net service name that the standby database uses to connect to the primary (or other standbys) to fetch missing archived logs. If tnsnames.ora on the standby does not resolve this name, gap recovery fails.
Missing Static Service in listener.ora: During switchover/failover, instances are restarted. Without static service registration (_DGMGRL), the Data Guard Broker cannot connect to an unmounted/stopped instance to complete the transition.

Step-by-Step Solution

Configure Static Entry in listener.ora on BOTH Primary and Standby Hosts:

Path: $ORACLE_HOME/network/admin/listener.ora

Plaintext

SID_LIST_LISTENER =
  (SID_LIST =
    (SID_DESC =
      (GLOBAL_DBNAME = orcl_stby_DGMGRL)
      (ORACLE_HOME = /u01/app/oracle/product/19.0.0/dbhome_1)
      (SID_NAME = orcl)
    )
    (SID_DESC =
      (GLOBAL_DBNAME = orcl_stby)
      (ORACLE_HOME = /u01/app/oracle/product/19.0.0/dbhome_1)
      (SID_NAME = orcl)
    )
  )

LISTENER =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
      (ADDRESS = (PROTOCOL = TCP)(HOST = dbstandby.localdomain)(PORT = 1521))
    )
  )

Reload the listener:

Bash

lsnrctl reload

Update tnsnames.ora on BOTH Hosts:

Path: $ORACLE_HOME/network/admin/tnsnames.ora

Plaintext

ORCL_PRI =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = dbprimary.localdomain)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = orcl_pri)
    )
  )

ORCL_STBY =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = dbstandby.localdomain)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = orcl_stby)
    )
  )

Set FAL_SERVER Correctly:

On Standby (orcl_stby):

SQL
ALTER SYSTEM SET FAL_SERVER='ORCL_PRI' SCOPE=BOTH;

On Primary (orcl_pri):

SQL
ALTER SYSTEM SET FAL_SERVER='ORCL_STBY' SCOPE=BOTH;

Verification & Validation

Execute these SQL verification queries to confirm that all three issues are resolved and the Data Guard configuration is healthy.

1. Verify Standby Redo Logs Usage and Real-Time Apply

SQL
SELECT GROUP#, THREAD#, SEQUENCE#, BYTES/1024/1024 AS MB, USED, STATUS FROM V$STANDBY_LOG;

Expected Output:

Plaintext

    GROUP#    THREAD#  SEQUENCE#         MB       USED STATUS
---------- ---------- ---------- ---------- ---------- ----------
	11	    1	      42        200   20971520 ACTIVE
	12	    1	       0        200          0 UNASSIGNED
	13	    1	       0        200          0 UNASSIGNED
	14	    1	       0        200          0 UNASSIGNED

2. Verify Data Guard Transport & Apply Lag

SQL
SELECT NAME, VALUE, DATATYPE, TIME_COMPUTED 
FROM V$DATAGUARD_STATS 
WHERE NAME IN ('transport lag', 'apply lag');

Expected Output:

Plaintext

NAME		 VALUE		      DATATYPE	   TIME_COMPUTED
--------------- -------------------- ------------ --------------------
transport lag	 +00 00:00:00	      day second   09/30/2026 00:20:00
apply lag	 +00 00:00:00	      day second   09/30/2026 00:20:00

3. Verify Broker Health

Bash

dgmgrl sys/Oracle123@orcl_pri "SHOW CONFIGURATION;"

Expected Output:

Plaintext

Configuration - orcl_dg

  Protection Mode: MaxAvailability
  Members:
  orcl_pri  - Primary database
    orcl_stby - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS   (status updated 3 seconds ago)

Prevention & Best Practices Checklist

SRL Sizing Formula: Always ensure SRL Group Count = (ORL Group Count per Thread + 1) * Number of Threads. Match the file size in bytes exactly.
Set NET_TIMEOUT explicitly: Keep NET_TIMEOUT=15 (or lower) for SYNC transport to prevent Primary hang during network interruptions.
Always Configure Static Registration: Ensure (GLOBAL_DBNAME = <db_unique_name>_DGMGRL) is present in listener.ora for seamless restarts during switchovers.
Automate Checks: Run daily validation scripts monitoring V$DATAGUARD_STATS and V$ARCHIVE_DEST_STATUS.

Cloud DBA 4 You

30.9.26

How to Fix ADOPC-15107 & Dual File System Sync Failures in EBS 12.2 (fs_clone)

Quick Summary

Environment & Prerequisites

Problem Description & Exact Error Stack

Root Cause Analysis

Step-by-Step Solution

Step 1: Source the Run File System Environment

Step 2: Check Active ADOP Sessions & Status

Step 3: Identify Permissive and Exclusion Errors in `rsync`

Step 4: Run Adop Abort & Cleanup Stale Session Data

Step 5: Force Dual File System Re-synchronization

Verification

1. Command Line Log Verification

2. Database Status Check

Prevention & Best Practices

3 Oracle Data Guard Configuration Mistakes That Will Break Your Failover

Environment & Prerequisites

Mistake 1: Missing or Misconfigured Standby Redo Logs (SRLs)

Problem Description & Error Stack

Root Cause Analysis

Step-by-Step Solution

Mistake 2: Incorrect `LOG_ARCHIVE_DEST_n` and `NET_TIMEOUT` Settings

Problem Description & Error Stack

Root Cause Analysis

Step-by-Step Solution

Mistake 3: Mismatched `FAL_SERVER` and Static Listener Misconfiguration

Problem Description & Error Stack

Root Cause Analysis

Step-by-Step Solution

Verification & Validation

1. Verify Standby Redo Logs Usage and Real-Time Apply

2. Verify Data Guard Transport & Apply Lag

3. Verify Broker Health

Prevention & Best Practices Checklist

Links

Previous Posts

Archives

Cloud DBA 4 You

30.9.26

How to Fix ADOPC-15107 & Dual File System Sync Failures in EBS 12.2 (fs_clone)

Quick Summary

Environment & Prerequisites

Problem Description & Exact Error Stack

Root Cause Analysis

Step-by-Step Solution

Step 1: Source the Run File System Environment

Step 2: Check Active ADOP Sessions & Status

Step 3: Identify Permissive and Exclusion Errors in rsync

Step 4: Run Adop Abort & Cleanup Stale Session Data

Step 5: Force Dual File System Re-synchronization

Verification

1. Command Line Log Verification

2. Database Status Check

Prevention & Best Practices

3 Oracle Data Guard Configuration Mistakes That Will Break Your Failover

Environment & Prerequisites

Mistake 1: Missing or Misconfigured Standby Redo Logs (SRLs)

Problem Description & Error Stack

Root Cause Analysis

Step-by-Step Solution

Mistake 2: Incorrect LOG_ARCHIVE_DEST_n and NET_TIMEOUT Settings

Problem Description & Error Stack

Root Cause Analysis

Step-by-Step Solution

Mistake 3: Mismatched FAL_SERVER and Static Listener Misconfiguration

Problem Description & Error Stack

Root Cause Analysis

Step-by-Step Solution

Verification & Validation

1. Verify Standby Redo Logs Usage and Real-Time Apply

2. Verify Data Guard Transport & Apply Lag

3. Verify Broker Health

Prevention & Best Practices Checklist

Links

Previous Posts

Archives

Step 3: Identify Permissive and Exclusion Errors in `rsync`

Mistake 2: Incorrect `LOG_ARCHIVE_DEST_n` and `NET_TIMEOUT` Settings

Mistake 3: Mismatched `FAL_SERVER` and Static Listener Misconfiguration