24.9.26

How to Fix ORA-01555: Snapshot Too Old in Oracle (Diagnostic & Prevention Guide)

 Quick Summary: ORA-01555 occurs when a long-running query requires consistent read data from undo segments, but that undo data has already been overwritten. This guide covers how to identify the failing query, trace undo tablespace pressure, and resolve the issue through UNDO tuning and query optimization.

1. Environment & Prerequisites

  • Database Versions: Oracle 11g, 12c, 18c, 19c, 23ai

  • OS Platform: Linux / Unix / Windows

  • Privileges Required: SYSDBA or SELECT_CATALOG_ROLE

2. Problem Description & Error Stack

During long batch jobs, night reporting runs, or data export tasks (expdp), the executing session fails and writes an entry to the alert log:

Plaintext
ORA-01555: snapshot too old: rollback segment number 12 with name "_SYSSMU12_123456789$" too small
ORA-02063: preceding line from REMOTE_DB

In the application log or SQL*Plus session, you will see:

Plaintext
SQL> SELECT * FROM large_orders_mv;
ERROR at line 1:
ORA-01555: snapshot too old: rollback segment number 8 with name "_SYSSMU8_342129038$" too small

3. Root Cause Analysis

Oracle uses Read Consistency (Multi-Version Concurrency Control / MVCC). When Query A starts at 10:00 AM, Oracle guarantees that Query A will only see data as it existed at 10:00 AM.

If Query B updates and commits rows at 10:05 AM, the old image of those rows moves to the UNDO tablespace. If Query A reaches those updated blocks at 10:30 AM, it must go to UNDO to reconstruct the 10:00 AM version.

ORA-01555 happens when:

  1. UNDO retention expired: The time elapsed exceeded UNDO_RETENTION, and Oracle reused those undo blocks for newer transactions.

  2. Delayed Block Cleanout: A massive uncommitted transaction modified blocks, and a subsequent SELECT query tried to clean them out long after the undo logs disappeared.

  3. Fetch-Across-Commit: A PL/SQL cursor loops over a dataset, executes UPDATE, and calls COMMIT inside the loop.

4. Step-by-Step Solution & Diagnostics

Step 1: Identify the Failing Query & Duration

First, check V$UNDOSTAT to see when the undo tablespace experienced high usage or retention steals around the time of the error.

SQL
SELECT 
    TO_CHAR(begin_time, 'YYYY-MM-DD HH24:MI') AS start_time,
    TO_CHAR(end_time, 'YYYY-MM-DD HH24:MI') AS end_time,
    ssolderrcnt AS snapshot_too_old_errors,
    nospaceerrcnt AS no_space_errors,
    maxquerylen AS max_query_seconds,
    tuned_undoretention AS auto_tuned_retention
FROM v$undostat
WHERE ssolderrcnt > 0
ORDER BY begin_time DESC;

If maxquerylen is larger than tuned_undoretention, your query simply takes longer to run than the historical undo data is retained.

Step 2: Check Your UNDO Tablespace Configuration

Check the current size, auto-extension settings, and parameter values:

SQL
SHOW PARAMETER undo;

Example Output:

Plaintext
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
undo_management                      string      AUTO
undo_retention                       integer     900
undo_tablespace                      string      UNDOTBS1

Next, verify if your UNDO tablespace is configured with fixed retention guarantees:

SQL
SELECT tablespace_name, retention FROM dba_tablespaces WHERE contents = 'UNDO';

Step 3: Apply the Fix

Depending on the underlying cause identified in your environment, apply one of the three solutions below.

Option A: Increase UNDO Retention and Space (Most Common)

If your batch query runs for 2 hours (7,200 seconds), set UNDO_RETENTION higher than the longest query duration:

SQL
-- Increase retention target to 4 hours (14,400 seconds)
ALTER SYSTEM SET undo_retention = 14400 SCOPE=BOTH;

Ensure the UNDO datafile has enough space to hold 4 hours of transaction history:

SQL
-- Allow UNDO tablespace to grow automatically if disk space permits
ALTER DATABASE DATAFILE '/u01/app/oracle/oradata/ORCL/undotbs01.dbf' 
AUTOEXTEND ON NEXT 100M MAXSIZE 30G;

Option B: Enforce Retention Guarantee

By default, Oracle can overwrite unexpired UNDO blocks if the UNDO tablespace runs out of space. Force Oracle to protect undo data for the full duration of UNDO_RETENTION:

SQL
ALTER TABLESPACE undotbs1 RETENTION GUARANTEE;

Warning: If UNDO space runs out with RETENTION GUARANTEE enabled, new DML transactions (INSERT/UPDATE/DELETE) will fail with ORA-30036 instead of queries failing with ORA-01555. Use this only when you have sufficient disk space.

Option C: Fix PL/SQL "Fetch-Across-Commit" Anti-Pattern

If the error occurs inside a PL/SQL procedure, inspect the code for commits inside a cursor loop:

Incorrect Code Pattern:

SQL
-- DO NOT DO THIS
FOR r IN (SELECT id, status FROM orders WHERE status = 'PENDING') LOOP
    UPDATE orders SET status = 'PROCESSED' WHERE id = r.id;
    COMMIT; -- <--- This breaks the cursor's read consistency over time!
END LOOP;

Correct Code Pattern (Bulk Processing):

SQL
-- Recommended Approach: Use Bulk Collect & FORALL
DECLARE
    TYPE t_order_ids IS TABLE OF orders.id%TYPE;
    v_ids t_order_ids;
BEGIN
    SELECT id BULK COLLECT INTO v_ids 
    FROM orders 
    WHERE status = 'PENDING';

    FORALL i IN 1..v_ids.COUNT
        UPDATE orders SET status = 'PROCESSED' WHERE id = v_ids(i);
        
    COMMIT; -- Commit ONCE after the batch is processed
END;
/

5. Verification

After increasing UNDO_RETENTION or refactoring the PL/SQL code, re-execute the target process and monitor V$UNDOSTATin real time:

SQL
SELECT 
    TO_CHAR(begin_time, 'HH24:MI:SS') AS sample_time,
    maxquerylen,
    tuned_undoretention,
    ssolderrcnt
FROM v$undostat
WHERE begin_time > SYSDATE - (1/24);

Expected Result: ssolderrcnt remains 0, and tuned_undoretention stays higher than maxquerylen.

6. Sizing Formula & Prevention

To calculate the exact UNDO size required for your workload, use this formula:

Run this query during peak load hours to calculate recommended sizing:

SQL
SELECT 
    ((UR * UPS * 8192) / (1024 * 1024)) AS recommended_undo_mb
FROM 
    (SELECT value AS UR FROM v$parameter WHERE name = 'undo_retention'),
    (SELECT MAX(undoblks/600) AS UPS FROM v$undostat);

Best Practices:

  • Keep statistics updated (DBMS_STATS.GATHER_TABLE_STATS) on large tables to prevent long-running queries caused by bad execution plans (FTS instead of Index Scans).

  • Never COMMIT inside a row-by-row cursor loop.

  • Monitor long-running queries via V$SESSION_LONGOPS.



Labels: , ,

11.9.26

Resolving RMAN-06059: Expected Archived Log Not Found. Repairing Broken Backup Pipelines

Labels: RMAN-06059, Archived Log Missing, Broken Backup, Log Deletion

1. Root Cause Identification

I opened the terminal node and verified the failure log. The RMAN output channel text was explicitly clear: RMAN-06059: expected archived log not found, loss of archived log compromises recoverability. This happens when a junior admin cleans out physical logs from the OS level using a raw 'rm' command instead of clearing them through the recovery catalog registry dashboard layout. This creates an immediate compliance gap and triggers a severe production downtime business insurance risk. To check the exact baseline state of our archived sequences, I jumped into the console interface and executed a full diagnostic crosscheck trace mapping:

RMAN> CROSSCHECK ARCHIVELOG ALL;

2. Immediate Failsafe Resolution

The crosscheck utility quickly flagged the deleted files as 'EXPIRED' inside the local control file metadata structure. To clean up the system repositories and let the main data streaming engines continue without locking up, I executed a safe catalog purge command directly on the active node profile:

RMAN> DELETE EXPIRED ARCHIVELOG ALL;

3. Long-Term Prevention Parameters

To avoid this problem from breaking our database backup data recovery services pipeline next month, we added a strict validation wrapper block to our automated bash shell scripts. This ensures a crosscheck runs automatically before any data copy triggers. This process keeps our enterprise cloud infrastructure optimization costs highly efficient and acts as a solid oracle licensing compliance audit defense mechanism by maintaining verified system trace histories. Always use RMAN to clean your file systems!

Labels: , , , , ,

7.7.26

Troubleshooting ORA-01555 Snapshot Too Old Errors in Oracle EBS

One of our finance users came to me today complaining that a critical end-of-month report kept failing halfway through. Looking at the request log, the culprit was obvious: ORA-01555: snapshot too old: rollback segment number with name... too small.

1. Understanding the Failure

This error simply means a long-running query needed to see old data blocks for consistency, but those blocks were overwritten in the Undo tablespace before the query could finish. To find out exactly how long the query was running and check our baseline retention times, I ran this diagnostic check:
SELECT tuned_undoretention, maxquerylen, undoblks FROM v$undostat;

2. The Live Production Fix

The maximum query length was heavily exceeding our default undo retention window parameters. To stop this from killing long finance reports, I dynamically extended our data parameters and scaled up our space bounds directly on the live database server node:
ALTER SYSTEM SET undo_retention=10800 SCOPE=BOTH;
ALTER DATABASE DATAFILE '/u01/oradata/prod/undo01.dbf' RESIZE 10G;

Setting `undo_retention` to 10800 forces Oracle to hold onto historical database undo blocks for a minimum of 3 hours. The user re-ran the processing transaction, and it completed successfully without a single snapshot dropout. Keep an eye on your monthly undo sizing!

Labels: , ,

6.7.26

How I Fixed a Critical ORA-04031 Error on EBS R12.2 Production

I ran into a brutal production cluster lockup on our Oracle E-Business Suite R12.2 environment today. Users were reporting sudden connection dropouts.

1. Root Cause Identification

I jumped onto the database node and checked the main alert log at:
$DIAG_HOME/diag/rdbms/ebsprod/ebsprod/trace/alert_ebsprod.log

The log was filled with a critical error: ORA-04031: unable to allocate 4096 bytes of shared memory. To verify the space fragmentation inside the reserved memory pool, my team ran this query:
SELECT name, free_space, request_failures FROM v$shared_pool_reserved;

2. Emergency Temporary Patch

The request failures counter was climbing rapidly. I applied a temporary patch to clean out the memory fragmentation:
ALTER SYSTEM FLUSH SHARED_POOL;

Warning: Doing this causes a brief, noticeable CPU spike for a few minutes while the system re-parses incoming SQL commands.

3. Long-Term Prevention Parameters

To fix this permanently, we updated our initialization parameter profile bounds to scale up the allocations safely:
ALTER SYSTEM SET shared_pool_size=4G SCOPE=SPFILE;
ALTER SYSTEM SET shared_pool_reserved_size=512M SCOPE=SPFILE;

Monitor your allocations closely!

Fixing Concurrent Manager Crashes After EBS R12.2 Cloning

We finished a rapid clone of our Oracle EBS R12.2 instance last night, but the Concurrent Managers refused to come up. Every time we kicked off the startup scripts, the ICM (Internal Concurrent Manager) immediately went into a deactivated status.

1. Checking the Real Logs

Instead of guessing, I went straight to the application tier diagnostic logs. The ICM log layout is found under your specific log directories:
$APPLCSF/$APPLLOG/NAME_MMDD.mgr

Inside, the log clearly stated that it could not initialize due to old node configurations stuck in the database layout tables. To clear out the stale configuration entries from the previous environment, I logged into SQL*Plus as the APPS user and ran the clean scripts:
EXEC FND_CONC_CLONE.SETUP_CLEAN;

2. Regenerating the Environment

Once the setup cleanup command executed successfully, I had to run Autoconfig to completely rebuild the system profile values and directory layouts:
sh $ADMIN_SCRIPTS_HOME/adautocfg.sh

After Autoconfig finished with a successful status code 0, I brought the managers back online using the standard control utility line script:
sh $ADMIN_SCRIPTS_HOME/adcmctl.sh start apps/apps_password

The Internal Concurrent Manager caught the correct database tables instantly and stayed up. Always remember to clear out stale configuration contexts post-clone!

Labels:

21.6.26

How to Fix Performance Degradation in Oracle EBS Concurrent Managers

 When Oracle E-Business Suite (EBS) performance degrades, the root cause is frequently found within the Concurrent Processing (CP) subsystem. If users are complaining that routine reports are stuck in "Pending" status, invoices are processing at a snail's pace, or your night batch jobs are bleeding into business hours, your Concurrent Managers are hitting a bottleneck.

Fixing this issue requires looking beyond general database tuning. You must optimize how the Concurrent Managers handle workload distribution, queue definitions, and internal purging.
In this comprehensive guide, we will break down the exact steps to diagnose and remediate performance degradation in Oracle EBS Concurrent Managers.
Phase 1: Diagnosing the Performance Bottleneck
Before tweaking settings, you must identify whether the slowdown is caused by system resource starvation, queue misconfigurations, or database contention.
1. Check for Queue Backlogs
Log in to Oracle Applications as the System Administrator responsibility and navigate to:
Concurrent > Manager > Administer
Look at the Target versus Actual process counts. If a manager's Target process count is higher than its Actual count, that manager is failing to launch internal workers, causing jobs to queue up indefinitely.
2. Run the Concurrent Manager Status Script
Run the standard Oracle-provided script from the application tier ($FND_TOP/sql/afcmstat.sql) to view a live status report of your managers, their queues, and active processes.
3. Analyze the Concurrent Manager Log Files
Review the Internal Concurrent Manager (ICM) log file found in the $APPLCSF/$APPLLOG directory (usually named ICM*.mgr). Look for repeating error messages, memory allocation failures, or timeout warnings like ALERTER: Max continuous errors exceeded.
Phase 2: Core Reasons for Performance Degradation
Through thousands of real-world DBA cases, performance drops in Concurrent Managers usually boil down to these four critical issues:
  • Bloated FND Tables: The underlying tracking tables (FND_CONCURRENT_REQUESTS and FND_CONCURRENT_PROCESSES) have grown to millions of rows, slowing down the internal queries the ICM runs to fetch the next job.
  • Improper Cache Size Configuration: If a manager's Cache Size is set too low, it frequently queries the database to grab new requests, causing high database contention.
  • Suboptimal Worker Allocations: Standard managers (like the Standard Manager or Conflict Resolution Manager) do not have enough target processes allocated to handle sudden spikes in user request volume.
  • Database Level Wait Events: Concurrent processes are being blocked at the database layer by locks, slow I/O, or high CPU utilization.
Phase 3: Step-by-Step Performance Tuning Solutions
Follow these systematic solutions to restore speed to your Oracle EBS Concurrent Processing environment.
Step 1: Clean and Purge Concurrent Request Tables (The #1 Fix)
The single most effective action a DBA can take is cleaning up the historical tracking tables.
  1. Schedule the standard concurrent program: Purge Concurrent Request and/or Manager Data.
  2. Run it weekly or nightly with the parameter Criterion = Age and Days = 7 or 30 (depending on your business retention requirements).
  3. If the tables are already massively bloated, running the purge program might hang. In that case, look into Oracle Support Note 213824.1 for manual truncation and rebuild scripts for FND_CONCURRENT_REQUESTS.
Step 2: Optimize Manager Definitions (Target Processes & Cache)
Adjusting the way queues process requests relieves massive software bottlenecks:
  1. Navigate to Concurrent > Manager > Define.
  2. Query the Standard Manager.
  3. Click on Work Shifts.
  4. Increase Target Processes: Raise the number of workers to match your server's hardware capability. (e.g., if it is set to 4, increase it to 8 or 12).
  5. Adjust Cache Size: Increase the Cache Size parameter to 10 or 20. This allows the manager to look ahead and cache multiple requests in memory at once, dramatically reducing queries to the database.
Step 3: Isolate Heavy Reports Using Dedicated Managers
If a few specific, massive custom reports (like heavy financial or inventory extracts) are locking up the Standard Manager queue, isolate them:
  1. Create a Custom Concurrent Manager dedicated only to those heavy programs.
  2. Assign specialization rules to the Standard Manager to Exclude those specific programs.
  3. Assign specialization rules to your Custom Manager to Include only those programs.
  4. This ensures that a single massive report will never freeze the entire business operations queue.
Step 4: Reorganize and Reindex FND Tables
Because FND_CONCURRENT_REQUESTS experiences constant INSERT, UPDATE, and DELETE actions daily, its indexes become highly fragmented.
  • Coordinate a maintenance window to rebuild indexes on FND_CONCURRENT_REQUESTS and FND_CONCURRENT_PROCESSES.
  • Gather fresh schema statistics using the program: Gather Schema Statistics for the APPLSYS schema.

Conclusion
Tuning Oracle EBS Concurrent Managers is not a one-time task; it requires proactive maintenance. By establishing a rigid automated purging schedule, resizing your target processes to fit your hardware capabilities, and isolating heavy custom code from your standard flows, you will drastically decrease request queue times and improve system responsiveness.

Labels: , , , , , , ,