tmp folder 100 percent full in linux

tmp folder 100 percent full in Linux

Applies to:

I am using: 

Red Hat Enterprise Linux 5.5

Oracle Application Install - Version: 11.5.10.0 and later

Description:

I recently came across a problem where the /tmp folder was listed as 100% full:

[root@prodapps01 ~]# df -h

Filesystem            Size  Used Avail Use% Mounted on

/dev/cciss/c0d0p2     9.7G  4.2G  5.1G  46% /

/dev/cciss/c0d0p8     2.0G  152M  1.7G   9% /home

/dev/cciss/c0d0p7     3.9G  3.9G  0M  100% /tmp

/dev/cciss/c0d0p6     7.8G  334M  7.1G   5% /var

/dev/cciss/c0d0p5      12G  3.2G  7.9G  29% /usr

/dev/cciss/c0d0p1     494M   32M  437M   7% /boot

Strangely, there were no files in the /tmp folder. This is how it happens. If you delete files from a folder and they still show up when you check a df -h disk status, then the deleted files are linked to currently running processes. You will have to stop or restart the processes that had those files open in the first place.

lsof | grep /tmp

Output of the above command shown below:

httpd  30027    oracle   57u      REG      104,7 24803361  40 /tmp/filemQMbGZ.TMP (deleted)

httpd  30033    oracle   75u      REG      104,7   773833    44 /tmp/file7s8VAe.TMP (deleted)

httpd  30962    oracle   71u      REG      104,7    72391     88 /tmp/filewzh7U1.TMP (deleted)

Solution:

You will need to kill the httpd processes it shows that are linked to the deleted files which will fix the problem.

To kill the httpd processes

killall -e -9 httpd

Or

service httpd restart

[root@prodapps01 ~]# df -h

Filesystem            Size  Used Avail Use% Mounted on

/dev/cciss/c0d0p2     9.7G  4.2G  5.1G  46% /

/dev/cciss/c0d0p8     2.0G  152M  1.7G   9% /home

/dev/cciss/c0d0p7     3.9G  420M  3.3G  12% /tmp

/dev/cciss/c0d0p6     7.8G  334M  7.1G   5% /var

/dev/cciss/c0d0p5      12G  3.2G  7.9G  29% /usr

/dev/cciss/c0d0p1     494M   32M  437M   7% /boot

Your comments, especially which will help us improve the functionality, will be greatly appreciated :)

unable to stop apache, adapcctl.sh: exiting with status 1

Adapcctl.Sh: Exiting With Status 1

Applies to:

Oracle Applications Technology Stack - Version: 11.5.10.2

Description:

When attempting to stop Apache using adapcctl.sh, the following error occurs:

ERROR
Service can not be stopped using this script

After doing some research work I found that the issue is caused by the following setup:
$IAS_ORACLE_HOME/Apache/Apache/logs/ directory contained both httpd.pid and
apache_runmode.properties
Files httpd.pid and httpd_pls.pid gets created when Apache is started in NORMAL mode.
File apache_runmode.properties gets created when Apache is started in RESTRICTED mode.
As both kind of files are present in Logs directory, adapcctl.sh script thinks Apache cannot be
started in both modes and fails to stop.

Solution:

To implement the solution, please execute the following steps:

1. Stop Apache / Kill all httpd processes if exists any.
2. Backup $IAS_ORACLE_HOME/Apache/Apache/logs
3. Delete all files under $IAS_ORACLE_HOME/Apache/Apache/logs/ directory.
4. Re-start Apache.
5. Retest the issue.

Your comments, especially which will help us improve the functionality, will be greatly appreciated :)

All Receiving Transactions Fail with rvtptcontrol Error

All Receiving Transactions Fail with rvtptcontrol Error

Applies to:

Oracle Application Install - Version: 11.5.10.0 and later.

Oracle Inventory Management - Version: 11.5.1 to 12.1.3

Description:

All Receiving transactions fail with rvtptcontrol. After some research work I found that the error occurs when many transactions fail with same error it is usually because the receiving executables (RVCTP and RCVOLTM) are out of sync.

Solution:

1.      Resolve INVALID Objects

1.1  Set environment

1.2  cd $ORACLE_HOME/rdbms/admin

1.3  sqlplus / as sysdba

1.4  SQL>@utlrp.sql

2.      Relink Receiving executables

2.1  cd $AD_TOP/bin

2.2  sh adrelink.sh force=y ranlib=y "PO RCVOLTM"

2.3  sh adrelink.sh force=y ranlib=y "PO RVCTP"

3        Bounce Receiving Transaction Manager:

3.1  (System Administrator > Concurrent > Manager > Administer)

3.2  Deactivate Receiving Transaction Manager

3.3  ps -ef | grep RCVOLTM (repeat until no processes are running)

3.4  Restart Receiving Transaction Manager

4        Perform transactions and retest the issue

5        If the transaction still fails, bounce the database and then perform another transaction.

6        If bouncing the database or flushing Shared Pool resolves the issue only for a few days (or a few weeks) and bounce database (or flush Shared Pool) must be repeated periodically to resolve the issue, please log a Service Request.

Your comments, especially which will help us improve the functionality, will be greatly appreciated :)

Adcfgclone.pl dbTier Failed: RC-00110

Adcfgclone.pl dbTier Failed: RC-00110

Applies to:

Oracle Application Install - Version: 11.5.10.0 and later.

Description:

On Applications 11.5.10.2, when attempting to clone using rapidclone the following error occurs:

ERROR:
--------
Log file located at ORACLE_HOME/appsutil/log/$SID_$HOSTNAME/ApplyDBTechStack_22472575.log
RC-00110: Fatal: Error occurred while relinking of ApplyDBTechStack
\ 40% completed
ERROR while running Apply...
ERROR: Failed to execute $RDBMS_ORACLE_HOME/appsutil/clone/bin/adclone.pl--

Found the following in Cloning log file:

dest : $ORACLE_HOME/oui/bin/runInstaller.sh
backup : $ORACLE_HOME/oui/bin/runInstaller.sh to
$ORACLE_HOME/appsutil//out/$SID_$HOSTNAME/runInstaller.sh
DEBUG : The destination file does not have write permission
$ORACLE_HOME/oui/bin/runInstaller.sh
DEBUG : Changing the permission of the file failed
AC-00444: Unable to write to file: java.io.FileNotFoundException:
$ORACLE_HOME/oui/bin/runInstaller.sh (Permission denied)
Raised by oracle.apps.ad.autoconfig.InstantiateFile



After doing some research work I found the following solution:

Solution:

To implement the solution, please execute the following steps:

1. Change the permissions on the file to 755:

chmod 755 $ORACLE_HOME/oui/bin/runInstaller.sh

2. Retest the issue

Note:
The new Target Context file should be created prior to this failure so you can rerun the failed command using the following syntax :
perl adcfgclone.pl dbTier /<path to $ORACLE_HOME>/appsutil/<SID>_<hostname>.xml

Your comments, especially which will help us improve the functionality, will be greatly appreciated :)

ORA-19909: Datafile 1 belongs to an orphan incarnation

ORA-19909: Datafile 1 belongs to an orphan incarnation

 Applies to: 

Oracle Database 10gR2

Description:

I have faced the error ORA-19909 while creating physical standby database through RMAN backup. When start recovery MRP process terminated with error.After doing some research work I found the following solution:

Solution:

After some research I have found the following solution:

The Problem occurs because the Primary Database has different incarnation with standby database.

On the Primary site:

Set environment

> rman  target /

RMAN> list incarnation;

using target database control file instead of recovery catalog

List of Database Incarnations

DB Key  Inc Key DB Name  DB ID            STATUS  Reset SCN  Reset Time

------- ------- -------- ---------------- --- ---------- ----------

1       1       PROD     143178490        PARENT  5988067599851 02-OCT-10

2       2       PROD     143178490        CURRENT 5988067601797 02-OCT-10

  

On the Standby site:

> rman  target /

RMAN> list incarnation

2> ;

using target database control file instead of recovery catalog

List of Database Incarnations

DB Key  Inc Key DB Name  DB ID            STATUS  Reset SCN  Reset Time

------- ------- -------- ---------------- --- ---------- ----------

1       1       PROD     143178490        PARENT  5988067599851 02-OCT-10

2       2       PROD     143178490        PARENT  5988067601797 02-OCT-10

3       3       PROD     143178490        CURRENT 3755054101457 11-MAY-11

RMAN> reset database to incarnation 2;

RMAN> exit

Now start recover and monitor alert log file. MRP process started successfully and recovers the archive log files.

Your comments, especially which will help us improve the functionality, will be greatly appreciated :)

Resize Redo Logs on Primary and Standby

RESIZE REDO LOGS ON PRIMARY AND STANDBY

Applies to: 

Oracle Database 10gR2

Description:

Need to increase the size of existing Redo Logs. In current environment there is one physical standby database.

Solution:

SQL> SELECT a.group#, a.member, b.bytes FROM v$logfile a, v$log b WHERE a.group# = b.group#;

SQL> select group#, status from v$log;

    GROUP# STATUS

---------- ----------------

         1 CURRENT

         2 INACTIVE

         3 INACTIVE

SQL> alter system switch logfile;

SQL> alter system switch logfile;

SQL> select group#, status from v$log;

    GROUP# STATUS

---------- ----------------

         1 INACTIVE

         2 INACTIVE

         3 CURRENT

-- To make primary redo log changes on primary and standbys: set

-- STANDBY_FILE_MANAGEMENT to MANUAL so file changes not transferred to standby

SQL> alter system set STANDBY_FILE_MANAGEMENT=MANUAL scope=both;

To stop Redo Apply, issue the following SQL statement on Standby:

SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL;

Drop the redo log group on Primary:

SQL> ALTER DATABASE DROP LOGFILE GROUP 1;

ALTER DATABASE DROP LOGFILE GROUP 1

*

ERROR at line 1:

ORA-01624: log 1 needed for crash recovery of instance ORA920 (thread 1)

ORA-00312: online log 1 thread 1: '<file_name>'

 
Perform a checkpoint to resolve the above issue if occur:

SQL> ALTER SYSTEM CHECKPOINT GLOBAL;

System altered.

SQL> ALTER DATABASE DROP LOGFILE GROUP 1;

Database altered.

Re-create dropped online redo log group on Primary:

Re-create the dropped redo log group with different size (if desired):

SQL> alter database add logfile group 1('/d01/oracle/papsdata/log01a.dbf', '/d01/oracle/papsdata/log01b.dbf') size 100M;

SQL> select group#, status from v$log;

    GROUP# STATUS

---------- ----------------

         1 UNUSED

         2 INACTIVE

         3 CURRENT

SQL> alter system switch logfile;

SQL> select group#, status from v$log;

    GROUP# STATUS

---------- ----------------

         1 CURRENT

         2 INACTIVE

         3 ACTIVE

Repeat the above step to re-create all the redo logs.

Drop Redo Logs on Standby

Check the Status of the Online Redo log Group.

SQL> SELECT GROUP#, STATUS FROM V$LOG;

GROUP# STATUS
---------- ----------------
1 CLEARING_CURRENT
3 CLEARING
2 CLEARING

If Status is CLEARING_CURRENT or CURRENT then you cannot drop Online Redo log Group. You will get ORA-01623 if you try to drop a Redo log Group with Status CLEARING_CURRENT or CURRENT.

For Status CLEARING, UNUSED, INACTIVE please follow below steps.
Clear the Online Redo Log Group:

SQL> ALTER DATABASE CLEAR LOGFILE GROUP 2;

If you have skipped the above step then you will get ORA-01624 while dropping the online redo log Group with Status CLEARING

Drop the Online Redo Log Group:

SQL>ALTER DATABASE DROP LOGFILE GROUP 2; 

Add Redo Logs on Standby

 Add Redo log File Group:

SQL> alter database add logfile group 2 ('/d01/oracle/papsdata/log02a.dbf',  '/d01/oracle/papsdata/log02b.dbf') size 100M reuse;

Database altered.

Add as many Redo Logfile Groups (or Members) you want to add.

Set STANDBY_FILE_MANAGEMENT to AUTO.

SQL> ALTER SYSTEM SET STANDBY_FILE_MANAGEMENT='AUTO' scope=both;

Start Redo Apply (Managed Recovery):

SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT FROM SESSION; 

Size of Redo Logs have been successfully increased on Primary and also on Physical Standby database.

Your comments, especially which will help us improve the functionality, will be greatly appreciated :) 

MRP: Background Media Recovery process shutdown on Physical standby Dataguard

MRP: Background Media Recovery process shutdown on Physical standby Dataguard

Applies to: 

Oracle Database 10gR2

Description:

RFS process successfully running, archive logs successfully shipped from Primary server to standby, but archive logs did not apply. MRP process going to shutdown automatically. After doing some research work I found the following solution:

Solution:

SQL> alter database recover managed standby database cancel;

SQL> alter database recover managed standby database cancel;

alter database recover managed standby database cancel

*

ERROR at line 1:

ORA-16136: Managed Standby Recovery not active

SQL> alter database recover managed standby database disconnect from session;

Alert log:

alter database recover managed standby database disconnect from session

Sat Jun  9 09:51:56 2012

Attempt to start background Managed Standby Recovery process (PROD)

MRP0 started with pid=18, OS id=6591

Sat Jun  9 09:51:56 2012

MRP0: Background Managed Standby Recovery process started (PROD)

Managed Standby Recovery not using Real Time Apply

MRP0: Background Media Recovery terminated with error 1111

Sat Jun  9 09:52:01 2012

Errors in file /d01/oracle/proddb/10.2.0/admin/PROD_pebsdbdrrt/bdump/prod_mrp0_6591.trc:

ORA-01111: name for data file 59 is unknown - rename to correct file

ORA-01110: data file 59: '/d01/oracle/proddb/10.2.0/dbs/UNNAMED00059'

ORA-01157: cannot identify/lock data file 59 - see DBWR trace file

ORA-01111: name for data file 59 is unknown - rename to correct file

ORA-01110: data file 59: '/d01/oracle/proddb/10.2.0/dbs/UNNAMED00059'

Sat Jun  9 09:52:01 2012

Errors in file /d01/oracle/proddb/10.2.0/admin/PROD_pebsdbdrrt/bdump/prod_mrp0_6591.trc:

ORA-01111: name for data file 59 is unknown - rename to correct file

ORA-01110: data file 59: '/d01/oracle/proddb/10.2.0/dbs/UNNAMED00059'

ORA-01157: cannot identify/lock data file 59 - see DBWR trace file

ORA-01111: name for data file 59 is unknown - rename to correct file

ORA-01110: data file 59: '/d01/oracle/proddb/10.2.0/dbs/UNNAMED00059'

Sat Jun  9 09:52:01 2012

MRP0: Background Media Recovery process shutdown (PROD)

Sat Jun  9 09:52:02 2012

Completed: alter database recover managed standby database disconnect from session

SQL> select thread#,max(sequence#) from v$log_history group by thread#;

   THREAD# MAX(SEQUENCE#)

---------- --------------

248980

SQL> select severity, error_code,message,to_char(timestamp,'DD-MON-YYYY HH24:MI:SS') from v$dataguard_status;

SQL> archive log list

SQL> ho ls -ltra /archive/logs/

SQL> select process,status,sequence# from v$managed_standby;

PROCESS   STATUS        SEQUENCE#

--------- ------------ ----------

ARCH      CONNECTED             0

ARCH      CONNECTED             0

RFS       IDLE                  0

RFS       IDLE                  0

SQL> ho ps -ef|grep mrp

oracle   30350 30330  0 09:24 pts/2    00:00:00 /bin/bash -c ps -ef|grep mrp

oracle   30352 30350  0 09:24 pts/2    00:00:00 grep mrp

SQL> select file#,name from v$datafile where name like '%UNNAMED%';

     FILE#

----------

NAME

--------------------------------------------------------------------------------

        59

/d01/oracle/proddb/10.2.0/dbs/UNNAMED00059

SQL> alter database create datafile '/d01/oracle/proddb/10.2.0/dbs/UNNAMED00059' AS '/data3/proddata/daw_INV_06.dbf';

SQL> alter database recover managed standby database cancel;

alter database recover managed standby database cancel

*

ERROR at line 1:

ORA-16136: Managed Standby Recovery not active

SQL> alter database recover managed standby database disconnect from session;

Database altered.

SQL> select thread#,max(sequence#) from v$log_history group by thread#;

   THREAD# MAX(SEQUENCE#)

---------- --------------

         1         248980

SQL> /

   THREAD# MAX(SEQUENCE#)

---------- --------------

         1         248981

Problem has been resolved, now MRP process started and recovering the archive logs.

Your comments, especially which will help us improve the functionality, will be greatly appreciated :)