SAP Note

Header Data

Symptom

You need to know concrete commands to check the whole database, single database objects (like tables, indexes, tablespaces, data files etc.) or backups for corruption. What are SAPs general recommendations concerning Oracle database consistency check procedures? For more information about consistency checks please refer to Note 540463.

Other Terms

ora-1578, ora-1499, ora-1498, ora-1410, ora-600 [12700], ora-8103

Reason and Prerequisites

Database block corruptions can occur due to failures in all technical system components like the operating system, hardequipment and the like. Such corruptions (as they are not caused by a faulty block modification by the database) may stay in the database unnoticed, because they affect blocks that aren't accessed for a long time. Since backups are usually only available for a specific retention time, it may be that the corruptions cannot be fixed anymore by using a recovery and thus data can be lost. Therefore it's extremely important to know for sure that the backups taken don't contain any corruption early enough, that is, before the retention time for the backups has expired. Also it's important to know as early as possible whether or not any corrupt blocks exist in the database and need to be recovered.

Solution

This note presents several methods to check the database and the backups for consistency. Also it contains recommendations on

l Regular consistency checks of the database

l Check strategy in case a corruption has been found in the database

l What check methods to use for the consistency check

l General hints for the analysis of the check results

This note is divided in the following chapters:

l Super-short recommendations - for those who just want to do the right things correctly.

l General recommendations - for those who want to review the recommendations and adapt their check strategy themselves.

l Detailed description of the analysis methods

l General hints and tips

Be aware: this note is all about finding corruptions. It does not cover what to do when a corruption has been found. For this information please see note 365481.

SUPER-SHORT-Recommendation: ===========================

¡ Run a full database consistency check at least once during a backup cycle. Use the transaction DB13 to Schedule this  ("Validate Structure", "Verify database").

¡ Perform consistency checks of the database backups! It is far more important to be sure that the backups are OK than to check the productive

    23345 - Consistency check of ORACLE database

Version   67     Validity: 19.07.2013 - active   Language   English

Released On 19.07.2013 11:15:48

Release Status Released for Customer

Component BC-DB-ORA Oracle

Priority Recommendations / Additional Info

Category Help for error analysis

database!

¡ Use RMAN for database backups and for the consistency check.  This is easily possible even when the backups had not been created with RMAN with the brbackup-option #-w only_rmw#.

¡ Make sure that DB Parameter db_block_checksum is set to TRUE or TYPICAL (starting with Oracle 11g), to improve the recognition of block(see note 923919).

¡ Use the BR*TOOLS for all the consistency check (brconnect, brbackup). Don#t perform the checks manually if this can be avoided.

¡ In case a corruption is found, please check THOROUGLY what the next steps in order to handle this corruption are (see note 365481 on that).

¡ DON#T just RECOVER the database if you don#t have an explicit recommendation for that from the SAP Oracle support!In many cases, unnecessary restore/recovery actions lead to big problems and data loss.

¡ Do open a support message at SAP in case there are any issues or open questions concerning database corruptions on application component BC

Recommendation 1 - "Regular check of the database" ================================================== SAP and Oracle highly recommend to regularly check your database and the backups of the database for consistency.Ideally the database backups should be restored to a separate hardware/storage and checked there with each of the following three check methods (all of them explained in this note):

¡ RMAN validate

¡ ANALYZE TABLE VALIDATE STRUCTURE and

¡ FULL EXPORT TO NULL

This check would prove that the backups can be successfully restored and recovered and that they don't contain any corruption. Also this allows a good estimation of the time expected for a full database restore/recovery. Also, this method takes away the additional system load from the production system. In case this separate check is not possible, the following checks should be considered as a minimum requirement: - Once per backup cycle a full check with RMAN and ANALYZE. The whole idea of this regular check is to find the corruptions before the backups with the corruption Since the check runs allocate system resources and may impact the overall system performance it's advisable to perform them at nonare any) of the system. - Once per backup cycle perform a restore/recovery check of the last backup. Recommendation 2 - "Check when a corruption has occurred on the system" =======================================================================

In case you face corruptions in the database that occurred unexpectedly (e.g. NOT because of the recovery of NOLOGGING index blocks) you should perform the following analysis steps in order to find out about the full extent of the corruption issue. Since the corruption checks will take quite some time as all stored data in the database will be read at least once, make sure to start the activities as soon as possible after the first corruption error message occurred. Some of the checks can and should be run in parallel to save time (which may save unnecessary downtime as well).Checks that can be run in parallel get the same letter in the following list (therefore, start all checks marked with "A" first and in parallel and continue with the checks marked with "B" afterwards): A.1. Perform an full RMAN consistency check: brbackup -u / -m all -w only_rmv  -e <parallel degree> -c If this does not work for some reason, use DBV (either via brbackup with "-w only_dbv" switch or manually) instead. A.2. Recover the last database backup to a separate location/server and perform a check of the  database there.Make sure to start this activity immediately if  you face a critical production down situation due to corruptions! B2. Perform a full ANALYZE consistency check: brconnect  -u / -c -f stats -v cascade -t all -e null -p <number> B2.5 While the ANALYZE check runs, start to find out which objects are affected by the corrupt blocks found by the RMAN/DBV check from step A.Use brconnect -u / -c -f lscorr or the SQL script "Which segment is stored in the corrupt blocks (RMAN)?" for this (see below). This is absolutely mandatory to properly understand the extent of damages and to correctly estimate whether or not the restore/recovery of a backup will be required or not. If the corrupt blocks belong to tables, table partitions, LOB-segments or LOB-indexes then a recovery may become necessary very likely!

Recommendation 3 - "Usage of tools" ===================================

SAP strongly recommends to use the BR*TOOLS to employ the check methods. This will allow for extended logging, build-in parallelization and eliminates the need to know the exact command syntax for the Oracle tools.Make sure to have the latest BR*TOOLS version installed, as this may help to save a lot of time in case you're currently facing a corruption.

Description of check methods ============================

Checkmethod ANALYZE  TABLE ... VALIDATE STRUCTURE Overview The analyze command checks tables and dependent indexes and also the references between indexes and tables.During the analysis the table is locked against changes up to Oracle 8.x. Starting with Oracle 9i it's possible to use the ONLINE option to prevent the permanent locks: ANALYZE TABLE "<owner>"."<table_name>"  VALIDATE STRUCTURE CASCADE ONLINE; Anyhow, for a short moment, the ANALYZE needs to acquire a lock. If this fails (e.g. because of some currently running INSERT/UPDATE/DELETE command an error is returned: "ORA-00054: resource busy and acquire with NOWAIT specified". In this case, retry the analyze command after the lock-holding transaction is finished. BRCONNECT automatically uses the ONLINE option! If you get a "ORA-14508: specified VALIDATE INTO table not found" while running ANALYZE , make sure to have the utilityrunning the Oracle catalog script $ORACLE_HOME/rdbms/admin/utlvalid.sql. This table is required for analyzing e.g. partitioned tables. See note 514178 for details on this. About the checks ANALYZE performs pre-defined consistency checks on Table- and Index-Entries. These checks include e.g. Index-table-reference consistency and row-length checks. ANALYZE can only be run when the database is open and in read/write mode, since the INVALID_ROWS table needs to be writeable.Since the checks are done based on rows, only used blocks of tables and indexes are checked. Data of LOB-Segments (that is, LOB-Data stored in a separate segment) is not checked by ANALYZE. To check this data use the EXPORT check method with 'exp' or starting with 11g 'expdp' command. The ANALYZE commands aborts with an error message as soon as the first corruption has been detected in a segment. This makes it hard to find out about the total number of corrupt blocks, therefore it#s highly recommended to run additional checks with RMAN/DBV. Verify the complete database To check the complete database, all tables of all users need to be checked. For that there are two options available: a) Check via BRCONNECT or b) Via a SQL*PLUS script. Both options simply run "ANALYZE TABLE/INDEX ... VALIDATE STRUCTURE [CASCADE]"  for every table and are therefore equal concerning the check result. a) Check via BRCONNECT (recommended) To use BRCONNECT for the ANALYZE check, make sure that these prerequisites are fullfilled: - Use the most current BR*TOOLS version that can be used with your database! -BRCONNECT version  6.10 PL 46/6.20 PL 15 or higher have to be used. See note 12741 for details. For SAP Kernels lower than 6.20 you can also use the 6.20 BRCONNECT! - for Oracle 9 only: DB Parameter O7_DICTIONARY_ACCESSIBILITY = TRUE has to be set To start the ANALYZE for the complete database run following command: brconnect  -u / -c -f stats -v cascade -t all -e null -p <number> '-p <number>' is the number of Oracle tables that are checked in parallel. Usually <number> can be chosen to be equal to the number of logical CPUs on the database server. The '-e null' flag ensures that no tables are excluded from the check. Any errors found will be logged in the brconnect log file (/oracle/<SID>/sapcheck/*.vst). In newer Oracle releases you may also find information about corruptions in the Alert.log file (see example below), so check this as well. Should you find any errors in there, that point out corruptions, handle these as described in note  365481. In case the BRCONNECT run is interrupted, it's possible to resume it with the -k (complete)  flag: brconnect  -u / -c -f stats -v cascade -t all -e null -p <number> -k See note 1235952 for details on this. b) Check via SQL*Plus script (to be used when brconnect cannot be employed) Download the script attached to this note (choose the right script for your database release!) to a working folder as 'analyze_complete_db_<n>.sql' and run it as follows: Oracle <=8.1: sqlplus "/ as sysdba" @analyze_complete_db_8.sql Oracle >=9.2: sqlplus "/ as sysdba" @analyze_complete_db_9.sql A log file 'analyze_cascade.log' will be written out to the current folder. Should you find any errors in there, that point out corruptions, handle these as described in note 365481.Please do also check the Alert.log in case of reported errors, as in more recent versions of Oracle additional information (e.g. what object, object type etc.) is affected. Example: [...] Fri Jul  9 16:30:54 2010 Hex dump of (file 42, block 77370) in trace file /oracle/EP2/saptrace/usertrace/ep2_ora_14134.trc Corrupt block relative dba: 0x0a812e3a (file 42, block 77370) Bad header found during buffer read Data in bad block: type: 0 format: 0 rdba: 0x00000000 last change scn: 0x0001.00000000 seq: 0x63 flg: 0x7d spare1: 0x0 spare2: 0x0 spare3: 0x0 consistency value in tail: 0x00000000 check value in block header: 0x0 computed block checksum: 0xc3 Reread of rdba: 0x0a812e3a (file 42, block 77370) found same corrupted data Fri Jul  9 16:33:07 2010

Corrupt Block Found          TSN = 6, TSNAME = PSAPBTABD          RFN = 42, BLK = 77370, RDBA = 176238138          OBJN = 130388, OBJD = 710473, OBJECT = EDISCDOC, SUBOBJECT =          SEGMENT OWNER = SAPR3, SEGMENT TYPE = Table Segment Fri Jul  9 16:38:45 2010 [...] Single segment check It's also possible to check just single segments, like a single table or a single index. Single segment check via BRCONNECT: To run a single segment check via brconnect, call it like this: brconnect -u / -c -f stats -t <table_name> -v cascade Single segment check via SQL command: Logon to sqlplus as SYSDBA and run the following command: Oracle <=8.1: analyze table "<OWNER>"."<TABLENAME>" validate structure cascade; Oracle >=9.2: analyze table "<OWNER>"."<TABLENAME>" validate structure cascade online; Should you find any errors in there that point out corruptions handle these as described in note 365481.It's also possible to just check indexes or single partitions which may save a lot of time in some cases. Please refer to the Oracle documentation for the full ANALYZE command reference. Additional Remarks Certain errors like ORA-8103 or ORA-1499 can only be detected by the ANALYZE-command. This makes the ANALYZE check more thorough compared with RMAN/DBV or Export but also implies a longer runtime. The ANALYZE command can only be executed, when the database is in OPEN state and it can only be run on segments that are stored in ONLINE tablespaces. For standby-databases it's possible to run the ANALYZE command when the standby-database is opened in READ ONLY mode. See note 817253 for details on this.During such a read only mode ANALYZE, error messages like the following may be given out: [...] BR0301W SQL error -1552 in thread 3 at location brc_dblog_write-5, SQL statement: 'INSERT INTO SAP_SDBAD (BEG, FUNCT, POS, LINE) VALUES ('20100528140042', 'vst' [...] These can be ignored as these are only BRCONNECT logging information that cannot be written into the CCMS tables. These error messages don't have any meaning concerning the database consistency. Recommendation The ANALYZE check should ideally run at least once during the backup cycle. If this is somehow not feasible to implement, at least check the objects located in tablespace SYSTEM and SYSAUX.The objects in these tablespaces usually cannot be recreated in case of a error and a time consuming recreation of the database (possibly with a full export/import of all application data)  may  become necessary should there be no good backup for these tablespaces.A repair of these segments is not supported (not even the rebuild of an index)! To run this check use brconnect as described in note 914174: brconnect -u / -c -f stats -t oradict_tab -v If the brconnect you're using does not support this command, you may as well run the attached sql-script "analyze_oracle_dictionary.txt". Rename it to "analyze_oracle_dictionary.sql" and call sqlplus like this: sqlplus "/ as sysdba" @analyze_oracle_dictionary.sql Afterwards make sure to check the spooled output "analyze_cascade.log" for any errors. Further information Oracle 11g provides an additional FAST-option to the analyze command: ANALYZE TABLE <table_name> VALIDATE STRUCTURE CASCADE FAST; Refer to the Oracle 11g sql command reference documentation for details. (http://download.oracle.com/docs/cd/E11882_01/server.112/e10592/statements_4005.htm#SQLRF53680) Currently SAP does not recommend to use this feature.

Checkmethod RMAN

Overview RMAN is the recommended tool for backup and restore/recovery of Oracle databases. It can backup and restore Oracle data files as well as archivelog files. In addition to the backup/recovery functionality RMAN provides the options to check Oracle files for consistency errors.To run the checks, RMAN uses ORACLE server processes just as standard client sessions do. This allows the usage of the same IO mechanisms (like DirectIO) that are used during standard DB operations. RMAN also facilitates multiblock IO for the consistency checks which gives a huge performance benefit compared with checks done by DBV. Additional advantages of RMAN compared to DBV are the frictionless check run during database online operations and the far better check result reporting options via database views (V$DATABASE_BLOCK_CORRUPTION). RMAN also automatically performs several checks when the database files are backed up with it. This eliminates the requirement of running a separate DBV check run against the database files. About the checks RMAN checks database blocks for technical consistency, that is, it is checked whether the block data structure complies with the expected data structure. For that the blocks are read via multiblock-IO comparable to a segment scan (e.g. full table scan or index fast full scan) and compared against the block structure model.. If there is a checksum stored with the blocks (which is the case when the parameter db_block_checksum was set to TRUE the last time when the block had been

written out to disk), this checksum is also checked and compared to the stored checksum. This checksum enables the check for data corruptions that are not within the block structure but within the data payload of the block.Therefore it's highly recommended to have the block checksum feature enabled (which is the default from Oracle 10g onwards). See note 923919 for further details on this. RMAN can check Oracle data files and archive log files that belong to the current database. It's also possible to check files that don't belong to the current database (e.g. files that have been restored from a backup).This is also true for files that haven't been backed up by RMAN before. That means: regardless whether RMAN is usually used for backup it's possible to employ it for consistency checks for backup files!Until (and including) Oracle 10g RMAN could not check online redolog-, control-, parameter- or wallet- Starting with Oracle 11g it's possible to check control files and server parameter files (SPFILEs) for consistency as well.Depending on the RMAN command used, either all blocks of a data file are read and checked (BACKUP AS COPY) or just those blocks below the data files high water mark (these are the blocks that had been used at least once). The latter case is the default and the blocks left out are just those that never had any data stored in them.RMAN, just like DBV, can only check consistency on block level. Checks for data consistency spawning multiple blocks (e.g. whether segment/tables freespace management is correct or whether indexes contain the correct pointers to table entries) are not possible with RMAN. Check the database with RMAN a) via BRBACKUP (recommended) To run a RMAN consistency check of the database with the BR*TOOLS you may use the following command: brbackup -u / -m all -w only_rmv  -e <parallel degree> -c Again, the '-e' flag specifies the parallel degree of this action. To limit the check to specific tablespaces or data files, change the -m flag accordingly, e.g. put in the tablespace name or the data file number. You can even specify a list of data files (e.g. -m 1,23,34 to check data files 1 and 23 and 34). b) via RMAN Since brbackup simply calls RMAN to execute the check it's also possible to run the check directly via RMAN. To do so execute the following commands: rman nocatalog RMAN> connect target / RMAN> backup check logical validate ( database ); To check single files or tablespaces you can alter the command e.g. like this: RMAN> backup check logical validate ( datafile 1,23,34 ); Or RMAN> backup check logical validate ( tablespace 'SYSAUX' ); RMAN stores information about found block corruptions in the control file of the database. You can access these information via the database view V$DATABASE_BLOCK_CORRUPTION. When the check is run via BRBACKUP the found block corruptions will be printed out automatically and saved in the log file o BRBACKUP.To review the corrupt block list via sqlplus you can use the query "Which object is assigned to the corrupt blocks (RMAN)?" (see below).As of note 1464091 (BR*TOOLS 7.20) the BR*TOOLS provide this information automatically. From BR*TOOLS 7.2 onwards, it's possible to display the list of corrupted blocks from V$DATABASE_BLOCK_CORRUPTION via: brconnect -u / -c -f lscorr Also, the BR*TOOLS check whether the corruption is beyond the segment high water mark, which is important to know, since those block corruptions usually can be ignored or at least don't mean immediate data loss. As of Oracle 11g RMAN also returns a summary check report to the command prompt for every checked data file after the check run.Additional Remarks Should the results of DBV and RMAN differ, usually the RMAN results should be preferred. Further Information Oracle Metalink 471716.1 - 11g New Feature V$DATABASE_BLOCK_CORRUPTION Enhancements and Rman Validate Command

Checkmethod DBVerify (DBV)

Overview DBV allows the block consistency check from "outside" of the database. It had been developed as a offlinedatabase process or even without a database at all. If run against files currently opened by a database instance it's possible that block corruptions are reported that aren't actually present. This effect can occur when the same block is modified by the Oracle process while DBV is checking it. Therefore, should corruptions be reported, the DBV check should be repeated to avoid false positives. In case the corruption is still reported, the check should be done against an OFFLINE file. As of Oracle 9i DBV can be used against opened data files also on Windows platforms. DBV should be used only when the check with RMAN is not possible, e.g. because the database cannot be mounted anymore. DBV check runs are usually much slower than RMAN check runs but don't provide additional or better check functions. As result report DBV provides an output similar to the following for each data file checked: DBVERIFY: Release 10.2.0.4.0 - Production on Thu Jul 16 13:19:30 2009 Copyright (c) 1982, 2007, Oracle.  All rights reserved. DBVERIFY - Verification starting : FILE = TDB.DATA1 DBVERIFY - Verification complete Total Pages Examined         : 8960 Total Pages Processed (Data) : 1184 Total Pages Failing   (Data) : 0                        * Total Pages Processed (Index): 2473 Total Pages Failing   (Index): 0                        *

Total Pages Processed (Other): 5298 Total Pages Processed (Seg)  : 0 Total Pages Failing   (Seg)  : 0                        * Total Pages Empty            : 5 Total Pages Marked Corrupt   : 0                        * Total Pages Influx           : 0 Highest block SCN            : 3282557 (0.3282557) It's important that in every line marked with '*' zero (0) blocks are reported! When used via BRBACKUP you will see the output above only when corruptions had been found. If you take the database backups via RMAN then block checks are already performed during the backup run. In this case you may omit separate DBV check runs for the database. About the checks Just as RMAN does, DBV checks the technical consistency of Oracle data file blocks. To do that the file is read block by block and compared against a reference block structure build into DBV. DBV also supports checksums and checks these for correctness if present. Unlike RMAN DBV cannot check redolog-, archivelog-, control-, parameter-, wallet- or RMAN-backup files.DBV only can check oracle data files. Due to the block wise check the same limitations to the checks apply as with RMAN. Only block-internal consistency can be checked. Dependencies spanning multiple blocks cannot be checked. Check the database with DBV For a complete database check all data files need to be checked. This can be done either by manually calling DBV manually for each data file or by using BRBACKUP. a) BRBACKUP command to check the complete database in OFFLINE mode: brbackup -w only_dbv -c -t offline -m all -e <parallel degree> ATTENTION:  For the DBV run via BRBACKUP it's necessary that the database can actually be started and opened without problems.BRBACKUP does this to get the full paths for the data files that should be checked. This is done even when the check mode OFFLINE is chosen!In case the database cannot be opened due to a problem, DBV needs to be started manually. b) BRBACKUP command to check the complete database ONLINE: brbackup -w only_dbv -c -t online -m all #e <parallel degree> The results of the DBV runs can be reviewed in the brbackup log files: UNIX: $ORACLE_HOME/sapbackup/b*.dbv WINDOWS: <DRIVE>:\oracle\<SID>\sapbackup\b*.dbv Should you find any errors in these log files, that point out corruptions, handle these as described in note 365481. c) DBV check without BRBACKUP In case the BRBACKUP tools cannot be used, DBV can be started manually. Ensure that the "logfile=<logfile name>" parameter is given for every run so that corruptions found are properly documented and can be reviewed afterwards. To generate the DBV commands for all data files the following SQL statement can be used (this statement can be executed even if the database can only be startup up to MOUNT state): set linesize 200 set pagesize 0 select 'DBV file='||name||' blocksize='||block_size ||' logfile=DBV_LOG_TS_'||ts#||'-FILE_'||file#||'.txt' from v$datafile; More detailed information on DBV and how to combine database backups with automatic DBV checks can be found in the online documentation for BRBACKUP. Single segment check DBV provides a feature to restrict the block check to a single segment. Compared to the ANALYZE check this option offers a better runtime and the check of all blocks of the segment, regardless whether already corruptions have been found or not. A use-case for this feature would be to check all blocks of a certain segment where ANALYZE already found one corruption. Also this DBV check would check blocks allocated by the segment but not yet used ones (beyond the segment high water mark). Use the SQL script "create DBV command for single segment checks" to run a single segment check with DBV. Additional Remarks DBV opens the data files with the default access options, which means it does not use file system-features like DirectIO, ConcurrentIO etc.Since DBV reads all blocks directly from disk, it may happen that these blocks are read through the file system cache of the operating system. This may in turn lead to memory shortage and paging on the database server. The additional IO of DBV can also lead to a reduced IO performance of the Oracle server processes. Therefore, in case you use DBV against a up and running database instance, closely monitor the database performance during the DBV run.

Export of the database

Overview The idea for this kind of consistency check is to try to simply read all data from a table, including LONG/LOB columns. Should this succeed a corruption may not affect the current data but only lead to problems when data should be changed or added. However, as it has been observed that successfully imported data was reported corrupt after an import, this check method can only serve as an addition to the ANALYZE and RMAN/DBV checks. Please note, as of 11g we strongly recommend to use EXPDP command. For more details please check the note 1712612.About the checks Since this technique is not specifically meant to be a corruption check, only the usual checks are performed that happen during data access anyway. As the full table is exported no index accesses are done and therefore, index corruptions cannot be checked by this method.Check the database with EXP or EPXDP as of 11g. The export of the database to the NULL device can be performed with the legacy export tool EXP only.

Since it's only required to know whether or not all data can be read, but not to have the exported data somewhere in an export file, the export is done to the NULL device. EXPDP, the datapump export client, does not support NULL devices. a) export via BRSPACE For this kind of special export via BRSPACE also see note 778426 Using EXP --------- UNIX: brspace -u / -c force -f tbexport -l exp  -o full -t "*" -u /dev/null WINDOWS: brspace -u / -c force -f tbexport -l exp -o full -t "*" -u \nul Using EXPDP ----------- UNIX: brspace -u / -c force -f tbexport -l expdp  -o full -t "*" WINDOWS: brspace -u / -c force -f tbexport -l expdp -o full -t "*" If no problems where found during the brspace export, the log file finishes with the line: "BR1005I BRSPACE completed successfully" b) export via EXP/EXPDP Using EXP --------- UNIX: exp system/manager full=y file=/dev/null volsize=0 buffer=3000000 log=<log filename> WINDOWS: exp system/manager full=y file=NUL: buffer=3000000 log=<log filename> Using EXPDP ----------- UNIX: expdp system/manager full=y dumpfile=<filename> logfile=<log filename> WINDOWS: expdp system/manager full=y dumpfile=<filename> logfile=<log filename> When running the export without BRSPACE make sure to review the log file afterwards. If no problems where found during the export, the log file finishes with the line: "Export terminated successfully without warnings." Additional Remarks The export with exp does not support internal parallelization and you cannot run two brspace exports at the same time.EXPDP does support parallelization but not the export to null-devices, so in case enough file system freespace is available, a parallel EXPDP export may be quicker than a EXP run to the NULL device. Further check options:

Special Check: "Archivelogs"

Archivelogs (offline redo logs) can be checks via these methods: 1. by recovering them into a database instance, e.g. a standby database. This is the most accurate check method, since an actual recovery of the data is performed. For a standby database that should just be used to check the archivelog files, the same amount of disk storage is required as on the production site. However, it's not necessary to have a SAP system running on this database nor it is required to have the same level of resources concerning memory or CPU available as on the production site. 2. by using the BRARCHIVE option '-w use_rmv' or '-w first_rmv' as described in note 1016173. These options will automatically call RMAN to check the archivelog files for block corruptions. This method is especially reliable when the parameter DB_BLOCK_CHECKSUM is set to TRUE/TYPICAL (see parameter recommendations and sap note 923919 for details). 3. by 'dumping' an archivelog file. If there is a error while performing the dump this usually is due to a archivelog corruption. In case no error is returned, the archivelog file still may contain corrupt data; therefore this method is not reliable and should only be used exceptional.

Special Check: "Soft"-Corruptions/NOLOGGING blocks "Soft"-corrupt are blocks that can be read from disc correctly but for which the Oracle database internally need to mark them as corrupt.This happens e.g. when data had been loaded into block with the NOLOGGING option and a recovery is performed. In this case the information necessary to make the blocks consistent are missing. In BW installations this NOLOGGING option is heavily used with INDEXES and thus needs to be kept in mind when performing recovery of a BW database. If a consistency check is performed with RMAN, then the "soft"-corrupt blocks are entered into the V$DATABASE_BLOCK_CORRUPTIONS table with CORRUPTION_TYPE='LOGICAL'. When DBV is used for the check, then you may recognize "soft"-corrupt blocks when the following conditions apply: 1) After a recovery there are 2) MANY corrupt blocks in relatively

3) FEW indexes. 4) The corrupt blocks cover whole block ranges (extents). E.g. blocks [5,6,7,8,9,10,11] and [127,128,129,130] and [1120,1121,1122] are reported as corrupt. Of course this kind of corruption cannot be solved by the restore and recovery of the affected data files. In fact the blocks are marked corrupt just during recovery. The only solution for this is to rebuild the indexes as explained in sap note 547464. In the SAP environment only indexes are used with the NOLOGGING option, since the indexes can always be rebuilt based on the table data. Therefore no data loss would occur due to the NOLOGGING option. When the backups are taken with the BR*TOOLS and the recovery is performed with BRRECOVER the indexes will automatically rebuild as part of the recovery activity. If the affected blocks belong to other segments than indexes or index-partitions, please open a support message to have this analyzed.Consistency check of backups: To ensure that the data written to the backup is actually corruption free, the BR*TOOLS provide two options to check the backed up data files: a) check during the creation of backups When backups are done via RMAN a block check is automatically done during the reading of the blocks that should be backed up.To validate that the backed up data is readable, use the BRARCHIVE/BRBACKUP flag '-w use_rmv' or '-w use_dbv'. It's also possible to just have the files restored to a separate folder and a binary comparison done on the files. For this use the flag 'further keywords). Details on this can be found in note 811638 and the online documentation for the BR*TOOLS. b) separate check of the backup files. To check backups of an already finished backup run, BRRESTORE can be used with the '-w' flag, e.g.: brrestore -u /  -b last -d disk -w use_rmv

Oracle PL/SQL package DBMS_SPACE_ADMIN Oracle delivers a standard PL/SQL package with the database catalog to check and repair for space management failures in ASSM and locally managed tablespaces. Please refer to the Oracle documentation and Oracle Metalink for a complete reference on this. The package should be used under guidance of Oracle development support only or if described in notes like 1151509, 1001558, 317554, 871455.A special consideration need to be given to the use of this package as of note 817262 "Checktool for segment corruptions in ASSM tablespaces".

Oracle PL/SQL package DBMS_REPAIR The standard Oracle package "DBMS_REPAIR" is a set of tools that should enable the data rescue from tables that contain corrupt blocks. It's NOT a tool to actually repair corrupt blocks but it should allow to ignore corrupt blocks during segment scans. With that eventually the data in the non-corrupt parts of the segment might be extracted and the segment could be rebuilt afterwards. Since the corrupted blocks are actually marked as CORRUPT by oracle, they are taken out of access for the database. This means that even if only parts of the blocks are corrupt it won't be possible anymore to access the remaining good rows from these blocks. Because of that SAP/Oracle don't recommend to use this tool in case of a corruption. The usage of it will happen on own risk and may lead to serious data loss and/or additional corruption. Further Information Oracle Metalink Doc ID: 428570.1 - Best Practices for Avoiding and Detecting Corruption http://download.oracle.com/docs/cd/B19306_01/server.102/b14231/repair.htm#ADMIN022

Oracle PL/SQL package DBMS_HM Oracle 11g provides a new set of features to automatically analyze and repair database consistency. Up to now the implemented checks don#t cover the consistency checks mentioned in this note and thus may only serve as additional checks. Oracle 11g - Health Checks with Health Monitor http://download.oracle.com/docs/cd/E11882_01/server.112/e10595/diag007.htm#ADMIN11270

General hints and remarks =========================

Materializing DBA_EXTENTS

In certain cases it can be beneficial to copy the DBA_EXTENTS view in order to make the lookup of block to segment assignments quicker. The following steps explain how this needs to be done:

1. create table TMP_EXTENTS tablespace SAPDATA as (select owner, segment_name, partition_name, segment_type,   file_id, block_id, blocks   from dba_extents where rownum<1);

2. insert /*+append*/ into TMP_EXTENTS (select owner, segment_name, partition_name, segment_type,   file_id, block_id, blocks   from dba_extents);

3. create index i_tmp_ext on TMP_EXTENTS (file_id, block_id, blocks) nologging tablespace sapdata;

4. analyze table TMP_EXTENTS compute statistics; or exec dbms_stats.gather_table_stats ('<user_name>', 'TMP_EXTENTS');

SQL Query: Which segment is stored in block x?"

select owner, segment_name, partition_name, segment_type, block_id, blocks

from tmp_extents where (<block_id> between block_id and (block_id + blocks - 1)) and file_id = <file_id>  and rownum < 2;

SQL Query: "Which segment is stored in the corrupt blocks (RMAN)?":

1. set linesize 200 set pagesize 100 col owner for a20 col segment_name for a20 col partition_name for a20 col file_name for a50

2. select ext.owner, ext.segment_name, ext.partition_name, ext.segment_type, ext.block_id, ext.blocks, dbc.corruption_type, fil.file#, fil.name as file_namefrom   dba_extents ext,   v$datafile fil,   v$database_block_corruption dbc where   (dbc.block# between ext.block_id and (ext.block_id + ext.blocks - 1) )   and dbc.file# = ext.file_id   and dbc.file# = fil.file# order by ext.file_id;

SQL Query: "create DBV command for single segment checks":

Make sure to replace the values for USER_NAME, PASSWORD and SEGMENT_NAME in the BASIS_INFO query before running the command! SELECT NULL OWNER, NULL SEGMENT_NAME, NULL BLOCKS, NULL COMMAND FROM DUAL WHERE 1 = 0 UNION ALL ( SELECT NULL OWNER, NULL SEGMENT_NAME, NULL BLOCKS, NULL COMMAND FROM DUAL WHERE 1 = 0 ) UNION ALL ( SELECT * FROM ( WITH BASIS_INFO AS ( SELECT     '<TABLE_OWNER>' USER_NAME,     '<PASSWORD>' PASSWORD,     '<SEGMENT_NAME>' SEGMENT_NAME   FROM     DUAL ) SELECT   S.OWNER,   S.SEGMENT_NAME,   S.BLOCKS,   'dbv blocksize=8192 userid=' || BI.USER_NAME || '/' || BI.PASSWORD ||     ' segment_id=' || TS.TS# || '.' || S.HEADER_FILE || '.' ||     S.HEADER_BLOCK || ' logfile= dbv_log_'|| TS.TS# || '.' || S.HEADER_FILE || '.' ||     S.HEADER_BLOCK||'.txt' COMMAND FROM   BASIS_INFO BI,   DBA_SEGMENTS S,   V$TABLESPACE TS WHERE   BI.USER_NAME = S.OWNER AND   BI.SEGMENT_NAME = S.SEGMENT_NAME AND   S.TABLESPACE_NAME = TS.NAME )); Example output: OWNER      SEGMENT_NAME     BLOCKS ---------- ------------ ---------- SAPR3      MCSI            486400 COMMAND ------------------------------------------------------------ dbv blocksize=8192 userid=SAPR3/SAP segment_id=23.242.363657 logfile=dbv_log_23.242.363657.txt

Validity

This document is not restricted to a software component or software component version

References

This document refers to:

SAP Notes

1566406   DBV enhancements in transaction layer

1464091   Minor functional enhancements in BR*Tools (3)

1395074  ORA-00201: Block, dba , marked corrupt for invalid

1235952   Minor functional enhancements in BR*Tools (2)

1151509   ORA-600[Kcbz_check_objd_typ_1] Upgrading from 9i to 10g

This document is referenced by:

SAP Notes (26)

Attachments

1116190   Handling of database corruptions by the SAP Support

1001558   DBA_SEGMENTS.BLOCKS wrong after parallel create index

923919   Advanced Oracle block checking features

914174   Minor functional enhancements in BR*Tools (1)

871455   Bad performance when accessing DBA and V$ views

817262   Check tool for segment corruption in ASSM tablespace

817253   FAQ: Open a Database Read only

724972  ORA-00200: Block, dba , already marked corrupted

605062   FAQ: Restore and recovery

591600   Error due to inappropriate column values

589320   Composite SAP note: DBV-00102

546006   Problems with Oracle due to operating system errors

527250   DBV-00100 Composite SAP note

514178   ORA-14508 when validating partitioned tables

496967   IMP-00020 with reorganization after ORA-01498

376755   ORA-00600 - what to do?

365481   Block corruptions

357194   ORA-00900 when importing SQL scripts

356481   RSTXTCPY: Long text names cannot be entered

354293   DBVerify reports corrupt block in freespace area

317554   BYTES, BLOCK, EXTENTS of DBA_SEGMENTS empty

96296   CC-ERROR: Database problem in client copy

12741   Current versions of BR*Tools

1395074  ORA-00201: Block, dba , marked corrupt for invalid

1001558   DBA_SEGMENTS.BLOCKS wrong after parallel create index

1151509   ORA-600[Kcbz_check_objd_typ_1] Upgrading from 9i to 10g

914174   Minor functional enhancements in BR*Tools (1)

589320   Composite SAP note: DBV-00102

591600   Error due to inappropriate column values

724972  ORA-00200: Block, dba , already marked corrupted

514178   ORA-14508 when validating partitioned tables

923919   Advanced Oracle block checking features

605062   FAQ: Restore and recovery

527250   DBV-00100 Composite SAP note

871455   Bad performance when accessing DBA and V$ views

1235952   Minor functional enhancements in BR*Tools (2)

546006   Problems with Oracle due to operating system errors

1464091   Minor functional enhancements in BR*Tools (3)

354293   DBVerify reports corrupt block in freespace area

12741   Current versions of BR*Tools

96296   CC-ERROR: Database problem in client copy

376755   ORA-00600 - what to do?

365481   Block corruptions

817262   Check tool for segment corruption in ASSM tablespace

817253   FAQ: Open a Database Read only

1566406   DBV enhancements in transaction layer

317554   BYTES, BLOCK, EXTENTS of DBA_SEGMENTS empty

356481   RSTXTCPY: Long text names cannot be entered

357194   ORA-00900 when importing SQL scripts

File Name File Size (KB) Mime Type

analyze_complete_db_8.txt 1 text/plain

analyze_complete_db_9.txt 1 text/plain

analyze_oracle_dictionary.txt 1 text/plain

DBV_single_segment_check.txt 1 text/plain