Wednesday, January 5, 2011

Another Performance mystery unfolded or not?

FULL TABLE SCANS on LAST PARTITION of RANGE BASED PARTITIONED TABLE
----------------------------------------------------------------------------------------------------

A few days back I came accross a rather strange scenario. Our application team started complaining of  BAD PERFORMANCE on a 3 node RAC server which until a day before was hardly 50% loaded. Transactions which used to complete in a second had started taking more than 5 seconds which, to a normal application may not sound as bad as it seems but for a financial application it pretty damn is...

Anyways, upon scrutizing the system I realized that major waits on DB were due to FULL TABLE SCANS on a 70GB WEEKLY partitioned table and the frequency of the query doing FTS was  approx. 50/sec.

Surprisingly the relevant indexes were present but not being used despite the existence of latest statistics. Upon further investigation it was dug out that the data was now being inserted in last partition of table whose HIGH_VALUE in dba_partitions was MAXVALUE since the weekly partitions had exhausted.

...SURPRISE SURPRISE...

The resolution was to SPLIT the MAXVALUE partition into valid current week partition. After having done that the tables turned and FULL TABLE SCANS vanished...

Strange.. I did not come across a single document in Oracle which could relate our experience to any particular feature/constraint of Oracle. But somehow the optimizer refused to use EXISTING INDEXES when data was being queried from LAST PARITION (MAXVALUE).