REUSE option in VSAM and IDCAM. Drawbacks of Reuse.

I have seen the REUSE parameter in the REPRO statement using IDCAMS. Backtracking the purpose of using it, came to know several features of this keyword which i am sharing here.
Some applications might need temporary files, which can be deleted after the program run is over.You may need these files in the form of a KSDS,ESDS or RRDS. Thus VSAM allows to  create a Reusable cluster or files  using this keyword in the definition of the cluster.

Lets see  what happens if we do not use the REUSE parameter ? 
In this case, we can reload the VSAM only once.Only way to reload the data is to DELETE DEFINE the vsam file and repro the data.Should we reload it again, we need to delete/define it again.  In order to remove this repeated  DELETE DEFINE steps, we can define the vsam with REUSE parameter.
Do we remember the term HURBA ? Check the  listcat  section once how we can know about it.
HURBA stands for  --> Highly Used Relative Byte Area.
When we delete define a VSAM, this HURBA remains as zero. It increments as we add records. Ideally this HURBA indicates the offset of the last byte in the data set.
When we use reuse option, this HURBA is reset back to zero. Logically we are deleting all the records thereby.
Thus,the REUSE parameter allows us to reload the vsam as if it were empty.

Although it sounds simple, but there are certain drawbacks for using REUSE parameter.

We know VSAM performance depends on the CI/CA splits. With this resue option, the CI/CA splits remain in place. Over the period of time the performance degrades.

Few points to remember :
1. We cant use REUSE in the repro when the cluset is not defined with REUSE option.
     However we can Alter the VSAM definition to make it resuable.

  //STEPA    EXEC  PGM=IDCAMS
  //SYSPRINT DD    SYSOUT=A
  //SYSIN    DD    *
             ALTER -
                TEST.VSAMDATASET -
                REUSE

2.  Cluster defined with Reuse keyword can not have alternate index defined on it.

What is SMS in mainframe ? SMS stands for Storage Management Subsystem

Introduction:
The Storage Management Subsystem (SMS) automates the use of storage for data sets.  The z/OS ,  system is not aware of how much space a dataset will need, where it will be stored(Disk or Tape) ,what will be its record format and other details. Z/oS storage team need to decide whether to backup the large files and also need to recall it when necessary. All these data management activities can be done either manually or through the use of automated processes which is nothing but SMS.  With SMS activated , the z/OS  programmer or storage admin team  may, for example, create model data definitions for typical data sets, so that SMS automatically assigns those  attributes to data sets when they are created.
The data sets allocated through SMS are called system-managed data sets or SMS-managed data sets. For example, suppose you want to create a new data set named DATA.LIST. If SMS is active, you could use JCL like this:

//NEWDS DD DSN=myid.test.dataset1,
// DISP=(NEW,CATLG),
// DATACLAS=DCSL002,
// STORCLAS=SRTCL00

In this case, z/OS can use characteristics from predefined data and storage classes when it creates the DATA.LIST data set.

If SMS is not active or not in use, you need to manually specify the space requirements and storage location for the new data set, and your JCL would look like this :

//NEWDS DD DSN=myid.test.dataset1,
// DISP=(NEW,KEEP),
 // SPACE=(CYL,(1,1)),
 // UNIT=SYSDA,
// VOL=SER=SHARED  

The advantages associated with an SMS environment.
 • The user is relieved of making decisions about resource allocation of data sets, since it is done by SMS.
 • SMS provides the capability of concatenating data sets of unlike devices. For example, a tape data set can be concatenated with a disk data set. This capability is not available in a non-SMS environment.
 • SMS managed data sets cannot be deleted unless they are first uncataloged. Due to this extra step, erroneous deletion of data sets is minimized.
 • Additional features are available in the use of IDCAMS in an SMS environment. For example, the ALTER command can be used to increase the limit of the number of generation data sets in a GDG. This feature is not available in a non-SMS system.
 • VSAM data sets created in an SMS environment offer more flexibility that those created through JCL in a non-SMS environment.

Read about the  SMS Parameters in the next post

What are KSDS, ESDS,RRDS, LDS VSAM ?. Concept and Structure of Vsams

Types of Vsam and its concepts:
Vsam supports the following types of file organizations:
1. ESDS  2. RRDS   3.KSDS
ESDS (Entry Sequenced Data Set)
ESDS can be accessed sequentially or directly. They can not be accessed skip sequentially. ESDS are mostly where we write logs and it can be accessed sequentially.
ESDS record can be located sequentially  by checking  each record, starting from the beginning of the data set.Also an ESDS record can be located directly if the relative byte address(RBA) of the record in known.This location can be derived from some previous search or by remembering the address where the record was inserted.

ESDS Data Insertion

 New  records must be inserted at the end of the file. Each record is inserted at the end of the latest CI.IF there is no enough space in the CI, then a new CI is started.
Deletion of records physically is not possible.If we do not need one record, we can mark it as deleted , but it will remain in the dataset until the ESDS is re built again.
Updation of the records can be done where they are provided the length remains the same.If the length changes, the record must be marked as deleted and added as a new record at the end.

RRDS (Relative Record Data Set)
RRDS is another type of VSAM.It consists of fixed length areas called slots. These are pre-formatted when the data is created, or whenever a new CA is created.Records are inserted into the slots.

RRDS and its slots

Advantages of RRDS over ESDS:
In RRDS, Records can be added and deleted within these slots.
Records can be directly accessed by specifying slot number known as RRN. The first slot has RRN 1  .We can use skip sequential processing for RRDS
The Application program can insert into any free slot which is know as Direct insertion. The application program can request  the record to be inserted into next free slot which is known as sequential insertion.
While deletion, a record can be deleted and slot can be re used.
RRDS Structure:
There is one RDF for every slot in RRDS. Each of the RDF holds the information whether its associated slot is empty or not

RRDS  Internal Structure

In RRDS, all the records must be of same length. Unlike ESDS, we can not have Spanned record in RRDS & We Can not have Free space in RRDS.
LDS: (Linear Data set)
Linear Data sets holds the records in the form of strings of data. There are no CIDF  or RDF. Most common usage of LDS in DB2, which uses LDS to store objects.
Internally Z/OS  uses  DATA IN VIRTUAL(DIV)  services or WINDOWS SERVICS to insert/update/ delete the data.

KSDS (Key Sequenced Data Set)
In a KSDS, we have a cluster which consists of two parts ; Data part and the Index Part
We need to have the data to be sorted when its inserted into a KSDS. The Key part can be 1-255 contiguous bytes and must be unique. Each Key points to the data part. As we Insert,update or delete a record, this Index component gets automatically.

KSDS Vsam Cluster

The Rounded Yellow part signifies the Vsam Name and is know as the VSAM Cluster. It relates the Data and the Index part.  It is not a file, just  a catalog entry.
Records in a KSDS can be updated and their length can be changed.If he record is reduced in length, then the remaining space can be used as free space, else if the record length increases, the record is moved to Free space to make room for the new updated record.

Check here on How to Define KSDS,ESDS,RRDS,LDS with IDCAMS

Vsam Basics and Vsam tutorial. (Control interval, Control Area, CIDF, RDF)

This post is just a short writeup for VSAMs.
What is a Vsam file?
Virtual Storage Access Method : This is one type of dataset and  APIs to access the dataset.It is an excellent database to keep the records as they  can be read sequentially,directly or skip sequentially.
We must know that VSAM dataset must always be kept on disk and not on Tape. IBM newly introduced an extension of the VSAM known as  Extended Format Vsam . To an application , a Vsam and extended Vsam are  all same.Extended vsam provides some addition features like compression of the data, sharing, improved performance and allows a maximum data set size if 128 terabytes.
What are control Intervals? 
It is the basic building block for  VSAM dataset. It holds one or more record. Concept is similar to block for sequential and partitioned data sets.  When we read/write record from a vsam file , we read the entire chunk of data, ie the entire CI is moved to memory from DASD, not just the single record

Control Interval which consists of bunch of records.

What are CIDF , RDF in VSAM ?
Apart from keeping records in a CI, Vsam places two additional fields which are needed to manage CI. They are CIDF and RDF.

Unused spce, RDF,CIDF in a VSAM

The Dotted space shows the unused space in a CI.
RDF (Record Descriptor field) stores the length of each record. VSAM with fixed length records needs two RDF. One to keep track of the number of records and the other to count the length of record.For variable length record, there is one per record
CIDF (Control interval definition field) is a 4 byte length field which holds the location and size of unused space in CI.(like the one shown in shaded space)

What is Control Area  ?
Group of CIs make one Control Area (CA).

Group of Control Interval is called as Control Area.

Read here  to check the  Parameters to define a Vsam with IDCAMS

Spanned Records in VSAM and CI Split:
While inserting a record, if  the Record size is larger than the CI size, it can not be stored. In such scenarios,  CI splits happen if the vsam is defined with SPANNED option. The records then span across more than one CI and is known as spanned record. Spanned record occupy the entire CI. Any unused space in the CI can not be used.

Spanned record

Can we create a VSAM file with IEFBR14 ?

The Answer is 'Yes', we can, but with some limitations. We all know or used to IDCAMS  when it comes to VSAM creation. But even IEFBR14 can create VSAM with some advantages and disadvantages!

Let have the JCL for our first run:
//STEP1    EXEC PGM=IEFBR14                    
//SYSPRINT DD SYSOUT=*                         
//DD1      DD DISP=(,CATLG),DSN=IXXXX.TEST.VSAM
//         LRECL=80,KEYLEN=5,KEYOFF=0,RECORG=KS,
//         SPACE=(CYL,(5,5)
/*
And here you go with the Output for KSDS VSAM:

Lets look for the parameters used in the JCL
LRECL=80  Defines the maximum record length of the VSAM in bytes
RECORG  Is the important parameter to note. Its value determines the type of VSAM dataset the job will create.
Here in the example we have
RECORG=KS which made it as a KSDS
RECORG=ES  will create a ESDS
RECORG=ES  will create Linear data set
RECORG=RR will create RRDS
'KEYLEN=' will define the length of the KSDS key
'KEYOFF='  will define the key offset length. Here the value of '0' indicates the first byte.
'SPACE='  will allocate the space .Here it is in cylinders, the primary and secondary being 5 each.
However we can also use SPACE=(TRK,(50,20))
We can use the AVGREC parameter along with the SPACE parameter like below.The value of AVGREC has some significance.
AVGREC=U  implies primary and secondary space numbers are  number of records
AVGREC=K  implies primary and secondary space numbers are  number of records in thousands(multiplied by1024)
AVGREC=M  implies primary and secondary space numbers are  number of records in Million(multiplied by 1024*1024)
Re writing the Above JCL once more
//STEP1    EXEC PGM=IEFBR14                    
//SYSPRINT DD SYSOUT=*                         
//DD1      DD DISP=(,CATLG),DSN=I15122.TEST.VSAM
//         LRECL=80,KEYLEN=5,KEYOFF=0,RECORG=KS,
//         SPACE=(60,(300,200)),AVGREC=U
/*
So here, the primary space large enough for  300*60=18000 bytes and secondary for 1200 bytes.
Disadvantage of using IEFBR14:
1. We cant specify CI Size
2. No Shareoptions
3.Seperate data and index component parameters

 Hope this helps a bit! :)

VSAM I/O RETURN CODE IS 108 - RPLFDBWD

Few Errors encountered while playing with VSAMs:

 ACTION ERROR ON TEST.VSAM
** VSAM I/O RETURN CODE IS 108 - RPLFDBWD = X'8408006C'
NUMBER OF RECORDS PROCESSED WAS 0
FUNCTION TERMINATED. CONDITION CODE IS 12

Explanation:You have defined your VSAM with Record length ,say (15 15)
You are trying to do REPRO from a dataset with Different LRECL

Solution: 
Create a flat file with the same LRECL as the VSAM file and use it in REPRO command

Reading/Interpret the LISTCAT command output in VSAM

When we create a VSAM,all its details are recorded in the catalog table in MVS. LISTCAT is a versatile command meaning 'List Catalog'. LISTCAT  command can be fired from command  shell (option 6 from ISPF menu ) or it can run in batch as a SYSIN parameter to the IDCAMS.

To run it from the command shell , go to option 6 and give 
LISTCAT ENT ('TST.MYVSAM.NAME') all
We will get the information's being displayed on the screen page by page as we hit enter.

To use it in batch we need the IDCAMS utility . Below is the sample JCL for LISTCAT command 
//STEP0010 EXEC PGM=IDCAMS,REGION=0M,COND=(0,NE)
//SYSPRINT DD SYSOUT=*                          
//SYSIN    DD *                                    
     LISTCAT ENT ('TST.MYVSAM.NAME') ALL 

Interpreting the LISTCAT command

Interpreting the listcat command

As we can see here the name of the cluster is TST.MYVSAM.DEPT.As we go through the spool we will see the information of  STORAGECLASS,MANAGEMENTCLASS, but those things are not of that much importance to us now. Go on and have a look down for


It implies that this VSAM is not password protected. Also the ASSOCIATIONS parameter will list down all the datasets with the same name pattern and we can see that it shows the data component and INDEX component. So our VSAM is probably a KSDS.
Look in the ATTRIBUTES Section.
ATTRIBUTES section will give a more clear picture on the VSAM definition, ie how it is defined .
So, from these statistics we can somewhat guess the definition.Like, for the above VSAM, the key is of length 16 and offset of the key position being 20.
((RKP parameter defines the relative position  of the key field in the base cluster.) )
The AVGLRECL and MAXLRECL will give us the  idea about the record size parameter used in definition which in this case is  191 and 1500 respectively.
SPEED and NOERASE are also keywords in VSAM world.
One more 'NONSPANNED' , specifies that the data in the component can not span across CIs
So from this screenshot lets note down what all information we collected
NAME(TST.MYVSAM.NAME)
SHR(2 3)
RECORDSIZE(191 01500)
KEYS(00016 00020)
SPEED
NOERASE
If we look a little right on the mainframe screen we will get the information about the CI size .

 So, the CISZ parameter will be of length 18432.  CISZ(18432).
Also , watch the parameter NOREUSE and UNORDERED. So three more features added to our cart for the vsam statistics; CISZ(18432) ,NOREUSE and UNORDERED
The CI/CA in VSAM indicates the number of CI in each CA .Thus the above information tell us that there are 45 CIs in each CA.
Go on for few more details in the STATISTICS section
This Section really gives us all the details regarding the statistics of the vsam in that moment. Will see more in details later. As of now check out the FREESPACE parameter % CI and % CA. Those are the parameters with which the vsam has been defined.
Ohh..!! one more i got in the Allocation section.
SPACE-TYPE ----CYLINDER
SPACE-PRI    -------60
SPACE-SEC   -------10
This is more or less self explanatory, ie, the allocation quantity is CYLLNDER and primary and secondary allocations are 60 and 10 respectively.
So few more added in our cart.
FREESPACE(025  008)
CYL(00000060  00000010) 
Some other useful information from STATISTICS section
1. (( FREESPC --16386048 : This indicates space in bytes in completely free CIs. Do not mix them up unused space in in CIs) )
2. REC-TOTAL :Total number of logical records in the data component in that instance listcat was run.
3. REC-INSERTED: Total number of logical records inserted in data component ,since the vsam was last created or re-organized.
4. REC-RETRIEVED : total logical records read from data part since its creation or reorganized.It takes into account  reads for update during REWRITE processing.
5. REC-UPDATED : Total number of logical records UPDATED or REWRITTEN back into the data component since its last creation or last it was reorganized. Does not include DELETED records.However if  a records is updated, and may be got deleted later ,is counted here.
6. SPLITS-CA /SPLITS-CI : Total number of CI and CA splits that happened since it was last created or reorganized.
Having covered all these till now, if we look into the VSAM parameters which we collected till now  ( highlighted in blue above), we can some what figure out the structure of the VSAM which will be like:
DEFINE CLUSTER (          -
   NAME(TST.MYVSAM.NAME) -
   SHR(2 3)               -
   RECORDSIZE(191 01500)  -
   NOREUSE                -
   INDEXED                -
   NOERASE                -
   UNORDERED              -
   NONSPANNED             -
   SPEED                  -
      DATA               -
       (                           -
       NAME(TST.MYVSAM.NAME.DATA) -
       VOLUMES(XXXX)               -
       CISZ(18432)                 -
       KEYS(00016 00020)           -
       FREESPACE(025  008)         -
       CYL(00000060  00000010)     -
        )                          -
       INDEX                       -
       (                            -
       NAME(TST.MYVSAM.NAME.INDEX) -
       VOLUMES(XXXXX)              -
       )                             

How to define KSDS, ESDS ,RRDS VSAM with IDCAMS.. Explanation of the VSAM parameters

VSAM is one file storage system used to store and retrieve data. It stores data, but not like a relational model or hierarchical model like DB2 or IMS. In fact, there are many structures in DB2, which are implemented using VSAM .We can use VSAM files in both batch and online(CICS) operations.
How do we create a vsam file?
The IBM Access Methods Services (AMS) utility,popular as IDCAMS is the utility to create and maintain VSAM files. In this context,we should know that VSAM is totally catalog driven.When ever we create a VSAM dataset,ie a VSAM file, an entry is recorded in the system catalog which stores the cluster informtion.
Vsam Cluster? what is it ? ::
In this VSAM world,simple VSAM files are often called clusters. A cluster is the set of catalog entries that represent a file.
So, how many types of VSAM can we create, rather how many types of VSAM are there?
The answer is 4 types:
1.ESDS    or Entry Sequenced dataset
2.KSDS    or Key Sequenced dataset
3.RRDS    or Relative record dataset
4.LDS     or Linear dataset
Frankly,till date i never came across the use of LDS or RRDS. The most Widely used VSAM are KSDS and ESDS.
Lets see the JCL to create a KSDS:
How to create a KSDS and the description of the parameters.

//STEP0020 EXEC PGM=IDCAMS
//SYSPRINT  DD SYSOUT=*           
//SYSIN     DD *
  DEFINE CLUSTER(             -                
       NAME(TEST.FILE.MYVSAM) -              
       RECSZ(300 2000)      -                
       KEYS(41 20)          -                
       MANAGEMENTCLASS(xxxx) -           
       INDEXED              -                
       SPEED                -                
       SHR  (2 3)           -                
       FSPC (24,7)          -                
       BUFSP (39936)        -                
       DATACLAS (DCVSEXT))  -                
       DATA(NAME(TEST.FILE.MYVSAM.DATA) -    
         CISZ (18432)         -              
         CYL  (200,100))       -             
       INDEX (NAME(TEST.FILE.MYVSAM.INDEX) - 
         CISZ (1024)          -              
         CYL  (10,10))
As we see, the CLUSTER have two components here. The DATA and the INDEX part.
The keywords are marked and highlighted in red.
Explanation of the VSAM parameters:

1. RECORDSIZE(300  2000)
Specifies the record length of the file. The first parameter specifies the average length of the record and second parameter tells the maximum record length.
RECORDSIZE(avg  max)
avg : Average length of records
max : Maximum length of records
e.g. :
 RECORDSIZE(300  2000) [Fixed Length records]
 RECORDSIZE(300  300)    [Variable Length records]
Note :  RECORDSIZE or RECSZ  can also be used in the IDCAMS define step.
We can use this parameter in the cluster level as well as in the data level.
If not used, VSAM will use its default recordsize which is  RECORDSIZE(4086 4086)
 
2. KEYS(41 20)
The first parameter defines the key length and the second parameter defines the offset, ie, the beginning of the key position from the starting of the record
Thus the general format is like :
        KEYS(length-of-the-key  offset-from-begining)

3. MANAGEMENTCLAS : Optional Clause

4. INDEXED: This specifies that the VSAM is a KSDS VSAM

5. SPEED or RECOVERY option is coded at the CLUSTER or DATA level.It just governs how the VSAM load data into the file.
   With the RECOVERY option, the VSAM pre-formats each Control area with binary  zeros before it  writes any control interval to it. When SPEED is specified, no such additional activity is performed and therefore the data loading is much faster with SPEED option.

6..FSPC (24,7): FSPC or FREESPACE(24,7) is same thing.
FREESPACE parameter applies to the KSDS.This free space can be used for adding new records or for expanding existing variable records.FREESPACE applies only to the data component. For the FREESPACE parameter, the first value defines the percentage of free space to be reserved within each control interval, and the second defines the number of control intervals within each control area that should be reserved as free space.
Default is FREESPACE(0 0)

7. SHR  (2 3)OR SHAREOPTIONS ( 2 3):
This parameter tells VSAM whether you want to let two or more jobs to process your file at the same time. It specifies how a VSAM dataset can be shared. This is a big topic to deal with .Read here to explore this feature.
8. BUFSP(39936):  The number of buffers that can be used to hold INDEX and DATA records.
9. CYLINDERS(Pri Sec)or CYL(200,100):Primary : Number of units of primary space to allocate.This amount is allocated once when the dataset is created.Secondary : Number of units of secondary space to allocate.

In the DEFINE CLUSTER command we need to mention the type of VSAM we are creating like:

INDEXED   for KSDS
NONINDEXED  for  ESDS
NUMBERED for  RRDS
LINEAR for  LDS

So, Here is the sample JCL for ESDS VSAM

DEFINE CLUSTER(                   -               
           NAME(TEST.RM3452.FILE) -             
           RECSZ(300 2000)        -                     
           NONINDEXED             -               
           SPEED                  -               
           SHR  (2 3)           
      DATA(NAME(TEST.RM3452.FILE.DATA) -   
             CISZ (18432)              -             
             CYL  (200,100))           -