Links

Installing the Model9 agent

Prerequisites

License key

Obtain your license key from Model9 by opening a “License" request in the Model9 service portal: https://model9.atlassian.net/servicedesk/customer/portals
The output of the z/OS command “D M=CPU” is required for obtaining a new license.

Java version

Verify that the z/OS Java is version 8 64-bit SR6 FP25 or above. Using an older Java version would result in an agent startup failure with ZM91005E message code.
Note: z/OS Java version 11 is not supported
Verify the Java version and change the Java installation directory using the following commands:
cd /usr/lpp/java/J8.0_64/bin
./java -version
java version "1.8.0_281"
Java(TM) SE Runtime Environment (build 8.0.6.25 - pmz6480sr6fp25-20210115_01(SR6 FP25))
IBM J9 VM (build 2.9, JRE 1.8.0 z/OS s390x-64-Bit Compressed References 20201218_462060 (JIT enabled, AOT enabled)
OpenJ9 - 4c03b71
OMR - 86a8e1a
IBM - 8c30c56)
JCL - 20210108_01 based on Oracle jdk8u281-b09

UTC date and time

The object storage protocol requires the z/OS USS and object storage UTC date and time to match. Use the following USS command to display and verify the UTC date and time:
date -u
Note
If the UTC date and time do not match, the agent will fail with an HTTP error code 403 while trying to connect to the object storage.

JZOS setup

The Model9 agent is invoked using the JZOS Batch Java Launcher which is a component of the IBM JVM for z/OS. To verify that the JZOS is correctly configured to run the Model9 agent, check that the JZOS JVMLDM86 module is located in your LINKLIST or use the STEPLIB DD.
Note
For more details about JZOS setup, refer to the JZOS Installation and User Guide: http://publibfp.dhe.ibm.com/epubs/pdf/ajvc0120.pdf

Step 1: Allocate and mount the ZFS filesystem

Create a dedicated ZFS filesystem for the Model9 Agent installation and mount point for the new ZFS, as demonstrated in the following JCL. Edit the JCL using the site’s standard naming convention for the mount point, ZFS and volume serial number if it is not managed by the SMS.
//M9AGTZFS JOB ACCT#,SYSPROG,TIME=NOLIMIT,REGION=0M,
// NOTIFY=&SYSUID,MSGLEVEL=(1,1),MSGCLASS=X
//* Create a Model9 installation directory
//CREATE EXEC PGM=BPXBATCH,
// PARM='SH mkdir -p /usr/lpp/model9'
//* Define and format a new zFS for Model9
//DEFINE EXEC PGM=IDCAMS
//SYSPRINT DD SYSOUT=*
//SYSIN DD *
DEFINE CLUSTER (NAME(SYS2.MODEL9.ZFS) -
VOLUMES(xxxxxx) -
LINEAR CYL(200 50))
/*
//FORMAT EXEC PGM=IOEAGFMT,REGION=0M,
// PARM=('-aggregate SYS2.MODEL9.ZFS -compat')
//SYSPRINT DD SYSOUT=*
//STDOUT DD SYSOUT=*
//STDERR DD SYSOUT=*
//* Mount the newly defined zFS to the Model9 installation directory
//MOUNT EXEC PGM=IKJEFT01,DYNAMNBR=10
//SYSTSPRT DD SYSOUT=*
//SYSTSIN DD *
MOUNT FILESYSTEM('SYS2.MODEL9.ZFS') +
MOUNTPOINT('/usr/lpp/model9') +
TYPE(ZFS) MODE(RDWR) /* AUTOMOVE */
//
Note
The parameters for the IOEAGFMT utility and the mount commands are case sensitive.
To automatically and permanently mount the ZFS after an IPL, add the mount command to the BPXPRMxx as shown in the following example:
MOUNT FILESYSTEM('SYS2.MODEL9.ZFS')
TYPE(ZFS)
MODE(RDWR)
MOUNTPOINT('/usr/lpp/model9') /* AUTOMOVE */
Note
For a Sysplex environment, remove the comment marks from the term AUTOMOVE in the JCL above to enable file system migration between Sysplex members.

Step 2: Upload the Model9 agent TAR file to the mainframe

Use an FTP utility to upload the Model9 agent’s installation tar file to the model9 directory created in the previous step. Use Passive Mode if supported by the FTP client. The tar file must be uploaded in binary mode as shown in the following example:
$ ftp mf-lp1
Connected to mf-lp1.
220-FTPD1 IBM FTP CS V2R2 at mf-lp1, 06:20:40 on 2017-02-23.
220 Connection will not timeout.
Name (mf-lp1:m9user): m9user
331 Send password please.
Password:
230 M9U is logged on. Working directory is "M9U.".
Remote system type is MVS.
ftp> cd /usr/lpp/model9/
250 HFS directory /usr/lpp/model9/ is the current working directory
ftp> bin
200 Representation type is Image
ftp> put model9-v2.8.2_build_ebcddfb-agent.tar
local: model9-v2.8.2_build_ebcddfb-agent.tar remote: model9-v2.8.2_build_ebcddfb-agent.tar
229 Entering Extended Passive Mode (|||1026|)
125 Storing data set /usr/lpp/model9/model9-v2.8.2_build_ebcddfb-agent.tar
250 Transfer completed successfully.
4483584 bytes sent in 00:02 (1.95 MiB/s)
ftp> quit
221 Quit command received. Goodbye.

Step 3: Extract the agent files from the TAR file

Use the tar command in the z/OS UNIX shell to extract the agent files. During extraction, all agent files are saved in the same directory and can only be run by users who have the following permissions in the FACILITY class:
  • BPX.FILEATTR.APF ACC(READ)
  • BPX.FILEATTR.PROGCTL ACC(READ)
  • BPX.FILEATTR.SHARELIB ACC(READ)
The directory name usually includes the agent’s release number. It is recommended to create an alias that does not include the release number for the directory. This makes future upgrades more transparent, as shown in the following example:
TSO OMVS
su
cd /usr/lpp/model9/
tar -xpf model9-v2.8.2_build_ebcddfb-agent.tar
# define the alias name "agent"
ln -s model9-v2.8.2_build_ebcddfb-agent agent

Step 4: Create and populate the configuration directory

Create and copy the Model9 sample configuration directory using the following commands:
cd /usr/lpp/model9
mkdir conf
cp agent/sampleConf/* conf/

Step 5: Create a program-controlled version of PAX

The Model9 z/OS UNIX files backup uses PAX. This functionality requires the creation of a program-controlled version of PAX. Use the following commands:
cd /usr/lpp/model9
mkdir bin
cp -p /bin/pax bin
# Set program-control flag
extattr +p bin/pax

Step 6: Install the Model9 Command Line Interface

The Model9 CLI (Command Line Interface) provides an interface for issuing Model9 commands from TSO / JCL. Install the feature by creating a listener directory under the agent’s main path:
cd /usr/lpp/model9/
mkdir listener
chmod 777 listener
chmod +t listener

Step 7: Copy the Model9 libraries from USS to PDS

Edit the JCL CPY#PDS located in /usr/lpp/model9/agent/installation/ to create the Model9 LOADLIB, SAMPLIB, and EXEC PDS files. Submit the JCL.
Customize the M9CLI rexx in the EXEC PDS to match installation standards:
fifodir = "/usr/lpp/model9/listener"
loaddir = "SYS2.MODEL9.V282.LOADLIB"
Note
Copy the M9CLI EXEC to your site’s local EXEC library.

Step 8: Define the agent to RACF

Use the sample JCL M9USERST in the SAMPLIB PDS to define the security settings required by the agent.
Note
The M9USERST JCL permits discrete profiles. If your site uses generic profiles, update the JCL accordingly. If your site uses unprotected resources, no further action is needed.
Check you security product database to validate the type of resources being used.

Optional:

By default, the M9USERST JCL contains a definition for the M9USER with the OPERATIONS attribute. Model9 also supports the RACF TRUSTED attribute in cases extended auditing is required. To perform this change follow the steps below:
  1. 1.
    Remove OPERATIONS from ADDUSER M9USER command in the M9USERST JCL.
  2. 2.
    Add TRUSTED(YES) attribute to all RDEFINE STARTED commands within the STDATA section, for example:
    RDEFINE STARTED M9AGENT.** STDATA(USER(M9USER) GROUP(SYS1) TRUSTED(YES))
  3. 3.
    Add an RDEFINE statement for the life cycle procedure as the TRUSTED attribute is only supported for Stated Tasks.
RDEFINE STARTED M9LIFECY.** STDATA(USER(M9USER) GROUP(SYS1) TRUSTED(YES))
Note
Review and update the JCL to match local site standards. The SHARED parameter is specified by default. The SHARED.IDS permission is required to run the JCL.
Submit the JCL and verify all the commands were performed, review the output in case of any issues.

Step 9: Customize the Model9 agent start procedure

Copy the sample JCL M9AGENT from the SAMPLIB PDS to a local PROCLIB member. Update the JCL:
Update
Description
DD STEPLIB
Model9 installation LOADLIB
PWD environment variable
Model9 agent installation path
CONF_HOME environment variable*
Model9 agent configuration directory path
* CONF_HOME can be used for activating more than one agent in the same LPAR. The parameter will allow the agents to use the same Model9 installation files and libraries but have a different configuration directory. The recommendation is to have one agent per LPAR and to have all agents in the same GRS-complex point to the same Model9 complex. However, additional agents in the same LPAR may be required if using a sub-plex, having both development and production environments, or pointing the agents to different cloud storage. CONF_HOME must precede the stdenv-main.sh statement. The following is a sample use of the parameter:
//STDENV DD *
export PWD=/usr/lpp/model9/agent
export CONF_HOME=$PWD/../conf
export ENV=agent
. $PWD/scripts/stdenv-main.sh
//
Note
Add the agent procedure to the system’s startup process.

Step 10: Update the Model9 agent configuration

Before starting the agent, update the following configuration files located in /usr/lpp/model9/conf

model9-stdenv.sh

Parameter
Description
Required
Default
JAVA_HOME
64-bit java home path
Yes
None
TIME_ZONE
Java time zone, e.g. America/New_York
Yes
None
TCPIP_NAME
TCP/IP stack used by the agent *
No
TCPIP
AGENT_MEMORY
Allocated Java heap space for the agent. 1 GB is the minimum value.
No
1g
LIFE_CYCLE_MEMORY
Allocated Java heap space for life cycle management. 256 MB is the minimum value.
No
256m
SAPI_MEMORY
Allocated Java heap space for the server API (M9SAPI). 56 MB is the minimum value.
No
56m
SAPI_NETWORK_DEBUG
Enable network level debug for server API (M9SAPI). Expected values: “true” | “false”
No
false
SAPI_KEYRING_NAME
The SAF keyring name to be used as a truststore during server API certificate validation. if specified, the keyring will be searched at: &SYSUID/<KeyRingName>
No
None
* If running in an environment with multiple TCP/IP stacks, use the DISPLAY TCPIP operator command to display the TCP/IP stack name as shown in the following example:

agent.yml

Parameter
Description
Required
Default
port
The agent’s listening port for server HTTPS communication
No
9999
keystore
An agent keystore in JKS format which contains only one key entry to be used by the agent for incoming HTTPS requests. See Optional: Using the organization's certification authority.
No
agent.jks
keystore_password
The provided keystore password
No
agent
truststore
An agent truststore in JKS format which the agent will use to trust incoming HTTPS requests. See Optional: Using the organization's certification authority.
No
keystore value
truststore_password
The provided truststore password
No
keystore_password value

Configure the object storage settings by providing the following parameters:

Parameter
Description
Required
Value
objstore.endpoint.url
The URL of the object storage (Should start with “http://” or “https://”)
yes
default: none
objstore.endpoint.userid
Access key to object storage
yes
default: none
objstore.endpoint.password
Secret key to object storage
yes
default: none
objstore.endpoint.password_encrypted
Determine if the object storage password is obfuscated or not
no
default: false
objstore.resources.container.name
Container/bucket name
yes
default: model9-data
objstore.endpoint.api.id
The object storage API name
no
default: s3
Amazon AWS: aws-s3
Microsoft Azure: azureblob
objstore.endpoint.api.s3.v4signatures
When using object storage that uses V4 signatures, set this parameter to ‘true’, in addition to api.id: s3
no
default: false Cohesity: true HCP-CS: true
objstore.endpoint.no.verify.ssl
When using the HTTPS protocol, set to ‘true’ to avoid SSL certificate verifications*
no
default: true
objstore.endpoint.tier
The object storage tier to use for created blobs.
This option should only be used when using AWS S3 or Azure Blob.
none will not request a specific tier for blobs; leaving the tier selection up to the object storage.
standard maps to AWS S3's "Standard" tier and Azure Blob's "Hot" tier.
infrequent maps to AWS S3's "Infrequent Access" tier and Azure Blob's "Cool" tier.
no
Default: none
Accepted values:
none/standard/infrequent
objstore.endpoint.api.s3.calculatemd5
Enable MD5 checksum calculation to be included in putObject requests.
*This is required for Object Lock enabled buckets
no
Default: false Accepted values: true/false
* To enable trust validation on the object storage certificate, use the objstore.endpoint.no.verify.ssl parameter and set it to false.
The agent.yml file may contain additional parameters, setting them will override the defaults:
Parameter Name
Description
Default value
objstore.endpoint.shared.archive.volume
Archived volume serial name
M9ARCH
objstore.functionality.backup. threadpool.size
Number of parallel threads for backup and archive runs. Increasing this parameter requires verifying that sufficient java heap space is defined for the agent .
20
load.balancing.group.name
A group of agents that can share the work of a single policy.
&SYSPLEX
restore.progress.interval.
percentage
Progress interval by percentage
10
restore.progress.minimum.size.mb
The minimum size in MB of a backup/archive action at which point the Agent starts reporting recall/restore progress
1048576
objstore.resources.complex.id
Defines the resources the agent can process.
“group-&SYSPLEX”
lifecycle.deleteExpired.gdg.minimumDays
The minimum number of days that a data set will be kept after being archived. After the number of minimum days has passed, the data set will be checked whether it is eligible for deletion.
7
lifecycle.deleteExpired.gdg.intervalDays
After the minimum number of days has passed, the data set will be checked whether it is eligible for deletion. The data set will be checked every X days.
4
lifecycle.deleteexpired.cds.catalogexpiration.minimumdays
The minimum number of days that a catalog controlled cloud data set will be kept in storage.
7
cli.functionality.backup.compression
The compression type used for CLI backup execution.
gzip
cli.functionality.archive.compression
The compression type used for CLI archive execution.
gzip
import.allocate_use_expdt_98000
Whether or not to add the EXPDT(98000) to the tape allocations.
For CA-1 import support change to: true
false
import.allocate_label_type
Which label type should be used by the tape allocation.
For CA-1 import support change to: blp
sl
dcollect.output.unit
DCOLLECT UNIT
VIO
dfdss.input_primary_cyl
DFDSS SYSIN primary size
1
dfdss.input_secondary_cyl
DFDSS SYSIN secondary size
10
dfdss.input_lrecl
DFDSS SYSIN LRECL
80
dfdss.input_block_size
DFDSS SYSIN block size
80
dfdss.input_bufno
DFDSS SYSIN BUFNO
1
dfdss.input_unit
DFDSS SYSIN UNIT
VIO
dfdss.output_primary_cyl
DFDSS SYSPRINT primary size
1
dfdss.output_secondary_cyl
DFDSS SYSPRINT secondary size
10
dfdss.output_lrecl
DFDSS SYSPRINT LRECL
134
dfdss.output_block_size
DFDSS SYSPRINT block size
1340
dfdss.output_bufno
DFDSS SYSPRINT BUFNO
1
dfdss.output_unit
DFDSS SYSPRINT UNIT
VIO
dfdss.optimize
Set the DFDSS OPTIMIZE value for data set backup and archive
3
discovery.missing_property.message.level
The message level used for resource level discovery failures while fetching properties. Accepted values: SEVERE WARNING INFO
WARNING
idcams.temp.unit
IDCAMS temporary unit
VIO
xcf.functionality.enabled
Controls the agent's XCF listener which enables the auto-recall and CDS functionality
true
volumeBackup_dfdss_warn_to_ok_msg_codes
Controls which DFDSS warning mesages will not affect the return code of the DFDSS operation
ADR411W ADR333W
model9.dfdss.statements.backup.forcecp.enabled
Enables the use of FORCECP for data set level backup (CLI & Policy). Accepted values: true false
true
model9.dfdss.statements.backup.forcecp.days
The FORCECP parameter days value.
Accepted values: 0-255
0
Note
If the target z/OS installation does not currently support VIO, add the following parameters to the agent.yml file and change the unit configuration options appropriately: dcollect.output.unit, dfdss.input_unit, dfdss.output_unit and idcams.temp.unit
Cloud data set parameters
Parameter
Description
Default value
cloud_datasets.supported_read_utility_name_pattern
Determines the enabled applications for cloud data set read usage
"(IEBGENR)|(IEBGENER)|(DSNUTILB)|(IEBCOPY)|(IDCAMS)"
cloud_datasets.supported_write_utility_name_pattern
Determines the enabled applications for cloud data set write usage
"(IEBGENR)|(IEBGENER)|(DSNUTILB)|(IEBCOPY)|(IDCAMS)"
Resource complex and load balancing parameters
Agents sharing the same resource complex can all interact with the same resources. Actions such as backup/archive/recall/restore/list are all dependent on the resource complex parameter. Agents not sharing the same resource complex will not be able to list/restore/recall backups/archives created on a different resource complex. By default, all agents in the same Sysplex share the same resource complex. To update the default value of the resource complex, add the parameter to your agent.yml configuration file with a new value. Within a resource complex, you can define one or more load-balancing groups. See the User and Administrator Guide for more details.
Note
An agent can only access resources backed up or archived to the complex ID defined in its configuration.

Optional: Obfuscate the object storage credentials

Model9 allows obfuscating the object storage secret key so it won't appear in clear text within the configuration files.
Use the obfuscated object storage secret key that was produced during the server installation. When using the obfuscated secret add the following setting to the agent's configuration file:
objstore.endpoint.password_encrypted: true

Step 11: Update the Model9 agent license

Update the license file at /usr/lpp/model9/conf/model9-licenses.txt. The file can accommodate several licenses, for example, the license of all the LPARS in the Sysplex and the Disaster Recovery site’s license.
  • Each license should be written on a separate line.
  • Comments should be preceded with a “#” sign.
When multiple licenses are specified in the file, the most fitting license will be used, according to the following rules:
  • The CPUID correlates with the CPUID of the executing LPAR. This is mandatory.
  • From the licenses that correlate with the CPUID of the executing LPAR, the one with the latest expiration date will be used.
The license details to be used will be displayed in the agent’s log and in the UI. In case the agent could not find a valid license, an error message will be displayed as a WTO. The full error message will be available in STDOUT.

Optional: Secure the agent-server communication

By default, the agent accepts requests from trusted clients using a self-signed certificate generated by Model9. In a production environment, it is recommended to replace the certificate with a signed organizational CA certificate.
See the Securing the server to agent communication section for more information.

Step 12: Install automatic recall

The Model9 Automatic recall intercept allows the recall of archived files from within z/OS in a transparent manner. To install the automatic recall intercept:
Update the PROGxx configuration by adding the following statements:
APF ADD DSNAME(SYS2.MODEL9.V282.LOADLIB) SMS
LPA ADD DSNAME(SYS2.MODEL9.V282.LOADLIB) MOD(ZM9CPTN)
LPA ADD DSNAME(SYS2.MODEL9.V282.LOADLIB) MOD(ZM9S26X)
EXIT ADD EXITNAME(ZM9P_S026) MOD(ZM9S26X) PARM('M9ARCH')
Apply the changes using the SET PROG=xx operator command from any console:
SET PROG=xx
Note
The intercept and exit must be loaded to the Dynamic LPA in order for them to function correctly, do not use MLPA or PLPA to load the modules.
Copy the M9HOOK and M9UNHOOK from the SAMPLIB PDS to a local PROCLIB member. Customize M9HOOK accordingly and activate it using the following command:
S M9HOOK
Note
Add the M9HOOK JCL to your standard IPL process.
If an existing Model9 intercept has been uninstalled, then a new copy of the ZM9CPTN module must be loaded to the Dynamic LPA using the following command:
SETPROG LPA ADD DSNAME=SYS2.MODEL9.V282.LOADLIB MOD=ZM9CPTN
When using another data management product together with Model9 Cloud Data Manager, add the following DD statement to the other product’s procedure to avoid collisions:
//ZM9$NORC DD DUMMY
Restart the address space after applying the DD.

Step 13: Life cycle management

The Model9 life cycle management JCL is responsible for automatically deleting data sets (archives, backups and imports) that have expired. A sample JCL, M9LIFECY, can be found in the Model9 SAMPLIB PDS.
  1. 1.
    Grant your site scheduler permission for surrogate access to the M9USER:
    RDEFINE SURROGAT M9USER.SUBMIT UACC(NONE)
    PERMIT M9USER.SUBMIT CLASS(SURROGAT) ID(<scheduler>) ACC(READ)
    SETR REFRESH RACLIST(SURROGAT)
  2. 2.
    Update the life cycle management JCL:
    Update
    Value description
    DD STEPLIB
    Model9 installation LOADLIB
    PWD environment variable
    Model9 agent’s installation path
  3. 3.
    Submit the life cycle management JCL in the simulate mode: Set --simulate to ‘yes’. The life cycle management JCL will list all the data sets that would have been deleted without actually deleting them.
  4. 4.
    Submit the life cycle management JCL in the initial mode: Set --simulate to ‘no’ or remove. The life cycle management JCL will scan all archived data sets with expiration dates in the past - up until today’s date. The next time the life cycle will run, it will resume where the previous process ended.

Optional: When using RACF TRUSTED attribute instead of OPERATIONS

The Model9 Life Cycle procedure, M9LIFECY, can be set to use the RACF TRUSTED attribute. This will require defining the JCL as a started task. Follow this step to configure it:
  1. 1.
    Replace the M9LIFECY's JOB card with:
    //M9LIFECY PROC
Note
Life cycle management JCL should be scheduled daily via the local site scheduler.

Step 14: Start the Model9 agent

Start the agent from any console by issuing the following command:
S M9AGENT
Verify that the agent was started successfully. The following messages should appear:
ZM91002I MODEL9 BACKUP AGENT VERSION 2.8.2 INITIALIZING
ZM91000I MODEL9 BACKUP AGENT INITIALIZED

Optional: Configure the Remote Server API M9SAPI

The optional Remote Server API feature allows the running of Model9 policies from within z/OS using standard JCL. To enable the Remote Server API feature:
  1. 1.
    Copy the sample JCL PROC M9SAPI from the SAMPLIB PDS to a local PROCLIB file and edit the procedure according to the instructions in the file. The JCL M9SAPIJ sample can be used to run a policy.
  2. 2.
    The Remote Server API uses HTTPS to communicate with the server. If a trusted certificate is not defined for the server, set VRFYCERT=NO in the M9SAPI procedure to skip certificate validations.
  3. 3.
    If a valid certificate is defined, make sure the user running the M9APIJ has the correct SAF keyring defined to enable validation of the server certificate. The SAPI_KEYRING_NAME setting defined in the sample model9-stdenv.sh configuration can be used to specify the keyring name.
    Note
    Communications between the Remote Server API and the Model9 Server are encrypted regardless of the VRFYCERT value.
  4. 4.
    When verifying the certificate used by the Remote Server API, it must be defined for a Resource Access Control Facility (RACF) using a keyring.

Optional: Configure ISPF to use M9ARCH as ML2 default volume

ISPF allows the definition of only one Level 2 migration volume. The default migration volume is MIGRAT. In order to use the Model9 automatic recall function, the default volume must be set to M9ARCH, using the ISPF Configuration utility. Failing to change the migration volume will result in the following errors in ISPF screen 3.4:
Command
Situation
Resolution
Browse / Edit / View
Automatic recall will be rightfully triggered but the screen will continue to display M9ARCH as the VOLSER, instead of the DASD. The message "Tape not supported" will appear on the top right of the ISPF screen.
Enter REFRESH on the command line. The screen will be refreshed, showing the correct VOLSER, and the file will be available.
LISTC or performing a REXX
Automatic recall will be unnecessarily triggered and the data set will be recalled
Use TSO LISTC instead of typing LISTC next to the data set name
Note
Performing this step will cause any other auto-recall software to suffer from these errors.
To change the default ISPF settings:
  1. 1.
    Start the ISPF Configuration utility (TSO ISPCCONF) and select the site defined in the configuration table.
  2. 2.
    Select Option 1 to modify the existing table.
  3. 3.
    Change the “Volume for Migrated Data Sets name” to M9ARCH - the default Model9 archive volume name.
  4. 4.
    Save the settings.
  5. 5.
    Review the newly created configuration table file.
Either create and install the SMP/E USERMOD containing the new configuration table or create a load module that resides on a shared library containing the ISPF settings.