How to Download and Install ARInside?

Hi colleagues,

Some of you ask how to download and install ARInside, therefore I would like to give you hints on that. First you need to download ARInside from http://arinside.org/. Open zipped stuff to a specific folder. This folder will be referenced in the following files therefore update references if you change it later.

Then you need to create an ini file in ARInside folder as the following. Name it as "settings_<server_name>.ini".

# Application Settings
TargetFolder     = <Full address to the destination folder; where you would like ARInside to be generated>
FileMode     = FALSE
ObjListXML     = DHS.xml
UTF-8     = TRUE
BlackList     = MyBlacklist
LoadServerInfoList   = TRUE
CompactFolder     = FALSE
DeleteExistingFiles  = FALSE


# Data Retrieval
LoadUserList = TRUE
UserForm = User
UserQuery = '1'!=$NULL$


LoadGroupList = TRUE
GroupForm = Group
GroupQuery = '1'!=$NULL$


LoadRoleList = TRUE
RoleForm = Roles
RoleQuery = '1'!=$NULL$
MaxRetrieve = 0


# Layout
CompanyName = <Company Name>
CompanyUrl = <Company URL>


Create a .bat file with the following content and rename it as "<server_name>.bat". This file can also be implemented on Unix side with a shell script. However it is better to avoid from putting ARInside on the same machine where your actual ARS implementation is.

cd\
cd <folder where you placed all your ARInside stuff>
ARInside.exe -i settings_<server_name>.ini -p <password> -l <username> -t <port_number> -s <server_name>
cd\


Now you are able to schedule this file with:
Windows Start Menu -> All Programs -> Accessories -> System Tools -> Scheduled Tasks.

Create a scheduled task when your ARS application has less traffic. You can do a similar scheduling with crontab on Unix systems.

Copying Files From One Server to Another

Hi folks,

Today I would like to provide two methods for copying files from one server to another. You can use this for synchronizing your scripts or some ARS folders between servers. The first method is SFTP (Secure FTP). You can type the following to connect to a server:

sftp <server_name>

Then you need to change to the correct directory on remote folder:

cd <directory_name>

Finally you can copy the file from local to remote via "put" or from remote to local via "get":

get <file_name>
put <file_name>

You can use wild cards like in Unix environment:

get *.log

The disadvantage of sftp is that directories can not be copied easily. As a second method for copying files and   directories, you can use "scp" command:

scp -r <user_name>@<server_name>:<full_path_to_the_file_or_directory> <destination_folder>

How to Enable Version Control in ARS?

Hello everyone,

Version control is one topic where you need to setup on development environments to prevent implementation conflicts. ARS provides a version control system where developer has to reserve an object before modifying it and release after modification. In AR Server Information form, you will see a tab called Version Control:

In this tab, you have the option to enable "Object Reservation". Enabling this option will create a record in form "AR System Version Control: Object Reservation" for each object modification. If you need to modify an object which is reserved by another colleague, you should not change it :) If you really need to do this, you can delete the record from this form and make a refresh on the object list in Remedy Developer.

Second option in Version Control tab is "Object Modification Log" which creates a log entry in form "AR System Version Control: Object Modification Log" in case an object is modified. Idea is very basic: You can track the changes made to a particular object. If you enable the option "Save Definition Files", then old definition files will be saved within the object modification log.

How to Convert Oracle Timestamp to Remedy Date?

Hello, I guess this is one of the questions which is popping up all the time. The best way for this is to design a form where you have on one side Date/Time Field and on the other side Integer field as below:

Behind the button in the middle, write an active link and assign Date/Time Field to Integer field in a SetFields action.

How to grep for ar system processes?

Hi guys,

Today I would like to give you a simple but useful command. With this command, you can see all Remedy processes running on the server, AR Server processes plus DSO processes:

ps -ef | grep arserv

How to make an API call by an external script?

Today I would like to share with you how to make an API call by a C script. Maybe it is worth to list what we will mainly do in the script:
1) Read the configuration file
2) Initialize the session
3) Verify the user we are connecting
4) Change AR Server connection queue
5) Set a flag (Radio button field) on a Remedy form
6) If there is an error while setting the flag, write the error message to another field.
Below is the code we have generated. You can actually change the parameters in blue color with yours.


#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <time.h>
#include "area.h"
#include "ar.h"
#include "arerrno.h"
#include "arextern.h"
#include "arstruct.h"
#include "ars_utils.h"


char ErrMsgStatus[2048];


void LogArsStatus(ARStatusList *);


int main(int argc, char * argv[])
{
ARNameType FormName;
ARStatusList arStatus;
ARControlStruct control;
char ProcessName [10];


AREntryIdType entryIdArr[1];
AREntryIdList entryIdL;
ARFieldValueList fieldL;

ARFieldValueStruct process;
ARFieldValueStruct error[2];


char ErrMsg [500];
char ConfigFile [128];
char requestID[16];

strcpy(requestID, argv[1]);
strcpy (ProcessName, "<PROCESS_NAME>");
strcpy(FormName, "<FORM_NAME>");


/*--------------------------------------------------------------------*/
/* Setup configuration file and control structure.
/*--------------------------------------------------------------------*/
strcpy(ConfigFile, "<CONFIG_FILE>");
if ((ConfigInit (ConfigFile)) != OKAY)
{
sprintf(ErrMsg, "Fatal error initialising config file\n");
ERROR (ERR_FATAL, ProcessName, ErrMsg, 0);

exit (1);
}
SetupControl (&control);


/*--------------------------------------------------------------------*/
/* Initialise Remedy ARS session
/*--------------------------------------------------------------------*/
if (ARInitialization(&control, &arStatus) >= AR_RETURN_ERROR)
{   
sprintf(ErrMsg, "Fatal error - initialising ARS session");
ERROR (ERR_FATAL, ProcessName, ErrMsg, 0);
LogArsStatus (&arStatus);
FreeARStatusList(&arStatus, FALSE);
return;
}
FreeARStatusList(&arStatus, FALSE);
  
sprintf(ErrMsg, "Initialized ARS session");
ERROR (ERR_INFO, ProcessName, ErrMsg, 0);


/*--------------------------------------------------------------------*/
/* Verify the user we are running this process with
/*--------------------------------------------------------------------*/
if (ARVerifyUser(&control, NULL, NULL, NULL, &arStatus) >= AR_RETURN_ERROR)
{
sprintf (ErrMsg, "Fatal error - verifying user '%s' on server '%s'", control.user, control.server);
ERROR (ERR_FATAL, ProcessName, ErrMsg, 0);
LogArsStatus (&arStatus);
FreeARStatusList(&arStatus, FALSE);
return;
}
FreeARStatusList(&arStatus, FALSE);


sprintf(ErrMsg, "Verified user '%s'", control.user);
ERROR (ERR_INFO, ProcessName, ErrMsg, 0);


/*--------------------------------------------------------------------*/
/* Chnage the AR Server Connection Queue
/*--------------------------------------------------------------------*/
if (ARSetServerPort(&control, control.server, <SERVER_PORT>, <QUEUE_NUMBER>, &arStatus) >= AR_RETURN_ERROR)
{
sprintf (ErrMsg, "Fatal error - changing the queue for user '%s' on server '%s'", control.user, control.server);
ERROR (ERR_FATAL, ProcessName, ErrMsg, 0);
LogArsStatus (&arStatus);
FreeARStatusList(&arStatus, FALSE);
return;
}
FreeARStatusList(&arStatus, FALSE);


sprintf(ErrMsg, "Queue is changed");
ERROR (ERR_INFO, ProcessName, ErrMsg, 0);


/*--------------------------------------------------------------------*/
/* Set Flag
/*--------------------------------------------------------------------*/
entryIdL.numItems = 1;
entryIdL.entryIdList = entryIdArr;
strcpy(entryIdArr[0] , requestID);


fieldL.numItems = 1;
fieldL.fieldValueList = &process;


process.fieldId = atol("<FLAG_FIELD_ID>");
process.value.u.intVal = 0;
process.value.dataType = AR_DATA_TYPE_INTEGER; /*AR_DATA_TYPE_ENUM*/


if (ARSetEntry(&control, FormName, &entryIdL, &fieldL, 0, 1, &arStatus) >= AR_RETURN_ERROR)
{
sprintf(ErrMsg, "Fatal error at processing the transaction '%s'.", requestID);
ERROR (ERR_WARN, ProcessName, ErrMsg, 0);
LogArsStatus (&arStatus);
FreeARStatusList(&arStatus, FALSE);

fieldL.numItems = 2;
fieldL.fieldValueList = error;
error[0].fieldId = atol("<ERROR_MESSAGE_FIELD_ID>");
error[0].value.u.charVal = ErrMsgStatus;
error[0].value.dataType = AR_DATA_TYPE_CHAR;


if (ARSetEntry(&control, FormName, &entryIdL, &fieldL, 0, 1, &arStatus) >= AR_RETURN_ERROR)
{
sprintf(ErrMsg, "Fatal error at writing Error Msg to the request '%s'.", requestID);
ERROR (ERR_WARN, ProcessName, ErrMsg, 0);
LogArsStatus (&arStatus);
FreeARStatusList(&arStatus, FALSE);
}
exit(1);
}
return 1;
}


void LogArsStatus(ARStatusList *Status)
{
    static char *MsgType;
    ARStatusStruct *tempPtr;
    int i;
    sprintf(ErrMsgStatus, "ARS Status List: %u items", Status->numItems);
    if (Status->numItems != 0)
    {
        tempPtr = Status->statusList;
        for (i = 0; i < (int)Status -> numItems; i++)
        {
            switch(tempPtr->messageType)
            {
                case AR_RETURN_OK:      MsgType = "NOTE   "; break;
                case AR_RETURN_WARNING: MsgType = "WARNING"; break;
                case AR_RETURN_ERROR:   MsgType = "ERROR  "; break;
                default:                MsgType = "?????  "; break;
            }


            sprintf(ErrMsgStatus, "%s\n   Type: %s   Message number: %d\n",
                    ErrMsgStatus, MsgType, tempPtr->messageNum);


            if (tempPtr->messageText == NULL)
            {
                sprintf(ErrMsgStatus, "%s   Message:\n", ErrMsgStatus);
            }
            else
            {
                sprintf(ErrMsgStatus, "%s   Message: %s", ErrMsgStatus, tempPtr->messageText);
            }
            tempPtr++;
        }
    }
}

By Zekeriya Dinçer

Operation Ranking in Load Balancer Environment

In Load Balancer ARS environments, there is a form called "AR System Server Group Operation Ranking" where you can set which tasks need to be handled by which server. The most important ones of these tasks are listed below:
- Administration
- Archive
- DSO
- E-mail Engine
- Escalation

As you can see in the screenshot, all servers for each operation are listed by default. According to which server you give the priority for an operation, you need to set the Rank field. E.g. If you would like DSO to be handled by Node1 server first, then you need to set value "1" for this row. If Node1 server is not available, then Node2, Node3 and finally Node4 server will try to handle DSO operations respectively.

According to this Operation Ranking form, the options in form "AR System Administration: Server Information" are set automatically. For instance, "Disable Archive" option will not be marked for Node1 server since Node1 is defined as the default server for Archiving.

Converting ARS Timestamp to Date

There is one useful function on database level called "unixts_to_date" where you can convert your ARS or Unix timestamps to date. Here is the code for it:


CREATE OR REPLACE FUNCTION ARADMIN.unixts_to_date(unixts IN PLS_INTEGER) RETURN DATE IS

        /**
         * Converts a UNIX timestamp into an Oracle DATE
         */
        unix_epoch DATE := TO_DATE('19700101000000','YYYYMMDDHH24MISS');
        max_ts PLS_INTEGER := 2145916799; -- 2938-12-31 23:59:59
        min_ts PLS_INTEGER := -2114380800; -- 1903-01-01 00:00:00
        oracle_date DATE;
       
        BEGIN           
            IF unixts> max_ts THEN
                RAISE_APPLICATION_ERROR(
                    -20901,
                    'UNIX timestamp too large for 32 bit limit'
                );
            ELSIF unixts <min_ts THEN
                RAISE_APPLICATION_ERROR(
                    -20901,
                    'UNIX timestamp too small for 32 bit limit' );
            ELSE
                oracle_date := unix_epoch + NUMTODSINTERVAL(unixts, 'SECOND');
            END IF;           
            RETURN (oracle_date);
END;

The usage for this function is below:
select unixts_to_date(1316466000) from dual;

Materialized Views for Reporting

Reporting is one of the important issues in AR System environment. If you are not able to take proper reports from your system, it is not meaningful to enter data, is not it? While you are writing select statements or even for updates, you can use ARS views e.g. select * from ar_system_email_messages. However you should remember that some workflow is running on this ARS form. When you are selecting from ARS view, you are accessing to the T table and you may lock the table. This actually would cause the workflow to run into errors.

For reporting, I recommend you to build a single interface although you may have more than one consumers. At least you can try to decrease the number of requested views as much as possible to avoid unnecessary DB load. To me the best structure for reporting is Materialized Views. Although its name is "view", it behaves like a table e.g. you can define index and triggers on it. It keeps all data on a separate place, therefore you do access to the ARS tables everytime you query.

Now let's come to the point how we populate data to Materialized Views. The name of the operation modifying a materialized view is called "Refresh". In Oracle, you can right click to the materialized view and you can see the "Refresh" option. There are different modes of refresh. If your view is basic and does not contain functional statements and complex joins, you can define Fast refresh. This option tracks the changes on the base tables of the view and refreshes the view considering these changes. It does not touch the unmodified data and that's why it is fast. There is also "Complete" mode. This builds the view from the beginning. You should know that there will be a huge data transfer with this mode. Here there is another concept called "Atomic" and "Non-atomic" refresh. If you select "Atomic" refresh, Oracle first detects the modified rows and changes these rows in the original view by inserts, updates and deletes. This is the reason why it is much slower compared to "Non-atomic" refresh. In our environment, we had a huge and complex view. For this view, "Complete Non-atomic" is the best refresh combination for materialized view. There is also another refresh mode called "Force". If this mode is selected, first Fast mode is taken to consideration. If it is not possible, a Complete refresh is done. Please note that refresh modes (Fast/Complete/Force) is defined while creating the view and refresh types (Atomic/Non-atomic) can be given in run time.

According to your need, you can build different materialized views e.g. you can keep last 7 days data in one view and last 6 months data in another view. In order to automate refreshes, you can define a stored procedure and call this by a scheduled job. Here is the piece of code which you can use for refresh in stored procedure:

DBMS_MVIEW.REFRESH('VIEW_NAME', ATOMIC_REFRESH=>FALSE);

What is Load Balancing and Load Balancer?

Load Balancing is a mechanism for distributing load among AR Servers. These AR Servers are generally node servers because they run the workflow and the load is usually high on them. Load Balancer server is a different physical or logical machine installed on the same environment. In our environment, there is a Perl script run by crontab on each server and it opens or closes port 7070 according to the availability of the server. Then the load balancer server checks port 7070 on a particular server and redirects requests if the port 7070 is open. Otherwise it means the server is not available and no requests are redirected to that server. Here is the code for opening and closing the port:

open (LOG,">>$LOGFILE");
print LOG "\n\n#####     Started: " . scalar localtime(). "   #####\n";


my $sock;
my $client;


my $flag=0, $debug=0;
while(1) {
    open (LOG,">>$LOGFILE");
    print LOG "#####     Check: " . scalar localtime(). "   #####\n";
    print "#####     Check: " . scalar localtime(). "   #####\n";
    close(LOG);
    $ARScrtl = ars_Login($ARSserver, $ARSuser, $ARSpassword);
    if ( ! $ARScrtl)
    {
        ######### Login fails ########
        open (LOG,">>$LOGFILE");
        print LOG "##### " . scalar localtime(). " Login failed: $ars_errstr #####\n";
        #print "##### " . scalar localtime(). " Login failed: $ars_errstr #####\n";
        close(LOG);
        close($client);
        close($sock);


        #end of cycle, wait for 10 sec. for next check cycle
        sleep(10);
    }
    else {
        ######## Login succeeds #######
        open (LOG,">>$LOGFILE");
        print LOG "##### " . scalar localtime(). " Login successful. #####\n";
        #print "##### " . scalar localtime(). " Login successful. #####\n";
        close(LOG);
        ars_Logoff($ARScrtl);


        $sock = new IO::Socket::INET (
                                  LocalHost => $LBChkServer,
                                  LocalPort => $LBChkPort,
                                  Proto => 'tcp',
                                  Listen => 1,
                                  Reuse => 1,
                                        #Blocking => 0
                                        Timeout => 10
                                 );


        close($client);
        if ( $client =$sock->accept() ) {
            #print LOG "New connection!\n";
            #sleep(3);
        }
        close($sock);
    }


    if($debug eq 1) {
        if($flag eq 0) {
                        $flag = 1;
        }
        else {
           $flag = 0;
        }
    }
    #sleep(1);
}


close($sock);
close(LOG);

Visibility Concept in AR System

In AR System, there exists a core functionality on Visibility of the entries in a form. As a base form AR System has Groups form that can be assigned to the users.

If successfully managed, AR System reserved fields Assignee Group and Dynamic Groups that range between 60000 to 60999 can be used to restrict access to the form entries. These groups will also be shown in the Permission Lists of the fields. So restricting the access can be done by giving permission to the Request ID($1$) field just the Assignee Group and the Dynamic Groups. The trick is to have fields in the form that we are applying our configuration with the group id as field id.

This case is useful when there exists two types of users as Write Permission and Read Permission. Let's make an example: Assume we have a form as "ABC Form". This form has all the required fields. We also have 2 distinct users who has only READ and who has only WRITE as Assignee group in User form.

1) We need to create two groups in the Group form as WriteSecurity, ReadSecurity assuming 60512 and 60513, respectively.

2) We need to create two more groups as READ 50810 and WRITE 50820.

3) We open our form "ABC Form" and edit the permissions of the Request ID($1$) and remove all the permissions and add only the ReadSecurity and WriteSecurity by giving the ReadSecurity only View Permission and WriteSecurity by giving Change Permission.

4) Add two fields as ReadSecurity and WriteSecurity with Field IDs 60512 and 60513, respectively.

5) Save the form.

6) Create a submit filter as when an entry is created the WriteSecurity field is set as WRITE groups ID and ReadSecurity is set as READ groups ID.

7) The users with READ and WRITE can see the form entries, the others can not and Read can only view and Write can modify the entries.

by Alper Gülbahar

Using Private Queues

Private queues are used to define a separate path for a request to the AR Server. If you have some script or API code reaching your AR Server, you may need to define a separate queue not to be blocked by the load of the AR Server. You can define a private queue under Server Information form -> Ports and Queues tab. Here you can state maximum and minimum threads to be used for this queue. When AR Server is started, private queue starts with the number of minimum threads and increase number of threads (if needed) till the maximum number of threads is reached. If you do not state in your API calls, Fast (390620) and List (390635) queues are used by default. In order to use a private queue (390654 in our case), you need to use the following schema in your code:

if (ARSetServerPort(&control, control.server, 4000, 390654, &arStatus) >= AR_RETURN_ERROR)
{
sprintf (ErrMsg, "Fatal error - changing the queue 390654 for user '%s' on server '%s'", control.user, control.server);
ERROR (ERR_FATAL, ProcessName, ErrMsg, 0);
LogArsStatus (&arStatus);
FreeARStatusList(&arStatus, FALSE);
return;
}
FreeARStatusList(&arStatus, FALSE);

Helpful Unix Commands

Now I would like to give you the Unix commands which I use most of the time:

1) Search for a term in all the files in the current folder:
find . -type f -exec grep -l "search_term" '{}' \;

2) Find files in the current folder whose size is greater than 300K:
find . -size +300000k

3) Exchange xxx with yyy in all files ending with Temp.sh:
perl -pi -e 's/xxx/yyy/g' *Temp.sh

4) Cut lines between 125 and 127 from testfile (you can also redirect it to another file):
sed -n '/125/,/127/p' testfile

5) Kill all processes which includes xxx in its name:
kill -9 `ps -ef | grep "xxx" | grep -v grep | awk '{print $2}'`

Happy working!!

Importing Data to an AR System Form on Database Level

You have more than one option to load data into an AR System form. You can use Remedy Import tool however you may need to build data on database level with a query for some data fix or administration tasks. E.g. There might be some number of corrupted tickets on your system and you may need to close them via an escalation. (The reason to use an escalation here is that tickets need to be closed by the workflow properly). First you have to create a regular AR System form, let's say "zz_Data_Fix" with default core fields plus character field named "ticket_number". Here I will also use a temporary table named zz_Intermediate for a better understanding. You populate this intermediate table with the query where you select the tickets which needs to be closed:

create table zz_Intermediate as
select ticket_number from mytickets where <condition for selecting tickets to be closed>

Here "mytickets" is the database table where you keep all your ticket data. You can also use joins in this query. Then you import ticket numbers into your zz_Data_Fix form (in order to find the schemaid of zz_Data_Fix, you can use the following: select schemaid from arschema where name = 'zz_Data_Fix'):

insert into zz_data_fix (request_id, ticket_number, create_date, last_modified_by, modified_date, status)
select rownum, ticket_number, 1, 'username', 1, 0 from zz_Intermediate


With running this query, records are loaded into the T table which keeps the data in AR System environment. (You have the option to directly import to T table but this way seems easier for me). However you also need to load H table which keeps status history info of every record. If you do not do that, you may not see your record in Remedy User.


insert into H1413 (entryid)
select C1 from T1413

Please note that form zz_Data_Fix is created from scratch therefore both T1413 and H1413 are empty. Some H tables might have different structures so you may need to adjust your query a little. After committing the above queries, you should see your ticket numbers when you make an unqualified search in Remedy User zz_Data_Fix form.

As the last step, you can write an escalation on zz_Data_Fix and make a Push Fields to your original ticket form with a PushFields if condition: $ticket_number$ = 'ticket_number' and modify any field you want.

Data Archiving

As the data grows in your AR System environment, you have to archive data for mainly performance reasons. For instance, it is not logical to keep 5 year-old ticket in ticket form. This data can be archived and can be accessed whenever needed, esp. for reporting. AR System provides a built-in archiving mechanism. It is very simple to set up archiving and it is very similar to create an escalation object.

The mechanism behind archiving is actually the same logic with escalations. In Remedy Developer, you open a form, then go to Main Menu -> Form -> Form Properties -> Archive. The best option for archiving would be "Copy to Archive and Delete from Source".  Archive state should be "Enable". If you define the archiving for the first time, AR System will create the archive form automatically. There are some options for archiving attachments and diary fields. Time definition is quite straight-forward. The most important part is the qualification. You can use something like:
"$LastModifiedDate$ < $TIMESTAMP$ -86400". This will archive all records whose last modified date is older than 7 days. One of the important tips is that you are allowed to define indexes on archive forms as you do in Regular forms. If you have a reporting based on archive forms, this can be very helpful.

Built-in archiving mechanism however is not always the best. When the archive form is created automatically by AR System, its type is "Archive". In this form, AR System puts some new fields like "Original Request ID" and "Original Create Date" not to lose record creation information. During the migrations you would get in trouble if you do not maintain the archive form whenever you modify the original form. AR System sometimes gives errors when you import archive form. Another disadvantage is that built-in mechanism archives all the fields. This enlarges the archive form and you may not need all the fields for your reporting purposes. Another proposal would be to implement your own archiving with escalations. You can define archiving forms manually and you can transfer the records between forms via DSO. In this way, you can choose the fields which you would like to archive. Migrations will be easier since the form type of the archiving forms will be normal "Regular" form.

Interpreting Log Files

Log files have great importance in AR System since they help you to identify the root-cause of an incident. Once you have a production release, logs might be full of errors but you have to order these errors according to their impact and then solve one by one till you have a clean log file. The most important error is that there some sort of loop on the system. You can see this in the log file as "Too many filter operations...". Once you see this, your system is in trouble and this error must be solved as soon as possible. In this case, you can turn on filter and escalation log and find the reason for this error. Another fatal error is the one related with licenses. If you do not have enough license, your system might behave abnormal, e.g. you would not create a licensed user.

Another point in log analysis is that AR System might throw the same error multiple times which cause a big log file. In order to read it, you have to go through many pages which is sometimes impossible. For this case, I recommend you to write a simple log parser, e.g. in Java, which goes through the whole log file and brings you distinct errors. In this way, your logs will be more readable.

It is also important to rotate the logs because they get larger everyday. It is a good idea to rotate logs, esp. arerror.log on daily basis. In this way, you can find errors easier when you have an incident. BMC always requests log files when you open a ticket to them. Having daily logs will also be an advantage because you can submit smaller logs to BMC. We use the following dummy shell script for rotation and zipping. You can adapt it to your environment:

#!/bin/sh
#------------------Rotating and Zipping 'XXX_error.log'--------------#
DAY=`date +%d`
MONTH=`date +%m`
YEAR=`date +%Y`
#------------------Rotation of 'XXX_error.log'----------------#
LOGFILEJ=api_error.log
for dir in <directory>
do
        if [ -x $dir ]; then
                cd $dir
                mv $LOGFILEJ $LOGFILEJ.$YEAR$MONTH$DAY
                echo "-------------------------------------------------"        
              echo "$YEAR-$MONTH-$DAY : '$LOGFILEJ' file is rotated";
        else
              echo "-------------------------------------------------"
                echo "$YEAR-$MONTH-$DAY : ERROR:'$LOGFILEJ' file cannot be rotated!";
        fi
done
#------------------Zipping of 'api_error.log'----------------#
for dir3 in  <directory>
do
        if [ -x $dir3 ]; then
                 cd $dir3
               gzip -c $LOGFILEJ.$YEAR$MONTH$DAY  > $LOGFILEJ.$YEAR$MONTH$DAY.gz
               rm $LOGFILEJ.$YEAR$MONTH$DAY
               mv $LOGFILEJ.$YEAR$MONTH$DAY.gz api_error/$LOGFILEJ.$YEAR$MONTH$DAY.gz
                 echo "-------------------------------------------------"       
               echo "$YEAR-$MONTH-$DAY : '$LOGFILEJ' file is zipped.";
        else
               echo "-------------------------------------------------"
                 echo "$YEAR-$MONTH-$DAY : ERROR:'$LOGFILEJ' file cannot be zipped!";
        fi
done
exit

Storing Configuration Files

Having the correct configuration is always a problem in AR System. You need to configure your ar.conf file according to the needs of your system. And once you have a stable system, it is worth to store this configuration file in some version control system, e.g. SVN or ClearCase. We experienced in the past that ar.conf was modified out of our control. We could not find whether someone changed this or file is corrupted by the system. The queue settings were removed and DSO password was changed. What can you do in these cases? If you have a strange problem on the system, (AR Server crashes or DSO does not work), please check the ar.conf file first and compare it with the previous version. This will definitely help you to detect whether your ar.conf file is correctly configured.

Backup Your AR System Workflow

You need to make sure that your AR System workflow is always secure because it is the code that you produce and you can not stand losing it any time. While we were in school, one of the mates was deleting the C code and then shouting with a fear on his face :) This is why backuping is always important in IT, whatever you develop. AR System provides you backup your workflow by exporting to the definition files. You can use the following code and put it into a Windows bat file in your PC to export all workflows. Of course you need to provide your parameters for username, password, server name and port number.
call "DefinitionExport.bat" -u <username> -p <password> -x <servername> -portnum <portnumber> -e "<servername>_Part1-%DATE%.def" -F -o "<servername>-Part1_%DATE%.log"
call "DefinitionExport.bat" -u <username> -p <password> -x <servername> -portnum <portnumber> -e "<servername>_Part2-%DATE%.def" -T -H -o "<servername>-Part2_%DATE%.log"
call "DefinitionExport.bat" -u <username> -p <password> -x <servername> -portnum <portnumber> -e "<servername>_Part3-%DATE%.def" -A -G -o "<servername>-Part3_%DATE%.log"
call "DefinitionExport.bat" -u <username> -p <password> -x <servername> -portnum <portnumber> -e "<servername>_Part4-%DATE%.def" -Q -M -B -D -N -o "<servername>-Part4_%DATE%.log"

Parameters telling which objects will be exported are explained (Integration Guide page 212).

How to find an AR System Field?

AR System usually does not provide you a simple method for field search. You need to enable "Object Relationships" in Server Information form. However this puts load on AR Server on large-scaled production environments and usually kept as disabled. What I would propose is to find the fields on database level. The query I use is below:
SELECT A.NAME, F.fieldname, f.fieldid, FC.pattern, fc.maxlength
FROM FIELD F, ARSCHEMA A, FIELD_CHAR FC
WHERE F.SCHEMAID = A.SCHEMAID 
AND (F.fieldid = fc.fieldid AND FC.schemaid = A.schemaid)
AND F.fieldname LIKE '%FieldName%'
order by A.name, F.fieldname
You should put the name of the field instead of "FieldName". The query is used only for character fields but you can adjust it according to the other field types. If you would like to search by Field ID, you can use the below query:
SELECT A.NAME, F.fieldname, f.fieldid, FC.pattern, fc.maxlength
FROM FIELD F, ARSCHEMA A, FIELD_CHAR FC
WHERE F.SCHEMAID = A.SCHEMAID 
AND (F.fieldid = fc.fieldid AND FC.schemaid = A.schemaid)
AND F.fieldid = 666666673
order by F.fieldname

Some Words on Scalability

As your system grows up, some tuning needs to be done. All Action Request System, application server and client parameters must be adjusted well in order to balance the load. AR System Server parameters are very important in that sense. Although BMC does not make any recommendations for a parameter set according to the user community scale, AR System Administrators should read the Configuration Guide carefully and must be aware of all parameters. One of the fatal parameters which can cause bottleneck on the system is "RPC-Non-Blocking-IO". This parameter is especially needs to be set if you experience odd server restarts, performance problems or if you are getting RPC errors from the server. As written in the Configuration Guide, this parameter enables the AR System on compliant systems to receive remote procedure calls in a non-blocking mode. You should set it in ar.conf file manually as below:
RPC-Non-Blocking-IO: T
We will touch scalability issues more in the future.

Monitoring Server Settings

If you are working with different teams on an AR System environment, you may need to be informed about the changes on server settings. As an example, one of your colleagues may have set the parameter "Maximum Entries Returned By GetList" which would cause a performance bottleneck on your system. Another example would be that one of the groups may have been deleted on the system which you are using on your form. In order to track these changes, I would recommend you to turn on "User/Group Changes" and "Server Setting Changes" on form "AR System Administration: Server Information".

Enabling this option, AR System would drop a footprint of the change to form "Server Events" and you can fetch this information whenever needed. The mapping for Event Types and Event Causes can be found in Configuration Guide.

Happy working! :)

Workflow Analysis with ARInside

AR System workflow is not easy to understand. You usually need to use the ARInside tool to make workflow analysis and find out how application behaves on either client or server side. ARInside is just like an overview of the workflow. If you open a form in ARInside, you can see which workflow is pushing to this form, which workflow is opening this form, references to this form and fields in the form. You can also see the relationships between the objects which provides you to track the workflow faster than Remedy Developer. For performance improvement tasks, you are able to track the indexes on the forms from ARInside.

AR System 7.5 Guides are available now!

Now you can access to the AR System 7.5 Guides from Useful Links.

AWR Reports

Checking the AWR reports regularly is a plus for administering AR System environments. If your business processes or workflows are complex, then you can get an AWR report from database system and identify the inefficient queries and bottlenecks of AR System. Important part of the AWR report for me is:
1) Query by Elapsed Time
2) Query by CPU Time
In the report, you can see how much DB query time a query takes and you can start to fix the query which takes the most. In Oracle databases, there is the Snapshot concept. Oracle regularly takes a snapshot of the database instance and stores it. You can generate an AWR report by giving two snapshot IDs, start and end. Probable statement for getting the AWR report would be:
SELECT output FROM TABLE (dbms_workload_repository.awr_report_text (<DB ID>, <InstanceID>, <BeginSnapID>, <EndSnapID>));

Monitoring of System Resources and Availability

If you have an AR System environment which consists of more than one server, it is important to monitor the system resources. An incident can happen at any time. AR Server or some other process may crash if physical server is out of resources. If you do not have enough input to investigate the root-cause, then you would wait till the same error occurs again. For this reason, it is worth to build any kind monitoring which is checking and logging the memory or resources for you. I am using a very simple shell script to check memory:


date >> /opt/remedy/node/ars5/db/check_memory.log
echo "______________________PRSTAT______________________________" >> /opt/remedy/node/ars5/db/check_memory.log
prstat -c 1 1 >> /opt/remedy/node/ars5/db/check_memory.log
echo "________________________TOP_______________________________" >> /opt/remedy/node/ars5/db/check_memory.log
top >> /opt/remedy/node/ars5/db/check_memory.log
echo "__________________________________________________________" >> /opt/remedy/node/ars5/db/check_memory.log
echo "__________________________________________________________" >> /opt/remedy/node/ars5/db/check_memory.log


For 32 bit AR Server processes, there is a limit of 4 GB of memory. It means it can grow to 4 GB memory. What I experienced is it tries to live with this 4 GB however if you force it or put more load on the server, then it would give a Segmentation Fault with Signal 11. For 64 bit AR Server process, BMC states that there is no memory limit for the process but my observation is that AR Server can not work efficiently if memory usage of 10 GB is exceeded. In the following articles, I will try to give you more hints regarding monitoring.

One Word on Maximizing

One of the important tips for efficient work is to maximize the Remedy User tool and the windows opening in the tool. Maximizing Remedy User provides you to see more space from your application world. If it is not maximized, you would not see a part of a form and every time you would need to click to some point to display it and this would put some time to the trash box. In the same manner, you have the ability to maximize the windows automatically by the settings as in the screenshot. In this way, you would see more fields and components on the forms.

Table Locking

One of the most important skills that you need to have is that you have to be familiar with the database. Please keep in mind that AR System is 100% database centric. Especially during the migrations and import operations, tools may lock AR System database tables, even your AR system or you may experience performance problems. In order to find whether any tables or objects are locked, you can use the following queries. Please keep in mind that these queries work for Oracle. You need to run these queries on the AR System database.

QUERY1
select
        C.OSUSER, c.sid, c.serial#, substr(object_name,1,30) OBJECT, c.username,
        substr(c.program,length(c.program)-10,length(c.program)) image,
        decode(b.type,
                'MR', 'Media Recovery',
                'RT', 'Redo Thread',
                'UN', 'User Name',
                'TX', 'Transaction',
                'TM', 'DML',
                'UL', 'PL/SQL User Lock',
                'DX', 'Distributed Xaction',
                'CF', 'Control File',
                'IS', 'Instance State',
                'FS', 'File Set',
                'IR', 'Instance Recovery',
                'ST', 'Disk Space Transaction',
                'TS', 'Temp Segment',
                'IV', 'Library Cache Invalidation',
                'LS', 'Log Start or Switch',
                'RW', 'Row Wait',
                'SQ', 'Sequence Number',
                'TE', 'Extend Table',
                'TT', 'Temp Table',
                b.type) lock_type,
        decode(b.lmode,
                0, 'None',           /* Mon Lock equivalent */
                1, 'Null',           /* NOT */
                2, 'Row-SELECT (SS)',     /* LIKE */
                3, 'Row-X (SX)',     /* R */
                4, 'Share',          /* SELECT */
                5, 'SELECT/Row-X (SSX)',  /* C */
                6, 'Exclusive',      /* X */
                to_char(b.lmode)) mode_held,
         decode(b.request,
                0, 'None',           /* Mon Lock equivalent */
                1, 'Null',           /* NOT */
                2, 'Row-SELECT (SS)',     /* LIKE */
                3, 'Row-X (SX)',     /* R */
                4, 'Share',          /* SELECT */
                5, 'SELECT/Row-X (SSX)',  /* C */
                6, 'Exclusive',      /* X */
                to_char(b.request)) mode_requested
from sys.dba_objects a, sys.v_$lock b, sys.v_$session c where
a.object_id = b.id1 and b.sid = c.sid and owner not in ('SYS','SYSTEM')

QUERY2
select     owner||'.'||object_name object, l.object_id
   ,oracle_username||' ('||s.status||')' oruser
   ,os_user_name osuser
   ,machine computer
   ,l.process unix
   ,s.sid sessionid,s.serial# serial
   ,r.name rollname
   ,to_char(s.logon_time,'yyyy/mm/dd hh24:mi:ss') time
from       v$locked_object l
   ,dba_objects o
   ,v$session s
   ,v$transaction t
   ,v$rollname r
where l.object_id = o.object_id
  and s.sid=l.session_id
  and s.taddr=t.addr
  and t.xidusn=r.usn
order by osuser, serial, object