Visibility Concept in AR System

In AR System, there exists a core functionality on Visibility of the entries in a form. As a base form AR System has Groups form that can be assigned to the users.

If successfully managed, AR System reserved fields Assignee Group and Dynamic Groups that range between 60000 to 60999 can be used to restrict access to the form entries. These groups will also be shown in the Permission Lists of the fields. So restricting the access can be done by giving permission to the Request ID($1$) field just the Assignee Group and the Dynamic Groups. The trick is to have fields in the form that we are applying our configuration with the group id as field id.

This case is useful when there exists two types of users as Write Permission and Read Permission. Let's make an example: Assume we have a form as "ABC Form". This form has all the required fields. We also have 2 distinct users who has only READ and who has only WRITE as Assignee group in User form.

1) We need to create two groups in the Group form as WriteSecurity, ReadSecurity assuming 60512 and 60513, respectively.

2) We need to create two more groups as READ 50810 and WRITE 50820.

3) We open our form "ABC Form" and edit the permissions of the Request ID($1$) and remove all the permissions and add only the ReadSecurity and WriteSecurity by giving the ReadSecurity only View Permission and WriteSecurity by giving Change Permission.

4) Add two fields as ReadSecurity and WriteSecurity with Field IDs 60512 and 60513, respectively.

5) Save the form.

6) Create a submit filter as when an entry is created the WriteSecurity field is set as WRITE groups ID and ReadSecurity is set as READ groups ID.

7) The users with READ and WRITE can see the form entries, the others can not and Read can only view and Write can modify the entries.

by Alper Gülbahar

Using Private Queues

Private queues are used to define a separate path for a request to the AR Server. If you have some script or API code reaching your AR Server, you may need to define a separate queue not to be blocked by the load of the AR Server. You can define a private queue under Server Information form -> Ports and Queues tab. Here you can state maximum and minimum threads to be used for this queue. When AR Server is started, private queue starts with the number of minimum threads and increase number of threads (if needed) till the maximum number of threads is reached. If you do not state in your API calls, Fast (390620) and List (390635) queues are used by default. In order to use a private queue (390654 in our case), you need to use the following schema in your code:

if (ARSetServerPort(&control, control.server, 4000, 390654, &arStatus) >= AR_RETURN_ERROR)
{
sprintf (ErrMsg, "Fatal error - changing the queue 390654 for user '%s' on server '%s'", control.user, control.server);
ERROR (ERR_FATAL, ProcessName, ErrMsg, 0);
LogArsStatus (&arStatus);
FreeARStatusList(&arStatus, FALSE);
return;
}
FreeARStatusList(&arStatus, FALSE);

Helpful Unix Commands

Now I would like to give you the Unix commands which I use most of the time:

1) Search for a term in all the files in the current folder:
find . -type f -exec grep -l "search_term" '{}' \;

2) Find files in the current folder whose size is greater than 300K:
find . -size +300000k

3) Exchange xxx with yyy in all files ending with Temp.sh:
perl -pi -e 's/xxx/yyy/g' *Temp.sh

4) Cut lines between 125 and 127 from testfile (you can also redirect it to another file):
sed -n '/125/,/127/p' testfile

5) Kill all processes which includes xxx in its name:
kill -9 `ps -ef | grep "xxx" | grep -v grep | awk '{print $2}'`

Happy working!!

Importing Data to an AR System Form on Database Level

You have more than one option to load data into an AR System form. You can use Remedy Import tool however you may need to build data on database level with a query for some data fix or administration tasks. E.g. There might be some number of corrupted tickets on your system and you may need to close them via an escalation. (The reason to use an escalation here is that tickets need to be closed by the workflow properly). First you have to create a regular AR System form, let's say "zz_Data_Fix" with default core fields plus character field named "ticket_number". Here I will also use a temporary table named zz_Intermediate for a better understanding. You populate this intermediate table with the query where you select the tickets which needs to be closed:

create table zz_Intermediate as
select ticket_number from mytickets where <condition for selecting tickets to be closed>

Here "mytickets" is the database table where you keep all your ticket data. You can also use joins in this query. Then you import ticket numbers into your zz_Data_Fix form (in order to find the schemaid of zz_Data_Fix, you can use the following: select schemaid from arschema where name = 'zz_Data_Fix'):

insert into zz_data_fix (request_id, ticket_number, create_date, last_modified_by, modified_date, status)
select rownum, ticket_number, 1, 'username', 1, 0 from zz_Intermediate


With running this query, records are loaded into the T table which keeps the data in AR System environment. (You have the option to directly import to T table but this way seems easier for me). However you also need to load H table which keeps status history info of every record. If you do not do that, you may not see your record in Remedy User.


insert into H1413 (entryid)
select C1 from T1413

Please note that form zz_Data_Fix is created from scratch therefore both T1413 and H1413 are empty. Some H tables might have different structures so you may need to adjust your query a little. After committing the above queries, you should see your ticket numbers when you make an unqualified search in Remedy User zz_Data_Fix form.

As the last step, you can write an escalation on zz_Data_Fix and make a Push Fields to your original ticket form with a PushFields if condition: $ticket_number$ = 'ticket_number' and modify any field you want.

Data Archiving

As the data grows in your AR System environment, you have to archive data for mainly performance reasons. For instance, it is not logical to keep 5 year-old ticket in ticket form. This data can be archived and can be accessed whenever needed, esp. for reporting. AR System provides a built-in archiving mechanism. It is very simple to set up archiving and it is very similar to create an escalation object.

The mechanism behind archiving is actually the same logic with escalations. In Remedy Developer, you open a form, then go to Main Menu -> Form -> Form Properties -> Archive. The best option for archiving would be "Copy to Archive and Delete from Source".  Archive state should be "Enable". If you define the archiving for the first time, AR System will create the archive form automatically. There are some options for archiving attachments and diary fields. Time definition is quite straight-forward. The most important part is the qualification. You can use something like:
"$LastModifiedDate$ < $TIMESTAMP$ -86400". This will archive all records whose last modified date is older than 7 days. One of the important tips is that you are allowed to define indexes on archive forms as you do in Regular forms. If you have a reporting based on archive forms, this can be very helpful.

Built-in archiving mechanism however is not always the best. When the archive form is created automatically by AR System, its type is "Archive". In this form, AR System puts some new fields like "Original Request ID" and "Original Create Date" not to lose record creation information. During the migrations you would get in trouble if you do not maintain the archive form whenever you modify the original form. AR System sometimes gives errors when you import archive form. Another disadvantage is that built-in mechanism archives all the fields. This enlarges the archive form and you may not need all the fields for your reporting purposes. Another proposal would be to implement your own archiving with escalations. You can define archiving forms manually and you can transfer the records between forms via DSO. In this way, you can choose the fields which you would like to archive. Migrations will be easier since the form type of the archiving forms will be normal "Regular" form.

Interpreting Log Files

Log files have great importance in AR System since they help you to identify the root-cause of an incident. Once you have a production release, logs might be full of errors but you have to order these errors according to their impact and then solve one by one till you have a clean log file. The most important error is that there some sort of loop on the system. You can see this in the log file as "Too many filter operations...". Once you see this, your system is in trouble and this error must be solved as soon as possible. In this case, you can turn on filter and escalation log and find the reason for this error. Another fatal error is the one related with licenses. If you do not have enough license, your system might behave abnormal, e.g. you would not create a licensed user.

Another point in log analysis is that AR System might throw the same error multiple times which cause a big log file. In order to read it, you have to go through many pages which is sometimes impossible. For this case, I recommend you to write a simple log parser, e.g. in Java, which goes through the whole log file and brings you distinct errors. In this way, your logs will be more readable.

It is also important to rotate the logs because they get larger everyday. It is a good idea to rotate logs, esp. arerror.log on daily basis. In this way, you can find errors easier when you have an incident. BMC always requests log files when you open a ticket to them. Having daily logs will also be an advantage because you can submit smaller logs to BMC. We use the following dummy shell script for rotation and zipping. You can adapt it to your environment:

#!/bin/sh
#------------------Rotating and Zipping 'XXX_error.log'--------------#
DAY=`date +%d`
MONTH=`date +%m`
YEAR=`date +%Y`
#------------------Rotation of 'XXX_error.log'----------------#
LOGFILEJ=api_error.log
for dir in <directory>
do
        if [ -x $dir ]; then
                cd $dir
                mv $LOGFILEJ $LOGFILEJ.$YEAR$MONTH$DAY
                echo "-------------------------------------------------"        
              echo "$YEAR-$MONTH-$DAY : '$LOGFILEJ' file is rotated";
        else
              echo "-------------------------------------------------"
                echo "$YEAR-$MONTH-$DAY : ERROR:'$LOGFILEJ' file cannot be rotated!";
        fi
done
#------------------Zipping of 'api_error.log'----------------#
for dir3 in  <directory>
do
        if [ -x $dir3 ]; then
                 cd $dir3
               gzip -c $LOGFILEJ.$YEAR$MONTH$DAY  > $LOGFILEJ.$YEAR$MONTH$DAY.gz
               rm $LOGFILEJ.$YEAR$MONTH$DAY
               mv $LOGFILEJ.$YEAR$MONTH$DAY.gz api_error/$LOGFILEJ.$YEAR$MONTH$DAY.gz
                 echo "-------------------------------------------------"       
               echo "$YEAR-$MONTH-$DAY : '$LOGFILEJ' file is zipped.";
        else
               echo "-------------------------------------------------"
                 echo "$YEAR-$MONTH-$DAY : ERROR:'$LOGFILEJ' file cannot be zipped!";
        fi
done
exit

Storing Configuration Files

Having the correct configuration is always a problem in AR System. You need to configure your ar.conf file according to the needs of your system. And once you have a stable system, it is worth to store this configuration file in some version control system, e.g. SVN or ClearCase. We experienced in the past that ar.conf was modified out of our control. We could not find whether someone changed this or file is corrupted by the system. The queue settings were removed and DSO password was changed. What can you do in these cases? If you have a strange problem on the system, (AR Server crashes or DSO does not work), please check the ar.conf file first and compare it with the previous version. This will definitely help you to detect whether your ar.conf file is correctly configured.

Backup Your AR System Workflow

You need to make sure that your AR System workflow is always secure because it is the code that you produce and you can not stand losing it any time. While we were in school, one of the mates was deleting the C code and then shouting with a fear on his face :) This is why backuping is always important in IT, whatever you develop. AR System provides you backup your workflow by exporting to the definition files. You can use the following code and put it into a Windows bat file in your PC to export all workflows. Of course you need to provide your parameters for username, password, server name and port number.
call "DefinitionExport.bat" -u <username> -p <password> -x <servername> -portnum <portnumber> -e "<servername>_Part1-%DATE%.def" -F -o "<servername>-Part1_%DATE%.log"
call "DefinitionExport.bat" -u <username> -p <password> -x <servername> -portnum <portnumber> -e "<servername>_Part2-%DATE%.def" -T -H -o "<servername>-Part2_%DATE%.log"
call "DefinitionExport.bat" -u <username> -p <password> -x <servername> -portnum <portnumber> -e "<servername>_Part3-%DATE%.def" -A -G -o "<servername>-Part3_%DATE%.log"
call "DefinitionExport.bat" -u <username> -p <password> -x <servername> -portnum <portnumber> -e "<servername>_Part4-%DATE%.def" -Q -M -B -D -N -o "<servername>-Part4_%DATE%.log"

Parameters telling which objects will be exported are explained (Integration Guide page 212).

How to find an AR System Field?

AR System usually does not provide you a simple method for field search. You need to enable "Object Relationships" in Server Information form. However this puts load on AR Server on large-scaled production environments and usually kept as disabled. What I would propose is to find the fields on database level. The query I use is below:
SELECT A.NAME, F.fieldname, f.fieldid, FC.pattern, fc.maxlength
FROM FIELD F, ARSCHEMA A, FIELD_CHAR FC
WHERE F.SCHEMAID = A.SCHEMAID 
AND (F.fieldid = fc.fieldid AND FC.schemaid = A.schemaid)
AND F.fieldname LIKE '%FieldName%'
order by A.name, F.fieldname
You should put the name of the field instead of "FieldName". The query is used only for character fields but you can adjust it according to the other field types. If you would like to search by Field ID, you can use the below query:
SELECT A.NAME, F.fieldname, f.fieldid, FC.pattern, fc.maxlength
FROM FIELD F, ARSCHEMA A, FIELD_CHAR FC
WHERE F.SCHEMAID = A.SCHEMAID 
AND (F.fieldid = fc.fieldid AND FC.schemaid = A.schemaid)
AND F.fieldid = 666666673
order by F.fieldname

Some Words on Scalability

As your system grows up, some tuning needs to be done. All Action Request System, application server and client parameters must be adjusted well in order to balance the load. AR System Server parameters are very important in that sense. Although BMC does not make any recommendations for a parameter set according to the user community scale, AR System Administrators should read the Configuration Guide carefully and must be aware of all parameters. One of the fatal parameters which can cause bottleneck on the system is "RPC-Non-Blocking-IO". This parameter is especially needs to be set if you experience odd server restarts, performance problems or if you are getting RPC errors from the server. As written in the Configuration Guide, this parameter enables the AR System on compliant systems to receive remote procedure calls in a non-blocking mode. You should set it in ar.conf file manually as below:
RPC-Non-Blocking-IO: T
We will touch scalability issues more in the future.

Monitoring Server Settings

If you are working with different teams on an AR System environment, you may need to be informed about the changes on server settings. As an example, one of your colleagues may have set the parameter "Maximum Entries Returned By GetList" which would cause a performance bottleneck on your system. Another example would be that one of the groups may have been deleted on the system which you are using on your form. In order to track these changes, I would recommend you to turn on "User/Group Changes" and "Server Setting Changes" on form "AR System Administration: Server Information".

Enabling this option, AR System would drop a footprint of the change to form "Server Events" and you can fetch this information whenever needed. The mapping for Event Types and Event Causes can be found in Configuration Guide.

Happy working! :)

Workflow Analysis with ARInside

AR System workflow is not easy to understand. You usually need to use the ARInside tool to make workflow analysis and find out how application behaves on either client or server side. ARInside is just like an overview of the workflow. If you open a form in ARInside, you can see which workflow is pushing to this form, which workflow is opening this form, references to this form and fields in the form. You can also see the relationships between the objects which provides you to track the workflow faster than Remedy Developer. For performance improvement tasks, you are able to track the indexes on the forms from ARInside.

AR System 7.5 Guides are available now!

Now you can access to the AR System 7.5 Guides from Useful Links.

AWR Reports

Checking the AWR reports regularly is a plus for administering AR System environments. If your business processes or workflows are complex, then you can get an AWR report from database system and identify the inefficient queries and bottlenecks of AR System. Important part of the AWR report for me is:
1) Query by Elapsed Time
2) Query by CPU Time
In the report, you can see how much DB query time a query takes and you can start to fix the query which takes the most. In Oracle databases, there is the Snapshot concept. Oracle regularly takes a snapshot of the database instance and stores it. You can generate an AWR report by giving two snapshot IDs, start and end. Probable statement for getting the AWR report would be:
SELECT output FROM TABLE (dbms_workload_repository.awr_report_text (<DB ID>, <InstanceID>, <BeginSnapID>, <EndSnapID>));

Monitoring of System Resources and Availability

If you have an AR System environment which consists of more than one server, it is important to monitor the system resources. An incident can happen at any time. AR Server or some other process may crash if physical server is out of resources. If you do not have enough input to investigate the root-cause, then you would wait till the same error occurs again. For this reason, it is worth to build any kind monitoring which is checking and logging the memory or resources for you. I am using a very simple shell script to check memory:


date >> /opt/remedy/node/ars5/db/check_memory.log
echo "______________________PRSTAT______________________________" >> /opt/remedy/node/ars5/db/check_memory.log
prstat -c 1 1 >> /opt/remedy/node/ars5/db/check_memory.log
echo "________________________TOP_______________________________" >> /opt/remedy/node/ars5/db/check_memory.log
top >> /opt/remedy/node/ars5/db/check_memory.log
echo "__________________________________________________________" >> /opt/remedy/node/ars5/db/check_memory.log
echo "__________________________________________________________" >> /opt/remedy/node/ars5/db/check_memory.log


For 32 bit AR Server processes, there is a limit of 4 GB of memory. It means it can grow to 4 GB memory. What I experienced is it tries to live with this 4 GB however if you force it or put more load on the server, then it would give a Segmentation Fault with Signal 11. For 64 bit AR Server process, BMC states that there is no memory limit for the process but my observation is that AR Server can not work efficiently if memory usage of 10 GB is exceeded. In the following articles, I will try to give you more hints regarding monitoring.

One Word on Maximizing

One of the important tips for efficient work is to maximize the Remedy User tool and the windows opening in the tool. Maximizing Remedy User provides you to see more space from your application world. If it is not maximized, you would not see a part of a form and every time you would need to click to some point to display it and this would put some time to the trash box. In the same manner, you have the ability to maximize the windows automatically by the settings as in the screenshot. In this way, you would see more fields and components on the forms.

Table Locking

One of the most important skills that you need to have is that you have to be familiar with the database. Please keep in mind that AR System is 100% database centric. Especially during the migrations and import operations, tools may lock AR System database tables, even your AR system or you may experience performance problems. In order to find whether any tables or objects are locked, you can use the following queries. Please keep in mind that these queries work for Oracle. You need to run these queries on the AR System database.

QUERY1
select
        C.OSUSER, c.sid, c.serial#, substr(object_name,1,30) OBJECT, c.username,
        substr(c.program,length(c.program)-10,length(c.program)) image,
        decode(b.type,
                'MR', 'Media Recovery',
                'RT', 'Redo Thread',
                'UN', 'User Name',
                'TX', 'Transaction',
                'TM', 'DML',
                'UL', 'PL/SQL User Lock',
                'DX', 'Distributed Xaction',
                'CF', 'Control File',
                'IS', 'Instance State',
                'FS', 'File Set',
                'IR', 'Instance Recovery',
                'ST', 'Disk Space Transaction',
                'TS', 'Temp Segment',
                'IV', 'Library Cache Invalidation',
                'LS', 'Log Start or Switch',
                'RW', 'Row Wait',
                'SQ', 'Sequence Number',
                'TE', 'Extend Table',
                'TT', 'Temp Table',
                b.type) lock_type,
        decode(b.lmode,
                0, 'None',           /* Mon Lock equivalent */
                1, 'Null',           /* NOT */
                2, 'Row-SELECT (SS)',     /* LIKE */
                3, 'Row-X (SX)',     /* R */
                4, 'Share',          /* SELECT */
                5, 'SELECT/Row-X (SSX)',  /* C */
                6, 'Exclusive',      /* X */
                to_char(b.lmode)) mode_held,
         decode(b.request,
                0, 'None',           /* Mon Lock equivalent */
                1, 'Null',           /* NOT */
                2, 'Row-SELECT (SS)',     /* LIKE */
                3, 'Row-X (SX)',     /* R */
                4, 'Share',          /* SELECT */
                5, 'SELECT/Row-X (SSX)',  /* C */
                6, 'Exclusive',      /* X */
                to_char(b.request)) mode_requested
from sys.dba_objects a, sys.v_$lock b, sys.v_$session c where
a.object_id = b.id1 and b.sid = c.sid and owner not in ('SYS','SYSTEM')

QUERY2
select     owner||'.'||object_name object, l.object_id
   ,oracle_username||' ('||s.status||')' oruser
   ,os_user_name osuser
   ,machine computer
   ,l.process unix
   ,s.sid sessionid,s.serial# serial
   ,r.name rollname
   ,to_char(s.logon_time,'yyyy/mm/dd hh24:mi:ss') time
from       v$locked_object l
   ,dba_objects o
   ,v$session s
   ,v$transaction t
   ,v$rollname r
where l.object_id = o.object_id
  and s.sid=l.session_id
  and s.taddr=t.addr
  and t.xidusn=r.usn
order by osuser, serial, object

Main Components

Maybe it is worth to start with main components. AR System is a server-client environment and composed of two components: AR Server and AR Client.

AR Server is responsible for processing the server requests. Do not  even think to write your own AR Server!! :) For sure it is optimized. AR Client is the tool where you can use the application built by AR System. (Most probably it will be an ITSM application). AR Client sends the user requests to the AR Server and gets the response back. Of course BMC has made it compatible to many environments and provided choices. If you would like to use it from your PC, Remedy User tool would fit for your purpose. It is a Windows based client tool. You also have the option to use Remedy Midtier to use your application from web. These two tools are typical AR Client tools.

AR System generally has more than one server: Node and Race. Node server is the one where you generally run your workflow and on Race server, you keep the configuration data and archive data. You usually need a synchronization between these two servers because you would use the configuration data on node server which is your main server. Or you may need to archive some data from node to race for a better performance. For these reasons, BMC invented DSO server which does this synchronization.

Remedy Developer is used for designing forms and workflow. All application design and application rules are made with using Remedy Developer.

For data importing, AR System provides a separate tool AR Import. With this tool, you can import form data easily and you do not need to consider database issues.

Welcome from Author

After 3 years of experience of ARS, I would like to share some tips here. Although you use an high level framework, it would sometimes be really hard to configure ARS or DO what you want. Sometimes you would need a magic hand touching on your shoulder and saying "Read the guides!!" :) Using this blog, I aim to share what we lived and how we managed. See you soon..