Shell Scripts


Lately I have been working on a Dev Ops project where you get to handle environment based tasks on a daily basis. On a normal AEM project these things are often ignored , as in most of the firms there is a dedicated DevOps team. When you get bombarded with lots of tasks daily , you evolve your work style to do things smartly and quickly using Scripts.  I thought of sharing few of the useful scripts here so that you can also use them in your day to day work without much hassle. Secondary aim of this blog post is to expose you to Shell scripts , if you are not using it yet and give you an inspiration to build your own script one day and share with everyone .  May you do things in a better fashion and with superfast speed.


  • Dispatcher Cache Clear This script could be used/modified for Configuring the Dispatcher Cache clear process . Execution of the script expects 2 parameters – command and project-name. To find all the files which needs to be cleared ‘find’ command is used with a regex. This ‘find’ operation is quite useful in other applications too, read more about it.
#!/bin/bash

if [ "$1" == "" -o "$2" == "" ]; then
  echo "usage: use 'ls' to list files, 'rm' to remove them for first parameter"
  echo "usage: add domain path in second  parameter"
  echo "e.g : ./clearCacheScript ls projectx"
  exit
elif [ "$1" == "ls" ]; then
  export cmd='find  -regex ".*\.\(html\|css\|jpg\|js\|gif\|png\|json\|zip\)"'
elif [ "$1" == "rm" ]; then
  export cmd='find  -regex ".*\.\(html\|css\|jpg\|js\|gif\|png\|json\|zip\)" -delete'
fi

echo $cmd

export dir="/mnt/var/www/etc/clientlibs/"$2
echo '*******Deleting files in' $dir
cd $dir;
eval $cmd;

export dir="/mnt/var/www/etc/designs/"$2
echo '*******Deleting files in' $dir
cd $dir;
eval $cmd;

export dir="/mnt/var/www/content/dam/"$2
echo '******Deleting files in' $dir
cd $dir;
eval $cmd;

export dir="/mnt/var/www/content/"$2
echo '******Deleting files in' $dir
cd $dir;
eval $cmd;


  • Thread Dumps For any performance based issue in AEM , taking a thread dump is like the first step to do. Thankfully helpx.adobe.com provides 2 excellent ways to take thread dumps :
  1. Using a Shell Script
  2. Thread Dump Collection and Analysis tool

The precondition here is that jstack should be configured on your AEM instance. Pasting the script from helpx.adobe.com for quick reference :

#!/bin/bash
if [ $# -eq 0 ]; then
echo >&2 "Usage: jstackSeries <pid> <run_user> [ <count> [ <delay> ] ]"
echo >&2 "    Defaults: count = 10, delay = 0.5 (seconds)"
exit 1
fi
pid=$1          # required
user=$2         # required
count=${3:-10}  # defaults to 10 times
delay=${4:-0.5} # defaults to 0.5 seconds
while [ $count -gt 0 ]
do
sudo -u $user top -H -b -n1 -p $pid >top.$pid.$(date +%H%M%S.%N) &
sudo -u $user jstack -l $pid >jstack.$pid.$(date +%H%M%S.%N)
sleep $delay
let count--
echo -n "."
done

  • Log Review This script might come handy when there is a need to find all the errors / warnings in log files , fetch a report out of it and sort out the errors/warnings based on their frequency of occurrence. It can also be used to process and monitor Request logs or access logs. Notice the application of grep and awk in this script. They are also quite useful commands.
#!/bin/bash

LOG_FOLDER=/Users/hakhan/Documents/AdobeLaptop/Work/AEMInstances/AEM6.1/crx-quickstart/logs

if [ "$1" == "" -o "$2" == "" ]; then
echo "1. usage: add type of logfile to be reviewed : error or access or request"
echo "2. usage: add regex to be used for the log file"
echo "e.g : ./logReviewScript error error.log*"
exit
fi

# Checks if the server is an Author or Publisher. This is not used here, but you can use this in future.
#PUB_OR_AUTH=`cat /etc/sysconfig/cq5 | grep -i R_AUTHOR`
#
#if [ $PUB_OR_AUTH == "R_AUTHOR=1" ]; then
# AEM_TYPE='author'
# cd /mnt/crx/author/crx-quickstart/logs
#else
# AEM_TYPE='publish'
# cd /mnt/crx/publish/crx-quickstart/logs
#fi

cd $LOG_FOLDER

echo "Loading..."
if [ "$1" == "error" ]; then
grep "*WARN*" $2 | awk '{print $1 " , " $3 " , " $5 " , " substr($0,index($0,$6))}'| sort | uniq -c | sort -nr > warnLogReview.csv
grep "*ERROR*" $2 | awk '{print $1 " , " $3 " , " $5 " , " substr($0,index($0,$6))}'| sort | uniq -c | sort -nr > errorLogReview.csv
echo "Completed Error/ Warning Log Review for files :: " $2
elif [ "$1" == "access" ]; then
awk '{print $5 " , " $6 "," $8 ","$7 "," $9 "," $10 "," $12}' access.log | sort | uniq -c | sort -nr > accessLogReview.csv
echo "Completed Access Log Review for files :: " $2
elif [ "$1" == "request" ]; then
awk '{id=$3 ;method=$5 ; path=$6 ; getline ; id2=$3; status=$5; type=$6; time=$7 ; print id "," method "," path "," "\n" id2 "," status "," type "," time ;}' $2 | awk ' { gsub ( /\[/, "" ) ; print } ' | awk ' { gsub ( /\]/, "" ) ; print } ' | sort -g > tempFile.csv
awk -F "," 's != $1 || NR ==1{s=$1;if(p){print p};p=$0;next} {sub($1,"",$0);p=p""$0;}END{print p}' tempFile.csv > requestLogReview.csv
echo "Completed Request Log Review for files :: " $2
fi

  • AEM Packager The AEM-Packager script deals with most of the use cases for package management in AEM. Its quite handy and simple to use. You can do a variety of operations using this script . All the actions here make you of cURL commands.
#!/bin/bash

# Default Variables
USER="admin"
PASSWORD="admin"
HOST="http://localhost:4502"

function usage
{
echo "usage: ./aem-packager.sh [list|install|uninstall|download|upload|upload_install|build|delete] [groupname|localpath when uploading] [packagename]"
}

list ()
{
echo "AVAILABLE PACKAGES:"
curl -u $USER:$PASSWORD $HOST/crx/packmgr/service.jsp?cmd=ls > input.xml
awk ' { gsub ( /\</, "" ) ; print } ' input.xml | awk '{sub(/\/.*/,""); print}' | cut -d\> -f2- | awk ' { gsub ( /\,/, "" ) ; print } ' > temp.csv
awk ' NR>6 { line1 =$1 ; getline ; line2=$1 ; getline ; line3=$1 ; getline ; line4 =$1 ; getline ; line5=$1 ; getline ; line6=$1 ; getline ; line7 =$1 ; getline ; line8=$1$2$3$4 ; getline ; line9=$1 ; getline ; line10 =$1$2$3$4 ; getline ; line11 =$1 ; getline ; line12 =$1$2$3$4 ; getline ; line13 =$1 ; print line1 "," line2 "," line3 "," line4 "," line5 "," line6 "," line7 "," line8 "," line9 "," line10 "," line11 "," line12 "," line13 ;}' temp.csv > aem-packages.csv
echo " Package List is Stored in the CSV File aem-packages.csv "
}

install_package ()
{
echo "Insalling PACKAGE: "$1:$2

if [ "$1" == "" -o "$2" == "" ]; then
echo "Please add Group Name and Package name for the package "
echo "e.g : ./aem-packager.sh install groupname packagename.zip"
exit
fi
curl -u $USER:$PASSWORD -X POST --fail "$HOST/crx/packmgr/service/.json/etc/packages/$1/$2?cmd=install"
echo ""
}

uninstall_package ()
{
echo "UnInstalling PACKAGE: "$1:$2

if [ "$1" == "" -o "$2" == "" ]; then
echo "Please add Group Name and Package name for the package "
echo "e.g : ./aem-packager.sh uninstall groupname packagename.zip"
exit
fi
curl -u $USER:$PASSWORD -X POST --fail "$HOST/crx/packmgr/service/.json/etc/packages/$1/$2?cmd=uninstall"
echo ""
}

upload_install ()
{
echo "Upload and Install PACKAGE from : " $1
if [ "$1" == ""]; then
echo "Please add package path "
echo "e.g : ./aem-packager.sh upload_install /Users/hakhan/Downloads/sample-2.0.zip "
exit
fi
curl -u $USER:$PASSWORD -F file=@"$1" -F force=true -F install=true $HOST/crx/packmgr/service.jsp
echo ""
}

upload ()
{
echo "Upload and Not Install PACKAGE : " $1
if [ "$1" == ""]; then
echo "Please add package path "
echo "e.g : ./aem-packager.sh upload /Users/hakhan/Downloads/sample-2.0.zip "
exit
fi
curl -u $USER:$PASSWORD -F file=@"$1" -F force=true -F install=false $HOST/crx/packmgr/service.jsp
echo ""
}

download ()
{
echo "Downloading PACKAGE: "$1:$2 "to path : " $3

if [ "$1" == "" -o "$2" == "" -o "$3" == "" ]; then
echo "Please add Group Name and Package name and Destination path "
echo "e.g : ./aem-packager.sh download groupname packagename.zip <path of folder>"
exit
fi
curl -u $USER:$PASSWORD $HOST/etc/packages/$1/$2 > $3
echo ""
}

build ()
{
echo "Building PACKAGE: "$1:$2

if [ "$1" == "" -o "$2" == "" ]; then
echo "Please add Group Name and Package name "
echo "e.g : ./aem-packager.sh build groupname packagename.zip"
exit
fi
curl -u $USER:$PASSWORD -X POST --fail "$HOST/crx/packmgr/service/.json/etc/packages/$1/$2?cmd=build"
echo ""
}

delete ()
{
echo "Deleting PACKAGE: "$1:$2

if [ "$1" == "" -o "$2" == "" ]; then
echo "Please add Group Name and Package name "
echo "e.g : ./aem-packager.sh delete groupname packagename.zip"
exit
fi
curl -u $USER:$PASSWORD -X POST --fail "$HOST/crx/packmgr/service/.json/etc/packages/$1/$2?cmd=delete"
echo ""
}

# Perform the actions
if [ "$1" = "list" ]; then
list
elif [ "$1" = "install" ] ; then
install_package $2 $3
elif [ "$1" = "upload_install" ] ; then
upload_install $2
elif [ "$1" = "upload" ] ; then
upload $2
elif [ "$1" = "download" ] ; then
download
elif [ "$1" = "build" ] ; then
build $2 $3
elif [ "$1" = "delete" ] ; then
delete $2 $3
elif [ "$1" = "uninstall" ] ; then
uninstall_package $2 $3
else
usage
fi

  • Request Log GraphRequest logs are targetted whenever performance of a site is monitored. You might be aware of using rlog.jar or more advanced tools like – AppDynamics , New Relic etc.  A much simpler and graphical way to plot the request log is by using a script by Jörg Hoh . You can copy the files from the GitHub link  and feed a request.log file to it. (Precondition is that you should have a perl module installed in your system . )

output

It gives the Graphical plot for all the requests in the log file. eg. If you will see in the above graphplot , you can target the requests around 16:00 hours to find out why there is a peak present in logs at that time. Furthermore, you can check what all processes are scheduled during that time-frame or check if the User activity is increased suddenly resulting in increase of response time. Thus the plot gave you a quick hint of what might be the reason of slow response of environment.

I hope the above scripts will be quite useful to you while working on AEM. Please feel free to use them or modify them as per your need. If you wish to add any other useful script do let me know.

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s