Lately I have been working on a Dev Ops project where you get to handle environment based tasks on a daily basis. On a normal AEM project these things are often ignored , as in most of the firms there is a dedicated DevOps team. When you get bombarded with lots of tasks daily , you evolve your work style to do things smartly and quickly using Scripts. I thought of sharing few of the useful scripts here so that you can also use them in your day to day work without much hassle. Secondary aim of this blog post is to expose you to Shell scripts , if you are not using it yet and give you an inspiration to build your own script one day and share with everyone . May you do things in a better fashion and with superfast speed.
- Dispatcher Cache Clear : This script could be used/modified for Configuring the Dispatcher Cache clear process . Execution of the script expects 2 parameters – command and project-name. To find all the files which needs to be cleared ‘find’ command is used with a regex. This ‘find’ operation is quite useful in other applications too, read more about it.
#!/bin/bash if [ "$1" == "" -o "$2" == "" ]; then echo "usage: use 'ls' to list files, 'rm' to remove them for first parameter" echo "usage: add domain path in second parameter" echo "e.g : ./clearCacheScript ls projectx" exit elif [ "$1" == "ls" ]; then export cmd='find -regex ".*\.\(html\|css\|jpg\|js\|gif\|png\|json\|zip\)"' elif [ "$1" == "rm" ]; then export cmd='find -regex ".*\.\(html\|css\|jpg\|js\|gif\|png\|json\|zip\)" -delete' fi echo $cmd export dir="/mnt/var/www/etc/clientlibs/"$2 echo '*******Deleting files in' $dir cd $dir; eval $cmd; export dir="/mnt/var/www/etc/designs/"$2 echo '*******Deleting files in' $dir cd $dir; eval $cmd; export dir="/mnt/var/www/content/dam/"$2 echo '******Deleting files in' $dir cd $dir; eval $cmd; export dir="/mnt/var/www/content/"$2 echo '******Deleting files in' $dir cd $dir; eval $cmd;
- Thread Dumps : For any performance based issue in AEM , taking a thread dump is like the first step to do. Thankfully helpx.adobe.com provides 2 excellent ways to take thread dumps :
The precondition here is that jstack should be configured on your AEM instance. Pasting the script from helpx.adobe.com for quick reference :
#!/bin/bash
if [ $# -eq 0 ]; then
echo >&2 "Usage: jstackSeries <pid> <run_user> [ <count> [ <delay> ] ]"
echo >&2 "    Defaults: count = 10, delay = 0.5 (seconds)"
exit 1
fi
pid=$1          # required
user=$2         # required
count=${3:-10}  # defaults to 10 times
delay=${4:-0.5} # defaults to 0.5 seconds
while [ $count -gt 0 ]
do
sudo -u $user top -H -b -n1 -p $pid >top.$pid.$(date +%H%M%S.%N) &
sudo -u $user jstack -l $pid >jstack.$pid.$(date +%H%M%S.%N)
sleep $delay
let count--
echo -n "."
done
- Log Review : This script might come handy when there is a need to find all the errors / warnings in log files , fetch a report out of it and sort out the errors/warnings based on their frequency of occurrence. It can also be used to process and monitor Request logs or access logs. Notice the application of grep and awk in this script. They are also quite useful commands.
#!/bin/bash
LOG_FOLDER=/Users/hakhan/Documents/AdobeLaptop/Work/AEMInstances/AEM6.1/crx-quickstart/logs
if [ "$1" == "" -o "$2" == "" ]; then
echo "1. usage: add type of logfile to be reviewed : error or access or request"
echo "2. usage: add regex to be used for the log file"
echo "e.g : ./logReviewScript error error.log*"
exit
fi
# Checks if the server is an Author or Publisher. This is not used here, but you can use this in future.
#PUB_OR_AUTH=`cat /etc/sysconfig/cq5 | grep -i R_AUTHOR`
#
#if [ $PUB_OR_AUTH == "R_AUTHOR=1" ]; then
# AEM_TYPE='author'
# cd /mnt/crx/author/crx-quickstart/logs
#else
# AEM_TYPE='publish'
# cd /mnt/crx/publish/crx-quickstart/logs
#fi
cd $LOG_FOLDER
echo "Loading..."
if [ "$1" == "error" ]; then
grep "*WARN*" $2 | awk '{print $1 " , " $3 " , " $5 " , " substr($0,index($0,$6))}'| sort | uniq -c | sort -nr > warnLogReview.csv
grep "*ERROR*" $2 | awk '{print $1 " , " $3 " , " $5 " , " substr($0,index($0,$6))}'| sort | uniq -c | sort -nr > errorLogReview.csv
echo "Completed Error/ Warning Log Review for files :: " $2
elif [ "$1" == "access" ]; then
awk '{print $5 " , " $6 "," $8 ","$7 "," $9 "," $10 "," $12}' access.log | sort | uniq -c | sort -nr > accessLogReview.csv
echo "Completed Access Log Review for files :: " $2
elif [ "$1" == "request" ]; then
awk '{id=$3 ;method=$5 ; path=$6 ; getline ; id2=$3; status=$5; type=$6; time=$7 ; print id "," method "," path "," "\n" id2 "," status "," type "," time ;}' $2 | awk ' { gsub ( /\[/, "" ) ; print } ' | awk ' { gsub ( /\]/, "" ) ; print } ' | sort -g > tempFile.csv
awk -F "," 's != $1 || NR ==1{s=$1;if(p){print p};p=$0;next} {sub($1,"",$0);p=p""$0;}END{print p}' tempFile.csv > requestLogReview.csv
echo "Completed Request Log Review for files :: " $2
fi
- AEM Packager : The AEM-Packager script deals with most of the use cases for package management in AEM. Its quite handy and simple to use. You can do a variety of operations using this script . All the actions here make you of cURL commands.
#!/bin/bash
# Default Variables
USER="admin"
PASSWORD="admin"
HOST="http://localhost:4502"
function usage
{
echo "usage: ./aem-packager.sh [list|install|uninstall|download|upload|upload_install|build|delete] [groupname|localpath when uploading] [packagename]"
}
list ()
{
echo "AVAILABLE PACKAGES:"
curl -u $USER:$PASSWORD $HOST/crx/packmgr/service.jsp?cmd=ls > input.xml
awk ' { gsub ( /\</, "" ) ; print } ' input.xml | awk '{sub(/\/.*/,""); print}' | cut -d\> -f2- | awk ' { gsub ( /\,/, "" ) ; print } ' > temp.csv
awk ' NR>6 { line1 =$1 ; getline ; line2=$1 ; getline ; line3=$1 ; getline ; line4 =$1 ; getline ; line5=$1 ; getline ; line6=$1 ; getline ; line7 =$1 ; getline ; line8=$1$2$3$4 ; getline ; line9=$1 ; getline ; line10 =$1$2$3$4 ; getline ; line11 =$1 ; getline ; line12 =$1$2$3$4 ; getline ; line13 =$1 ; print line1 "," line2 "," line3 "," line4 "," line5 "," line6 "," line7 "," line8 "," line9 "," line10 "," line11 "," line12 "," line13 ;}' temp.csv > aem-packages.csv
echo " Package List is Stored in the CSV File aem-packages.csv "
}
install_package ()
{
echo "Insalling PACKAGE: "$1:$2
if [ "$1" == "" -o "$2" == "" ]; then
echo "Please add Group Name and Package name for the package "
echo "e.g : ./aem-packager.sh install groupname packagename.zip"
exit
fi
curl -u $USER:$PASSWORD -X POST --fail "$HOST/crx/packmgr/service/.json/etc/packages/$1/$2?cmd=install"
echo ""
}
uninstall_package ()
{
echo "UnInstalling PACKAGE: "$1:$2
if [ "$1" == "" -o "$2" == "" ]; then
echo "Please add Group Name and Package name for the package "
echo "e.g : ./aem-packager.sh uninstall groupname packagename.zip"
exit
fi
curl -u $USER:$PASSWORD -X POST --fail "$HOST/crx/packmgr/service/.json/etc/packages/$1/$2?cmd=uninstall"
echo ""
}
upload_install ()
{
echo "Upload and Install PACKAGE from : " $1
if [ "$1" == ""]; then
echo "Please add package path "
echo "e.g : ./aem-packager.sh upload_install /Users/hakhan/Downloads/sample-2.0.zip "
exit
fi
curl -u $USER:$PASSWORD -F file=@"$1" -F force=true -F install=true $HOST/crx/packmgr/service.jsp
echo ""
}
upload ()
{
echo "Upload and Not Install PACKAGE : " $1
if [ "$1" == ""]; then
echo "Please add package path "
echo "e.g : ./aem-packager.sh upload /Users/hakhan/Downloads/sample-2.0.zip "
exit
fi
curl -u $USER:$PASSWORD -F file=@"$1" -F force=true -F install=false $HOST/crx/packmgr/service.jsp
echo ""
}
download ()
{
echo "Downloading PACKAGE: "$1:$2 "to path : " $3
if [ "$1" == "" -o "$2" == "" -o "$3" == "" ]; then
echo "Please add Group Name and Package name and Destination path "
echo "e.g : ./aem-packager.sh download groupname packagename.zip <path of folder>"
exit
fi
curl -u $USER:$PASSWORD $HOST/etc/packages/$1/$2 > $3
echo ""
}
build ()
{
echo "Building PACKAGE: "$1:$2
if [ "$1" == "" -o "$2" == "" ]; then
echo "Please add Group Name and Package name "
echo "e.g : ./aem-packager.sh build groupname packagename.zip"
exit
fi
curl -u $USER:$PASSWORD -X POST --fail "$HOST/crx/packmgr/service/.json/etc/packages/$1/$2?cmd=build"
echo ""
}
delete ()
{
echo "Deleting PACKAGE: "$1:$2
if [ "$1" == "" -o "$2" == "" ]; then
echo "Please add Group Name and Package name "
echo "e.g : ./aem-packager.sh delete groupname packagename.zip"
exit
fi
curl -u $USER:$PASSWORD -X POST --fail "$HOST/crx/packmgr/service/.json/etc/packages/$1/$2?cmd=delete"
echo ""
}
# Perform the actions
if [ "$1" = "list" ]; then
list
elif [ "$1" = "install" ] ; then
install_package $2 $3
elif [ "$1" = "upload_install" ] ; then
upload_install $2
elif [ "$1" = "upload" ] ; then
upload $2
elif [ "$1" = "download" ] ; then
download
elif [ "$1" = "build" ] ; then
build $2 $3
elif [ "$1" = "delete" ] ; then
delete $2 $3
elif [ "$1" = "uninstall" ] ; then
uninstall_package $2 $3
else
usage
fi
- Request Log Graph : Request logs are targetted whenever performance of a site is monitored. You might be aware of using rlog.jar or more advanced tools like – AppDynamics , New Relic etc. A much simpler and graphical way to plot the request log is by using a script by Jörg Hoh . You can copy the files from the GitHub link and feed a request.log file to it. (Precondition is that you should have a perl module installed in your system . )
It gives the Graphical plot for all the requests in the log file. eg. If you will see in the above graphplot , you can target the requests around 16:00 hours to find out why there is a peak present in logs at that time. Furthermore, you can check what all processes are scheduled during that time-frame or check if the User activity is increased suddenly resulting in increase of response time. Thus the plot gave you a quick hint of what might be the reason of slow response of environment.
- User/ Groups Migration Script : Sometimes there is a need to import all the Users / Groups from one environment to another and it gets tedious to add each principal user / group one by one. This script comes quite handy in that scenario. It will create and build a package of all the Users / Groups in an environment which are fetched via cURL command. That package can later be downloaded and ported to other higher environments or can be used for Migrations. Similar scripts can be formed to create other cURL query based searches.
#!/bin/bash
CREDS='admin:admin'
HOST=localhost
PORT=4502
QUERYURL=/bin/querybuilder.json
DATE=$(date +%Y%m%d%H%M%S)
PACKAGENAMEUSERS="users-${DATE}"
PACKAGENAMEGROUPS="groups-${DATE}"
PARAMSUSERS='path=/home/users&property=jcr%3aprimaryType&property.value=rep%3aUser&p.limit=-1&p.hits=full'
PARAMSGROUP='path=/home/groups&property=jcr%3aprimaryType&property.value=rep%3aGroup&p.limit=-1&p.hits=full'
usage ()
{
echo "Usage: add type of Package you want to Export : user or group"
echo "e.g : ./userMigration user"
exit
}
statusCheck ()
{
if [[ "$2" != "true" ]];then
echo "$1 : Incorrect Curl command or Parameters . Please verify."
exit
fi
}
#Method to Grab the List of Users in the AEM instance.
listUsers ()
{
echo "Getting List of Users:"
OUTPUT=$(curl --write-out -s -u ${CREDS} "http://${HOST}:${PORT}${QUERYURL}?${PARAMSUSERS}")
STATUS=$(jq '.success' <<< "${OUTPUT}")
statusCheck "ListUsers" ${STATUS}
addFilters "${OUTPUT}" $1
}
#Method to Grab the List of Groups in the AEM instance.
listGroups ()
{
echo "Getting List of Groups:"
OUTPUT=$(curl --write-out -s -u ${CREDS} "http://${HOST}:${PORT}${QUERYURL}?${PARAMSGROUP}")
STATUS=$(jq '.success' <<< "${OUTPUT}")
statusCheck "ListGroups" ${STATUS}
addFilters "${OUTPUT}" $1
}
# Method to Create a package with the Given PackageName and Date Stamp.
createPackage ()
{
echo "Creating Package:"
OUTPUT=$(curl --write-out -s -u ${CREDS} \
"http://${HOST}:${PORT}/crx/packmgr/service/.json/etc/packages/users/$1.zip?cmd=create" \
-d packageName=$1 \
-d groupName=users)
STATUS=$(jq '.success' <<< "${OUTPUT}")
statusCheck "Create" ${STATUS}
}
# Method to add Filters to the created package.
addFilters ()
{
OUTPUT=$1 | jq .
RESULTS=$(jq '.results' <<< ${OUTPUT})
STATUS=$(jq '.success' <<< ${OUTPUT})
USERPATHS=$(jq '.hits[]."jcr:path"' <<< ${OUTPUT} | sed 's/"//g' )
USERNAMES=$(jq '.hits[]."rep:principalName"' <<< ${OUTPUT} | sed 's/"//g' )
ARRAYUSERPATHS=( $USERPATHS )
ARRAYUSERNAMES=( $USERNAMES )
FILTER="["
for ((i=0; i<${RESULTS}; i++)); do
FILTER="${FILTER}{\"root\":\"${ARRAYUSERPATHS[${i}]}\", \"rules\":[{\"modifier\":\"exclude\",\"pattern\":\"${ARRAYUSERNAMES[${i}]}/.tokens\"}]}, "
done
FILTER="${FILTER:0:${#FILTER}-2}]"
echo "Add Filter to Package"
OUTPUT=$(curl --write-out -s -u ${CREDS} \
"http://${HOST}:${PORT}/crx/packmgr/update.jsp" \
-F "path=/etc/packages/users/$2.zip" \
-F "packageName=$2" \
-F "groupName=users" \
-F "_charset_=UTF-8" \
-F "filter=${FILTER}")
STATUS=$(jq '.success' <<< "${OUTPUT}")
statusCheck "Filter" ${STATUS}
}
# Method to build a package created above.
buildPackage ()
{
echo "Build Package:"
OUTPUT=$(curl --write-out -s -u ${CREDS} -X POST \
"http://${HOST}:${PORT}/crx/packmgr/service/.json/etc/packages/users/$1.zip?cmd=build")
STATUS=$(jq '.success' <<< "${OUTPUT}")
statusCheck "Build" ${STATUS}
}
# Perform the actions
if [ "$1" = "user" ]; then
createPackage ${PACKAGENAMEUSERS}
listUsers ${PACKAGENAMEUSERS}
buildPackage ${PACKAGENAMEUSERS}
elif [ "$1" = "group" ] ; then
createPackage ${PACKAGENAMEGROUPS}
listGroups ${PACKAGENAMEGROUPS}
buildPackage ${PACKAGENAMEGROUPS}
else
usage
fi
I hope the above scripts will be quite useful to you while working on AEM. Please feel free to use them or modify them as per your need. If you wish to add any other useful script do let me know.
Now all the Scripts are available in the GITHUB Project – https://github.com/hashimkhan786/aem-shell-scripts
CQ5 AEM Tricks of Trade 
Good job!
LikeLike
Hi,
could you please explain how to write the rewrite rules in webserver and basic concepts of rewrite rules
like how to shorten the url and what is $1, what is RewriteCond %{REQUEST_URI} !^/apps
basic concepts where to use in apache webserver
LikeLike
Hi ,
This seems to be a very broad question. You can read more about it online in similar links to understand it better :
http://mrcoles.com/blog/simple-way-understand-mod-rewrite/
https://www.sitepoint.com/guide-url-rewriting-2/
https://httpd.apache.org/docs/current/rewrite/intro.html
LikeLike
Pingback: The Perfect AEM Deployment | CQ5 AEM Tricks of Trade
Hi Hashim,
I am new to this bash and also sh. Can you help me to understand it from basic? Do we need aem.packemgr.sh installed on the machine where we will run this script? I am trying to use upload_install method to work with package. Can you throw some light to make me understand how to use it?
LikeLike
Hi Neha,
You need to have environment box access to run bash scripts. You can run them in a Unix / Mac-OS box or use CygWin for windows. Read more https://www.shellscript.sh/
LikeLike