In the last few projects, I have seen and learnt a lot of best practices and potential flaws in an AEM infrastructure. I have been lucky enough to work with some interesting clients – who have given me many difficult scenarios and system related issues – which eventually led me to ponder over the flaws in a project and to think of some best practices / tricks which can help us overcome them. I am sure some of you must be following one or other best practices in your project and I would like to hear about them in comments below. Recently I have read a wonderful white paper- by TAD REEVES from Rackspace – which is the basis for this post.

In a normal AEM development project, a lot of companies follow the Build , Deliver and Operate Model. Quite often the Operate /Maintain part is left with client teams or System engineers. And many a times we overlook how frequent changes should be done after the project gets over. Ideally the complete AEM platform should be robust enough to accept frequent Business changes with minimal development effort. This particular need requires a good process for every subsequent release .

AEM Release Process :

1. Local Development :  Every developer should have a running AEM instance at their local instance which should match as much as possible with Production environment. If you are using GIT , make sure to carve out a feature-release before working on any feature . Keep on committing at your feature-branch and merge to develop only when the feature is complete and thoroughly reviewed by peers or architect. This will ensure good code quality.
2. Shared Development CI (Dev Environment.) : As soon as the code is merged to develop , a continuous integration can be setup in Jenkins to deploy code to Dev environment. This will have the SNAPSHOT of the build. You can configure the Build Triggers in a Jenkins Job to achieve this – either periodically or on every commit/ merge.
3. Unit Testing: Once the build for feature is present in Dev environment, its the responsibility of developers to Unit test that feature thoroughly before giving it to QA or pushing to Staging.
4. Release to QA / Staging : Once all the features are done for the release , you should carve out a branch in GIT for a release version. Change the package version and deploy again on DEV environment. Next , you should configure Jenkins Job to pick only those packages which are deployed on DEV environment and are working fine. You should use CRX plugin for Jenkins and configure it to pick all the packages from DEV author/Publish environment and deploy to QA environment. This would ensure that only verified packages are going to QA. here are the configuration steps: https://wiki.jenkins.io/display/JENKINS/CRX+Content+Package+Deployer+Plugin    The parametrized configuration would look like this – (list of packages is coming from DEV )
5. Deploy to Staging :  After the deployed package is QA verified , a similar Jenkins Jobs should be created deploy-to-UAT-from-QA such that only a QA verified package goes to Staging.
6. Performance / Load/ Security testing on UAT/ Staging : The staging environment should match the Production environment as much as possible- in terms of Content and infrastructure. Here we should execute a round of Load testing, Performance testing or Security testing before moving ahead with Production deployment. I know this sounds cumbersome , but once the scripts are ready it should be a smooth task to execute. Also make sure that Log Analysis is done during this stage and the Build should only be pushed to Prod once there are NO Errors and tolerable Warnings. You may use this https://hashimkhan.in/2016/10/16/useful-shell-scripts-for-aem/ for Log Analysis.
7. Deployment to Production: Although this step is the most important one, yet it get its stability from the process which we followed above. If the initial steps are done with due diligence this should be a cake walk. Before starting any deployment on Prod , make sure to take a Volume Backup of the repository. Again there should be a Jenkins Job deploy-to-Prod-from-UAT which will pick only those packages which are tested on UAT thoroughly.  Lets say you have 4 Publishers in Prod environment , the final deployment should happen in a Rolling Manner i.e. Take 2 publishers off the Load Balancer , deploy the code there , put them back to Load balancer , take other 2 publishers off the Load Balancer , deploy code to the next two, and put back to Load balancer. You can also do this one publisher at a time , depending on the Load. Restart the servers if needed. Perform Dispatcher / Akamai cache clearance as per need.
8. Rollbacks  : To prepare for the worst , we should always be ready with a fallback mechanism. The Prod volume backups we took before deployment should be pushed back if something fails terribly.

Things to Keep in Mind:

1. Version Control: Do version control of your code using the available tools such as GIT, Subversion, etc. You should also checkin the Dispatcher Configurations there and devise a mechanism to deploy Dispatcher Configurations in Apache using Jenkins.
2. Versioned Packages:  Only deploy packages in an environment. Whether its configurations / code / content, nothing should be done manually . Every change should come via package. And it should follow the same course Dev>QA>UAT>Prod. Never deploy SNAPSHOTs in higher environments, always carve out versions for the release and then deploy.
3. Content Sync : Ideally content is authored on Production environment by most of the clients, so there should be a weekly / biweekly sync from Prod to lower environments – QA/ UAT . Make sure that staging environments are as much as possible close to Prod environment. Ideally infrastructure wise, staging and prod should be identical, but it might not be feasible for most companies due to cost constraints. This can be overcome using proper calculations during Load/ performance testing on stage.
4.  Reviews & Monitoring :  Make sure to use a good APM tool like AppDynamics / New Relic to keep a check on the environment . You should also use Log monitoring tools such as Splunk / Nagios.