In the last few projects, I have seen and learnt a lot of best practices and potential flaws in an AEM infrastructure. I have been lucky enough to work with some interesting clients – who have given me many difficult scenarios and system related issues – which eventually led me to ponder over the flaws in a project and to think of some best practices / tricks which can help us overcome them. I am sure some of you must be following one or other best practices in your project and I would like to hear about them in comments below. Recently I have read a wonderful white paper- by TAD REEVES from Rackspace – which is the basis for this post.
In a normal AEM development project, a lot of companies follow the Build , Deliver and Operate Model. Quite often the Operate /Maintain part is left with client teams or System engineers. And many a times we overlook how frequent changes should be done after the project gets over. Ideally the complete AEM platform should be robust enough to accept frequent Business changes with minimal development effort. This particular need requires a good process for every subsequent release .
AEM Release Process :
- Local Development : Every developer should have a running AEM instance at their local instance which should match as much as possible with Production environment. If you are using GIT , make sure to carve out a feature-release before working on any feature . Keep on committing at your feature-branch and merge to develop only when the feature is complete and thoroughly reviewed by peers or architect. This will ensure good code quality.
- Shared Development CI (Dev Environment.) : As soon as the code is merged to develop , a continuous integration can be setup in Jenkins to deploy code to Dev environment. This will have the SNAPSHOT of the build. You can configure the Build Triggers in a Jenkins Job to achieve this – either periodically or on every commit/ merge.
- Unit Testing: Once the build for feature is present in Dev environment, its the responsibility of developers to Unit test that feature thoroughly before giving it to QA or pushing to Staging.
- Release to QA / Staging : Once all the features are done for the release , you should carve out a branch in GIT for a release version. Change the package version and deploy again on DEV environment. Next , you should configure Jenkins Job to pick only those packages which are deployed on DEV environment and are working fine. You should use CRX plugin for Jenkins and configure it to pick all the packages from DEV author/Publish environment and deploy to QA environment. This would ensure that only verified packages are going to QA. here are the configuration steps: https://wiki.jenkins.io/display/JENKINS/CRX+Content+Package+Deployer+Plugin The parametrized configuration would look like this – (list of packages is coming from DEV )
- Deploy to Staging : After the deployed package is QA verified , a similar Jenkins Jobs should be created deploy-to-UAT-from-QA such that only a QA verified package goes to Staging.
- Performance / Load/ Security testing on UAT/ Staging : The staging environment should match the Production environment as much as possible- in terms of Content and infrastructure. Here we should execute a round of Load testing, Performance testing or Security testing before moving ahead with Production deployment. I know this sounds cumbersome , but once the scripts are ready it should be a smooth task to execute. Also make sure that Log Analysis is done during this stage and the Build should only be pushed to Prod once there are NO Errors and tolerable Warnings. You may use this https://hashimkhan.in/2016/10/16/useful-shell-scripts-for-aem/ for Log Analysis.
- Deployment to Production: Although this step is the most important one, yet it get its stability from the process which we followed above. If the initial steps are done with due diligence this should be a cake walk. Before starting any deployment on Prod , make sure to take a Volume Backup of the repository. Again there should be a Jenkins Job deploy-to-Prod-from-UAT which will pick only those packages which are tested on UAT thoroughly. Lets say you have 4 Publishers in Prod environment , the final deployment should happen in a Rolling Manner i.e. Take 2 publishers off the Load Balancer , deploy the code there , put them back to Load balancer , take other 2 publishers off the Load Balancer , deploy code to the next two, and put back to Load balancer. You can also do this one publisher at a time , depending on the Load. Restart the servers if needed. Perform Dispatcher / Akamai cache clearance as per need.
- Rollbacks : To prepare for the worst , we should always be ready with a fallback mechanism. The Prod volume backups we took before deployment should be pushed back if something fails terribly.
Things to Keep in Mind:
- Version Control: Do version control of your code using the available tools such as GIT, Subversion, etc. You should also checkin the Dispatcher Configurations there and devise a mechanism to deploy Dispatcher Configurations in Apache using Jenkins.
- Versioned Packages: Only deploy packages in an environment. Whether its configurations / code / content, nothing should be done manually . Every change should come via package. And it should follow the same course Dev>QA>UAT>Prod. Never deploy SNAPSHOTs in higher environments, always carve out versions for the release and then deploy.
- Content Sync : Ideally content is authored on Production environment by most of the clients, so there should be a weekly / biweekly sync from Prod to lower environments – QA/ UAT . Make sure that staging environments are as much as possible close to Prod environment. Ideally infrastructure wise, staging and prod should be identical, but it might not be feasible for most companies due to cost constraints. This can be overcome using proper calculations during Load/ performance testing on stage.
- Reviews & Monitoring : Make sure to use a good APM tool like AppDynamics / New Relic to keep a check on the environment . You should also use Log monitoring tools such as Splunk / Nagios.
7 thoughts on “The Perfect AEM Deployment”
Wow Hashim – thanks for the shout-out! Your article brings mine quite a bit of missing detail, with your Jenkins details and analysis links. Just for posterity, the original post I’d done is this one, if anyone is curious: https://blog.rackspace.com/white-paper-creating-the-ideal-aem-deployment-process
LikeLiked by 1 person
Using Multifield in a dialog, works fine when it were created for the first time, But when we try to update any one of the multi field’s field to a new value, the other untouched fields are removed from the crx repository.
I am using AEM6.3
Are you using custom multifield or OOTB implementation of multifield? The values shouldn’t be removed unless you save the dialog again. It could be due to some errors while reading the properties from node.
Try to add some console logs in your widget and debug what is happening.
Thanks for this article!
i needed some information on setting up new environment(Dev and QA) for my project which is currently being developed in my and my teammates local AEM instances,
we need to build this project using jenkins, using git so that every developer can work collectively in team and can push there modules on dev and followed by qa environment respectively.
we do not have any system engineers, architect, DevOps or clients to set up these for us but we have AWS server already.
can you please help us with Steps/Links/Detailed Information to achieve this?
Any help would be appreciated
Its actually difficult to give you advice without knowing things about the environment – how many Authors/ Publishers do you need, architecture etc. That’s where System engineers and architects come in.
You can set up your GIT build using Jenkins http://www.tothenew.com/blog/how-to-setup-jenkins-for-a-maven-project/ You can use IPs of AWS server to deploy via Jenkins.
For every feature of the project, create a different branch of GIT, and merge to develop branch only when that code is properly reviewed. Apply continuous integration to Develop branch to your DEV/ QA environment.
Follow the above post to move code from 1 environment to another.
“Ideally content is authored on Production environment by most of the clients, so there should be a weekly / biweekly sync from Prod to lower environments – QA/ UAT”
What (automated)process would you recommend to do a sync of content from prod to QA/UAT?
When you do such sync, will it not override the code(dialog code for instance)
Yes, that’s like the best practice which should be followed. You can use many ways for it to happen.
Assuming all your CQ-authors are changing content in the Production environment.
So there could be 2 major approaches: either you can schedule / manually do a Volume backup of Prod and create QA/ UAT environment, and then deploy new Code to QA/ UAT.
You can package out the difference of Content from Prod and deploy to QA/ UAT. This will just deploy Content. https://adobe-consulting-services.github.io/acs-aem-commons/features/packagers/query-packager/index.html