Skip Main Navigation

Using Private Dependencies in Cloud Manager Builds

With the Cloud Manager, Adobe provides a viable tool to self-manage AEM environments in the cloud. In general,  the approach is that all AEM customisation code is built from source and then deployed to the Cloud Manager managed environments. While this works well for small to medium-sized projects, larger customers often run into one of the following scenarios:

 

  • Multiple teams run on separate release cycles, yet all those modules have to run on the same AEM platform in production. Having all teams work on one repository does not scale.
  • There are multiple platforms for the same customer and re-using AEM specific artifacts is desired to not duplicate the code (e.g. shared AEM components, parent poms).
  • Customer managed packages that have a broader scope than just AEM have to be integrated (e.g. frontend modules like living style guides, ready-made connectors to internal systems, etc.).
  • Packages from third-party vendors that are not open source and not available via Adobe or Maven Central repository have to be integrated (e.g. a Translation Service Provider)

 

Previously, allowing a private repository in Cloud Manager was possible but it was a manual task that had to be taken care of on the Adobe side, by your project's CSE. Recently, however, Adobe added a feature to Adobe IO's cloud manager integration to set environment variables to the build in order to make this aspect self-service. With this alone, it’s possible to enforce repository-provided settings.xml by using the file ".mvn/maven.config". 

 

However, duplicating Adobe's settings.xml in the repo is not ideal. For example, if a development team has an optimised settings.xml that they generally use, for the Cloud Manager repo, this would be overwritten with the Adobe defaults. Also, Adobe may change their master version of the settings.xml at any time which would break the Cloud Manager build (e.g. if their repository server is migrated to another domain). To avoid these issues a custom Maven extension comes into play.

Setup a private Maven repository using pipeline variables and the custom extension maven-ext-repos-from-env

Instead of duplicating the settings.xml and adding the credentials for the repository there, simply add the file ".mvn/extensions.xml" with the following content to your repository:

 

<extensions xmlns="http://maven.apache.org/EXTENSIONS/1.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/EXTENSIONS/1.0.0 http://maven.apache.org/xsd/core-extensions-1.0.0.xsd">
    <extension>
        <groupId>biz.netcentric.maven.extension</groupId>
        <artifactId>maven-ext-repos-from-env</artifactId>
        <version>1.0.1</version>
    </extension>
</extensions>


This Maven extension is open source (https://github.com/Netcentric/maven-ext-repos-from-env) and available via Maven Central. In the case that  no environment variables with prefix "MVN_SETTINGS_REPO" are set, this extension will simply do nothing. However, for the Cloud Manager case, the environment variables for the private repo may be set as follows:

 

aio cloudmanager:set-pipeline-variables \
   <PIPELINE_ID> \
   --programId=<PROGRAM_ID> \
   --variable \
     MVN_SETTINGS_REPO_URL https://artifacts.corp.com/cloud-manager/ \
     MVN_SETTINGS_REPO_USERNAME cloudmanager \
   --secret \
     MVN_SETTINGS_REPO_PASSWORD <REPO_PASSWORD>

 

Running the build in Cloud Manager with the extension and the environment variables will automatically amend the Maven execution with:

 

  • A remote repository
  • A server entry to provide credentials for the repository

 

Your Cloud Manager build will show logs as follows:

 

...
[INFO] Repository added from system environment variables: https://artifacts.corp.com/cloud-manager/ (id: sysEnvRepo user: cloudmanager)
[INFO] Scanning for projects...
[INFO] Downloading from sysEnvRepo: https://artifacts.corp.com/cloud-manager/com/corp/aem/aem-parent/4.3.8/aem-parent-4.3.8.pom
[INFO] Downloaded from sysEnvRepo: https://artifacts.corp.com/cloud-manager/com/corp/aem/aem-parent/4.3.8/aem-parent-4.3.8.pom (51 kB at 39 kB/s)
...

 

See documentation at https://github.com/Netcentric/maven-ext-repos-from-env/blob/develop/README.md for more details (e.g. on the prerequisites for using aio cloudmanager:set-pipeline-variables).

What about private NPM dependencies?

For npm dependencies, the situation is slightly easier since Cloud Manager does not come with its own npm configuration and npm supports environment variables for both npmrc location and within npmrc itself. But first, in the same way as for Maven, let's set up a secret as pipeline variable to make it available for the build:

aio cloudmanager:set-pipeline-variables \
   <PIPELINE_ID> \
   --programId=<PROGRAM_ID> \
   --secret \
     NPM_PRIVATE_REPO_AUTH <npm-auth>

 

This authentication information can then be referenced from the .npmrc file in the source repository. In the following example, the scope "corp" is downloaded  from a private repository:

 

@corp:registry=https://artifacts.corp.com/cloud-manager/
//artifacts.corp.com/cloud-manager/:_auth=${NPM_PRIVATE_REPO_AUTH}
//artifacts.corp.com/cloud-manager/:always-auth=true

 

If the .npmrc is set up in the frontend module's root folder of the source repository, it will become active for Cloud Manager as well as for all developers. If the developers (lie for the Maven use case) already have a slightly different ~/.npmrc file optimised for development, the following snippet can be used in the Maven pom to solely enable this for Cloud Manager.

 

<profile>
    <id>cloud-manager</id>
    <activation>
        <property>
            <!-- The env variable CM_BUILD is set on any cloud manager build -->
            <name>env.CM_BUILD</name>
        </property>
    </activation>
    <build>
        <plugins>
            <plugin>
                <groupId>com.github.eirslett</groupId>
                <artifactId>frontend-maven-plugin</artifactId>
                <configuration>
                    <environmentVariables>
                        <!-- using name .npmrc-cloudmanager to make it self-explanatory -->
                        <NPM_CONFIG_USERCONFIG>${project.basedir}/.npmrc-cloudmanager</NPM_CONFIG_USERCONFIG>
                    </environmentVariables>
                </configuration>
            </plugin>
        </plugins>
    </build>
</profile>

A note about security

Cloud Manager runs on Azure and IP protection is currently not an option for private repositories. This means you have to make your private repository available on the internet. The following security measures are recommended to lock down the repository:

 

  • Only allow access with authentication (the obvious point)
  • Restrict access to only allow GET requests
  • Lock down the allowed paths to the absolute minimum

References


Georg Henzler

Principal Solution Architect

Georg has grown up with the web space from the very beginning and has helped enterprises to design their solution for it from the time when images were first added to their web sites. Today he shapes the way Netcentric delivers IT projects technically by continuously evolving the development process and automating infrastructure to the last slash. Amongst being a committer for Apache Sling (the web framework in AEM) he has contributed to many other open source projects and talks regularly at tech conferences. He loves Netcentric for its culture of "Share, discuss and bring ideas to life"!

More Developer Circle