A Hitchhiker’s Guide to Spinnaker Plugins

Photo by Daniel Stenholm on Unsplash.

Spinnaker is a multi-cloud continuous delivery platform. It is an open source product the development of which incorporates years of expertise from companies like Netflix and Google.

One powerful feature of Spinnaker is that it is highly customisable. In the hope of helping builders of Spinnaker plugins this blogpost presents an overview of the platform, things to have in mind when building plugins, and challenges you may face.

Architecture

Spinnaker is composed of a number of independent micro-services (also see its official documentation).

Spinnaker’s Architecture from https://spinnaker.io/docs/reference/architecture/microservices-overview.

Below is a list of components which are often customised when building plugins:

• Deck is the browser-based UI.
• Gate is the API gateway. This is the component through which the Spinnaker UI and any API callers communicate with Spinnaker.
• Orca is the orchestration engine.
• Clouddriver is responsible for Cloud Operations. Think AWS and Kubernetes.
• Echo is Spinnaker’s eventing service. It is used, among others, for notifications.
• Finally, Halyard is Spinnaker’s configuration service.

Plugins

Photo by Clint Patterson on Unsplash.

Spinnaker follows a pluggable architecture which allows users to extend its components. This can be achieved through the Plugin Framework For Java (PF4J) and Extension Points.

Extension Points

In Spinnaker terms, services such as Orca, provide interfaces which are called Extension Points. Implementations of these Extension Points are called Extensions, a collection of which forms a plugin. Strictly speaking, a plugin is associated with a single Spinnaker Service. The largest entity in the Spinnaker plugin world is a bundle, which is a collection of plugins.

Imagine you want to implement a custom stage that deploys resources to an in-house container orchestration platform. For this, you could extend the StageDefinitionBuilder Extension Point and create a plugin for Orca. In addition to that, you would need to customise CloudDriver and Deck. These plugins would then be packaged together to form a bundle.

Project Setup

If you follow the plugin examples, you will notice that they use a certain tech stack and they follow a similar project structure. To begin with, you can clone one of these repositories and build on top of them. If you would like to customise Deck you could use the provided scaffolding which will generate your Deck plugin’s project structure.

In terms of programming languages, server-side components such as CloudDriver and Orca are written in Kotlin, while the front-end (Deck) is written in React/Typescript.

As regards the build system, sample projects use Gradle.

Versions

When building a new plugin you need to be careful with the versions of the core libraries. You may notice the Version Compatibility section in existing plugins. What helps is that Spinnaker releases are documented alongside Spinnaker subcomponents’ versions which have been tested with each release.

Local Plugin Setup

Photo by Andre Tan on Unsplash.

Running Spinnaker Locally

There are many guides on how to deploy and manage Spinnaker.

The first choice to make, as long as your workloads are running on Kubernetes, is whether to use Halyard or the Operator (everything is an operator nowadays!). Although my experience with the operator is limited, I would recommend looking into it for production environments. There is even a guide on how to migrate from Halyard to the Operator.

For a local setup the official documentation includes instructions on how to install Spinnaker on a Multipass VM using Minnaker. Personally, I installed Spinnaker on a local Docker Desktop Kubernetes cluster by following the instructions from the Minnaker repository.

To access the Spinnaker components locally you can either port forward or create Load Balancers. Once you have a local setup Spinnaker’s documentation includes all the steps you need to locally build and run your plugins.

Local Overrides

To customise the Spinnaker setup and its components’ configurations you can use Custom Profiles. With profiles, you can, among others:

• Extend the locally running components to include your plugin after you link the two projects
• Configure Spinnaker to connect to AWS accounts and talk to Kubernetes clusters
• Configure Spinnaker to connect to an internal GitHub instance

The way to do this with Halyard is by providing the X-local.yml files. When using the Kubernetes Operator you can provide profiles in the specs.

Testing

Photo by Nicolas Thomas on Unsplash.

As with most components which are part of a bigger system testing your plugins can be achieved at different levels.

Unit Tests

First, you can unit test methods, classes, etc with libraries like JUnit.

Integration Tests

Integration tests provide more confidence as to whether your plugins load and behave as expected on runtime. The official documentation covers integration testing with references to open source plugins.

To facilitate integration testing Spinnaker libraries come with test harnesses. The modules that include these test utils are named X-tck, where X is the module for which test harnesses are provided. For example, you can use the orca-api-tck to write integration tests for your Orca plugin.

On this note, often I had the need to mock certain calls. For instance, when writing integration tests for a new stage in Orca which would call CloudDriver (in specific, its oort and kato services). For this you can customise the required beans and mock any calls to these services.

Compability Testing

A type of testing that I found useful when building plugins is compatibility testing. Spinnaker provides a neat way for this which allows you to run your integration tests vs different versions of Spinnaker.

Manual Testing

We should strive towards automating testing and preventing issues the soonest possible. However, a quick sanity check is always a good idea, especially in this architecture which involves many components.

For the example we mentioned above where we build a stage that deploys to an in-house container orchestration platform, testing end-to-end could be done as follows:

• Run the plugin locally or even better in a Spinnaker instance that can be used for testing purposes.
• Verify the front-end which includes the custom stage.
• Trigger a deployment by pointing the Spinnaker instance to the in-house platform (again, a test environment of the platform).
• Verify the deployment in the platform.
• Verify the expected behavior in the UI. That is, any feedback loop provided to the users.

Troubleshooting

Photo by Christopher Gower on Unsplash.

I found troubleshooting issues with Spinnaker plugins cumbersome without prior experience. Below you can find hints for common issues.

Server Components

First, make sure your server components are healthy. As long as you have created Load Balancers for the externalised services and your locally built ones are running you can access their health check on their associated port (e.g. http://localhost:7002/health for CloudDriver).

Each of the components exposes APIs that can be used for verification. For example, you can call CouldDriver’s /credentials API to verify your AWS and Kubernetes integration.

It is also a good idea to check the logs and make sure there are no exceptions or warnings.

Something to have in mind is that the source code of any Spinnaker component is available on GitHub. This is great in case you would like to debug or to contribute back.

Rendering Issues

For front-end issues, it is recommended to download the React Developer Tools Chrome extension. This allows you to debug React components and Deck.

If Deck is up and running but your plugin is not appearing in the UI, it is possible that it is due to caching in the browser. Try clearing the cached content or use Incognito mode.

Memory Issues

It is common to run into memory issues when running the locally developed Spinnaker components. To overcome these issues you may need to increase the memory heap of the IDE.

Another memory-related issue that may occur is the Gradle process running out of Java heap space memory. For this try to increase the Gradle JVM memory (org.gradle.jvmargs in your gradle.properties file).

Challenges

Photo by Quino Al on Unsplash.

Building and maintaining Spinnaker plugins requires investment. My experience so far has not been as smooth as I expected. Below is a summary of the main challenges I have faced.

Running Plugins Locally

Setting up Spinnaker and testing plugins locally is a tedious process. Once you have the setup up and running I would recommend documenting all the steps in your project’s repository. I am personally a huge fan of documentation, even if this slows down deliverables.

Memory Footprint

The memory footprint of Spinnaker is surprisingly high. Requiring 15–20GB of memory to run Spinnaker locally is prohibitive for many engineers, especially on personal laptops. This has improved over time and with the Operator but unfortunately running the localised components in a fully-fledged IDE adds another layer of memory pressure.

Libraries and Versions

To ensure compatibility between Spinnaker versions and libraries it is recommended to import the BOM for each plugin you are building. I ran into several issues with libraries like Jackson where using a different version causes runtime issues.

On the library front, I felt there is a certain amount of unnecessary complexity in the Spinnaker components. A good example are the provided testing libraries which rely on a wrapper instead of building on top of JUnit. Personally, I am a bit against wrappers and the necessity to bring in more libraries to projects. In my experience such libraries introduce complexity, mental overhead, and lack of maintenance is a risk introduced to your systems.

Versioning is also slightly puzzling. In most of the components’ repositories you will find multiple versioning schemes (e.g. v-X, version-X). There is limited documentation on which one to use and this comes in addition to the multiple Spinnaker versions (open source vs Armory’s licensed versions).

Documentation

There is extensive documentation on Spinnaker from both the Community and Armory. The repositories with plugin examples are handy.

Netflix engineers and the community have put tremendous effort to decouple the internal implementation and to document it. There is still a fair amount of undocumented library code in the codebase but this also means there are opportunities for new joiners to contribute.

Bintray Sunset

Migrations are painful and JFrog’s Bintray sunset has certainly impacted many projects. One of them is Spinnaker, the libraries of which were published to Bintray. Although most artifacts are now available in different registries I ran into issues with references to Bintray or older artifacts not being in any registries.

Conclusion

Spinnaker’s customisation features and the fact that it is an open-source platform make it perfect for experimentation. This being my first time building components of a pluggable architecture made me realise the uniqueness of these architectures and their challenges. All in all, building plugins for Spinnaker is fun. The learning curve is not steep but not flat either. My key takeaway is that once you build your first plugin additional ones should be much more straightforward.

References

• Spinnaker Architecture Overview: https://spinnaker.io/docs/reference/architecture/microservices-overview/
• Plugin User Guide: https://spinnaker.io/docs/guides/user/plugins-users/
• Spinnaker Plugin Examples: https://github.com/spinnaker-plugin-examples
• Compatibility Testing: https://spinnaker.io/docs/guides/developer/plugin-creator/testing/compatibility-testing/