Years ago, we developed a software development communities and processes analytics PaaS PoC based in GrimoireLab technologies. We called it Cauldron, and recently we have started working on a big update, that includes many new features, including Open Distro for Elascticsearch! Let’s see what’s coming…
A short history of Cauldron
Bitergia’s aim is to help organizations understand the communities and processes of the software development projects that matter to them, and was since Bitergia was founded in 2012. Of course, to achieve this aim Bitergia uses and contributes open source tools, like MetricsGrimoire, VizGrimoire, and more recently GrimoireLab (we familiarly call them the Grimoire tools).
But 5 years ago, we realized that there could be other ways to help people and organizations beyond having free, open source tools, and providing consultancy services on top of them. Some people were asking for a PaaS for Grimoire tools. (We joked about the GaaS name from time to time).
We need to carefully take one step at a time, because our resources are limited. So, we started our first PoC based on legacy MetricsGrimoire code and some new UX ideas to present it on OSCON 2014.
Building on the lessons learned, we started working on a whole new platform, the one that has become GrimoireLab. One of the core components of it is Elasticsearch because, beyond its cool features, it has a nice tool to visualize and play with the data: Kibana.
One year later, at OSCON 2015, we presented a new PoC, based in the new stack, and it’s been unchanged for years: cauldron.io was born. With a limited support from our operations team, users have been using it for years, and we have received occasional feature requests.
Now, 3 years later, thanks to a national government grant, we have resources to re-think and update the old Cauldron. Welcome to the alpha version of a new Cauldron!
What’s new?
First of all, it’s very alpha! It’s a preliminary version of a MVP (investors are welcome! 😀 ). All data can be removed without prior notice, and for sure, it’s full of bugs, so feel free to submit issues 😉 (Have I already said it’s 100% free, open source?)
The whole UX is totally different from the old Cauldron
In the previous version, you were able to analyze data only from the 30 latest active repositories within one GitHub organization. Now (at least during the alpha period), you can analyze several projects. You can define each project by a collection of GitHub organizations, GitHub users, or GitLab groups, or GitHub/GitLab repositories, and even a set of git repositories.
Now, you have more information about the status of the data collection. Once the data sources repositories are added, they appear in a detailed list full of information and controls. The “Status” column indicates whether data is already available. The “Last refresh” column indicates whether data is already available and how old the data is. The “Refresh” Button allows to manually start a new data collection process for refreshing the data.
Open Distro for Elasticsearch
Yes, we want to test many new things in this new Cauldron, and one of them is Open Distro for Elasticsearch. Why?
During the years, we have been using Elasticsearch and Kibana as key components of the GrimoireLab open source project. We have contributed to Kibana core, developed plugins, and integrated third-party tools. And we have been always committed to open source principles. We ended up developing a soft-fork of Kibana in GrimoireLab, to have customization that were unique to our use case.
Meanwhile, Elastic, the company behind Elasticsearch and Kibana, has kept doing an amazing job improving their stack, but also adding some non-free, non-open source components in favor of their subscription business model for X-Pack (opened by Elastic, but not as free, open source software). This business strategy has blocked us from testing some new features in GrimoireLab, and even worse, from integrating these proprietary features in our services for customers that specifically require us to use and provide only 100% free, open source software.
With one developer working initially on the new Cauldron, we have considered moving to other platforms, like Grafana, or back to our initial approach of having our self-developed visualization platform. And in the middle of all of this discussion, Open Distro for Elasticsearch was announced. Everything became clear to us: we should use Open Distro for the new Cauldron.
The current Cauldron alpha reuses some new visualizations, but thanks to Open Distro Kibana features, you should be able to use many of its goodies, like alerting, SQL queries, or multi-tenant.
Incorporating Open Distro for Elasticsearch technology into Cauldron wasn’t a big challenge due to the community that maintains it. And the documentation is kept up to date and includes simple deployment examples in a few steps and even for use with other technologies such as Kubernetes (something we want to test in the near future).
Feel free to use the new Cauldron alpha and provide feedback!
What’s next?
Let’s dive into technical aspects of Cauldron.
For the analysis of the repositories, Cauldron uses components from GrimoireLab. It is a 100% free, open source software from the CHAOSS project, under the Linux Foundation. And there are some new features and components to be announced at the next CHAOSSCon in San Diego on August 20th. I wouldn’t miss the Dive into GrimoireLab workshop to know more 😉
One of the amazing features of GrimoireLab is that it can analyze data from 30+ different data sources to provide a holistic view of an open source community. This alpha version of Cauldron currently uses only three of these data sources, namely Git, GitHub, and GitLab. However, it is planned to enable more sources, including Discourse, Slack, and Meetup. If you think we should add others, please, ask for them! (or leave comments to this post)
Last, but not least, this new Cauldron will be our tool to try new things and technologies (like Kubernetes mentioned before) or test new ways to consume the data (for example with Grafana for those used to it), that will result in better support and services for our current and future customers. Actually, we are discussing how to develop a business model associated with Cauldron, and we’d love hearing your ideas or comments to this post.
