Setting up a quick demo for an open source project

Share this Post

Last week I was invited to participate in the Open Source Weekends meetup, so I set up a quick GrimoireLab demo for them. It was a surprise, part of my talk about the history of Bitergia. You can see the slides online:

The talk included a quick intro to open development analytics concept, some historical Bitergia references, and quick introductions to GrimoireLab and CHAOSS projects.

But the one more thing during the talk was a live demo of GrimoireLab with real data about the activity, community and performance of Open Source Weekends projects. Do you want to know how I’ve set it up?

Let’s go for it!

Open Source Weekends projects

Open Source Weekends (AKA: OSW) is a young community, based mostly in Madrid, that meets during a weekend once a month. They are doing a great work helping people to start their own open source projects or collaborate in the existing ones under their umbrella. These projects and initiatives are organized as guilds, with very agile mindset (if that concept exists), probably influenced by Spotify’s view of Agile Team Organization.

They have the following main infrastructure already supported by GrimoireLab to manage their community:

A set of GitHub organizations: OSWeekends, GingerCode, and GoblinDBRocks
A Meetup group
A set of channels in an Slack workspace

How to set up a GrimoireLab demo

I know that current documentation needs a lot of work, and the community is working on it. And I am not the most technical person in the office, so, I am sure there will be some areas of improvement in the following description.

Get the tools

First, I need an Elasticsearch and a Kibiter (a GrimoireLab’s Kibana fork with some steroids) instance running to store and visualize the data. To get it, I’ve used a docker-compose.yml file like this:


elasticsearch:
  image: bitergia/elasticsearch:6.1.0
  command: /elasticsearch/bin/elasticsearch -E network.bind_host=0.0.0.0
  ports:
    - "9200:9200"
  environment:
    - ES_JAVA_OPTS=-Xms2g -Xmx2g

kibiter:
  image: bitergia/kibiter:6.1.0
  links:
    - elasticsearch
  ports:
   - "5601:5601"
  environment:
    - PROJECT_NAME=jsmanrique
    - NODE_OPTIONS=--max-old-space-size=1200
    - ELASTICSEARCH_URL=http://elasticsearch:9200

And to run it, simply:


$ docker-compose up -d

To save you some time (I discovered that later), ensure you have MySQL or MariaDB installed and running also in your machine. Note: For the next demo, I’ll try to include it as part of the docker-compose.yml to reduce set up complexity.

The rest of the pieces are Python packages, and to get them, just:


(venv)$ pip install grimoire-mordred

A couple of comments:

I strongly recommend you to run this in a Python3 virtual environment
Use Python3

With grimoire-mordred I get all the GrimoireLab Python packages I need to run the analytics platform. So, it’s time to start configuring the scope of the analysis.

Configuration files

First, a .json file that describes the ecosystem to be analyzed. An ecosystem (OSW for example) is described by the projects under its umbrella (OSWeekends, GingerCode and GoblinDB for example in this case) and these are described by the data sources (git, github, meetup and slack in this case) and the repositories to get the data from. So, it ends looking like this (the file is truncated to make it more readable):


{
    "osweekends": {
        "git": [
            "https://github.com/OSWeekends/Guilds.git",
            "https://github.com/OSWeekends/memezinga.git",
            ...
            "https://github.com/OSWeekends/electric-skateboard.git",
            "https://github.com/OSWeekends/Open-Communities-summit.git"
        ],
        "github": [
            "https://github.com/OSWeekends/Guilds",
            "https://github.com/OSWeekends/memezinga",
            ...
            "https://github.com/OSWeekends/electric-skateboard",
            "https://github.com/OSWeekends/Open-Communities-summit"
        ],
	"meetup": [
		"Open-Source-Weekends"
	],
        "slack": [
                "C2QL47LJ3",
                ...
                "C8GLKD8M9"
        ]
    },
    "GingerCode": {
        "git": [
            "https://github.com/GingerCode/gingercode.org.git",
            ...
            "https://github.com/GingerCode/Art.git"
        ],
        "github": [
            "https://github.com/GingerCode/gingercode.org",
            ...
            "https://github.com/GingerCode/Art"
        ]
    },
    "GoblinDBRocks": {
        "git": [
            "https://github.com/GoblinDBRocks/GoblinDB.git",
            "https://github.com/GoblinDBRocks/GoblinDBRocks.github.io.git",
            "https://github.com/GoblinDBRocks/Art.git"
        ],
        "github": [
            "https://github.com/GoblinDBRocks/GoblinDB",
            "https://github.com/GoblinDBRocks/GoblinDBRocks.github.io",
            "https://github.com/GoblinDBRocks/Art"
        ]
    }
}

Second, a Mordred configuration file is needed. In my case, it looks like this (API tokens are masked for security reasons):


[general]
short_name = OSWeekends
update = false
min_update_delay = 10
debug = true
logs_dir = /tmp/logs

[projects]
projects_file = projects-osweekends.json

[es_collection]
url = http://localhost:9200
user =
password =

[es_enrichment]
url = http://127.0.0.1:9200
user =
password =
autorefresh = true

[sortinghat]
host = localhost
user = root
password = root
database = osw_shdb
load_orgs = false
unaffiliated_group = Unknown
affiliate = true
autoprofile = [github,git]
matching = [email-name]
sleep_for = 0

[panels]
kibiter_time_from= "now-90d"
kibiter_default_index= "git"

[phases]
collection = true
identities = true
enrichment = true
panels = true

[git]
raw_index = git-osw-raw
enriched_index = git-osw
studies = [enrich_demography]

[github]
raw_index = github-osw-raw
enriched_index = github-osw
api-token = ************************************
sleep-for-rate = true
sleep-time = 300

[meetup]
raw_index = meetup-osw-raw
enriched_index = meetup-osw
api-token = **********************************
sleep-for-rate = true
#sleep-time = 300

[slack]
raw_index = slack-osw-raw
enriched_index = slack-osw
api-token = ********************************

And finally, a menu.yml is needed to have a side bar for easy panels navigation. Mine looks like this one. Note: In next releases, this file won’t be needed, because it’s already part of grimoire-mordred package

Go for it

After that, I am ready to run the analysis:


(venv)$ mordred -c osweekends-mordred.cfg

As outcomes, you’ll get:

A dashboard with a side menu based for easy navigation through the different data panels
A Kibana environment to play with the data, with some extras like the network visualization plugin (100% free, open source software)
An Elasticsearch server with some indexes you can query using Elasticsearch API with Python, javascript, etc.

Community network analysis demo — Community network analysis during April’s OSWeekends

What’s next?

I hope anyone will be able to set up a demo from here, and if not, just let us know by commenting this post.

Of course, if you have also ideas about how it could be improved, that would be great! Please, let us know about them, and feel free to publish issues in the main GrimoireLab repository.