Lack of Maintainer Support for Concourse

[ChrisJBurns]

So, it's probably become apparent to folks who have kept track of this concern over the last year that Concourse hasn't got the support that would most would require from a popular CI product.

Can I ask the question honestly regarding if there are any plans to increase the Concourse support? Looking at the Pulse insights over the last month, 4 PR's were merged, out of them only 1 was an actual code change, the rest were just updating of dev certs, security patches and rebases. There were also 8 issues raised and 1 closed. There's always going to be more raised than closed, that I'm aware of, but the rate at which issues are being closed / resolved is pretty low. I'm sure if we extended this to the last year, it would show similar statistics.

In combination with this, the issues keep creeping up in numbers, with a large amount of them going unanswered/unresolved. Not to mention the discussions also mostly going unanswered.

I understand the time and care it takes to maintain a popular product, especially an open-source one, but at this point, we have to start looking at other options that are more frequently maintained if there isn't a plan to increase resources for supporting Concourse long-term.

[DanielJonesEB]

If there's sufficient demand of people willing to pay for Concourse support/maintenance, and we can corral them all together, then there are people who would be willing satisfy that demand given appropriate compensation.

There were attempts to get Concourse moved into a foundation, but the answer from VMware was "we're not going to do that as it provides us no business benefit."

EngineerBetter asked VMware if we could solve a problem for them by 'taking Concourse support off their hands' - we would have invested in spinning up Concourse support, maintenance, and ideally feature development in return for having been introduced to all of VMware's customers who would be likely to be willing to pay for support. We saw that as a win/win/win, where VMware no longer have to invest in Concourse to keep paying Cloud Foundry customers happy, EngineerBetter gets a stable revenue stream, and customers get a supported product. I had conversations with Concourse-advocates in VMware, about whether we could get all the Concourse users together to put such a proposition to them. Sadly, it never happened.

For years at EngineerBetter we'd wanted to find a way of sustainably supported and contributing to Concourse, but the problem was that Pivotal/VMware gave it away for free, so no customer was willing to pay for it. I was reluctant to gamble EngineerBetter finances on 'build it and they will come', and do sometimes wonder if in retrospect that was a mistake. Perhaps if we'd been more proactive in offering commercial support and maintenance back in 2017/2018, it could have tipped Concourse's trajectory towards wider adoption. Heck, we were even worked two different product ideas, Concourse-on-demand and multi-tenant-Concourse-as-a-Service-on-Kubernetes-with-scale-to-zero. EngineerBetter is no longer a going concern, having been acquired and snuffed out by Container Solutions.

I've spoken to ex-Pivots who were interested in a 'lifestyle business' of supporting Concourse. Having now worked with other CI/CD systems that just cannot achieve the same outcomes as Concourse, I wonder if there's a possibility that it could have a wider audience if there was a sustainable business model behind it.

This is a serious and genuine offer - if you work for a large organisation that would be willing to pay realistic amounts of money for Concourse support and maintenance, get in touch.

PCF/TAS/whatever-it's-called-now customers regularly pay 6-to-8 figures for their licenses, depending on their footprint. Who knows how much that might go up under the Broadcom leadership. Concourse is, in most of those installations, a vital component to keeping platforms deployed and up-to-date.

If each remaining PCF/TAS customer was willing to pay 10% of their CF budget for Concourse support, I'm sure that could be a viable business concern. The Concourse team I think peaked at eight engineers and a PM. Having four engineers full-time would be a significant uptick in velocity from where the project is now, and the costs of that, supporting staff and overheads in the UK is going to in the order of GBP500-750k/year. So we'd need to get enough users together who could commit that kind of spend on an annually-recurring support agreement in order to be viable.

[jenniferplusplus]

I'm at one of those companies that built an unsupportable mess on top of github actions, and I'm currently in the process of trying to untangle it. The most painful part is all the steps that cross between different repositories, k8s clusters, and argo apps. On paper, concourse is a great fit for us, and I'm trying to encourage adoption in at least those parts of the work flow. I think a managed concourse is definitely something people here could be sold on. We're already spending the money, it's just going to an absurd github bill. And I know I personally would be having a much easier time if the project was more active, with or without a managed provider to lean on.

FWIW, I think the painful part I described is pretty common among teams following gitops patterns (and others, but definitely gitops). Existing tools work great and provide a good experience up to the first deployment somewhere, and then there's nothing. There's no way to tie the dev, staging, and prod environments together into a coherent pipeline. There are some funded start ups working on products that fill this gap (Akuity and System Initiative that I know of), but I think concourse is already very well suited to do much the same thing, and is much more mature.

[ChrisJBurns]

@jenniferplusplus I agree, the single view of an entire application definitely is what Concourse has that other CI systems don't - I've really love the UI and always have (although there are some weird bugs in it). I've found that using CI systems like GitlabCI and Github Actions is great for open source tooling that are somewhat isolated in nature, but when it comes to applications that exist with the enterprise SDLC (such as microservices for websites), it's not easy (and in some cases impossible) to track that in a single linear pipeline as they all have their own testing phase within an organisation. Spinnaker was something I believe that Netflix came up with because they saw this problem of CI and CD and realised that there weren't really tools that offer that single pane of glass view - even then they used Jenkins for their builds and then Spinnaker to somewhat track the "pipeline to prod" views.

We use adhere to GitOps too, and the fact that the resource checks in Concourse allow for us to check when new versions of an application has been deployed to a cluster so that it can kick off the new phase of testing is a god send. We had to write our own resource to do this, but I agree with @DanielJonesEB point on the daggerverse capability, it would be great to have this in Concourse also so folks can share and collaborate on what normally are the same old things (build a maven project, build an image, etc etc). I even wrote added support for using Cosign to sign images using the registry-image-resource and haven't got a single comment from any authors regarding this, so it isn't encouraging for contributors to spend time fixing / adding new features when the work they do wouldn't even be acknowledged from the maintainers (not the maintainers fault of course, time is a precious resource in the open-source world, especially when you aren't being paid to maintain the repository).

I completely agree with @DanielJonesEB point as well on the fact it doesn't need to compete with the likes of Tekton (it's pretty much the only thing in the CI space that does what it does right now), however it can start with just resolving the issues people are getting that have already adopted Concourse and getting what would be seen as the more obvious issues out of the way such as #403. Then traction on bugs / new issues, and just general increased velocity of changes over time would somewhat restore a bit of faith of the future of the project and a somewhat track in the sand could start to appear.

But as @DanielJonesEB has described, there have been some very motivated people over the years that have tried to get this ship moving but have come up against other obstacles.

[DanielJonesEB]

I'm definitely going off-topic here, but I've got time to kill and enjoy reflecting on this stuff.

So many of the things you both write resonate with me and my experiences.

the single view of an entire application definitely is what Concourse has that other CI systems don't

Absolutely! This drives so much organisational behaviour. Without a single view of the pipeline to production, people just don't think of it as a thing. Instead every team worries about their own bit, and no-one is responsible for the whole path. I've seen this so many times as a consultant. When the pipeline is a thing that you can visualise and point to, people treat it differently. The 'sausage machine' that makes software is evident, instead of being this invisible monstrosity with delays, cycles, and 'inventory' piling up.

Often when I've consulted on organisational efficiency, one of the first things to do is draw a value stream map. A good set of Concourse pipelines are a living, breathing value stream map. I don't think my friends at Armakuni would mind be saying that there was a time when they were looking at extending the Concourse UI to pull in data from Jira (eew!) and annotate pipelines with wait times and work times. That would be such a great lens into how a software machine delivers value.

Now, on the single view part - I think this is a double-edged sword, and something that made me realise that Concourse is at loggerheads with popular practice. A legitimate criticism of Concourse is that it encourages monolithic pipelines: one YAML file that knows about (and is thus coupled to, even if it's unidirectional) all the things. I can see why some would consider this an organisational anti-pattern.

However, some rebuttals:

• Some poor git has to work out what got deployed where, and why production is broken. That person needs a view like a monolithic Concourse pipeline.
• Whether you design it explicitly and monolithically, or allow it to emerge in a distributed fashion, there is still a path to production. Would you rather know what it is, or have all the individual components do their own shizzle and hope for the best?
• Some people (like @jenniferplusplus and myself!) want to prove that the composite system works as a whole. I care that the entire system works, not just that one team's microservice worked when it was contract-tested. I know a Senior Staff at a very prominent UK company with 1,000 developers, 500 microservices, and constant production bugs because they don't do end-to-end/system/integration testing. If you can test everything together, you should, until you reach the scale where it becomes impractical.

Is Concourse's tendency towards monolithic pipeline a forcing function? Is encouraging a single definition of the path to production, coupled to all the things, a good idea? In continuous delivery we block the pipeline and have a 'stop the line' moment - a forcing function to make sure we don't make a bad situation worse by continuing to push code when something is broken. Sometimes these patterns are valuable because of the behaviour that they drive. In the case of monolithic pipelines, they drive coordination and shared responsibility.

Is there an alternative? Yes. It occurred to me a few years ago that if every hipster wants to have isolated CI workflows for just their microservice, then you can get the benefits of Concourse's UI by building it after the fact. Some sort of system for reverse-engineering that pipeline view from distributed workflows.

Existing tools work great and provide a good experience up to the first deployment somewhere, and then there's nothing.

I think I might be misunderstanding your intent here @jenniferplusplus and going off in a different direction, but the thing that resonates with me about your statement is that Concourse is great for living, breathing, ongoing things. I saw a Concourse pipeline as a vital part of an application (or in my domain, we tended to deploy platforms and all the apps on top of them). People expect their mobile apps to just auto-update. Enterprise customers should expect their platforms and third-party software to just update itself too. Much like software without tests is 'working by accident', systems without a living, breathing CI/CD pipeline are ephemeral flashes of lightning in a bottle. They're not temporally resilient.

there have been some very motivated people over the years that have tried to get this ship moving but have come up against other obstacles

You make a good point. I'd been wanting to share that people really tried more with the intention of letting y'all know that someone cared. But it's a good reflection point that if those people (and I was a minor contributor to that effort) couldn't make it work, then maybe that's a sign.

[jenniferplusplus]

@DanielJonesEB I think we're understanding each other, yes. I've been describing Concourse as a task orchestrator. It can monitor almost any arbitrary thing and schedule tasks in response. And then monitor the tasks, and take action in response to the state and outcome of those. And it does a reasonably good job of making that whole arrangement visible, and even allows a good amount of human intervention. All of which is exactly the missing piece in so many CI/CD systems.

I also think you're right that the abstract and hard-to-see nature of those existing CI/CD systems makes it hard for a lot of people to conceptualize it as a thing.

[pidster]

A few thoughts...

Whatever it's called, the thing that runs CI is a critical piece of developer infrastructure. Everyone needs one, so the addressable market size is substantial.

The complexity of process and requirements in most contemporary applications means that, no matter the complexity of what you're building, you are building two applications, (not one), because you build an application (a build & test pipeline) that produces the second one, the (business) application. I think that undervaluing this, and not treating it as a product problem, is the main factor that leads to the kind of problems @jenniferplusplus described above.

What happens in most CI pipelines (that we should probably call CI applications) is a bit of a mystery from a zero trust perspective, this needs a radical improvement.

@DanielJonesEB your list above is a pretty good start FWIW.

[DanielJonesEB]

I appreciate the replies in this thread!

not treating it as a product problem

Something I see all over the place (as a consultant brought in to help dysfunctional development orgs, so selection bias is at work here) is a lack of anyone thinking of the software development process as a product. It's like everyone's busy thinking how to make the end application, and no-one's thinking about the assembly line. So many developers, team leaders, development managers, heads of engineering, CTOs and VPs seem to have a complete blind-spot on the higher-order thinking of building the machine that builds the software - whether that's the human and policy parts, or the automated parts.

Which is a long-winded way of saying: I agree, and I think it's a symptom of a larger problem. Which maybe what led the folks at Armakuni to consider seeing an expanded version of Concourse as a lens through which one can see the delivery machine.

[jenniferplusplus]

I have similar questions and concerns, myself. There's clearly still some movement on the project. And based on my having just done it, we can still make some enhancements. But I also worry that it's not much. I think Concourse can satisfy a lot of really critical unserved needs for devops/platform teams. Really any organization that's following modern development practices for building and operating a service.

I would like to help build and maintain it, but it's not clear what the governance structure is anymore, and that makes it hard to approach. I would like to help, but I'm not sure how I would do more than minor tweaks as a newcomer.

[taylorsilva]

Ex-Pivotal and Concourse maintainer here. I pop in every now and then when work isn't too stressful. I still like working on this project when I can fit it in on weekends, which has sadly not been as frequent as I would like. It's been sad to see the corporate support wane. I'm going to try and reach out to some vmware contacts to see what the picture on the other side is right now. Maybe there's a conversation to be had...

---

I've got a soft spot for Concourse. I'd find it easier to leave in the past if it wasn't for 1) the sense of missed potential; 2) the fact that trying to get other things to achieve the same outcome is a raging pain in the backside; EDIT and 3) the fact there are people using a thing that need support and might be willing to pay for it.

Well said @DanielJonesEB! Exactly captures my feelings about the project as well and why I keep coming back around. This quote from one of Nat's Simpler Machines newsletter has a similar sentiment:

The current CI/CD landscape is, honestly, a disaster, especially for handling large pipelines of the type that Concourse is best at handling. There is no good alternative to Concourse for managing large distributed systems deployed to VMs.

---

I would like to get some organizational structure back into the project. I have relevant project knowledge and expertise that would be useful to kick-start things in this repo again. I can merge stuff in and assuming https://github.com/concourse/governance still works, I can elevate others permissions as well if folks want to help maintain.

I really want to do something to keep this project going, but I feel a bit lost. I don't want to end up being another burned out OSS dev, which is why I don't spend much time here. That said, I can definitely spend more time here consistently. Figuring out a path forward with a group of people would go a long way for the project I think.

[jenniferplusplus]

My recent experience suggests that something is broken in concourse/governance. I keep getting automatically invited and then kicked from the organization.

Not the point, I know. But I wanted to mention, since you brought it up.

[taylorsilva]

Honestly, that repo broke so much even after it was first written lol The idea is great, but execution has been lackluster 😭

[anEXPer]

I am, as far as I know, the most recently-departed "technology leader" "responsible" for Concourse at VMware prior to the Broadcom acquisition.
With the caveats that the personnel involved have definitely changed at least some since the acquisition, and the management thinking is clearly very different in other areas, here's what I can tell you in answer to this:

1. The currently documented governance model and roadmap are falsehoods-by-decay. Conceivably this could be changed by attempts to actually use the governance model, but I have doubts. The actual governing entity is, at this time, roughly, product leadership associated with some Broadcom business unit or another.
2. The actual funding and fate of the basic Concourse maintenance you have observed is directly tied to the revenue of Tanzu Application Service (TAS)/Pivotal Application Service (PAS)/Pivotal Cloud Foundry (PCF) - for those unfamiliar, these are all names for the same thing, the principal commercial packaging of Cloud Foundry (CF).
   a. CF R&D and TAS R&D are both deeply dependent on Concourse and will remain so absent a huge investment; my hand-wavey estimate of the implication of that is that basic maintenance can reasonably be expected to continue for at least three and probably more like five years, just based on public information about customers and generalizations about how such customers (large banks, etc) move between systems.
   b. Further, many PCF customers are also dependent on Concourse for their installation and management of the platform, and will remain so, in much the same way as above.
3. Nothing I know, public or private, points to any plans at Broadcom for increased investment in Concourse, and various things I know weigh against it.
4. There are various foundations that would be good homes for Concourse. My attempt to move it to the Cloud Foundry Foundation or Cloud Native Computing Foundation foundered on the rocks not even of hard policy opposition, but soft prioritization/cost prohibition. There was also a lack of evidence that concerns about Concourse's fate could be impactful to TAS's revenue or costs.

I love Concourse. Part of the reason I left VMware is that I could not execute their policy in connection to it. I would love to see, and would happily lend support to, a movement to exert pressure towards giving Concourse the longevity and development it deserves.

A final note: Concourse's deep dependency on Garden, among other CFF-owned codebases, make CFF my first choice, but many places would be better than the current state.