« Changelog History


Concourse v5.0.0 Release Notes

Release Date: 2019-03-06 // about 5 years ago

  • 🚀 This release is a doozy. You should probably read these release notes in full - there are a ton of substantial new features and a good (bad?) amount of breaking changes.

    Sorry this took so long! The holiday season took its toll, but we also got a bit overzealous with piling feature work on master, and well, we restructured the entire project and re-created its pipeline from scratch, so that didn't help.

    🚀 On the plus side, the project restructure is now done, and we'll be implementing a new release process soon that should prevent these kinds of hold-ups from happening again in the future.

    🚀 Special thanks to the many individuals in the community who took part in this release - whether you submitted a PR, helped triage issues, helped people out on the forums or in Discord, or simply cheered us on, every little bit helps keep the project humming along. We deeply appreciate it, and look forward to delivering y'all a better and better CI system - hopefully, more continuously.

    🍱 🔗 feature, breaking

    ⬆️ We have done a major internal overhaul of how resource versions are stored. As a result, the version history for each resource across your pipelines will be re-set upon upgrading to v5.0.

    ⬆️ The upgrade does however preserve the state of which versions were disabled, and the data relating versions to builds they were inputs to and outputs of.

    🔧 In versions prior to v5.0, resource version history was associated to a pipeline resource by name. This meant that if you changed a resource's configuration or type, those old versions would actually stick around, even though they may technically no longer be appropriate.

    With v5.0, resource versions are now tied directly to an anonymous "resource config" - basically the source: and type: for the resource. Pipeline resources instead point to a config, and if their source: or type: changes, they'll point to a new config with its own version history.

    This improves the correctness of the system as a whole, eliminating the need to ever "purge" the history of a resource.

    In addition, now that versions are tied directly to their configs, check containers are also shared across teams, reducing the overall container count. As a result however we limited who can fly intercept check containers.

    🏗 Building on this change, we are currently experimenting with improvements that can now be made to reduce the overall checking overhead across a Concourse cluster that has many equivalent resource definitions across pipelines and teams. This is currently off by default while we learn more about the implications - see Global Resources for more information.

    🍱 🔗 fix, breaking

    🔧 We have removed --allow-all-users as almost every use has been a misuse. You must configure users explicitly now instead. This was done for development environments but even those were trivial to switch to a local user whitelist.

    🔧 If you were setting this flag before, you probably didn't mean to - setting this with GitHub oAuth configured, for example, would allow literally everyone to be a part of your team and manage your pipelines.

    ⬆️ After upgrading, any teams that had this configured will preserve the behavior from before - they will continue to allow all users. The next time the teams are configured, however, you will have to specify something else, as the CLI no longer has the flag.

    🍱 🔗 feature, breaking

    The concourse binary distribution has been rejiggered. Rather than a self-contained binary, we now ship it as a .tgz containing the binary and its dependencies pre-extracted. The .tgz should be extracted somewhere like /usr/local, resulting in /usr/local/concourse/bin/....

    👷 The main benefit of this is simplification and faster startup. The concourse worker command no longer needs to extract resource types/etc. on start, so this speeds that up quite a bit.

    🚀 The concourse binary no longer directly embeds Garden-runC code, and instead ships alongside the gdn binary, copied from their releases. This simplifies the interface for configuring Garden and allows us to leverage their build process rather than risking deviation.

    The "breaking" aspect of this is that if you have been passing esoteric flags to Garden you'll have to switch to using a config file via --garden-config instead, or pass them as env vars (e.g. CONCOURSE_GARDEN_FOO_BAR) - flags are no longer supported as those relied on directly embedding their code.

    🍱 🔗 feature, breaking

    🏗 Workers can now be configured to periodically rebalance so that they don't end up all forwarding through a single web node. This is done by setting the --rebalance-interval flag on concourse worker. The rebalancing makes sure to drain in-flight connections and should not disrupt any in-flight builds.

    🚚 Along the way, we removed support for direct worker registration. The --peer-ip flag is no longer available on concourse worker. To transition to 5.0, just remove the flag - the worker will register via forwarding instead.

    👷 Forwarding is more secure as it doesn't require opening your workers up to inbound traffic. It's easier for us to just focus on one registration method and make sure it works well.

    👷 This also sets us up for enforcing TLS for all traffic to the forwarded workers in the future (#2415).

    🍱 🔗 feature, breaking

    🚀 The Concourse BOSH release has been redesigned and is now centered around the concourse binary.

    ⚠ > warning: Be sure to recreate your workers after or during the deploy, as the location that the worker stores volumes has changed and the old volume directory will not be cleaned up, effectively leaking disk usage.

    ⚠ > warning: The additional_resource_types property can no longer be configured. We plan to add another mechanism for co-located resources in future releases.

    🚀 The concourse release no longer needs to be deployed alongside a garden-runc BOSH release, and instead embeds the gdn binary directly.

    🚀 Along the way, we have adopted BPM and now use it for deploying the web node. We also enforce a higher nofile limit which should make large-scale deployments more...scaley.

    🍱 🔗 fix, breaking

    Two flags have been modified to be more consistent with other flag syntax:

    🌐 concourse web --vault-auth-param foo=bar should now be specified as concourse web --vault-auth-param foo:bar (note the :).

    🌐 concourse web --tsa-team-authorized-keys team=path/to/key should now be specified as concourse web --tsa-team-authorized-keys team:path/to/key (note the :).

    🍱 🔗 feature

    The Concourse GitHub repository has been completely restructured. This isn't really a feature per se, but it should make contributing a lot easier.

    ⚡️ More on this on our blog post: The Great Process Update of 2018.

    🍱 🔗 feature

    🏗 A new resource, the registry-image resource, has been added to the core. This resource is intended to replace the docker-image resource image for image pulling and pushing (but not building).

    🐳 This resource improves on the docker-image resource in a few ways:

    🐳 It doesn't run Docker to fetch the image - it's written in pure Go, using the google/go-containerregistry package. This makes the implementation much less error-prone.

    Because it doesn't run Docker, it doesn't need a privileged container. The fewer privileged containers in your cluster, the better - especially in light of [v4.2.3](https://github.com/concourse/recent CVE fixes/releases/tag/v4.2.3).

    By focusing solely on fetching and pushing, the resource is much smaller and simpler. It also has test coverage!

    The output has pretty colors.

    This all results in much faster, more efficient, and resilient image fetching. We recommend everyone to try switching your image_resources and Resource Types over - in most cases this is just a matter of replacing type: docker-image with type: registry-image.

    We intend to deprecate and phase out support for the docker-image resource in favor of the registry-image resource. We can't really do this until there's a solid direction for image building - preferably with a task, not a resource. This is a more natural split, and supports building images without pushing them - a long awaited ask of the docker-image resource.

    🏗 An experimental task for this is available at concourse/builder. This is not yet official, but we've using it in our own pipeline and it's been pretty solid. Feel free to give it a try!

    🏗 The next step from here is to actually kick off an RFC for reusable tasks - we're still collecting our thoughts for that in (RF)RFC #7. Once this is done we can formalize concourse/builder.

    🍱 🔗 feature

    We have introduced the first phase of role-based access control!

    👀 Right now there are only a few statically defined roles. We started off by supporting the common request of having read-only team members ('team viewer'), and adding a slightly less powerful 'team member' role. See User Roles & Permissions for more information.

    Here's a quick rundown of how things have changed:

    Existing team auth config will be transitioned to the Team Owner role - that is, anyone that can authenticate prior to the upgrade will now be authenticated as an owner of their team. This role is the closest equivalent to what they could do before.

    The main team still has special admin power, with the slight tweak that only users that are an owner of the main team have admin capabilties.

    Before, teams members could rename or destroy their own team. Team owners no longer have this power - only admins can do this.

    The Team Member role is a new role that allows users to have full read and write powers within the team, except for being able to modify the team itself.

    🏗 The Team Viewer role is a new role that allows users to browse the team's pipelines, builds, resources, etc. without permitting any sensitive operations (like fly get-pipeline or triggering builds).

    ⬆️ For a detailed breakdown of each role's capabilties, see the Permission Matrix. To learn how to configure these roles after upgrading, see Setting User Roles.

    If you're curious about the design process for this feature, check out RFC #3 (RBAC)!

    🍱 🔗 feature

    We have replaced resource pausing with resource pinning.

    📌 Resource pausing had the effect of disabling the periodic checking for the paused resource. However we found that in most cases it was being used in combination with disabling versions to effectively pin a resource to the most recent available version.

    However, with global resource versions, each resource actually points to a shared history, so pausing checking wouldn't be enough - if any other pipelines had the same resource, new versions would still arrive!

    So instead, versions can now be pinned individually via the web UI or via the pipeline config (see version). Pinned resources will also skip periodic checking, but now even if the checking still happens (because some other pipeline had it un-pinned), the resource will stay pinned to the desired version.

    📌 A comment can also be left on pinned versions for explaining to your team-mates why you decided to pin the resource.

    ⬆️ During the 5.0 upgrade, paused resources will be automatically transitioned to their pinned equivalent, by pinning the resource to the most recent available version. A comment will be left on any resources that are migrated so that it's clear to pipeline users.

    🍱 🔗 feature

    👀 Task ((vars)) received a bit of an overhaul, thanks to a PR by @ralekseenkov!

    Values for task ((vars)) can now be provided during fly execute!

    In addition, values may be provided to a task step in a pipeline via vars.

    Tasks can now have ((vars)) pretty much anywhere in their config, not just in image_resource.

    In all cases, vars can also be satisifed via a credential manager, the same as before.

    Admittedly, there is now some cause for confusion with params. This may see clarification with reusable tasks. In addition, pipeline ((params)) will now be referred to as pipeline ((vars)) instead, for consistency.

    🍱 🔗 feature

    🍱 🔗 feature

    • 👷 Any volumes or containers that disappeared from their worker (possibly due to a worker being re-created and then coming back under the same name) will now be automatically reaped from the database. This makes it easier for Concourse to recover from this situation rather than erroring with file not found or unknown handle errors.

    🍱 🔗 feature

    🔊 Logs emitted by Concourse components will now be...slightly prettier? They're still JSON (sorry), but the timestamps and log levels are at least human-readable.

    ⚡️ If you've got anything parsing your logs, make sure to update it accordingly!

    🍱 🔗 feature

    👀 Concourse will now automatically retry fetching credentials when the request to the credential manager fails, thanks to a PR by @ralekseenkov!

    0️⃣ By default Concourse will retry 5 times, waiting 1 second between each attempt. This can be adjusted with the --secret-retry-attempts and --secret-retry-interval flags on concourse web.

    🍱 🔗 feature

    👍 Tasks are now permitted to have inputs, outputs, and caches with overlapping paths. This was a hold-over from older versions of the container runtime that did not support this.

    This means that for simple tasks that e.g. make a commit a git repo, you no longer need to copy the input to the output. Yay!

    🍱 🔗 feature

    • The put step can now be explicitly given a list of inputs to use, rather than using all of them. This can be used to dramatically speed up builds that have a ton of artifacts prior to a put.

    🍱 🔗 feature

    🌲 The fly login flow has been reworked a bit to better support logging in to a remote session. There's now a prettier landing page that detects when the token transfer fails by allowing you to copy the token to your clipboard instead.

    The auto-login prompt will also no longer ask for the token, because that disrupts the normal flow of the command. Previously it would ask for a token but then eat half of the keystrokes from then on. Now it just won't ask for a token.

    🍱 🔗 feature

    • 🏁 The concourse binary now has a generate-key subcommand to assist with - you guessed it - key generation. This is more portable to other platforms (I'm looking at you, Windows) and is more likely to generate keys that Concourse can actually accept (I'm looking at you, OpenSSH 7.8).

    🍱 🔗 feature

    • 🐧 The concourse worker command can now be given a --garden-use-houdini flag on Linux to use the "no-op" Houdini Garden backend for those odd cases where you don't really want containerization. (Use sparingly.)

    🍱 🔗 feature

    • 🏗 The timestamps shown in the build header will now transition to absolute instead of relative when the build is over 24 hours old. It wasn't very useful to see things like 128d 15h 30m ago when trying to compare old builds. Thanks for the PR, @Twiknight!

    🍱 🔗 fix

    You may have seen a scary error cropping up around your resources now and then. Something like worker_resource_config_check__resource_config_check_sessio_fkey references unreticulated spline.

    🛠 We fixed it. That thing doesn't even exist anymore. Don't worry about it.

    🍱 🔗 fix

    With Concourse 4.x configured with an oAuth provider such as GitHub, a user could log in via GitHub even if they weren't technically a member of any team. They couldn't do anything, mind you, but it was confusing that they were allowed to log in in the first place.

    This is no longer permitted.

    Similarly, fly login will also check to make sure you've successfully logged in to the target team and return an error if the team isn't in your token.

    🍱 🔗 fix

    🔧 The AWS SSM credential manager and the AWS SecretsManager credential manager previously had a turf war going on over the AWS_REGION environment variable. They both declared it as their own, meaning if you set it they would both try to be configured, which would fail.

    They now have separately namespaced env vars instead.

    🍱 🔗 fix

    • 🏗 fly intercept will now give a better error when it fails to execute the command (e.g. because bash isn't installed in the image).

    🍱 🔗 feature

    • 👷 fly execute can now specify input mappings via -m, which is useful when running with --inputs-from-job when the job renames some inputs.

    🍱 🔗 fix

    • 🚚 fly execute with --include-ignored will no longer blow up when files are removed locally.

    🍱 🔗 feature

    • The error message when a task's file refers to an unknown artifact source (i.e. the foo in foo/ci/task.yml) has been made more descriptive.

    🍱 🔗 feature

    • 👷 There's a new fly command for landing workers remotely, called... fly land-worker. This will initiate the landing process via the API and will ultimately result in the worker process exiting. (Which may end up being re-started by whatever process monitor you use, but hey, it landed.)

    🍱 🔗 feature

    • 👷 The web UI now explains why some get steps have a yellow icon, via a handy-dandy tooltip. (Spoiler: it means the job has never run with that version before!)

    🍱 🔗 fix

    🍱 🔗 feature

    • 🏗 fly watch can now be called with --timestamps to show per-line timestamps in the build output. Thanks for the PR, @pivotal-kahin-ng!

    🍱 🔗 fix

    • fly get-pipeline will now throw an error if the specified pipeline does not exist, rather than returning an empty pipeline config.

    🍱 🔗 fix

    • 🛠 Fixed various subtle UI issues with the dashboard page: #2430, #2434, #2435.

    🍱 🔗 fix

    • fly login will no longer prompt for your auth method when a username/password are given via flags. It'll deduce that you're trying to do local auth.

    🍱 🔗 fix

    • Task caches are now supported on Windows!

    🍱 🔗 fix

    • 🛠 Fixed an internal bug that made UNIQUE constraints for resource_configs ineffective (#2509). This was fairly low-impact, but database integrity matters!

    🍱 🔗 feature

    • 👍 BitBucket auth support has been re-introduced thanks to PRs to Dex and Concourse by @edtan!

    🍱 🔗 fix

    • 👷 The /api/v1/resources and /api/v1/jobs endpoints will now return [] instead of null when there are no resources or jobs, thanks to a PR by @pivotal-kahin-ng.

    🍱 🔗 feature

    • 👀 The dashboard page will now indicate whether you are seeing a pipeline because it's exposed by showing an ominous "eye" icon.

    🍱 🔗 fix

    • 🛠 Fixed handling of auth configs set from empty env vars - previously this would result in bogus Dex configuration (e.g. github:, with no org or team) and sometimes cause things to misbehave.

    🍱 🔗 fix

    • 💻 The legibility and anti-aliasing of text in the web UI has been improved.

    🍱 🔗 fix

    Cleaned up some dashboard behavior when there are no pipelines:

    👀 you can now see which team you're a member of, rather than one big 'no pipelines set' page

    the bar along the bottom will now show up

    💻 there's a fancy ASCII art UI now

    the search function is no longer shown (since there's nothing to search)

    the HD view has been disabled and just redirects to / instead, since there was nothing for it to show

    🍱 🔗 fix

    • 💻 The username part of the top bar will no longer detonate when viewed on a tiny mobile browser.

    🍱 🔗 fix

    • 📇 When a resource's metadata is super wide, it will remain cordoned off to the side rather than uncomfortably squishing the resource's get output. Thanks for the fix, @stigtermichiel!

    🍱 🔗 fix

    • Concourse will now send TCP keepalives for connections to the database. This will allow it to detect when the connection has been interrupted ungracefully. Thanks for the PR, @SimonXming!

    🍱 🔗 fix

    • 💻 The manifest.json href in the web UI used to be relative to the URL, meaning it was broken on any page except /. This is now fixed.

    🍱 🔗 fix

    • 🏗 The web node used to leak both a connection and a goroutine for each build that completed when configured to drain build logs to syslog. This is now fixed. Sorry about that!

    🍱 🔗 fix

    • The resources and resource types returned by fly get-pipeline will now be in a deterministic order, thanks to a PR by @edtan!

    🍱 🔗 feature

    • fly curl is a new command to assist with (hopefully occasional) manual API requests to Concourse. Thanks for the PR and collaboration, @simonjohansson!

    🍱 🔗 fix

    • The --tsa-authorized-keys flag is now optional, for situations where all authorized keys are associated to teams (via --tsa-team-authorized-keys). Thanks for the fix, @tlwr!

    🍱 🔗 fix

    • The fly status command will now let you know if your token has expired, rather than happily reporting that everything is fine.

    🍱 🔗 feature

    • A fly userinfo command has been added which will let you know which teams you are logged in to and which roles you have in each team.

    🍱 🔗 fix

    • 🛠 The positioning of the "no results" text when searching on the dashboard has been fixed.