Maintaining a consistent data infrastructure is challenging, especially at scale. You build an ETL pipeline used across multiple repositories. Then, months later, you need to roll out a new feature that must be applied to all of them. Manually updating each repo is time-consuming, error-prone, and risky: miss one, and you introduce drift; update one incorrectly, and you might break a production flow.
Template scaffolding tools like Cruft and Copier solve this by enabling automatic updates to existing projects. Instead of just generating initial project structures, they can push changes across your entire platform.
This post compares both tools, focusing on their practical differences for teams managing multiple projects and the lessons we learned implementing automated template updates in production.
One of the most widely used tools for project scaffolding in the Python ecosystem is Cookiecutter. It lets you define reusable project templates using Jinja2. You generate new projects by filling in values like project_name or author. Cookiecutter is simple to use and mature, having been around for a long time.
However, Cookiecutter has a significant limitation: it only works at the time of project creation. Once a project has been generated, there’s no built-in way to track updates to the template. You also can’t apply those changes to existing projects. This becomes a problem when you’re working at scale.
Imagine you have a shared template used across dozens of services, pipelines, or internal tools. Over time, that template changes. Maybe you need to include new CI/CD jobs, adjust configuration defaults, or add new features. Without an automated way to propagate those changes, you’re left manually applying edits to every project. That’s manageable with two or three repositories. But it becomes nearly impossible when the number reaches dozens or hundreds.
This is the gap that Cruft fills. In our case, we had already been generating projects with a Cookiecutter template, so integrating Cruft was straightforward because it builds directly on top of Cookiecutter. It adds functionality to track and update any Cookiecutter-based projects. When you create a project using Cruft, it records metadata about the template. This includes the Git source, the exact commit used, and the context variables. All of this gets stored in a .cruft.json file inside the generated project. This link allows Cruft to later fetch updates from the template and apply them to your project.
When updating, Cruft checks the current state of your project against the latest version of the template. It regenerates the template with the same context and compares the result to your local codebase. Then it applies the differences. If changes conflict with local edits, you’re prompted to resolve them using your regular Git merge workflow.
That said, running cruft update and resolving conflicts manually for each repo is still a fair bit of work. To cut down on developer work, we integrated Cruft into our CI/CD pipeline. If there are any changes to the template, a merge request is automatically created once a week. This includes all changes since the last update. The responsible developer can then review, approve, and resolve any merge conflicts as needed.
The main challenge we ran into was how Cruft handles merge conflicts.
When a conflict occurs during an update, Cruft generates a .rej file to indicate that it couldn’t apply certain changes from the template. This caused a couple of problems for us.
First, the original file stays unchanged, even if the update partially fails. That means our CI/CD pipeline might still succeed, even though the update wasn’t fully applied. This is especially problematic because Cruft still updates the commit hash in .cruft.json to the latest version. This happens even if the template changes aren’t relevant to a specific repository. As a result, a merge request might be accepted and merged with unresolved conflicts. Important changes could be silently skipped.
Second, .rej files are inconvenient to deal with. Unlike conflict markers that show changes inline and are easy to resolve, .rej files require manual patching. For example, an inline conflict might look like this:
Python:
1<<<<<<< before updating
2def greet(name):
3 print("Hello, " + name + "!")
4=======
5# New version from the updated template:
6def greet(name):
7 print(f"Hi there, {name}!")
8>>>>>>> after updating
This only requires the developer to edit this specific file. They just need to keep the code that is useful for their project. For most minor changes, this can be handled directly within the IDE or on platforms like GitLab or GitHub, allowing conflicts to be resolved fairly quickly.
There’s no option in Cruft to switch to a more user-friendly conflict resolution method. While this has been raised in an open GitHub issue a few years ago, no fix has been merged yet.
Because these issues were significant for our workflow, we decided to try out Copier, a more modern scaffolding tool that handles updates differently.
Copier hasn’t been around for as long as Cookiecutter, but it has quickly gained popularity. This is due to its modern approach to code scaffolding. It is built with Python and powered by Jinja2, just like Cookiecutter. Copier also has a built-in way to update your projects. So there’s no need to rely on two different repositories. Beyond updates, Copier offers several improvements. It also provides a better user interface for prompting template variables, including type-safe prompts with validation. It also supports conditional file inclusion based on user responses. It also has a more intuitive conflict resolution system that shows clear diffs instead of generating .rej files. And if .rej files are preferred anyway, there is the option to change the default output of merge conflicts.
There is a lot more room for configuration. You can define input types like strings, booleans, and integers. You can set intelligent defaults. You can also include or exclude entire files and directories based on how users answer prompts.
If you’re starting with an existing Cookiecutter template, you’ll need to adjust some syntax and structure. Following the Copier documentation, this is very manageable. Variable names and prompt questions are set up a little differently. Most important is to correctly migrate the .cruft.json file to .copier-answers.yml. This is where Copier will look for the project’s metadata.
The approach for including a job in the CI/CD to create a merge request is very similar to Cruft. The main difference is in the commands used. Here is an example of the job in GitLab CI/CD:
YAML:
1create_merge_request:
2 stage: copier
3 image: python:3.11-slim
4 before_script:
5 - apt-get update
6 - apt-get -y install git curl
7
8 # https://git-scm.com/docs/git-credential-store
9 - git config --global credential.helper store
10 - echo "https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.com" > ~/.git-credentials
11
12 - git config --global user.email "<email>"
13 - git config --global user.name "<username>"
14
15 script:
16 - copier update --defaults
17 - git restore --staged .
18
19 - |
20 if [ "$CI_COMMIT_REF_NAME" == "main" ]; then
21 git checkout -b copier-update
22
23 # Check for changes
24 if git diff --quiet; then
25 echo "No changes detected. Exiting."
26 exit 0
27 fi
28
29 COMMIT_MSG="chore: New updates to Copier template detected"
30 DESCRIPTION="This MR has been created to remain up-to-date with the Copier template."
31
32 # Push changes to git
33 git remote set-url origin https://gitlab-ci-token:${SEMANTIC_RELEASE_GL_TOKEN}@gitlab.com/${CI_PROJECT_PATH}.git
34 git add .
35
36 git commit -m "$COMMIT_MSG"
37 git push origin copier-update
38
39
40 curl --request POST --header "PRIVATE-TOKEN: ${PROPERLY_SCOPED_TOKEN}"
41 --form "source_branch=copier-update"
42 --form "target_branch=main"
43 --form "title=$COMMIT_MSG"
44 --form "description=$DESCRIPTION"
45 "https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/merge_requests"
46 --form "remove_source_branch=true"
47 --form "squash=true"
48 fi
49 rules:
50 - if: $CI_PIPELINE_SOURCE == "schedule"
All in all, Cruft makes sense if you already have an existing Cookiecutter template. It also works well if your workflow involves minimal overlap between the files you customize per project and the files that get updated in the template. Essentially, situations where merge conflicts are rare. In these scenarios, and if you don’t need complex template logic, Cruft’s direct Cookiecutter compatibility can be an advantage.
However, for most other use cases, Copier is the clear way to go. It offers more freedom in how you handle updates. It provides better conflict resolution tools. It also includes modern templating features that make both template creation and project maintenance easier. The initial migration effort from Cookiecutter to Copier pays off in the long run. You get improved developer experience and more reliable update processes. If you’re starting fresh or frequently encounter merge conflicts with your current setup, investing in Copier will likely save your team significant time and frustration down the road.
Copier has now been integrated into our workflow for the last 4 months and has already saved a lot of hours that would have otherwise been spent updating the existing projects.
Or would you like to know more about automation?
Contact us, we are happy to help you.