Spegel –

Announcing Spegel v0.6.0

Tue, 16 Dec 2025 00:00:00 +0000

The v0.6.0 release of Spegel has been focused on increasing the test coverage and stability of Spegel first and foremost. Integration tests have been refactored and extended to ensure compatibility with both Containerd and Kubernetes. The work made it painfully aware that there was a lack of documentation about which versions are supported. This has been remedied with the expansion of the compatibility documentation alongside matrix integration tests. With the current release the Containerd versions that have been tested are 1.7.x, 2.0.x, 2.1.x, and 2.2.x. The Kubernetes versions tested are 1.32.x, 1.33.x, and 1.34.x. Worth noting that this is the last release of Spegel which will officially support Containerd 1.7.x and Kubernetes 1.32.x. Spegel may still work with older versions in the future but that is coincidental and now verification will be done in the future.

I (phillebaba) will be taking some time off from working on Spegel, to prioritize my personal life and focus on work that will allow me to pay the bills. From now until the beginning of February you should not expect any responses in either Slack of GitHub issues. I have also cancelled the community calls until February. My hope is that this release will fix some of the stability issues that have been reported over the last few months. If you are in need of support during this time you can reach out to me at contact@kvick.dev to inquire about support contracts.

I wish you all a happy holidays free from outages! 🎅

Containerd and Kubernetes Integration Tests

Testing is a critical component of delivering stable software. The challenge Spegel has is that it integrates with other tools that have many actively used versions. Both Containerd and Kubernetes offer different feature sets and behave differently depending on the version. Previously Spegel has depended on the e2e tests with Kind to verify integrations with both. This partially worked but had the downside that we could not easily control the Kubernetes or Containerd version used. Over the last month the tests have been refactored and split into separate Containerd and Kubernetes integration tests. This enables the Kubernetes tests to be simplified which enables it to run faster, while moving some of the complex test cases to the Containerd integration tests.

Doing this also enabled matrix testing. The same tests can now be run against all the officially supported Containerd and Kubernetes versions. The tests take some time to run, so PRs will only run the tests against the latest version. To get full coverage nightly tests have been added which will also be run before any release is cut.

The investment in updating the testing has already paid off as it has surfaced two different bugs in the Containerd integration. The hope is that it will ensure more stable releases in the future.

Sweeping Provider

A critical component of Spegel is the DHT which enables discovery of content on other peers. Each peer advertises the content present locally with a set TTL. This means that the content needs to be re-advertised before the TTL expires. Up until now this process would run on a fixed timer that would advertise all the content in one go. Which would create a spike in CPU and memory usage. Thanks to the great work done by the maintainers of go-libp2p-kad-dht there is a new solution called sweeping provider that reduces the resource spikes by batching content advertisement in an intelligent manner. The old solution has been completely replaced by the new provider so expect to see some reduction in top level resource consumption.

Dual Stack Support

Alongside the new provider implementation some of the address advertisement logic was refactored. A lot of this code has not been touched since the initial implementation. When initially implemented only single stack Kubernetes clusters were considered. That being clusters only using IPv4 or IPv6. While not as common there are people running dual stack clusters, which would at random times not work. To resolve this new address selection logic has been added which is aware of the stack which is locally supported and returns addresses with a priority for IPv6.

Announcing Spegel v0.4.0

Sun, 21 Sep 2025 00:00:00 +0000

Summer has come and gone, so it has been a while since the last release of Spegel. This has given a lot of time for refactoring and general cleanup of tests and other functionality. A lot of this work is a requirement to implement more complicated features in the future. The v0.4.0 release still contains some changes that are worth highlighting, especially as it contains a deprecation which will be removed in a future release.

OCI Volumes

While not a new feature per se, checks have been added to the e2e tests to verify that OCI volumes are compatible with Spegel. The feature is implemented using the existing CRI API, which means that no additional changes were required in Spegel to support the feature. Officially supporting OCI volumes opens up a lot of doors, especially for AI use cases, as the artifacts tend to be large and pulling within the cluster will be faster than egressing to an external registry. The OCI volume feature is still in beta hence disabled by default. Support for OCI volumes will become more relevant as the feature graduates allowing other projects like kserve start supporting the volume type, along with greater community adoption.

Resolve Latest Tag Deprecation

A challenge some users face when reusing image tags together with Spegel, is how to update the digest which the tag resolves to. While it can be debated whether or not reusing tags is a good practice, it is something that parts of the community do as part of their release strategy. While the suggested solution in the Updating Latest Tag guide works for most, there are those that are not able to use mutating webhooks. Early on the resolveLatestTag flag was added to stop Spegel from resolving any image using the latest tag while resolving all other tags. This works well as long as the tag that is being reused is latest, but does not work with any other tag name.

There was a need to support more generic use cases which allows ignoring other tag names or even pattern matching. A new configuration parameter called registryFilters was added by @pepordev in #1003 that allows regex pattern filtering of registry requests. This allows a lot of customizability in the types of images that need to be ignored by Spegel, not just only specific tag names. This would allow disabling tag resolution for the dev tag for images from docker.io.

registryFilters:
 - ^docker\.io/.+:dev$

This new feature means that resolveLatestTag is no longer needed as the same functionality can be expressed using the new feature. The old parameter is now deprecated and will be removed in the next release. If you are using it please update your configuration. The following registry filter will achieve the same behavior as the flag did.

registryFilters:
 - :latest$

Verifying Spegel

It has been a long time coming, but we finally have a reasonable method of verifying Spegel’s core functionality after an installation. Previous documentation had suggested looking at request logs to determine if Spegel is properly mirroring requests or not. Which not so surprisingly was a horrible UX, especially for new users. The distributed nature of Spegel along with the requirement to keep Spegel stateless has made it challenging to implement a good solution. The changes done in #986 changed the default logging verbosity of request logs to be opt in. Removing the ability to verify request logs easily, forcing the implementation of a new solution.

The debug web view is now enabled by default, serving the page on the same port as metrics and tracing. The debug web view has a new metric called last successful mirror which indicates if the Spegel instance has successfully served content. If the value is pending, it means that this has yet to occur.

There is now a new Verifying Spegel Is Working guide which walks through how to force an image pull on a specific node and then verify that it was successfully mirrored by Spegel.

OCI refactoring

The proxy component of the registry has undergone some refactoring to be more OCI aware. This was necessary to implement current and future features. This release contains two fixes that were only possible with the refactoring. Previously if a mirror request fails partially through downloading a layer, the request would be retried with a new mirror. The problem with this implementation is that the request would be retried from the beginning, causing the served content to be incorrect. Being aware of the request type and optional range requests enables retriable requests that starts after the last byte that was written.

Additionally timeouts have been added to all requests other than GET requests for blobs. Networking is assumed to be reasonably fast within private networking, allowing the proxy to be slightly more aggressive with it’s timeout. All forwarded HEAD requests have a timeout of 1 second while GET requests have a timeout out of 2 seconds. The goal is to move on to the next mirror quickly rather than indefinitely waiting for a response from a Spegel instance that is either not running or is already overloaded. The only forwarded request that does not have a timeout set is GET requests for blobs. Figuring out a suitable timeout for the request is a bit more complicated as the content served can be anywhere between 1 Kilobyte to 1 Terabyte. There is ongoing work to implement a timeout strategy for these requests to avoid having requests wait forever.

Announcing Spegel v0.3.0

Fri, 06 Jun 2025 00:00:00 +0000

Containerd v2.1.0 unlocks a major piece of the puzzle for Spegel. This release removes key limitations that previously held back development. A big part of the work in v0.3.0 has been making sure everything plays well with the new version. That meant refactoring internals and updating the end-to-end tests to run against it. It will take some time before this version makes its way into most cloud providers, but once it does, Spegel will be ready.

Content Events

Containerd now supports content events, introduced in #11006. What this means for you as the end user is faster advertisement of layers. Previously, Spegel wouldn’t be notified about new layers until the entire image had been pulled. With large images, the delay between one layer being successfully pulled and the next could be several minutes. This meant that even if the layer was already present on the node, we wouldn’t advertise it. The issue became especially noticeable when multiple pods using the same image were started at the same time.

Large parts of the event logic have now been refactored to support both image and layer events. As a result, layers are advertised as soon as they are available. While this may not improve performance across the board, it can make a noticeable difference in specific use cases. Once the benchmarking tool is further developed, we’ll be able to measure more precisely what impact early layer advertisement has on overall Spegel performance.

Dial Timeout

Containerd now supports dial timeouts for mirrors, introduced in #11106. Previously, if the local Spegel instance failed to start for any reason, Containerd would wait indefinitely for the TCP connection to be established. If Spegel never came up, Containerd would never move on to another mirror or fall back to the upstream registry. This could effectively block the node from pulling any images, which is obviously not great. With the new dial timeout parameter, we can set a limit to avoid this situation. Spegel now includes this parameter when writing mirror configurations. Since the traffic is local, connection times are expected to be low. After some testing, one second was found to be more than enough to establish a TCP connection within the cluster. While real-world cases have been rare, this is still a valuable stability improvement.

Containerd Future Work

Containerd includes two new features that will impact Spegel in future releases. The first is OCI volume support, which allows containers to mount other OCI artifacts as volumes. These are pulled using the same mechanism as regular container images, so they should work seamlessly with Spegel. The documentation will be updated once the end-to-end tests have been extended to cover this.

The second is multipart fetching, which was merged after a year in development. This feature splits layer pulling into smaller chunks and downloads them in parallel, improving pull speed. Although currently opt-in, multipart fetching could bring significant performance gains when combined with Spegel. Exploring its impact will be an important next step.

Announcing Spegel v0.2.0

Tue, 29 Apr 2025 00:00:00 +0000

The focus of this release has been on cleaning up the codebase, increasing test coverage, and simplifying future extensions. A few minor, non-breaking bugs introduced in v0.1.0 have also been fixed. Beyond that, the two main improvements that matter to end users are: a new cleanup mechanism added to the Helm chart, and the introduction of prebuilt binary releases for Spegel.

Cleanup of Mirror Configuration

The lack of proper cleanup has been a major pain point for users testing or benchmarking Spegel. Previously, Spegel would write the containerd mirror configuration to the host, but would not remove it when the Helm chart was uninstalled. This left behind mirror settings pointing to a registry that no longer existed. The result varied, but typically added unwanted latency as containerd attempted to reach a mirror that would never respond. In clusters that regularly rotate nodes, this wasn’t an issue since new nodes wouldn’t have the stale configuration. Still, this was bad practice on Spegel’s part.

The challenge was finding a solution that works with Helm’s uninstall hooks in combination with daemonsets. Helm only natively supports pod and job resources if you want it to wait for completion during uninstall. Spegel needed a way to run cleanup logic on every node and ensure it completed before uninstalling.

One option was to create a job that would in turn launch cleanup jobs for each node and wait for them to complete. But that approach is overly complex and can leave behind resources in failure scenarios.

Instead, Spegel now takes advantage of a quirk in Helm: it allows a pod and a daemonset to be created as part of the uninstall hook. The daemonset removes the mirror configuration from each host and then starts an HTTP server to respond to health probes. The accompanying pod runs a simple prober that checks all the daemonset pods. Once all respond successfully, we assume cleanup is complete. This setup allows Helm to manage all uninstall-related resources and requires no additional Kubernetes expertise from users.

This cleanup feature is enabled by default in the latest version of the Helm chart.

Binary Releases

Some users have expressed interest in running Spegel directly on the host, outside of Kubernetes. To support this use case, the release process now includes standalone binaries. This required some refactoring of the build pipeline. Spegel is now built outside of Docker, and the Docker image build step simply copies the compiled binary into the image.

Host-level support is still experimental. The main challenge is finding the right solution for the bootstrap process. Currently, the best option is to use the static bootstrapper, though it has some limitations. It depends on specific peers being available. This can lead to issues if those peers are down.

Work will continue on improving the experience for running Spegel on the host, and documentation will follow once the current limitations are addressed.

Refactoring the Registry Proxy

There have been two major changes to the registry mirror implementation.

First, the use of the standard library HTTP proxy has been completely removed. It never fully aligned with Spegel’s mirroring logic and was unnecessarily bulky. Spegel does not need all the features of a general-purpose proxy. It handles a narrow and well-defined set of proxying behaviors. Since Spegel is aware of the request types and supported paths, it can make assumptions a generic proxy cannot. The proxy has been replaced with a lightweight HTTP client and some custom logic instead. This also prepares the codebase for future features like the referrers API, which will require an HTTP client anyway.

The second change is the removal of request source detection. Before v0.2.0, if Spegel received a request for content that had not yet been mirrored, it would inspect the client IP. If the request came from another Spegel instance, it could allow proxying the traffic to itself if selected during lookup. If the request was local, it would avoid routing to itself and only consider other nodes. This logic was introduced early in Spegel’s development and has become less useful over time. It is now better for Spegel to immediately return content if it exists locally. In addition, the logic for determining the client IP was unreliable and often failed depending on the CNI in use. Overall, the feature added little value and mostly caused confusion.

Announcing Spegel v0.1.0

Sat, 22 Mar 2025 00:00:00 +0000

Spegel’s journey began with its first commit in February 2023, sparked by a curiosity about OCI and Containerd, along with a challenge to explore alternative image distribution methods. In the two years since, it has evolved into a valuable tool for everyone from homelab enthusiasts to multinational corporations. Seeing the GitHub stars climb has been rewarding, a clear indicator that adoption is growing and people recognize its value.

Since Spegel’s initial release, breaking changes have been avoided, and for the most part, the project has evolved smoothly. However, the time has come to address past missteps. The v0.1.0 release marks an important milestone, enabling necessary changes while also committing to proper versioning and stability moving forward. It’s important to review the changelog to understand if these breaking changes might impact you. From here on, future releases will follow semantic versioning, with new features introduced through minor version increases. While this release could have come sooner, uncertainty in Spegel’s implementation made it challenging. But, as they say, better late than never.

Before diving into the new features and changes, a quick request. If your company is using Spegel, adding your organization to the adopters page would be extremely valuable. One of the biggest challenges in open source is understanding who the end users are. If you find value in Spegel and want to support its continued development, consider sponsoring the project. Your support helps ensure the future of Spegel and its ongoing improvements.

We’re building a stronger community of Spegel users, and we’d love for you to be part of it. A Slack channel has been created within the Kubernetes Slack organization, and we hold community meetings every Tuesday at 17:00 CET. To learn how to join us, check out the community documentation.

Mirror all registries

One of the most significant breaking changes in v0.1.0 is that Spegel will now configure all registries to be mirrored by default. This shift has been a long time coming and is the primary driver behind this release. Initially, only a limited set of default registries were configured to minimize potential negative impacts if something went wrong. Over the past two years, however, confidence in Spegel has grown, and most users now prefer to mirror all images from all registries instead of just a select few.

Along with this change, several configuration values have been renamed for clarity. The setting spegel.appendMirrors is now spegel.prependMirrors to more accurately describe its function. Similarly, spegel.registries has been renamed to spegel.mirroredRegistries, and spegel.additionalMirrorRegistries has been renamed to spegel.additionalMirrorTargets. These changes aim to reduce confusion and more clearly distinguish between the registries to be mirrored and the target mirrors where requests will be sent.

Debug Web UI

A challenge with Spegel has been gaining insights into its operation. New users often struggle to determine whether Spegel is working correctly. This is a significant issue. Until now, users have had to rely on container logs and HTTP status codes to figure out if an image was found and pulled successfully. To simplify this process, a new debug web UI has been added. It is disabled by default but can be easily enabled by setting spegel.debugWebEnabled: true. To access it, simply port forward to a Spegel pod on its metrics port and browse to http://localhost:9090/debug/web.

While the features are still limited, you can already initiate an image pull from other Spegel nodes to check that images are found and pulled correctly. We welcome any feedback. If there is additional information that would help with troubleshooting, please consider creating an issue.

Memory requests and limits

The Helm chart now includes default memory requests and limits to address feedback about excessive memory usage by Spegel. Linux file reading relies on the VFS cache, which keeps file chunks in memory for efficiency. In a containerized environment, VFS cache usage contributes to the container’s overall memory consumption. Since the VFS cache is only cleared under memory pressure, Spegel may appear to consume increasing amounts of memory until it reaches the available memory on the node. While some users have manually set memory limits to control this, the majority likely have not. To prevent confusion about excessive memory usage, the Helm chart now sets a default memory limit of 128Mi, ensuring the cache is cleared when necessary. This change helps support the majority of users and avoids future issues.

New bootstrapper

This feature was originally part of v0.0.29 but was intended for the v0.1.0 release. Due to the delay of v0.1.0, it was made available earlier. The new bootstrapper now uses a headless DNS service to find peers within the cluster. This change addresses the fact that Kubernetes leader election was never designed to be used by a DaemonSet in large Kubernetes clusters. Users running clusters with 500+ nodes were facing performance issues, as all nodes were communicating with the API server. The old Kubernetes bootstrap method has now been fully removed, eliminating any Go Kubernetes dependencies. This new method is more lightweight and should significantly improve startup times.

Future

There’s a lot of exciting work ahead to make Spegel faster and more efficient. For a glimpse of the major features being prioritized, check out the roadmap. Looking ahead, a v1.0.0 release of Spegel is in the plans, though it’s still some time away. This release will likely align with the wider adoption of Containerd v2, which would allow Spegel to drop support for Containerd v1.7 and earlier versions. The timing will depend on how quickly cloud providers and end users adopt Containerd v2.

Spegel development will continue to grow and evolve, with exciting new features ahead. Thanks to your support, we’re building something truly special. Here’s to another year of keeping things fast and local, without the need to pull from upstream registries!