Architecture
Spegel can run as a stateless application by exploiting the fact that an image pulled by a node is not immediately garbage collected. Spegel is deployed as a Daemonset on each node which acts as both the registry and mirror. Each instance is reachable both locally through a host port and a Service. This enables Containerd to be configured to use the localhost interface as a registry mirror and for Spegel instances to forward requests to each other.
Images are composed of multiple layers which are stored as individual files on the node disk. Each layer has a digest which is its identifier. Every node advertises the digests which are stored locally on disk. Kademlia is used to enable a distributed advertisement and lookup of digests. An image pull consists of multiple HTTP requests with one request per digest. The request is first sent to Spegel when an image is pulled if it is configured to act as the mirror for the registry. Spegel will lookup the digest within the cluster to see if any node has advertised that they have it. If a node is found the request will be forwarded to that Spegel instance which will serve the file with the specified digest. If a node is not found a 404 response will be returned and Containerd will fallback to using the actual remote registry.
In its core Spegel is a pull only OCI registry which runs locally on every Node in the Kubernetes cluster. Containerd is configured to use the local registry as a mirror, which would serve the image from within the cluster or from the source registry.
Flow Diagrams
The flow diagrams provide a comprehensive set of diagrams explaining Spegel’s architecture, flows, and operations.
High-Level Cluster Architecture
Shows how Spegel pods form a P2P network within the cluster. Containerd interacts with Spegel for image pulls and handles fallback to external registry when needed.
graph TB
subgraph "External"
ER["External Registry"]
end
subgraph "Kubernetes Cluster"
subgraph "Node 1"
SP1["Spegel Pod"]
CD1["Containerd"]
SP1 <-->|interacts| CD1
CD1 -->|fallback| ER
end
subgraph "Node 2"
SP2["Spegel Pod"]
CD2["Containerd"]
SP2 <-->|interacts| CD2
CD2 -->|fallback| ER
end
subgraph "Node 3"
SP3["Spegel Pod"]
CD3["Containerd"]
SP3 <-->|interacts| CD3
CD3 -->|fallback| ER
end
SP1 <-->|P2P Network| SP2
SP2 <-->|P2P Network| SP3
SP3 <-->|P2P Network| SP1
end
Pod Component Architecture
Details the internal components of a Spegel pod and their relationships, showing how the registry service, P2P components, and state management interact with each other and with containerd.
graph TB
subgraph "Spegel Pod"
subgraph "Registry Service"
RS[HTTP Server /v2/]
RH[Request Handler]
RS --> RH
end
subgraph "P2P Components"
P2P[P2P Router]
DHT[DHT Provider]
BS[Bootstrapper]
P2P --> DHT
BS --> P2P
end
subgraph "State Management"
ST[State Tracker]
MT[Metrics]
ST --> MT
end
CD[Containerd Client]
RH --> P2P
ST --> P2P
CD --> ST
end
subgraph "Node Components"
CDD[Containerd Daemon]
CS[Content Store]
CDD --> CS
end
CD --> CDD
Image Pull Flow
Shows the sequence of operations during an image pull request, demonstrating both successful peer pulls and fallback to external registry.
sequenceDiagram
participant CD as Containerd
participant SR as Spegel Registry
participant P2P as P2P Router
participant PR as Peer Registry
participant ER as External Registry
Note over SR,P2P: 20ms default resolve timeout
Note over SR,P2P: 3 default resolve retries
CD->>SR: GET /v2/{name}/manifests/{ref}
SR->>P2P: Resolve(key, allowSelf, retries)
alt Peer Found
P2P-->>SR: Return Peer Address
SR->>PR: Request Content
PR-->>SR: Stream Content
SR-->>CD: Return Content
CD->>CS: Store Content
else No Peers Available (within 20ms)
SR-->>CD: 404 Not Found
CD->>ER: Request from External
ER-->>CD: Return Content
CD->>CS: Store Content
end
P2P Network Formation
Shows how nodes discover each other and form the P2P network through leader election and peer sharing.
sequenceDiagram
participant N1 as Node 1
participant N2 as Node 2
participant N3 as Node 3
participant LE as Leader Election
Note over N1,LE: 10s lease duration
Note over N1,LE: 5s renew deadline
Note over N1,LE: 2s retry period
N1->>LE: Participate in Election
N2->>LE: Participate in Election
N3->>LE: Participate in Election
LE->>N1: Elected Leader
N2->>N1: Discover Leader
N3->>N1: Discover Leader
N1->>N2: Share Peer List
N1->>N3: Share Peer List
N2->>N3: Establish P2P Connection
Note over N1,N3: P2P Network Formed
State Management and Content Advertisement
Shows how content availability is maintained and advertised in the P2P network, including periodic refresh cycles and event-driven updates.
sequenceDiagram
participant ST as State Tracker
participant CD as Containerd
participant P2P as P2P Router
participant DHT as DHT Network
participant MT as Metrics
Note over ST,DHT: Content TTL: 10 minutes
Note over ST,DHT: Refresh: Every 9 minutes
loop Every 9 minutes
ST->>CD: List Images
CD-->>ST: Image List
loop For each image
ST->>P2P: Advertise(image_keys)
P2P->>DHT: Provide(keys)
end
ST->>MT: Update Metrics
end
CD-->>ST: Image Event (Create/Update/Delete)
ST->>P2P: Update Advertisement
ST->>MT: Update Metrics
Content Resolution Process
Shows how content is located and retrieved from peers in the network, including peer selection and retry mechanisms.
sequenceDiagram
participant SR as Spegel Registry
participant P2P as P2P Router
participant DHT as DHT Network
participant PR1 as Peer 1
participant PR2 as Peer 2
SR->>P2P: Resolve(content_key)
P2P->>DHT: FindProviders(key)
par Parallel Resolution
DHT-->>P2P: Found Peer 1
DHT-->>P2P: Found Peer 2
end
P2P->>SR: Return First Available Peer
Note over SR,PR2: Default 20ms timeout
Note over SR,PR2: 3 retry attempts
alt Try Peer 1
SR->>PR1: Request Content
PR1-->>SR: Stream Content
else Peer 1 Fails
SR->>PR2: Request Content
PR2-->>SR: Stream Content
end
Data Flow Paths
Shows the content paths and system control flows, including peer transfers and fallback mechanisms.
graph LR
subgraph "Content Paths"
CD[Containerd]
SP[Spegel]
P[Peers]
ER[External Registry]
CS[Content Store]
CD -->|Request| SP
SP -->|Check| P
P -->|Content| SP
SP -->|Return| CD
CD -->|Store| CS
SP -->|404| CD
CD -->|Fallback| ER
end
subgraph "P2P Operations"
P2P[P2P Network]
DHT[DHT]
ST[State Tracker]
P2P -->|Advertise| DHT
DHT -->|Discover| P2P
ST -->|Update| P2P
end
Failure Handling
Shows how different types of failures are handled in the system.
sequenceDiagram
participant CD as Containerd
participant SR as Spegel Registry
participant P2P as P2P Router
participant PR as Peer
participant ER as External Registry
Note over SR,ER: Failure Scenarios
alt Peer Not Found
CD->>SR: Request Content
SR->>P2P: Resolve(key)
P2P--xSR: No Peers Available
SR-->>CD: 404 Not Found
CD->>ER: Fallback Request
end
alt Peer Connection Failed
SR->>PR: Request Content
PR--xSR: Connection Failed
SR->>P2P: Resolve(key) Retry
P2P-->>SR: Alternative Peer
end
alt Content Corrupted
SR->>PR: Request Content
PR-->>SR: Stream Content
SR--xCD: Verification Failed
CD->>ER: Fallback Request
end
Metrics Collection
Shows how metrics are collected and organized across the system components.
graph TB
subgraph "Metrics Sources"
RQ[Registry Requests]
P2P[P2P Operations]
ST[State Changes]
end
subgraph "Metric Types"
CT[Counters]
HT[Histograms]
GT[Gauges]
end
subgraph "Prometheus Metrics"
MR[mirror_requests_total]
RD[resolve_duration_seconds]
AI[advertised_images]
AK[advertised_keys]
RL[request_latency]
IF[requests_inflight]
end
RQ --> CT
RQ --> HT
P2P --> HT
P2P --> GT
ST --> GT
CT --> MR
HT --> RD
HT --> RL
GT --> AI
GT --> AK
GT --> IF