SILO vs. MinIO Behavior Changes
Standard S3 applications usually switch from MinIO to SILO without code changes, and modern xl / xl-single data volumes mount in place without rewriting object data. However, migrating from upstream MinIO to SILO requires a coordinated full-cluster stop (mixed-version rolling replacement across the product boundary is unsupported), and security and consistency repairs deliberately tighten deployment identity, IAM policy evaluation, request signing, Object Lock, multi-pool mutations, and background jobs.
This guide covers every user-perceivable behavior change from the upstream fork baseline minio/minio@27742d469462 (2025-12-03, after the final upstream tag RELEASE.2025-10-15T17-29-55Z) through SILO Server 20261112 (RELEASE.2026-11-12T00-00-00Z, source 44e848fedf8c, embedding Console v2.7.2 0e7953ff59c4, paired with mcli 20261112 a37a9f054fba, silo-pkg v3.16.1, Helm chart 7.0.5, and Go 1.27.4). For the 24-category executive index, see the Compatibility Overview; for step-by-step cutover commands, see the Migration Guide.
See whether existing MinIO data drives mount directly, which topologies require a full stop, and what rollback requires.
Filter the exact upgrade gates for Upstream MinIO, SILO 202608–09, SILO 20260916–1026, or SILO 20261109.
Inspect concrete changes across deployment, IAM, S3 authz, encryption, multipart, replication, pools, ILM, batch, mcli, and Console.
Run pre-cutover drains, offline backups, policy audits, per-node S3 data-path checks, and consistent rollback preparation.
Can an Existing MinIO Cluster Run Directly on SILO?
When evaluating an in-place takeover, separate three distinct questions: whether SILO reads existing disks, whether nodes can be swapped one by one while serving traffic, and whether existing policies, integrations, and background jobs keep identical semantics afterward:
| Scenario or Boundary | Verdict | Behavior on SILO 20261112 | Operational Requirement |
|---|---|---|---|
Modern single-drive (xl-single) and distributed (xl) data volumes |
Direct Mount | .minio.sys/format.json and XL metadata headers remain compatible; existing objects, versions, multipart uploads, encryption headers, and stored IAM records load in place without rewriting object data |
Inspect .minio.sys/format.json read-only (xl or xl-single) and confirm healthy read quorum across all erasure sets before stopping old processes |
Legacy fs or gateway directories |
Unsupported | Removed upstream in 2022-10-29; SILO rejects legacy fs directories at startup (note: modern single-directory server /data uses xl-single, which mounts directly) |
Migrate data via object export/mirror following the FS/Gateway migration guide |
| Mixed-binary rolling cluster (MinIO + SILO nodes running together) | Blocked | Distributed bootstrap verifies peer binary checksums, private internode StorageREST removed /rmpl (ReadMultiple), and peer IAM/replication protocols advanced |
Cut over from upstream MinIO to SILO as a coordinated full-cluster stop, never node-by-node mixed binaries |
| Single pool whose erasure sets span different host subsets (typically ≥ 18 hosts in automatic layout) | Full Stop | Pre-20261109 processes routed single-pool locks to erasure set 0 instead of the target set; mixed old/new processes use disjoint lock domains and lose mutual exclusion |
Drain traffic and stop every old process before starting 20261109 or later |
| Multi-pool cluster rolling between SILO releases | Drain Writes | Cross-pool conditional writes, lock ordering, bulk version purges, and suspended-bucket null-version lock checks require all nodes on 20261112 for uniform guarantees |
Drain or pause concurrent S3 writes across mixed-version multi-pool rolling windows |
| Multi-site replication and shared IAM cutover | Coordinated | Durable IAM revocation tombstones (20260916+) and tri-state service-account inline policies (20261112) can be overwritten if an unupgraded peer site edits accounts and resyncs |
Avoid editing service accounts or IAM policies on old sites during mixed-site rollouts; reconcile policies after all sites upgrade |
Unfinished Batch jobs (expire, replicate, keyrotate) |
Drain First | 20261109+ writes submitter-bound SILO-BATCH v2 manifests and reports; legacy running/queued/paused jobs are not guaranteed to resume across the boundary, and ≤ 20261026 rejects v2 records |
Stop new batch submissions, let active jobs finish or cancel them before upgrading, and resubmit once all nodes run 20261112 |
Native S3 archive restore (POST ?restore) |
All Nodes | Two-phase lock-free restore (20261109+) requires every peer in the cluster to advertise the capability and be reachable; otherwise POST ?restore returns 503 |
Upgrade all nodes and confirm peer connectivity before running archive restores |
| Custom IAM policies, OIDC/LDAP, TLS/KMS, and notifications | Audit Config | Object-only ARNs no longer grant 12 bucket-level writes, version deletes require s3:DeleteObjectVersion, self-service password change checks admin:ChangeMyPassword, HMAC OIDC is rejected, and legacy DB notify requires a DSN |
Audit representative policies and test login, KMS decryption, replication, and event targets before reopening client traffic |
| Swapping back to an older binary for rollback | Backup Required | Readable objects do not prove safe downgrade: startup, scanner passes, IAM revocations, Batch v2 manifests, and bucket tombstones update .minio.sys state that older binaries reject or misinterpret |
Capture an offline consistent backup of .minio.sys, IAM exports, and deployment manifests before cutover; restore all peers together if rolling back |
Quick Check by Starting Version
Select your current running version to focus on the upgrade gates that apply directly to you; refer to the 12 domain tables below for full details.
Migrating directly from upstream MinIO (such as the final tag RELEASE.2025-10-15T17-29-55Z) crosses every fork-era delivery and security boundary in one cutover:
- Cutover mode and disk verification: Confirm
.minio.sys/format.jsonisxlorxl-single, capture an offline backup, and execute a coordinated full-cluster stop (do not mix MinIO and SILO nodes). See the documented single-node migration case. - Deployment and artifact identity (Section 1): Switch images to
docker.io/pgsty/silo:RELEASE.2026-11-12T00-00-00Z, server binary/systemd unit tosilo, and client package tomcli. Replace hard-coded/usr/bin/miniopaths withsilo. Preserve Helm resource names and explicitly pinimage.tag. Removeadmin update,mcli update, and SUBNET automation. - IAM policies and external authentication (Section 2, Section 3):
- Statements containing only
arn:aws:s3:::bucket/*no longer grant 12 bucket-level write actions (PutBucketPolicy,PutBucketLifecycle,DeleteBucket, etc.); add barearn:aws:s3:::bucketto the same statement. - Permanent deletes carrying
versionIdrequires3:DeleteObjectVersion;Allow s3:*+Deny s3:DeleteObjectpolicies must also addDeny s3:DeleteObjectVersion. - Self-service password updates check
admin:ChangeMyPassword(the built-inreadonlypolicy no longer permits self-service password changes). Policy writes enforce strict JSON decoding and reject bare ARN namespace prefixes. - OIDC rejects symmetric
client_secretHMAC (HS256/384/512) tokens and RSA JWKs above 4096 bits. ConfigureMINIO_API_TRUSTED_PROXIESwhen trustingX-Forwarded-For/X-Real-IP/Forwarded.
- Statements containing only
- Integrations and background jobs (Sections 8–10): Convert legacy PostgreSQL / MySQL notification settings to
connection_string/dsn_string; review per-bucket CORS rules (now actively enforced); drain unfinished Batch jobs before stopping; verify signed S3 reads/writes against every node after startup.
You have already crossed the silo / mcli rename and initial fork security gates (bare bucket ARNs, DeleteObjectVersion, OIDC asymmetric JWKS). Upgrading to 20261112 focuses on changes introduced since 20260916:
- IAM and password authorization (Section 2):
20260916splitadmin:ChangeMyPasswordand introduced durable IAM revocation tombstones (follow the IAM upgrade guide across peer sites);20261018hardened delegated Admin escalation and root-derived credentials;20261112enforces strict policy JSON decoding and preserves service-account inline policies when omitted ({}resets to inheritance). - S3 tag headers and archive-member authorization (Section 3): Local and active-active proxied
GET/HEADresponses omitx-amz-tagging-count,X-Amz-Tagging, andX-Amz-Object-Lock-*unless the caller holds the corresponding tag or lock read permission; ZIP member reads authorize the outer ZIP’s stored tags without persisting an on-disk index. - Topology stop gates and background jobs (Sections 6–10): Single-pool clusters spanning distinct host subsets across erasure sets (typically ≥ 18 hosts) require a full stop to align lock domains; drain or cancel unfinished Batch jobs before upgrading (
SILO-BATCHv2 manifests);POST ?restorereturns503while any peer is unupgraded; drain concurrent S3 writes across multi-pool rolling windows. - HTTP, TLS, and Console (Sections 11–12): HTTP/1 request headers enforce an absolute read deadline; Go
1.27.4and-distrolessUBI CA roots apply; embedded Console advances tov2.7.2(cursor pagination, 10 UI languages, folder usage scans, tri-state inline-policy editor).
When upgrading from the September or October releases to 20261112, check these operational gates:
- Deployments that enabled optional
20260916strictmultipart listing (Section 5): Retired in20261018+(ignored with a startup warning);ListMultipartUploadsuses the default cache/exact-key path without durable enumeration across restarts. - Single-pool lock domains, Batch v2, and native restore (
20261109gates):- Drain and stop every old process before starting
20261112if a single pool’s erasure sets do not all share the exact same host set (normally ≥ 18 hosts); - Drain or cancel all unfinished Batch jobs before upgrading and resubmit afterward (
≤ 20261026rejectsSILO-BATCHv2 manifests); - Native
POST ?restorereturns503until every peer runs20261109+and is reachable.
- Drain and stop every old process before starting
20261112authorization, lock, and multi-pool repairs:- Service-account edits preserve existing inline policies when omitted and require explicit
{}(or turning off Restrict beyond user policy) to restore inheritance — avoid editing service accounts on old peer sites during rollout; - Strict policy JSON decoding rejects unknown fields and trailing data across IAM, STS, bucket policies, Console, and
mcli; - Every explicit-
VersionIdentry inDeleteObjectsruns sequentially through the single-object path across all buckets, increasing latency for large version purges; - Locked
nullversions in versioning-suspended or prefix-excluded buckets are protected against ordinary overwrites and shadowed deletes; compressed+encryptedRangereads and cross-pool historical self-copies are repaired.
- Service-account edits preserve existing inline policies when omitted and require explicit
20261109 already includes single-pool lock-domain alignment, SILO-BATCH v2 manifests, and two-phase native restore. Upgrading to 20261112 focuses on these targeted deltas (see 20261112 upgrade notes):
- Service-account inline-policy preservation and reset (SN-2026-025): Updating a service account or access key with an omitted/
nullpolicy preserves its existing inline policy (including pre-20261111empty deny-all inline policies); pass{}or{"Statement":[]}explicitly to clear the inline policy and restore inheritance. Do not edit service accounts on20261109peer sites during a mixed-site rollout. - Strict policy JSON and SSE-KMS context validation: Server, Console
v2.7.2, andmcli 20261112reject unknown JSON fields, trailing payloads, and emptyPrincipal: {}in policy writes; invalidX-Amz-Server-Side-Encryption-Contextheaders return400 InvalidArgumentinstead of500, whilekeyrotatepersists KMS context and stream-rewrites legacyXLV1objects ontoXLV2. - S3 tag headers, ZIP member authz, and signing bounds (SN-2026-022 / -023 / -024): Local and active-active proxied
GET/HEADresponses omitx-amz-tagging-count,X-Amz-Tagging, andX-Amz-Object-Lock-*when tag or lock read permission is absent; tag-authorized reads preserve5xxstorage errors on read-quorum failure instead of returning false403(#294); ZIP member reads verify outer ZIP tags in memory without mutating the object; authenticatedSTREAMING-UNSIGNED-PAYLOAD-TRAILERenforces the 15-minute skew window, and SigV4 rejects mismatched scope dates and > 64 KiB trailers. - Object Lock, bulk version deletes, and pools: Versioning-suspended buckets protect locked
nullversions against overwrites and delete markers;DeleteObjectsruns sequentially on all explicit-VersionIdentries across all buckets; pause concurrent S3 writes across multi-pool rolling windows; fixes cover compressed+encryptedRangereads (decryptSkip > 0), cross-pool historical self-copy (#343), pre-2024-04-28xl.metaandUsesDataDirrevision normalization (#332,#344), and decommission nested child-key rescans plus within-quorum offline-drive emptiness checks (#334). - Helm upgrade: Pass
--set image.tag=RELEASE.2026-11-12T00-00-00Zexplicitly when upgrading Helm chart7.0.5with--reuse-values.
1. Deployment, Artifacts, Identity & Outbound Services
| Scenario | Since | Previous Behavior → SILO Behavior | Impact & Action |
|---|---|---|---|
| Packages, executables, and systemd unit | 20260806+ |
minio / minio.service / /usr/local/bin/minio → Server package, binary, and unit use silo (silo.service); client package and binary use mcli; no /usr/bin/minio alias is installed |
Update startup scripts, health probes, and automation; see Native Package Migration |
| OS service user and environment file precedence | 20260806+ |
Default account minio-user → Fresh package installs create silo:silo; /etc/default/silo takes precedence when present, falling back to /etc/default/minio |
Keep existing data ownership or grant silo:silo access to drives, certs, and logs; avoid leaving conflicting settings in both env files |
| Local configuration and TLS certificate fallback | 20260806+ |
Fixed ~/.minio → Fresh processes default to ~/.silo, falling back to ~/.minio when ~/.silo is absent, and still checking ~/.minio/certs when ~/.silo/certs has no certs |
Pass --config-dir or --certs-dir explicitly in service units so an accidental empty ~/.silo does not shadow legacy config |
Container image, entrypoint, and -distroless variant |
20260806+ / 20261109+ |
minio/minio with /usr/bin/minio → docker.io/pgsty/silo (classic UBI and -distroless variants); entrypoint translates argv minio server ..., but only /usr/bin/silo exists on disk |
Replace hard-coded /usr/bin/minio commands or probes with silo; -distroless omits a shell and mcli, so use ["CMD", "silo", "healthcheck", "ready"] |
Helm chart upgrade and resource naming (7.0.5) |
20260806+ / 20261112 |
Upstream minio chart names and security defaults → SILO chart defaults to silo resource names and non-root security context, and --reuse-values retains the old image.tag |
Pin nameOverride, fullnameOverride, serviceAccount.name, and securityContext when adopting existing PVCs, and pass --set image.tag=RELEASE.2026-11-12T00-00-00Z |
| In-place self-update, SUBNET, telemetry, and Windows ZIP | 20260804+ / 20261026+ |
mc admin update, mc update, SUBNET uploads, and seeded play alias → Online self-update, callhome, telemetry, and SUBNET are disabled with explicit errors; Windows archives use .zip |
Remove scheduled update and SUBNET jobs; roll out upgrades through packages, container images, or your orchestrator |
| TLS defaults and CA trust bundle | 20260903+ / 20261109+ |
Older Go TLS defaults and broader roots → Uses Go 1.27.4 TLS defaults, and -distroless matches the UBI ca-certificates bundle (removing retired GlobalSign R2 and distrusted roots) |
Add custom/internal CA certificates under certs/CAs/ and verify handshakes against KMS, OIDC, LDAP, etcd, webhooks, and replication peers |
| Go module embedding and shared library path | 20260903+ |
Imported github.com/minio/pkg/v3 → Server module remains github.com/minio/minio, while maintained source directly requires github.com/pgsty/silo-pkg/v3 (v3.16.1) |
Affects only Go code that embeds internal server/shared packages directly; standard S3 SDK users are unaffected |
2. Authentication, IAM, Policies & Proxy Trust
| Scenario | Since | Previous Behavior → SILO Behavior | Impact & Action |
|---|---|---|---|
| 12 protected bucket-level writes require bare bucket ARN (SN-2026-004) | 20260804+ |
Object-only arn:aws:s3:::bucket/* granted bucket policy, lifecycle, replication, versioning, and delete-bucket writes → PutBucketPolicy, DeleteBucketPolicy, PutBucketObjectLockConfiguration, PutBucketVersioning, PutReplicationConfiguration, PutBucketLifecycle, DeleteBucket, ForceDeleteBucket, PutBucketCors, DeleteBucketCors, PutBucketQOS, and PutInventoryConfiguration require bare arn:aws:s3:::bucket in the same Allow statement |
Add "arn:aws:s3:::bucket" alongside "arn:aws:s3:::bucket/*" in provisioning policies (temporary compatibility escape hatch: MINIO_API_LEGACY_BUCKET_RESOURCE_MATCH=on) |
User and group status changes split Enable* vs. Disable* (SN-2026-009) |
20260903+ |
SetUserStatus and UpdateGroupStatus always checked Enable* → Enabling checks admin:EnableUser / admin:EnableGroup; disabling checks admin:DisableUser / admin:DisableGroup |
Grant admin:DisableUser and admin:DisableGroup to operator roles that suspend users or groups |
| Self-service password change split from user creation | 20260916+ |
admin:CreateUser covered both creating users and changing one’s own password, and built-in readonly allowed self-service password change → Self-service password change checks admin:ChangeMyPassword, and built-in readonly no longer grants it |
Attach admin:ChangeMyPassword explicitly if read-only users must rotate their own passwords; see Password Permissions |
| Tri-state service-account and access-key inline policies (SN-2026-025) | 20261112 |
Metadata-only edits omitting a policy could clear an inline restriction and widen the key back to full parent inheritance → Omitted/null policy preserves the existing inline policy (including pre-20261111 empty deny-all inline policies); explicit {} or {"Statement":[]} removes it and restores inheritance; nonempty policies missing Version receive the default |
Upgrade Console v2.7.2 and mcli 20261112; do not edit service accounts on unupgraded peer sites during rollout, and review effective policies afterward |
| Strict policy JSON decoding and bare ARN rejection (SN-2026-005) | 20260804+ / 20261112 |
Tolerated unknown JSON fields, trailing tokens, and bare arn:aws:s3::: prefixes → IAM named policies, service-account session policies, STS Policy, bucket policies, imports, and replication reject unknown fields, trailing payloads, empty Principal: {}, NotPrincipal, and bare ARN prefixes; stored IAM policies still load |
Fix typos in hand-written or templated policy JSON (such as Conditions or Principals) and validate policy bundles before importing |
| Built-in policy override Reset | 20261026+ |
Deleting an overridden built-in policy (such as readonly or readwrite) could remove the record without restoring the native definition → Reset removes the override and immediately restores the native built-in policy while retaining in-use checks |
Overridden built-in policies can now be safely reset to factory defaults |
Policy merge Deny preservation and fail-closed partial load |
20261018+ / 20261109+ |
Multi-policy deduplication or partial policy read failures could drop an explicit Deny → Policy merges preserve every explicit Deny, and unreadable attached policy records fail closed for affected requests |
If previously permitted calls return AccessDenied, check attached policies for an explicit Deny that was previously masked |
| Delegated Admin containment and root-derived protection (SN-2026-017) | 20261018+ |
Session-restricted or self-denied admin credentials could mint broader child keys or touch root-derived accounts → Evaluates the actor’s live effective permissions and self-denies, and blocks non-root admins from reading, mutating, or deleting root or root-derived credentials | Ensure IAM automation credentials hold full effective grants for policies they assign; manage root service accounts only as root |
| Durable IAM revocation tombstones and scoped STS invalidation (SN-2026-013) | 20260916+ / 20261018+ |
Deleted users/keys lacked durable tombstones across peer sites, while disabling one user could flush unrelated STS sessions → Persists and replicates IAM revocation records across peers, and invalidates only credentials derived from the affected principal | Follow the IAM Upgrade Guide on multi-site deployments so old peers do not resurrect deleted accounts |
| IAM ZIP import status preservation and non-transactional execution (SN-2026-025) | 20261109+ / 20261112 |
Imports could re-enable disabled accounts or swallow group errors → Validates the entire policy file up front, preserves disabled user/service-account/group status, and reports group errors; the overall ZIP import remains non-transactional |
Export a backup (mcli admin cluster iam export) before importing and inspect the exit status and item report |
| OIDC asymmetric JWKS requirement and 4096-bit RSA cap (CVE-2026-33322) | 20260321+ / 20261018+ |
Accepted client_secret symmetric HMAC (HS256/384/512) ID tokens and unbounded RSA JWK sizes → Requires asymmetric JWKS verification only and rejects RSA keys larger than 4096 bits |
Configure your IdP to sign tokens with RS256/ES256 (RSA ≤ 4096 bits) and test OIDC login before cutover |
| LDAP STS uniform errors, rate limiting, and long-DN login (CVE-2026-33419) | 20260417+ / 20261026+ |
Distinguished unknown users from bad passwords and failed group login when normalized DNs exceeded 255 bytes → Auth failures return uniform InvalidParameterValue with per-source rate limiting (429 ThrottlingException, Retry-After: 6), and long-DN group logins continue past missing optional local records |
Honor Retry-After on 429 responses and configure MINIO_IDENTITY_LDAP_STS_TRUSTED_PROXIES behind reverse proxies so clients do not share one rate-limit bucket |
| Source-IP proxy trust boundary (SN-2026-018) | 20260804+ |
Trusted client-supplied X-Forwarded-For / X-Real-IP / Forwarded unconditionally for aws:SourceIp → Adds MINIO_API_TRUSTED_PROXIES (none or trusted proxy/peer CIDRs); unset retains legacy trust-any behavior for compatibility |
Set MINIO_API_TRUSTED_PROXIES=none on direct deployments, or list exact reverse-proxy and cluster-peer CIDRs |
| External authorization plugin bounds and boolean check | 20261109+ |
Unbounded external Access Plugin response size/latency → Caps plugin responses at 1 MiB and 1 minute and permits access only on an explicit boolean allow result |
Keep custom authorization webhook responses small, fast, and strictly boolean |
3. Request Signing & S3 Object Authorization
| Scenario | Since | Previous Behavior → SILO Behavior | Impact & Action |
|---|---|---|---|
SigV4 X-Amz-* SignedHeaders coverage and scope-date check (SN-2026-011) |
20260916+ / 20261112 |
Permitted unsigned X-Amz-* headers or Credential scope dates differing from the signed request timestamp → Every sent X-Amz-* header (and X-Minio-Replication-* header) must appear in SignedHeaders, and the credential scope date must match the UTC request timestamp |
Standard AWS SDKs and mcli already comply; fix custom SigV4 signers or reverse proxies that inject X-Amz-* headers after signing |
| Header-only presigned upload SHA-256 verification (SN-2026-012) | 20260916+ |
Presigned uploads carrying a concrete X-Amz-Content-Sha256 digest could skip full body verification → Enforces exact SHA-256 verification against the uploaded payload |
Sign arbitrary browser uploads with UNSIGNED-PAYLOAD, or supply the exact payload SHA-256 |
STREAMING-UNSIGNED-PAYLOAD-TRAILER presigned rejection and skew window (CVE-2026-41145, SN-2026-023) |
20260804+ / 20261112 |
Accepted query-presigned unsigned-trailer uploads and skipped skew/unsigned-header checks on header-authenticated requests → Query-presigned mode is rejected (SignatureVersionNotSupported); authenticated requests enforce the 15-minute skew window and checkUnsignedHeaders (bucket-policy anonymous uploads remain supported) |
Use regular PUT or signed aws-chunked uploads for presigned URLs, and keep client clocks synchronized via NTP |
| Snowball archive extraction pre-authentication (CVE-2026-40344) | 20260804+ |
PutObjectExtract could begin reading tar entries before completing authz → Authenticates and authorizes before untar(), rejecting unauthorized callers with 403 |
Affects unauthorized requests only; authorized Snowball batch uploads work unchanged |
| Unshadowed policy condition sources (SN-2026-003) | 20260804+ |
Raw headers or query params could shadow s3:VersionId, s3:ExistingObjectTag/<key>, s3:RequestObjectTag/<key>, or s3:signatureAge → Evaluates conditions strictly from server-verified version IDs, stored tags, and signature timestamps |
Prevents condition-spoofing bypasses; update test harnesses if they simulated conditions via forged headers |
Explicit version deletion requires s3:DeleteObjectVersion (SN-2026-010) |
20260903+ |
DeleteObject?versionId=... and DeleteObjects entries with VersionId succeeded with s3:DeleteObject alone → Explicit version removals require s3:DeleteObjectVersion (evaluated per entry in bulk delete) |
Grant s3:DeleteObjectVersion to version-pruning roles; add Deny s3:DeleteObjectVersion alongside Deny s3:DeleteObject in immutable-history policies |
GET/HEAD (local and active-active proxy) tag and lock header filtering (SN-2026-022) |
20261112 |
s3:GetObject alone exposed x-amz-tagging-count, X-Amz-Tagging (ACCESS), or peer X-Amz-Object-Lock-* headers → Omits tag headers unless s3:GetObjectTagging / s3:GetObjectVersionTagging (with no explicit Deny) allows, and omits peer lock headers unless s3:GetObjectRetention / s3:GetObjectLegalHold allows |
Object body and standard metadata reads are unchanged; grant tag or lock read actions if clients inspect those response headers |
Version-addressed reads, tagging, and GetObjectAttributes (SN-2026-024) |
20261112 |
Versioned reads/tagging could miss explicit version-action denies or actual version tags, and GetObjectAttributes missed dual action or zero-byte SSE-C checks → Explicit s3:GetObjectVersion* denies take precedence on GET/tagging (retaining legacy base-action grants when no version Deny matches); GetObjectAttributes requires both Attributes and GetObject (s3:GetObjectVersionAttributes + s3:GetObjectVersion when versionId is set) against actual version tags and SSE-C keys |
Grant both s3:GetObjectVersionAttributes and s3:GetObjectVersion to callers using GetObjectAttributes?versionId=... |
Preserving 5xx on tag-authorized read-quorum failures (#294) |
20261112 |
When s3:ExistingObjectTag/* policies were active, drive read-quorum failures could evaluate against empty tags and return a false 403 AccessDenied → Non-ObjectNotFound storage errors propagate as 5xx (such as 503 SlowDown) |
SDK retry loops now back off properly on transient quorum loss instead of aborting on false 403 |
ListObjectsV2 / ListObjectVersions (metadata=true) and tagging proxies |
20261112 |
Inconsistent per-version or peer-replica tag authorization → Ordinary metadata listing checks current-object permissions; ListObjectVersions(metadata=true) checks each version; active-active tagging proxies (GET/PUT/DELETE) authorize every present replica and fail closed on unreadable targets |
Keep IAM policies synchronized across active-active replication sites |
CopyObject, SelectObjectContent, and ZIP member views (SN-2026-024) |
20261112 |
Source reads could skip stored tags, CopyObject (COPY directive) could mis-evaluate destination request-tag conditions, and ZIP member views could bypass outer tags while persisting an index on disk → Authorizes stored source tags, evaluates COPY destination conditions on copied tags, and authorizes ZIP member GET/HEAD/listing against the ZIP’s stored tags (plus s3:ListBucket for listing) using an in-memory index without mutating the ZIP |
Read-only callers can inspect authorized ZIP archives without write permissions; pass x-minio-extract: true at upload time if a persisted index is desired |
4. S3 Reads, Writes, Encryption, Checksums & Select
| Scenario | Since | Previous Behavior → SILO Behavior | Impact & Action |
|---|---|---|---|
Corrupted or undecryptable Range read status code |
20260804+ / 20261018+ |
Pre-first-byte bit-rot or decryption failures could commit 200/206 before aborting the stream → Failures detected before committing headers return HTTP 500 |
Clients will not cache truncated 206 bodies as valid partial content; alert on 5xx and run mcli admin heal |
Indexed Range reads on compressed + server-side encrypted objects |
20261112 |
Indexed Range reads into compressed SSE-S3, SSE-KMS, or SSE-C objects failed when the target S2 block started inside the first 64 KiB encrypted frame (decryptSkip > 0) → Properly skips intra-frame prefix bytes without rewriting stored objects |
Existing compressed and encrypted objects become readable via arbitrary Range requests immediately upon upgrade |
| New SSE-C uploads and encrypted multipart size reporting | 20260903+ / 20261018+ |
Combining SSE-C with server-side compression risked replica/size mismatches, and encrypted multipart sizes could reflect ciphertext bytes → New SSE-C uploads bypass server-side compression (historical compressed SSE-C objects remain readable), and HeadObject/GetObjectAttributes report accurate plaintext sizes and part counts |
Compress payloads client-side before uploading with SSE-C if storage compression is required |
| SSE-C in-place key rotation and zero-byte verification (SN-2026-006) | 20260903+ |
Zero-byte SSE-C reads skipped customer-key verification, and in-place CopyObject key rotation could corrupt ciphertext or report source-key checksums → Verifies customer keys on zero-byte SSE-C reads/attributes and re-encrypts/checksums cleanly under the destination key |
Ensure probes against zero-byte SSE-C objects supply the customer encryption key headers |
Strict X-Amz-Server-Side-Encryption-Context validation |
20261112 |
Empty, malformed Base64/UTF-8/JSON, trailing tokens, or non-string member values on PutObject, CopyObject, CreateMultipartUpload, or browser POST could return 500 InternalError → Rejects invalid encryption contexts with 400 InvalidArgument |
Verify custom KMS context headers encode a flat JSON object of string/null values in valid Base64 |
S3 Select 1 MiB record size cap (CVE-2026-39414) |
20260618+ |
Oversized CSV / JSON Lines records could trigger unbounded allocations or SIMD overreads → Enforces AWS’s 1 MiB maximum record size (OverMaxRecordSize) |
Stream objects larger than 1 MiB per line directly via GetObject instead of S3 Select |
| Streaming trailer size cap and negative decoded length check | 20261112 |
Inconsistent handling of negative X-Amz-Decoded-Content-Length or truncated/oversized streaming trailers → Rejects negative decoded lengths and truncated or > 64 KiB trailers |
Blocks malformed streaming requests only; standard SDKs are unaffected |
Cross-pool historical self-copy and federated CopyObject (#343) |
20260916+ / 20261112 |
Self-copying a historical version from an older pool into a newer pool (including encrypted or multi-block S2 compressed objects) or federated cross-cluster copies could corrupt logical sizes or seek indexes → Preserves logical streams, encryption re-wrapping, checksums, and multi-block S2 seek indexes | Metadata updates and version rollbacks across expanded storage pools work without manual workarounds |
5. Multipart Uploads, Conditional Operations & Bulk Delete
| Scenario | Since | Previous Behavior → SILO Behavior | Impact & Action |
|---|---|---|---|
| Multipart part order, duplicate parts, and checksum combinations | 20260804+ / 20260903+ |
CompleteMultipartUpload could accept duplicate or out-of-order PartNumber entries → Rejects duplicate or descending part numbers with InvalidPartOrder, rejects CRC64NVME + COMPOSITE, and returns accurate FULL_OBJECT checksums plus ChecksumType |
Sort and deduplicate part lists in custom multipart clients, and tolerate ChecksumType in completion XML |
Multipart upload isolation, Abort, and stale cleanup |
20261018+ / 20261109+ |
Concurrent part retries, AbortMultipartUpload, and stale upload cleanup could hit lock-order deadlocks or timestamp skew → Aligns lock ordering and upload identity/timestamp tracking |
Eliminates high-concurrency multipart hangs transparently |
Retirement of experimental 20260916 strict multipart listing |
20261018+ |
20260916 offered an optional strict persistent multipart listing mode (1,000-entry cap) → Retired in 20261018+; the setting logs a warning and falls back to the default cache/exact-key path |
Remove assumptions of restart-persistent ListMultipartUploads enumeration if you opted into 20260916 strict mode |
Multi-pool conditional writes (PUT, CreateMultipartUpload, CompleteMultipartUpload) |
20260916+ / 20261109+ |
If-None-Match / If-Match could miss versions on another pool or treat unreadable pool metadata as “not found” and overwrite → Verifies authoritative state across pools and returns 503 when pool metadata is unreadable |
Expect 503 (with safe SDK retry) on conditional writes during drive/node outages even if GET still hits a healthy pool |
Single-object conditional delete DeleteObject (If-Match) |
20260916+ |
DeleteObject ignored If-Match → Enforces If-Match against the target object or addressed versionId; bulk DeleteObjects still ignores per-entry <ETag> |
Stop sending stale or placeholder If-Match headers on DeleteObject calls |
Single-pool DeleteObjects path containment (SN-2026-020) |
20261112 |
Single-pool bulk delete lacked uniform lexical .. segment rejection → Rejects dot-segment traversal keys before storage path resolution |
Affects malformed .. object keys only |
Sequential DeleteObjects path for explicit VersionId and locked null versions |
20261109+ / 20261112 |
Parallel fast-path bulk deletes could bypass cross-pool or suspended-bucket lock checks → 20261109 serialized version deletes in Object Lock buckets; starting in 20261112, every DeleteObjects entry with an explicit VersionId in any bucket (and unversioned deletes that can touch a locked null version) runs sequentially through the single-object path |
Large permanent version-purge batches take longer than unversioned delete-marker batches; increase client timeouts or use smaller batch sizes |
6. Object Lock & Replication
| Scenario | Since | Previous Behavior → SILO Behavior | Impact & Action |
|---|---|---|---|
| Replication header signing and stripping (CVE-2026-34204, SN-2026-008) | 20260417+ / 20261112 |
Client-supplied X-Minio-Source-* or unsigned X-Minio-Replication-* headers could influence replica metadata → Requires SigV4 SignedHeaders coverage across X-Minio-Replication-* plus s3:ReplicateObject / s3:ReplicateDelete, stripping unauthorized headers post-auth |
Never send internal X-Minio-Source-* or X-Minio-Replication-* headers from regular S3 clients |
PutObjectRetention base and bypass permission check (SN-2026-015) |
20261018+ |
Shortening retention or changing lock mode could miss combined base + governance-bypass checks on edge paths → Requires both s3:PutObjectRetention and s3:BypassGovernanceRetention under the write lock |
Ensure governance-override admin roles carry both s3:PutObjectRetention and s3:BypassGovernanceRetention |
Locked null-version protection in suspended or prefix-excluded buckets (SN-2026-016, #336) |
20261018+ / 20261112 |
Suspending versioning or excluding prefixes could allow ordinary overwrites, delete markers, or shadowed unversioned DELETE calls to replace or hide a locked null version → Blocks ordinary overwrites, delete markers, and permanent deletes against locked null versions across single-pool and multi-pool clusters |
Retained or legally held null versions in suspended buckets cannot be overwritten under the same key until their lock expires or is cleared |
Fail-closed x-minio-force-delete on bucket lock metadata errors |
20261112 |
Unreadable bucket Object Lock configuration could allow x-minio-force-delete to proceed without lock verification → Fails closed when bucket lock metadata cannot be read |
Restore storage read quorum before retrying administrative force-deletes |
Target-side replicated DELETE lock enforcement and COMPLIANCE floor |
20261109+ / 20261112 |
Replicated permanent deletes or peer timestamps could bypass or shorten unexpired destination locks → Target-side replicated DELETE enforces destination retention/legal hold, and incoming peer timestamps cannot strip or shorten unexpired COMPLIANCE retention |
Keep NTP clocks and retention policies aligned across replication peers; refused deletes surface in replication retry/failure metrics |
| Per-field tag/retention/legal-hold ordering and default bucket retention | 20260916+ / 20261112 |
Shared coarse timestamps could let tag edits clobber lock state, and setting Legal Hold could skip default bucket retention → Tracks independent timestamps for tags, retention, and legal hold and applies default bucket retention alongside Legal Hold | Prevents concurrent tag/lock overwrites on future writes; already-overwritten historical metadata is not reconstructed automatically |
Staged replica integrity checks, multipart checksums, and resync terminal states |
20261026+ / 20261112 |
Staged replicas lacked pre-commit SHA-256/SSE-C/ActualSize verification, and composite CRC multipart replicas omitted FULL_OBJECT type → Verifies content hash, SSE-C state, and compressed ActualSize before renameData, forwards FULL_OBJECT, and fixes replicate resync state reporting and MRF counters |
Upgrade source and destination clusters together and monitor mcli replicate status |
7. Multi-Pool Storage, Listing, Migration & Healing
| Scenario | Since | Previous Behavior → SILO Behavior | Impact & Action |
|---|---|---|---|
| Authoritative cross-pool reads, writes, and permanent deletes | 20260916+ / 20261026+ |
Partial pool visibility could return false 404 when a newer version lived on an unreadable pool, or remove the authoritative copy before stale pool copies → Returns 503 when unreadable pools cannot rule out a newer version, and deletes non-authoritative copies before the authoritative copy while protecting shared remote-tier references |
Treat 503 as retryable in clients and backup jobs; never treat a failed read or listing as proof of deletion |
Heterogeneous EC-layout LIST and uncertain-entry recovery |
20261018+ |
Listing across pools with different erasure stripe sizes could drop uncertain entries or turn errors into empty pages → Confirms eligible uncertain LIST entries or returns an explicit error (no cross-page snapshot is promised) |
Check error returns on every listing page before running downstream mirror deletions |
Empty-only directory cleanup after failed renameData |
20261109+ |
Failed write commits could either leave empty ghost directories or risk removing concurrent sibling versions → Removes only empty object directories during failed renameData cleanup |
Prevents ghost directories without risking sibling object versions |
Pool rebalance metadata placement, tags, and stop convergence |
20260916+ / 20261112 |
In-place replica metadata updates during pool draining/rebalance could misplace data or drop tags/delete markers, and rebalance stop could stall → Keeps in-place metadata updates on the authoritative pool (moving to an active pool only when encryption/checksum recomputation is required), preserves tags/markers, continues past single-bucket errors, and converges stop state cleanly |
Avoid scheduling massive bucket re-encryption jobs concurrently with pool rebalance |
Pool decommission nested child-key rescan, bounded progress, and offline-drive check (#334) |
20261112 |
Coexisting parent and child keys (e.g., a/b and a/b/c) could leave child keys unmigrated, offline drives blocked emptiness verification, and zero-progress passes could loop indefinitely → Rescans hidden nested child keys, stops passes without physical progress with cancellation errors, and tolerates offline drives within each erasure set’s read quorum when verifying cleanup |
Follow the updated Pool Decommission Guide; offline drives must remain within read quorum |
heal --dry-run read-only guarantee |
20261109+ |
Edge healing paths could still write metadata during --dry-run → Strictly prevents disk or metadata mutations during --dry-run |
Safe to run mcli admin heal --dry-run for non-invasive health audits |
Removal of private internode StorageREST /rmpl (ReadMultiple) (CVE-2026-42600) |
20260618+ |
Exposed private ReadMultiple peer RPC → Removed /rmpl and added lexical path containment and allocation bounds across internode storage RPCs (SN-2026-002) |
Transparent to S3 clients, but prevents mixing upstream MinIO and SILO nodes inside one distributed cluster |
8. Bucket Configuration, CORS & Notifications
| Scenario | Since | Previous Behavior → SILO Behavior | Impact & Action |
|---|---|---|---|
| Per-bucket CORS persistence and enforcement | 20260903+ |
Bucket ?cors APIs were placeholders in upstream community builds and always fell back to global CORS → PutBucketCors, GetBucketCors, and DeleteBucketCors persist, export/import, replicate, and override global MINIO_API_CORS_ALLOW_ORIGIN for that bucket |
Inspect existing buckets with mcli cors get before upgrading in case stale test CORS rules were stored previously |
| Serialized bucket metadata updates, stale-cache guard, and deletion tombstones | 20260903+ / 20260916+ / 20261112 |
Concurrent policy/lifecycle/tag edits could overwrite each other, GetConfig could overwrite newer config from stale cache, and multi-site config deletions could be resurrected by peers → Serializes writers under metadata.lock, guards BucketMetadataSys.GetConfig against stale cache overwrites, and supports opt-in MINIO_SITE_REPLICATION_METADATA_TOMBSTONES once all sites upgrade |
Upgrade all replicated sites before enabling MINIO_SITE_REPLICATION_METADATA_TOMBSTONES consistently |
| PostgreSQL and MySQL notification DSN requirement | 20260903+ |
Accepted legacy discrete host/port/user settings for SQL notification targets → Requires explicit connection_string (PostgreSQL) or dsn_string (MySQL) at startup |
Consolidate legacy SQL notification parameters into a connection string before cutover |
| NATS / AMQP setting registration and credential redaction (SN-2026-019) | 20260804+ / 20260916+ |
Rejected valid NATS/AMQP keys or echoed raw URI passwords in validation errors → Registers supported keys and redacts embedded credentials in error and debug output | Update log parsers if they matched unredacted notification URI error text |
9. Lifecycle (ILM), Remote Tiering & Archive Restore
| Scenario | Since | Previous Behavior → SILO Behavior | Impact & Action |
|---|---|---|---|
Standalone ExpiredObjectDeleteMarker and exact-key all-version purge guards |
20261026+ / 20261109+ |
Versioned remote-tier expiration and whole-key purges could delete shared remote-tier blobs prematurely or skip standalone delete-marker rules → Delete-marker-only expiry preserves prior versions’ remote-tier data; exact-key all-version purge re-verifies every version and refuses whole-key removal if any version is locked, tiered, or restored; standalone DelMarkerExpiration rules execute properly |
Review ILM rules before upgrading to verify DelMarkerExpiration and all-version expiry prefixes match intent |
Hiding internal remote-tier FreeVersion records |
20261026+ |
Internal FreeVersion tier-cleanup markers could appear in user version listings → Hidden from user-facing listings |
Cleaner ListObjectVersions output with no internal tiering markers |
Two-phase lock-free native restore (POST ?restore) and all-peer gate |
20261109+ |
Remote tier restore held object locks across slow downloads and lacked explicit duplicate-in-progress semantics → Uses two-phase lock-free download/commit, returns 409 RestoreAlreadyInProgress on concurrent duplicate requests, fixes restored-copy expiry precision, and returns 503 on POST ?restore while any peer is unupgraded or unreachable |
Complete the upgrade across every cluster node and verify peer health before issuing POST ?restore calls |
Pre-2024-04-28 xl.meta header and UsesDataDir revision normalization (#332, #344) |
20261112 |
Partially healed pre-2024-04-28 objects or expired restored copies could compute divergent metadata revisions across drives due to header version or time-dependent xlFlagUsesDataDir bits, splitting read quorum → Normalizes legacy xl.meta headers and clears xlFlagUsesDataDir from revision comparison |
Resolves recurring quorum splits on aged historical objects automatically without rewriting stored data |
10. Batch Jobs (expire, replicate, keyrotate)
| Scenario | Since | Previous Behavior → SILO Behavior | Impact & Action |
|---|---|---|---|
| Submitter-bound live authorization and mid-run revalidation | 20261109+ |
Submitted batch jobs ran under broad internal privileges even if the submitter was later disabled or narrowed → Binds jobs durably to the submitter, revalidates credential status and effective policies at start and during execution, restricts unavailable request-context conditions, and pauses/retries when an external authz plugin is temporarily unreachable | Submit long-running batch jobs using dedicated persistent admin service accounts rather than short-lived STS sessions |
SILO-BATCH v2 manifest and report gate |
20261109+ |
Unversioned / v1 batch manifests lacked submitter binding → Writes SILO-BATCH v2 manifests and reports; unfinished legacy jobs are not guaranteed to resume across the upgrade, and 20261026 or earlier rejects v2 records |
Stop new batch submissions and drain or cancel all running, queued, or paused jobs before upgrading; resubmit after all nodes run 20261112 |
Per-version lock checks, contiguous progress, and complete-with-skips |
20261109+ |
Skipped locked/unauthorized versions could advance checkpoints past unfinished items or report plain completion → Checks locks and permissions per version, advances checkpoints contiguously, reports complete-with-skips when items are skipped, and runs a full rescan when resuming replicate jobs |
Update batch monitoring scripts to inspect complete-with-skips and skip/failure counters |
keyrotate KMS context persistence, metadata preservation, and XLV1 → XLV2 rewrite |
20261109+ / 20261112 |
Batch key rotation could drop configured KMS Encryption Context, alter pool placement/tags/Expires, or fail on legacy XLV1 objects → Persists KMS context across execution, retry, and reload (validating UTF-8 before rekeying), preserves pool, version, tags, multipart/tier headers, and expiry, and stream-rewrites legacy XLV1 objects onto XLV2 with sealed plaintext MD5 ETags |
Review YAML job definitions against the updated Batch Key Rotation Guide |
11. HTTP Timeouts, Observability, Health Checks & mcli Client
| Scenario | Since | Previous Behavior → SILO Behavior | Impact & Action |
|---|---|---|---|
| Absolute HTTP/1 request-header read deadline | 20260916+ |
Slow byte-by-byte headers could extend rolling deadlines indefinitely → Applies an absolute read deadline to HTTP/1 headers while keeping rolling idle timeouts for request bodies | Mitigates Slowloris header stalls without affecting large streaming uploads; check idle pre-connected proxy pools |
Dynamic webhook/Kafka logger reload and bad queue_dir safety |
20261109+ |
Dynamic audit/logger updates could miss peer reload or panic on an unwritable queue_dir → Synchronizes peer reloads, returns configuration errors instead of panicking on bad queue_dir, and caps AMQP frame sizes |
Verify cluster-wide logger status with mcli admin info after live config updates |
| Prometheus metric fixes and new diagnostic counters | 20260903+ / 20260916+ |
Inaccurate effective bucket-quota metrics, unguarded concurrent CPU collection, and missing MRF/IAM revocation counters → Retains minio_* metric names, fixes effective quota reporting and CPU collection concurrency, adds MRF drop queue-entry counters and IAM revocation sync metrics, and switches Console dashboards to Metrics V3 |
Review custom Grafana panels and alerts (note that MRF drop counters count queue entries, not unique lost objects) |
silo healthcheck vs. S3 data-path readiness |
20260903+ / 20261109+ |
No built-in binary probe subcommand → Adds silo healthcheck ready / live / cluster; passing health probes does not replace signed S3 read/write verification |
Use silo healthcheck ready for container probes, and run bounded signed S3 read/write checks against every node during upgrades |
mcli config path, exit codes, move safety, and {} policy reset |
20260806+–20261112 |
Upstream mc read ~/.mc, exited 0 on several partial failures, could delete sources on same-IP alias self-moves, and could not pass {} on access-key edits → Invoking mcli uses ~/.mcli (pgsty/mc containers keep /root/.mc and pre-create 1777 /.mc for non-root UIDs); cp/mv/mirror/rm/imports return non-zero on item failures; mv rejects ambiguous same-scheme/port/IP moves; mcli 20261112 accepts --policy '{}' to restore inheritance and enforces strict policy JSON decoding; adds read-only mcli checksum verify |
Copy ~/.mc to ~/.mcli once (or set MC_CONFIG_DIR) on hosts switching to mcli; check script handling of non-zero exit codes and growing totalCount/totalSize JSON progress fields; see MCLI Compatibility |
12. Embedded & Standalone Web Console
| Scenario | Since | Previous Behavior → SILO Behavior | Impact & Action |
|---|---|---|---|
| Full administration UI retained | v2.0.0+ |
Upstream pared community Console down to an object browser → SILO retains full cluster, bucket, lifecycle, replication, notification, tiering, IAM, IdP, KMS, and diagnostic administration | Full browser administration remains built in; S3 APIs are unaffected |
| Cursor-paginated object browser and single-page selection scope | v2.4.0+ |
Loaded unbounded object lists for client-side sorting/select-all → Cursor pagination (default 100, options 50–1,000) with first/previous/next controls; sorting, name filtering, and select-all apply only to the current page; rewind and deleted-object views cap at 1,000 versions with truncation notices | Use mcli CLI commands rather than browser select-all for bulk operations across thousands of objects |
| Tri-state service-account inline-policy editor and validation (SN-2026-025) | v2.7.1 / v2.7.2 |
Metadata-only access-key edits could snapshot inherited policies as inline copies or accept empty policy variants → Tracks inherited vs. inline policies separately; metadata edits preserve inheritance; turning off Restrict beyond user policy sends {} to restore inheritance (requires Server 20261112+); blocks empty {} / Statement while restricted and renders save errors inside the modal |
Upgrade standalone Console deployments to v2.7.2 alongside Server 20261112 |
| Permission-aware controls, explicit Governance bypass, and error fidelity | v2.4.1+–v2.7.2 |
Opaque permission errors, session logouts on transient network errors, and masked versioning errors → Reflects effective session capabilities, adds an explicit Governance bypass toggle, separates network/5xx errors from 401 session expiry, and surfaces wrapped S3 error codes (such as InvalidBucketState) |
Console API consumers can also read added accountAccessKey and size: 0 fields |
| Restricted share-download proxy and native streaming ZIP (SN-2026-014) | v2.4.1+ / v2.5.1+ |
Broad URL handling on anonymous share downloads → Restricts share-download targets to configured cluster endpoints and supports bounded streaming ZIP multi-object downloads with pre-byte S3 error status preservation | Verify configured external Console/S3 URLs when running behind reverse proxies |
Live bucket inventory, folder usage scan, 1 MiB text preview, and 10 UI languages |
v2.5.1+–v2.7.2 |
Cached account-info bucket lists after create/delete, no bounded folder sizing, and EN/ZH only → Reads live bucket inventory (Server 20261026+), adds cancellable folder/bucket usage scans (working over plain HTTP and optional usage-worker), safe 1 MiB text preview, 10 bundled languages including Arabic RTL, host-preserving CONSOLE_SECURE_TLS_REDIRECT on 0.0.0.0, OIDC array group claims, and valid UTF-8 credential exports |
Stop older standalone console usage-worker processes before starting v2.7.0+ workers; see Console Compatibility |
13. Withdrawn Experiments & Non-Goals
The items below either appeared briefly in development / a single release before being withdrawn, or represent boundaries that upgrading does not perform automatically:
| Item | Current Status & Scope | Practical Takeaway |
|---|---|---|
| Access-frequency inter-pool tiering | Explored briefly after the fork and removed completely before 20260916; never shipped in a public Server release (removal record) |
Standard lifecycle expiration, remote object-store tiering (S3/GCS/Azure/SILO), rebalance, and decommission remain fully supported |
Optional 20260916 strict multipart listing mode |
Retired in 20261018+; setting the switch logs a warning and uses the default cache/exact-key path |
Do not build workflows assuming restart-persistent ListMultipartUploads enumeration |
Experimental write-generation fencing (consistency_strict build tag) |
Not enabled in published releases; request timeouts or lease loss do not prove underlying disk I/O has stopped | Maintain standard erasure quorum and operational runbooks rather than assuming distributed write fencing |
| Automatic resurrection of lost historical metadata or policy clauses | Repairs protect new writes, copies, key rotations, and policy edits, without guessing or reconstructing tags, locks, remote-tier blobs, or Deny clauses already lost on older builds |
Audit critical buckets after cutover using mcli checksum verify, mcli admin heal --dry-run, and IAM policy review |
14. Pre-Upgrade & Rollback Checklist
Follow these five steps when cutting over from upstream MinIO or upgrading an earlier SILO deployment to SILO 20261112:
- Verify disk format, quorum health, and rollout mode: Inspect
.minio.sys/format.jsonread-only (xlorxl-single) and runmcli admin infoto confirm healthy read quorum across every pool and erasure set. Schedule a coordinated full-cluster stop when migrating from upstream MinIO or when upgrading a single pool whose erasure sets span distinct host subsets (typically ≥ 18 hosts) from≤ 20261026; pause concurrent S3 writes across multi-pool rolling windows. - Drain background jobs and capture an offline recovery point: Stop new Batch submissions and let running, queued, or paused Batch jobs finish (or cancel them). Record exact image digests / binary versions and rendered systemd / Compose / Helm manifests, export IAM and bucket metadata (
mcli admin cluster iam export,mcli admin cluster bucket export), and capture a consistent backup of.minio.sysand config files once writes are paused. - Audit configuration, policies, and external TLS/auth: Replace hard-coded
/usr/bin/miniopaths withsiloand pass--set image.tag=RELEASE.2026-11-12T00-00-00Zwithhelm upgrade --reuse-values. Check custom IAM policies for bare bucket ARNs on the 12 bucket writes,s3:DeleteObjectVersion,admin:ChangeMyPassword, strict JSON syntax, and service-account inline restrictions; verify asymmetric OIDC JWKS (RSA ≤ 4096 bits), SQL notification DSNs,MINIO_API_TRUSTED_PROXIES, and external CA roots. - Cut over atomically and verify the S3 data path on every node: After
silo healthcheck readysucceeds, run bounded signed S3 write, read, conditional, andRangechecks directly through every individual node (covering representative multipart, encrypted, and compressed historical objects), then validate Console login, representative sub-user credentials, replication queues, and notification delivery. - Resume background operations and reconcile multi-site policies: Once every cluster node and replicated peer site is running
20261112, resubmit Batch jobs, re-enable archivePOST ?restoreworkflows, and verify effective service-account policies across replicated sites.
Mounting existing disks in place does not mean you can swap back to an older binary later without restoring state. Startup routines, background scanners, IAM edits (durable revocation tombstones and tri-state inline policies), SILO-BATCH v2 manifests, and bucket metadata tombstones update .minio.sys state in ways older MinIO or SILO binaries may reject or misinterpret. If you must roll back, restore your pre-cutover consistent backup across all participating nodes and replication peers together — never downgrade a single node’s binary or image in isolation.
15. Audit Scope & References
- Fork comparison baselines: Server
minio/minio@27742d469462(2025-12-03), clientminio/mc@77f82e18b540, Consolesilo-console@feff71e48e39. - Current target stack (
2026-11-12): SILO Server 20261112 (44e848fedf8c), Console v2.7.2 (0e7953ff59c4), mcli 20261112 (a37a9f054fba),silo-pkg v3.16.1(433f7eefa714). - Companion references: Compatibility Overview · Component Version Matrix · Migration Guide · Native Package Migration · Server Audit Ledger · MCLI Compatibility · Console Compatibility · Security Advisories.