# Product Updates Storage System Documentation

**Last Updated**: 2026-04-08  
**Status**: Production Ready ✅

## Overview

Product Updates uses a **single source of truth (SSOT)** on disk:

- **Default root:** `v2/data/produkt-updates/`
  - `produkt_updates.json`
  - `images/` (uploaded assets)
- **Override:** environment variable **`ORDIO_PRODUKT_UPDATES_ROOT`** (absolute path, e.g. outside webroot).

`findReadableDataFile()` and `findWritableDataFile()` return the **same path**. There is **no** multi-location JSON fallback and **no** `/tmp/` data fallback in normal operation.

**Migration from legacy** `wp-content/uploads/produkt-updates/`:

1. `php v2/scripts/dev-helpers/audit-produkt-updates-storage.php` (inventory)
2. `php v2/admin/produkt-updates/migrate-to-ssot-storage.php --dry-run` then `--migrate` (add `--force` to overwrite target JSON)
3. `php v2/admin/produkt-updates/normalize-produkt-updates-image-paths.php` as needed (`--dry-run` / `--write`)

**Nginx:** see [NGINX_PRODUKT_UPDATES.md](./NGINX_PRODUKT_UPDATES.md).

## Deployment safety (Git / CI)

- **Do not** rsync or deploy over **`v2/data/produkt-updates/produkt_updates.json`** or **`v2/data/produkt-updates/images/`** (or the path set by `ORDIO_PRODUKT_UPDATES_ROOT`). Those are live CMS content; `.gitignore` excludes them from the repo by default.
- **Do not** ship or rsync a file at **`v2/data/produkt_updates.json`** on production (it triggers “legacy JSON” warnings and is easy to confuse with SSOT). A **read-only sample** for local reference lives at **`v2/data/samples/produkt_updates.sample.json`**; use `migrate-to-ssot-storage.php` to seed SSOT from a chosen source when needed.
- After path fixes, run **`php v2/admin/produkt-updates/normalize-produkt-updates-image-paths.php --dry-run`** then **`--write`**. Canonical public URLs stay **`/produkt-updates/bilder/{file}`**; files live under SSOT `images/`.
- **Public vs filesystem:** Browsers use `/produkt-updates/bilder/{file}` (`produkt-updates-bilder.php`). Do not use bare `/produkt-updates/{filename}.webp` for assets — that namespace is for **post slugs**.
- Monitoring: **`php v2/api/produkt-updates-image-persistence-check.php --fail-on-issues`**, **`php v2/scripts/dev-helpers/verify-produkt-updates-ssot.php --fail-on-issues`**.
- **OWASP-style note:** user uploads live under a dedicated tree; public delivery goes through the bilder script with path allowlisting (`produktUpdatesImagePathIsUnderServedRoot`).
- **GitHub Actions:** [`.github/workflows/production-deployment.yml`](../../../.github/workflows/production-deployment.yml) rsync **excludes** `v2/data/produkt-updates/` and `v2/data/backups/` so deploys do not replace live CMS files or reset ownership (rsync previously synced `.gitkeep` with the runner UID and could leave the SSOT directory owned by a non–php-fpm user, breaking writes).

### Production ops checklist (SSH)

Use this when the admin shows storage warnings, **503** on `produkt-updates-upload.php`, or **500** on `produkt-updates-autosave.php`.

1. **Identify the PHP user** (Debian/Ubuntu typical): `ps aux | grep 'php-fpm: pool'` — workers are often `www-data`.
2. **Confirm SSOT paths:** default `{app}/v2/data/produkt-updates/` with `produkt_updates.json` and `images/`. Atomic writes need a **writable directory** for `.tmp` and `.lock` next to the JSON file (not only a writable file with a non-writable parent).
3. **Fix ownership (example):**  
   `chown -R www-data:www-data /var/www/lexikon/v2/data/produkt-updates`  
   `chmod 775 /var/www/lexikon/v2/data/produkt-updates /var/www/lexikon/v2/data/produkt-updates/images`  
   `chmod 664 /var/www/lexikon/v2/data/produkt-updates/produkt_updates.json`  
   Replace `www-data` and `/var/www/lexikon` with your pool user and app root.
4. **Verify:** `sudo -u www-data php v2/scripts/dev-helpers/verify-produkt-updates-ssot.php --fail-on-issues` (from app root).
5. **Optional dedicated root:** set **`ORDIO_PRODUKT_UPDATES_ROOT`** (absolute path) in the php-fpm pool environment if SSOT must live outside the deploy tree; reload php-fpm.
6. **Legacy JSON on disk:** duplicate `produkt_updates.json` files under `wp-content/` or `v2/data/` confuse operators; after SSOT is verified, rename to e.g. `produkt_updates.json.archived-YYYYMMDD` (keep backups elsewhere as needed).

**Hosting decision (Ordio production, 2026-04):** SSOT stays at the default **`v2/data/produkt-updates/`** under the application root; **`ORDIO_PRODUKT_UPDATES_ROOT` is unset**. Revisit a dedicated path outside `/var/www/lexikon` if the deploy tree ever becomes read-only for PHP or multiple nodes must share one CMS dataset.

### Shared images and when files are deleted

- One file under SSOT `images/` (or legacy wp uploads during transition) may be referenced by **more than one** feature or improvement: `featured_image`, legacy `images[]`, or inline `<img src>` in `page_content` / `description`.
- **Replace, clear featured image, or delete a feature/improvement row** updates JSON only. **Files stay on disk** for reuse until an editor deletes them from the **image library** (confirm in UI).
- **Only explicit library delete** ([`v2/api/produkt-updates-upload.php`](../../v2/api/produkt-updates-upload.php) `action=delete`) **removes bytes** and runs **`cleanupImageFromJson()`** to strip matching refs from content. This matches common CMS practice (WordPress-style media library: detach vs delete).
- **Usage counting** for “In Use” badges: **`produktUpdatesCountBasenameReferencesInData()`** in [`v2/includes/produkt-updates-paths.php`](../../v2/includes/produkt-updates-paths.php) (CLI: `php v2/scripts/dev-helpers/test-produkt-updates-image-refcount.php`). It does **not** trigger deletion.
- **Orphan files** (on disk but unreferenced in JSON) may accumulate after many post deletions; clean up via library or a separate maintenance script if needed.
- **Library delete + HTML cleanup:** `cleanupImageFromJson()` removes whole `<img>` tags when `src` contains the matching basename, using patterns for Quill-style markup (`src` first or after other attributes). It does **not** strip **`srcset`**-only references; rare complex `src` encodings may need manual JSON edit.

### Troubleshooting: Mediathek-Löschung, HTTP 503, „Upload-Verzeichnis nicht beschreibbar“

**Symptom:** Beim Löschen aus der Bildbibliothek im Admin erscheint ein Fehler wie „Upload-Verzeichnis nicht beschreibbar … v2/data/produkt-updates/ oder ORDIO_PRODUKT_UPDATES_ROOT“, und die Konsole zeigt **503** auf `POST /v2/api/produkt-updates-upload.php`.

**Ursachen und Fix:**

1. **Berechtigungen (häufig in Produktion):** PHP (z. B. `www-data`, `nginx`, `php-fpm`) braucht **Schreibrechte** auf `{root}/` und **`{root}/images/`**, um Dateien zu löschen (POSIX: Schreibrecht auf das **übergeordnete Verzeichnis** zum `unlink`). Außerdem muss **`produkt_updates.json`** beschreibbar sein, damit nach dem Löschen `cleanupImageFromJson()` die Verweise atomar aktualisieren kann.
2. **Umgebungsvariable:** Wenn der Standardpfad nicht beschreibbar ist (z. B. nach Deploy nur App-Code beschreibbar), **`ORDIO_PRODUKT_UPDATES_ROOT`** auf einen **persistenten, absoluten Pfad** setzen (Volume außerhalb des read-only Trees) und Verzeichnisse anlegen bzw. Rechte setzen.
3. **Diagnose-URLs (auth wie jeweils dokumentiert):**
   - [`/v2/api/produkt-updates-diagnostics.php`](../../../v2/api/produkt-updates-diagnostics.php) — `resolved_paths`, `storage`, `dirs_ok` / Hinweise
   - [`/v2/api/produkt-updates-storage-health.php`](../../../v2/api/produkt-updates-storage-health.php) — Monitoring-Health
   - [`/v2/api/produkt-updates-server-investigation.php`](../../../v2/api/produkt-updates-server-investigation.php) — Server-Kontext
4. **CLI:** `php v2/scripts/dev-helpers/verify-produkt-updates-ssot.php --fail-on-issues`
5. **SSH-Checkliste (Produktion):** PHP-Prozess-User ermitteln → `chown`/`chmod` auf `{root}` und `{root}/images/` → sicherstellen, dass Deploy/rsync **nicht** überschreibt (siehe Abschnitt *Deployment safety* oben) → Löschung in der Mediathek erneut testen.

**Verhalten im Code (Referenz):** Neue **Uploads** prüfen weiterhin `findWritableImageDir()` und liefern bei fehlendem SSOT-`images/`-Schreibzugriff **503**. **Mediathek-Löschung** (`action=delete`) nutzt diese Upload-Sperre nicht mehr als globales Vorschalttor; stattdessen prüft sie u. a. `produktUpdatesJsonPersistenceWritable()` (JSON aktualisierbar) und meldet **`unlink`-Probleme** mit einer eigenen Meldung (Verzeichnisrechte).

## Storage Architecture

### SSOT layout

| Item | Path |
|------|------|
| Config | `v2/config/produkt-updates-storage.php` |
| JSON | `{root}/produkt_updates.json` |
| Images | `{root}/images/` |
| Atomic writes + lock | `v2/includes/produkt-updates-json.php` |

`{root}` = `produktUpdatesStorageRoot()` (default `v2/data/produkt-updates`, override via `ORDIO_PRODUKT_UPDATES_ROOT`).

### Path resolution

- **`findWritableDataFile()`** and **`findReadableDataFile()`** both return **`produktUpdatesDataFilePath()`**.
- **`findWritableImageDir()`** — SSOT `images/` only.
- **`findReadableImage()`** — SSOT first, then legacy `wp-content/uploads/produkt-updates/` (read-only) during migration.

### Directory creation

`produktUpdatesEnsureStorageDirectories()` creates `{root}` and `{root}/images` with `0755` when possible.

## Image Storage

> **Note**: See [IMAGE_REFERENCE_STRATEGY_ANALYSIS.md](./IMAGE_REFERENCE_STRATEGY_ANALYSIS.md). **Implemented 2026-02**: Primary location uses direct paths for SEO and performance.

### Writes and reads

- **Writes:** only `{root}/images/` (SSOT).
- **Reads:** SSOT `images/` first, then legacy `wp-content/uploads/produkt-updates/` until migration removes duplicates.

**`getAllImageDirs()`** returns SSOT plus legacy (if different) for library listing and deletes.

### Image serving

- **Canonical URL:** `/produkt-updates/bilder/{filename}` → `produkt-updates-bilder.php` (validates path with `produktUpdatesImagePathIsUnderServedRoot()`).
- **Legacy proxy:** `/v2/api/serve-produkt-updates-image.php?file=` still resolves via `findReadableImage()`; prefer bilder URLs in JSON.

### Dynamic image flows (end-to-end reference)

| Step | Component | Role |
|------|-----------|------|
| Upload | [`v2/api/produkt-updates-upload.php`](../../../v2/api/produkt-updates-upload.php) | Writes to `{root}/images/`, optional WebP via `optimizeImage()`, returns canonical `/produkt-updates/bilder/` URLs; `storage_location` in JSON reflects SSOT classification (`produktUpdatesClassifyImageStorageLocation`). |
| Delete | Same API `action=delete` | Unlinks file (+ WebP sidecar when applicable) under all `getAllImageDirs()` locations; runs `cleanupImageFromJson()` (atomic JSON write). |
| Set featured | [`v2/pages/produkt_updates_admin.php`](../../../v2/pages/produkt_updates_admin.php) `set_feature_image` | `normalizeFeaturedImageForStorage()` + `resolveImagePathForValidation()` + `saveUpdatesData()`. |
| Library list | [`v2/api/produkt-updates-diagnostics.php`](../../../v2/api/produkt-updates-diagnostics.php) | `listExistingImages()` + JSON merge + `dedupeProduktUpdatesExistingImagesBuckets()`; each row `url` is **canonical** (`canonicalizeProduktUpdatesImageRef`). |
| “In use” / badges | [`v2/api/produkt-updates-get-image-usage.php`](../../../v2/api/produkt-updates-get-image-usage.php) | Keys = canonical paths; `type` values: `feature_featured`, `feature_gallery`, `feature_inline`, `improvement_*`. |
| Public delivery | [`v2/pages/produkt-updates-bilder.php`](../../../v2/pages/produkt-updates-bilder.php), [`v2/api/serve-produkt-updates-image.php`](../../../v2/api/serve-produkt-updates-image.php) | Path allowlisting via `produktUpdatesImagePathIsUnderServedRoot()` / `findReadableImage()`. |

## Current production setup (after SSOT migration)

### Data file

**Location:** `{root}/produkt_updates.json` (default `v2/data/produkt-updates/produkt_updates.json`)

### Images

**Location:** `{root}/images/`

**Serving:** `/produkt-updates/bilder/{filename}` (see [NGINX_PRODUKT_UPDATES.md](./NGINX_PRODUKT_UPDATES.md) on Nginx).

### Feature / improvement publication fields

- Optional **`publication_status`**: `draft` keeps the row off all public surfaces (pages, feeds, carousel, sitemap, LLMS helpers). Omitted or **`published`** uses date-based visibility.
- **Scheduling**: `published` (or legacy absent) plus a **future** `published_date` hides the item until that calendar date in **Europe/Berlin** (single source of truth: [`v2/helpers/produkt-updates-public-visibility.php`](../../v2/helpers/produkt-updates-public-visibility.php)).
- **Persistence**: Full saves go through **`saveUpdatesData()`** in [`v2/includes/produkt-updates-save-updates-data.php`](../../v2/includes/produkt-updates-save-updates-data.php) (atomic write, regeneration). Month layout regeneration is [`v2/includes/produkt-updates-regenerate-months.php`](../../v2/includes/produkt-updates-regenerate-months.php). The admin page must require this save include **before** any code path calls `saveUpdatesData()` or `regenerateMonthsFromContent()` (including SSOT read/migrate on authenticated GET). CLI check: `php v2/scripts/dev-helpers/verify-produkt-updates-admin-includes.php`.
- **Debounced autosave** (admin UI): [`v2/api/produkt-updates-autosave.php`](../../v2/api/produkt-updates-autosave.php) merges fields and reuses the same save path; new rows without `id` are created as **draft** stubs.

## Migration Process

### When Migration is Needed

Migration is needed when:

- Data/images are stored in `/tmp/` (volatile)
- Admin panel shows "Storage Status" warnings
- Data disappears after server restart

### Data Migration

**Script**: `/v2/admin/produkt-updates/migrate-to-persistent-storage.php`

**Usage**:

```
GET /v2/admin/produkt-updates/migrate-to-persistent-storage.php?action=check
GET /v2/admin/produkt-updates/migrate-to-persistent-storage.php?action=migrate
```

**Process**:

1. Checks current data file location
2. Identifies target persistent location
3. Creates backup of current data
4. Copies data to persistent location
5. Verifies data integrity
6. Updates file permissions
7. Provides rollback option

### Image Migration

**Script**: `/v2/admin/produkt-updates/migrate-missing-images.php`

**Usage**:

```
GET /v2/admin/produkt-updates/migrate-missing-images.php?action=migrate&password=...
```

**Process**:

1. Identifies images referenced in JSON but missing from primary location
2. Searches all known image directories (including old locations)
3. Copies found images to primary persistent location
4. Updates JSON `featured_image` paths to new location
5. Reports migration statistics

## Storage Status Monitoring

### Admin Panel Display

**Location**: Settings section → Storage Status card

**Displays**:

- Current data file location
- Current image directory location
- Persistence status (persistent vs volatile)
- Warnings if using volatile storage
- Links to migration scripts if needed

### Health Check Endpoint

**URL**: `/v2/api/produkt-updates-storage-health.php`

**Returns**:

- Storage status (ok, warning, critical)
- Data file location and persistence
- Image directory location and persistence
- Server restart impact analysis
- Recommendations

## File Permissions

### Required Permissions

**Directories**:

- `755` (drwxr-xr-x) - readable and executable by all, writable by owner
- Owner: `www-data:www-data` (web server user)

**Files**:

- `644` (-rw-r--r--) - readable by all, writable by owner
- Owner: `www-data:www-data` (web server user)

### Permission Issues

**Common Problem**: Owner mismatch

- Deployment user creates files: `1001:1001`
- Web server needs: `www-data:www-data`
- Result: Files exist but not writable

**Solution**: Use WordPress uploads directory (already has correct permissions)

## Atomic Writes

### Implementation

**Process**:

1. Write to temporary file: `produkt_updates.json.tmp`
2. Verify file integrity
3. Atomic rename: `rename($temp_file, $file)`
4. Clean up temp file on error

**Benefits**:

- Prevents partial writes
- Prevents data corruption
- Prevents race conditions

**Code Pattern**:

```php
$temp_file = $file . '.tmp';
file_put_contents($temp_file, $json_data, LOCK_EX);
// Verify integrity
$temp_content = file_get_contents($temp_file);
if ($temp_content === $json_data) {
    rename($temp_file, $file); // Atomic operation
}
```

## File Locking

### When to Use

- Migration scripts
- Bulk operations
- Concurrent write operations

### Implementation

```php
$lock_file = __DIR__ . '/../../data/.migration.lock';
$lock_handle = fopen($lock_file, 'c');
if (!$lock_handle) {
    return false;
}

if (!flock($lock_handle, LOCK_EX | LOCK_NB)) {
    fclose($lock_handle);
    return false; // Operation already in progress
}

try {
    // Perform operation
} finally {
    flock($lock_handle, LOCK_UN);
    fclose($lock_handle);
}
```

## Backup Strategy

### Automatic Backups

**Before Bulk Operations**:

- Creates backup in `v2/data/backups/`
- Filename: `produkt_updates_YYYY-MM-DD_HH-MM-SS.json`
- Preserves original file

### Manual Backups

**Recommended**:

- Backup before major changes
- Backup before migrations
- Regular backups (weekly/monthly)

**Backup Location**: `v2/data/backups/`

## Admin manual QA checklist (images)

Run on staging after deploy or storage changes (especially migration).

1. **Diagnostics:** Open `/v2/api/produkt-updates-diagnostics.php` — confirm `resolved_paths.image_dir.storage_kind` is `ssot`, notes mention SSOT root, no volatile-image warnings.
2. **Upload:** In Produkt-Updates-Admin → feature → Manage image → Upload; file appears under Library; Network response includes `storage_location: "ssot"` (or `legacy_wp` / `volatile_tmp` if expected).
3. **Set featured (library):** Select from library → save; list thumbnail updates; JSON `featured_image` is `/produkt-updates/bilder/…` (or WebP if sidecar exists).
4. **Insert by URL:** Use canonical path or Ordio `https://` URL that resolves on disk; saving an **existing** feature rejects missing files (400). Inline Quill insert allows relative URLs per client rules.
5. **Replace / clear:** Replace featured image; confirm old file remains on disk (by design) until library delete.
6. **Library delete:** Select image in use → delete → confirm modal lists usages; after delete, JSON refs cleared and file removed from writable dirs.
7. **Badges:** Library shows German usage labels (Vorschaubild / Galerie / Fließtext) when applicable; thumbnail can still be empty if image is only inline — expected.
8. **CLI:** `php v2/scripts/dev-helpers/verify-produkt-updates-ssot.php --fail-on-issues` (and migration scripts from overview) on the server.

## Upload security (OWASP-aligned checklist)

Aligned with [OWASP File Upload Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/File_Upload_Cheat_Sheet.html) themes:

| Control | Implementation |
|---------|----------------|
| Authentication | [`produkt-updates-upload.php`](../../../v2/api/produkt-updates-upload.php) requires `produkt_updates_admin_authenticated` session. |
| Size limit | 5 MB per file (adjust if product policy changes). |
| Content typing | `finfo` MIME check + `getimagesize()` + allowed raster types; GD optimization after move. |
| Filename | Generated/sanitized basename; traversal rejected on delete; canonical paths for JSON. |
| Storage location | Writes only under SSOT `{root}/images/` via `findWritableImageDir()`. |
| Delivery | No raw filesystem URL; `produkt-updates-bilder.php` allowlists realpath under SSOT + legacy read roots. |
| SSRF | Insert-by-URL does **not** server-fetch arbitrary URLs; optional future “import URL” would need allowlists and caps. |

## Troubleshooting

### Data File Not Found

**Symptoms**: Admin panel shows empty, public pages show old content

**Causes**:

- Data in `/tmp/` (lost on restart)
- Permission issues
- Path resolution mismatch

**Solutions**:

1. Check storage status in admin panel
2. Run migration script if needed
3. Verify file permissions
4. Check path resolution logic

### Images Not Displaying

**Symptoms**: Images show broken/placeholder on public pages

**Causes**:

- Images in `/tmp/` (lost on restart)
- Path resolution mismatch
- Image proxy endpoint issues

**Solutions**:

1. Check image directory location
2. Run image migration script
3. Verify image proxy endpoint
4. Check `findReadableImage()` includes WordPress uploads

### Permission Errors

**Symptoms**: Cannot save data, upload images fail

**Causes**:

- Directory not writable
- Owner mismatch
- Incorrect permissions

**Solutions**:

1. Use WordPress uploads directory (already writable)
2. Check directory permissions
3. Verify owner is `www-data:www-data`
4. Run discovery script to find writable locations

## Featured Image Persistence

### Overview

The Product Updates system stores `featured_image` fields for each feature entry. These fields must persist across server restarts and data regeneration operations.

### Field Preservation

**Critical**: The `featured_image` field is explicitly preserved during all data operations:

1. **During Regeneration**: The `regenerateMonthsFromContent()` function explicitly preserves `featured_image` fields when rebuilding the months structure
2. **During Migration**: The `migrateImagesToFeaturedImage()` function never overwrites existing `featured_image` fields
3. **During Save**: Pre-save and post-save validation ensures `featured_image` fields are not lost

### Path Formats

Featured images can be stored in two formats:

1. **Proxy URLs**: `/v2/api/serve-produkt-updates-image.php?file=filename.jpg`

   - Used for images served through the proxy endpoint
   - Filename is extracted and resolved to actual file location
   - Preferred format for consistent access

2. **Direct Paths**: `/wp-content/uploads/produkt-updates/image.jpg`
   - Direct filesystem paths
   - Validated against actual file existence

### Validation

The system includes multiple validation layers:

- **Pre-save validation**: Counts `featured_image` fields before regeneration
- **Post-regeneration validation**: Verifies fields persist after structure rebuild
- **Post-save verification**: Confirms fields are saved correctly to JSON file
- **Path resolution**: Validates that image paths resolve to existing files

### Monitoring

Featured image integrity is monitored through:

1. **Health Check Endpoint**: `/v2/api/produkt-updates-storage-health.php`

   - Checks featured_image field counts
   - Validates path resolution
   - Reports missing or invalid paths

2. **Diagnostic Script**: `/v2/api/produkt-updates-image-persistence-check.php`

   - Comprehensive analysis of all featured_image fields
   - Path resolution testing
   - File existence validation

3. **Test Script**: `/v2/admin/produkt-updates/test-image-persistence.php`
   - Simulates server restart scenario
   - Verifies featured_image persistence after regeneration

### Troubleshooting Featured Image Loss

**Symptoms**: Featured images disappear after server restart

**Causes**:

- Data file stored in volatile location (`/tmp/`)
- Regeneration function not preserving fields
- Path resolution failures

**Solutions**:

1. Verify data file is in persistent location (not `/tmp/`)
2. Run diagnostic script to identify issues
3. Check health check endpoint for warnings
4. Review error logs for regeneration warnings

### Best Practices

1. **Always use persistent storage**: Ensure data file is not in `/tmp/`
2. **Monitor health checks**: Regularly check health endpoint for issues
3. **Use direct paths**: Prefer `/wp-content/uploads/produkt-updates/{filename}` for featured images (proxy is fallback only)
4. **Test after changes**: Run test script after code changes affecting regeneration

## Phase 9: Cleanup (After Grace Period)

After 1–2 months of stable direct serving (2026-01 implementation):

1. **Remove proxy references** from health checks, diagnostic scripts, and test scripts
2. **Optionally remove**: `v2/api/serve-produkt-updates-image.php` and its entry in `docs/rule-glob-validation.json`
3. **Monitor**: Check error logs for 404s on image URLs; verify no broken images in production
4. **Run**: `grep -r "serve-produkt-updates-image" .` to find remaining references

## Related Documentation

- [Storage Solution Summary](./STORAGE_SOLUTION_SUMMARY.md) - Quick reference
- [Production Storage Analysis](./PRODUCTION_STORAGE_ANALYSIS.md) - Detailed analysis
- [Diagnostic Tools](./PRODUCT_UPDATES_DIAGNOSTIC_TOOLS.md) - Storage diagnostic tools
- [Troubleshooting Guide](./PRODUCT_UPDATES_TROUBLESHOOTING.md) - Common issues