# Progress Tracking System

**Last Updated:** 2025-11-20

## Overview

The progress tracking system provides real-time progress updates for template generation, especially useful for long-running operations with large templates.

## Features

- **Real-Time Updates**: Track progress percentage and current stage
- **Stage Tracking**: Monitor progress through different generation stages
- **Metrics Collection**: Track cells processed, memory usage, generation time
- **Time Estimation**: Estimate remaining time based on current progress
- **Async Support**: Support for asynchronous generation with progress polling
- **Error Tracking**: Track failures and error details

## Usage

### Basic Progress Tracking

```php
require_once __DIR__ . '/helpers/template-progress.php';

// Initialize progress tracker
$progress = new OrdioTemplateProgress();
$progressId = $progress->getProgressId();

// Update progress
$progress->update(25, 'generating', [
    'metrics' => ['cells_processed' => 100]
]);

// Complete
$progress->complete([
    'filename' => 'template.xlsx',
    'file_size' => 1024000
]);
```

### API Integration

#### Start Generation with Progress Tracking

```
POST /v2/systems/excel-template-generator/api/generate.php?track_progress=true
```

Response:

```json
{
  "success": true,
  "progress_id": "progress_abc123...",
  "message": "Generation started",
  "progress_url": "/v2/systems/excel-template-generator/api/progress.php?id=progress_abc123..."
}
```

#### Poll Progress

```
GET /v2/systems/excel-template-generator/api/progress.php?id=progress_abc123...
```

Response:

```json
{
  "success": true,
  "progress": {
    "id": "progress_abc123...",
    "status": "processing",
    "percentage": 45,
    "stage": "saving",
    "started_at": "2025-11-20 19:00:00",
    "updated_at": "2025-11-20 19:00:05",
    "elapsed_seconds": 5,
    "estimated_remaining_seconds": 6,
    "metrics": {
      "cells_processed": 500,
      "sheets_processed": 2,
      "memory_usage": "50 MB",
      "generation_time": 5.2
    },
    "stages": {
      "generating": {
        "started_at": "2025-11-20 19:00:00",
        "updated_at": "2025-11-20 19:00:03",
        "percentage": 50
      },
      "saving": {
        "started_at": "2025-11-20 19:00:03",
        "updated_at": "2025-11-20 19:00:05",
        "percentage": 45
      }
    }
  }
}
```

#### Completed Progress

```json
{
  "success": true,
  "progress": {
    "id": "progress_abc123...",
    "status": "completed",
    "percentage": 100,
    "stage": "completed",
    "completed_at": "2025-11-20 19:00:10",
    "result": {
      "filename": "template.xlsx",
      "file_size": 1024000,
      "generation_time": 10.5,
      "memory_peak": "75 MB"
    }
  }
}
```

## Progress Stages

### Standard Stages

1. **initializing** (0-10%): Setting up template generator
2. **generating** (10-80%): Generating sheets and cells
3. **saving** (80-95%): Writing to file
4. **completed** (100%): Generation complete

### Custom Stages

You can define custom stages based on your template structure:

```php
$progress->update(30, 'processing_sheet_1');
$progress->update(60, 'processing_sheet_2');
$progress->update(90, 'applying_styles');
```

## Metrics

### Tracked Metrics

- **cells_processed**: Number of cells generated
- **sheets_processed**: Number of sheets processed
- **memory_usage**: Current memory usage
- **generation_time**: Time elapsed since start

### Custom Metrics

Add custom metrics during progress updates:

```php
$progress->update(50, 'generating', [
    'metrics' => [
        'formulas_applied' => 25,
        'styles_applied' => 100,
        'validation_rules' => 10
    ]
]);
```

## Time Estimation

The progress API automatically calculates:

- **Elapsed Time**: Time since generation started
- **Estimated Remaining**: Estimated time to completion

Based on current progress percentage and elapsed time.

## Error Handling

### Failed Progress

```json
{
  "success": true,
  "progress": {
    "id": "progress_abc123...",
    "status": "failed",
    "stage": "failed",
    "failed_at": "2025-11-20 19:00:05",
    "error": "Memory limit exceeded",
    "error_data": {
      "memory_limit": "128M",
      "peak_memory": "150M"
    }
  }
}
```

## Frontend Integration

### JavaScript Example

```javascript
async function generateTemplateWithProgress(templateDefinition) {
  // Start generation
  const startResponse = await fetch(
    "/v2/systems/excel-template-generator/api/generate.php?track_progress=true&async=true",
    {
      method: "POST",
      body: JSON.stringify(templateDefinition),
      headers: { "Content-Type": "application/json" },
    }
  );

  const { progress_id, progress_url } = await startResponse.json();

  // Poll progress
  const pollInterval = setInterval(async () => {
    const progressResponse = await fetch(progress_url);
    const { progress } = await progressResponse.json();

    // Update UI
    updateProgressBar(progress.percentage);
    updateStatus(progress.stage);

    if (progress.status === "completed") {
      clearInterval(pollInterval);
      handleCompletion(progress.result);
    } else if (progress.status === "failed") {
      clearInterval(pollInterval);
      handleError(progress.error);
    }
  }, 1000); // Poll every second
}
```

## Cleanup

### Automatic Cleanup

Progress files are automatically cleaned up after 24 hours. You can manually clean old progress files:

```php
OrdioTemplateProgress::cleanOldProgress(86400); // 24 hours
```

### Cron Job Setup

```bash
# Clean old progress files daily at 3 AM
0 3 * * * php /path/to/v2/scripts/clean-progress.php
```

## Best Practices

1. **Poll Frequency**: Poll every 1-2 seconds for responsive updates
2. **Progress Granularity**: Update progress at meaningful milestones
3. **Error Handling**: Always check for failed status
4. **Cleanup**: Regularly clean old progress files
5. **Security**: Validate progress IDs to prevent unauthorized access

## Limitations

- **File-Based Storage**: Progress is stored in JSON files (not suitable for high-concurrency)
- **No Real-Time Push**: Requires polling (consider WebSockets for real-time updates)
- **Single Server**: Progress files are server-specific (not suitable for load-balanced setups)

## Future Enhancements

- Database-backed progress storage
- WebSocket support for real-time updates
- Redis-based progress tracking for distributed systems
- Progress aggregation and analytics