# Slack Integration Troubleshooting Guide

**Last Updated:** 2026-03-06

Comprehensive troubleshooting guide for Ordio Loop Slack integration issues.

## Quick Diagnostics

### Run Diagnostic Script

The fastest way to identify issues:

```bash
php v2/scripts/affiliate/diagnose-slack-integration.php --verbose
```

This script checks:
- cURL extension availability
- Configuration status
- Webhook URL format
- Payload generation
- Webhook connectivity
- Error log entries

### Test Webhook

Send a test message to verify connectivity:

```bash
# Basic test
php v2/scripts/affiliate/test-slack-webhook.php

# Verbose output
php v2/scripts/affiliate/test-slack-webhook.php --verbose

# Test all notification types
php v2/scripts/affiliate/test-slack-webhook.php --all
```

## Production Debugging

### Step-by-Step Production Diagnosis

When notifications aren't working in production, follow these steps:

1. **Run Comprehensive Diagnostics**
   ```bash
   php v2/scripts/affiliate/diagnose-slack-integration.php --verbose
   ```
   This will show:
   - Configuration status
   - Webhook URL validity
   - Connectivity test results
   - Recent error log entries

2. **Check PHP Error Logs**
   ```bash
   # Search for Slack-related errors
   grep "[Affiliate Slack]" /var/log/php-fpm/error.log | tail -20
   # or Apache
   grep "[Affiliate Slack]" /var/log/apache2/error.log | tail -20
   ```
   Look for:
   - "Notification skipped" messages with reasons
   - "Invalid webhook URL format" errors
   - "cURL extension not available" errors
   - Correlation IDs for tracking specific failures

3. **Check Dedicated Log File**
   ```bash
   # View recent entries
   tail -n 50 v2/logs/affiliate-slack.log
   
   # Monitor in real-time
   tail -f v2/logs/affiliate-slack.log
   
   # Search for failures
   grep -i "failed\|error\|skipped" v2/logs/affiliate-slack.log | tail -20
   ```

4. **Verify Configuration in Production**
   ```bash
   # Check if constants are defined
   php -r "require_once 'v2/config/slack-config.php'; echo 'ENABLED: ' . (SLACK_LOOP_UPDATES_ENABLED ? 'true' : 'false') . PHP_EOL; echo 'URL configured: ' . (defined('SLACK_LOOP_UPDATES_WEBHOOK_URL') && SLACK_LOOP_UPDATES_WEBHOOK_URL !== '' ? 'yes' : 'no') . PHP_EOL;"
   ```

5. **Test Webhook Manually**
   ```bash
   # Test basic connectivity
   php v2/scripts/affiliate/test-slack-webhook.php --verbose
   
   # Test partner registration notification
   php v2/scripts/affiliate/test-partner-registration-slack.php --type=email
   ```

6. **Check Environment Variables**
   ```bash
   # If using environment variables, verify they're set
   env | grep SLACK
   ```

7. **Verify File Permissions**
   ```bash
   # Check log directory is writable
   ls -la v2/logs/
   test -w v2/logs && echo "Writable" || echo "NOT writable"
   ```

### Common Production Issues

**Issue: Configuration not loaded**
- **Symptom**: `SLACK_LOOP_UPDATES_ENABLED` constant not defined
- **Solution**: Ensure `v2/config/slack-config.php` is included before calling `sendLoopUpdateToSlack()`
- **Check**: Verify `require_once` statements in registration endpoints

**Issue: Environment variable override**
- **Symptom**: Webhook URL empty despite config file having value
- **Solution**: Check if `SLACK_LOOP_UPDATES_WEBHOOK_URL` environment variable is set to empty string
- **Check**: Run diagnostic script to see actual configuration values

**Issue: Webhook URL expired**
- **Symptom**: HTTP 404 or 410 errors in logs
- **Solution**: Regenerate webhook in Slack API dashboard
- **Update**: Set new URL in environment variable or `slack-config.local.php`

**Issue: Log file not created**
- **Symptom**: No entries in `v2/logs/affiliate-slack.log`
- **Solution**: Check directory permissions, ensure `v2/logs/` exists and is writable
- **Check**: Errors will still appear in PHP error log even if dedicated log fails

## Common Issues

### No Messages in #loop-updates

**Symptoms:**
- Partner registrations occur but no Slack notifications appear
- Test script fails or shows errors

**Diagnosis Steps:**

1. **Check Configuration**
   ```bash
   php v2/scripts/affiliate/diagnose-slack-integration.php
   ```
   Look for:
   - `SLACK_LOOP_UPDATES_ENABLED` is `true`
   - Webhook URL is configured and valid format

2. **Check Error Logs**
   ```bash
   # Check dedicated log file
   tail -f v2/logs/affiliate-slack.log
   
   # Check PHP error log
   grep "[Affiliate Slack]" /var/log/php-fpm/error.log
   # or
   grep "[Affiliate Slack]" /var/log/apache2/error.log
   ```

3. **Test Webhook Connectivity**
   ```bash
   php v2/scripts/affiliate/test-slack-webhook.php --verbose
   ```

**Common Causes:**

- **Webhook URL expired or invalid**
  - Solution: Regenerate webhook in Slack API dashboard
  - Update URL in environment variable or `slack-config.local.php`

- **Configuration disabled**
  - Solution: Set `SLACK_LOOP_UPDATES_ENABLED=true` in environment or config

- **cURL errors (network, SSL, timeout)**
  - Solution: Check network connectivity, SSL certificates, firewall rules
  - Review error logs for specific cURL error codes

- **Invalid webhook URL format**
  - Solution: Ensure URL starts with `https://hooks.slack.com/services/`
  - Format: `https://hooks.slack.com/services/T.../B.../...`

### cURL Errors

**Common cURL Error Codes:**

- **CURLE_COULDNT_CONNECT (7)**: Cannot connect to Slack servers
  - Check firewall rules
  - Verify network connectivity
  - Check DNS resolution

- **CURLE_COULDNT_RESOLVE_HOST (6)**: DNS resolution failed
  - Check DNS configuration
  - Verify `hooks.slack.com` resolves correctly

- **CURLE_OPERATION_TIMEOUTED (28)**: Request timeout
  - Check network latency
  - Increase timeout if needed (default: 10s)

- **CURLE_SSL_CONNECT_ERROR (35)**: SSL handshake failed
  - Check SSL certificate configuration
  - Verify system time is correct
  - Update CA certificates if needed

**Solutions:**

1. Check cURL version and SSL support:
   ```bash
   php -r "print_r(curl_version());"
   ```

2. Test connectivity manually:
   ```bash
   curl -v https://hooks.slack.com/services/T.../B.../...
   ```

3. Review error logs for specific error codes and messages

### HTTP Errors

**HTTP 400 Bad Request:**
- Invalid payload format
- Missing required fields
- Check payload validation in logs

**HTTP 401 Unauthorized:**
- Webhook URL is invalid or expired
- Regenerate webhook in Slack API dashboard

**HTTP 404 Not Found:**
- Webhook URL is incorrect
- Verify URL matches Slack API dashboard

**HTTP 429 Too Many Requests:**
- Rate limit exceeded
- Retry logic handles this automatically (3 attempts with backoff)
- Check if too many notifications are being sent

**HTTP 500/502/503 Server Errors:**
- Slack API temporary issues
- Retry logic handles this automatically
- Check Slack status page if persistent

### Payload Validation Errors

**Symptoms:**
- Error log shows "Invalid payload" or "missing fields"
- Specific notification types fail

**Diagnosis:**

Check error logs for missing fields:
```
[Affiliate Slack] Invalid payload for type registration: missing fields email, name
```

**Solutions:**

1. Verify payload structure matches notification type requirements
2. Check that all required fields are present in payload
3. Review `validateSlackPayload()` function for required fields per type

**Required Fields by Type:**

- `registration` / `registration_oauth`: `partner_id`, `name`, `email`
- `lead_attributed`: `partner_id`, `email`
- `partner_level_up`: `partner_id`, `name`, `old_level`, `new_level`
- `deal_closed_won`: `partner_id`, `deal_name`, `deal_mrr`, `partner_share`
- `lead_qualified`: `partner_id`, `email`
- `deal_status_change`: `partner_id`, `deal_name`, `old_status`, `new_status`
- `mrr_milestone`: `partner_id`, `milestone`, `current_mrr`

### Configuration Issues

**Webhook URL Not Configured:**

1. Check environment variable:
   ```bash
   echo $SLACK_LOOP_UPDATES_WEBHOOK_URL
   ```

2. Check local config file:
   ```bash
   cat v2/config/slack-config.local.php
   ```

3. Check fallback in `slack-config.php` (if no env/local config)

**Integration Disabled:**

1. Check environment variable:
   ```bash
   echo $SLACK_LOOP_UPDATES_ENABLED
   ```

2. Verify webhook URL format (must start with `https://hooks.slack.com/services/`)

3. Check `SLACK_LOOP_UPDATES_ENABLED` constant value

## Diagnostic Commands

### Check Notification Status (Admin Only)

```bash
# Via API endpoint (requires admin authentication)
curl -b cookies.txt https://www.ordio.com/v2/api/slack-notification-status.php

# With custom time period (hours)
curl -b cookies.txt "https://www.ordio.com/v2/api/slack-notification-status.php?hours=48"
```

### Monitor Log File in Real-Time

```bash
tail -f v2/logs/affiliate-slack.log
```

### Search Error Logs

```bash
# Find all Slack-related errors
grep -i "affiliate slack" /var/log/php-fpm/error.log | tail -20

# Find recent failures
grep -i "failed\|error" v2/logs/affiliate-slack.log | tail -20

# Find specific correlation ID
grep "SLACK-20260306-123456" v2/logs/affiliate-slack.log
```

### Check Notification Statistics

```php
<?php
require_once 'v2/helpers/affiliate-slack-monitor.php';

// Get stats for last 24 hours
$stats = getSlackNotificationStats(24);
print_r($stats);

// Get summary
$summary = getSlackNotificationSummary(24);
print_r($summary);
```

## Advanced Troubleshooting

### Enable Verbose Logging

The diagnostic script includes verbose mode:

```bash
php v2/scripts/affiliate/diagnose-slack-integration.php --verbose
```

This shows:
- Detailed configuration values (sanitized)
- Full error messages
- Recent log entries
- cURL version and SSL info

### Test Specific Notification Types

```bash
# Test registration notification
php -r "
require 'v2/config/slack-config.php';
require 'v2/helpers/affiliate-slack.php';
sendLoopUpdateToSlack('registration', [
    'partner_id' => 'AP-TEST-123456',
    'name' => 'Test Partner',
    'email' => 'test@example.com'
]);
"
```

### Manual Webhook Test

Test webhook directly with curl:

```bash
curl -X POST https://hooks.slack.com/services/T.../B.../... \
  -H 'Content-Type: application/json' \
  -d '{
    "text": "Test message",
    "blocks": [
      {
        "type": "section",
        "text": {
          "type": "mrkdwn",
          "text": "Manual test from command line"
        }
      }
    ]
  }'
```

### Check Retry Logic

The integration includes automatic retry logic (3 attempts with exponential backoff). Check logs for retry attempts:

```bash
grep "attempt" v2/logs/affiliate-slack.log | tail -20
```

## Monitoring & Alerts

### Health Check

Run periodic health checks:

```bash
php -r "
require 'v2/helpers/affiliate-slack-monitor.php';
checkSlackNotificationHealth(5, 50.0, 24);
"
```

This checks:
- Minimum 5 failures in last 24 hours, OR
- Success rate below 50%

If threshold exceeded, sends email alert to `hady@ordio.com`.

### Set Up Cron Job for Health Checks

Add to crontab:

```bash
# Check Slack notification health every 6 hours
0 */6 * * * php /path/to/v2/helpers/affiliate-slack-monitor.php -c checkSlackNotificationHealth
```

## Escalation Procedures

### Level 1: Basic Diagnostics

1. Run diagnostic script
2. Check error logs
3. Test webhook connectivity
4. Verify configuration

### Level 2: Advanced Troubleshooting

1. Review detailed error logs
2. Test specific notification types
3. Check network connectivity
4. Verify SSL certificates

### Level 3: Slack API Issues

1. Check Slack status page: https://status.slack.com
2. Verify webhook URL in Slack API dashboard
3. Regenerate webhook if needed
4. Contact Slack support if persistent

## Prevention

### Regular Maintenance

1. **Weekly**: Review notification statistics
2. **Monthly**: Check error logs for patterns
3. **Quarterly**: Verify webhook URL is still valid
4. **Annually**: Rotate webhook URL for security

### Monitoring Setup

1. Set up health check cron job
2. Monitor notification success rate
3. Set up alerts for repeated failures
4. Review logs regularly

## Related Documentation

- [SLACK_LOOP_UPDATES.md](SLACK_LOOP_UPDATES.md) - Integration overview
- [SLACK_MONITORING.md](SLACK_MONITORING.md) - Monitoring guide
- [DEPLOYMENT_CHECKLIST.md](DEPLOYMENT_CHECKLIST.md) - Deployment steps