HH/docs/SLA_SYSTEM_SETUP_AND_TESTING_GUIDE.md

# SLA System Setup and Testing Guide

## Overview

This guide covers the complete SLA (Service Level Agreement) system for complaints, including setup, configuration, testing, and monitoring.

## Table of Contents

1. [System Architecture](#system-architecture)
2. [Model Structure](#model-structure)
3. [SLA Configuration](#sla-configuration)
4. [Escalation Rules](#escalation-rules)
5. [Testing the System](#testing-the-system)
6. [Monitoring and Troubleshooting](#monitoring-and-troubleshooting)
7. [Production Setup](#production-setup)

---

## System Architecture

### Components

The SLA system consists of the following key components:

1. **Complaint Model** - Stores complaint details and SLA tracking
2. **SLA Configuration** - Defines time limits for response/resolution
3. **Escalation Rules** - Defines when and how to escalate complaints
4. **Complaint Updates** - Timeline of all complaint activities
5. **Celery Tasks** - Background jobs for SLA reminders and checks
6. **Signals** - Automatic triggers on complaint creation and updates

### Workflow

```
Complaint Created
    ↓
Calculate Due Date (based on SLA config)
    ↓
Set First/Second Reminder Times
    ↓
Celery Beat Checks Every 15 Minutes
    ↓
Send First Reminder (if enabled and time reached)
    ↓
Send Second Reminder (if enabled and time reached)
    ↓
Escalate (if overdue and escalation rules match)
    ↓
Update Complaint Timeline
```

---

## Model Structure

### Complaint Model

Located in: `apps/complaints/models.py`

Key SLA-related fields:

```python
class Complaint(models.Model):
    # Basic fields
    reference_number = models.CharField(max_length=50, unique=True)
    hospital = models.ForeignKey(Hospital, on_delete=models.PROTECT)
    department = models.ForeignKey(Department, null=True, on_delete=models.PROTECT)
    category = models.ForeignKey(ComplaintCategory, on_delete=models.PROTECT)
    staff = models.ForeignKey(Staff, null=True, on_delete=models.PROTECT)
    
    # Priority and severity
    priority = models.CharField(choices=PRIORITY_CHOICES, default='medium')
    severity = models.CharField(choices=SEVERITY_CHOICES, default='medium')
    
    # Status and SLA tracking
    status = models.CharField(choices=STATUS_CHOICES, default='open')
    
    # SLA deadlines
    due_at = models.DateTimeField(null=True, blank=True)
    
    # SLA reminder tracking
    first_reminder_sent_at = models.DateTimeField(null=True, blank=True)
    second_reminder_sent_at = models.DateTimeField(null=True, blank=True)
    
    # Escalation tracking
    escalation_level = models.IntegerField(default=0)
    escalated_at = models.DateTimeField(null=True, blank=True)
    escalated_to = models.ForeignKey(
        User,
        null=True,
        blank=True,
        on_delete=models.SET_NULL,
        related_name='escalated_complaints'
    )
```

### SLA Configuration Model

```python
class SLAConfig(models.Model):
    hospital = models.ForeignKey(Hospital, on_delete=models.CASCADE)
    department = models.ForeignKey(Department, null=True, on_delete=models.CASCADE)
    category = models.ForeignKey(ComplaintCategory, on_delete=models.CASCADE)
    
    # Time limits (in hours)
    response_time_hours = models.IntegerField()
    resolution_time_hours = models.IntegerField()
    
    # First reminder
    enable_first_reminder = models.BooleanField(default=True)
    first_reminder_hours_before = models.IntegerField(default=24)
    
    # Second reminder
    enable_second_reminder = models.BooleanField(default=True)
    second_reminder_hours_before = models.IntegerField(default=12)
    
    is_active = models.BooleanField(default=True)
```

### Escalation Rule Model

```python
class EscalationRule(models.Model):
    hospital = models.ForeignKey(Hospital, on_delete=models.CASCADE)
    level = models.IntegerField()
    name = models.CharField(max_length=200)
    
    # Trigger conditions
    trigger_type = models.CharField(
        choices=[
            ('overdue', 'Overdue'),
            ('after_reminder', 'After Reminder Sent'),
        ],
        default='overdue'
    )
    hours_overdue = models.IntegerField(null=True, blank=True)
    hours_after_reminder = models.IntegerField(null=True, blank=True)
    
    # Escalation target
    escalate_to = models.CharField(
        choices=[
            ('department_manager', 'Department Manager'),
            ('hospital_admin', 'Hospital Administrator'),
            ('regional_admin', 'Regional Administrator'),
        ],
        default='department_manager'
    )
    
    # Priority/severity filters
    applicable_priorities = models.JSONField(default=list)
    applicable_severities = models.JSONField(default=list)
    
    is_active = models.BooleanField(default=True)
```

### Complaint Update Model

```python
class ComplaintUpdate(models.Model):
    complaint = models.ForeignKey(Complaint, on_delete=models.CASCADE)
    
    update_type = models.CharField(
        choices=[
            ('status_change', 'Status Change'),
            ('comment', 'Comment'),
            ('escalation', 'Escalation'),
            ('reminder_sent', 'Reminder Sent'),
            ('sla_breach', 'SLA Breach'),
        ]
    )
    
    old_status = models.CharField(max_length=50, blank=True)
    new_status = models.CharField(max_length=50, blank=True)
    message = models.TextField()
    created_by = models.ForeignKey(User, null=True, on_delete=models.SET_NULL)
```

---

## SLA Configuration

### Setting Up SLA Configurations

SLA configurations define time limits for responding to and resolving complaints based on:
- Hospital
- Department (optional - applies to all departments if null)
- Complaint Category

### Configuration Steps

1. **Access Admin Panel**
   ```
   http://localhost:8000/admin/complaints/slaconfig/
   ```

2. **Create SLA Configuration**
   - Select Hospital
   - Select Category (required)
   - Select Department (optional)
   - Set Response Time (hours)
   - Set Resolution Time (hours)
   - Configure First Reminder
   - Configure Second Reminder
   - Mark as Active

3. **Example Configuration**

   For critical complaints:
   - Response Time: 4 hours
   - Resolution Time: 24 hours
   - First Reminder: 2 hours before due
   - Second Reminder: 1 hour before due

   For medium complaints:
   - Response Time: 24 hours
   - Resolution Time: 72 hours
   - First Reminder: 24 hours before due
   - Second Reminder: 12 hours before due

### Default Configurations

The system includes default SLA configurations for all complaint categories:

| Category | Response Time | Resolution Time | First Reminder | Second Reminder |
|----------|--------------|-----------------|---------------|-----------------|
| Clinical Care | 24 hours | 72 hours | 24 hours | 12 hours |
| Staff Behavior | 12 hours | 48 hours | 12 hours | 6 hours |
| Facility | 24 hours | 72 hours | 24 hours | 12 hours |
| Wait Time | 4 hours | 24 hours | 4 hours | 2 hours |
| Billing | 24 hours | 72 hours | 24 hours | 12 hours |
| Communication | 12 hours | 48 hours | 12 hours | 6 hours |
| Other | 24 hours | 72 hours | 24 hours | 12 hours |

---

## Escalation Rules

### Understanding Escalation

Escalation automatically assigns complaints to higher-level staff when:
1. Complaint becomes overdue
2. Specified time after reminder has passed

### Setting Up Escalation Rules

1. **Access Admin Panel**
   ```
   http://localhost:8000/admin/complaints/escalationrule/
   ```

2. **Create Escalation Rule**
   - Select Hospital
   - Set Escalation Level (1, 2, 3, etc.)
   - Name the rule
   - Choose Trigger Type (Overdue or After Reminder)
   - Set Hours parameter
   - Select Escalation Target
   - Filter by Priority/Severity (optional)
   - Mark as Active

### Example Escalation Rules

**Level 1: Department Manager**
- Trigger: Overdue
- Hours overdue: 0 (immediately when due time passes)
- Escalate to: Department Manager

**Level 2: Hospital Admin**
- Trigger: After Reminder
- Hours after reminder: 12
- Escalate to: Hospital Administrator
- Only for: Critical and High priority

**Level 3: Regional Admin**
- Trigger: Overdue
- Hours overdue: 48
- Escalate to: Regional Administrator
- Only for: Critical severity

---

## Testing the System

### Automated Testing

Run the automated SLA test suite:

```bash
python test_sla_functionality.py
```

This test verifies:
- ✓ SLA configuration retrieval
- ✓ Due date calculation
- ✓ First reminder logic
- ✓ Second reminder logic
- ✓ Escalation eligibility
- ✓ Timeline tracking

### Manual Testing

#### Test 1: Complaint Creation and SLA Calculation

1. Create a complaint via admin panel or API
2. Verify:
   - `due_at` field is populated
   - `first_reminder_sent_at` and `second_reminder_sent_at` are null
   - `escalation_level` is 0
   - Timeline entry shows "Complaint created and registered"

```bash
# View complaint details
python manage.py shell -c "
from apps.complaints.models import Complaint
c = Complaint.objects.first()
print(f'ID: {c.id}')
print(f'Title: {c.title}')
print(f'Status: {c.status}')
print(f'Due At: {c.due_at}')
print(f'Escalation Level: {c.escalation_level}')
print(f'First Reminder Sent: {c.first_reminder_sent_at}')
print(f'Second Reminder Sent: {c.second_reminder_sent_at}')
"
```

#### Test 2: First Reminder

1. Create a complaint with SLA config
2. Wait for first reminder time or modify `due_at` to trigger
3. Check complaint for:
   - `first_reminder_sent_at` is populated
   - Timeline entry shows "First SLA reminder sent"

```bash
# Check reminder status
python manage.py shell -c "
from apps.complaints.models import Complaint
from django.utils import timezone
import logging

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

complaint = Complaint.objects.first()
if complaint:
    from apps.complaints.tasks import check_and_send_sla_reminders
    result = check_and_send_sla_reminders()
    print(f'Result: {result}')
    
    # Refresh complaint
    complaint.refresh_from_db()
    print(f'First Reminder Sent At: {complaint.first_reminder_sent_at}')
    print(f'Second Reminder Sent At: {complaint.second_reminder_sent_at}')
"
```

#### Test 3: Second Reminder

1. Ensure complaint has first reminder sent
2. Wait for second reminder time or modify timestamps
3. Verify:
   - `second_reminder_sent_at` is populated
   - Timeline entry shows "Second SLA reminder sent"

#### Test 4: Escalation

1. Create overdue complaint (set `due_at` in the past)
2. Ensure escalation rules exist
3. Run escalation check:

```bash
python manage.py shell -c "
from apps.complaints.models import Complaint
from django.utils import timezone
import logging

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

complaint = Complaint.objects.first()
if complaint:
    # Make it overdue
    complaint.due_at = timezone.now() - timezone.timedelta(hours=2)
    complaint.save()
    
    from apps.complaints.tasks import check_and_escalate_overdue_complaints
    result = check_and_escalate_overdue_complaints()
    print(f'Result: {result}')
    
    # Refresh complaint
    complaint.refresh_from_db()
    print(f'Escalation Level: {complaint.escalation_level}')
    print(f'Escalated To: {complaint.escalated_to}')
    print(f'Escalated At: {complaint.escalated_at}')
"
```

#### Test 5: Seeding Test Data

Generate test complaints for SLA testing:

```bash
# Seed 10 complaints (7 Arabic, 3 English)
python manage.py seed_complaints --count 10

# Seed 50 complaints with different distribution
python manage.py seed_complaints --count 50 --arabic-percent 50 --staff-mention-percent 70

# Dry run to preview
python manage.py seed_complaints --count 20 --dry-run

# Clear existing and seed fresh
python manage.py seed_complaints --count 10 --clear
```

---

## Monitoring and Troubleshooting

### Checking SLA Status

View all complaints with SLA information:

```bash
python manage.py shell -c "
from apps.complaints.models import Complaint
from django.utils import timezone
from django.db.models import Q

now = timezone.now()

# All open complaints
open_complaints = Complaint.objects.filter(status='open')
print(f'Total Open Complaints: {open_complaints.count()}')

# Overdue complaints
overdue = open_complaints.filter(due_at__lt=now)
print(f'Overdue: {overdue.count()}')

# Due within 24 hours
soon = open_complaints.filter(due_at__gte=now, due_at__lte=now + timezone.timedelta(hours=24))
print(f'Due within 24h: {soon.count()}')

# First reminder sent but not second
first_only = open_complaints.filter(first_reminder_sent_at__isnull=False, second_reminder_sent_at__isnull=True)
print(f'First reminder only: {first_only.count()}')

# Escalated complaints
escalated = open_complaints.filter(escalation_level__gt=0)
print(f'Escalated: {escalated.count()}')
"
```

### Celery Task Monitoring

Check if Celery Beat is running:

```bash
# Check Celery status
celery -A config inspect active

# View scheduled tasks
celery -A config inspect scheduled

# View registered tasks
celery -A config inspect registered
```

### Email Configuration

Test email sending:

```bash
python test_email_sending.py
```

Ensure email settings in `.env`:
```env
EMAIL_BACKEND=django.core.mail.backends.smtp.EmailBackend
EMAIL_HOST=smtp.gmail.com
EMAIL_PORT=587
EMAIL_USE_TLS=True
EMAIL_HOST_USER=your-email@gmail.com
EMAIL_HOST_PASSWORD=your-app-password
DEFAULT_FROM_EMAIL=your-email@gmail.com
```

### Common Issues

**Issue 1: Reminders not sending**

- Check Celery Beat is running
- Verify Celery logs: `tail -f logs/celery.log`
- Check email configuration
- Verify SLA config has reminders enabled

**Issue 2: Escalation not working**

- Verify escalation rules exist and are active
- Check complaint matches rule criteria (priority/severity)
- Ensure complaint is actually overdue
- Check Celery logs for errors

**Issue 3: Wrong due dates**

- Verify SLA configuration exists for complaint's category/hospital
- Check timezone settings in Django
- Review signal handlers in `apps/complaints/signals.py`

---

## Production Setup

### 1. Configure Celery Beat

Ensure Celery Beat is configured in `config/celery.py`:

```python
from celery.schedules import crontab

app.conf.beat_schedule = {
    'check-sla-reminders-every-15-minutes': {
        'task': 'apps.complaints.tasks.check_and_send_sla_reminders',
        'schedule': crontab(minute='*/15'),
    },
    'check-overdue-complaints-every-30-minutes': {
        'task': 'apps.complaints.tasks.check_and_escalate_overdue_complaints',
        'schedule': crontab(minute='*/30'),
    },
}
```

### 2. Run Celery Workers

```bash
# Start Celery worker
celery -A config worker -l INFO

# Start Celery Beat scheduler
celery -A config beat -l INFO --scheduler django_celery_beat.schedulers:DatabaseScheduler
```

### 3. Configure Production Email Settings

Use production email service (e.g., SendGrid, AWS SES):

```python
EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend'
EMAIL_HOST = 'smtp.sendgrid.net'
EMAIL_PORT = 587
EMAIL_USE_TLS = True
EMAIL_HOST_USER = 'apikey'
EMAIL_HOST_PASSWORD = os.getenv('SENDGRID_API_KEY')
DEFAULT_FROM_EMAIL = 'noreply@alhammadi.com.sa'
```

### 4. Set Up Monitoring

- Monitor Celery task queue size
- Track SLA breach rate
- Alert on high escalation rates
- Monitor email delivery success rate

### 5. Database Indexes

Ensure database has proper indexes for SLA queries:

```python
class Complaint(models.Model):
    class Meta:
        indexes = [
            models.Index(fields=['status', 'due_at']),
            models.Index(fields=['status', 'created_at']),
            models.Index(fields=['hospital', 'status']),
            models.Index(fields=['escalation_level']),
        ]
```

### 6. Backup Strategy

- Regular database backups
- Backup SLA configurations
- Archive old complaints
- Keep escalation rule history

---

## Summary

The SLA system provides:

✓ **Automated Time Tracking** - Automatically calculates and tracks due dates
✓ **Double Reminder System** - First and second reminders configurable per SLA
✓ **Smart Escalation** - Automatic escalation based on rules
✓ **Complete Audit Trail** - Full timeline of all activities
✓ **Multi-level Configuration** - Different SLAs by hospital, department, and category
✓ **Bilingual Support** - Full Arabic/English support for emails and notifications
✓ **Flexible Rules** - Configure different priorities and severities
✓ **Background Processing** - Non-blocking Celery tasks
✓ **Production Ready** - Tested and ready for deployment

---

## Next Steps

1. **Configure SLA Settings** - Set appropriate time limits for your organization
2. **Test with Real Data** - Use production-like data for testing
3. **Monitor Initial Period** - Closely monitor SLA performance for first week
4. **Adjust Rules** - Fine-tune escalation rules based on observed patterns
5. **Train Staff** - Educate staff on SLA system and their responsibilities
6. **Set Up Dashboards** - Create monitoring dashboards for SLA metrics

---

## Additional Resources

- **SLA Testing Guide**: `docs/SLA_TESTING_GUIDE.md`
- **SLA System Overview**: `docs/SLA_SYSTEM_OVERVIEW.md`
- **SLA Testing Plan**: `docs/SLA_TESTING_PLAN.md`
- **Complaint Seeding Guide**: `docs/COMPLAINT_SEEDING_GUIDE.md`
- **Email Templates**: `templates/complaints/emails/`

---

*Last Updated: January 14, 2026*