# Fix: Re-import Complaints with Missing Data

## Overview
Full re-import of all complaints (2022-2025) to fix missing timeline dates, satisfaction, resolution_outcome, and other data.

## Steps

### 1. Delete all existing complaints
```python
Complaint.objects.all().delete()
```

### 2. Fix `import_historical_complaints.py` (2022-2024)

**2a. Fix COLUMN_MAPPING — add missing columns:**
```python
COLUMN_MAPPING = {
    # ... existing mappings ...
    "form_sent_date": 12,          # إرسال نموذج الشكوى → form_sent_at
    "activated_date": 17,          # تفعيل الشكوى → activated_at
    # col 20 (date_sent) stays → forwarded_to_dept_at (was wrongly mapped to activated_at)
    "escalation_reason": 35,       # Reason of Escalation → metadata
    "recommendation": 58,          # Recommendation/Action plan → recommendation_action_plan
}
```

**2b. Fix Complaint.objects.create() — add fields:**
- `form_sent_at=form_sent_date` (col 12)
- `activated_at=activated_date` (col 17, NOT col 20)
- `forwarded_to_dept_at=date_sent` (col 20, was wrongly in activated_at)
- `satisfaction=normalize_satisfaction(satisfaction_val)` (col 56, write to model field NOT just metadata)
- `recommendation_action_plan=recommendation` (col 58)

**2c. Add satisfaction normalization helper:**
```python
def _normalize_satisfaction(self, val):
    val = str(val or "").strip().lower()
    mapping = {"satisfied": "satisfied", "dissatisfied": "dissatisfied", "no response": "no_response"}
    return mapping.get(val, "")
```

### 3. Fix `import_2025_complaints_basic.py`

**3a. Fix HEADER_ALIASES — fix satisfaction and add missing columns:**
```python
HEADER_ALIASES = {
    # ... existing correct mappings ...
    "satisfaction": ["Satisfied/Dissatisfied"],  # FIXED: was pointing to wrong column
    "rightful_side": ["The Rightful Side"],      # NEW: col 66
    "complaint_subject": ["موضوع الشكوى الأساسية"],  # NEW: col 53
    "form_sent_date": ["إرسال نموذج الشكوى"],    # NEW: col 16
    "activated_date": ["تفعيل الشكوى"],          # NEW: col 21
    "sent_date": ["تم ارسال الشكوى"],            # NEW: col 24
    "first_reminder": ["First Reminder Sent"],    # NEW: col 28
    "second_reminder": ["Second Reminder Sent"],  # NEW: col 32
    "escalated_date": ["Escalated"],              # NEW: col 36
    "closed_date": ["Closed"],                    # NEW: col 40
    "resolved_date": ["Resolved"],                # NEW: col 44
    "delay_reason": ["سبب تأخير القسم بالرد"],    # NEW: col 58
    "closure_delay": ["سبب تأخير اغلاق الشكوى خلال 72 ساعه"],  # col 59
    "action_taken": ["الاجراء المتخذ من قبل القسم المعني"],      # col 62
    "action_result": ["نتيجة الاجراء المتخذ بعد التحقيق"],       # col 63
    "recommendation": ["Recommendation/Action plan"],             # col 64
    "solutions": ["حلول واقتراحات"],                              # col 67
}
```

**3b. Fix _process_sheet() — parse new timeline dates:**
Add parsing for all new date fields (form_sent_date, activated_date, sent_date, first_reminder, second_reminder, escalated_date, closed_date, resolved_date).

**3c. Fix status determination:**
```python
# Current: only checks response_date → open/resolved
# Fixed: check all dates like historical script
if closed_date:
    status = "closed"
elif resolved_date:
    status = "resolved"
elif escalated_date:
    status = "in_progress"
else:
    status = "open"
```

**3d. Fix Complaint.objects.create() — add all missing fields:**
- `activated_at=activated_date`
- `form_sent_at=form_sent_date`
- `forwarded_to_dept_at=sent_date`
- `reminder_sent_at=first_reminder`
- `second_reminder_sent_at=second_reminder`
- `escalated_at=escalated_date`
- `closed_at=closed_date`
- `resolved_at=resolved_date`
- `satisfaction=normalize_satisfaction(satisfaction_val)`
- `resolution_outcome=normalize_rightful_side(rightful_side)`
- `complaint_subject=complaint_subject`
- `explanation_delay_reason=delay_reason`
- `delay_reason_closure=closure_delay`
- `action_taken_by_dept=action_taken`
- `action_result=action_result`
- `recommendation_action_plan=recommendation or solutions`

### 4. Run re-import
```bash
python manage.py shell -c "from apps.complaints.models import Complaint; Complaint.objects.all().delete()"
python manage.py import_all_complaints --hospital-code=HH-N
python manage.py backfill_sent_to_department
```

### 5. Verify
- Check complaint counts match
- Check timeline dates populated
- Check satisfaction field populated
- Check resolution_outcome populated
- Check department detail page shows correct data