HH/.opencode/plans/fix-complaint-reimport.md

Fix: Re-import Complaints with Missing Data

Overview

Full re-import of all complaints (2022-2025) to fix missing timeline dates, satisfaction, resolution_outcome, and other data.

Steps

1. Delete all existing complaints

Complaint.objects.all().delete()

2. Fix import_historical_complaints.py (2022-2024)

2a. Fix COLUMN_MAPPING — add missing columns:

COLUMN_MAPPING = {
    # ... existing mappings ...
    "form_sent_date": 12,          # إرسال نموذج الشكوى → form_sent_at
    "activated_date": 17,          # تفعيل الشكوى → activated_at
    # col 20 (date_sent) stays → forwarded_to_dept_at (was wrongly mapped to activated_at)
    "escalation_reason": 35,       # Reason of Escalation → metadata
    "recommendation": 58,          # Recommendation/Action plan → recommendation_action_plan
}

2b. Fix Complaint.objects.create() — add fields:

• form_sent_at=form_sent_date (col 12)
• activated_at=activated_date (col 17, NOT col 20)
• forwarded_to_dept_at=date_sent (col 20, was wrongly in activated_at)
• satisfaction=normalize_satisfaction(satisfaction_val) (col 56, write to model field NOT just metadata)
• recommendation_action_plan=recommendation (col 58)

2c. Add satisfaction normalization helper:

def _normalize_satisfaction(self, val):
    val = str(val or "").strip().lower()
    mapping = {"satisfied": "satisfied", "dissatisfied": "dissatisfied", "no response": "no_response"}
    return mapping.get(val, "")

3. Fix import_2025_complaints_basic.py

3a. Fix HEADER_ALIASES — fix satisfaction and add missing columns:

HEADER_ALIASES = {
    # ... existing correct mappings ...
    "satisfaction": ["Satisfied/Dissatisfied"],  # FIXED: was pointing to wrong column
    "rightful_side": ["The Rightful Side"],      # NEW: col 66
    "complaint_subject": ["موضوع الشكوى الأساسية"],  # NEW: col 53
    "form_sent_date": ["إرسال نموذج الشكوى"],    # NEW: col 16
    "activated_date": ["تفعيل الشكوى"],          # NEW: col 21
    "sent_date": ["تم ارسال الشكوى"],            # NEW: col 24
    "first_reminder": ["First Reminder Sent"],    # NEW: col 28
    "second_reminder": ["Second Reminder Sent"],  # NEW: col 32
    "escalated_date": ["Escalated"],              # NEW: col 36
    "closed_date": ["Closed"],                    # NEW: col 40
    "resolved_date": ["Resolved"],                # NEW: col 44
    "delay_reason": ["سبب تأخير القسم بالرد"],    # NEW: col 58
    "closure_delay": ["سبب تأخير اغلاق الشكوى خلال 72 ساعه"],  # col 59
    "action_taken": ["الاجراء المتخذ من قبل القسم المعني"],      # col 62
    "action_result": ["نتيجة الاجراء المتخذ بعد التحقيق"],       # col 63
    "recommendation": ["Recommendation/Action plan"],             # col 64
    "solutions": ["حلول واقتراحات"],                              # col 67
}

3b. Fix _process_sheet() — parse new timeline dates: Add parsing for all new date fields (form_sent_date, activated_date, sent_date, first_reminder, second_reminder, escalated_date, closed_date, resolved_date).

3c. Fix status determination:

# Current: only checks response_date → open/resolved
# Fixed: check all dates like historical script
if closed_date:
    status = "closed"
elif resolved_date:
    status = "resolved"
elif escalated_date:
    status = "in_progress"
else:
    status = "open"

3d. Fix Complaint.objects.create() — add all missing fields:

• activated_at=activated_date
• form_sent_at=form_sent_date
• forwarded_to_dept_at=sent_date
• reminder_sent_at=first_reminder
• second_reminder_sent_at=second_reminder
• escalated_at=escalated_date
• closed_at=closed_date
• resolved_at=resolved_date
• satisfaction=normalize_satisfaction(satisfaction_val)
• resolution_outcome=normalize_rightful_side(rightful_side)
• complaint_subject=complaint_subject
• explanation_delay_reason=delay_reason
• delay_reason_closure=closure_delay
• action_taken_by_dept=action_taken
• action_result=action_result
• recommendation_action_plan=recommendation or solutions

4. Run re-import

python manage.py shell -c "from apps.complaints.models import Complaint; Complaint.objects.all().delete()"
python manage.py import_all_complaints --hospital-code=HH-N
python manage.py backfill_sent_to_department

5. Verify

• Check complaint counts match
• Check timeline dates populated
• Check satisfaction field populated
• Check resolution_outcome populated
• Check department detail page shows correct data