# Survey Tracking Implementation - Final Summary

## Overview

This document provides a comprehensive summary of the survey tracking and analytics system implementation for tracking patient survey engagement throughout their healthcare journey.

## Tracking Requirements Met

✅ **When patient receives survey link** - `sent_at` timestamp + `status='sent'`

✅ **How many times patient opens the link** - `open_count` field + `last_opened_at` timestamp

✅ **How many open and fill the survey** - Completed surveys with `status='completed'`

✅ **How many open and don't fill** - Abandoned surveys with `status='abandoned'`

✅ **Time from sending to filling** - Calculated from `sent_at` to `completed_at`

## Implementation Details

### 1. Database Changes

**SurveyInstance Model Updates:**
- `open_count` (IntegerField) - Number of times survey was opened
- `last_opened_at` (DateTimeField) - Most recent open timestamp
- `time_spent_seconds` (IntegerField) - Total time patient spent on survey
- `status` field enhanced with new statuses:
  - `sent` - Survey has been sent
  - `viewed` - Patient opened the survey
  - `in_progress` - Patient is actively answering (auto-detected)
  - `completed` - Patient completed the survey
  - `abandoned` - Patient opened but didn't complete (auto-detected)
  - `expired` - Token expired
  - `cancelled` - Survey cancelled

**SurveyTracking Model (New):**
- Tracks granular events throughout survey lifecycle
- Event types: page_view, survey_started, question_answered, survey_completed, survey_abandoned
- Captures device/browser info, IP, location, time metrics
- Flexible metadata field for custom data

### 2. Automatic Status Detection

#### in_progress Status (Automatic)
- **Trigger**: First patient interaction with any question
- **Implementation**: JavaScript tracking in frontend
- **Endpoint**: `POST /surveys/s/{access_token}/track-start/`
- **Result**: Status changes to `in_progress` + creates tracking event

#### abandoned Status (Automatic)
- **Trigger**: Survey not completed within 24 hours (configurable)
- **Implementation**: Celery background task + management command
- **Command**: `python manage.py mark_abandoned_surveys`
- **Result**: Status changes to `abandoned` + creates tracking event with metadata

### 3. Analytics Capabilities

**Key Metrics Available:**
- Total surveys sent
- Total surveys opened
- Total surveys completed
- Total surveys abandoned
- Open rate (percentage)
- Completion rate (percentage)
- Abandonment rate (percentage)
- Average completion time (minutes)
- Breakdown by delivery channel (SMS, email)

**Analytics Functions:**
- `get_survey_engagement_stats()` - Overall engagement metrics
- `get_patient_survey_timeline()` - Individual patient history
- `get_survey_completion_times()` - Individual completion times
- `get_survey_abandonment_analysis()` - Abandonment patterns
- `get_hourly_survey_activity()` - Activity by hour of day

### 4. API Endpoints

**Analytics API:**
- `GET /api/surveys/api/analytics/engagement_stats/`
- `GET /api/surveys/api/analytics/patient_timeline/`
- `GET /api/surveys/api/analytics/completion_times/`
- `GET /api/surveys/api/analytics/abandonment_analysis/`
- `GET /api/surveys/api/analytics/hourly_activity/`
- `GET /api/surveys/api/analytics/summary_dashboard/`

**Tracking API:**
- `POST /surveys/s/{access_token}/track-start/` - Track survey start
- `GET /api/surveys/api/tracking/by_survey/` - Get tracking events

### 5. Admin Interface

**Enhanced SurveyInstance Admin:**
- Display open count, time spent, status badges
- Inline tracking events viewer
- Filters by status, delivery channel
- Detailed fieldsets for tracking data

**New SurveyTracking Admin:**
- View all tracking events
- Filters by event type, device, browser
- Search by IP address, patient name
- Links to related survey instances

### 6. Frontend Tracking

**JavaScript Implementation:**
- Automatic detection of first interaction
- Tracks when patient starts answering
- Sends data to backend without user action
- Works with all question types (rating, NPS, text, etc.)

**Template Updates:**
- `templates/surveys/public_form.html` - Tracking JavaScript added
- Monitors form for interactions (click, input, change)
- Updates progress bar in real-time

## Usage Examples

### Get Overall Statistics
```python
from apps.surveys.analytics import get_survey_engagement_stats

stats = get_survey_engagement_stats(days=30)
print(f"Total sent: {stats['total_sent']}")
print(f"Total opened: {stats['total_opened']}")
print(f"Total completed: {stats['total_completed']}")
print(f"Total abandoned: {stats['total_abandoned']}")
print(f"Open rate: {stats['open_rate']}%")
print(f"Completion rate: {stats['completion_rate']}%")
```

### Get Patient Timeline
```python
from apps.surveys.analytics import get_patient_survey_timeline

timeline = get_patient_survey_timeline(patient_id=123)
for entry in timeline:
    time_to_complete = (entry['completed_at'] - entry['sent_at']).total_seconds() / 60
    print(f"Survey: {entry['survey_name']}")
    print(f"Sent: {entry['sent_at']}")
    print(f"Opened: {entry['opened_at']}")
    print(f"Completed: {entry['completed_at']}")
    print(f"Time to complete: {time_to_complete:.1f} minutes")
    print(f"Opens: {entry['open_count']}")
```

### Run Abandoned Survey Detection
```bash
# Manual run
python manage.py mark_abandoned_surveys

# Dry run (preview)
python manage.py mark_abandoned_surveys --dry-run

# Custom hours
python manage.py mark_abandoned_surveys --hours 48
```

## Configuration

Add to Django settings:
```python
# Hours before marking survey as abandoned
SURVEY_ABANDONMENT_HOURS = 24
```

Add to Celery beat schedule:
```python
app.conf.beat_schedule = {
    'mark-abandoned-surveys': {
        'task': 'apps.surveys.tasks.mark_abandoned_surveys',
        'schedule': crontab(hour=2, minute=0),  # Daily at 2 AM
        'kwargs': {'hours': 24}
    }
}
```

## Files Created/Modified

### New Files
- `apps/surveys/analytics.py` - Analytics functions
- `apps/surveys/analytics_views.py` - Analytics API views
- `apps/surveys/analytics_urls.py` - Analytics URL patterns
- `apps/surveys/migrations/0003_add_survey_tracking.py` - Database migration
- `apps/surveys/management/commands/mark_abandoned_surveys.py` - Abandoned survey command
- `test_survey_tracking.py` - Test script
- `test_survey_status_transitions.py` - Status transition tests

### Modified Files
- `apps/surveys/models.py` - Added tracking fields and SurveyTracking model
- `apps/surveys/admin.py` - Enhanced admin interface
- `apps/surveys/serializers.py` - Added analytics serializers
- `apps/surveys/public_views.py` - Added track_survey_start view
- `apps/surveys/urls.py` - Added tracking endpoint
- `apps/surveys/tasks.py` - Added mark_abandoned_surveys task
- `templates/surveys/public_form.html` - Added tracking JavaScript
- `requirements.txt` - Added user-agents dependency

### Documentation
- `docs/SURVEY_TRACKING_IMPLEMENTATION.md` - Complete implementation guide
- `docs/SURVEY_TRACKING_GUIDE.md` - User guide
- `docs/SURVEY_TRACKING_SUMMARY.md` - Feature summary
- `docs/SURVEY_TRACKING_FINAL_SUMMARY.md` - This document

## Migration

Run the database migration:
```bash
python manage.py migrate surveys
```

## Testing

Test scripts available:
```bash
# Basic tracking test
python test_survey_tracking.py

# Status transition test
python test_survey_status_transitions.py
```

## Deployment Checklist

- [ ] Run database migrations
- [ ] Install dependencies (user-agents)
- [ ] Configure SURVEY_ABANDONMENT_HOURS in settings
- [ ] Add Celery beat schedule for abandoned surveys
- [ ] Test in_progress status detection
- [ ] Test abandoned survey detection
- [ ] Verify analytics API endpoints
- [ ] Test admin interface
- [ ] Monitor tracking events in production

## Performance Considerations

- Database indexes added for efficient queries
- Consider caching for frequently accessed analytics
- Archive old tracking events after 90 days
- Aggregate daily statistics for long-term reporting

## Benefits

1. **Complete Visibility**: Track every stage of survey lifecycle
2. **Automatic Detection**: No manual intervention needed for status updates
3. **Actionable Insights**: Identify drop-off points and optimize surveys
4. **Patient Engagement**: Understand when and how patients complete surveys
5. **Data-Driven Decisions**: Make informed decisions based on real metrics

## Support

For detailed implementation information, see:
- `docs/SURVEY_TRACKING_IMPLEMENTATION.md` - Technical implementation details
- `docs/SURVEY_TRACKING_GUIDE.md` - Usage guide
- API documentation: `/api/docs/`
- Admin interface: `/admin/surveys/`