HH/docs/SIMULATOR_LOGGING_IMPLEMENTATION.md

# Simulator Data Logging Implementation

## Overview

This implementation adds persistent data logging and a web-based viewer for the Survey and Patient Journey Simulator. All simulator requests (Email, SMS, and HIS Events) are now automatically logged to the database and can be viewed through a user-friendly interface.

## Features

### 1. Automatic Data Logging

All simulator requests are automatically logged to the database with the following information:

- **Email Simulator**: Recipient, subject, message preview, status, response data
- **SMS Simulator**: Recipient phone number, message preview, status, response data
- **HIS Journey Events**: Patient MRN, encounter ID, hospital code, event type, visit type, department

### 2. Processing Metrics

Each log entry captures:
- Request timestamp
- Processing time (in milliseconds)
- Request status (success, sent, failed, partial)
- Error messages (if applicable)
- IP address and user agent

### 3. Web-Based Log Viewer

Access the simulator logs at: `/simulator/logs/`

#### Features:
- **Statistics Dashboard**: View total requests, success rate, average processing time
- **Channel Breakdown**: See counts for Email, SMS, and HIS Events
- **Status Breakdown**: Success, failed, and partial request counts
- **Advanced Filtering**: Filter by channel, status, hospital, visit type, date range
- **Search**: Search by request ID, patient MRN, recipient, or subject
- **Pagination**: Handle large datasets with customizable page sizes
- **Detailed View**: Click any log entry to see full request/response data

### 4. Log Detail View

Each log entry can be viewed in detail showing:
- Full request payload (formatted JSON)
- Full response data (formatted JSON)
- Related objects (patient, journey, survey if available)
- Error details (if applicable)
- Message preview
- Copy to clipboard functionality

## Database Schema

### SimulatorRequestLog Model

```python
class SimulatorRequestLog(models.Model):
    # Request identification
    request_id = models.AutoField(primary_key=True)
    timestamp = models.DateTimeField(auto_now_add=True)
    
    # Channel identification
    channel = models.CharField(choices=['email', 'sms', 'his_event'])
    
    # Request data
    payload = models.JSONField(default=dict)
    
    # Response data
    status = models.CharField()  # success, sent, failed, partial
    response_data = models.JSONField(null=True, blank=True)
    
    # Processing details
    processing_time_ms = models.IntegerField(null=True, blank=True)
    error_message = models.TextField(null=True, blank=True)
    
    # Related objects
    patient_id = models.CharField(max_length=100, null=True, blank=True)
    journey_id = models.CharField(max_length=100, null=True, blank=True)
    survey_id = models.CharField(max_length=100, null=True, blank=True)
    hospital_code = models.CharField(max_length=50, null=True, blank=True)
    
    # Email/SMS specific
    recipient = models.CharField(max_length=255, null=True, blank=True)
    subject = models.CharField(max_length=500, null=True, blank=True)
    message_preview = models.TextField(null=True, blank=True)
    
    # HIS event specific
    event_type = models.CharField(max_length=100, null=True, blank=True)
    visit_type = models.CharField(max_length=50, null=True, blank=True)
    department = models.CharField(max_length=100, null=True, blank=True)
    
    # Metadata
    ip_address = models.GenericIPAddressField(null=True, blank=True)
    user_agent = models.TextField(null=True, blank=True)
```

## API Endpoints

### Logging Endpoints (Automatic)

These endpoints automatically log all requests:

- `POST /simulator/api/email/` - Email simulator
- `POST /simulator/api/sms/` - SMS simulator
- `POST /simulator/api/his-events/` - HIS events handler

### UI Endpoints

- `GET /simulator/logs/` - View all simulator logs
- `GET /simulator/logs/<id>/` - View specific log details
- `POST /simulator/logs/clear/` - Clear all logs (admin only)

### Utility Endpoints

- `GET /simulator/health/` - Simulator health check and statistics
- `GET /simulator/reset/` - Reset simulator statistics (in-memory only)

## Usage Examples

### 1. Send Email and View Logs

```bash
curl -X POST http://localhost:8000/simulator/api/email/ \
  -H "Content-Type: application/json" \
  -d '{
    "to": "patient@example.com",
    "subject": "Survey Invitation",
    "message": "Please complete our survey..."
  }'
```

Then view the log at: `http://localhost:8000/simulator/logs/`

### 2. Send SMS and View Logs

```bash
curl -X POST http://localhost:8000/simulator/api/sms/ \
  -H "Content-Type: application/json" \
  -d '{
    "to": "+966501234567",
    "message": "Your survey is ready..."
  }'
```

### 3. Send HIS Events and View Logs

```bash
curl -X POST http://localhost:8000/simulator/api/his-events/ \
  -H "Content-Type: application/json" \
  -d '{
    "events": [{
      "encounter_id": "ENC-2024-001",
      "mrn": "MRN-12345",
      "national_id": "1234567890",
      "first_name": "Ahmed",
      "last_name": "Mohammed",
      "phone": "+966501234567",
      "email": "patient@example.com",
      "event_type": "OPD_STAGE_1_REGISTRATION",
      "timestamp": "2024-01-20T10:30:00Z",
      "visit_type": "opd",
      "department": "Cardiology",
      "hospital_code": "ALH-main"
    }]
  }'
```

### 4. View Simulator Health

```bash
curl http://localhost:8000/simulator/health/
```

Response:
```json
{
  "status": "healthy",
  "timestamp": "2024-01-20T10:30:00Z",
  "statistics": {
    "total_requests": 15,
    "email_requests": 5,
    "sms_requests": 7,
    "recent_requests": [...]
  }
}
```

## Web Interface

### Log List View

Access: `/simulator/logs/`

Features:
- Statistics cards showing:
  - Total requests
  - Success/failed/partial counts
  - Success rate percentage
  - Average processing time
  - Survey invitations sent
- Channel breakdown (Email, SMS, HIS Events)
- Status breakdown
- Hospital breakdown (for HIS events)
- Advanced filters
- Paginated results table

### Log Detail View

Access: `/simulator/logs/<id>/`

Features:
- Basic information (channel, status, timestamp, processing time)
- Related objects (patient, journey, survey if available)
- Request payload (formatted JSON)
- Response data (formatted JSON)
- Error details (if applicable)
- Message preview
- Copy to clipboard for JSON data

## Maintenance

### Clearing Logs

To clear all simulator logs (admin only):

```bash
# Via UI
Visit /simulator/logs/ and click "Clear All Logs"

# Via API
POST /simulator/logs/clear/
```

### Database Management

```bash
# Create migrations
uv run python manage.py makemigrations simulator

# Apply migrations
uv run python manage.py migrate simulator

# Reset database (WARNING: Deletes all data)
uv run python manage.py migrate simulator zero
uv run python manage.py migrate simulator
```

## Performance Considerations

1. **Indexing**: The model includes indexes on frequently queried fields:
   - `timestamp`
   - `channel`
   - `status`
   - `patient_id`
   - `journey_id`
   - `survey_id`
   - `hospital_code`
   - `event_type`
   - `visit_type`

2. **Pagination**: Use pagination to handle large datasets efficiently

3. **Query Optimization**: The views use `select_related` for related objects to avoid N+1 queries

4. **JSON Storage**: Full request/response data is stored as JSON for flexibility

## Security

1. **Authentication**: Log viewer requires user login (`@login_required` decorator)

2. **Authorization**: Clear logs feature restricted to superusers and PX admins

3. **IP Tracking**: Client IP addresses are logged for auditing

4. **User Agent**: User agent information is captured for debugging

## Testing

### Test Email Simulator Logging

```bash
curl -X POST http://localhost:8000/simulator/api/email/ \
  -H "Content-Type: application/json" \
  -d '{
    "to": "test@example.com",
    "subject": "Test Email",
    "message": "This is a test email"
  }'
```

Verify log at: `http://localhost:8000/simulator/logs/`

### Test SMS Simulator Logging

```bash
curl -X POST http://localhost:8000/simulator/api/sms/ \
  -H "Content-Type: application/json" \
  -d '{
    "to": "+966501234567",
    "message": "This is a test SMS"
  }'
```

### Test HIS Events Logging

```bash
curl -X POST http://localhost:8000/simulator/api/his-events/ \
  -H "Content-Type: application/json" \
  -d '{
    "events": [{
      "encounter_id": "TEST-001",
      "mrn": "TEST-MRN",
      "national_id": "1234567890",
      "first_name": "Test",
      "last_name": "Patient",
      "phone": "+966501234567",
      "email": "test@example.com",
      "event_type": "OPD_STAGE_1_REGISTRATION",
      "timestamp": "2024-01-20T10:30:00Z",
      "visit_type": "opd",
      "department": "Cardiology",
      "hospital_code": "ALH-main"
    }]
  }'
```

## Troubleshooting

### Logs Not Appearing

1. Check if the database migration was applied:
   ```bash
   uv run python manage.py showmigrations simulator
   ```

2. Check for errors in logs:
   ```bash
   tail -f logs/django.log
   ```

3. Verify the model is imported in views.py

### Slow Query Performance

1. Add more indexes if needed
2. Use date range filters to reduce dataset size
3. Consider archiving old logs

### Memory Issues

1. Reduce page size in the UI
2. Implement log rotation/archival
3. Add cleanup jobs to remove old logs

## Future Enhancements

1. **Export Functionality**: Export logs to CSV/Excel
2. **Log Archival**: Automatic archival of old logs
3. **Analytics Dashboard**: Charts and graphs for log statistics
4. **Real-time Updates**: WebSocket support for real-time log updates
5. **Alerting**: Email/SMS alerts for failed requests
6. **Performance Metrics**: More detailed performance tracking
7. **Log Search**: Full-text search across all log fields
8. **API Documentation**: Swagger/OpenAPI documentation for endpoints

## Files Modified/Created

### Created:
- `apps/simulator/models.py` - Database model
- `apps/simulator/ui_views.py` - Web views
- `templates/simulator/log_list.html` - List view template
- `templates/simulator/log_detail.html` - Detail view template

### Modified:
- `apps/simulator/views.py` - Added logging to all endpoints
- `apps/simulator/urls.py` - Added UI routes

## Summary

This implementation provides a complete solution for logging and viewing simulator data. All requests are automatically captured, stored in the database, and can be viewed through a user-friendly web interface with advanced filtering and search capabilities. The system is designed to be performant, secure, and maintainable.