HH/docs/EMAIL_SENDING_FIX.md

# Email Sending Fix - Complete Summary

## Problem Identified

Emails were not being sent in the development environment due to conflicting email backend configurations:

### Configuration Conflict

1. **`.env` file**: Configured to use SMTP backend
   ```
   EMAIL_BACKEND=django.core.mail.backends.smtp.EmailBackend
   ```

2. **`config/settings/base.py`**: Read EMAIL_BACKEND from .env → SMTP

3. **`config/settings/dev.py`**: **OVERRRODE** with console backend
   ```python
   EMAIL_BACKEND = 'django.core.mail.backends.console.EmailBackend'
   ```

4. **Simulator API**: Attempted to use Django's `send_mail()` which used console backend, but then tried to send via SMTP server that doesn't support STARTTLS

### Result

- Emails printed to console instead of being sent via simulator API
- SMTP connection errors: "STARTTLS extension not supported by server"
- All email requests failed with 500 status

## Solution Implemented

### 1. Updated `config/settings/dev.py`

**Changed**: Commented out console backend override to allow simulator API to work

```python
# Email backend for development
# Use simulator API for email (configured in .env with EMAIL_API_ENABLED=true)
# Emails will be sent to http://localhost:8000/api/simulator/send-email
# and displayed in terminal with formatted output
# EMAIL_BACKEND = 'django.core.mail.backends.console.EmailBackend'  # Disabled for simulator API
```

**Effect**: Now uses the configuration from `.env` which enables EMAIL_API

### 2. Updated `apps/notifications/services.py`

**Changed**: Modified `send_email()` and `send_sms()` to check for API enabled first

```python
@staticmethod
def send_email(email, subject, message, html_message=None, related_object=None, metadata=None):
    # Check if Email API is enabled and use it (simulator or external API)
    email_api_config = settings.EXTERNAL_NOTIFICATION_API.get('email', {})
    if email_api_config.get('enabled', False):
        return NotificationService.send_email_via_api(...)
    
    # Fallback to Django email backend if API disabled
    ...
```

**Effect**: Prioritizes API-based sending when enabled, falls back to Django's send_mail() otherwise

### 3. Updated `apps/simulator/views.py`

**Changed**: Modified email simulator to BOTH display formatted output AND send real emails via SMTP

```python
# Display formatted email to terminal
print(f"\n{'╔' + '═'*68 + '╗'}")
print(f"║{' ' * 15}📧 EMAIL SIMULATOR{' ' * 34}║")
# ... formatted output ...
print(f"╚{'═'*68}╝\n")

# Send real email via Django SMTP
send_mail(
    subject=subject,
    message=message,
    from_email=settings.DEFAULT_FROM_EMAIL,
    recipient_list=[to_email],
    html_message=html_message,
    fail_silently=False
)
```

**Effect**: Simulator displays emails in terminal with beautiful formatted output AND sends real emails via SMTP

### 4. Updated `.env` File

**Changed**: Disabled TLS for SMTP server

```bash
# Before
EMAIL_USE_TLS=True

# After
EMAIL_USE_TLS=False
```

**Effect**: Allows connection to SMTP server at 10.10.1.110:2225 which doesn't support STARTTLS

## Current Email Flow (Development)

```
NotificationService.send_email()
  ↓
Checks EMAIL_API_ENABLED in settings
  ↓
If enabled → Uses send_email_via_api()
  ↓
Sends POST request to http://localhost:8000/api/simulator/send-email
  ↓
Simulator receives request
  ↓
Displays formatted email in terminal
  ↓
Sends real email via SMTP (10.10.1.110:2225)
  ↓
Returns 200 OK with success response
  ↓
NotificationLog created with status='sent'
```

## Test Results

All tests passed successfully:

```
1. Testing plain text email... ✅
   Log ID: 476a0fce-9a26-4244-877c-62e696c64169
   Recipient: test@example.com

2. Testing HTML email... ✅
   Log ID: f2bd7cbf-b5ee-4f02-9717-a3c61b46f88d
   Recipient: test@example.com

3. Testing SMS sending... ✅
   Log ID: edc987b6-aca6-4368-b3e3-8d42b3eb9dd5
   Recipient: +966501234567
```

## Server Log Output

```
INFO [Email Simulator] Sending email to test@example.com: Test Email - Plain Text
INFO [Email Simulator] Email sent via SMTP to test@example.com
INFO [Email Simulator] Email sent successfully to test@example.com
INFO [Simulator] EMAIL Request #1: sent
INFO "POST /api/simulator/send-email HTTP/1.1" 200 170
```

## Formatted Output Example

### Email Simulator Output
```
╔════════════════════════════════════════════════════════════════════╗
║               📧 EMAIL SIMULATOR                                   ║
╠════════════════════════════════════════════════════════════════════╣
║ Request #: 1                                                      ║
╠════════════════════════════════════════════════════════════════════╣
║ To:      test@example.com                                         ║
║ Subject: Test Email - Plain Text                                   ║
╠════════════════════════════════════════════════════════════════════╣
║ Message:                                                          ║
║         This is a test email sent via simulator API.                ║
╚════════════════════════════════════════════════════════════════════╝
```

### SMS Simulator Output
```
╔════════════════════════════════════════════════════════════════════╗
║               📱 SMS SIMULATOR                                    ║
╠════════════════════════════════════════════════════════════════════╣
║ Request #: 1                                                       ║
╠════════════════════════════════════════════════════════════════════╣
║ To:      +966501234567                                           ║
║ Time:    2026-01-12 18:57:13                                      ║
╠════════════════════════════════════════════════════════════════════╣
║ Message:                                                          ║
║         This is a test SMS sent via simulator API.                  ║
╚════════════════════════════════════════════════════════════════════╝
```

## Configuration

### Required `.env` Settings

```bash
# Email Configuration
EMAIL_BACKEND=django.core.mail.backends.smtp.EmailBackend
EMAIL_HOST=10.10.1.110
EMAIL_PORT=2225
EMAIL_USE_TLS=False  # Disabled for this SMTP server
EMAIL_HOST_USER=
EMAIL_HOST_PASSWORD=
DEFAULT_FROM_EMAIL=noreply@px360.sa

# Enable Email API (simulator or external)
EMAIL_API_ENABLED=true

# Email Simulator API URL
EMAIL_API_URL=http://localhost:8000/api/simulator/send-email
EMAIL_API_KEY=simulator-test-key
EMAIL_API_AUTH_METHOD=bearer

# Enable SMS API (simulator or external)
SMS_API_ENABLED=true

# SMS Simulator API URL
SMS_API_URL=http://localhost:8000/api/simulator/send-sms
SMS_API_KEY=simulator-test-key
SMS_API_AUTH_METHOD=bearer
```

## Usage

### Send Email (Plain Text)
```python
from apps.notifications.services import NotificationService

log = NotificationService.send_email(
    email='user@example.com',
    subject='Welcome to PX360',
    message='Thank you for registering!'
)
```

### Send Email (HTML)
```python
html_message = """
<html>
<body>
    <h1>Welcome!</h1>
    <p>Thank you for registering with <strong>PX360</strong>.</p>
</body>
</html>
"""

log = NotificationService.send_email(
    email='user@example.com',
    subject='Welcome to PX360',
    message='Plain text version',
    html_message=html_message
)
```

### Send SMS
```python
log = NotificationService.send_sms(
    phone='+966501234567',
    message='Your survey is ready. Please complete it today!'
)
```

## Benefits

1. **Dual-Mode Operation**: Displays formatted output AND sends real emails
2. **No SMTP Errors**: Fixed STARTTLS issue by disabling TLS for development
3. **Formatted Output**: Beautiful terminal display for both email and SMS
4. **Logged to Database**: All notifications logged in NotificationLog table
5. **API-First Architecture**: Easy to switch to external APIs (SendGrid, Twilio) in production
6. **Retry Logic**: Built-in retry with exponential backoff for API failures
7. **Testing Friendly**: Easy to verify emails are being sent

## Production Configuration

To use actual email/SMS providers in production:

```bash
# Email (e.g., SendGrid)
EMAIL_API_ENABLED=true
EMAIL_API_URL=https://api.sendgrid.com/v3/mail/send
EMAIL_API_KEY=your-sendgrid-api-key
EMAIL_API_AUTH_METHOD=bearer

# SMS (e.g., Twilio)
SMS_API_ENABLED=true
SMS_API_URL=https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Messages.json
SMS_API_KEY=your-twilio-api-key
SMS_API_AUTH_METHOD=basic
```

## Testing

Run the test script to verify email sending:

```bash
python test_email_sending.py
```

Expected output:
```
======================================================================
Testing Email Sending via Simulator API
======================================================================

1. Testing plain text email... ✅ Plain text email sent successfully!
2. Testing HTML email... ✅ HTML email sent successfully!
3. Testing SMS sending... ✅ SMS sent successfully!

======================================================================
Test Complete!
======================================================================
```

## Files Modified

1. `config/settings/dev.py` - Disabled console backend override
2. `apps/notifications/services.py` - Updated to prioritize API sending
3. `apps/simulator/views.py` - Changed to print formatted output AND send via SMTP
4. `.env` - Disabled TLS for SMTP server
5. `test_email_sending.py` - Created test script
6. `docs/EMAIL_SENDING_FIX.md` - Complete documentation

## Next Steps

- Consider implementing email templates for better formatting
- Add email preview functionality in admin panel
- Implement email tracking and analytics
- Add bounce and complaint handling for production
- Set up webhook notifications for delivery status
- Configure a more secure SMTP server for production with TLS enabled

## Troubleshooting

### Emails still not sending?

1. Check `.env` file has `EMAIL_API_ENABLED=true`
2. Verify server is running on port 8000
3. Check logs: `tail -f logs/px360.log`
4. Test simulator directly: `curl http://localhost:8000/api/simulator/health-check`
5. Verify SMTP server is accessible: `telnet 10.10.1.110 2225`

### Can't see formatted output?

The formatted output is printed to the terminal where the Django development server is running, not to the log file. Make sure you're watching the correct terminal.

### Email sent but not received?

1. Check spam/junk folder
2. Verify email address is correct
3. Check SMTP server logs
4. Verify `DEFAULT_FROM_EMAIL` is properly configured
5. Some email providers may reject emails from certain senders

## Date

Fixed on: January 12, 2026
Updated: January 12, 2026 - Added real SMTP sending capability