Marwan Alwali 27a71b658a first commit

2025-11-02 14:35:35 +03:00

18 KiB

Raw Permalink Blame History

ZATCA E-Invoice Enhancement Recommendations

Executive Summary

Your AgdarCentre e-invoice implementation has been significantly enhanced with ZATCA compliance features. This document provides recommendations for further improvements and production readiness.

✅ What Has Been Implemented

Phase 1: Generation Phase (Mandatory since Dec 4, 2021)

✅ Invoice Counter (ICV) - Sequential, tamper-resistant
✅ Hash Chain (PIH) - SHA-256 linking of invoices
✅ QR Code Generation - TLV encoded, Base64 format (5 fields)
✅ Invoice Type Classification - Standard/Simplified distinction
✅ Credit/Debit Note Support - With billing references

Phase 2: Integration Phase (Mandatory from Jan 1, 2023 in waves)

✅ XML Generation - UBL 2.1 format
✅ FATOORA API Integration - Clearance & Reporting APIs
✅ CSID Management - Storage, validation, renewal tracking
✅ Automated Tasks - Celery tasks for submission and monitoring
✅ Failure Handling - Retry logic and error tracking

🔧 Critical Enhancements Needed

1. Cryptographic Signing (ECDSA) - HIGH PRIORITY ⚠️

Current Status: Not implemented

Requirements:

ECDSA signing using secp256k1 curve
Private key generation and secure storage
Public key extraction
Digital signature generation

Recommended Implementation:

# Install cryptography library
pip install cryptography ecdsa

# In finance/crypto_service.py
from ecdsa import SigningKey, SECP256k1
import hashlib

class CryptoService:
    @staticmethod
    def generate_key_pair():
        """Generate ECDSA key pair."""
        private_key = SigningKey.generate(curve=SECP256k1)
        public_key = private_key.get_verifying_key()
        return private_key, public_key
    
    @staticmethod
    def sign_invoice(invoice_hash: str, private_key) -> str:
        """Sign invoice hash with private key."""
        signature = private_key.sign(
            invoice_hash.encode('utf-8'),
            hashfunc=hashlib.sha256
        )
        return base64.b64encode(signature).decode('utf-8')

Action Items:

Install cryptography libraries
Create crypto_service.py module
Implement key generation
Implement signing function
Store private keys securely (encrypted)
Update Invoice.save() to sign simplified invoices

2. Tenant VAT Number Field - HIGH PRIORITY ⚠️

Current Status: Referenced but not in Tenant model

Issue: Code references tenant.vat_number but field doesn't exist

Recommended Fix:

# In core/models.py - Tenant model
vat_number = models.CharField(
    max_length=15,
    unique=True,
    validators=[
        RegexValidator(
            regex=r'^3\d{13}3$',
            message='VAT number must be 15 digits starting and ending with 3'
        )
    ],
    verbose_name=_("VAT Registration Number")
)

Action Items:

Add vat_number field to Tenant model
Create migration
Update admin interface
Add validation

3. PDF/A-3 Generation - MEDIUM PRIORITY

Current Status: Not implemented

Requirements:

Generate PDF/A-3 compliant documents
Embed XML inside PDF
Include QR code image
Bilingual layout (Arabic + English)

Recommended Libraries:

pip install reportlab PyPDF2 qrcode pillow

Implementation Approach:

# In finance/pdf_service.py
from reportlab.pdfgen import canvas
from reportlab.lib.pagesizes import A4
import qrcode

class PDFService:
    @staticmethod
    def generate_invoice_pdf(invoice):
        """Generate PDF/A-3 with embedded XML."""
        # 1. Generate PDF layout
        # 2. Add QR code image
        # 3. Embed XML as attachment
        # 4. Return PDF bytes
        pass

Action Items:

Install PDF libraries
Create PDF template
Implement QR code image generation
Implement XML embedding
Add Arabic font support

4. Invoice Counter Auto-Increment - HIGH PRIORITY ⚠️

Current Status: Manual counter management in views

Issue: Counter should auto-increment in model save

Recommended Fix:

# In finance/models.py - Invoice.save()
def save(self, *args, **kwargs):
    """Override save to auto-generate counter, hash, and QR."""
    # Auto-increment counter if new invoice
    if not self.pk and not self.invoice_counter:
        from finance.csid_manager import InvoiceCounterManager
        self.invoice_counter = InvoiceCounterManager.get_next_counter(self.tenant)
        self.previous_invoice_hash = InvoiceCounterManager.get_previous_invoice_hash(self.tenant)
    
    # Generate hash
    if not self.invoice_hash:
        self.invoice_hash = self.generate_invoice_hash()
    
    # Generate QR code
    self.qr_code = self.generate_qr_code()
    
    super().save(*args, **kwargs)

Action Items:

Move counter logic to model save
Add transaction locking to prevent race conditions
Test concurrent invoice creation

5. Arabic Language Support - MEDIUM PRIORITY

Current Status: Partial (name_en/name_ar fields exist)

Requirements:

All invoice data must be in Arabic
Bilingual support (Arabic + English)
RTL layout for Arabic
Arabic numerals or Hindi numerals

Recommended Implementation:

# In finance/models.py
class Invoice:
    # Add Arabic fields
    notes_ar = models.TextField(blank=True)
    
    def get_seller_name_ar(self):
        """Get seller name in Arabic."""
        return getattr(self.tenant, 'name_ar', self.tenant.name)
    
    def get_patient_name_ar(self):
        """Get patient name in Arabic."""
        if hasattr(self.patient, 'full_name_ar'):
            return self.patient.full_name_ar
        return str(self.patient)

Action Items:

Add Arabic fields to all models
Create Arabic invoice templates
Implement RTL CSS
Add Arabic translations
Test Arabic PDF generation

6. Offline Mode Support - MEDIUM PRIORITY

Current Status: Not implemented

Requirements:

Queue invoices when offline
Auto-submit when connection restored
Local storage of pending submissions
Sync status tracking

Recommended Implementation:

# In finance/models.py
class InvoiceSubmissionQueue(models.Model):
    """Queue for offline invoice submissions."""
    invoice = models.ForeignKey(Invoice, on_delete=models.CASCADE)
    submission_type = models.CharField(max_length=20)  # CLEARANCE/REPORTING
    attempts = models.PositiveIntegerField(default=0)
    last_attempt = models.DateTimeField(null=True)
    status = models.CharField(max_length=20)  # PENDING/SUBMITTED/FAILED
    error_message = models.TextField(blank=True)

Action Items:

Create submission queue model
Implement offline detection
Create background sync task
Add UI indicators for offline mode

7. CSID Onboarding Integration - HIGH PRIORITY ⚠️

Current Status: CSID model exists, but no onboarding flow

Requirements:

OTP generation integration
CSR (Certificate Signing Request) generation
Compliance checks
Production CSID retrieval

Recommended Implementation:

# In finance/onboarding_service.py
class OnboardingService:
    @staticmethod
    def generate_csr(tenant, egs_info):
        """Generate Certificate Signing Request."""
        # Use OpenSSL or cryptography library
        pass
    
    @staticmethod
    def request_compliance_csid(csr, otp):
        """Request compliance CSID from ZATCA."""
        # POST to /compliance endpoint
        pass
    
    @staticmethod
    def request_production_csid(compliance_csid, request_id):
        """Request production CSID from ZATCA."""
        # POST to /production/csids endpoint
        pass

Action Items:

Create onboarding service
Implement CSR generation
Add OTP input UI
Implement compliance checks
Add CSID renewal workflow

8. Enhanced Error Handling - MEDIUM PRIORITY

Current Status: Basic error handling in tasks

Enhancements Needed:

Detailed error categorization
User-friendly error messages
Error recovery suggestions
Admin notifications for critical errors

Recommended Implementation:

# In finance/error_handler.py
class ZATCAErrorHandler:
    ERROR_CODES = {
        '400': 'Invalid request - check invoice data',
        '401': 'Authentication failed - check CSID',
        '413': 'Invoice too large - reduce size',
        '429': 'Too many requests - rate limited',
        '500': 'ZATCA server error - retry later',
        '503': 'Service unavailable - retry later',
    }
    
    @staticmethod
    def handle_error(response, invoice):
        """Handle ZATCA API error."""
        error_code = str(response.get('status_code', ''))
        error_msg = ZATCAErrorHandler.ERROR_CODES.get(error_code, 'Unknown error')
        
        # Log error
        # Create notification
        # Suggest recovery action
        pass

Action Items:

Create error handler class
Map all ZATCA error codes
Add user notifications
Create error recovery guide

9. Compliance Dashboard - LOW PRIORITY

Current Status: Basic financial reports exist

Enhancements:

ZATCA submission statistics
Compliance rate tracking
Failed submission alerts
CSID status monitoring
Invoice sequence validation

Recommended Features:

Real-time compliance metrics
Submission success rate charts
Failed invoice list with retry options
CSID expiry calendar
Audit log viewer

Action Items:

Create compliance dashboard view
Add ZATCA-specific metrics
Implement charts and graphs
Add export functionality

10. Testing & Validation - HIGH PRIORITY ⚠️

Current Status: No automated tests for ZATCA features

Recommended Tests:

# In finance/tests/test_zatca.py
class ZATCAComplianceTests(TestCase):
    def test_qr_code_generation(self):
        """Test QR code TLV encoding."""
        pass
    
    def test_hash_chain_integrity(self):
        """Test invoice hash chain."""
        pass
    
    def test_invoice_counter_sequence(self):
        """Test counter increments correctly."""
        pass
    
    def test_xml_generation(self):
        """Test UBL 2.1 XML generation."""
        pass
    
    def test_clearance_api(self):
        """Test clearance API integration."""
        pass
    
    def test_reporting_api(self):
        """Test reporting API integration."""
        pass

Action Items:

Create test suite for ZATCA features
Test QR code generation
Test hash chain
Test XML generation
Test API integration (sandbox)
Test failure scenarios

📋 Implementation Priority Matrix

Immediate (Week 1)

✅ Add vat_number field to Tenant model
✅ Move counter auto-increment to model save
✅ Implement cryptographic signing (ECDSA)
✅ Create basic test suite

Short-term (Week 2-3)

✅ Implement onboarding service
✅ Add enhanced error handling
✅ Create PDF/A-3 generation
✅ Add offline mode support

Medium-term (Month 1-2)

✅ Full Arabic language support
✅ Compliance dashboard
✅ Comprehensive testing
✅ Performance optimization

🔒 Security Recommendations

1. CSID Secret Storage

Current: Plain text in database Recommended: Encrypted storage

from cryptography.fernet import Fernet

class SecureCSIDStorage:
    @staticmethod
    def encrypt_secret(secret: str) -> str:
        """Encrypt CSID secret."""
        key = settings.ENCRYPTION_KEY
        f = Fernet(key)
        return f.encrypt(secret.encode()).decode()
    
    @staticmethod
    def decrypt_secret(encrypted: str) -> str:
        """Decrypt CSID secret."""
        key = settings.ENCRYPTION_KEY
        f = Fernet(key)
        return f.decrypt(encrypted.encode()).decode()

2. Private Key Protection

Store in hardware security module (HSM) if available
Use Django's SECRET_KEY for encryption
Never log or expose private keys
Implement key rotation policy

3. API Authentication

Use environment variables for credentials
Implement rate limiting
Add request signing
Monitor for suspicious activity

📊 Monitoring & Alerting

Recommended Metrics to Track:

Submission Success Rate
- Target: >99%
- Alert if <95%
Average Submission Time
- Target: <2 seconds
- Alert if >5 seconds
Failed Submissions
- Alert on any 400 errors (data issues)
- Auto-retry on 500 errors (server issues)
CSID Expiry
- Alert 30 days before expiry
- Alert 7 days before expiry
- Alert on expiry day
Invoice Sequence Gaps
- Daily validation
- Alert on any gaps detected

Recommended Tools:

Sentry for error tracking
Prometheus for metrics
Grafana for dashboards
PagerDuty for critical alerts

🧪 Testing Strategy

1. Unit Tests

# Test QR code generation
def test_qr_code_tlv_encoding():
    invoice = create_test_invoice()
    qr = invoice.generate_qr_code()
    
    # Decode and verify
    decoded = base64.b64decode(qr)
    assert decoded[0] == 1  # Tag 1
    assert decoded[1] > 0    # Length
    # ... verify all tags

# Test hash chain
def test_invoice_hash_chain():
    inv1 = create_invoice()
    inv2 = create_invoice()
    
    assert inv2.previous_invoice_hash == inv1.invoice_hash

2. Integration Tests

# Test ZATCA API (sandbox)
def test_clearance_api_integration():
    invoice = create_standard_invoice()
    zatca = ZATCAService(use_sandbox=True)
    
    success, response = zatca.submit_for_clearance(
        invoice, test_csid, test_secret
    )
    
    assert success == True
    assert 'clearedInvoice' in response

3. Load Tests

Test concurrent invoice creation
Test batch submission performance
Test API rate limits

📱 User Interface Enhancements

1. Invoice Form

Add:

Invoice type selector (Standard/Simplified)
QR code preview
ZATCA submission status indicator
Retry button for failed submissions

2. Invoice Detail Page

Add:

QR code display (scannable)
ZATCA submission history
XML download button
PDF download button
Clearance/Reporting status badge

3. Compliance Dashboard

Add:

Submission success rate chart
Failed invoices list
CSID status widget
Compliance alerts

🌐 Arabic Language Implementation

Required Changes:

Invoice Templates

<!-- invoice_detail.html -->
<div class="invoice-header" dir="rtl" lang="ar">
    <h1>فاتورة ضريبية</h1>
    <p>رقم الفاتورة: {{ invoice.invoice_number }}</p>
</div>

Model Fields

# Add Arabic fields
seller_name_ar = models.CharField(max_length=200)
buyer_name_ar = models.CharField(max_length=200)
line_item_description_ar = models.CharField(max_length=500)

XML Generation

# Include Arabic text in XML
name_elem.text = invoice.tenant.name_ar or invoice.tenant.name

🔄 Workflow Improvements

1. Automated Clearance for Standard Invoices

# In finance/signals.py
@receiver(post_save, sender=Invoice)
def auto_submit_standard_invoice(sender, instance, created, **kwargs):
    """Auto-submit standard invoices for clearance."""
    if created and instance.invoice_type == 'STANDARD':
        # Submit for clearance immediately
        submit_invoice_to_zatca.delay(str(instance.id))

2. Scheduled Reporting for Simplified Invoices

# In AgdarCentre/celery.py
from celery.schedules import crontab

app.conf.beat_schedule = {
    'auto-submit-simplified-invoices': {
        'task': 'finance.zatca_tasks.auto_submit_simplified_invoices',
        'schedule': crontab(minute='*/30'),  # Every 30 minutes
    },
    'check-csid-expiry': {
        'task': 'finance.zatca_tasks.check_csid_expiry',
        'schedule': crontab(hour=9, minute=0),  # Daily at 9 AM
    },
}

📦 Required Dependencies

Add to requirements.txt:

# ZATCA E-Invoice
cryptography>=41.0.0
ecdsa>=0.18.0
requests>=2.31.0
qrcode>=7.4.2
Pillow>=10.0.0
reportlab>=4.0.0
PyPDF2>=3.0.0
lxml>=4.9.0  # For better XML handling

🚀 Deployment Checklist

Before Production:

Obtain production CSID from ZATCA
Configure production API endpoints
Set up secure key storage
Enable SSL/TLS
Configure backup system
Set up monitoring and alerts
Train staff on new features
Create user documentation
Perform load testing
Complete security audit

Environment Variables:

# .env
ZATCA_USE_SANDBOX=False
ZATCA_CSID=your-production-csid
ZATCA_SECRET=your-production-secret
ZATCA_VAT_NUMBER=300000000000003
ENCRYPTION_KEY=your-encryption-key

📞 Support & Resources

ZATCA Resources:

Developer Portal: https://sandbox.zatca.gov.sa/
Documentation: https://zatca.gov.sa/en/E-Invoicing/
Support Email: info@zatca.gov.sa
Phone: 19993 (Local), +966112048998 (International)

Technical Support:

SDK Download: https://zatca.gov.sa/en/E-Invoicing/SystemsDevelopers/
API Documentation: Available on Developer Portal
Compliance Toolbox: Web-based validator available

🎯 Success Metrics

Key Performance Indicators:

Compliance Rate: >99% of invoices successfully submitted
Submission Time: <2 seconds average
Error Rate: <1% of submissions
CSID Uptime: 100% (no expired CSIDs)
Sequence Integrity: 0 gaps in invoice counter

Monthly Review:

Review failed submissions
Analyze error patterns
Update error handling
Optimize performance
Train staff on issues

📝 Next Steps

Immediate Actions:

Add vat_number field to Tenant model
Implement ECDSA signing
Move counter logic to model save
Create basic test suite

This Week:

Test in ZATCA sandbox
Implement error handling
Add UI enhancements
Create user documentation

This Month:

Implement PDF/A-3 generation
Add Arabic language support
Complete onboarding flow
Prepare for production

Document Version: 1.0
Last Updated: October 27, 2025
Status: Implementation Complete - Enhancements Recommended

18 KiB Raw Permalink Blame History

ZATCA E-Invoice Enhancement Recommendations

Executive Summary

✅ What Has Been Implemented

Phase 1: Generation Phase (Mandatory since Dec 4, 2021)

Phase 2: Integration Phase (Mandatory from Jan 1, 2023 in waves)

🔧 Critical Enhancements Needed

1. Cryptographic Signing (ECDSA) - HIGH PRIORITY ⚠️

2. Tenant VAT Number Field - HIGH PRIORITY ⚠️

3. PDF/A-3 Generation - MEDIUM PRIORITY

4. Invoice Counter Auto-Increment - HIGH PRIORITY ⚠️

5. Arabic Language Support - MEDIUM PRIORITY

6. Offline Mode Support - MEDIUM PRIORITY

7. CSID Onboarding Integration - HIGH PRIORITY ⚠️

8. Enhanced Error Handling - MEDIUM PRIORITY

9. Compliance Dashboard - LOW PRIORITY

10. Testing & Validation - HIGH PRIORITY ⚠️

📋 Implementation Priority Matrix

Immediate (Week 1)

Short-term (Week 2-3)

Medium-term (Month 1-2)

🔒 Security Recommendations

1. CSID Secret Storage

2. Private Key Protection

3. API Authentication

📊 Monitoring & Alerting

Recommended Metrics to Track:

Recommended Tools:

🧪 Testing Strategy

1. Unit Tests

2. Integration Tests

3. Load Tests

📱 User Interface Enhancements

1. Invoice Form

2. Invoice Detail Page

3. Compliance Dashboard

🌐 Arabic Language Implementation

Required Changes:

🔄 Workflow Improvements

1. Automated Clearance for Standard Invoices

2. Scheduled Reporting for Simplified Invoices

📦 Required Dependencies

🚀 Deployment Checklist

Before Production:

Environment Variables:

📞 Support & Resources

ZATCA Resources:

Technical Support:

🎯 Success Metrics

Key Performance Indicators:

Monthly Review:

📝 Next Steps

Immediate Actions:

This Week:

This Month:

18 KiB

Raw Permalink Blame History