🚀 Automated Serverless Data Pipeline (ASDP)

📖 Description

The Automated Serverless Data Pipeline (ASDP) is an event-driven data ingestion system built using Infrastructure as Code (IaC) with Terraform. The pipeline automatically processes CSV datasets uploaded to an Amazon S3 bucket by triggering an AWS Lambda function written in Python.

The Lambda function parses and transforms the CSV data, then stores the processed records in an Amazon DynamoDB table. The entire solution is designed to run locally using an AWS cloud emulator (Floci) for development and testing without deploying resources to AWS.

---

🏗️ Architecture

---

✨ Features

• Event-driven serverless architecture
• Infrastructure provisioning using Terraform
• Automatic S3 event triggers
• CSV parsing with Python AWS Lambda
• Data storage in Amazon DynamoDB
• Local AWS service emulation for offline development

---

🛠️ Prerequisites

Before getting started, ensure the following tools are installed on your system:

• Terraform
• AWS CLI
• Python 3.10
• PowerShell (Windows) or any terminal
• Floci (or LocalStack) - Local AWS service emulation

---

🚀 Installation

1. Clone the Repository

git clone https://github.com/chandra-64/ASDP
cd ASDP

2. Start the Local AWS Emulator

Launch Floci (or your preferred local AWS emulator) to make the mock AWS services available.

3. Deploy the Infrastructure

Navigate to the Terraform directory and provision the required resources.

cd terraform

terraform init

terraform apply -auto-approve

---

▶️ Usage

Step 1: Configure Local AWS Credentials

Set temporary environment variables to satisfy AWS CLI authentication requirements.

PowerShell

$env:AWS_ACCESS_KEY_ID="mock_key"
$env:AWS_SECRET_ACCESS_KEY="mock_secret"
$env:AWS_DEFAULT_REGION="us-east-1"

---

Step 2: Upload a CSV File

Upload the sample CSV dataset to the S3 bucket.

aws --endpoint-url=http://localhost:4566 s3 cp test_data.csv s3://data-pipeline-bucket/

Uploading the file automatically triggers the Lambda function, which processes the CSV file and stores the records in DynamoDB.

---

Step 3: Verify Processed Data

Confirm that the CSV records have been successfully inserted into DynamoDB.

aws --endpoint-url=http://localhost:4566 dynamodb scan --table-name ProcessedData

Example response:

---

🧹 Clean Up

When you are finished testing, destroy all locally provisioned resources.

terraform destroy -auto-approve

---

🛠️ Technologies Used

Technology	Purpose
Terraform	Infrastructure as Code (IaC)
AWS Lambda (Python)	Serverless data processing
Amazon S3	Object storage and event source
Amazon DynamoDB	NoSQL database
AWS CLI	Resource management
Floci / LocalStack	Local AWS service emulation

---

📄 License

This project is licensed under the MIT License.