Skip to content

DEV.md

Latest commit

 

History

History
215 lines (148 loc) · 5.59 KB

DEV.md

File metadata and controls

215 lines (148 loc) · 5.59 KB
Raw

Developer Guide

This guide provides information for contributors who want to build and test the Apache Beam SDK for Astra DB.

Prerequisites

Required

  • Java 11 or higher - The project is built with Java 11
  • Maven 3.6+ - Build tool for compiling and running tests
  • Astra DB Token - Required for running integration tests

Astra DB Token Setup

The integration tests require an Astra DB token to create and access test databases. You have two options to provide this token:

Option 1: Environment Variable (Recommended)

Set the ASTRA_DB_APPLICATION_TOKEN environment variable:

# Linux/macOS
export ASTRA_DB_APPLICATION_TOKEN="AstraCS:your-token-here"

# Windows (Command Prompt)
set ASTRA_DB_APPLICATION_TOKEN=AstraCS:your-token-here

# Windows (PowerShell)
$env:ASTRA_DB_APPLICATION_TOKEN="AstraCS:your-token-here"

Option 2: Astra CLI

Install and configure the Astra CLI:

# Install Astra CLI
curl -Ls "https://dtsx.io/get-astra-cli" | bash

# Configure with your token
astra setup

# Verify configuration
astra db list

The Astra CLI stores your token in ~/.astrarc which the tests will automatically detect.

Token Lookup Order

The test framework looks for the token in this order:

  1. ~/.astrarc configuration file (if Astra CLI is installed)
  2. ASTRA_DB_APPLICATION_TOKEN environment variable

If neither is found, tests will fail with:

IllegalStateException: Create environment variable ASTRA_DB_APPLICATION_TOKEN with your token

Getting an Astra DB Token

  1. Log in to Astra DB
  2. Navigate to your organization settings
  3. Go to "Token Management"
  4. Create a new token with "Database Administrator" role
  5. Copy the token (starts with AstraCS:)

Building the Project

Compile Only

mvn clean compile

Run Unit Tests Only

Unit tests don't require Astra DB connection:

mvn test -Dtest=SimpleRateLimiterTest

Run All Tests (Including Integration Tests)

Integration tests will automatically create a test database in your Astra account:

mvn clean test

Note: Integration tests will:

  • Create a temporary Astra database (if needed)
  • Run tests against the database
  • The database may remain after tests (check your Astra console)

Skip Tests

mvn clean install -DskipTests

Generate Javadoc

mvn javadoc:javadoc

The generated documentation will be in target/site/apidocs/.

Project Structure

beam-sdks-java-io-astra/
├── src/
│   ├── main/
│   │   └── java/
│   │       └── org/apache/beam/sdk/io/astra/
│   │           └── db/
│   │               ├── AstraDbIO.java          # Main IO connector
│   │               ├── ReadFn.java             # Read operations
│   │               ├── CqlSessionHolder.java   # Session management
│   │               ├── mapping/                # Entity mapping
│   │               ├── options/                # Configuration options
│   │               │   ├── RateLimiter.java    # Rate limiting interface
│   │               │   └── SimpleRateLimiter.java
│   │               ├── transforms/             # Data transformations
│   │               └── utils/                  # Utilities
│   └── test/
│       └── java/
│           └── org/apache/beam/sdk/io/astra/
│               ├── AbstractAstraTest.java      # Base test class
│               ├── AstraDbIOTest.java          # Integration tests
│               └── db/options/
│                   └── SimpleRateLimiterTest.java
├── pom.xml                                     # Maven configuration
└── README.md                                   # User documentation

Code Style

  • Follow existing code formatting
  • Use 2-space indentation
  • Add Javadoc for public APIs
  • Include unit tests for new features
  • Ensure all tests pass before submitting PR

Common Issues

Tests Fail with "Token not found"

Solution: Set up your Astra token using one of the methods above.

Tests Fail with "Database creation timeout"

Solution: Check your Astra account limits. Free tier has limits on concurrent database creation.

Compilation Errors with AutoValue

Solution: Run mvn clean compile to regenerate AutoValue classes.

Javadoc Warnings

Solution: Ensure all public classes, methods, and fields have proper Javadoc comments.

Running Specific Tests

# Run a specific test class
mvn test -Dtest=AstraDbIOTest

# Run a specific test method
mvn test -Dtest=AstraDbIOTest#test01ReadWithTable

# Run multiple test classes
mvn test -Dtest=AstraDbIOTest,SimpleRateLimiterTest

Debugging

Enable Debug Logging

Add to src/test/resources/logback-test.xml:

<logger name="org.apache.beam.sdk.io.astra" level="DEBUG"/>

Debug in IDE

  1. Import project as Maven project
  2. Set breakpoints in code
  3. Run tests in debug mode
  4. Ensure ASTRA_DB_APPLICATION_TOKEN is set in run configuration

Contributing

  1. Fork the repository
  2. Create a feature branch
  3. Make your changes
  4. Add/update tests
  5. Ensure all tests pass
  6. Update documentation
  7. Submit a pull request

License

This project is licensed under the Apache License 2.0. See LICENSE file for details.

Support