HBASE-30087: Create a nightly Jenkins test for the Read-Replica feature

[kgeisz]

HBASE-30087: Create a nightly Jenkins test for the Read-Replica feature

Note: Some code in the pull request was written using Claude Opus 4.6.

Introduction

• Apache HBase Read Replica Cluster Feature
• Read Replica Test Plan Document
• HBase Website Documentation for Read Replica Cluster

HBase's Read-Replica feature was merged into master in PR #8364. This pull request introduces integration tests for Read-Replica via Jenkins and Docker containers. Jenkins and Docker containers have been useful for testing this feature because it works around the META_TABLE_NAME issue mentioned in HBASE-29691 and PR #7730.

In a Read-Replica setup, clusters share the same storage location and need to have different META_TABLE_NAMEs. The table names are distinguished using the hbase.meta.table.suffix configuration property. However, this still leads to problems when using MiniHBaseCluster to test a multi-cluster Read-Replica setup because these clusters run in the same JVM and end up sharing the same static META_TABLE_NAME variable.

How It Works

• A Read-Replica stage has been added to the dev-support/Jenkinsfile.
• The dev-support/read-replica directory contains various files for building a Docker image, running containers with HBase procecces, and running integration test scripts written in Python. It also contains files and directories used by HBase, such as conf directories.
• The hbase_docker_client.py file is the most important Python file. Every test script uses this file to communicate with each hbase-docker container. It does so by using docker exec to run commands in a container's HBase shell.

The tests are triggered within a stage in HBase's nighly Jenkins run. The stage starts by cloning the current HBase repository to dev-support/read-replica. From here, the repo gets used to build a Docker image designed to run HBase in a Read-Replica setup. After the image has been built, two Docker containers are started, where one starts as the active cluster (read-write mode), and the other container starts as a replica cluster (read-only mode). These clusters share a data-store directory containing the hbase.rootdir. It is a mounted volume between each container and the local filesystem, and it is created by the Python scripts before container startup. The directory is also given 777 permissions in order to avoid HBase startup failures. There are other volumes as well for easy access, such as each container's conf directory, a utils directory for bulkloading data, and a logs directory for preserving HBase logs from each container.

Once the containers are up and running, a series of Python scripts are run as integration tests. They test expected behavior for a Read-Replica cluster setup, such as verifying valid/invalid startup, blocking writes on replica clusters, being able to refresh meta and HFiles on replica clusters to make them consistent with the active cluster, read-only flag flipping (changing hbase.global.readonly.enabled via dynamic configuration), verifying bug fixes, etc. If the Jenkins stage finishes successfully or a failure occurs at any point, then the HBase logs are saved to the Jenkins output directory, the containers are stopped, and the Docker image is deleted.

Other Information

• docker-compose.yml defines and configures two hbase-docker containers capable of running in a read-replica setup.
• build_images.sh uses Dockerfile to build an hbase-docker image. This script assumes there is an hbase repo in the same directory.
• .env defines environment variables used by docker-compose.yml, build_images.sh, and Python scripts.
• proto_compiler.py copies ActiveClusterSuffix.proto from the hbase repo and compiles it. The generated output is used for verifying the active.cluster.suffix.id file.
• utils is a directory containing scripts for bulkloading data into HBase.
• The cluster1 and cluster2 directories contain cluster-specific conf files and HBase log output directories. They each contain their own hbase-site.xml, log4j2.properties, and zoo.cfg files. The directories are defined as mounted volumes in docker-compose.yml. This is especially useful for changing the value of hbase.global.readonly.enabled in order to change a cluster's read-only mode. The mounted logs directory is useful because it helps preserve the HBase logs so they can be copied to the Jenkins output directory if a test failure occurs.

[Apache9]

Better add this to nightly or integration tests. Typically we do not require integration test when merging a PR.

[anmolnar]

lgtm.

[kgeisz]

Hi @Apache9, thanks for the feedback! I updated the PR by adding a read-replica stage to dev-support/Jenkinsfile. I removed the GitHub Action workflow file as well. Unfortunately, I am not an HBase committer, so I cannot test this file in an upstream Jenkins sandbox. I did, however, test the read-replica stage using a local Jenkins Docker container. The Jenkinsfile I used was a tuncated version of the upstream Jenkinsfile that only has code relevent to the read-replica test stage (I left out Yetus stages, etc.).

Is there any way this updated Jenkinsfile can be tested with upstream Jenkins before merging the PR?
cc. @anmolnar @taklwu @ndimiduk @apurtell

[anmolnar]

Still looks good to me.

[Copilot]

🟡 Changes recommended

Malformed ZooKeeper configuration, version-dependent image packaging, ineffective timeouts, and missing license headers currently prevent reliable nightly execution.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds nightly Docker-based integration coverage for HBase Read-Replica clusters.

Changes:

• Adds a two-cluster Docker test environment.
• Adds Python tests for replication, read-only transitions, bulk loading, and region splits.
• Integrates execution and artifact collection into Jenkins nightly builds.

File summaries

File	Description
dev-support/Jenkinsfile	Adds the nightly Read-Replica stage.
dev-support/hbase_nightly_read_replica_test.sh	Orchestrates build, tests, and cleanup.
dev-support/read-replica/.env	Defines test environment settings.
dev-support/read-replica/Dockerfile	Builds the HBase test image.
dev-support/read-replica/build-images.sh	Invokes the Docker build.
dev-support/read-replica/docker-compose.yml	Defines both HBase containers.
dev-support/read-replica/requirements.txt	Pins Python dependencies.
dev-support/read-replica/cluster1/conf/hbase-site.xml	Configures the active cluster.
dev-support/read-replica/cluster1/conf/log4j2.properties	Configures active-cluster logging.
dev-support/read-replica/cluster1/conf/zoo.cfg	Configures active-cluster ZooKeeper.
dev-support/read-replica/cluster2/conf/hbase-site.xml	Configures the replica cluster.
dev-support/read-replica/cluster2/conf/log4j2.properties	Configures replica-cluster logging.
dev-support/read-replica/cluster2/conf/zoo.cfg	Configures replica-cluster ZooKeeper.
dev-support/read-replica/python/__init__.py	Defines the Python package.
dev-support/read-replica/python/proto/proto_compiler.py	Generates Python protobuf bindings.
dev-support/read-replica/python/scripts/__init__.py	Defines the scripts package.
dev-support/read-replica/python/scripts/verify_hbase_start.py	Verifies cluster readiness.
dev-support/read-replica/python/scripts/test_create_drop_behavior.py	Tests table creation and deletion.
dev-support/read-replica/python/scripts/test_put_get_delete_behavior.py	Tests data mutation and refresh behavior.
dev-support/read-replica/python/scripts/test_read_only_flag_flipping.py	Tests repeated role transitions.
dev-support/read-replica/python/scripts/test_dual_active_cluster_startup.py	Tests rejection of dual-active startup.
dev-support/read-replica/python/scripts/test_cannot_promote_second_active_cluster.py	Tests rejection of a second promotion.
dev-support/read-replica/python/scripts/test_bulkloaded_data_and_region_splits.py	Tests bulk loads and region splits.
dev-support/read-replica/python/src/__init__.py	Exports shared Python helpers.
dev-support/read-replica/python/src/environment_loader.py	Loads required environment values.
dev-support/read-replica/python/src/hbase_docker_client.py	Implements Docker and HBase shell operations.
dev-support/read-replica/python/src/logger_config.py	Centralizes test logging.
dev-support/read-replica/python/src/utils.py	Provides shared test workflows.
dev-support/read-replica/utils/bulkload.sh	Generates and bulk-loads HFiles.
dev-support/read-replica/utils/tsv_generator.py	Generates bulk-load TSV data.

Review details

• Files reviewed: 30/30 changed files
• Comments generated: 11
• Review effort level: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[taklwu]

@Apache9 @ndimiduk sorry for pinging you guys directly, how do we add a nightly Jenkins job after this PR merge? is it automated?

[taklwu]

LGTM

[taklwu]

nit: do we really need this init file ? I saw few of them are empty with just the ASF header

[taklwu]

nit: this itself could be a single commit to start the container with hbase cluster and client

[kgeisz]

I made some changes to how the tests are run. The tests are now run with pytest, which allows us to generate xml and html test results for Jenkins. Here are some screenshots from my local Jenkins deployment to give an idea of what the results looks like:

[taklwu]

@kgeisz let's learnt from #8601.

1. create a jenkins feature branch , review and merge this PR
2. create the jenkins job based on the feature branch , keep monitoring for 1 weeks or 2.
3. create the merge PR into master.

[kgeisz]

Changed the destination branch to HBASE-30087 so the Jenkins changes can be tested before they are merged into master.