Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 12 additions & 19 deletions examples/cos-sdk/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,40 +17,33 @@ checked out version.
### IBM COS SDK configuration

The scripts expect the following values:
* shared credentials file `~/.bluemix/cos_credentials` or target file from `COS_CREDENTIALS_FILE` environment variable
* `IBM_API_KEY_ID` environment variable set to the `apikey` from your COS service credentials
* `IBM_SERVICE_INSTANCE_ID` environment variable set to the `resource_instance_id` from your COS service credentials
* `CLOUDANT_IAM_API_KEY` environment variable set to API key with permission to the Cloudant instance
* (optional) `CLOUDANT_IAM_TOKEN_URL` environment variable set to the URL of token endpoint (defaults to `https://iam.cloud.ibm.com`)

#### IBM COS

When using IBM Cloud Object Storage create a service credential with __disabled__ `Include HMAC Credential` option.

Copy the credentials into `~/.bluemix/cos_credentials` or generate it using the `ibmcloud` CLI tool:
To retrieve your COS credentials using the `ibmcloud` CLI tool:
```bash
ibmcloud resource service-key-create <credentials-name> --instance-name <cos-instance-name>

ibmcloud resource service-key <credentials-name> --output JSON | jq '.[].credentials' > ~/.bluemix/cos_credentials
ibmcloud resource service-key <credentials-name> --output JSON | jq '.[].credentials'
```
More info on generating the credentials:
https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-service-credentials

#### Service Credentials file structure:
```json
{
"apikey": "<API_KEY>",
"endpoints": "https://control.cloud-object-storage.cloud.ibm.com/v2/endpoints",
"iam_apikey_description": "Auto-generated for key crn:v1:...f9d5b",
"iam_apikey_id": "ApiKey-6f...b1",
"iam_apikey_name": "<NAME>",
"iam_role_crn": "...Writer",
"iam_serviceid_crn": "crn:v1:...",
"resource_instance_id": "crn:v1:..."
}
From the output, set the environment variables:
```bash
export IBM_API_KEY_ID=<apikey value>
export IBM_SERVICE_INSTANCE_ID=<resource_instance_id value>
```

#### IBM COS

Run the scripts with the `--cos_url` option pointing to your COS instance S3 endpoint.
Run the scripts with the `--cos_url` option pointing to your COS instance S3 endpoint and `--cos_region` set to the region of your bucket (e.g. `us-south`, `eu-de`).

Corresponding endpoint URLs can be found under the link found in the Service Credentials file or on the IBM Cloud UI (`endpoints` field).

Expand All @@ -68,7 +61,7 @@ The source database and destination bucket are required options.
The minimum needed to run the backup scripts are thus:

```bash
node cos-backup-file.js -s 'https://~replaceWithYourUniqueHost~.cloudantnosqldb.appdomain.cloud/sourcedb' -b 'examplebucket' --cos_url 's3.eu-de.cloud-object-storage.appdomain.cloud'
node cos-backup-file.js -s 'https://~replaceWithYourUniqueHost~.cloudantnosqldb.appdomain.cloud/sourcedb' -b 'examplebucket' --cos_url 's3.eu-de.cloud-object-storage.appdomain.cloud' --cos_region 'eu-de'
```

The object created in the bucket for the backup file will be
Expand All @@ -88,7 +81,7 @@ The target database URL, source bucket, and backup object name are required opti
The minimum needed to run the restore scripts are thus:

```bash
node cos-restore-file.js -t 'https://~replaceWithYourUniqueHost~.cloudantnosqldb.appdomain.cloud/targetdb' -b 'examplebucket' -o 'couchbackup-sourcedb-2024-01-25T09:45:11.730Z' --cos_url 's3.eu-de.cloud-object-storage.appdomain.cloud'
node cos-restore-file.js -t 'https://~replaceWithYourUniqueHost~.cloudantnosqldb.appdomain.cloud/targetdb' -b 'examplebucket' -o 'couchbackup-sourcedb-2024-01-25T09:45:11.730Z' --cos_url 's3.eu-de.cloud-object-storage.appdomain.cloud' --cos_region 'eu-de'
```

## Progress and debug
Expand All @@ -97,7 +90,7 @@ To see detailed progress of the backup/restore and upload/download or additional
use the `DEBUG` environment variable with label `couchbackup-cos` e.g.

```bash
DEBUG='couchbackup-cos' node cos-backup-file.js -s 'https://~replaceWithYourUniqueHost~.cloudantnosqldb.appdomain.cloud/sourcedb' -b 'couchbackup-example' --cos_url "s3.eu-de.cloud-object-storage.appdomain.cloud"
DEBUG='couchbackup-cos' node cos-backup-file.js -s 'https://~replaceWithYourUniqueHost~.cloudantnosqldb.appdomain.cloud/sourcedb' -b 'couchbackup-example' --cos_url "s3.eu-de.cloud-object-storage.appdomain.cloud" --cos_region "eu-de"
```

```
Expand Down
32 changes: 17 additions & 15 deletions examples/cos-sdk/cos-backup-file.js
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,8 @@
// The script generates the backup object name by combining together the path
// part of the database URL and the current time.

const IBM_COS = require('ibm-cos-sdk');
const { S3Client, HeadBucketCommand } = require('ibm-cos-sdk-v2');
const { Upload } = require('@ibm-cos/lib-storage');
const fs = require('fs');
const VError = require('verror');
const tmp = require('tmp');
Expand All @@ -38,6 +39,7 @@ function main() {
bucket: { alias: 'b', nargs: 1, demandOption: true, describe: 'Destination bucket' },
prefix: { alias: 'p', nargs: 1, describe: 'Prefix for backup object key', default: 'couchbackup' },
cos_url: { nargs: 1, demandOption: true, describe: 'IBM COS S3 endpoint URL' }, // An endpoint from 'endpoints' list in cos_credentials
cos_region: { nargs: 1, demandOption: true, describe: 'IBM COS S3 endpoint region' },
shallow: { describe: 'Backup the documents winning revisions only', type: 'boolean' }
})
.help('h').alias('h', 'help')
Expand All @@ -49,23 +51,23 @@ function main() {
const backupName = new url.URL(sourceUrl).pathname.split('/').filter(function(x) { return x; }).join('-');
const backupKeyPrefix = `${argv.prefix}-${backupName}`;
const backupKey = `${backupKeyPrefix}-${new Date().toISOString()}`;
const cosEndpoint = argv.cos_url;
const cosEndpoint = argv.cos_url.startsWith('https://') ? argv.cos_url : `https://${argv.cos_url}`;
const cosRegion = argv.cos_region;
const cloudantApiKey = process.env.CLOUDANT_IAM_API_KEY;
const mode = argv.shallow ? 'shallow' : 'full';
const backupTmpFile = tmp.fileSync();

/*
* Creds are from ~/.bluemix/cos_credentials, generated by ibmcloud CLI tool
* Creds are from IBM_API_KEY_ID and IBM_SERVICE_INSTANCE_ID environment variables.
* See: https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-service-credentials
*
* corresponding 'endpoint' can be found on IBM Cloud UI at the COS instance,
* or under the 'endpoints' link provided in the generated file (~/.bluemix/cos_credentials)
* corresponding 'endpoint' and 'region' can be found on IBM Cloud UI at the COS instance.
* */
const config = {
endpoint: cosEndpoint,
credentials: new IBM_COS.SharedJSONFileCredentials(),
region: cosRegion,
};
const COS = new IBM_COS.S3(config);
const COS = new S3Client(config);
debug(`Creating a new backup of ${sourceUrl} at ${backupBucket}/${backupKey}...`);
bucketAccessible(COS, backupBucket)
.then(() => {
Expand All @@ -87,15 +89,15 @@ function main() {
* Return a promise that resolves if the bucket is available and
* rejects if not.
*
* @param {IBM_COS.S3} s3 IBM COS S3 client object
* @param {S3Client} s3 IBM COS S3 client object
* @param {any} bucketName Bucket name
* @returns Promise
*/
function bucketAccessible(s3, bucketName) {
const params = {
Bucket: bucketName
};
return s3.headBucket(params).promise()
return s3.send(new HeadBucketCommand(params))
.then(() => { debug('Bucket is accessible'); })
.catch((reason) => {
console.error(reason);
Expand Down Expand Up @@ -145,7 +147,7 @@ function createBackupFile(sourceUrl, backupTmpFilePath, cloudantApiKey, mode) {
/**
* Upload a backup file to an IBM COS bucket.
*
* @param {IBM_COS.S3} cos Object store client
* @param {S3Client} cos Object store client
* @param {any} backupTmpFilePath Path of backup file to write.
* @param {any} bucket Object store bucket name
* @param {any} key Object store key name
Expand All @@ -160,17 +162,17 @@ function uploadNewBackup(cos, backupTmpFilePath, bucket, key) {
Key: key,
Body: inputStream
};
const options = {
const upload = new Upload({
client: cos,
params: params,
partSize: 5 * 1024 * 1024, // max 5 MB part size (minimum size)
queueSize: 5 // allow 5 parts at a time
};

const upload = cos.upload(params, options);
});
upload.on('httpUploadProgress', (progress) => {
debug(`IBM COS S3 upload progress: ${JSON.stringify(progress)}`);
});

return upload.promise()
return upload.done()
.then(() => {
debug('Upload succeeded!');
})
Expand Down
29 changes: 16 additions & 13 deletions examples/cos-sdk/cos-backup-stream.js
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,8 @@
// part of the database URL and the current time.

const stream = require('stream');
const IBM_COS = require('ibm-cos-sdk');
const { S3Client, HeadBucketCommand } = require('ibm-cos-sdk-v2');
const { Upload } = require('@ibm-cos/lib-storage');
const VError = require('verror').VError;
const couchbackup = require('@cloudant/couchbackup');
const debug = require('debug')('couchbackup-cos');
Expand All @@ -37,6 +38,7 @@ function main() {
bucket: { alias: 'b', nargs: 1, demandOption: true, describe: 'Destination bucket' },
prefix: { alias: 'p', nargs: 1, describe: 'Prefix for backup object key', default: 'couchbackup' },
cos_url: { nargs: 1, demandOption: true, describe: 'IBM COS S3 endpoint URL' }, // An endpoint from 'endpoints' list in cos_credentials
cos_region: { nargs: 1, demandOption: true, describe: 'IBM COS S3 endpoint region' },
shallow: { describe: 'Backup the documents winning revisions only', type: 'boolean' }
})
.help('h').alias('h', 'help')
Expand All @@ -48,22 +50,22 @@ function main() {
const backupName = new url.URL(sourceUrl).pathname.split('/').filter(function(x) { return x; }).join('-');
const backupKeyPrefix = `${argv.prefix}-${backupName}`;
const backupKey = `${backupKeyPrefix}-${new Date().toISOString()}`;
const cosEndpoint = argv.cos_url;
const cosEndpoint = argv.cos_url.startsWith('https://') ? argv.cos_url : `https://${argv.cos_url}`;
const cosRegion = argv.cos_region;
const cloudantApiKey = process.env.CLOUDANT_IAM_API_KEY;
const mode = argv.shallow ? 'shallow' : 'full';

/*
* Creds are from ~/.bluemix/cos_credentials, generated by ibmcloud CLI tool
* Creds are from IBM_API_KEY_ID and IBM_SERVICE_INSTANCE_ID environment variables.
* See: https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-service-credentials
*
* corresponding 'endpoint' can be found on IBM Cloud UI at the COS instance,
* or under the 'endpoints' link provided in the generated file (~/.bluemix/cos_credentials)
* corresponding 'endpoint' and 'region' can be found on IBM Cloud UI at the COS instance.
* */
const config = {
endpoint: cosEndpoint,
credentials: new IBM_COS.SharedJSONFileCredentials(),
region: cosRegion,
};
const COS = new IBM_COS.S3(config);
const COS = new S3Client(config);
debug(`Creating a new backup of ${sourceUrl} at ${backupBucket}/${backupKey}...`);
bucketAccessible(COS, backupBucket)
.then(() => {
Expand All @@ -82,15 +84,15 @@ function main() {
* Return a promise that resolves if the bucket is available and
* rejects if not.
*
* @param {IBM_COS.S3} s3 IBM COS S3 client object
* @param {S3Client} s3 IBM COS S3 client object
* @param {any} bucketName Bucket name
* @returns Promise
*/
function bucketAccessible(s3, bucketName) {
const params = {
Bucket: bucketName
};
return s3.headBucket(params).promise()
return s3.send(new HeadBucketCommand(params))
.then(() => { debug('Bucket is accessible'); })
.catch((reason) => {
console.error(reason);
Expand Down Expand Up @@ -123,11 +125,12 @@ function backupToS3(sourceUrl, s3Client, s3Bucket, s3Key, cloudantApiKey, mode)
Key: s3Key,
Body: streamToUpload
};
const options = {
const upload = new Upload({
client: s3Client,
params: params,
partSize: 5 * 1024 * 1024, // max 5 MB part size (minimum size)
queueSize: 5 // allow 5 parts at a time
};
const upload = s3Client.upload(params, options);
});
upload.on('httpUploadProgress', (progress) => {
debug(`IBM COS upload progress: ${JSON.stringify(progress)}`);
});
Expand Down Expand Up @@ -155,7 +158,7 @@ function backupToS3(sourceUrl, s3Client, s3Bucket, s3Key, cloudantApiKey, mode)
.on('changes', batch => debug('Couchbackup changes batch: ', batch))
.on('written', progress => debug('Fetched batch:', progress.batch, 'Total document revisions written:', progress.total, 'Time:', progress.time))
);
return Promise.all([backupPromise, upload.promise()]);
return Promise.all([backupPromise, upload.done()]);
}

main();
33 changes: 20 additions & 13 deletions examples/cos-sdk/cos-restore-file.js
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
// Small script which restores a Cloudant or CouchDB database from an IBM Cloud Object Storage (COS)
// bucket using an intermediary file on disk

const IBM_COS = require('ibm-cos-sdk');
const { S3Client, HeadObjectCommand, GetObjectCommand } = require('ibm-cos-sdk-v2');
const VError = require('verror');
const couchbackup = require('@cloudant/couchbackup');
const debug = require('debug')('couchbackup-cos');
Expand All @@ -33,24 +33,26 @@ function main() {
bucket: { alias: 'b', nargs: 1, demandOption: true, describe: 'Source bucket containing backup' },
object: { alias: 'o', nargs: 1, demandOption: true, describe: 'Backup Object name in IBM COS' },
cos_url: { nargs: 1, demandOption: true, describe: 'IBM COS S3 endpoint URL' },
cos_region: { nargs: 1, demandOption: true, describe: 'IBM COS S3 endpoint region' },
})
.help('h').alias('h', 'help')
.epilog('Copyright (C) IBM 2025')
.argv;

const restoreBucket = argv.bucket;
const objectKey = argv.object;
const cosEndpoint = argv.cos_url;
const cosEndpoint = argv.cos_url.startsWith('https://') ? argv.cos_url : `https://${argv.cos_url}`;
const cosRegion = argv.cos_region;
const targetUrl = argv.target;

const cloudantApiKey = process.env.CLOUDANT_IAM_API_KEY;
const restoreTmpFile = tmp.fileSync();

const config = {
endpoint: cosEndpoint,
credentials: new IBM_COS.SharedJSONFileCredentials(),
region: cosRegion,
};
const COS = new IBM_COS.S3(config);
const COS = new S3Client(config);
restoreProcess(COS, restoreBucket, objectKey, targetUrl, cloudantApiKey, restoreTmpFile.name)
.then(() => {
debug('Restore completed successfully');
Expand All @@ -73,7 +75,7 @@ async function restoreProcess(COS, restoreBucket, objectKey, targetUrl, cloudant

/**
* Check if object is accessible in COS
* @param {IBM_COS.S3} s3
* @param {S3Client} s3
* @param {string} bucketName
* @param {string} objectKey
*/
Expand All @@ -83,7 +85,7 @@ async function objectAccessible(s3, bucketName, objectKey) {
Bucket: bucketName,
};
try {
await s3.headObject(params).promise();
await s3.send(new HeadObjectCommand(params));
debug(`Object '${objectKey}' is accessible`);
} catch (reason) {
debug(reason);
Expand All @@ -93,20 +95,25 @@ async function objectAccessible(s3, bucketName, objectKey) {

/**
* Download backup file from COS to local temporary file
* @param {IBM_COS.S3} COS
* @param {S3Client} COS
* @param {string} restoreBucket
* @param {string} objectKey
* @param {string} restoreTmpFilePath
*/
async function createRestoreFile(COS, restoreBucket, objectKey, restoreTmpFilePath) {
debug(`Downloading from ${restoreBucket}/${objectKey} to ${restoreTmpFilePath}`);

const inputStream = COS.getObject({
Bucket: restoreBucket,
Key: objectKey
}).createReadStream({
highWaterMark: 16 * 1024 * 1024 // 16MB buffer
});
let inputStream;
try {
const response = await COS.send(new GetObjectCommand({
Bucket: restoreBucket,
Key: objectKey
}));
inputStream = response.Body;
} catch (err) {
debug('COS GetObjectCommand error:', err);
throw new VError(err, 'Failed to fetch object from COS');
}

const outputStream = fs.createWriteStream(restoreTmpFilePath, {
highWaterMark: 16 * 1024 * 1024 // 16MB buffer
Expand Down
Loading