Skip to content

scripts

Directory actions

More options

Directory actions

More options

Latest commit

 

History

History

scripts

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
conf
conf
 
 
deep_compare
deep_compare
 
 
schema
schema
 
 
.gitignore
.gitignore
 
 
README.md
README.md
 
 
dot.bigqueryrc
dot.bigqueryrc
 
 
export_ada_pots.sh
export_ada_pots.sh
 
 
export_block.sh
export_block.sh
 
 
export_block_hash.sh
export_block_hash.sh
 
 
export_collateral.sh
export_collateral.sh
 
 
export_datum.sh
export_datum.sh
 
 
export_delegation.sh
export_delegation.sh
 
 
export_epoch_param.sh
export_epoch_param.sh
 
 
export_epoch_stake.sh
export_epoch_stake.sh
 
 
export_ma_minting.sh
export_ma_minting.sh
 
 
export_meta.sh
export_meta.sh
 
 
export_param_proposal.sh
export_param_proposal.sh
 
 
export_pool_offline_data.sh
export_pool_offline_data.sh
 
 
export_pool_owner.sh
export_pool_owner.sh
 
 
export_pool_retire.sh
export_pool_retire.sh
 
 
export_pool_update.sh
export_pool_update.sh
 
 
export_redeemer.sh
export_redeemer.sh
 
 
export_rel_addr_txout.sh
export_rel_addr_txout.sh
 
 
export_rel_stake_hash.sh
export_rel_stake_hash.sh
 
 
export_rel_stake_txout.sh
export_rel_stake_txout.sh
 
 
export_reward.sh
export_reward.sh
 
 
export_schema_version.sh
export_schema_version.sh
 
 
export_script.sh
export_script.sh
 
 
export_stake_deregistration.sh
export_stake_deregistration.sh
 
 
export_stake_registration.sh
export_stake_registration.sh
 
 
export_tx.sh
export_tx.sh
 
 
export_tx_consumed_output.sh
export_tx_consumed_output.sh
 
 
export_tx_hash.sh
export_tx_hash.sh
 
 
export_tx_in_out.sh
export_tx_in_out.sh
 
 
export_tx_metadata.sh
export_tx_metadata.sh
 
 
export_withdrawal.sh
export_withdrawal.sh
 
 
fix_big_json_numbers.py
fix_big_json_numbers.py
 
 
functions.sh
functions.sh
 
 
run_bq_pg_compare.sh
run_bq_pg_compare.sh
 
 
run_bq_pg_deep_compare.sh
run_bq_pg_deep_compare.sh
 
 
run_bq_update.sh
run_bq_update.sh
 
 
run_update_epoch.sh
run_update_epoch.sh
 
 
split_long_lists.py
split_long_lists.py
 
 
test_split_long_lists.py
test_split_long_lists.py
 
 
update_block.sh
update_block.sh
 
 
update_block_hash.sh
update_block_hash.sh
 
 
update_collateral.sh
update_collateral.sh
 
 
update_datum.sh
update_datum.sh
 
 
update_delegation.sh
update_delegation.sh
 
 
update_epoch_ada_pots.sh
update_epoch_ada_pots.sh
 
 
update_epoch_delegation.sh
update_epoch_delegation.sh
 
 
update_epoch_epoch_param.sh
update_epoch_epoch_param.sh
 
 
update_epoch_epoch_stake.sh
update_epoch_epoch_stake.sh
 
 
update_epoch_epoch_stake_addr.sh
update_epoch_epoch_stake_addr.sh
 
 
update_epoch_epoch_stake_pool.sh
update_epoch_epoch_stake_pool.sh
 
 
update_epoch_ma_minting.sh
update_epoch_ma_minting.sh
 
 
update_epoch_param_proposal.sh
update_epoch_param_proposal.sh
 
 
update_epoch_pool_offline_data.sh
update_epoch_pool_offline_data.sh
 
 
update_epoch_pool_update.sh
update_epoch_pool_update.sh
 
 
update_epoch_rel_addr_txout.sh
update_epoch_rel_addr_txout.sh
 
 
update_epoch_rel_stake_txout.sh
update_epoch_rel_stake_txout.sh
 
 
update_epoch_reward.sh
update_epoch_reward.sh
 
 
update_epoch_reward_addr.sh
update_epoch_reward_addr.sh
 
 
update_epoch_reward_pool.sh
update_epoch_reward_pool.sh
 
 
update_ma_minting.sh
update_ma_minting.sh
 
 
update_pool_offline_data.sh
update_pool_offline_data.sh
 
 
update_pool_owner.sh
update_pool_owner.sh
 
 
update_pool_retire.sh
update_pool_retire.sh
 
 
update_pool_update.sh
update_pool_update.sh
 
 
update_redeemer.sh
update_redeemer.sh
 
 
update_rel_addr_txout.sh
update_rel_addr_txout.sh
 
 
update_rel_stake_hash.sh
update_rel_stake_hash.sh
 
 
update_rel_stake_txout.sh
update_rel_stake_txout.sh
 
 
update_script.sh
update_script.sh
 
 
update_stake_deregistration.sh
update_stake_deregistration.sh
 
 
update_stake_registration.sh
update_stake_registration.sh
 
 
update_tx.sh
update_tx.sh
 
 
update_tx_consumed_output.sh
update_tx_consumed_output.sh
 
 
update_tx_hash.sh
update_tx_hash.sh
 
 
update_tx_in_out.sh
update_tx_in_out.sh
 
 
update_tx_metadata.sh
update_tx_metadata.sh
 
 
update_withdrawal.sh
update_withdrawal.sh
 
 

README.md

Scripts

This folder contains the various scripts that are used in the BigQuery project. There are update, validation and export scripts. Most of the secrets/config that are used in the scripts are passed through environment variables in json format.

Update scripts

In this folder you will find all the scripts used for syncing (or "updating") of the data between db-sync and BigQuery. Most of the scripts are bash scripts. The scripts can be split into 3 categories:

  • The scripts that update a table at the slot level. All the scripts with the naming convention update_<table_name>.sh fall into this category.
  • The scripts that update a table at the epoch level. Certain tables need to be updated only every epoch. One example would be the rewards table, as we have new rewards only every new epoch. The scripts in this category have the naming conventions of update_epoch_<table_name>.sh.
  • The aggregate scripts: these are the ones that call the scripts of the previous 2 categories: for example to run the update after an epoch boundary for all the tables. The scripts run_bq_update.sh and run_update_epoch.sh fall into this category.

Validation scripts

The validation scripts can be found in the deep_compare folder. These scripts validate that the data in BigQuery have a one to one correspondence with the data in db-sync. They perform a deep comparison of the data using hashing.

Export scripts

The export scripts follow the naming pattern: export_<table_name>.sh. They are essentially backfill scripts used to populate in bulk the BigQuery tables. They are usually updating a table with all the data until a recent epoch.