diff --git a/.github/workflows/notify-docs.yml b/.github/workflows/notify-docs.yml new file mode 100644 index 00000000..bf2d668d --- /dev/null +++ b/.github/workflows/notify-docs.yml @@ -0,0 +1,24 @@ +name: Notify bedbase docs + +# Tells databio/doc-hubs that this repo's docs changed, so docs.bedbase.org +# rebuilds. All the logic is in the reusable workflow; this file only names +# which site and which spoke. See databio/doc-hubs README, "Add a spoke". + +on: + push: + # Must equal this spoke's `ref` in hubs/bedbase/spokes.toml: the hub builds + # that ref, so notifying from any other branch would rebuild the same bytes. + branches: [master] + paths: + - 'docs/guide/**' # the spoke's docs tree, docs.yml included + - '.github/workflows/notify-docs.yml' # so a notifier fix proves itself + workflow_dispatch: + +jobs: + notify: + uses: databio/actions/.github/workflows/notify-doc-hub.yml@master + with: + site: bedbase + spoke: bedhost + secrets: + DOCS_DISPATCH_TOKEN: ${{ secrets.DOCS_DISPATCH_TOKEN }} diff --git a/docs/guide/README.md b/docs/guide/README.md new file mode 100644 index 00000000..18802e96 --- /dev/null +++ b/docs/guide/README.md @@ -0,0 +1,47 @@ +

bedhost

+ +[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black) +[![Github badge](https://img.shields.io/badge/source-github-354a75?logo=github)](https://github.com/databio/bedhost) + + +`bedhost` is a Python FastAPI module for the API that powers BEDbase. +It needs a path to the *bedbase configuration file*, which can be provided either via `-c`/`--config` argument or read from `$BEDBASE_CONFIG` environment variable. + +## Introduction + +You can find the formal OpenAPI documentation and interactive interface at . This document provides more conceptual introduction and explanations to how to use the API effectively. + + +## General API organization + +### Data types + +BEDbase stores two types of data, which we call *records*. They are 1. BEDs, and 2. BEDsets. BEDsets are simply collections of BEDs. Each record in the database is either a BED or a BEDset. + +### Endpoint organization + +The endpoints are divided into 3 groups: + +1. `/bed` endpoints are used to interact with metadata for BED records. +2. `/bedset` endpoints are used to interact with metadata for BEDset records. +3. `/objects` endpoints are used to download metadata and get URLs to retrieve the underlying data itself. These endpoints implement the [GA4GH DRS standard](https://ga4gh.github.io/data-repository-service-schemas/). + +Therefore, to get information and statistics about BED or BEDset records, or what is contained in the database, look through the `/bed` and `/bedset` endpoints. But if you need to write a tool that gets the actual underlying files, then you'll need to use the `/objects` endpoints. The type of identifiers used in each case differ. + +## Record identifiers vs. object identifiers + +Each record has an identifier. For example, `0000120fe8c5334bb0ce759dfcf06c3b` is a BED identifier. You can use this identifier for the metadata endpoints. To download files, you'll need something slightly different -- you need an *object identifier*. This is because each BED record includes multiple files, such as the original BED file, the BigBed file, analysis plots, and so on. To download a file, you will construct what we call the `object_id`, which identifies the specific file. + +### How to construct object identifiers + +Object IDs take the form `..`. An example of an object_id for a BED file is `bed.0000120fe8c5334bb0ce759dfcf06c3b.bed_file` + +So, you can get information about this object like this: + +`GET` [https://api.bedbase.org/v1/objects/bed.0000120fe8c5334bb0ce759dfcf06c3b.bed_file](https://api.bedbase.org/v1/objects/bed.0000120fe8c5334bb0ce759dfcf06c3b.bed_file) + +Or, you can get a URL to download the actual file with: + +`GET` [https://api.bedbase.org/v1/objects/bed.0000120fe8c5334bb0ce759dfcf06c3b.bed_file/access/http](https://api.bedbase.org/v1/objects/bed.0000120fe8c5334bb0ce759dfcf06c3b.bed_file/access/http) + + diff --git a/docs/guide/build_image.md b/docs/guide/build_image.md new file mode 100644 index 00000000..926ca632 --- /dev/null +++ b/docs/guide/build_image.md @@ -0,0 +1,38 @@ +# Build the container locally + +Running with `uvicorn` provides auto-reload. To configure, this assumes you have previously set up `databio/secrets`. + +1. Source `.env` file to populate the environment variables referenced in the configuration file. +2. Start `bedhost` using `uvicorn` and pass the configuration file via the `BEDBASE_CONFIG` env var. + + +```console +source environment/production.env +BEDBASE_CONFIG=deployment/config/api.bedbase.org.yaml uvicorn bedhost.main:app --reload +``` + +You can change the database you're connecting to by using a different config file: +- Using a local config: `BEDBASE_CONFIG=../bbconf/tests/data/config.yaml uvicorn bedhost.main:app --reload` +- With the dev database: `BEDBASE_CONFIG=deployment/config/api-dev.bedbase.org.yaml uvicorn bedhost.main:app --reload` + +Now, you can access the service at [http://127.0.0.1:8000](http://127.0.0.1:8000). Example endpoints: +- http://127.0.0.1:8000/v1/bed/bbad85f21962bb8d972444f7f9a3a932/metadata?full=true +- http://127.0.0.1:8000/v1/bed/bbad85f21962bb8d972444f7f9a3a932/metadata/plots?full=true +- http://127.0.0.1:8000/v1/objects/bed.bbad85f21962bb8d972444f7f9a3a932.chrombins +- http://127.0.0.1:8000/v1/bed/list?limit=10&offset=0 + + +## Running the server in Docker + +### Building image + +- Primary image: `docker build -t databio/bedhost -f Dockerfile .` +- Dev image `docker build -t databio/bedhost:dev -f dev.Dockerfile .` +- Test image: `docker build -t databio/bedhost:dev -f test.Dockerfile .` + +Existing images can be found [at dockerhub](https://hub.docker.com/r/databio/bedhost). + + +## Deploying updates automatically + +- [Deploying bedbase](./deployment.md). \ No newline at end of file diff --git a/docs/guide/changelog.md b/docs/guide/changelog.md new file mode 100644 index 00000000..a9476db7 --- /dev/null +++ b/docs/guide/changelog.md @@ -0,0 +1,180 @@ +# Changelog + +This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html) and [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) format. + +## [0.13.0] -- 2026-08-20 +### Added: +- `/v1/objects/exports` and `/v1/objects/files` endpoints listing published bulk-metadata exports and standalone analysis files [API] +- `BEDHOST_INIT_ML` environment variable to skip loading ML models [API] +### Changed: +- Statistics cache moved to app state (thread safe), with a fallback for empty datasets [API] +- Blocking endpoints run in a threadpool instead of stalling the event loop [API] +- Missing markdown files return 404 instead of 500 [API] +- Logging configured once, in `main.py` + +## [0.12.7] -- 2026-04-22 +### Changed: +- UMAP page improvements and fixed link previews [UI] +- Migrated to Starlette 1.0 and yacman 1.0 + +## [0.12.6] -- 2026-04-16 +### Fixed: +- Build fix + +## [0.12.5] -- 2026-04-15 +### Changed: +- UI updates and search improvements + +## [0.12.4] -- 2026-02-06 +### Fixed: +- Search by external id +### Changed: +- Updated UMAP UI [UI] + +## [0.12.3] -- 2026-01-21 + +## [0.12.2] -- 2026-01-21 +### Added: +- BED classifier and reference genome validator in the BED analyzer [UI] +### Changed: +- Updated home page [UI] + +## [0.12.1] -- 2025-12-22 +### Added: +- Hybrid text search combining dense and sparse vector search [API] +### Changed: +- Backend improvements that reduce container size + +## [0.12.0] -- 2025-12-01 +### Added: +- umap calculation [API] +- BED-analyzer (qc), which is backed by gtars-wasm package [UI] +- New improved home page [UI] +- Added visualization of the UMAP, and interactive [UI] + + +## [0.11.0] -- 2025-09-11 +### Added: +- Added umap visualizations of bed embeddings +- Added bedbase-verse metrics +- Added filtering by assay and genome in the semantic search +- Added bed file qc check before running bed to bed search + +### Changed: +- Changed bivec search to text semantic search +- Multiple UI bug fix + + +## [0.10.0] -- 2025-04-21 +### Added: +- Added usage statistics +- Added Bed file statistic page + +### Changed: +- Updated bed compliance and data formats + + +## [0.9.0] -- 2025-01-03 +### Added: +* Added a new class `CreateBEDsetRequest` in `bedhost/data_models.py` to handle BEDset creation requests. +* Introduced a new API endpoint `/v1/bed/{bed_id}/neighbours` to get nearest neighbors for a BED record in `bedhost/routers/bed_api.py`. +* Implemented a new API endpoint `/v1/bedset/create/` to create a new BEDset by providing a registry path to the PEPhub project in `bedhost/routers/bedset_api.py`. + +### Changed: +* Refactored `text_to_bed_search` function to include additional logic for handling specific queries in `bedhost/routers/bed_api.py`. + +### UI improvements: +* Added Most similar files table to bed page +* Improved Mobile friendly ui to both bed and bedset page +* Improved metadata tables on bed page +* Added `Download pdf` button for plots +* Improved search tables +* Added creation of bedset UI + + +## [0.8.0] -- 2024-11-07 + +### Added: +- Added endpoint showing available genomes +- Added endpoint listing bed_ids with missing plots + +## [0.7.0] -- 2024-10-23 + +### Added: +- New text2bed search (bivec search) +- Added track_hub endpoints and pointing link +- Added pep generating endpoint for bedsets + +## [0.6.0] -- 2024-10-15 + +- Multiple ui improvements and fixes +- Updated bed metadata endpoint: added `annotation` to metadata return model, with standard schema +- Updated metadata in search endpoints. +- Added embed endpoint. [#136](https://github.com/databio/bedhost/issues/136) + + +## [0.5.0] -- 2024-06-11 + +- Improved Bed search (speed and quality) +- Added licenses +- UI tweaks +- Added universes and bed tokens to the database +- Added embedding endpoint + +## [0.4.0] -- 2024-04-08 + +- Support of new bbconf. +- Updated endpoints. + + +## [0.3.0] -- 2023-03-01 + +- switch to pydantic2 +- updated requirements +- updated docs + + +## [0.2.0] -- 2023-10-17 +- remove all graphql +- remove local static hosting of UI +- update to new pipestat-based bbconf (pending) +- major refactor of API that introduces backwards-incompatible changes + +## [0.1.3] -- 2023-09-01 +- allow all origins + +## [0.1.2] -- 2023-02-06 +### change +- change `/bedset/my_bedset/file_paths`endpoint from GET to POST + +## [0.1.1] -- 2021-10-30 +### change +- `/bed/genomes` and `bedset/genomes`: improve speed + +## [0.1.0] -- 2021-10-25 +### add +- GraphQL endpoints +### change +- endpoints update due to `bbconf` and `pipestat` changes + +## [0.0.6] -- 2021-05-17 +### add +- Add endpoints that serve: + - a list of genome assemblies in bedsets and bedfiles table + - bed files by search term(s) + - remote file path (http / s3) + +## [0.0.5] -- 2021-04-15 +### add +- Add examples of API endpoints +### fix +- resolve `/about` page not found when typing/editing url in the address bar. + +## [0.0.4] -- 2021-04-01 +### add +- add endpoint for region-based query +### fix +- construction of local file/img path + +## [0.0.3] -- 2021-02-22 +- Initial project release diff --git a/docs/guide/deployment.md b/docs/guide/deployment.md new file mode 100644 index 00000000..3cefa817 --- /dev/null +++ b/docs/guide/deployment.md @@ -0,0 +1,64 @@ +# Deploying bedbase.org + +This repository deploys the API for bedbase. It will run these services: + +1. production API: https://api.bedbase.org/ +2. dev API: https://api-dev.bedbase.org/ + +This repo will deploy a new service by following these steps: + +1. Build an image by packaging the bedhost image (from dockerhub) with the bbconf file in this repository. +2. Push that image to AWS. +3. Deploy it to yeti cluster with aws task def. + +## Build the container + +Here we use the `databio/bedhost` container on dockerhub, and just add the configuration file in this repo to it, so build is super fast. + +``` +docker build -t databio/bedhost-configured -f deployment/Dockerfiles/primary.Dockerfile . +``` + +Or for dev: + +``` +docker build -t databio/bedhost-configured-dev -f deployment/Dockerfiles/dev1.Dockerfile . +``` + +## Run it locally to test + +First, source the .env file to set env vars in the calling environment. +Then, use `--env-file` to pass those env vars through to the container + +``` +source environment/production.env +docker run --rm --network="host" \ + --env-file environment/docker.env \ + databio/bedhost-configured-dev +``` + +Here's another example for running the container: + +``` +docker run --rm --init -p 8000:8000 --name bedstat-rest-server \ + --network="host" \ + --volume "$(pwd)/deployment/config/api.bedbase.org.yaml:/bedbase.yaml" \ + --env-file environment/docker.env \ + --env BEDBASE_CONFIG=/bedbase.yaml \ + databio/bedhost uvicorn bedhost.main:app --reload +``` + +## Building the Amazon-tagged version + +You could build and push to ECR like this if you need it... but the github action will do this for you. + +Authenticate with AWS ECR: +``` +aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 235728444054.dkr.ecr.us-east-1.amazonaws.com +``` + +Build/tag/push image: +``` +docker build -t 235728444054.dkr.ecr.us-east-1.amazonaws.com/bedhost -f deployment/Dockerfiles/primary.Dockerfile . +docker push 235728444054.dkr.ecr.us-east-1.amazonaws.com/bedhost +``` diff --git a/docs/guide/docs.yml b/docs/guide/docs.yml new file mode 100644 index 00000000..a983a9c9 --- /dev/null +++ b/docs/guide/docs.yml @@ -0,0 +1,6 @@ +site_name: BEDhost +nav: + - BEDhost overview: README.md + - Building docker image: build_image.md + - Deploy API: deployment.md + - Changelog: changelog.md