Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 7 additions & 4 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -160,12 +160,15 @@ Many tests use expected outputs saved directly in the source tree:
make run-website-devserver
```
- Mock vulnerability records are located in [`go/cmd/website-devserver/testdata/`](go/cmd/website-devserver/testdata/). Add or edit `.json` records and `.meta.yaml` companion files to immediately see changes on page refresh.
- **Python Website with Datastore Emulator (Legacy)**:
Run the legacy Python website server against a local Datastore emulator:
- **Run against Cloud Datastore**:
Run the Go website server against production Datastore:
```bash
make run-website-emulator
make run-website
```
Or against staging Datastore:
```bash
make run-website-staging
```
- Add custom mock testcases inside [`gcp/website/testdata/osv/`](gcp/website/testdata/osv/).

### Local API Server Development (Go-native)
- To run the public OSV API server locally using the native Go implementation alongside the ESPv2 proxy (which transcodes HTTP/JSON REST requests to gRPC):
Expand Down
11 changes: 5 additions & 6 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -186,16 +186,15 @@ make run-website

#### Running a local UI instance

For contributors without access to the GCP project, you can use the website emulator which does
not require Google Cloud project access. This emulator uses a local datastore
and loads data from a local directory.
For contributors without access to the GCP project, you can use the website devserver which does
not require Google Cloud project access or a Datastore emulator. It serves the Go website using a live flat mock dataset with hot reloading.

```shell
make run-website-emulator
make run-website-devserver
```

You can add testcase records to `gcp/website/testdata/osv/` to test odd cases.
See [gcp/website/testdata/osv/README.md](gcp/website/testdata/osv/README.md)
Mock vulnerability records are located in `go/cmd/website-devserver/testdata/`.
See [go/cmd/website-devserver/testdata/README.md](go/cmd/website-devserver/testdata/README.md)
for more information on the format of these records.

### Linting and formatting
Expand Down
15 changes: 3 additions & 12 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -94,19 +94,10 @@ build-website-frontend:
cd website/frontend3 && pnpm install && pnpm run build
cd website/blog && hugo --buildFuture -d ../dist/static/blog

run-website: build-website-frontend ## Run local Python website against prod Datastore
cd gcp/website && $(install-cmd) && GOOGLE_CLOUD_PROJECT=oss-vdb OSV_VULNERABILITIES_BUCKET=osv-vulnerabilities $(run-cmd) python main.py

run-website-staging: build-website-frontend
cd gcp/website && $(install-cmd) && GOOGLE_CLOUD_PROJECT=oss-vdb-test OSV_VULNERABILITIES_BUCKET=osv-test-vulnerabilities $(run-cmd) python main.py

run-website-emulator: build-website-frontend ## Run local Python website against emulator

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm do we need to figure out a local emulator?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see #5843

cd gcp/website && $(install-cmd) && DATASTORE_EMULATOR_PORT=5002 $(run-cmd) python frontend_emulator.py

run-go-website: build-website-frontend ## Run local Go website against prod Datastore
run-website: build-website-frontend ## Run local Go website against prod Datastore
cd go && GOOGLE_CLOUD_PROJECT=oss-vdb OSV_VULNERABILITIES_BUCKET=osv-vulnerabilities go run ./cmd/website -static-dir ../website/dist -docs-dir ../docs

run-go-website-staging: build-website-frontend
run-website-staging: build-website-frontend
cd go && GOOGLE_CLOUD_PROJECT=oss-vdb-test OSV_VULNERABILITIES_BUCKET=osv-test-vulnerabilities go run ./cmd/website -static-dir ../website/dist -docs-dir ../docs

run-website-devserver: build-website-frontend ## Run local Go website development server against local mock dataset
Expand All @@ -117,7 +108,7 @@ stage-website-assets: build-website-frontend
cp -r website/dist/* go/cmd/website/dist/
cp docs/osv_service_v1.swagger.json go/cmd/website/docs/

run-go-website-prod: stage-website-assets
run-website-prod: stage-website-assets
cd go && GOOGLE_CLOUD_PROJECT=oss-vdb OSV_VULNERABILITIES_BUCKET=osv-vulnerabilities go run -tags embedstatic ./cmd/website


Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,9 +44,9 @@ consists of:
| `gcp/datastore` | The datastore index file (`index.yaml`) |
| `gcp/functions` | The Cloud Function for publishing PyPI vulnerabilities (maintained, but not developed) |
| `gcp/indexer` | The determine version `indexer` |
| `gcp/website` | The backend of the osv.dev web interface, with the frontend in `frontend3` <br /> Blog posts (in `blog`) |
| `gcp/website` | Frontend assets for the osv.dev website (in `frontend3`) and blog posts (in `blog`) |
| `gcp/workers/` | Workers for bisection and impact analysis (`worker`, `importer`, `alias`) <br /> `cron/` jobs for database backups and processing oss-fuzz records |
| `go/` | Go module for shared libraries and commands (`cmd/exporter`, `cmd/recordchecker`) |
| `go/` | Go module for shared libraries and commands (`cmd/importer`, `cmd/worker`, `cmd/exporter`, `cmd/website`, `cmd/api`, etc.) |
| `osv/` | The core OSV Python library, used in basically all Python services <br /> OSV ecosystem package versioning helpers in `ecosystems/` <br /> Datastore model definitions in `models.py` |
| `tools/` | Misc scripts/tools, mostly intended for development (datastore stuff, linting) <br /> The `indexer-api-caller` for indexer calling |
| `vulnfeeds/` | Go module for (mostly) the NVD CVE conversion <br /> The Alpine feed converter (`cmd/alpine`) <br /> The Debian feed converter (`tools/debian`, which is written in Python) |
Expand Down
30 changes: 27 additions & 3 deletions deployment/build-and-stage.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -396,18 +396,42 @@ steps:
args: ['push', '--all-tags', 'gcr.io/oss-vdb/nvd-cve-osv']
waitFor: ['build-nvd-cve-osv', 'cloud-build-queue']

# Build website frontend assets
- name: 'node:24.18'
entrypoint: 'bash'
args:
- '-c'
- |
corepack enable pnpm
pnpm install --frozen-lockfile --ignore-scripts
pnpm run build:prod
dir: 'gcp/website/frontend3'
id: 'build-frontend3'
waitFor: ['setup']

- name: 'gcr.io/oss-vdb/ci'
args: ['hugo', '--buildFuture', '-d', '../dist/static/blog']
dir: 'gcp/website/blog'
id: 'build-hugo'
waitFor: ['setup']

# Build/push Website image
- name: 'gcr.io/cloud-builders/docker'
entrypoint: 'bash'
args: ['-c', 'docker pull gcr.io/oss-vdb/osv-website:latest || exit 0']
id: 'pull-website'
waitFor: ['setup']
- name: 'gcr.io/cloud-builders/docker'
args: ['buildx', 'build', '--build-arg', 'BUILDKIT_INLINE_CACHE=1',
args: ['buildx', 'build',
'-t', 'gcr.io/oss-vdb/osv-website:latest', '-t', 'gcr.io/oss-vdb/osv-website:$COMMIT_SHA',
'-f', 'gcp/website/Dockerfile', '--cache-from', 'gcr.io/oss-vdb/osv-website:latest', '--pull', '.']
'--target', 'website',
'--build-context', 'bindings=../bindings',
'--build-context', 'website-dist=../gcp/website/dist',
'--build-context', 'docs=../docs',
'-f', 'Dockerfile', '--cache-from', 'gcr.io/oss-vdb/osv-website:latest', '--pull', '.']
dir: 'go'
id: 'build-website'
waitFor: ['pull-website', 'build-first-package-finder']
waitFor: ['pull-website', 'build-frontend3', 'build-hugo', 'build-first-package-finder']
- name: 'gcr.io/cloud-builders/docker'
args: ['push', '--all-tags', 'gcr.io/oss-vdb/osv-website']
waitFor: ['build-website', 'cloud-build-queue']
Expand Down
4 changes: 4 additions & 0 deletions deployment/clouddeploy/osv-website/run-prod.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,10 @@ spec:
env:
- name: OSV_VULNERABILITIES_BUCKET
value: 'osv-vulnerabilities'
- name: OSV_LINTER_BUCKET
value: 'osv-public-import-logs'
- name: OSV_API_URL
value: 'api.osv.dev'
- name: REDISHOST
value: '10.85.52.228'
- name: REDISPORT
Expand Down
6 changes: 5 additions & 1 deletion deployment/clouddeploy/osv-website/run-staging.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,10 @@ spec:
env:
- name: OSV_VULNERABILITIES_BUCKET
value: 'osv-test-vulnerabilities'
- name: OSV_LINTER_BUCKET
value: 'osv-test-public-import-logs'
- name: OSV_API_URL
value: 'api.test.osv.dev'
- name: REDISHOST
value: '10.189.34.180'
- name: REDISPORT
Expand All @@ -28,7 +32,7 @@ spec:
secretKeyRef:
name: google-oauth-client-secret
key: 'latest'
- name: FLASK_SECRET_KEY
- name: SESSION_SECRET_KEY
valueFrom:
secretKeyRef:
name: flask-secret-key
Expand Down
15 changes: 14 additions & 1 deletion go/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -22,9 +22,10 @@
#
# cd go
# docker build -t osv/importer --target importer --build-context bindings=../bindings -f Dockerfile .
# docker build -t osv/website --target website --build-context bindings=../bindings --build-context website-dist=../website/dist --build-context docs=../docs -f Dockerfile .
#
# Select which service to build using the --target flag (e.g. importer, worker, exporter,
# relations, recordchecker, generatesitemap, custommetrics, gitter, first_package_finder, api).
# relations, recordchecker, generatesitemap, custommetrics, gitter, first_package_finder, api, website).
# ====================================================================================

# ========================================================
Expand Down Expand Up @@ -148,3 +149,15 @@ RUN CGO_ENABLED=0 go build -o /app/api ./cmd/api/
FROM gcr.io/distroless/static-debian12@sha256:a9fcaedd4c9b59e12dd65d954f0b5044f19b0647a8a3712e77205df9e7b102cd AS api
COPY --from=api-build /app/api /
ENTRYPOINT ["/api"]

# ========================================================
# Target: Website
# ========================================================
FROM builder AS website-build
COPY --from=website-dist . /workspace/go/cmd/website/dist/
COPY --from=docs osv_service_v1.swagger.json /workspace/go/cmd/website/docs/
RUN CGO_ENABLED=0 go build -tags embedstatic -o /app/website ./cmd/website/

FROM gcr.io/distroless/static-debian12@sha256:a9fcaedd4c9b59e12dd65d954f0b5044f19b0647a8a3712e77205df9e7b102cd AS website
COPY --from=website-build /app/website /
ENTRYPOINT ["/website"]