Implement Wiley workflow 'create' step by jonavellecuerdo · Pull Request #263 · MITLibraries/dspace-submission-composer

jonavellecuerdo · 2026-09-03T20:16:34Z

Purpose and background context

The Wiley workflow requires a batch creation process that must be capable of performing the following functions:

Accept a CSV file of DOIs as input
Can download bitstreams (PDFs) from Wiley via API
Can download metadata from Crossref
Can determine whether a DOI was seen from a previous batch run for Wiley

How this addresses that need:

Add required env vars for Wiley workflow
Add ItemSubmission method for retrieving all submissions for a given workflow
Add download method to S3 client
Create Wiley workflow module

Highlights

This assumes that Wiley will create a batch folder in the DSC S3 bucket, named "YYYY-MM-wiley-am" to represent monthly uploads of a CSV file containing all DOIs associated with MIT author accepted manuscripts.
- Conversations with Wiley will likely occur later in Q1 or early Q2 during which we will configure their access to the DSC S3 bucket and discuss specifics for their file uploads.
Similar to the Digitized Theses workflow, this workflow:
- Mints a new batch ID by appending the run date to the original batch id
- Prepares the batch in a local temporary directory then syncs the prepared batch to S3.
  Though they share similarities, I would propose any work to create a mutual parent for these two workflows for sometime later!
In the early stages of this work, checking whether an item identifier (a DOI) was associated with an record in DynamoDB with status="ingest_success" was performed per item submission, but that resulted in the batch creation process taking 40+ mins to run. For this reason, I opted to retrieve the item identifiers from records in DynamoDB and store them in a list before proceeding to prepare each item submission. With this change, the batch creation process now takes ~3 mins to complete!

✨ Note: 693 of the 1K+ lines changed are pyproject.toml and uv.lock related.

How can a reviewer manually see the effects of these changes?

Review added unit tests for Wiley workflow
Ran the create step using the DSO step function for batch_id=2022-10-wiley-am. The CSV file I uploaded represents the most recent CSV file in the wiley-dev S3 bucket (s3://wiley-files-dev-222053980223/archived/MIT_Automatic_Article_List_10.10.2022.csv). I renamed the file to one that the DSC app expects.
- See step function execution
- See CloudWatch logstream
  - Many of the log messages indicate item submissions already submitted to DSpace due to Wiley record migration 🚀 , showing that the DSC can evaluate item submissions across batches for the Wiley workflow!

From logstream:

- Reminder: There are currently 1092 completed Wiley item submissions (migrated from wiley-deposits table) (DSO-12).

026-09-04 15:31:52,953 INFO dsc.workflows.wiley.workflow.prepare_batch(): There are 1092 completed Wiley item submissions

Pointing out this create summary that gets printed as part of Wiley.prepare_batch. Planning to create a follow-up ticket to print summaries during the create step across all workflows, which will allow us to reorganize when this log appears!

2026-09-04 15:33:23,943 INFO dsc.workflows.wiley.workflow.prepare_batch(): Created items for batch '2022-10-wiley-am-20260904T153152Z': 
{
    "total": 1353,
    "created": 294,
    "skipped": 1053,
    "errors": 6
}

This means, 1053 of the records from the most recent file we received from Wiley were previously processed via the v1 workflow with https://github.com/MITLibraries/wiley-deposits/blob/main/. I would take the numbers above with a grain of salt and a closer look at these counts will be more valuable once we get our first upload from Wiley! The main point of sharing these numbers is to show that the DSC workflow can skip items previously ingested/completed.

Includes new or updated dependencies?

YES

Changes expectations for external applications?

YES - Implementing the DSC workflow for Wiley will allow us to sunset the standalone https://github.com/MITLibraries/wiley-deposits app.

What are the relevant tickets?

https://mitlibraries.atlassian.net/browse/IN-1097

Code review

Code review best practices are documented here and you are encouraged to have a constructive dialogue with your reviewers about their preferences and expectations.

Why these changes are being introduced: * The Wiley workflow requires a batch creation process that must be capable of performing the following functions: 1. Accept a CSV file of DOIs as input 2. Can download bitstreams (PDFs) from Wiley via API 3. Can download metadata from Crossref 4. Can determine whether a DOI was seen from a previous batch run for Wiley How this addresses that need: * Add required env vars for Wiley workflow * Add ItemSubmission method for retrieving all submissions for a given workflow * Add download method to S3 client * Create Wiley workflow module Side effects of this change: * This requires the addition of two new env vars required by the Wiley workflow: - WILEY_METADATA_API_URL - WILEY_BITSTREAM_API_URL Relevant ticket(s): * https://mitlibraries.atlassian.net/browse/IN-1097

Copilot

🟡 Changes recommended

Incomplete sync handling, capped historical lookup, and unchecked operational failures can produce missing or duplicate submissions.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Implements the Wiley batch-creation workflow for DOI-based submissions.

Changes:

Downloads Wiley PDFs and Crossref metadata from CSV input.
Adds workflow-wide submission lookup and S3 download support.
Registers Wiley configuration, tests, and dependencies.

File summaries

File	Description
`dsc/workflows/wiley/workflow.py`	Implements Wiley batch creation.
`dsc/workflows/wiley/transformer.py`	Adds transformer placeholder.
`dsc/workflows/wiley/__init__.py`	Exports Wiley components.
`dsc/workflows/__init__.py`	Registers the workflow.
`dsc/item_submission.py`	Adds workflow submission lookup.
`dsc/utils/aws/s3.py`	Adds S3 file downloads.
`dsc/utils/aws/__init__.py`	Exports AWS utilities.
`dsc/config.py`	Adds Wiley settings and changes logging setup.
`pyproject.toml`	Updates dependency constraints.
`tests/workflows/wiley/test_workflow.py`	Tests Wiley helper behavior.
`tests/workflows/wiley/__init__.py`	Initializes the test package.
`tests/test_item_submission.py`	Updates creation expectations.

Review details

Files reviewed: 12/13 changed files
Comments generated: 10
Review effort level: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

ghukill

Requesting changes, primarily for a docstring request for the workflow class. There are additional suggestions, but those didn't trigger the requesting of changes.

Nice work! All said, pretty easy to reason about. I'm unsure how much my prior knowledge helped... but the code is easy to follow.

ghukill · 2026-09-04T16:56:51Z

+        # get list of DOIs for completed item submissions
+        skip_list = self._get_completed_item_submission_ids()
+        logger.info(f"There are {len(skip_list)} completed Wiley item submissions")


Excellent idea to retrieve all at once, and then reuse. Quite the performance gain it sounds like!

Indeed...Do you think it will continue to be performant, say we get to retrieving 10K item identifier values from the DynamoDB table? 😅

I do. Seems much better to download ~10k identifiers (assuming we can limit to this workflow only, or at least scan to it and only retrieve that much data) than make ~10k requests to Dynamo!

ghukill · 2026-09-04T17:05:33Z

+        # copy csv of DOIs into temp batch folder
+        s3_client = S3Client()
+        s3_client.download_file(
+            s3_uri=f"s3://{CONFIG.s3_bucket_submission_assets}/{original_batch_path}MIT_Authored_Articles_Wiley.csv",


Is MIT_Authored_Articles_Wiley.csv the naming convention "that the DSC app expects"? Is this what you mentioned the other day about hardcoding, but perhaps we'd revisit?

[nods heavily] Yes! When I look at the files in the v1 Wiley workflow S3 bucket, I didn't see any consistency in the filenames. I'm hoping we can request a format that they can agree to!

ghukill · 2026-09-04T17:07:58Z

+}
+
+
+class Wiley(Workflow):


Perhaps I'll change my tune, but as I kind of dig into this prepare_batch() PR, I'm finding that it could be really helpful if the workflow docstring provided a high level overview of how this workflow works.

It doesn't need to be exhaustive, but maybe touch on things like:

this workflow pulls from 3rd party APIs

how the batch + batch assets are created; my nascent understanding is that it's created by the workflow, pulling a file from S3 that it's expecting to be there... matched on YYYY-MM in the prefix?

etc.

Forgive me if I'm missing this somewhere, but it feels like in code, not just confluence, we should explain how this workflow works.

Agreed, I think you're part way there with some of the details in the commit message

Ahh, I thought I hit all the boxes re: docstrings! I know you've made this comment before--will make sure to add one.

ghukill · 2026-09-04T17:09:01Z

+        raise NotImplementedError
+
+    def prepare_batch(self, *, synced: bool = False) -> tuple[list, ...]:  # noqa: ARG002
+        """Prepare a batch folder in the DSC S3 bucket.


I think it'd be handy if the docstring explained that a) we use thread parallelization to query the Wiley API, and b) why it's needed.

ghukill · 2026-09-04T17:15:59Z

+        try:
+            self._download_bitstream(
+                item_identifier=item_submission.item_identifier,
+                output_dir=output_dir,
+            )
+            self._get_crossref_metadata(
+                item_identifier=item_submission.item_identifier,
+                output_dir=output_dir,
+            )
+        except (
+            exceptions.ItemBitstreamsNotFoundError,
+            exceptions.ItemMetadataNotFoundError,
+        ) as exception:
+            item_submission.status = ItemSubmissionStatus.CREATE_FAILED
+            item_submission.status_details = str(exception)
+        else:
+            item_submission.status = ItemSubmissionStatus.CREATE_SUCCESS


Any reason why bitstream + metadata are grouped under the same try? Is it to allow the fallthrough else statement?

Even if so, given those are pretty major operations, it feelsl like they would warrant their own try/except block.

An option could be pretty boring and explicit, with an early return if an error is had:

try: self._download_bitstream( item_identifier=item_submission.item_identifier, output_dir=output_dir, ) except exceptions.ItemBitstreamsNotFoundError as exception: item_submission.status = ItemSubmissionStatus.CREATE_FAILED item_submission.status_details = f"Bitstream download failed: {exception}" return item_submission try: self._get_crossref_metadata( item_identifier=item_submission.item_identifier, output_dir=output_dir, ) except exceptions.ItemMetadataNotFoundError as exception: item_submission.status = ItemSubmissionStatus.CREATE_FAILED item_submission.status_details = f"Crossref metadata fetch failed: {exception}" return item_submission item_submission.status = ItemSubmissionStatus.CREATE_SUCCESS

This would change the behavior your mention of a successful PDF download, but not metadata, where the PDF would enter the batch but would just be ignored. If that's an improvement, great! If a regression, than this form wouldn't work.

All said, totally optional.

ghukill · 2026-09-04T17:18:13Z

+            logger.exception(f"Failed to retrieve content from {url}")
+            raise exceptions.ItemBitstreamsNotFoundError from exception
+
+        content_type = response.headers.get("content-type", "")


Empty strings always make me very nervous.

What about:

content_type = response.headers.get("content-type") if not content_type or not content_type.startswith("application/pdf"): logger.error( f"Expected PDF but retrieved {content_type or 'no content type'} instead" ) raise exceptions.ItemBitstreamsNotFoundError

ghukill · 2026-09-04T17:20:04Z

+        filepath = (
+            Path(output_dir)
+            / item_identifier.replace("/", "-")
+            / f"{item_identifier.replace('/', '-')}.pdf"
+        )


Maybe normalize the identifier once, then reuse?

normalized_item_identifier = item_identifier.replace("/", "-") filepath = ( Path(output_dir) / normalized_item_identifier / f"{normalized_item_identifier}.pdf" )

ghukill · 2026-09-04T17:21:19Z

+        logger.info(f"Created batch folder in temporary directory: {tmp_dir.name}")
+        return str(tmp_batch_path)
+
+    def _download_bitstream(self, item_identifier: str, output_dir: str) -> None:


I realize it's not explicitly used (at least I don't think it is), but what if this method returned the filepath string of where it wrote the file? That can be handy for testing, logging, or even invoking directly. Totally optional.

Good call and I would rename this method, bitstream is a DSpace term so I would assume this is downloading from DSpace not the Wiley server, maybe _get_file_from_wiley_server or _get_pdf_from_wiley_server?

ghukill · 2026-09-04T17:22:00Z

+            file.write(response.content)
+            logger.info(f"Saved PDF to {file.name}")
+
+    def _get_crossref_metadata(self, item_identifier: str, output_dir: str) -> None:


See comments above for Wiley PDF downloading! Very similar method. Whatever is applied above, recommended to copy that for this method.

Agreed on harmonizing these methods

ehanson8

Agreeing with @ghukill's comments and adding some of my own!

ehanson8 · 2026-09-04T18:02:52Z

+}
+
+
+class Wiley(Workflow):


Agreed, I think you're part way there with some of the details in the commit message

ehanson8 · 2026-09-04T18:08:29Z

+        # create temporary directory
+        tmp_batch_path = self._create_tmp_batch_dir()
+
+        # copy csv of DOIs into temp batch folder


Why do we need to download these to a temp folder? I think they could be read in memory and then you wouldn't need the new download_file method

ehanson8 · 2026-09-04T18:14:36Z

+    def _create_tmp_batch_dir(self) -> str:
+        """Create temporary directory for batch preparation."""
+        tmp_dir = tempfile.TemporaryDirectory(delete=False)
+        tmp_batch_path = Path(tmp_dir.name) / self.batch_id
+        os.makedirs(tmp_batch_path)
+        logger.info(f"Created batch folder in temporary directory: {tmp_dir.name}")
+        return str(tmp_batch_path)


This could also be removed if you read the CSVs in memory

ehanson8 · 2026-09-04T18:16:57Z

+        logger.info(f"Created batch folder in temporary directory: {tmp_dir.name}")
+        return str(tmp_batch_path)
+
+    def _download_bitstream(self, item_identifier: str, output_dir: str) -> None:


Good call and I would rename this method, bitstream is a DSpace term so I would assume this is downloading from DSpace not the Wiley server, maybe _get_file_from_wiley_server or _get_pdf_from_wiley_server?

ehanson8 · 2026-09-04T18:17:48Z

+        filepath = (
+            Path(output_dir)
+            / item_identifier.replace("/", "-")
+            / f"{item_identifier.replace('/', '-')}.pdf"
+        )


ehanson8 · 2026-09-04T18:18:25Z

+            file.write(response.content)
+            logger.info(f"Saved PDF to {file.name}")
+
+    def _get_crossref_metadata(self, item_identifier: str, output_dir: str) -> None:


Agreed on harmonizing these methods

ehanson8 · 2026-09-04T18:19:38Z

+        filepath = (
+            Path(output_dir)
+            / item_identifier.replace("/", "-")
+            / f"{item_identifier.replace('/', '-')}.json"
+        )


Echoing @ghukill 's recommendation for reusing the normalized identifier

jonavellecuerdo requested a balanced review from Copilot September 4, 2026 12:39

Copilot started reviewing on behalf of jonavellecuerdo September 4, 2026 12:40 View session

Copilot AI reviewed Sep 4, 2026

View reviewed changes

Address comments from Copilot review

d2ea93d

jonavellecuerdo marked this pull request as ready for review September 4, 2026 15:39

jonavellecuerdo requested a review from a team as a code owner September 4, 2026 15:39

jonavellecuerdo requested review from ehanson8 and ghukill September 4, 2026 15:39

ghukill requested changes Sep 4, 2026

View reviewed changes

ehanson8 reviewed Sep 4, 2026

View reviewed changes

		}


		class Wiley(Workflow):

		}


		class Wiley(Workflow):

Uh oh!

Conversation

jonavellecuerdo commented Sep 3, 2026 • edited by atlassian Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose and background context

How can a reviewer manually see the effects of these changes?

Includes new or updated dependencies?

Changes expectations for external applications?

What are the relevant tickets?

Code review

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ghukill left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ehanson8 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jonavellecuerdo commented Sep 3, 2026 •

edited by atlassian Bot

Loading