diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 577c0c4..22c7a00 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -8,7 +8,7 @@ name: CI # doc/POLICY.md section 11 and doc/REQUIREMENTS.md section 20. # # No credential is configured here and nothing reaches an external service: the -# suite must need neither (doc/POLICY.md Invariant 6). Examples tagged :network +# suite must need neither (doc/POLICY.md Invariant 7). Examples tagged :network # are excluded by default, and none of the Gemfile's optional groups is # installed, so no plugin's own gem is a condition of this workflow passing. # diff --git a/doc/AI_TUTORIAL.md b/doc/AI_TUTORIAL.md index f36ee49..7634820 100644 --- a/doc/AI_TUTORIAL.md +++ b/doc/AI_TUTORIAL.md @@ -48,10 +48,14 @@ is the order of the work. The store plugin's position is not an aesthetic choice. `FilterJoin` produces one item with **no link** — it is several articles at once, so there is no page -it points at — and the store plugins are keyed on the link and drop an item -without one. `StorePermalink`, `StoreFullText` and `StoreDigest` therefore -belong **before** `FilterJoin`, where there is still one item per article to -record. Put one after it and the Recipe stores nothing and publishes nothing. +it points at. `StorePermalink` and `StoreFullText` use the shared link-based +store path and drop an item without a link, so both belong **before** +`FilterJoin`. `StoreDigest` is different: it identifies content from the +configured fields and can store an item whose link is nil. In this tutorial it +still belongs before `FilterJoin`, because the job is to record each source +article before they are joined and before later work is repeated. Putting +`StoreDigest` after `FilterJoin` changes its meaning to de-duplicating the whole +joined digest rather than the individual articles. ## 2. Build it without AI first diff --git a/doc/PLUGINS.md b/doc/PLUGINS.md index 52d7280..bbc46a5 100644 --- a/doc/PLUGINS.md +++ b/doc/PLUGINS.md @@ -1488,9 +1488,10 @@ look new again. #### StoreFullText — **Supported** `store/full_text.rb`. Records title, link, description and `content_encoded`, -and passes on only what is new. Deduplicates on link **or** title, so a -republished article with a new URL is not stored twice. Pair with -`FilterFullFeed` to archive article bodies. +and passes on only items whose new record was saved successfully. Deduplicates +on link **or** title, so a republished article with a new URL is not stored +twice. A database write failure ends the run; the unsaved item is not passed +downstream. Pair with `FilterFullFeed` to archive article bodies. | Key | Type | Meaning | | --- | --- | --- | diff --git a/doc/POLICY.md b/doc/POLICY.md index f3b8227..5eaff24 100644 --- a/doc/POLICY.md +++ b/doc/POLICY.md @@ -381,7 +381,7 @@ appear in the order shown below. An executable uses this canonical form: #!/usr/bin/env ruby # -*- coding: utf-8 -*- # Name:: automatic -# Author: id774 (More info: http://id774.net) +# Author: id774 (More info: https://id774.net) # Source Code:: https://github.com/id774/automaticruby # License:: The GPL version 3, or LGPL version 3 (Dual License). # Contact:: idnanashi@gmail.com diff --git a/doc/VERSIONS b/doc/VERSIONS index 7b50aa7..212a72c 100644 --- a/doc/VERSIONS +++ b/doc/VERSIONS @@ -13,6 +13,7 @@ v26.09 (Release Date: TBD) title and link values. - Preserve SubscriptionText TSV column positions and ignore empty input rows. - Preserve local file paths when StoreFile hands file URIs to PublishAmazonS3. +- Stop StoreFullText from passing items downstream when database writes fail. v26.08 (2026-08-22) ------------------- diff --git a/plugins/store/full_text.rb b/plugins/store/full_text.rb index abefae8..dcf5cf9 100644 --- a/plugins/store/full_text.rb +++ b/plugins/store/full_text.rb @@ -5,7 +5,7 @@ # License:: The GPL version 3, or LGPL version 3 (Dual License). # Contact:: idnanashi@gmail.com # Created:: Feb 26, 2012 -# Updated:: Aug 15, 2026 +# Updated:: Sep 12, 2026 # Copyright:: Copyright (c) 2012-2026 Automatic Ruby Developers. require_relative 'database' @@ -46,18 +46,14 @@ def model_class # what is new. def run for_each_new_feed do |feed| + Blog.create!( + title: feed.title, + link: feed.link, + description: feed.description, + content: feed.content_encoded, + created_at: Time.now.strftime('%Y/%m/%d %X') + ) Automatic::Log.puts('info', "Saving FullText: #{feed.link}") - begin - Blog.create( - title: feed.title, - link: feed.link, - description: feed.description, - content: feed.content_encoded, - created_at: Time.now.strftime('%Y/%m/%d %X') - ) - rescue StandardError => e - Automatic::Log.puts('warn', "Skip feed due to fault in save: #{e.message}") - end end end end diff --git a/spec/plugins/filter/claude_spec.rb b/spec/plugins/filter/claude_spec.rb index fccdca9..d1dbe69 100644 --- a/spec/plugins/filter/claude_spec.rb +++ b/spec/plugins/filter/claude_spec.rb @@ -65,7 +65,7 @@ def one_item # Stands in for the network: records the connection and the request, answers # with what the example queued, and opens nothing. The default suite reaches - # no network (doc/POLICY.md Invariant 6). + # no network (doc/POLICY.md Invariant 7). def serve(*responses) requests = posted opened = connections diff --git a/spec/plugins/filter/gemini_spec.rb b/spec/plugins/filter/gemini_spec.rb index ca88753..8f9ee46 100644 --- a/spec/plugins/filter/gemini_spec.rb +++ b/spec/plugins/filter/gemini_spec.rb @@ -64,7 +64,7 @@ def one_item # Stands in for the network: records the connection and the request, answers # with what the example queued, and opens nothing. The default suite reaches - # no network (doc/POLICY.md Invariant 6). + # no network (doc/POLICY.md Invariant 7). def serve(*responses) requests = posted opened = connections diff --git a/spec/plugins/filter/kimi_spec.rb b/spec/plugins/filter/kimi_spec.rb index 026ec12..5629c54 100644 --- a/spec/plugins/filter/kimi_spec.rb +++ b/spec/plugins/filter/kimi_spec.rb @@ -68,7 +68,7 @@ def one_item # Stands in for the network: records the connection and the request, answers # with what the example queued, and opens nothing. The default suite reaches - # no network (doc/POLICY.md Invariant 6). + # no network (doc/POLICY.md Invariant 7). def serve(*responses) requests = posted opened = connections diff --git a/spec/plugins/filter/open_ai_spec.rb b/spec/plugins/filter/open_ai_spec.rb index 04c2d9b..df2bb90 100644 --- a/spec/plugins/filter/open_ai_spec.rb +++ b/spec/plugins/filter/open_ai_spec.rb @@ -62,7 +62,7 @@ def one_item # Stands in for the network: records the connection and the request, answers # with what the example queued, and opens nothing. The default suite reaches - # no network (doc/POLICY.md Invariant 6). + # no network (doc/POLICY.md Invariant 7). def serve(*responses) requests = posted opened = connections diff --git a/spec/plugins/filter/sakura_ai_spec.rb b/spec/plugins/filter/sakura_ai_spec.rb index b60a43f..269dfab 100644 --- a/spec/plugins/filter/sakura_ai_spec.rb +++ b/spec/plugins/filter/sakura_ai_spec.rb @@ -65,7 +65,7 @@ def one_item # Stands in for the network: records the connection and the request, answers # with what the example queued, and opens nothing. The default suite reaches - # no network (doc/POLICY.md Invariant 6). + # no network (doc/POLICY.md Invariant 7). def serve(*responses) requests = posted opened = connections diff --git a/spec/plugins/store/full_text_spec.rb b/spec/plugins/store/full_text_spec.rb index e7cc23d..0070dbd 100644 --- a/spec/plugins/store/full_text_spec.rb +++ b/spec/plugins/store/full_text_spec.rb @@ -5,7 +5,7 @@ # License:: The GPL version 3, or LGPL version 3 (Dual License). # Contact:: idnanashi@gmail.com # Created:: Feb 26, 2012 -# Updated:: Oct 16, 2014 +# Updated:: Sep 12, 2026 # Copyright:: Copyright (c) 2012-2026 Automatic Ruby Developers. require File.expand_path(File.dirname(__FILE__) + '../../../spec_helper') @@ -134,6 +134,26 @@ Automatic::Plugin::Blog.count.should eq 1 end + it "should raise when storing a new blog entry fails" do + instance = Automatic::Plugin::StoreFullText.new({"db" => @db_filename}, + AutomaticSpec.generate_pipeline { + feed { + item "http://blog.id774.net/blogs/feed/", + "dummy title", + "aaa bbb ccc http://test2.id774.net ddd eee", + "Mon, 07 Mar 2011 15:54:11 +0900" + } + } + ) + + Automatic::Plugin::Blog.should_receive(:create!).and_raise( + ActiveRecord::StatementInvalid.new("write failed") + ) + lambda { + instance.run + }.should raise_error(ActiveRecord::StatementInvalid, /write failed/) + end + it "should store 2 records for the independent entries" do instance = Automatic::Plugin::StoreFullText.new({"db" => @db_filename}, AutomaticSpec.generate_pipeline { diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb index 8b47442..9645718 100644 --- a/spec/spec_helper.rb +++ b/spec/spec_helper.rb @@ -56,7 +56,7 @@ # Examples tagged :network reach real hosts. They are excluded from the # default suite and from CI, because the suite must need neither a network - # nor a credential (doc/POLICY.md Invariant 6). Several of them point at + # nor a credential (doc/POLICY.md Invariant 7). Several of them point at # hosts that no longer serve what they expect, which is a further reason not # to make them a gate. Run them deliberately with: # @@ -99,12 +99,12 @@ module AutomaticSpec OPTIONAL_PLUGIN_GEMS = %w[activerecord feedbag nokogiri sanitize sqlite3].freeze class << self - # Load a plugin, or report that its dependency is absent. + # Load a plugin, or report that its own optional dependency is absent. # - # A plugin whose gem is not installed -- because the service it talks to no - # longer exists, and no currently published gem speaks to it -- is never - # stubbed into passing (doc/POLICY.md Invariant 7). Its spec is skipped - # instead, and the reason is printed, which is the honest signal. + # Supported (external) plugin specs use this when the plugin's declared + # optional gem is not installed in the current bundle. That absence is not + # simulated into success: the spec is skipped and names the missing + # dependency (doc/POLICY.md Invariant 8). # # Only Automatic::OptionalDependencyError, raised by require_optional for # the plugin's own missing gem, is skipped this way. A plain LoadError