Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ name: CI
# doc/POLICY.md section 11 and doc/REQUIREMENTS.md section 20.
#
# No credential is configured here and nothing reaches an external service: the
# suite must need neither (doc/POLICY.md Invariant 6). Examples tagged :network
# suite must need neither (doc/POLICY.md Invariant 7). Examples tagged :network
# are excluded by default, and none of the Gemfile's optional groups is
# installed, so no plugin's own gem is a condition of this workflow passing.
#
Expand Down
12 changes: 8 additions & 4 deletions doc/AI_TUTORIAL.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,10 +48,14 @@ is the order of the work.

The store plugin's position is not an aesthetic choice. `FilterJoin` produces
one item with **no link** — it is several articles at once, so there is no page
it points at — and the store plugins are keyed on the link and drop an item
without one. `StorePermalink`, `StoreFullText` and `StoreDigest` therefore
belong **before** `FilterJoin`, where there is still one item per article to
record. Put one after it and the Recipe stores nothing and publishes nothing.
it points at. `StorePermalink` and `StoreFullText` use the shared link-based
store path and drop an item without a link, so both belong **before**
`FilterJoin`. `StoreDigest` is different: it identifies content from the
configured fields and can store an item whose link is nil. In this tutorial it
still belongs before `FilterJoin`, because the job is to record each source
article before they are joined and before later work is repeated. Putting
`StoreDigest` after `FilterJoin` changes its meaning to de-duplicating the whole
joined digest rather than the individual articles.

## 2. Build it without AI first

Expand Down
7 changes: 4 additions & 3 deletions doc/PLUGINS.md
Original file line number Diff line number Diff line change
Expand Up @@ -1488,9 +1488,10 @@ look new again.
#### StoreFullText — **Supported**

`store/full_text.rb`. Records title, link, description and `content_encoded`,
and passes on only what is new. Deduplicates on link **or** title, so a
republished article with a new URL is not stored twice. Pair with
`FilterFullFeed` to archive article bodies.
and passes on only items whose new record was saved successfully. Deduplicates
on link **or** title, so a republished article with a new URL is not stored
twice. A database write failure ends the run; the unsaved item is not passed
downstream. Pair with `FilterFullFeed` to archive article bodies.

| Key | Type | Meaning |
| --- | --- | --- |
Expand Down
2 changes: 1 addition & 1 deletion doc/POLICY.md
Original file line number Diff line number Diff line change
Expand Up @@ -381,7 +381,7 @@ appear in the order shown below. An executable uses this canonical form:
#!/usr/bin/env ruby
# -*- coding: utf-8 -*-
# Name:: automatic
# Author: id774 (More info: http://id774.net)
# Author: id774 (More info: https://id774.net)
# Source Code:: https://github.com/id774/automaticruby
# License:: The GPL version 3, or LGPL version 3 (Dual License).
# Contact:: idnanashi@gmail.com
Expand Down
1 change: 1 addition & 0 deletions doc/VERSIONS
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ v26.09 (Release Date: TBD)
title and link values.
- Preserve SubscriptionText TSV column positions and ignore empty input rows.
- Preserve local file paths when StoreFile hands file URIs to PublishAmazonS3.
- Stop StoreFullText from passing items downstream when database writes fail.

v26.08 (2026-08-22)
-------------------
Expand Down
20 changes: 8 additions & 12 deletions plugins/store/full_text.rb
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
# License:: The GPL version 3, or LGPL version 3 (Dual License).
# Contact:: idnanashi@gmail.com
# Created:: Feb 26, 2012
# Updated:: Aug 15, 2026
# Updated:: Sep 12, 2026
# Copyright:: Copyright (c) 2012-2026 Automatic Ruby Developers.

require_relative 'database'
Expand Down Expand Up @@ -46,18 +46,14 @@ def model_class
# what is new.
def run
for_each_new_feed do |feed|
Blog.create!(
title: feed.title,
link: feed.link,
description: feed.description,
content: feed.content_encoded,
created_at: Time.now.strftime('%Y/%m/%d %X')
)
Automatic::Log.puts('info', "Saving FullText: #{feed.link}")
begin
Blog.create(
title: feed.title,
link: feed.link,
description: feed.description,
content: feed.content_encoded,
created_at: Time.now.strftime('%Y/%m/%d %X')
)
rescue StandardError => e
Automatic::Log.puts('warn', "Skip feed due to fault in save: #{e.message}")
end
end
end
end
Expand Down
2 changes: 1 addition & 1 deletion spec/plugins/filter/claude_spec.rb
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ def one_item

# Stands in for the network: records the connection and the request, answers
# with what the example queued, and opens nothing. The default suite reaches
# no network (doc/POLICY.md Invariant 6).
# no network (doc/POLICY.md Invariant 7).
def serve(*responses)
requests = posted
opened = connections
Expand Down
2 changes: 1 addition & 1 deletion spec/plugins/filter/gemini_spec.rb
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ def one_item

# Stands in for the network: records the connection and the request, answers
# with what the example queued, and opens nothing. The default suite reaches
# no network (doc/POLICY.md Invariant 6).
# no network (doc/POLICY.md Invariant 7).
def serve(*responses)
requests = posted
opened = connections
Expand Down
2 changes: 1 addition & 1 deletion spec/plugins/filter/kimi_spec.rb
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ def one_item

# Stands in for the network: records the connection and the request, answers
# with what the example queued, and opens nothing. The default suite reaches
# no network (doc/POLICY.md Invariant 6).
# no network (doc/POLICY.md Invariant 7).
def serve(*responses)
requests = posted
opened = connections
Expand Down
2 changes: 1 addition & 1 deletion spec/plugins/filter/open_ai_spec.rb
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ def one_item

# Stands in for the network: records the connection and the request, answers
# with what the example queued, and opens nothing. The default suite reaches
# no network (doc/POLICY.md Invariant 6).
# no network (doc/POLICY.md Invariant 7).
def serve(*responses)
requests = posted
opened = connections
Expand Down
2 changes: 1 addition & 1 deletion spec/plugins/filter/sakura_ai_spec.rb
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ def one_item

# Stands in for the network: records the connection and the request, answers
# with what the example queued, and opens nothing. The default suite reaches
# no network (doc/POLICY.md Invariant 6).
# no network (doc/POLICY.md Invariant 7).
def serve(*responses)
requests = posted
opened = connections
Expand Down
22 changes: 21 additions & 1 deletion spec/plugins/store/full_text_spec.rb
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
# License:: The GPL version 3, or LGPL version 3 (Dual License).
# Contact:: idnanashi@gmail.com
# Created:: Feb 26, 2012
# Updated:: Oct 16, 2014
# Updated:: Sep 12, 2026
# Copyright:: Copyright (c) 2012-2026 Automatic Ruby Developers.

require File.expand_path(File.dirname(__FILE__) + '../../../spec_helper')
Expand Down Expand Up @@ -134,6 +134,26 @@
Automatic::Plugin::Blog.count.should eq 1
end

it "should raise when storing a new blog entry fails" do
instance = Automatic::Plugin::StoreFullText.new({"db" => @db_filename},
AutomaticSpec.generate_pipeline {
feed {
item "http://blog.id774.net/blogs/feed/",
"dummy title",
"aaa bbb ccc http://test2.id774.net ddd eee",
"Mon, 07 Mar 2011 15:54:11 +0900"
}
}
)

Automatic::Plugin::Blog.should_receive(:create!).and_raise(
ActiveRecord::StatementInvalid.new("write failed")
)
lambda {
instance.run
}.should raise_error(ActiveRecord::StatementInvalid, /write failed/)
end

it "should store 2 records for the independent entries" do
instance = Automatic::Plugin::StoreFullText.new({"db" => @db_filename},
AutomaticSpec.generate_pipeline {
Expand Down
12 changes: 6 additions & 6 deletions spec/spec_helper.rb
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@

# Examples tagged :network reach real hosts. They are excluded from the
# default suite and from CI, because the suite must need neither a network
# nor a credential (doc/POLICY.md Invariant 6). Several of them point at
# nor a credential (doc/POLICY.md Invariant 7). Several of them point at
# hosts that no longer serve what they expect, which is a further reason not
# to make them a gate. Run them deliberately with:
#
Expand Down Expand Up @@ -99,12 +99,12 @@ module AutomaticSpec
OPTIONAL_PLUGIN_GEMS = %w[activerecord feedbag nokogiri sanitize sqlite3].freeze

class << self
# Load a plugin, or report that its dependency is absent.
# Load a plugin, or report that its own optional dependency is absent.
#
# A plugin whose gem is not installed -- because the service it talks to no
# longer exists, and no currently published gem speaks to it -- is never
# stubbed into passing (doc/POLICY.md Invariant 7). Its spec is skipped
# instead, and the reason is printed, which is the honest signal.
# Supported (external) plugin specs use this when the plugin's declared
# optional gem is not installed in the current bundle. That absence is not
# simulated into success: the spec is skipped and names the missing
# dependency (doc/POLICY.md Invariant 8).
#
# Only Automatic::OptionalDependencyError, raised by require_optional for
# the plugin's own missing gem, is skipped this way. A plain LoadError
Expand Down
Loading