Rescued HTML scraping recipe from old rascal branch

courses/RascalAmendmentProposals/RAP17/RAP17.md

@@ -0,0 +1,31 @@
+---
+title: RAP 17 - Drop symbol sequence syntax and semantics
+sidebar_position: 17
+---
+
+| RAP | 16 |
+| :---- | :---- |
+| Title | Drop sequence symbols | 
+| Author | Jurgen Vinju |
+| Status | Draft |
+| Type | Language Design  |
+
+## Abstract
+
+The sequence symbol in syntax definitions, `(A B)` defines a nameless non-terminal which parses and `A` followed by a `B`. In context-free syntax contexts a lalayout node is added. An arbitrary amount of elements is allowed. Sequence is one of "regular" symbols next to lists, separated lists, alternatives and optionals.
+
+The proposal is to remove this feature entirely from Rascal.
+
+## Motivation
+
+* For technical reasons in the parser the sequence non-terminal requires extensive special cases in multiple places in the implementation of Rascal. It is therefore still not fully supported in concrete syntax fragments. It does not work as an outermost type and it does not work as a spliced variable in a concrete pattern or a concrete expression. It also does not work as the toplevel type to the `parse` function.
+It can be made to work but at the cost of weird special cases in several places. While the other regular expressions have been dealt with in a natural way, this is not possible for the sequence symbol. 
+* Nested sequences in a grammar do not contribute to the readability of a grammar.
+* Nested sequences in a grammar make downstream processing in Rascal harder.
+* A simple production rule for the sequence: `syntax MySequence = A B;` solves the problem.
+
+## Compatibility
+
+* Sequence is not used by the bootstrap sequence because it does not occur in the Rascal grammar for Rascal
+* None of our examples use sequence
+* Removing sequence will produce a parse error in the grammars that use it.

courses/Recipes/Languages/HTML/HTML.md

@@ -0,0 +1,14 @@
+---
+title: HTML
+details:
+  - Scraping
+
+---
+
+#### Synopsis
+
+Tools for HTML Processing
+
+#### Description
+
+(((TOC)))

courses/Recipes/Languages/HTML/Scraping/Scraping.md

@@ -0,0 +1,171 @@
+---
+title: HTML Scraping
+keywords:
+    - scraping
+    - "pattern matching"
+    - recursion
+    - html
+---
+
+#### Synopsis
+
+Scraping HTML is to recover raw data from HTML documents
+
+#### Description
+
+In this example we see HTML as just another language that may contain relevant information. In the case
+of HTML it is smart to reuse existing parsers, so we use an ((AbstractSyntax)) format for HTML. It is
+described in ((lang::html::AST)) and its IO interface is described in ((lang::html::IO)).
+
+In this demo we extract information from the website of the [Centraal Bureau voor Statistiek (CBS)](https://www.cbs.nl), the Dutch national centre for statistics. 
+
+We found an interesting [page](https://longreads.cbs.nl/nederland-in-cijfers-2022/hoeveel-fietsen-we-gemiddeld-per-week/) that lists how much biking the Dutchies do on average weekly:
+
+```rascal-shell,errors
+import lang::html::IO;
+import IO;
+
+// we use a local copy instead of the live content for stability reasons
+htmlExample = getResource("Languages/HTML/Scraping/fietsen.html");
+page = readHTMLFile(htmlExample);
+```
+
+As you can see the output is truncated with `...`, to see more we can use ((IO-iprintln)):
+
+```rascal-shell,continue
+iprintln(page)
+```
+
+We used Chrome's "Inspect" feature to figure out that the div class `datatable-container` is of interest.
+So let's select that using a deep match operator and bind that div to the `tab` variable:
+
+```rascal-shell,continue
+if (/tab:div(_,class=/datatable-container/) := page)
+  iprintln(tab);
+```
+
+We used a deep match pattern and then a regular expression pattern to select all `class` attributes that have `datatable-container` somewhere in the string.
+
+Every row in the table contains data, except the header row. Let's convert this entire table 
+to a relation of type `rel[str persoonskenmerken, real fietskilometers]`. 
+
+We create the match pattern by step-wise refinement. First let's just 
+list all the rows:
+
+```rascal-shell,continue
+if (/tab:div(rows,class=/datatable-container/) := page) { // <1>
+    for (/r:tr(_) := rows) { // <2>
+        println(r);
+    }
+}
+```
+
+* <1> binds the children of the div to `rows`;
+* <2> uses deep match `/` to quickly jump to all the nested `tr` nodes;
+
+Now we refined the pattern to filter out the non-header rows:
+
+```rascal-shell,continue
+if (/tab:div(rows,class=/datatable-container/) := page) { 
+    for (/r:tr([text(_),th(_,scope="row"), text(_), td(_), text(_)]) := rows) { // <3>
+        println(r);
+    }
+}
+```
+
+* <3> we matching only those `tr` that have two children, one `th` and one `td`. To be sure we also limit the first `th` to have the `scope` attribute equal to `"row"`.
+
+Now it's time to get the final data out. The category is in the first column and the numbers are in the second.
+We could make the query deeper and more complex, but we choose to add another nesting level for the sake of clarity:
+
+```rascal-shell,continue
+if (/tab:div(rows,class=/datatable-container/) := page) { 
+    for (/r:tr([text(_),category:th(_,scope="row"), text(_), number:td(_), text(_)]) := rows) { 
+        if (/text(str c) := category, /text(str n) := number) {
+            println("<c> --- <n>");
+        }
+    }
+}
+```
+
+Now we have scraped the data out of the HTML syntax tree, we have to convert it to 
+raw data. But the Dutch use comma's as decimal separators:
+
+```rascal-shell,continue,error
+import String;
+import util::Math;
+toReal("18,79");
+toReal("18.79");
+replaceAll("18,79", ",", ".")
+```
+
+```rascal-shell,continue
+rel[str persoonskenmerken, real fietskilometers] myData = {};
+if (/tab:div(rows,class=/datatable-container/) := page) { 
+    for (/r:tr([text(_),category:th(_,scope="row"), text(_), number:td(_), text(_)]) := rows) { 
+        if (/text(str c) := category, /text(str n) := number) {
+            println("<c> --- <n>");
+            myData += <c, toReal(replaceAll(n, ",", "."))>;
+        }
+    }
+}
+myData;
+```
+
+Now we have the data in a format that we can compute with:
+```rascal-shell,continue
+myData<persoonskenmerken>
+import Set;
+theSum = sum(myData<fietskilometers>);
+relativeData = { <pk, round(avg / theSum * 100.0, 0.1) > | <pk, avg> <- myData};
+```
+
+To keep this analysis for the future, for example when new data is published on the site, we
+can store the query in a function. It is also ready to be rewritten from structured programming
+style into a functional comprehension. Let's do that first:
+
+```rascal-shell,continue
+{ <c, toReal(replaceAll(n, ",", "."))>                      
+| /tab:div(rows,class=/datatable-container/) := page        
+, /r:tr([text(_),category:th(_,scope="row"), text(_), number:td(_), text(_)]) := rows 
+, /text(str c) := category, /text(str n) := number          
+}
+```
+
+The patterns have _not_ changed, only they have been copied to the generator/filter side
+of a ((Set-Comprehension)):
+* <1> here we have the resulting tuple that uses `c` and `n` which have been selected by pattern matching
+* <2> this is the first selector that finds the table in the page
+* <3> here we iterate over the rows that are not the header
+* <4> finally we project out the text from the two cells.
+
+Now we wrap it all up in a reusable function:
+```rascal-shell,continue
+rel[str persoonskenmerken, real fietskilometers] scrapeFietsKilometers(loc address=|https://longreads.cbs.nl/nederland-in-cijfers-2022/hoeveel-fietsen-we-gemiddeld-per-week/|) 
+    = { <c, toReal(replaceAll(n, ",", "."))>                      
+        | /tab:div(rows,class=/datatable-container/)        := readHTMLFile(address)        
+        , /r:tr([text(_),category:th(_,scope="row"), text(_), number:td(_), text(_)]) := rows 
+        , /text(str c) := category, /text(str n)            := number          
+    };
+scrapeFietsKilometers()
+```
+
+Every time the function is called, the HTML is retrieved again from the site. We coded the
+URL in a default parameter, just in case a similar page exists that we might try our analysis
+on. 
+
+#### Benefits
+
+* Rascal has a lot of powerful ((PatternMatching)) operators to dissect a HTML page with;
+* Skills used in the analysis of programming languages, like traversal and pattern matching, are equally useful for HTML scraping;
+* Deep matching and ((Statements-Visit)) skip over all uninteresting content without depending on it. The more you use these "structure shy" primitives, the more robust the scraper will be against sudden changes in the HTML.
+
+#### Pitfalls
+
+* HTML scraping is a *brittle* business. If the page changes, then it's likely the query will not work
+anymore. The function will start returning empty sets of tuples in that case, most likely. If we look at the
+structural dependencies then the word `datatable-container` is very important. Also this query matches only
+tables with two columns, and the first cell is always a `th` and the second cell is `td`. Finally the actual
+data is stored in a single text cell under the `th` and `td`. If any of these properties change, this scraper 
+breaks. However, if _anything else_ changes, the scraper keeps working;
+* The HTML parser skips SVG elements;