Adam Retter

adam@evolvedbinary.com

XML Prague
@ University of Economics, Prague
2024-06-08

@adamretter

Towards
RESTful XQuery
2.0

About me

Director of Evolved Binary
- UK - Software, Consultancy, Training, and R&D
Co-founder and Co-owner of eXist Solutions
- Germany - TEI Software
Software Engineer / Prolific Open Source contributor
Enjoys Research and Development
Involved in several conference boards and peer-review panels
W3C XQuery Working Group - Invited expert
Founder of EXQuery group, and creator of RESTXQ
Me: www.adamretter.org.uk

What's in a title?

There is no XQuery version 2.0!
- 1.0, 1.1, 3.0, and 3.1
In case you were napping...
- There was no version 1.0 of this talk that you missed!

It seems like only yesterday!

XML Prague 2012
- RESTful XQuery: Standardised XQuery 3.0 Annotations for REST

Towards RESTful XQuery 2.0

Builds upon:
- The paper: RESTful XQuery: Standardised XQuery 3.0 Annotations for REST
- The standard: RESTXQ 1.0: RESTful Annotations for XQuery
Poses and answers the question:

Is RESTXQ 1.0 still fit for purpose today?
Reviews the intervening progress in:
- XQuery standardisation
- REST (and HTTP) Software Application Frameworks
- Alternatives to REST
- Feedback on RESTXQ 1.0
Proposes a set of enhancements to RESTXQ 1.0
- RESTXQ 2.0 standard?

Setting The Scene

(RESTXQ 1.0)

REST - Representational State Transfer

Client and/or Server have some "resources"
- e.g. documents
- Need to exchange these (or parts thereof)
Transfer a representation of their state
- Representation is a serialization: e.g. XML, JSON, etc...
- Usually transferred over HTTP
No direct coupling client of client and server state
- i.e. No Server Session object
HATEOAS
- Simplified: Use Hypermedia (i.e. traversable links) to navigate between resources
RESTful - to adhere to REST principles!

HTTP - Hypertext Transfer Protocol

POST /catalogue/clothing/product HTTP/1.1
Accept: application/xml
Content-Type: application/xml
Host: fictional-shop.com
Connection: keep-alive

<product id="12345">
	<name>A nice hat</name>
    
    ...
    
</product>

HTTP/1.1 201 Created
Location: http://fictional-shop.com/catalogue/clothing/product/1234
Cache-Control: no-cache
Server: Elemental/1.0.0
Date: Sat Jun 8 10:34:53 2024
Connection: Keep-Alive
Content-Type: application/xml;charset=UTF-8
Content-Length: 20

<product id="1234"/>

RESTXQ 1.0 (2012)

Connects XQuery and HTTP together
- Targeted the upcoming W3C XQuery 3.0 standard (2014)
- Targeted HTTP 1.1 (1997)
Prescribes but does not enforce RESTful principles
- Intentionally omits any mention of a "session"
Enables building HTTP API (and Web Apps) in XQuery

RESTXQ 1.0 - Three Main Concepts:

A set of named XQuery Annotations
- Added to an XQuery Function
  - Promotes it to a "Resource Function"
- Two classes of annotation:
  1. Resource Function Constraints
  2. Resource Function Parameters
Additional serialization rules
- Metadata for constructing an HTTP Response
A few XPath utility functions

RESTXQ 1.0 - Processing Model:

RESTXQ 1.0
BY EXAMPLE

(Briefly!)

RESTXQ 1.0 - Simplest Resource Function

xquery version "3.0";

module namespace my = "http://my-namespace";

declare namespace rest = "http://exquery.org/ns/restxq";

declare
	%rest:path("/hello")
function my:hello() {
  <hello>To all at XML Prague 2024</hello>
};

Path Annotation
- (One of the Resource Function Constraint Annotations)
Matches the URI path: /hello
- e.g. http://example.evolvedbinary.com/hello
- Base URI is set by the implementing server!

RESTXQ 1.0 - Path Annotation

declare
	%rest:path("/catalogue/{$category}/product/{$pid}")
function my:product($category as xs:string, $pid as xs:integer) {

  fn:collection("/products/" || $category)/product[@id eq $pid]
};

Only one per Resource Function
Matched against the path of the URI within the HTTP Request
Supports simple URI Templates
- Capture a Path Segment
- Inject the value as a parameter to the Resource Function
- Affords some type conversion

e.g. Matches the URI path: /catalogue/hats/product/5645

RESTXQ 1.0 - Method Annotation

declare
    %rest:GET
	%rest:path("/catalogue/{$category}/product/{$pid}")
function my:product($category as xs:string, $pid as xs:integer) {

  fn:collection("/products/" || $category)/product[@id eq $pid]
};

Zero or More per Resource Function
(One of the Resource Function Constraint Annotations)
Matched against the HTTP Method of the HTTP Request
- Only HTTP 1.1
- OPTIONS, HEAD, GET, POST, PUT, and DELETE

e.g. Matches GET requests for the URI path: /catalogue/hats/product/5645

RESTXQ 1.0 - Content Negotiation 1/2

declare
    %rest:PUT("{$new-product}")
	%rest:path("/catalogue/{$category}/product")
    %rest:consumes("application/xml", "text/xml")
function my:new($category as xs:string, $new-product as document-node(element(product))) {

  xmldb:store("/products/" || $category, $new-product)
};

Consumes Annotation
- (One of the Resource Function Constraint Annotations)
- Matched against the Content-Type Header within the HTTP Request
  - Typically used with POST or PUT Method

e.g. Matches:
- PUT request
- For the URI path: /catalogue/hats/product
- With the Content-Type: application/xml request header

RESTXQ 1.0 - Content Negotiation 2/2

declare
	%rest:path("/catalogue/{$category}/product/{$pid}")
    %rest:produces("application/xhtml+xml")
    %output:method("xhtml")
function my:product($category as xs:string, $pid as xs:integer) {

  fn:collection("/products/" || $category)/product[@id eq $pid]
};

Produces Annotation
- (One of the Resource Function Constraint Annotations)
- Matched against the Accept Header within the HTTP Request
  - Often used with %output: serialization annotations

e.g. Matches GET requests for the URI path: /catalogue/hats/product/5645

RESTXQ 1.0 - Resource Function Parameter Annotations

declare
    %rest:GET
	%rest:path("/catalogue/{$category}/product}")
    %rest:path-param("name", "{$name-query}")
    %rest:path-param("min-score", "{$min-score}", "0.25")
function my:search($category as xs:string, $name-query as xs:string*, $min-score as xs:double*) {

  for $product score $score in fn:collection("/products/" || $category)/product[@id eq $pid]
      [name contains text ($name-query[1] using stemming)]
  where $score ge $min-score[1]
  return
    $product
};

All are optional... unlike constraints!
- Query - HTTP URI Query string parameters
- Form - HTML form field parameters
- Header - HTTP Header parameters
- Cookie - HTTP Cookie Header parameters

RESTXQ 1.0 - Serialization

Reuses XSLT and XQuery Serialization 3.0
- Allows Serialization Parameters to be used as Function Annotations
- e.g. %output:method("xml")
Resource Function may return a rest:response element
- Must be the first item in the result sequence
- Metadata to inform creation of the HTTP Response

declare namespace http = "http://expath.org/ns/http-client";

declare
	%rest:path("/tea")
function my:easter-egg() {
  (<rest:response>
  	  <http:response status="418" message="Tea Time!">
        <http:header name="Server" value="Tea-Bot/1.0.0"/>
        <http:header name="X-With-Milk" value="Never"/>
      </http:response>
    </rest:response> ,
    <img src="https://paradeantiques.co.uk/images/c1860-masonic-treacle-glaze-twin-spouted-teapot-01.jpg"/>)
};

Proposals for RESTXQ 2.0

(My favourites...)

Improved URI Templates in Path Annotation

A URI Template can now capture more than one path segment
Three modes of operation:
- Simple Matching
  - Similar to RESTXQ 1.0
- Basic Regular Expression Matching
  - Maps to a single Resource Function parameter - xs:anyAtomicType
- Capturing Regular Expression Matching
  - Maps to an XDM array(xs:anyAtomicType) type by default
  - Each capturing group maps to a corresponding array entry
  - RESTXQ type conversion rules are applied to each array entry
  - Group 0, i.e. the entire matching pattern, can be optionally appended to the array
Optional additional type conversion information can be added

Path Annotation

For the path: /product/4567/summary
- $pid := xs:integer("4567")

Basic Regular Expression Matching

declare
	%rest:path("/product/{$pid=[0-9]+}/summary")
function local:product-summary($pid as xs:integer)

declare
	%rest:path("/animal/{$common-name}/{$sub-path=.+}")
function local:animal-root($common-name, $sub-path)

For the path: /animal/cat/sleep/anywhere
- $common-name := "cat"
  $sub-path := "sleep/anywhere"
Could alternatively write:

%rest:path("/animal/{$common-name}{$sub-path=.+}")

Path Annotation

Capturing Regular Expression Matching

For the path: /product/HAT9876/summary
- $pid := ["HAT", "9876"]

declare
	%rest:path("/product/{$pid=([A-Z]{3,})([0-9]+)}/summary")
function local:product-summary($pid)

For the path: /product/COAT-123456/summary
- $pid := ["COAT", "123456", "COAT-123456"]

declare
	%rest:path("/product/{$pid==([A-Z]{3,})-([0-9]+)}/summary")
function local:product-summary($pid)

Path Annotation

Capturing Regular Expression Matching with Type Information

For the path: /product/HAT9876/summary
- $pid := [xs:NCName("HAT"), xs:integer("9876")]

declare
	%rest:path("/product/{$pid(xs:NCName, xs:integer)=([A-Z]{3,})([0-9]+)}/summary")
function local:product-summary($pid)

Support for JSON

XQuery 3.1: An XML Query Language added support for "JSON"
- XQuery and XPath Data Model 3.1 added Map and Array types
- XPath and XQuery Functions and Operators 3.1 added functions for loading/parsing JSON
- XSLT and XQuery Serialization 3.1 added JSON serialisation
RESTXQ should be updated to target version 3.1 of XQuery and associated specs.
JSON as the HTTP Response Body
- Simple... Serialization 3.1 spec. has us covered!
JSON in the HTTP Request Body...

JSON in the HTTP Request Body

Can already be accessed from POST and PUT Method Annotations
- RESTXQ 1.0 delivers as xs:base64Binary (or implementation defined mapping)
Add a mapping rule to all Method Annotations that take a body parameter
- If Content-Type: application/json
- Parse following the rules of fn:parse-json
- Resource Function parameter type must be: map(*) or array(*)
Controlling the parsing
- Allow additional arguments to the body parameter name:

declare
    %rest:PUT("{$new-product(liberal=true,duplicates=reject,escape=true)}")
	%rest:path("/catalogue/{$category}/product")
    %rest:consumes("application/json")
function my:hello($category as xs:string, $new-product as map(*)) {

  xmldb:store("/products/" || $category, $new-product)
};

Support for Multipart

Before Maps and Arrays it was unclear how to efficiently support this
An HTTP Request or Response body can be split into distinct multiple parts
- Each has its own set of headers and body
- Each part may itself be a multipart...

Parse HTTP Requests with a multipart/form-data body
Make accessible via Form Parameter Annotation (%rest:form-param)
- RESTXQ 1.0 only supported application/x-www-form-urlencoded: xs:anyAtomicValue*
- RESTXQ 2.0: When parameter type is xs:anyAtomicType:
  - Inject and convert the body of the part; file data is xs:base64Binary
- RESTXQ 2.0: When parameter type is map(xs:string, item()):
  - Extract from the part and convert into a map containing: the content disposition parameters, headers, and body
  - If the part is itself multipart, then recurse into its body...

multipart/form-data

Multipart in the HTTP Request Body

Parse HTTP Requests with a multipart/* body
- Except multipart/form-data, use %rest:form-param instead!
- Minimal compliance:
  - multipart/mixed
  - multipart/related
  - multipart/alternative
- Probably only needed for HTTP POST?
Make accessible via Method Parameter Annotation (%rest:POST)
- The resource function parameter must accept a map(xs:string, item())+
  - Each part is extracted and convert into a map containing: the content disposition parameters, headers, and body
  - If the part is itself multipart, then recurse into its body...

multipart/*

Multipart in the HTTP Request Body

Multipart Map Format

{
  parameters: [
    {
    	name:  "my-content-disposition-parameter-name",
        value: "my-content-disposition-parameter-name"
    },
    
    // For example:
    {
      "name": "filename",
      "value": "filename if multipart/form file upload"
    }
  ],
  
  headers: [
    {
    	name:  "my-header-name",
        value: "my-header-value"
    }
  ],
  
  "body":  xs:string | xs:base64Binary | map(*)+
}

Multipart in the HTTP Response Body

Define a new Serialization Method: rest:multipart
- e.g. output:method("rest:multipart)"
- Resource Function result is a sequence of maps (optional <rest:response> first)
  - Each map is a part (of the multipart response)
  - Each map may contain additional serialization parameters

{
  "rest:response" {
    "output:serialization-parameters": {
      "output:method": "xml"
    }
  },
  
  parameters: [ ... ],
  
  headers: [ ... ],
  
  "body":  xs:string | xs:base64Binary | map(*)+
}

Further Proposals in the Paper

Options for Backwards Compatibility
Support for HTTP Patch
Support for any HTTP Method
Improved HTTP Header Parsing
Support for Server Side Quality Factors in Content Negotiation
Inclusion of Quality Factors in Matching Resource Functions
Serialization Parameters within the Response
Support for Handling XQuery Errors

Still to think about...

A possible Map representation of rest:response
Support CSV in HTTP Request Body
Server Sent Events
Web Sockets
Code Generation of RESTXQ XQuery Module stub from OpenAPI
Development and publication of a clear and complete standard for RESTXQ 2.0

Questions?

Accompanying Paper (2024):
Towards RESTful XQuery 2.0
RESTXQ 2.0 specification source code (NOT STARTED):
RESTXQ 2.0: RESTful XQuery
RESTful XQuery Paper (2012):
RESTful XQuery: Standardised XQuery 3.0 Annotations for REST
(YouTube Video)
RESTXQ 1.0 specification:
RESTXQ 1.0: RESTful Annotations for XQuery

Adam Retter

adam@evolvedbinary.com

@adamretter

Towards RESTful XQuery 2.0

About me

What's in a title?

It seems like only yesterday!

Towards RESTful XQuery 2.0

Setting The Scene

(RESTXQ 1.0)

REST - Representational State Transfer

HTTP - Hypertext Transfer Protocol

RESTXQ 1.0 (2012)

RESTXQ 1.0 - Three Main Concepts:

RESTXQ 1.0 - Processing Model:

RESTXQ 1.0 BY EXAMPLE

(Briefly!)

RESTXQ 1.0 - Simplest Resource Function

RESTXQ 1.0 - Path Annotation

RESTXQ 1.0 - Method Annotation

RESTXQ 1.0 - Content Negotiation 1/2

RESTXQ 1.0 - Content Negotiation 2/2

RESTXQ 1.0 - Resource Function Parameter Annotations

RESTXQ 1.0 - Serialization

Proposals for RESTXQ 2.0

(My favourites...)

Improved URI Templates in Path Annotation

Path Annotation

Basic Regular Expression Matching

Path Annotation

Capturing Regular Expression Matching

Path Annotation

Capturing Regular Expression Matching with Type Information

Support for JSON

JSON in the HTTP Request Body

Support for Multipart

multipart/form-data

Multipart in the HTTP Request Body

multipart/*

Multipart in the HTTP Request Body

Multipart Map Format

Multipart in the HTTP Response Body

Further Proposals in the Paper

Still to think about...

Questions?

Thank You

Towards
RESTful XQuery
2.0

RESTXQ 1.0
BY EXAMPLE