Random inputs for a metadata extension

These informal notes elaborate on an earlier coarse proposal in the attempt to address some of the so called FITS perceived shortcomings

• mainly 4D (unextendable headers)
• and 4K (handling of metadata)

• but also 5B (support to Unicode)

• and collaterally also:
  4F (data organizer),
• 4I (strong kwd typing)
• 3G (thumbnails)
• 3 (large distributed datasets)

• it may also be an alternate way to deal with 4A (long keyword names and long keyword values)

• I also add a note on 2 (streaming support) alhtough it has little relationship with the rest

Table of content

• Layout of "enhanced" FITS files
• the METADATA extension
• Unicode support
• the MIME extension
• the INDEX HDU
• the AtEnd convention

I haven't considered above a particular type of extension which could be termed "dataless HDU", which could be used to store just a collection of ancillary keywords.

• layout of dataless HDU

  HDUm	extension header
  	no data array

The possible organization of the MHDU is described in the next section. The present subsection describes only the arrangement and classification of FITS file layouts as result from the insertion of MHDUs.

The metadata in the MHDU could be relevant to the entire FITS file, or to each individual "true" HDU (i.e. non-dataless). In the latter case each true HDU can be followed by its own MHDU.
However this would break the easy editability requirement, since the "private" MHDUs will be followed by other entire HDUs and not be last in the FITS file.

I argument that typical working files should be either PIFs (plain images) or SEFs (single BINTABLEs or TABLEs) or anyhow relatively simple MEFs. As such a working file should have just a global MHDU at the end. A working file is one which is intended to be manipulated (in place) by its users.

On the other hand, for archival and transfer purposes (of just for data organization), it might be appropriate to have complex MEFs, which could be made "packing together" several working files (inclusive of their MHDU). We could define such files as FITS Archives (FAR).
FARs should not be edited by general users, but just by their creators.
We could also imagine a "far" and "unfar" utility to create a FAR from a collection of individual FITS files, and to extract selected individual FITS files from the FAR. A FAR should contain, ideally as first useful HDU, an index HDU(IHDU) with the list, size and location of the archived components. The possible organization of the IHDU is described in a separate section.

What follows is a tentative preliminary classification of possible "enhanced" FITS files.

• (a) a dataless HDU without data array, containing just a (long) sequence of header keywords, also supporting the newest enhancements (long kwd names, long kwd values, extended-reduced character set)
• (b) a BINTABLE extension with XTENSION='BINTABLE' and EXTNAME ='METADATA', provided the standard for BINTABLE is extended to support the TFORMn for Unicode (U and V) described in the next section
• (c) an extension of the new standard type XTENSION='METADATA' which will be defined mimicking closely the BINTABLE extension, but limiting support to columns of TFORMn equal to nA, nU and nVc(the latter will be valid only in the new extension type, will be used to support Unicode, and are described in the next section

• The data array of a MHDU shall have at least two columns, which shall be named KEYWORD and VALUE (and shall appear in this order ?)
• The data array of a MHDU could have an additional column, named KTYPE, which is described below (and shall appear as third column ?)
• The data array of a MHDU could have additional columns, whose name and details are not regulated by the standard. However their presence is not recommended.
  Or should we make an exceptions for things currently handled in the keyword comments following the slash on the card image (KCOMMENT and units KUNIT) ?

• By default, columns KEYWORD and VALUE should have types TFORM1='kA' and TFORM2='vA' supporting the restricted set (decimal 32 to 126) of ASCII values
• It is TBD whether further restrictions on the range of ASCII values for KEYWORD (i.e. the keyword name) shall be enforced in accordance to usage for customary header keyword (current or new standard).
  It is equally TBD whether special interpretations shall be dictated by the standard for some particular character (e.g. a dot delimiting items in a namespace hierarchy).
• If support to (Unicode) characters beyond the restricted ASCII set is desired, columns KEYWORD and VALUE could have types TFORM1='kU' and TFORM2='vU' or TFORM1='jVk' and TFORM2='wVv'.
• It is TBD whether support to Unicode for KEYWORD (name) is desirable or whether it shall be forbidden or discouraged.
• The number k in TFORM1='kA', 'kU' or 'jVk' corresponds to the maximum length of the keyword name in characters (for ASCII) or code points (for Unicode). It shall be established by the creator of the particular FITS file, to indicate the maximum length actually used or envisaged in the particular file. k shall not be less than 8 (for consistency with the earliest FITS standard) and it is recommended it shall be not less than 54 TBD (for consistency with new standard on keyword name length), but can be longer if desired.
• Trailing blanks (and characters after a NULL, see 7.3.3.1 pag. 23 of the FITS standard) in a keyword name are ignored. Similarly leading and imbedded blanks shall not be allowed.
• The number v in TFORM2='vA', 'vU' or 'wVv' corresponds to the maximum length of the keyword value in characters (for ASCII) or code points (for Unicode). It shall be established by the creator of the particular FITS file, to indicate the maximum length actually used or envisaged in the particular file.
• Trailing blanks (and characters after a NULL, see 7.3.3.1 pag. 23 of the FITS standard) in a keyword value are ignored. It is TBD the handling of leading and imbedded blanks, or of other special characters like quotes (for analogy or at variance with 4.2.1 of the FITS standard)
• The numbers j and w in TFORM1='jVk' and TFORM2='wVv' indicate the maximum number of bytes reserved for the UTF-8 Unicode string. It shall be established by the creator of the particular FITS file in such a way to support all actual strings used in the file. When updating the file, the failure to store a string of k or v code points in j or w bytes shall generate an error and truncate the string to the largest length compatible with the number of available bytes.

• In absence of the KTYPE column, the data type of keyword values shall be interpreted as in section 4.2 of the FITS standard, except that strings shall not be surrounded by quotes, and ... other exceptions TBD
  In particular this means all keywords are interpreted as scalar values
• If support to explicit typingof keywords and to array-valued keywords it desired, the column KTYPE shall be present.
• KTYPE shall habe type TFORM3='tA' with t such to allow the values below (usually t between 1 and 4 should suffice).
• KTYPE shall assume one of the values A, nI, nJ, nK, nE, nD, nC, nM, nL to indicate the value is respectively a string, a 16-, 32- or 64-bit integer, a 32- or 64-bit real, a 64- or 128-bit complex.
  (or one could perhaps support only a reduced set of types)
• n, if absent, defaults to 1 and means the keyword is scalar. String keywords are only scalar. The value shall be encoded in ASCII according to the rules in section 4.2 of the FITS standard, except that strings shall not be surrounded by quotes, and ... other exceptions TBD. Unicode is required only for string valued quantities beyond allowed ASCII values.
• If n is present and larger than 1, the keyword is interpreted as array-valued, with dimension n. All n values of the array shall be present in the VALUE column, encoded in ASCII according to the rules in section 4.2 of the FITS standard or anyhow the rules for scalar-valued VALUE . Consecutive values shall be separated by at least one blank. Consecutive blanks are not significant and should be avoided.
  n shall not assume zero or negative values
• It is care of the creator to make sure the ASCII representation of the value is consistent with the declared KTYPE.

• The presence of a metadata extension in a FITS file shall be advertised in the header of the HDU to which it refers by the presence of the METADATA keyword, which could assume the following TBD values:
  • METADATA= 'GLOBAL' if a single MHDU is present as last extension in the FITS file. This shall be declared in the PHDU of an EPIF (TBD for ESEF, EMEF or FAR), or each non primary HDU in an ESEF, EMEF or FAR, TBD)
  • METADATA= 'LOCAL' if a specific MHDU follows immediately each data HDU (this applies to EMEF and FAR)
  • METADATA= 'GROUP' if a specific MHDU follows a logical unit (group of consecutive HDUs) in a FAR

The header differs only in the first 1-2 lines and includes from 13-14 to 15-16 keywords (i.e. no more than one 2880-byte block).

In the following example dump I anticipate the display of the third optional column KTYPE as second for the sake of clarity.
Splitting of keyword value on several lines is a browser-dependent display artifact, but long values actually are intended to occupy a single table row in the FITS file.

On the other hand the need to support non-Latin scripts, or just a few odd characters here and there, is not a driving scientific need. Therefore I see no reason to add general Unicode support to FITS. Unicode support can remain confined essentially to keyword values, and even there would probably have a limited scope.
That's why in the previous section I limited the introduction of an Unicode data format only within the scope of the METADATA extension (suggesting to make it nominally different from the general BINTABLE).

My experience with Unicode is limited, therefore what follows is to be regarded as highly tentative. Opinions and suggestions by real experts will be greatly appreciated.

According to the Unicode FAQ each Unicode code point (i.e. character or glyph) is univocally described by a 21-bit code. These codes are mapped to formats like UTF-8, UTF-16 or UTF-32. Since 21 bits are a rather odd size, the code charts give an hexadecimal description, which can be used in HTML escapes like Ⴀ or Java escapes like \u10A0 (mainly to handle a few odd characters in a name, like sometimes one uses LaTeX escapes, but cannot be considered a general solution).

UTF-32 looks the simplest though most space-expensive solution to deal with Unicode. Each code point is encoded as 4 bytes, which is always wasteful (32 bits instead of 21) and very wasteful for ASCII text (32 bits instead of 8). Conversely it has the advantage that the length in bytes b is fixed and easily predictable from the length in code points p : b=4p. However UTF-32 (and UTF-16) have the endianness problem.

UTF-8 is probably the most widespread solution dealing with Unicode (at least for plain text processing like e-mail etc.). It is a byte-oriented protocol, exempt from endiannes problems. It is also the most compact representation, with one code point being represented by 1, 2, 3 or 4 bytes in decreasing usage frequency order. In particular pure ASCII (7-bit ASCII including the restricted set commonly used by FITS) is preserved identically in UTF-8. The main "difficulty" is that one cannot a priori predict the length in bytes of the encoding of a string of p code points, unless one actually performs the conversion of a specific string.

I propose for the METADATA extension (and optionally for the entire BINTABLE extension) to support at least one, or both, of the following TFORM:

• kU shall correspond to the UTF-32 representation of a k codepoint string. The length of the field in bytes will be 4k bytes. One should decide whether to support in the FITS standard big-endian, little-endian or unmarked UTF-32 which makes use of BOM

• jVk shall correspond to the UTF-8 representation of a k codepoint string. The length of the field in bytes will be j bytes, where j cannot be less than k (j=k is suitable for 7-bit ASCII, and could perhaps be assumed as default) and the actual value of j has to established by the FITS file creator according to one's needs. An attempt to write an arbitrary string of k codepoints which cannot fit in j bytes shall generate an error, and a corrective actions of writing the longest l<k string (truncated at an integral number of codepoints) which fits in the available length j.

The default action of a FITS reader (displayer) on an Unicode sequence containing non-ASCII characters could be to output it as ASCII inserting HTML or Java escape sequences.

Note that I am not considering any more a possibility I thought of in the past, i.e. to store the actual Unicode text (of variable byte length) in the "heap area" (e.g. as a PB(m) variable-length array), since some people complain about using variable-length arrays for real data, and probably won't like at all to use them for "prompt usage" information like metadata keywords.

The wish to have preview thumbnails attached to a given FITS file was raised in the shortcoming list (item 3G). Of course it does not make sense to have a preview for a TABLE or BINTABLE, so I assume the request referred to FITS images only.
A quick preview, like e.g. a bitmapped image, is something supported e.g. in PostScript files, although not compulsory, and not often used.
Potentially a preview could be managed by a FITS viewer (lik ds9) reading the FITS file and displaying it at low resolution (entire file at zoom setting such to fill current size windows). But possible the requestor meant a preview in one of the common image display formats like GIF, JPEG or PNG.

Therefore ultimately the request translates as a way to attach a non-FITS file to a FITS file, i.e. in a modern way to manage foreign files.

Such (relatively) "modern" way needs not to be unlike the way generic files are attached to an e-mail message, i.e. MIME. Like a Mail User Agent disposes of attachment of arbitrary MIME types, spawning an external viewer or handler, also a FITS viewer or reader could act accordingly encountering a "MIME extension" in a FITS file (I presume this should be prepended to the actual FITS HDU to which it refers.

MIME is defined by RFC 2045 (which I read) and RFC 2046 RFC 2047 RFC 4288 RFC 4289 RFC 2049 (which I did not read).

Essentially exchange of a MIME file implies the construction of an RFC-822-like MIME header, followed by a blank line, following by the file content encoded in one of the possible encodings. There are Linux utilities like base64 to perform BASE64 encoding, and like metamail to analyse a MIME-formatted file and dispatch it to the appropriate viewer.
This means handling of MIME files could be handled mostly by existing utilities, e.g.

• in a writer
• encode the input file to a temporary file (e.g. base64 infile > outfile)
• compute the size in bytes of the temporary file
• write an appropriate MIME header
• In the FITS file write the standard extension FITS header keywords for the MIME extension, reporting the size in bytes.
• copy the MIME header to a set of MIMEnnnn FITS keywords, padding the unused part of the FITS header as usual
• append the temporary file as is, padding it with binary zeros to reach an integral number of 2880-byte blocks.

• in a reader
• extract the MIMEnnnn keywords, reconstruct a valid MIME header and pipe it to metamail or equivalent utility
• pipe a blank line to metamail or equivalent
• pipe the data array (excluding zero-padding) to metamail or equivalent

The MIME header consist of "keyword-value' pairs where the kewyord name is separated by a semicolon from the value. Usually a MIME keyword fits in e.g. a 72-byte line, but it might be continued (wrapped) on more lines if the first character of the next line is blank. For the purpose of FITS I do not propose to match MIME keywords to FITS keywords (the new long-name long-value keywords), but to map the entire MIME header to a sequence of standard (old style) keywords, eventually cutting MIME header records longer than 68 characters to such length and wrapping them to the next lines. Wrapping should occur rather seldom. Provision is made for up to 9999 MIME header lines (after wrapping).

A MIME extension could have a FITS header like this, where the red keywords are mandatory, and with a single parameter while the bluish MIME header is just an example.

The data array of the HDU will be the encoded byte stream corresponding to the MIME file, padded with binary zeros to make up an integral number of 2880-byte blocks.

Alternatively one could imagine instead something like this (which in principle needs not to be a new standard extension, could be just a BINTABLE with EXTNAME='MIME'):

the MIME header follows in the data array as a single-column BINTABLE (or ASCII table), where each row contains a full MIME keyword (name and value).
The encoded file follows instead in the heap area, padded with binary zeros to make up an integral number of 2880-byte blocks.

There are possibly some similarities in the usage with the hierarchical grouping convention which are worth a deeper analysis.

Concerning the "support to Terabyte datasets which span more than one filesystem", I do not regard that it is satisfied, nor that it should be satisfied. If a dataset is so large that it requires a special hardware arrangement, it is also likely that it might need to be replicated only in a few dedicated places inside a project collaboration, and not distributed world-wide to generic users, which might instead be interested in retrieving portions of it (image cutouts, database table subsets, etc.). Therefore this matter is outside of the scope of FITS.

However, talking of "distributed datasets", one might imagine that "components" of the dataset can be located in different sites, and do not need or are not wanted to be replicated at each user's site (imagine for instance a calibration dataset managed in a central site, where the latest update will always be available).
This case could be supported e.g. in a FAR if we allow that some "virtual" HDUs in a FITS file are not physically present within the file, but are instead pointed by some form of URL or URI, as suggested above.