A possible [tclvfs] - [CMcC] 20041029 Storage of metadata (data about data) is an important and undersupported facility of file systems. Tcl supports the [[[file] attributes] command to store some system-defined metadata, and [tclvfs] permits this to be redefined and extended. What this allows is [vfs] which intercepts the [[file attributes]] command and loads/stores attribute (name,value) pairs in some parallel store. Such a vfs would be stackable over other filesystems, so as to provide generic metadata for any filesystem. This would allow applications to simply assume that they could store arbitrary metadata. The question is what form of parallel store would be best? * '''parallel hierarchy''' - metadata is stored in a file hierarchy which parallels the structure of the original, so there's a mapping from ${path} -> ${path}.metadata which contains an array dump of name->value pairs. ''CONS'': expensive/slow, ''PROS'': persistence for free. * '''hidden directory''' - metadata stored in special files hidden in a per-directory ''.metadata'' directory. ''CONS'': expensive/slow, invasive (less so than invasive-interpolation, below), permissions problems, ''PROS'': faster than parallel-hierarchy, metadata is joined with with data (less so than invasive-interpolation) * '''persistent array''' - metadata is stored in an array $metadata($path) as an [[array get]] form per file, to be loaded/stored once, then accessed from memory. Could use [tie] for persistence. ''CONS'': doesn't scale well, persistence needs work ''PROS'': fast. * '''metakit''' - metadata stored in a metakit file, loaded/stored as required. ''CONS'': not pure tcl, slower than persistent-array, ''PROS'': scales, faster than parallel-FS. * '''invasive interpolation''' - metadata stored at the head of each file in (say) RFC822-style name:value lines. ''PROS'': some data (e.g. emails) are already of this form, some data (e.g. caches) can be coerced to this form ''CONS'': invasive, wrecks general files for general (non-tcl) use. * '''multifork files''' - files may have several "forks", one of which is the traditional file and another of which can be used to store metadata. Example: Mac resource forks. Possible implementation on top of a monofork file system: each "file with metadata" is really a directory, where each fork is a separate file, i.e., there could be one $file/data with the file as such and one $file/about with the metadata. PROS/CONS: Similar to "hidden directory". * (add more here) ---- [SEH] -- 4feb05 -- I've been thinking a lot lately about how useful a metadata vfs would be. I think it could lead to a whole new way of developing applications: instead of using relational databases for storage designed to provide a single stovepiped function, one could design and improve new applications on an ongoing basis that access the same filesystem space and each make use of its own preferred attributes. There's no reason why a single file couldn't serve as a weblog entry, a bug report, a todo list item, a mail message, a calendar event, a wiki page, a usenet post, an FAQ contribution, a code patch submission and a documentation paragraph; it would simply be a matter of which application you use to access the filesystem space. I favor a RFC822-style header storage format. Steve Cassidy has made persuasive arguments for this option while discussing metadata storage for [CANTCL]. (Namely, it's well-defined, well-understood, and can be used by any other application that can access a file.) If you didn't want to mix your metadata with your file contents, you could optionally define a "-contents" attribute whose value was the location of the original file and store the attributes separately. That is to say, a vfs with arbitrary attributes would allow you some leeway in choosing your storage method, you wouldn't have to choose one of the above options irrevocably. Such a vfs could do more than get/set name,value pairs. It could allow optional hooks to procedures associated with getting and setting attribute values. Imagine an attribute called "md5", or "sha1". Such an attribute could be calculated on the fly at the time it's requested, rather than calculated and stored once with the attendant risk of becoming inaccurate with the next file write. Such a filesystem stacked on top of a [metakit] vfs could take advantage of the automatic file contents indexing metakit already does to do fast SQL-style queries based on attribute values, thus allowing the design of issue tracking, project management and ERP-type applications, with search and reporting functions. You might be able to do away with relational databases entirely for small to medium sized systems, and best of all you'd get your application integration for free, instead of it being an expensive and complicated extra step. [CMcC] one lovely thing about metakit as a substrate for a vfs is that you can add arbitrary metadata to it quite easily (and dynamically) as new columns which can then be treated as [[file attributes]] ---- [DKF]: [Metadata] editing - a way for users to ''really'' cause havok! ---- [[[Category VFS]]]