# CSDIF Proof of Concept ## Community Science Data Interchange Format --- <br/> <br/> <div style="float: right">  </div> Matthijs Kooijman - MJS Amersfoort Harmen Zijp - MJS Amersfoort Egon Kastelijn - SMAL Zeist Slides: https://md.giplt.nl/csdif-meetkoppel-2026 Website: https://www.csdif.info Email newsletter: https://www.csdif.info/en/newsletter Matrix chatroom: https://matrix.to/#/#csdif:matrix.org Note: - Sponsored by SpukSLA, Zeist, Amersfoort, Provincie Utrecht --- # Collecting data  - Start with simple and fixed table: Time, temperature, humidity, position - Adding fields (lux, battery, PM1, PM10, ...) does not scale - Changing sensors loses metadata - New experiments (tree circumference) do not really fit - Exchange data! Note: - Familiar story for regulars: But now with progress - Add lux, PM - Different PM sensor, more fields, what sensor is used? - Weird data, how is this sensor mounted exactly? --- # What do we need? Collect, store and expose observational data. Needs to: - be flexible: handle different experiments - have metadata: give meaning to data - be standardized: support data exchange Ideally also based on existing technologies and standards. Note: - SMAL Zeist handled "flexible" better from start (right Egon?) - We tried building something ourselves from scratch, but hard choices... --- # Introducing CSDIF - Metastandard: Specifies how to use SensorThings and SensorML - SensorThings for structure and API - SensorML for metadata about stations and sensors - <!-- .element: class="fragment" data-fragment-index="1" --> Based on Observations & Measurements conceptual model - An *Observer* makes an of *Observation* (such as *Measurements*) of a *Property* of a *Feature* (something that is being observed). - <!-- .element: class="fragment" data-fragment-index="2" --> Such as: *Observer* = meetstation-123, *Observation* = 15°C, *Property* = Temperature, *Feature* = Air inside this room - <!-- .element: class="fragment" data-fragment-index="3" --> Or: *Observer* = Matthijs, *Observation* = "they look interested", *Property* = State of mind, *Feature* = People attending this talk --- # Process - 2023/2024: Explored various existing standards, selected SensorThings and SensorML as most promising - 2024/2025: Wrote Request-For-Comments document outlining the goals and ingredients - 2025 Built first proof-of-concept to explore ideas and technologies - 2026+: Use PoC to answer open questions, solicit feedback, write CSDIF specification  --- # CSDIF Proof-of-concept  --- # PoC Parts 1. MJS: Use off-the-shelf SensorThings server for storage (FROST-Server) - Now: Convert existing data before storage - Later: Experiment with CSDIF-aware measurement stations 2. SMAL: Build custom SensorThings layer on existing storage 3. Interactive map to explore MJS and SMAL data - Bonus: RIVM Samen Meten already uses SensorThings Note: - Samen Meten has limited metadata - Next: Highlight experiences with PoC based around questions we wrote in the RFC or discovered in the PoC. --- # Challenge: Complexity - Unused model complexity - Specification complexity - Query / model complexity for own implementation - Terminology complexity - Solutions: Subset, documentation, examples Note: - Solutions are probably not perfect --- # Challenge: Intended usecase - RFC written for data exchange - But realized storage and data access also important for adoption - Needing three different databases raises barrier - All three have subtly different requirements - verbosity vs storage size - query efficiency vs storage size vs processing power - genericity vs low barrier to entry --- # Challenge: Mapping existing data - Stable Identifier generation - Mapping queries onto existing data - Data privacy and security --- # Challenge: Flexibility - SensorThings and SensorML are flexible by using arbitrary URLs - to describe our stations: ```json Identifiers = [ { "definition":"http://sensorml.com/ont/swe/property/ModelNumber", "label":"Model Number", "value":"Si7021" }, ... ] ``` - to describe what is being measured: ```json ObservedProperty = { "definition":"http://qudt.org/vocab/quantitykind/Temperature", "name":"Temperature", "description":"Temperature", } ``` - and a lot more - Uses terms from externally defined "ontologies" (aka vocabularies). --- # URLS are nice, but - What ontologies to use? - There are many - Some old (used by examples, but no longer online) - None of them complete - Maybe start our own? - How to further qualify? PM1 vs PM10? Air temperature vs water temperature? - Experience welcome! --- # Outcomes - MJS CSDIF Data store - Live at https://data-test.meetjestad.net/SensorThings/v1.1/ - Code at https://github.com/meetjestad/mjs_backend_design/tree/backend-prototype-frost-sta - SMAL Zeist CSDIF Data converter - Live at https://sensorthings-api.meten-natuurlijk.nl/sensorthings-api/SensorThingsService/v1.0 - Code at https://codeberg.org/Cooperatief-Meten-Natuurlijk-UA (more to be published) - SensorThings API validator - Code at https://codeberg.org/Cooperatief-Meten-Natuurlijk-UA/sensorthings-validator - CSDIF standard development - RFC at https://www.csdif.info/downloads/CSDIF-001-RFC%20Rev2.pdf - PoC experiences at https://www.csdif.info/en/document-csdif-002-poc --- # Next steps - Use PoC for additional experiments - CSDIF-aware measurement stations - High-volume performance (10yr MJS data) - Express additional metadata - Find minimally required query features - Translate PoC experiences into CSDIF spec - Invite others (you!) to play with our APIs - Spuksla done, but CSDIF part of: - UTMOST project with KNMI - FAIR Data Logger with University of Twente --- # Thanks! Slides: https://md.giplt.nl/csdif-meetkoppel-2026 Website: https://www.csdif.info Email newsletter: https://www.csdif.info/en/newsletter Matrix chatroom: https://matrix.to/#/#csdif:matrix.org --- # Questions? Wishes? Suggestions? Slides: https://md.giplt.nl/csdif-meetkoppel-2026 Website: https://www.csdif.info Email newsletter: https://www.csdif.info/en/newsletter Matrix chatroom: https://matrix.to/#/#csdif:matrix.org