Changelog

Unreleased

2025/07/01 v0.0.24

  • DMS: Added option ignore_ddl to ignore DDL events

  • DMS: Added direct column mapping

  • DMS: Started using direct column mapping by default

  • DMS: Started using the mapping value OBJECT when defining column types

2025/06/23 v0.0.23

  • Dependencies: Migrated from zyp to tikray. It’s effectively the same, but provided using a dedicated package now

  • CI: Added support for Python 3.13

  • DMS: Fixed handling of primary keys

  • DMS: Added support for the previously missing drop-table operation

  • DMS: Fixed primary key / where clause rendering

2024/10/28 v0.0.22

  • DynamoDB/Testing: Use CrateDB nightly again

  • DynamoDB: Use ON CONFLICT DO NOTHING clause on CDC operations of type INSERT, to mitigate errors when events are relayed redundantly from retries after partially failed batches on the Lambda processor.

2024/10/09 v0.0.21

  • MongoDB: Fixed BSON decoding of {"$date": 1180690093000} timestamps

  • DynamoDB/Testing: Use CrateDB 5.8.3 for because 5.8.4 can no longer store ARRAYs with varying inner types into OBJECT(IGNORED) columns.

2024/09/30 v0.0.20

  • DynamoDB: Change CrateDB data model to use (pk, data, aux) columns Attention: This is a breaking change.

  • MongoDB: Handle too large $date.$numberLong values gracefully

  • Zyp/Moksha: Improve error reporting when rule evaluation fails

2024/09/26 v0.0.19

  • DynamoDB CDC: Fix MODIFY operation by propagating NewImage fully

  • Zyp/Moksha: Improve error reporting when rule evaluation fails

2024/09/25 v0.0.18

  • MongoDB: Improved MongoDBCrateDBConverter.decode_canonical to also decode non-UUID binary values

  • Zyp/Moksha/jq: to_object function now respects a zap option, that removes the element altogether if it’s empty

  • Zyp/Moksha/jq: Improve error reporting at MokshaTransformation.apply

  • MongoDB: Improved MongoDBCrateDBConverter.decode_extended_json

2024/09/22 v0.0.17

  • MongoDB: Fixed edge case when decoding MongoDB Extended JSON elements

  • Zyp: Added capability to skip rule evaluation when disabled: true

2024/09/19 v0.0.16

  • MongoDB: Added MongoDBFullLoadTranslator and MongoDBCrateDBConverter

  • Zyp: Fixed execution of collection transformation

  • Zyp: Added software test and documentation about flattening lists

  • MongoDB: Use bson package to parse BSON CANONICAL representation

  • MongoDB: Complete and verify BSON data type mapping end-to-end

  • MongoDB: Use improved decoding machinery also for MongoDBCDCTranslator

  • Dependencies: Make MongoDB subsystem not strictly depend on Zyp

  • Zyp: Translate a few special treatments to jq-based MokshaTransformation again

  • Zyp: Improve documentation

  • Packaging: Add missing MANIFEST.in

2024/09/10 v0.0.15

  • Added Zyp Treatments, a slightly tailored transformation subsystem

2024/09/02 v0.0.14

  • Replace poor man’s relation name quoting with implementation quote_relation_name from sqlalchemy-cratedb package.

  • DynamoDB: Add special decoding for varied lists, storing them into a separate OBJECT(IGNORED) column in CrateDB

  • DynamoDB: Improve to_sql() to accept list of records

2024/08/27 v0.0.13

  • DMS/DynamoDB: Use parameterized SQL WHERE clauses instead of inlining values

2024/08/26 v0.0.12

  • DMS/DynamoDB/MongoDB: Use SQL with parameters instead of inlining values

2024/08/23 v0.0.11

  • DynamoDB: Fix serializing OBJECT and ARRAY representations to CrateDB

2024/08/22 v0.0.10

  • DynamoDB: Fix Map representation to CrateDB.

2024/08/22 v0.0.9

  • DynamoDB: Fix String Set and Number Set representation for CrateDB

  • DynamoDB: Fix serializing empty strings

2024/08/20 v0.0.8

  • DynamoDB: Apply rough type evaluation and dispatching when computing values for UPDATE statements

2024/08/17 v0.0.7

  • DynamoDB: Fixed a syntax issue with text data type in UPDATE statements

2024/08/16 v0.0.6

  • Changed UPDATE statements from DMS not to write the entire data column. This allows defining primary keys on the sink table.

2024/08/16 v0.0.5

  • Changed UPDATE statements from DynamoDB not to write the entire data column. This allows defining primary keys on the sink table.

2024/08/14 v0.0.4

  • Added Zyp Transformations, a minimal transformation engine based on JSON Pointer (RFC 6901).

  • Added documentation using Sphinx and Read the Docs

2024/08/05 v0.0.3

  • Added transformer for AWS DMS to CrateDB SQL

  • Dropped support for Python 3.7

2024/07/19 v0.0.2

  • Added transformer for MongoDB CDC to CrateDB SQL conversion

  • Rename “Airrohr” decoder to “Sensor.Community”

2024/07/16 v0.0.1

  • Added decoders for Airrohr, Tasmota, and TTS/TTN from Kotori DAQ

  • Added transformer for DynamoDB CDC to CrateDB SQL conversion