PySpark timezone offset from ISO 8601 without UDF

Tue, 15 Oct 2024 19:39:40 +0200

Background

During my work at Stone Three upgrading some of our data pipelines with Azure DataBricks, I was quite disappointed to learn that Spark 3.5.3’s only timezone-capable timestamp type always stores timestamps converted to UTC, and always displays these timestamps converted to the session-global timezone.

Admittedly, the situation is equally bad with PostgreSQL’s TIMESTAMPTZ type, leading to designs where the actual timezone offset must be stored as a separate timezone offset column.

Having a timezone-aware timestamp type that could store the input timezone natively, and which would always display with its full timezone information, would have been much more useful. In our case, we not only need to know the exact timepoint that one of our vision sensors generated a measurement, but we also need to know what the exact local time was for that specific measurement.

Conservative rendering and liberal parsing of ISO 8601 timestamps in Python

Fri, 05 Aug 2022 15:28:00 +0200

You’ll see in Python bug 37962 that datetime.isoformat() and datetime.fromisoformat() use a restricted subset of the ISO 8601 timestamp formatting standards.

This is similar to the subset followed by W3C.

Recently, we ran into the specific issue that fromisoformat() rejects inputs where the timezone is specified as +hhmm instead of +hh:mm.

Both are allowed by ISO 8601, but only the latter is part of the restricted subset.

Following the Robustness principle (also known as Postel’s law) in the design of our APIs, we would prefer to

Timezones on vxlabs

PySpark timezone offset from ISO 8601 without UDF

Background

Conservative rendering and liberal parsing of ISO 8601 timestamps in Python