Pytz: The Fastest Footgun in the West (2018)(blog.ganssle.io)
blog.ganssle.io
Pytz: The Fastest Footgun in the West (2018)
https://blog.ganssle.io/articles/2018/03/pytz-fastest-footgun.html
124 comments
> New in version 3.9.
But 3.9 is no longer new. If you're still on 3.8 you're running out of time to upgrade as official support is ending next year.
The project has also been backported to 3.6+, so you can still use it if you're stuck on a legacy/unsupported version.
See https://www.python.org/downloads/ and https://pypi.org/project/backports.zoneinfo/
The project has also been backported to 3.6+, so you can still use it if you're stuck on a legacy/unsupported version.
See https://www.python.org/downloads/ and https://pypi.org/project/backports.zoneinfo/
I have no choice, because 3.9 isn't supported on Windows 7. I've always wondered what feature of Windows was so important that they had to drop compatibility with older versions, it's intentionally a very stable interface.
There is a discussion with some insight into the technical reasons why they don't support Windows 7: https://discuss.python.org/t/windows-7-support-for-python-3-... (summary: api-ms-win-core-path-l1-1-0.dll is not available on Windows 7, so they'd have to switch from static import to feature detection + dynamic import in several places; and the Python team don't have access to Windows 7 so they can't do testing).
An unofficial Python installer for Windows 7 exists: https://github.com/adang1345/PythonWin7 (it uses an unofficial implementation of api-ms-win-core-path-l1-1-0.dll).
An unofficial Python installer for Windows 7 exists: https://github.com/adang1345/PythonWin7 (it uses an unofficial implementation of api-ms-win-core-path-l1-1-0.dll).
Thanks for that insight. The part about not being able to do testing doesn't surprise me, it's to be expected.
Windows 7 support ended on 14 Jan 2020. It makes sense not to officially support it.
Have you tried building 3.9 from source to see if there's a particular API that's unsupported?
Have you tried building 3.9 from source to see if there's a particular API that's unsupported?
Sure, I wouldn't expect to get any help from the Python folks if I had some trouble. I was hoping for more of a don't care philosophy. But the installer goes out of its way to disallow installing it.
No, I haven't tried to build from source. That's a rabbit hole I'm not particularly interested in jumping down.
No, I haven't tried to build from source. That's a rabbit hole I'm not particularly interested in jumping down.
These might be interesting to you: https://github.com/indygreg/python-build-standalone/releases
Available to all in a backport: https://pypi.org/project/backports.zoneinfo/
Tangential, but man time zones are just really hard. Some theoretically “simple” rules seem to routinely violate human intuition, and the almanac actually changes based on laws and stuff.
Keeping everything in Zulu ms or ns and rendering into timezones as late as possible is a useful guideline as far as it carries one, but sometimes it’s not enough, this is one of a zillion reasons why e.g. calendaring is just really hard.
I’ve long since mentally tagged “timezone handling”, in any language, as a slow zone: let’s stop and really think this through.
Keeping everything in Zulu ms or ns and rendering into timezones as late as possible is a useful guideline as far as it carries one, but sometimes it’s not enough, this is one of a zillion reasons why e.g. calendaring is just really hard.
I’ve long since mentally tagged “timezone handling”, in any language, as a slow zone: let’s stop and really think this through.
It's even harder. Is the time the user entered actually an always-local time?
As in, if I set my alarm for 08:00, I'd expect it to wake me up 08:00 local regardless if I'm in Oslo or I've traveled to say New York.
However if I make a calendar event things are less clear cut. For meetings and such it makes sense that the time is global time (hence stored in UTC), but for tasks it can be be either or, depending.
As in, if I set my alarm for 08:00, I'd expect it to wake me up 08:00 local regardless if I'm in Oslo or I've traveled to say New York.
However if I make a calendar event things are less clear cut. For meetings and such it makes sense that the time is global time (hence stored in UTC), but for tasks it can be be either or, depending.
This is the part that isn’t even technically hard per se. It’s just hard because there’s literally no way to know what the computer should do because there’s no way to know what a human would even expect.
In many cases reasonable assumptions can be made, e.g. a calendar event for two people in different time zones to meet on a video call almost certainly needs to be scheduled for a proper time (or likely UTC, which is close enough since if you can ignore leap seconds). Although even then I wonder what the two attendees would expect if, between the scheduling of the event and the event itself, both of their local time zones changed their DST rules such that their UTC offset changed by +1 hours. If only one time zone changed its UTC offset we might assume that the previously agreed upon UTC time would hold, but it’s hard to say what the humans would expect if both of their time zones shifted the same way.
In many cases reasonable assumptions can be made, e.g. a calendar event for two people in different time zones to meet on a video call almost certainly needs to be scheduled for a proper time (or likely UTC, which is close enough since if you can ignore leap seconds). Although even then I wonder what the two attendees would expect if, between the scheduling of the event and the event itself, both of their local time zones changed their DST rules such that their UTC offset changed by +1 hours. If only one time zone changed its UTC offset we might assume that the previously agreed upon UTC time would hold, but it’s hard to say what the humans would expect if both of their time zones shifted the same way.
I've been in this situation, and what people expect are that the "main office" time holds, even for remote teams that should know better.
Then again, you don't want all your meetings changing by an hour half the year to avoid a one-week discrepancy.
Then again, you don't want all your meetings changing by an hour half the year to avoid a one-week discrepancy.
If there’s a clear “main office” that’s probably a reasonable assumption, and it’s probably a reasonable feature for calendars to explicitly allow scheduling an event for a specific time zone (instead of converting every datetime to UTC or similar before storing it, which is the usual recommendation among programmers).
I'm pretty sure that both MS and Google default to the time of whoever set up the meeting.
I think it's fixed now, but DB (german transit company) used to always use your local time in its app, making it incredibly confusing to book tickets or even check on tickets you already have. I planned a trip from Japan and the train times showing up were something like 2:30 in the morning.
Another bad example is using Windows bootcamp on a mac. Windows writes local time to the RTC and MacOS writes universal time, so when switching between MacOS and Windows, it's always wrong at first.
Another bad example is using Windows bootcamp on a mac. Windows writes local time to the RTC and MacOS writes universal time, so when switching between MacOS and Windows, it's always wrong at first.
Even worse with transit in Vienna/Austria where if you buy tickets in advance for a specific day or date range they seem to take your local time into account if those tickets are bought while in another timezone. That's for tickets where time shouldn't even matter because they are for full days (from midnight to midnight).
If you're in Pacific time and you buy the ticket for a date in the future (let's say June 1st) before 3pm your local time everything works fine. But if you buy the same ticket for June 1st after 3pm Pacific local time (which is already the next day in Vienna) the ticket that you're buying isn't for June 1st but actually for June 2nd (only visible once you receive the confirmation).
So apparently they're somehow translating between timezones for the displayed dates, although that doesn't make sense: If I buy a ticket for June 1st on Vienna public transport I don't care what my local time is. Even if I'm in California while buying that ticket I still want it to be valid on June 1st instead of June 2nd.
If you're in Pacific time and you buy the ticket for a date in the future (let's say June 1st) before 3pm your local time everything works fine. But if you buy the same ticket for June 1st after 3pm Pacific local time (which is already the next day in Vienna) the ticket that you're buying isn't for June 1st but actually for June 2nd (only visible once you receive the confirmation).
So apparently they're somehow translating between timezones for the displayed dates, although that doesn't make sense: If I buy a ticket for June 1st on Vienna public transport I don't care what my local time is. Even if I'm in California while buying that ticket I still want it to be valid on June 1st instead of June 2nd.
> Windows writes local time
I seem to recall there's a registry key to control that and have it store UTC to the RTC
edit: found that one https://superuser.com/questions/975717/does-windows-10-suppo...
I seem to recall there's a registry key to control that and have it store UTC to the RTC
edit: found that one https://superuser.com/questions/975717/does-windows-10-suppo...
I do this, If you have a unix install on the same host, It really helps keep a sane time config. there are ways to do it the other way around. sysctl kern.utc_offset on openbsd. But the unix way feels better to me, so I change windows.
> But the unix way feels better to me, so I change windows.
I'm with you as well. Sometimes you don't have a choice when dual-booting, MS-DOS only wants to work in local time while on the opposite end of the spectrum MacOS only wants to work in UTC, but if I have the choice I'm going to change Windows to use UTC.
I'm with you as well. Sometimes you don't have a choice when dual-booting, MS-DOS only wants to work in local time while on the opposite end of the spectrum MacOS only wants to work in UTC, but if I have the choice I'm going to change Windows to use UTC.
Good to know, thanks!
I was baffled when I first entered my schedule for a conference into Google Maps, only to find when I flew across the country that all the times were wrong. I understand that it's a reasonable choice to use the local time zone at event creation, but it sure wasn't what I expected (I vaguely thought it would be like your alarm clock example, where it would move with the local time).
I once made a flight by the slimmest of margins possible (the gate crew stopped closing the check in because they saw me sprinting through the terminal hall), because of this and only one hour difference between where I started from and where I went.
I mean obviously wherever a date time entry can be found, there should be a TZ drop-down and a DST checkbox.
lusers who don’t know what those things are will learn soon enough.
I really mean it. The ambiguity is a problem.
lusers who don’t know what those things are will learn soon enough.
I really mean it. The ambiguity is a problem.
I've done the same.
Google did 100% the correct thing, but it goes to show that this is just an inherently tricky thing not limited to whether you are writing code.
Google did 100% the correct thing, but it goes to show that this is just an inherently tricky thing not limited to whether you are writing code.
Thank you for the examples - now I know to not try to make and publish an alarm clock or calendar app/site/feature...
What happens exactly if you set your alarm for 1:30AM on DST changes. Does it go off twice in Autumn and skip altogether in Spring?
The whole DST time change is predicated on the fact that almost nobody sets alarms (or does anything time-sensitive) at 130AM local time. So yeah, if you do that, that’ll happen.
Is that actually true though? Does, say, an iPhone actually fire the alarm both times the clock flips to 1:30am? I tried Googling it but I could only find info about missing alarms entirely when springing forward. I also found some info about how cron works but even then the answer was “it depends which version of cron you’re using.”
You don't even need timezones or DST to run into issues along these lines.
Let's say you have have a repeating 7am alarm on your phone. What should happen if you want to set it to 8am for one day? I mean, without walking you through a dialog that looks like: do you mean today 8, or tomorrow 8?
Let's say you have have a repeating 7am alarm on your phone. What should happen if you want to set it to 8am for one day? I mean, without walking you through a dialog that looks like: do you mean today 8, or tomorrow 8?
> For meetings and such it makes sense that the time is global time (hence stored in UTC)
I think this is tricky too - if you have a 9AM meeting, if your time zone switches, should the time of your meeting change too? Most of the time, no, but if you have a meeting with other people in far away time zones, maybe you don't want it to change.
I think this is tricky too - if you have a 9AM meeting, if your time zone switches, should the time of your meeting change too? Most of the time, no, but if you have a meeting with other people in far away time zones, maybe you don't want it to change.
Doesn’t “meeting” imply multiple people, and the only time you’d need to worry about time zone issues is when those multiple people are not all in the same time zone?
Some people using "meeting" to block off time for the sole purpose of being unavailable for someone else to schedule you in a meeting.
That's hard enough for a calendar program to figure out, but also it can also be affected by whether other meetings involved multiple time zones, especially when something is recurring.
I really like what google calendar does, which is to allow you to display both.
> the almanac actually changes based on laws and stuff.
some of my perennial favorites:
- in samoa, there was never a december 30th 2011.
- in indiana, time zone observance is determined per county
- in arizona, the reservation land observes dst, and the state doesn’t, and reservation lands can enclose non-reservation lands.
- lebanon recently had a legislative dispute about dst such that some entities in the nation did or did not observe dst for about a week.
some of my perennial favorites:
- in samoa, there was never a december 30th 2011.
- in indiana, time zone observance is determined per county
- in arizona, the reservation land observes dst, and the state doesn’t, and reservation lands can enclose non-reservation lands.
- lebanon recently had a legislative dispute about dst such that some entities in the nation did or did not observe dst for about a week.
> - in samoa, there was never a december 30th 2011.
That was interesting, so I had to read about it. Apparently they skipped a day to be closer to Asian time zones. They did the same thing in the opposite direction in 1892 by having two 4th of Julys to be closer to American timezones. Now they kinda just walked back on that.
Interestingly I think the Python datetime approach can handle both of these with no problem. The rest of your list sounds... a lot tougher.
That was interesting, so I had to read about it. Apparently they skipped a day to be closer to Asian time zones. They did the same thing in the opposite direction in 1892 by having two 4th of Julys to be closer to American timezones. Now they kinda just walked back on that.
Interestingly I think the Python datetime approach can handle both of these with no problem. The rest of your list sounds... a lot tougher.
If you're storing historical dates, September 1752 only had 19 days.
The transition from the Julian to Gregorian calendar also happened on vastly different dates in different countries. Most of Western Europe changed in 1582, the British Empire (and its American colonies) in 1752 as you said, and other countries switched even later like Russia in 1918.
oh, historical time is an absolute mess. i have yet to find a library that handles BC dates except maybe astropy — and they have different priorities!
> Zulu ms or ns
Is that timestamps as in unix timestamps? That won't work with birthdates for people more than 53 years old for example; a colleague mentioned a weird issue for someone born in 1938, but that was because the timezone in that location at that time was something like GMT+1:23, then the Germans came around and standardized it to +1:20, then later it was normalized to +1:00.
tl;dr, time and date can only be accurate if time, date and location are known. Actually, some places have moved to different timezones over time as well, so you can't just say "this time has a timezone of GMT", you have to ask "what is the timezone of this location at this time/date"
Is that timestamps as in unix timestamps? That won't work with birthdates for people more than 53 years old for example; a colleague mentioned a weird issue for someone born in 1938, but that was because the timezone in that location at that time was something like GMT+1:23, then the Germans came around and standardized it to +1:20, then later it was normalized to +1:00.
tl;dr, time and date can only be accurate if time, date and location are known. Actually, some places have moved to different timezones over time as well, so you can't just say "this time has a timezone of GMT", you have to ask "what is the timezone of this location at this time/date"
To get to/from that local time representation, you can still store it as a timestamp (to know when it actually happened) and use a timezone database that tells you what the people living there in that moment labeled that time as, right?
You still have the fundamental issue of storing it correctly in the first place. Do most languages time libraries account for localized temporal shifts of this nature by default? It does seem true that you could query an authoritative source, but on entry:
If I say the event happened 60 years ago exactly, and it’s 1pm, what do I mean by “exactly?” Do I mean the day 60 years ago, at 1pm? Or 60 years ago, taking into account timezone shifts? Do I, the person entering this information, even know or care about this distinction? If you enter an event as 60 years ago, would your average joe expect that the time would shift by 23 minutes, or would they think there’s something wrong with the software?
What about timezone differences that vary depending not only on geography, but also on the ethnicity of the people associated with the event?
If I say the event happened 60 years ago exactly, and it’s 1pm, what do I mean by “exactly?” Do I mean the day 60 years ago, at 1pm? Or 60 years ago, taking into account timezone shifts? Do I, the person entering this information, even know or care about this distinction? If you enter an event as 60 years ago, would your average joe expect that the time would shift by 23 minutes, or would they think there’s something wrong with the software?
What about timezone differences that vary depending not only on geography, but also on the ethnicity of the people associated with the event?
> Do I, the person entering this information, even know or care about [the time of the event I'm referring to]?
I'm sorry but if you don't even know yourself what you mean, the computer cannot know it either. The topic here is how to store a datetime, not how to guess meanings.
> What about timezone differences that vary depending not only on geography, but also on the ethnicity of the people associated with the event?
Again, display conversion. Have these natives put their date system into the timezone database and make the UI have a selector for this timezone. Under the hood it still happened at a certain actual moment in time which is what should be stored in the database.
No significant difference from any other timezone that you have to apply to a unix timestamp to get a display time, or conversely, apply to input data to get the timestamp to store.
I'm sorry but if you don't even know yourself what you mean, the computer cannot know it either. The topic here is how to store a datetime, not how to guess meanings.
> What about timezone differences that vary depending not only on geography, but also on the ethnicity of the people associated with the event?
Again, display conversion. Have these natives put their date system into the timezone database and make the UI have a selector for this timezone. Under the hood it still happened at a certain actual moment in time which is what should be stored in the database.
No significant difference from any other timezone that you have to apply to a unix timestamp to get a display time, or conversely, apply to input data to get the timestamp to store.
> What about timezone differences that vary depending not only on geography, but also on the ethnicity of the people associated with the event?
Typically, recently, this problem would occur in disputed territories where the “legal time” at a certain point differs because it is claimed by different polities with different timezone rules, so its more of political allegiance than ethnicity (though those correlate).
Typically, recently, this problem would occur in disputed territories where the “legal time” at a certain point differs because it is claimed by different polities with different timezone rules, so its more of political allegiance than ethnicity (though those correlate).
You are correct but also don't have to look that far: in Indiana USA, different counties have different daylight-saving rules, so "August 13 1pm" is a different time depending in which county you are (or live, or work).
You mean ethnicities that are still deprived of nationhood, so the empire they are subjects of assigns different timezones to them compared to other ethnicities they rule ?? (Any examples ??)
The classic example (and it may in fact be the only extant one) is the occupied West Bank, where Israelis keep a different time than Palestinians (sometimes): https://www.972mag.com/the-worlds-only-ethnic-time-zone/
Yeah, ISTR that there are (or have recently been) some at least theoretical (that is, where there were political claims with different timezones, but not necessarily populations in the affected area using both) ones besides that, but that may well be the only current one.
There's some informal use of a Xinjiang timezone among Uyghurs. China as a whole only has one official timezone.
[1] https://www.timeanddate.com/worldclock/china/urumqi
[more background here] https://www.theatlantic.com/china/archive/2013/11/china-only...
[1] https://www.timeanddate.com/worldclock/china/urumqi
[more background here] https://www.theatlantic.com/china/archive/2013/11/china-only...
I'd have yet another interpretation: the exact time when Earth needed precisely 60 revolutions around the sun to reach its current location.
For things in the past, Zulu is fine. But you won't know Zulu for a date in the future, because Timezone changes may happen.
So Zulu for the past, datetime+location for the future.
So Zulu for the past, datetime+location for the future.
Time zones, character encoding, localization. I admire people who dig in and really understand them deeply, but as far as I’m concerned they are all exposed high voltage wires that I avoid going near.
> I’ve long since mentally tagged “timezone handling”, in any language, as a slow zone: let’s stop and really think this through.
I will second you ok that: date & Time handling across timezones was one of the 2 hardest things I've ever worked on that are deceptively simple on the surface. The other was adding localization/making an app translatable to any language after shipping the first version with hard-coded text.
I will second you ok that: date & Time handling across timezones was one of the 2 hardest things I've ever worked on that are deceptively simple on the surface. The other was adding localization/making an app translatable to any language after shipping the first version with hard-coded text.
Same here. Worrying about current timezones is a hard enough problem. Getting accurate historical timezones handling working is an even harder problem. Getting consistent results across Windows, Mac and Linux is an even harder problem. Needed it for proper handling of historical time series for financial data.
Non-programmers don't care about this nearly as much as we do. The prevailing attitude is "Ok sure there are edge cases, but so what? My use case is simple".
And after thinking about it a lot more, I think this attitude is actually more sensible than mine used to be. Why do we as programmers keep trying to fit a square peg into a round hole? Timezones are a political construct, not a mathematical one.
We should address edge cases by deferring to people as much as possible. Oh, you find it annoying to enter an offset? Don't worry about it, it's optional! Your dentist will expect you for your cleaning at 8:00am +/- 5 hours.
If we keep smoothing over the rough edges for our users, the problem will only get worse. Better to make timezones as painful as possible for as many users as possible. They'll eventually vote for something less obnoxious.
And after thinking about it a lot more, I think this attitude is actually more sensible than mine used to be. Why do we as programmers keep trying to fit a square peg into a round hole? Timezones are a political construct, not a mathematical one.
We should address edge cases by deferring to people as much as possible. Oh, you find it annoying to enter an offset? Don't worry about it, it's optional! Your dentist will expect you for your cleaning at 8:00am +/- 5 hours.
If we keep smoothing over the rough edges for our users, the problem will only get worse. Better to make timezones as painful as possible for as many users as possible. They'll eventually vote for something less obnoxious.
> We should address edge cases by deferring to people as much as possible. Oh, you find it annoying to enter an offset? Don't worry about it, it's optional! Your dentist will expect you for your cleaning at 8:00am +/- 5 hours.
> If we keep smoothing over the rough edges for our users, the problem will only get worse. Better to make timezones as painful as possible for as many users as possible. They'll eventually vote for something less obnoxious.
This is gonna go over great with the people making purchasing decisions.
> If we keep smoothing over the rough edges for our users, the problem will only get worse. Better to make timezones as painful as possible for as many users as possible. They'll eventually vote for something less obnoxious.
This is gonna go over great with the people making purchasing decisions.
As much as I would love this too from a programmer's perspective, reading [1] convinced me that time zones are a necessary evil we just kinda have to deal with. They could be improved by a lot, though. First thing I'd do is get rid of DST everywhere.
[1]: https://qntm.org/abolish
[1]: https://qntm.org/abolish
Thanks for the reference. I have a number of issues with that essay, but fundamentally, once we have people on Mars we'll have exactly this issue. There's no easy way around it and we'll always have to negotiate hours that work for both parties (which is what I basically have to do with my family who are about 90º phase shifted from me).
Implement everything in Zulu, then watch as the frontend ticket for proper rendering stays on the bottom of the backlog, because you always have more concerning problems than users having to do mental math on times.
Are you giving this as genuine advice, or are you suggesting implementing time zones early instead as a form of penance?
[deleted]
I agree. Local time with timezone is really a presentation-layer concern.
For server-side and storage layer, it's saner for everyone to simply store in UTC or epoch seconds/nanoseconds.
For server-side and storage layer, it's saner for everyone to simply store in UTC or epoch seconds/nanoseconds.
Time zones are themselves a footgun. There is no actual reason for morning to start at approximately the same time everywhere. 9am can be 9am everywhere. In some places 9am can be sunrise, in some it can be sunset, and in some it can be the middle of the night. No conversions, no confusion, no resetting of watches...
We don't change the months in the southern hemisphere. Australia is cold in July, Europe is cold in January, this doesn't seem to bother us.
We don't change the months in the southern hemisphere. Australia is cold in July, Europe is cold in January, this doesn't seem to bother us.
> There is no actual reason
There may be no technical reason for it, but having a common reference point for times of the day in terms of human experience and behavior is an actual reason. Saying you were walking down the street at 2am or 3am is immediately distinct from saying you were walking down the street at 10pm or 11pm in ways many people understand. Having to qualify it with the date and geographic location and/or sunrise or sunset patterns would be a much bigger PITA than your average person will ever experience because of time zone inconsistencies. It's just not that important for most people to be able to pinpoint a timestamp to one exact time throughout the world.
We've got GMT. Nothing is stopping anybody from using that to schedule their lives. There's a reason people don't.
There may be no technical reason for it, but having a common reference point for times of the day in terms of human experience and behavior is an actual reason. Saying you were walking down the street at 2am or 3am is immediately distinct from saying you were walking down the street at 10pm or 11pm in ways many people understand. Having to qualify it with the date and geographic location and/or sunrise or sunset patterns would be a much bigger PITA than your average person will ever experience because of time zone inconsistencies. It's just not that important for most people to be able to pinpoint a timestamp to one exact time throughout the world.
We've got GMT. Nothing is stopping anybody from using that to schedule their lives. There's a reason people don't.
It's a lot easier to remember "north and south are opposite" than it is to remember, "Oh right, 9 am is morning in England, early morning in New York, and the middle of the night in Los Angeles." And so on for every possible time of day.
Whereas it's (relatively) easy to guess, "Los Angeles is in California, that's three hours behind me in New York, so it would be 6 am there at 9 am here, so it's too early to call them yet." Not perfect, but not too bad for simple cases, and easy to do in your head most of the time.
Timezones are about standardizing _human_ time, not actual time.
Whereas it's (relatively) easy to guess, "Los Angeles is in California, that's three hours behind me in New York, so it would be 6 am there at 9 am here, so it's too early to call them yet." Not perfect, but not too bad for simple cases, and easy to do in your head most of the time.
Timezones are about standardizing _human_ time, not actual time.
> Tangential, but man time zones are just really hard
That can be expanded. Time and dates is just fundamentally hard.
That can be expanded. Time and dates is just fundamentally hard.
I think keeping time in UTC as primary data for as long as possible is a really bad idea, as explained by Jon Skeet quite a while ago: https://codeblog.jonskeet.uk/2019/03/27/storing-utc-is-not-a...
I know some of the HN crowd are gonna wag their finger at me but the moment I have to deal with timestamps in Python, even for the most simplest of cases, I parachute in pandas. `pd.Timestamp` (and its friends like `pd.Timedelta`) in my opinion are the most powerful timestamp toolkit available anywhere.
To me it's very often easily worth the extra 50-150MB of install size that just works. I've used pandas for around 7 years now and it was inconsistent and buggy at the beginning. But these days, it's practically flawless. Also I'm aware that pd.Timestamp is built on top of many awesome libraries.
To me it's very often easily worth the extra 50-150MB of install size that just works. I've used pandas for around 7 years now and it was inconsistent and buggy at the beginning. But these days, it's practically flawless. Also I'm aware that pd.Timestamp is built on top of many awesome libraries.
Pendulum is quite good[0]. Relatively intuitive interface.
When I left Rails, this is one area I really missed.
Time.use_zone("Singapore") { (Time.zone.now - 3.days).beginning_of_week }
This readable line in datetime utils or even pendulum, is such a pain.
[0] https://github.com/sdispater/pendulum
When I left Rails, this is one area I really missed.
Time.use_zone("Singapore") { (Time.zone.now - 3.days).beginning_of_week }
This readable line in datetime utils or even pendulum, is such a pain.
[0] https://github.com/sdispater/pendulum
Handling timestamps well was one of the motivations for building pandas in the first place (Wes McKinney started pandas while he was working at an investment management company).
I'm not a Pandas person - is it possible to install the Pandas timestamp functionality as a smaller library, or do you need to install the whole shebang?
I had investigated if this was possible to do but I couldn't find a way to do it myself. It's possible that pd.Timestamp might be intricately tied interally to vectorization optimizations using numpy or others. Therefore separating out just the timestamp stuff from pandas would be a lot of work.
Python is in many ways the anti-JavaScript when it comes to packaging. Where JavaScript will have a installable package that contains a single trivial function, Python packages will bundle hundreds of functions, all vaguely related to an area, with no easy way of just installing what you actually need.
My experience has been exactly the opposite, with two exceptions: pandas and numpy.
Generally every other library works in terms of standard python types where possible. Very few other places will you see things like a re-implementation of a time library.
Generally every other library works in terms of standard python types where possible. Very few other places will you see things like a re-implementation of a time library.
with two exceptions
...and scipy and all the scikit libraries and whole bunch of others. 'Accidentally' pulling in hundreds of MBs of libraries you never use for a 100 line data processing script is par for the course in python land.
...and scipy and all the scikit libraries and whole bunch of others. 'Accidentally' pulling in hundreds of MBs of libraries you never use for a 100 line data processing script is par for the course in python land.
My experience is that Pendulum does a good job of handling date/time data as a drop-in replacement for the standard library classes.
> pd.Timestamp is built on top of many awesome libraries
... pytz for example ;-)
... pytz for example ;-)
As long as it hides the footguns, you are good to go. Pytz is not technically flawed.
Well, pytz actually is technically flawed at the moment because it cannot read TZif Version 2+ files, so it breaks for dates >2038 (and it would break if you tried to point it at system data generated with zic's "slim" mode), see this: https://github.com/stub42/pytz/issues/48
But you are right that pandas' use of pytz is benign and as far as I know doesn't suffer the same usability issues.
But you are right that pandas' use of pytz is benign and as far as I know doesn't suffer the same usability issues.
Yes, and? Doesn't mean it inherits all the usability flaws of pytz.
No, I think pandas actually abstracts away pytz's footguns, so that's a good thing.
Django has similar functionality:
from django.utils import timezone
now = timezone.now()
later = timezone.now() + timezone.timedelta(minutes=1)
from django.utils import timezone
now = timezone.now()
later = timezone.now() + timezone.timedelta(minutes=1)
[deleted]
It's a bit unfair to complain that pytz doesn't use the "standard" interface, when that standard interface didn't exist when it was created. The necessary calls for syncing the time zone with the date on datetime creation were a relatively recent addition.
I don't think it is unfair.
If you are looking to solve a timezone problem right now, then you care about what the standard interface is right now... not "when the library was created".
If you are looking to solve a timezone problem right now, then you care about what the standard interface is right now... not "when the library was created".
The documentation is very clear on how to use it to solve your timezone problem. The trouble comes when people expect it to "just work" without understanding how it works.
Until now I never took a close look at how Python deals with timezones. Now I did, and it turns out the way Python handles arithmetic on timezone-aware datetimes is not what I expected. An example to illustrate:
For (C), I would expect 2023-03-26 04:59:59+02:00: that is the point in time 2 hours after (A). As you can see by the times in UTC, (C) is only 1 hour later then (A) instead of 2.
Both feel really wrong.
The article compares pointer arithmetic using Python's time zone model vs pytz's time zone model, and concludes that pytz is worse because you have to normalize the result for the time zone to be adjusted. But the pytz result is correct (according to my expectations, at least), while Python's standard approach does adjust the time zone but without properly adjusting the time (and hence is incorrect, according to my expectations). It seems the author only looked at the time zone, without checking the time itself.
I can work around it by converting to UTC and back, fortunately. I had a quick look at Pendulum and it seems it has functions to do arithmetic properly.
from zoneinfo import ZoneInfo
from datetime import datetime, timedelta, timezone
localtz = ZoneInfo('Europe/Brussels')
utctz = timezone.utc
a_local = datetime(2023, 3, 26, 1, 59, 59, tzinfo=localtz) # right before change to DST
b_local = a_local + timedelta(seconds=2) # 2 seconds later; should be right after change to DST
c_local = a_local + timedelta(hours=2) # 2 hours later
a_utc = a_local.astimezone(utctz)
b_utc = b_local.astimezone(utctz)
c_utc = c_local.astimezone(utctz)
print('datetimes:')
print(f'(A) {a_local} (utc: {a_local.astimezone(utctz)})')
print(f'(B) {b_local} (utc: {b_local.astimezone(utctz)})')
print(f'(C) {c_local} (utc: {c_local.astimezone(utctz)})')
print('deltas:')
print(f'(B-A) local: {b_local-a_local} utc: {b_utc-a_utc}')
print(f'(C-A) local: {c_local-a_local} utc: {c_utc-a_utc}')
datetimes:
(A) 2023-03-26 01:59:59+01:00 (utc: 2023-03-26 00:59:59+00:00)
(B) 2023-03-26 02:00:01+01:00 (utc: 2023-03-26 01:00:01+00:00)
(C) 2023-03-26 03:59:59+02:00 (utc: 2023-03-26 01:59:59+00:00)
deltas:
(B-A) local: 0:00:02 utc: 0:00:02
(C-A) local: 2:00:00 utc: 1:00:00
For (C), I would expect 2023-03-26 04:59:59+02:00: that is the point in time 2 hours after (A). As you can see by the times in UTC, (C) is only 1 hour later then (A) instead of 2.
Both feel really wrong.
The article compares pointer arithmetic using Python's time zone model vs pytz's time zone model, and concludes that pytz is worse because you have to normalize the result for the time zone to be adjusted. But the pytz result is correct (according to my expectations, at least), while Python's standard approach does adjust the time zone but without properly adjusting the time (and hence is incorrect, according to my expectations). It seems the author only looked at the time zone, without checking the time itself.
I can work around it by converting to UTC and back, fortunately. I had a quick look at Pendulum and it seems it has functions to do arithmetic properly.
I've run into this at work and I handle by always add timedeltas in UTC or just design everything to work in UTC.
Unfortunately, some people dislike the former because it's 3 lines of code (convert to UTC, add delta, convert to local). I've had the same data scientist reject 3 MRs fixing this sort of issue in his code. At least it only causes a once per year error or a blip that no one notices.
Always using UTC doesn't work always depending on requirements. The worst was interacting with an API that required precisely every hour of the following day to be sent without good documentation.
Unfortunately, some people dislike the former because it's 3 lines of code (convert to UTC, add delta, convert to local). I've had the same data scientist reject 3 MRs fixing this sort of issue in his code. At least it only causes a once per year error or a blip that no one notices.
Always using UTC doesn't work always depending on requirements. The worst was interacting with an API that required precisely every hour of the following day to be sent without good documentation.
Interesting. I got to thinking why a negative fold (e.g. -1) is not used to indicate local times that do not exist due to standard time to DST transition (moving clocks forward in spring). It is in fact discussed by the PEP: https://peps.python.org/pep-0495/#strict-invalid-time-checki...
The PEP provides instructions for detecting invalid times (that work as long as you use the standard library zoneinfo database.) In short - for invalid times, the UTC offset will be different for different fold versions (1/0) but with converse order than for times that exist doubly:
The PEP provides instructions for detecting invalid times (that work as long as you use the standard library zoneinfo database.) In short - for invalid times, the UTC offset will be different for different fold versions (1/0) but with converse order than for times that exist doubly:
# Non-existent wall time (std -> DST):
>>> print(datetime(2023, 3, 26, 1, 30, tzinfo=ZoneInfo('Europe/London'), fold=0).utcoffset())
0:00:00
>>> print(datetime(2023, 3, 26, 1, 30, tzinfo=ZoneInfo('Europe/London'), fold=1).utcoffset())
1:00:00
# Ambigious wall time (DST -> std):
>>> print(datetime(2023, 10, 29, 1, 30, tzinfo=ZoneInfo('Europe/London'), fold=0).utcoffset())
1:00:00
>>> print(datetime(2023, 10, 29, 1, 30, tzinfo=ZoneInfo('Europe/London'), fold=1).utcoffset())
0:00:00I was going to complain that Python for all of its "batteries included" lacks a simple facility in the standard library to "please give me a timezone for EDT", but apparently one has been added in 3.9: https://docs.python.org/3/library/zoneinfo.html#module-zonei...
The real problem that still is not fixed is that `datetime.date.now()` and others do not do the right thing and encode the system timezone. Rather they record a time without timezone. I have seen this cause so many problems because people are just not aware.
Even worse is that utcnow() returns timezone unaware dates, when it could have easily returned dates with timezone set to utc. Leading to accidentally mix of local and utc times.
Actually, naïve times are considered aware in the system time zone. There's no good way to create a `tzinfo` object that behaves like "the system time zone" and doesn't have unfortunate consequences. See this article: https://blog.ganssle.io/articles/2022/04/naive-local-datetim...
It's not an ideal situation, but given path dependence I don't think it can be improved much from here.
It's not an ideal situation, but given path dependence I don't think it can be improved much from here.
For a second, I thought Jack Ganssle [1] got himself a hipster domain and started to do Python.
[1]: http://www.ganssle.com/tem-back.htm
[1]: http://www.ganssle.com/tem-back.htm
Is something like this why you can only book a ticket on Lufthansa with your correct birth date if you set your computer's time zone to German time?
What are your favorite datetime related Python packages? (not necessarily for tz handling)
I like dateparser for certain use cases:
https://dateparser.readthedocs.io/en/latest/
I like dateparser for certain use cases:
https://dateparser.readthedocs.io/en/latest/
Freezegun for unit testing.
Can stop and step time predictably. Any code using datetime functions will get the time controlled by the test, with no additional abstractions or other boilerplate needed.
Can stop and step time predictably. Any code using datetime functions will get the time controlled by the test, with no additional abstractions or other boilerplate needed.
arrow
This arrow[0], correct? Initially, I thought you meant Apache Arrow and was confused.
https://arrow.readthedocs.io/en/latest/ (looks good!)
https://arrow.readthedocs.io/en/latest/ (looks good!)
Yes, exactly!
I've read the title and thought "oh this seems to be a weird name for a footgun. It would totally make sense if it was about timezones". Then I clicked the link.
Working with date/times in python made me miss Joda-Time/Java time libraries. For my money it's the most well designed date/time API I've worked with.
> Published Mon 19 March 2018 (updated Wed 01 April 2020)
This needs a (2018) (or (2020)) @dang
This needs a (2018) (or (2020)) @dang
Pytz throwing an exception on ambiguous datetimes was the first footgun I discovered with it.
That is the opposite of a foot gun.
That depends if you got it into production or not first :)
A foot nerf gun
python, the footgun factory
> The title of this post claims that pytz is the fastest footgun in the west, by which I mean that pytz is a quite well-optimized library, and historically it has been faster than dateutil.tz
Wat. The author has no idea what a foot gun is.
Pytz is a must because using the standard library and timezone manipulations often leads to bugs. There are nuances around “naive” date times where one can easily just add a fake timezone value, not realizing that the time didn’t actually get localized to the timezone.
There are other libraries such as Arrow and Pendulum worth checking out but pytz is probably enough.
Wat. The author has no idea what a foot gun is.
Pytz is a must because using the standard library and timezone manipulations often leads to bugs. There are nuances around “naive” date times where one can easily just add a fake timezone value, not realizing that the time didn’t actually get localized to the timezone.
There are other libraries such as Arrow and Pendulum worth checking out but pytz is probably enough.
Eh? They're pointing out it's an extremely well optimised and fast library, and then pointing out a number of ways in which it's a footgun because it doesn't behave as you'd expect, e.g.
NYC = pytz.timezone('America/New_York')
dt = datetime(2018, 2, 14, 12, tzinfo=NYC)
print(dt)
# 2018-02-14 12:00:00-04:56
thats horrific
Horrific in the Evil Dead sense, where it’s both horrific and really very funny.
> Wat. The author has no idea what a foot gun is.
In the article, "fastest" is italicized. The sentence is about whether the footgun is fast, not whether the footgun is a footgun.
In the article, "fastest" is italicized. The sentence is about whether the footgun is fast, not whether the footgun is a footgun.
Did you read the article? The author explains exactly why they think Pytz is not enough, and why they think dateutil.tz (not the standard library) is a better alternative.
To be clear, when I wrote this article, I had not yet created the `zoneinfo` module in the standard library. All the recommendations for `dateutil.tz` apply to `zoneinfo` as well, except that `zoneinfo` is also faster than `pytz`.
Thank you so much for your work on zoneinfo. I spend my days working on a legacy Django app and can't wait to switch from pytz to zoneinfo. In my experience, timezones are always a struggle so your efforts to make them easier to deal with is hugely appreciated.
[0]: https://docs.python.org/3/library/zoneinfo.html