At the moment, I post fairly infrequently. The whole site is written using emacs org mode. Most of the posts have to do with emacs and data stuff (often doing data stuff in emacs).
To conduct this research, we needed to know: (1) which members of congress made which stock trades, and (2) which members of congress belonged to which congressional committees. The data for (1) was available from the the senate/house stock watchers sites; the data for (2) came from the ProPublica Congress API. There was no primary key available for linking the two datasets: the best we had to work with were the names of the members of congress.
This would be fine, if the names were represented uniquely and consistently. This was not the case. You can't join "Mitch McConnell" to "A. Mitchell McConnell, Jr." without a bit of work.
Manually matching every single name from the first data source to every single name in the second would be tedious, time consuming, and error prone. Instead, we used the Levenshtein distance to compute a similarity metric between each name in the first dataset and each name in the second. Simply using the best match according to this metric correctly matched more than 95% of the names, and made it incredibly simple to review the list and manually fix the few incorrect matches.
I wrote this article after a colleague pointed out that the Pandas DataFrame.to_sql() method uses row-by-row INSERTs. There are plenty of good reasons for this, and the to_sql method works great with many different SQL database flavors, but it's not fast.
This article compares the performance of different methods for writing a Pandas DataFrame to a PostgreSQL database using the to_sql method on DataFrames ranging from 100 rows to 10,000,000 rows.
To me it just looks like it's too early to tell. Did quits go up very quickly in 2021? Sure! But that comes on the heels of a massive spike in layoffs that occurred in 2020. It is at least a possibility that the current situation is a response to that.
One point I didn't go into is the fact that the labor force participation rate also dropped steeply in 2020 and hasn't recovered to pre-pandemic levels yet. So that could create labor shortages that are not necessarily represented in the quits rate.
(I wrote this) I wish I had clearer answers to those questions! The conclusion is just a little unsatisfying from a writing perspective—it's just too early to tell exactly what's going on. The quits data through most of 2021 don't look all that different from what we'd expect based on the pre-pandemic trend, though there were definitely more than anticipated, especially later in the year. But this is following a massively-disruptive pandemic that put a lot of people out of work and in general had a dramatic impact on the employment situation.
I think the most interesting part is the decrease in layoffs that coincided with the increase in quits. People aren't leaving that much more than before, but when they leave, they're doing so on their own terms.
Maybe I've been especially fortunate, or I'm just not understanding the question right, but I've felt respected pretty much from the beginning of my career (aside from grad school, which was not great, in many respects). Or at least I've felt "treated with respect" -- not sure if that's exactly the same thing. But I've always been given a fair amount of independence at work, and I've generally been able to solve whatever I'm assigned to solve, or clearly articulate the challenges, allowing those with different/more expertise to help out. And I've never been made to feel ashamed or inadequate because of either the work I've completed or the work I've needed to seek out additional help to complete.
This perspective always bothers me. It's the same with the recent Don't Look Up. The people who will watch and and understand it aren't the people who actually need to get the message. They're both bland movies that present a point of view they know their viewers will agree with. They then get praised--undeservingly, in my opinion--for presenting a "bold" perspective.
I use a lot of U.S. government data sources (EPA climate data; BLS employment statistics; etc.). I also use a fair amount of international greenhouse gas emissions data, such as from the UNFCCC greenhouse gas inventory datasets.
Pain points: data disappearing, moving, or being updated without notice and without indication of a change. Numbers from the same API endpoints or URL changing unexpectedly and without explanation can be an unwelcome surprise.
I use bit.io (https://bit.io -- I work there) to deal with these problems. It's an online PostgreSQL database; very easy to use with e.g. psycopg2/SQLalchemy in Python or DBI+dbplyr in R. Before any analysis, I copy the necessary data over to a repo/schema in bit.io, fill in the documentation with the dates on which I obtained the data, and use that as the source of "ground truth" for the analysis.
The Washington Post did some really great work on generating a variety of comparison datasets: https://www.washingtonpost.com/climate-environment/interacti.... You're right, though -- it's really hard to avoid the issue of political influence in climate data. None of the data can exist in a vacuum; it all has (geo)political implications.
We recently wrote an article (https://l.bit.io/o-cop26) about methane emissions and the COP26 commitment to cut emissions. During the writing of that article, we found some serious inconsistencies in some of the data sources.
Discussions of data quality and validation in data science tend to end with recommendations for a few data validation checks, such as making sure data come from trusted sources; handling missing values; and investigating outliers. These sorts of checks are important, but they won't save an analysis from perfectly-formatted data from a trusted source that happens to be wrong for reasons that can't be found in the dataset itself. Even data of apparently good quality can lead to faulty conclusions.
This article delves into this question by exploring a case study. The U.N. publishes greenhouse gas emissions data supplied each year by parties to the UNFCCC (United Nations Framework Convention on Climate Change). The data are consistent, up-to-date, and well formatted, and the U.N. is a reliable source of official data. However, there is good reason to believe the data submitted by some countries is not accurate. There are other trusted data sources that show startlingly large differences from the U.N. data. In particular, we found that Russia's Methane emissions data were highly inconsistent with the World Resources Institute (WRI) Climate Analysis Indicators Tool (CAIT) data, even though these data were quite similar to the U.N. data for other countries.
Sure, in a constrained sense at least. There are things I wouldn't do/say because I don't especially want to do/say them in a professional context. But I seldom, if ever, feel that I need to act in a truly inauthentic way at work.
A little bit of false enthusiasm now and then? Sure, it makes it easier to get through the day. In both a professional and personal context.
(Article author here) Agreed! Instead we passed a law locking the size of the House at 435 members. And the present political interest in changing this seems limited at best.
Interesting point, though: smaller states end up at the extremes of both over- and under-representation under the current system (though there does appear to be a systematic bias in favor of small states). I wrote about that in a previous article: https://l.bit.io/census-apportionment-bias. Larger states tend to be close to the national average constituent-to-population ratio while smaller states are more likely to be very over- or under-represented.
At the moment, I post fairly infrequently. The whole site is written using emacs org mode. Most of the posts have to do with emacs and data stuff (often doing data stuff in emacs).