How a 20-year-old kernel feature helped USDS improve VA’s network(medium.com)
medium.com
How a 20-year-old kernel feature helped USDS improve VA’s network
https://medium.com/the-u-s-digital-service/how-a-20-year-old-kernel-feature-helped-usds-improve-vas-network-33109cbcb2e6
65 comments
I agree. I'd probably look at it from TCP layer shortly after initial failures to diagnose if not from he start. Especially when dealing with communication between a cloud provider and on-prem gear and infrastructure. However, it's tempting to exhaust all other avenues depending on how likely the on-prem ops folks are to punt the issue.
I actually did look at the TCP layer early on. However, I didn't pay close attention to the TS Val. From the packet dumps, it just appeared that the TCP window had stopped sliding. I couldn't conclude that NSOC's router was at fault.
Getting NSOC on-board is a big deal. After all, they deal with the entire VA network with 100,000+ employees. If you think about it from their perspective, why is USDS' TCP connections so special?
Getting NSOC on-board is a big deal. After all, they deal with the entire VA network with 100,000+ employees. If you think about it from their perspective, why is USDS' TCP connections so special?
Network level troubleshooting is incredibly difficult, especially for individuals who don't have a networking background. Even showing someone how to read wireshark often isn't enough.
I just wanted to politely point out though, in this case, I think there should have been an indications of a network failure in this analysis early on, from the standpoint that TCP frames were sent to the server which were not acknowledged. This would depend on the point where you capture the traffic naturally, but the lack of acknowledgement would be a strong indicator that traffic is not reaching the server, or that replies are not reaching your capture point.
So while the TS Val may be the cause of the drops, I think the packet drops should have stood out when seeing the traffic being black holed, and likely the same segments getting re-transmitted continuously.
And for anyone out their who thinks this is easy to catch, I'd say this is very easy to miss, because you need to have a good understanding of how TCP works in the first place, to know what not working looks like.
I just wanted to politely point out though, in this case, I think there should have been an indications of a network failure in this analysis early on, from the standpoint that TCP frames were sent to the server which were not acknowledged. This would depend on the point where you capture the traffic naturally, but the lack of acknowledgement would be a strong indicator that traffic is not reaching the server, or that replies are not reaching your capture point.
So while the TS Val may be the cause of the drops, I think the packet drops should have stood out when seeing the traffic being black holed, and likely the same segments getting re-transmitted continuously.
And for anyone out their who thinks this is easy to catch, I'd say this is very easy to miss, because you need to have a good understanding of how TCP works in the first place, to know what not working looks like.
True, but Wireshark will highlight dodgy TCP frames (retransmits, dups, etc) which should give a small clue to look further. I agree that it is necessary to understand how TCP works (or have access to someone who does) in order to run Internet services.
I applaud you for not putting the obvious work-around in place:
- Inserting "sleep 300" into the startup process.
- Adding a cronjob to reboot the servers once a week.
I kid, but I'm sure we've all seen hacks like this.
- Inserting "sleep 300" into the startup process.
- Adding a cronjob to reboot the servers once a week.
I kid, but I'm sure we've all seen hacks like this.
I really hope USDS can help introduce interdepartmental digital transfers within the federal government.
To give an example of how frustrating it can be... I went through the visa -> green card -> citizenship process. No two departments talk to one another, and when they do they seemingly transmit information on paper which is then transcribed by hand introducing errors/typos.
For example USCIS does not talk to the SSA digitally at all. I filled in a single form which was used for both my Visa/Green Card and to apply for a Social Security Card on my behalf, my name was spelt correctly on the visa, but got typo-ed during entry into the SSA's system (then the emphasis placed on me to prove their error, even though other US government departments don't have the typo, including any official ID I hold or naturalisation certificate).
Additionally when you earn citizenship the USCIS won't tell anyone. You have to get your piece of paper and physically go tell each department one by one about the change, otherwise nothing will happen.
Why doesn't the federal government just have a big database? Or failing that, why does one department not electronically transfer records to another department? Why are people still hand re-entering information already held digitally?
To give an example of how frustrating it can be... I went through the visa -> green card -> citizenship process. No two departments talk to one another, and when they do they seemingly transmit information on paper which is then transcribed by hand introducing errors/typos.
For example USCIS does not talk to the SSA digitally at all. I filled in a single form which was used for both my Visa/Green Card and to apply for a Social Security Card on my behalf, my name was spelt correctly on the visa, but got typo-ed during entry into the SSA's system (then the emphasis placed on me to prove their error, even though other US government departments don't have the typo, including any official ID I hold or naturalisation certificate).
Additionally when you earn citizenship the USCIS won't tell anyone. You have to get your piece of paper and physically go tell each department one by one about the change, otherwise nothing will happen.
Why doesn't the federal government just have a big database? Or failing that, why does one department not electronically transfer records to another department? Why are people still hand re-entering information already held digitally?
> Why doesn't the federal government just have a big database?
Because that tends to end very badly. I've been in Fed MIS systems since I joined the USAF in 1981, and it's the same just about everywhere except for a few of the research labs.
Remember the OPM data breach, where the background check info for over 20 million people went walkabout? I was one of the lucky winners, and it's a result of the management-by-spreadsheet mentality that's everywhere in the government. On paper, they were fine. In actual fact, not so much.
If the security checklist says "you must have an audit system in place", and you have the system installed and running, you pass. Nothing is said about ever looking through the logs for atypical behavior that might indicate a breach.
You want to buy new software? If it's not Oracle or Microsoft, good luck. If there's not a contractual vehicle in place to use for the purchase, forget it. Whether or not the software is fit for purpose has no bearing on the matter.
Because that tends to end very badly. I've been in Fed MIS systems since I joined the USAF in 1981, and it's the same just about everywhere except for a few of the research labs.
Remember the OPM data breach, where the background check info for over 20 million people went walkabout? I was one of the lucky winners, and it's a result of the management-by-spreadsheet mentality that's everywhere in the government. On paper, they were fine. In actual fact, not so much.
If the security checklist says "you must have an audit system in place", and you have the system installed and running, you pass. Nothing is said about ever looking through the logs for atypical behavior that might indicate a breach.
You want to buy new software? If it's not Oracle or Microsoft, good luck. If there's not a contractual vehicle in place to use for the purchase, forget it. Whether or not the software is fit for purpose has no bearing on the matter.
This sounds incredibly frustrating! Almost all of my USDS projects have involved moving data across agencies, including quite a few with USCIS. It's definitely one of our most common challenges, and USDS is often called into help because we are uniquely positioned to work across departments.
As you probably realized when going through the process, USCIS has historically been 98% paper [1] (I've been to the underground limestone cave where they store a lot of it), and we've been working hard to help them modernize the entire agency, including an online application for naturalization [2] and the corresponding backend processing systems.
There are a lot of reasons why it's so hard to get agencies to work together, but making it better starts with modernizing individual systems and processes, especially when we're starting with paper that, obviously, can't be transferred seamlessly. For the most part, USCIS doesn't store your entire case file digitally (yet), and even the metadata is stored in a bunch of different systems, which includes (of course) an actual mainframe. (I've seen the mainframe too; it has pretty sweet green LED strips, and not much else going for it.)
I'm actually not familiar with how USCIS triggers SSA cards, but I'm going to ask. As it happens, USCIS and SSA do interact electronically in some situations. USCIS runs the E-Verify program, which talks with SSA, as described in this dense but refreshingly public privacy document: https://www.dhs.gov/sites/default/files/publications/privacy.... We've also helped USCIS introduce and improve data exchanges with State, including an early engagement on modernizing the immigrant visa process [3], which you went through, and work on refugee admissions [4].
One of the things I love most about my time at USDS is how many civil servants have embraced and championed best practices for building digital services to best serve the American people. The former director of USCIS, in particular, intuitively understood how technology can improve the immigration process and continued to push us and the agency until their final day in office. As USCIS makes more benefits applications available online, they'll have more data in a readily accessible digital format. They'll be able to streamline the user experience as you progress through the process over the years, and by the end of it, they shouldn't need to ask you a whole lot. It's been great to see human-centered design being championed over and over.
Congratulations on becoming a citizen! Want to help us continue to improve the immigration system? We could use the help: https://www.usds.gov/join
[1] https://medium.com/the-u-s-digital-service/technology-is-hel...
[2] https://my.uscis.gov/exploremyoptions/us_citizen_through_nat...
[3] https://obamawhitehouse.archives.gov/blog/2015/07/15/bringin...
[4] https://www.usds.gov/report-to-congress/2016/refugee-admissi...
As you probably realized when going through the process, USCIS has historically been 98% paper [1] (I've been to the underground limestone cave where they store a lot of it), and we've been working hard to help them modernize the entire agency, including an online application for naturalization [2] and the corresponding backend processing systems.
There are a lot of reasons why it's so hard to get agencies to work together, but making it better starts with modernizing individual systems and processes, especially when we're starting with paper that, obviously, can't be transferred seamlessly. For the most part, USCIS doesn't store your entire case file digitally (yet), and even the metadata is stored in a bunch of different systems, which includes (of course) an actual mainframe. (I've seen the mainframe too; it has pretty sweet green LED strips, and not much else going for it.)
I'm actually not familiar with how USCIS triggers SSA cards, but I'm going to ask. As it happens, USCIS and SSA do interact electronically in some situations. USCIS runs the E-Verify program, which talks with SSA, as described in this dense but refreshingly public privacy document: https://www.dhs.gov/sites/default/files/publications/privacy.... We've also helped USCIS introduce and improve data exchanges with State, including an early engagement on modernizing the immigrant visa process [3], which you went through, and work on refugee admissions [4].
One of the things I love most about my time at USDS is how many civil servants have embraced and championed best practices for building digital services to best serve the American people. The former director of USCIS, in particular, intuitively understood how technology can improve the immigration process and continued to push us and the agency until their final day in office. As USCIS makes more benefits applications available online, they'll have more data in a readily accessible digital format. They'll be able to streamline the user experience as you progress through the process over the years, and by the end of it, they shouldn't need to ask you a whole lot. It's been great to see human-centered design being championed over and over.
Congratulations on becoming a citizen! Want to help us continue to improve the immigration system? We could use the help: https://www.usds.gov/join
[1] https://medium.com/the-u-s-digital-service/technology-is-hel...
[2] https://my.uscis.gov/exploremyoptions/us_citizen_through_nat...
[3] https://obamawhitehouse.archives.gov/blog/2015/07/15/bringin...
[4] https://www.usds.gov/report-to-congress/2016/refugee-admissi...
While I found this article very interesting, I feel like something is missing here.
So linking this issues to a Cisco bug is very interesting, that dropping connections would cause the application to lock up / crash, while all the connections to the database were dead.
My question is why would the application lock up and the servers would crash?
I don't see it very often, but when striving for high availability and strong resiliency (which isn't reasonable for everyone), issues need to be looked at in great detail. So I would be trying to look at the second side of the story, which is why was there crashes encountered under these circumstances, and are there other plausible triggers that could cause a similar set of circumstances.
Disabling timestamps does avoid the Cisco bug, but a similar set of triggers could be encountered anytime the VPN connection dropped, or if the firewall failed over without the state tables in sync, or any number of other network conditions.
And don't take me wrong, I don't know if the OP did this, but based on the article, I would lean towards disabling timestamps as a workaround, and this might still be an indicator that something in the app isn't behaving correctly when the database is unavailable.
So linking this issues to a Cisco bug is very interesting, that dropping connections would cause the application to lock up / crash, while all the connections to the database were dead.
My question is why would the application lock up and the servers would crash?
I don't see it very often, but when striving for high availability and strong resiliency (which isn't reasonable for everyone), issues need to be looked at in great detail. So I would be trying to look at the second side of the story, which is why was there crashes encountered under these circumstances, and are there other plausible triggers that could cause a similar set of circumstances.
Disabling timestamps does avoid the Cisco bug, but a similar set of triggers could be encountered anytime the VPN connection dropped, or if the firewall failed over without the state tables in sync, or any number of other network conditions.
And don't take me wrong, I don't know if the OP did this, but based on the article, I would lean towards disabling timestamps as a workaround, and this might still be an indicator that something in the app isn't behaving correctly when the database is unavailable.
You are dead on. We do have a bug where we are not recovering the Oracle connectivity correctly. It is on our radar to address the issue. https://github.com/department-of-veterans-affairs/caseflow-m...
However, There is actually another 50% of the story that I never posted. VACOLS is a really old Oracle DB (from the 80s) that is out of our control. Somehow, it has a "feature" where you can only make one TCP connection to it every 2-3 second. So if we lose connection to the database, it will take many seconds to recover. At that point, our ELB health-check would've fired and restarted our EC2 instances. This is why recoverability of the database connection is not an immediate priority.
Here's how we preallocate the VACOLS connection pool to workaround this throttling feature. https://github.com/department-of-veterans-affairs/caseflow/b...
The infrastructure we operate in are very challenging (and interesting) because of legacy systems. That's why common sense engineering often may not apply in USDS.
However, There is actually another 50% of the story that I never posted. VACOLS is a really old Oracle DB (from the 80s) that is out of our control. Somehow, it has a "feature" where you can only make one TCP connection to it every 2-3 second. So if we lose connection to the database, it will take many seconds to recover. At that point, our ELB health-check would've fired and restarted our EC2 instances. This is why recoverability of the database connection is not an immediate priority.
Here's how we preallocate the VACOLS connection pool to workaround this throttling feature. https://github.com/department-of-veterans-affairs/caseflow/b...
The infrastructure we operate in are very challenging (and interesting) because of legacy systems. That's why common sense engineering often may not apply in USDS.
I also bet, those challenging legacy systems in many case are way better built than what "modern" systems would provide. Sure there will be whacky things to work around, but I've seen my share of whacky engineering in brand new systems too. Common sense engineering seems to be few and far between these day's.
Kudos on having something interesting to work on.
Kudos on having something interesting to work on.
>dropping connections would cause the application to lock up / crash
plenty of programs/libraries do that, for example ffmpeg, and by extension MPlayer. It so happens Google/YT has anti resource starvation measures in place on their content servers in case someone opens TCP session and keeps it paused with ZeroWindow packets for over 3 minutes. Guess how MPlayer handles/signals pausing a stream :(
plenty of programs/libraries do that, for example ffmpeg, and by extension MPlayer. It so happens Google/YT has anti resource starvation measures in place on their content servers in case someone opens TCP session and keeps it paused with ZeroWindow packets for over 3 minutes. Guess how MPlayer handles/signals pausing a stream :(
It's like a 2017 version of the 500 mile email bug
https://www.ibiblio.org/harris/500milemail.html
https://www.ibiblio.org/harris/500milemail.html
The Jiffies root cause leads to an interesting idea: an Glossary of Magic Constants where all kinds of important constants, limits, and overflows are tracked to aid in debugging. You could imagine a search engine where "tcp connection drops after 5 minutes" lists every piece of software and firmware with 5 minute and 300 second constants.
So OEIS for programming? Largely seems covered by Google: punch in an oddly specific number and someone will probably have discussed it on Stack Exchange.
Reminds me of trying to debug long-lived SSH tunnels which would fail every 2:11:15 hours. Right down to the hardcoded value in net.ipv4.tcp
Would this (disabling TCP Timestamps) affect TCP Performance with other OSes in regard of their respective TCP Window Auto Scaling Implementations? I believe Linux uses DRS (1) and doesn't necessarily depend on TCP Option TS for TCP Window Auto Scaling and FreeBSD has got this (2) commit ~ 2 Months ago.
(1) http://public.lanl.gov/radiant/pubs.html#DRS (2) https://svnweb.freebsd.org/base?view=revision&revision=31667...
(1) http://public.lanl.gov/radiant/pubs.html#DRS (2) https://svnweb.freebsd.org/base?view=revision&revision=31667...
Yup, it would. Disabling the TS option was just a stopgap measure to make our deployments stable for the time being.
Of course, setting priorities.. I was just wondering how different OSes would behave under these circumstances. For instance AWS S3 also doesn't support TCP Timestamps and this had a rather big impact on e.g. FreeBSDs TCP Performance until recently.
A team at Veterans Affairs is my customer. They have been tasked to integrate with some AWS hosted intranet system. Other than points of integration, we know very little about it. This article seems to be a big clue.
Some people might not have realized USDS is still around since it was best known for the Healthcare.gov rescue under Obama. But it's still here, and still hiring people to work on problems like this www.usds.gov/join
Hmm.... I apply every 6 months or so and get the thumbs down. Not sure what they're looking for. I've got 30 years of every kind of experience (dev, DBA, network, security, product mgmt, analytics/data science, business mgmt, and more) with good credentials and they never bite. I wish I knew more what the ideal profile was; I'd love to help out!
I can provide a little general insight on this. Here are some things that might lead one to get rejected from USDS Engineering without an interview:
- You're too junior (not your case, I understand.)
- It is not clear that you've actually written software recently as a professional. (While we're looking for senior people and do have management needs, over time we've struggled with hiring managers -vs- growing them because government is such a radically different environment that success managing in the private sector is a poor indicator. So engineering managers are evaluated primarily as engineers first, managers after they pass)
- Your skills seem too specialized in areas we do not have needs. (Much of government technology is super old and much of what is wrong with it is technical debt and decay, not cutting edge technical challenges. For that reason we prefer generalists. Again this doesn't sound like your case)
- Not enough web development work (There is an on-going debate about this, but realistically citizen facing services tend to mean websites and the infrastructure that supports them. In the past we have hired engineers who were unable to adjust to web development and we couldn't find them enough work to play to their strengths. So while we're open to software engineers from other disciplines, there's still a lot of inconsistency in how the engineers judging your resume weigh this issue. Our attempts to correct this are ongoing.)
- You've applied as an engineer and emphasized non-engineering accomplishments (Since we're a civic tech organization sometimes people curate their resumes to play up their social good activities instead of their engineering. This is without a doubt the wrong move. If our engineers don't think you can write code they will not clear you for a technical interview.)
- You've applied for the wrong role or it's not clear what role you would fit into (This seems like it might your case. USDS has three types of roles [well five, but two are not really relevant here]: Engineering, Design (which includes visual, UX research, and content strategy), and Strategy/Operations (which includes both our front office administration and people who are coming in with significant government/policy/legal/product management experience. While we definitely have people who straddle lines [PMs with engineering backgrounds, designers who can program, etc] all those people still applied and were evaluated for one specific community.)
- You're too junior (not your case, I understand.)
- It is not clear that you've actually written software recently as a professional. (While we're looking for senior people and do have management needs, over time we've struggled with hiring managers -vs- growing them because government is such a radically different environment that success managing in the private sector is a poor indicator. So engineering managers are evaluated primarily as engineers first, managers after they pass)
- Your skills seem too specialized in areas we do not have needs. (Much of government technology is super old and much of what is wrong with it is technical debt and decay, not cutting edge technical challenges. For that reason we prefer generalists. Again this doesn't sound like your case)
- Not enough web development work (There is an on-going debate about this, but realistically citizen facing services tend to mean websites and the infrastructure that supports them. In the past we have hired engineers who were unable to adjust to web development and we couldn't find them enough work to play to their strengths. So while we're open to software engineers from other disciplines, there's still a lot of inconsistency in how the engineers judging your resume weigh this issue. Our attempts to correct this are ongoing.)
- You've applied as an engineer and emphasized non-engineering accomplishments (Since we're a civic tech organization sometimes people curate their resumes to play up their social good activities instead of their engineering. This is without a doubt the wrong move. If our engineers don't think you can write code they will not clear you for a technical interview.)
- You've applied for the wrong role or it's not clear what role you would fit into (This seems like it might your case. USDS has three types of roles [well five, but two are not really relevant here]: Engineering, Design (which includes visual, UX research, and content strategy), and Strategy/Operations (which includes both our front office administration and people who are coming in with significant government/policy/legal/product management experience. While we definitely have people who straddle lines [PMs with engineering backgrounds, designers who can program, etc] all those people still applied and were evaluated for one specific community.)
I wonder if the environment of experience is significant? USDS positions itself like a startup (even their page has a section on "dress code" which mentions being like "any other startup"). Someone whose experience is primarily enterprise or BigCo might be less appealing. It would be interesting to see a roster of current USDS FTEs and their backgrounds (I didn't see a "Who's Who" on their page, but didn't look extensively).
Hey, I'm the current director of engineering, so I'll toss in my 2 cents because I want to make sure this is really, really clear.
I could not care less about what which company/non-profit/agency/whatever you worked for. I just care about what you have stepped up and done in that environment. We will gladly consider a BigCo person, as long as they can demonstrate how they managed to get awesome stuff done at BigCo (e.g. ported a legacy system to a modern stack) because that is relevant to what we do. We have also passed on folks from FancyStartup.com because they show signs that they would struggle in an environment this complex.
Fundamentally, who we look for are people who can go into just about any situation and figure out a reasonable way to deal with it so that we can get services working for the people who need them. And we look for this in our designers, engineers, talent team, procurement specialists, and product and strat ops folks. See mbellotti's comment for great info on some engineering-specific tips.
The Federal government is a giant bureaucracy. I would actually love to see more BigCo applicants with experience succeeding in tough environments because that's what we do here. We need a good balance of people who know how to deal with things the way they are and people who know how to build things the way they should be. The best candidates are folks who can do both. We make the best decisions we can based on resumes and interviews, and it sucks when we may miss out on a super swell person. But it's bound to happen sometimes.
To be clear: I don't care how old you are, what company you worked for, what part of the country you're from, what school you went to (or if you didn't go to school at all), or any other artificial criteria. In fact, I'm really not interested in hiring the same profile over and over again. That's ineffective and honestly, boring. All I care about is finding people who demonstrate that they can creatively and scrappily solve the types of problems we tend to see in the types of environments/situations in which we tend to land. Period.
So if any of this sounds like you, consider applying. This country and it's citizens could really use your help. And make sure your resume shows us that this sounds like you. :)
I could not care less about what which company/non-profit/agency/whatever you worked for. I just care about what you have stepped up and done in that environment. We will gladly consider a BigCo person, as long as they can demonstrate how they managed to get awesome stuff done at BigCo (e.g. ported a legacy system to a modern stack) because that is relevant to what we do. We have also passed on folks from FancyStartup.com because they show signs that they would struggle in an environment this complex.
Fundamentally, who we look for are people who can go into just about any situation and figure out a reasonable way to deal with it so that we can get services working for the people who need them. And we look for this in our designers, engineers, talent team, procurement specialists, and product and strat ops folks. See mbellotti's comment for great info on some engineering-specific tips.
The Federal government is a giant bureaucracy. I would actually love to see more BigCo applicants with experience succeeding in tough environments because that's what we do here. We need a good balance of people who know how to deal with things the way they are and people who know how to build things the way they should be. The best candidates are folks who can do both. We make the best decisions we can based on resumes and interviews, and it sucks when we may miss out on a super swell person. But it's bound to happen sometimes.
To be clear: I don't care how old you are, what company you worked for, what part of the country you're from, what school you went to (or if you didn't go to school at all), or any other artificial criteria. In fact, I'm really not interested in hiring the same profile over and over again. That's ineffective and honestly, boring. All I care about is finding people who demonstrate that they can creatively and scrappily solve the types of problems we tend to see in the types of environments/situations in which we tend to land. Period.
So if any of this sounds like you, consider applying. This country and it's citizens could really use your help. And make sure your resume shows us that this sounds like you. :)
I think that startup mentality might bite them in the arse.
I saw "React on Ruby" and winced.
There is nothing wrong with that platform as a "We are in a market where things will change radically in two years" but for the VA? Where things might change once a decade, that's a recipe for pain.
Look at where the Web was 5 years ago (hell React didn't exist) never mind 10.
Angular is 7 years old, KnockoutJS is 7, jQuery is the grandaddy at 11, React is 4.
Not a criticism (they are clearly doing important impactful work) more a concern.
If someone said to me "You will have to support this for at least 10 years" the choices I made would be extremely conservative.
I saw "React on Ruby" and winced.
There is nothing wrong with that platform as a "We are in a market where things will change radically in two years" but for the VA? Where things might change once a decade, that's a recipe for pain.
Look at where the Web was 5 years ago (hell React didn't exist) never mind 10.
Angular is 7 years old, KnockoutJS is 7, jQuery is the grandaddy at 11, React is 4.
Not a criticism (they are clearly doing important impactful work) more a concern.
If someone said to me "You will have to support this for at least 10 years" the choices I made would be extremely conservative.
To me, this looks like the bigger potential problem:
>U.S. Digital Service members join us for what we call a tour of duty. We are seeking candidates interested in joining the U.S. Digital Service fulltime, ideally for at least 12 months. In some cases, we can accommodate candidates who can only commit to a shorter amount of time. Three months is the minimum time commitment we can accommodate. All members of the U.S. Digital Service hold "term-limited" positions, which means that at the end of a prescribed term, the candidate's employment with that agency must end.
You have to move to DC -- without relocation assistance -- knowing that you're only going to work for USDS with an expiration date? Seems like that'd really shrink the net of candidates to me. I know it kind of kills my interest, personally.
>U.S. Digital Service members join us for what we call a tour of duty. We are seeking candidates interested in joining the U.S. Digital Service fulltime, ideally for at least 12 months. In some cases, we can accommodate candidates who can only commit to a shorter amount of time. Three months is the minimum time commitment we can accommodate. All members of the U.S. Digital Service hold "term-limited" positions, which means that at the end of a prescribed term, the candidate's employment with that agency must end.
You have to move to DC -- without relocation assistance -- knowing that you're only going to work for USDS with an expiration date? Seems like that'd really shrink the net of candidates to me. I know it kind of kills my interest, personally.
Yeah, this is hard. I worked from home for five years before joining USDS, and I wish we could support remote work, but we're grafted onto agency projects that are so often based in D.C. that it's just not possible.
The "term-limited" positions are actually quite nice. Generally, you can serve for up to four years. USDS is almost 3 years old, and anecdotally it feels like people tend to leave by the end of their first two-year term because, well, it's hard and it can be exhausting. You lose effectiveness over time, and returning to the private sector is important to re-strengthen your skills and knowledge.
As it happens, the excellent career civil servants we work with also appreciate knowing we're there to help them, not take their jobs. :-)
The "term-limited" positions are actually quite nice. Generally, you can serve for up to four years. USDS is almost 3 years old, and anecdotally it feels like people tend to leave by the end of their first two-year term because, well, it's hard and it can be exhausting. You lose effectiveness over time, and returning to the private sector is important to re-strengthen your skills and knowledge.
As it happens, the excellent career civil servants we work with also appreciate knowing we're there to help them, not take their jobs. :-)
What also concerns me about that is maintenance. You're constantly bringing in new people to build new things who have no knowledge of what people in previous "tours" built. The overheard of all the handoffs and knowledge transfers that needs to happen seems unfortunately high.
While the USDS does build things, the model is to have them partner with career civil servants and contractors and get them to implement industry best practices. So there is still overhead when handing off between tours, but the bigger problem is finding capable contractors and vested partners at the agency's who can champion the new way of doing things.
Our team is very careful at choosing technology that is stable and well proven. In fact, it took a lot of debate for us to move from Rails static pages to React SPA.
As for government being slow and only change once a decade, you might be underestimating engineers in the government.
Here's the disability claim processing project I worked on when I was on SSA Digital Service. https://federalnewsradio.com/ask-the-cio/2017/01/ssa-turns-c...
The team I worked with managed to pull off React/Node.js/Redis/Zookeeper microservice framework over AWS using Jenkins as CI/CD pipeline. And most of the team came from an older enterprise stack using Java and IBM WebSphere. They were able to adapt to the new framework.
Be conservative, yes. But being overly conservative is part of the reasons why government is so far behind today.
As for government being slow and only change once a decade, you might be underestimating engineers in the government.
Here's the disability claim processing project I worked on when I was on SSA Digital Service. https://federalnewsradio.com/ask-the-cio/2017/01/ssa-turns-c...
The team I worked with managed to pull off React/Node.js/Redis/Zookeeper microservice framework over AWS using Jenkins as CI/CD pipeline. And most of the team came from an older enterprise stack using Java and IBM WebSphere. They were able to adapt to the new framework.
Be conservative, yes. But being overly conservative is part of the reasons why government is so far behind today.
Interesting, when I said once a decade I didn't particulary mean changing anything I meant that the system as a whole has to last a decade.
I've stuff in production I wrote in 2007, it's the same system but lots of the parts have been replaced (bit of a Theseus/Triggers Broom philosophical point) over time.
I agree on the overly conservative point, what I find helps there is to consider the migration away from anything new you are considering, I usually ask myself the following question - "If the entire dev team for foobar disappeared tomorrow, how much would it hurt" if the answer is "not much because I can maintain it myself while migrating away" then thats very different to "a lot because I don't understand it that well and/or it's massively complex".
Enterprise is a different world (even for a medium sized one like I work for) because (frankly) a lot of the programmers are doing it purely for the money and/or simply aren't that competent.
Not all of them by any means but I run into a lot of code that is just plain horrible, not "not the way I'd have done it" so much as "how does this thing even work?" and "Who possibly thought this was the right approach?".
I'm weird, I like LoB software development - I find that done well the feedback loop is very gratifying, you get to have an immediate affect on the companies bottom line and you have happier employees also the problems have a lot of hidden complexity which is intellectually stimulating.
I've stuff in production I wrote in 2007, it's the same system but lots of the parts have been replaced (bit of a Theseus/Triggers Broom philosophical point) over time.
I agree on the overly conservative point, what I find helps there is to consider the migration away from anything new you are considering, I usually ask myself the following question - "If the entire dev team for foobar disappeared tomorrow, how much would it hurt" if the answer is "not much because I can maintain it myself while migrating away" then thats very different to "a lot because I don't understand it that well and/or it's massively complex".
Enterprise is a different world (even for a medium sized one like I work for) because (frankly) a lot of the programmers are doing it purely for the money and/or simply aren't that competent.
Not all of them by any means but I run into a lot of code that is just plain horrible, not "not the way I'd have done it" so much as "how does this thing even work?" and "Who possibly thought this was the right approach?".
I'm weird, I like LoB software development - I find that done well the feedback loop is very gratifying, you get to have an immediate affect on the companies bottom line and you have happier employees also the problems have a lot of hidden complexity which is intellectually stimulating.
I think that much of the reason for the existence of USDS is that government agencies like the VA should not be stuck with 10-year-old [1] technology. The idea is that U.S. citizens should be able to expect the same level of technology from their government that they get from Facebook, Twitter, Apple, or Google. They're explicitly trying to change the culture where you do things once for hundreds of millions of dollars and then it's never revisited because the first time was such a clusterfuck.
[1] Actually more like 50 year old technology, if my friends at the USDS are to be believed. In some cases, they're replacing systems where SOP is to manually type in data, print it out, fax it over to another department, and then type it in again to a different system.
[1] Actually more like 50 year old technology, if my friends at the USDS are to be believed. In some cases, they're replacing systems where SOP is to manually type in data, print it out, fax it over to another department, and then type it in again to a different system.
Those kinds of inefficiencies are /everywhere/ in government. They tend to mirror the limited ways departments communicate (see: Conway's Law). Having structured data available isn't a given, nor is the ability to send it across networks that often predate the internet.
I remember an incident where a critical feature went down for days because a backhoe severed the only available link between two agencies. It's hard to overstate how unusual it is by government standards to operate the way modern startups do-- e.g., put everything in the cloud and let Amazon handle your availability problems.
I remember an incident where a critical feature went down for days because a backhoe severed the only available link between two agencies. It's hard to overstate how unusual it is by government standards to operate the way modern startups do-- e.g., put everything in the cloud and let Amazon handle your availability problems.
There's nothing that would prohibit maintaining a Backbone or Knockout app (or just one written with a bunch of non-spaghetti-code jQuery) today, and it's hard to say that any other choice for writing a piece of software with a GUI would have fared better. Why do you think that using React will have a worse result than that?
I think that of the kinds of tools people are using to make web applications in 2017, React and Rails are probably in the more conservative, most likely to be maintainable in 10 years category. (I wouldn't believe this about Rails except it's been so popular for the past 10 years.)
I think that of the kinds of tools people are using to make web applications in 2017, React and Rails are probably in the more conservative, most likely to be maintainable in 10 years category. (I wouldn't believe this about Rails except it's been so popular for the past 10 years.)
> There's nothing that would prohibit maintaining a Backbone or Knockout app (or just one written with a bunch of non-spaghetti-code jQuery) today
Well that kind of depends, I still have stuff in production with knockout and for some stuff I still use it but it's not just a matter of the framework/library it's the ancillary tooling.
We went through grunt, gulp, browserify and webpack fairly quickly, we could have stayed with any of them but no-one else did, if you stand still you end up been left miles behind when you eventually do want to move on.
The dozens of different bits approach has a lot of advantages but it has some severe downsides as well.
Modern JS is a bit of a red queen problem, you have to constantly adapt your codebases just to stay even on a 5-10 basis.
It's a problem in my world (enterprise LoB stuff in the browser), you want to be somewhat conservative while still be able to have some assurance of long term viability and some of the nice toys.
My approach has been to trade off dependencies as much as possible, I moved to TypeScript for a lot of stuff since it emits good JS so if it ever did go away I could just output the most recent JS and base from that (and I doubt TypeScript is going away in the next few years, MS has invested heavily in it as a platform for internal stuff).
Meanwhile over on the other side of things I recently ran something that was written in 'pure' Java 1.2 on the modern VM with not as many issues as you'd think (that platform is 20 years old).
I recently inherited a large enterprise system that is classic jQuery/no framework, it 'works' on a modern browser but it's an unholy unstructured mess and getting any kind of iteration velocity on it is a complete pain and that was written in over a few years finishing two years ago.
The developers just hadn't adapted to the modern landscape well at all and their momentum was terrible and getting worse.
Well that kind of depends, I still have stuff in production with knockout and for some stuff I still use it but it's not just a matter of the framework/library it's the ancillary tooling.
We went through grunt, gulp, browserify and webpack fairly quickly, we could have stayed with any of them but no-one else did, if you stand still you end up been left miles behind when you eventually do want to move on.
The dozens of different bits approach has a lot of advantages but it has some severe downsides as well.
Modern JS is a bit of a red queen problem, you have to constantly adapt your codebases just to stay even on a 5-10 basis.
It's a problem in my world (enterprise LoB stuff in the browser), you want to be somewhat conservative while still be able to have some assurance of long term viability and some of the nice toys.
My approach has been to trade off dependencies as much as possible, I moved to TypeScript for a lot of stuff since it emits good JS so if it ever did go away I could just output the most recent JS and base from that (and I doubt TypeScript is going away in the next few years, MS has invested heavily in it as a platform for internal stuff).
Meanwhile over on the other side of things I recently ran something that was written in 'pure' Java 1.2 on the modern VM with not as many issues as you'd think (that platform is 20 years old).
I recently inherited a large enterprise system that is classic jQuery/no framework, it 'works' on a modern browser but it's an unholy unstructured mess and getting any kind of iteration velocity on it is a complete pain and that was written in over a few years finishing two years ago.
The developers just hadn't adapted to the modern landscape well at all and their momentum was terrible and getting worse.
I got the impression they are looking for Agile evangelists mixed with... something. I really don't know.
They're probably looking for much kids/fewer years experience (they keep bloviating about being startup like after all). Based on GS pay they cap out at a salary that is not very high for an industry veteran (they talk about steps being skipped only in "exceptional" circumstances). I guess they assume that if you've got all those years and are still applying you must not be very good. A GS15 is only 128k in the costly DC area. I work for a nonprofit in a much much lower cost of living area and make about that and I'm nowhere near the top of any payscale!
Oh and no relo assistance either lol. To work in the capital of corruption. Sorry but I would have to be braindead to surround myself by that environment knowing I was just a blue collar lackey. And if equivocating "public" service and being Trump's bitch works for you, I am happy for you.
Oh and no relo assistance either lol. To work in the capital of corruption. Sorry but I would have to be braindead to surround myself by that environment knowing I was just a blue collar lackey. And if equivocating "public" service and being Trump's bitch works for you, I am happy for you.
I am turning 34 coming this September. Not much of a kid anymore (even though I act like one).
Not true. GS15 caps at 161k at DC area. It is a very respectable salary. http://www.fedweek.com/pay-tables/2017-gs-pay-table-washingt...
For engineering, USDS predominately hires senior engineers with years of experience. The reason is because we help troubleshoot some of the biggest crises in the government.
Not true. GS15 caps at 161k at DC area. It is a very respectable salary. http://www.fedweek.com/pay-tables/2017-gs-pay-table-washingt...
For engineering, USDS predominately hires senior engineers with years of experience. The reason is because we help troubleshoot some of the biggest crises in the government.
Sure, but isn't 161k the max, and there's no room to grow beyond that? For many people who work in the SF bay area (for example) and have 10+ years of experience, even 161k may represent a pay cut. DC is certainly cheaper cost-of-living-wise, but not by a lot. And if 161k is the max, where do you go from there, especially if you won't have a job after a few years due to how their "tour of duty" thing? Spend even more of your own money to move back to the bay area, and try to reestablish your old salary from before you left?
If your counting the money it will never be worth it. It makes no financial sense to go work for the USDS, you will get paid less, you will work in a much worse environment, with more frustration then you can imagine. However you will directly impact the lives of millions of Americans. People who would have died, might live. Educations can be obtained for those who might never have had one. Doctors will get paid more efficiently and will take on more medicare and medicaid patients.
You might not have as much disposable income, but you can live a good life in a nice area for 161k. Thats more them most people in the DC area make.
You might not have as much disposable income, but you can live a good life in a nice area for 161k. Thats more them most people in the DC area make.
I am glad to see it's still around. I was worried that it would be cut with the change in administrations since it reports into the White House. The USDS is a shining example of civil service and the best of government.
Has work there changed with the new administration? Wondering how much USDS tech goals change with different administrations.
Hey there, I'm the USDS Administrator. Most of our projects are the same, like making sure that veterans can get their health benefits. We've found partners in the White House who really want to make government work better for the American people. An example is Chris Liddell, who was CFO for Microsoft. Every administration is going to be different and have different interests, but we've been able to find common ground on projects, and those projects are helping people who need it.
Our mission has remained very consistent: use design and technology best practices to improve government services. The new administration has different policy priorities, but it has been remarkable to see the bipartisan support for our mission. Our work has remained largely the same. The major difference is fewer technologists are raising their hands for public service now, which constrains our ability to improve things.
> The major difference is fewer technologists are raising their hands for public service now
I suspect you are making a false, or at least unwarranted, generalization from “federal executive branch” to “public” here.
I suspect you are making a false, or at least unwarranted, generalization from “federal executive branch” to “public” here.
Good point! Lots of ways to serve.
Its amazing to see how "solving" the problems can often not solve the problem. Immediately when faced with a error that happened after five minutes I might just put a sleep(301) in the startup script, but that totally would have masked the issue for others. Also amazing foresight by the kernel team to think ahead and make this wrap explicit.
Author here. Completely agreed. My jaws dropped when I saw the INITIAL_JIFFIES. The kernel developers really saved our butt.
I could not imagine debugging this problem if INITIAL_JIFFIES was randomized. It may takes days/weeks/months for this bug to appear.
I could not imagine debugging this problem if INITIAL_JIFFIES was randomized. It may takes days/weeks/months for this bug to appear.
Similarly, Unreal Engine 4 offsets platform time (a double) by some large value so if it's stored in a float, accuracy errors will be exposed almost immediately. Looking it up, the offset starts out large enough that the epsilon is two seconds.
Do you have a link with more info? I'd love to read more about this.
Sorry, no. It's not something documented other than a cryptic comment in the source code ( FPlatformTime::Seconds() ) assuming some knowledge of floating point number gotchas.
Edit: Here's a more detailed post o made about the specific gotcha if you're interested: https://community.gamedev.tv/t/why-is-fplatformtime-seconds-...
Edit: Here's a more detailed post o made about the specific gotcha if you're interested: https://community.gamedev.tv/t/why-is-fplatformtime-seconds-...
Here's the patch that applies this: https://www.kernel.org/pub/linux/kernel/people/akpm/patches/...
Nice! I was looking for it, and it predates git. Thank you!!
I had the pleasure of meeting and working with many amazing USDS engineers. Lots of talents, many are truly dedicated to the higher purpose and truly believe in the mission of serving our country. It's a shame that because of the current administration, people are less and less interested in the government.
The government's current IT infrastructure crisis is not caused by any one administrations. The root cause goes back decades. Things like "Improving Veterans' lives so they don't have to wait 5-10 years for an Appeals decision" shouldn't be political.
I can honestly say that the projects I've been involved with in USDS are the most impactful and meaningful projects I've worked on in my entire life.
I can honestly say that the projects I've been involved with in USDS are the most impactful and meaningful projects I've worked on in my entire life.
I just wanted to say that I love these type of postmortem stories. Thanks for sharing!
I love bugs like these. Make it crash is often the hardest part of solving any bug and without this you'd have never known. There is one nasty bit to this story though: the NSOC was running outdated firmware on their Cisco's and wouldn't have known about it if an outside party had not alerted them to this fact. That's pretty sloppy on their end.
1) Connections hanging over a frame relay network that one day started dropping packets over a certain size. Work-around was adjusting the MTU until I was able to convince the frame relay network operator that something was broken in their network. Initially it was confusing because an interactive telnet session over the network would work fine till you did something like "ls -l" or tried to read a man page which generated enough text to send a full size packet, then the connection would hang.
2) Unable to reach a Verizon e-mail paging gateway but only when connecting from a Linux box. An OS X box on the same network as the Linux box could reach the gateway fine. Turned out Verizon had a firewall rejecting connections where the ECN bit was set. Linux was setting ECN, OS X was not.
3) Solaris box A could initiate a connection to box B, but not the other way. After A talked to B, B could then talk to A, but only for a short period. Someone had deleted A's own MAC from A's ARP table, so A wasn't replying to ARP requests for itself. But if A connected to B, B would keep A's MAC in its own table till it timed out after which B couldn't initiate connection to A any more.
4) All manner of misconfigurations over the years where you learn to recognize the symptoms: misconfigured netmask size; misconfigured duplex; duplicate IP address on same network. You rarely see these any more.
5) The infamous 500-mile e-mail. :-)
6) And my favorite - https://www.pagerduty.com/blog/the-discovery-of-apache-zooke...