The DoD is prioritizing open source software(ritwikgupta.me)
ritwikgupta.me
The DoD is prioritizing open source software
https://ritwikgupta.me/dod-opportunities-for-open-source-software/
49 comments
If you take money out of the equation, how will you convince the generals to select/help enhance your project when you cannot offer them a half million dollar a year VP job at your software project when they retire? I'm not joking or being cynical, this is how DoD acquisition works: the revolving door from General to VP to civilian DoD manager to Senior VP for a defense contractor to whatever political appointment can be wrangled. I would LOVE to see OSS succeed at the five-sided puzzle palace but unless it's a civilian DoD contractor pushing it (and charging a boatload in the process) I'm just skeptical this will change much.
In my opinion, the waves of change towards OSS are already being stirred by DoD military, civilian, and contractor personnel alike. I think this also aligns with industry adopting an open-source-first, provide services/hosting model as well.
Anecdata, but most DoD program managers (CIV, CTR, or MIL) I know who work on software programs are generally driven by a need to solve the problems that they face. They gladly embrace open source solutions when they're made available easily.
Anecdata, but most DoD program managers (CIV, CTR, or MIL) I know who work on software programs are generally driven by a need to solve the problems that they face. They gladly embrace open source solutions when they're made available easily.
Q. How does the Open Source community benefit from this?
By enabling a contractual path to supporting FOSS projects/foundations with real funding, the DoD becomes a valid (authorized) player to support the community. Additionally, the memo enables (and encourages) DoD personnel to contribute to open source software as part of their job's responsibilities. This brings thousands of developers into the OSS community who otherwise may not have engaged/may not have been allowed to engage.
> (1) Government employees may contribute to existing OSS projects (including being the primary maintainer) as part of their official duties, so long as they consult with their supervisor first to ensure a common-sense approach for contributions that preserve Operations Security (OPSEC) and further the Government's interests.
> (1) Government employees may contribute to existing OSS projects (including being the primary maintainer) as part of their official duties, so long as they consult with their supervisor first to ensure a common-sense approach for contributions that preserve Operations Security (OPSEC) and further the Government's interests.
Worst case? They don't. They adopt OSS where possible to cut costs, fluff resumes, reduce effort and it ends up in a private repo somewhere only a few people will ever see. Best case? They create a public fork that fixes bugs and makes it suitable for their use case. Source: anecdata from being in the kitchen while the sausage was made.
The same way Open Source benefits from the NSA's current and former contributions.
A: Like with most of these programs, a few large organizations like google and microsoft stand to make a lot more money, but the people at the heart of the problem (the actual maintainers) stand to lose.
AT the bare minimum, even doing it yourself, it costs more money to create a company and apply for 501C3 status than the paltry sums of money most projects receive in donations.
AT the bare minimum, even doing it yourself, it costs more money to create a company and apply for 501C3 status than the paltry sums of money most projects receive in donations.
I completely agree that there's some overhead associated with my proposed route, and really only makes sense if your project is large enough such that this overhead is already being incurred. I believe there is still a large gap between projects supported by large tech companies and companies that fit this bill, and in that gap are projects that can benefit from this change.
There are also various c3 and c6 (and similar statuses outside the US) fiscal sponsors that may work for some projects that don't want to set up a whole organization.
By having yet another discussion about OSI's version of the term "Open Source", and in particular about whether restriction in military use is covered by the "Fourth Freedom" (or wherever they postulate absence of purpose restrictions).
NB there's no such thing as an "Open Source community", just individuals and orgs with different goals: some want to show off, some want additional maintainers if at all possible, some want to give away their complement, some want to establish an alternative to a competitor's product, some want to promote a language or format, some want xGPL for political reasons, yet others don't care at all
NB there's no such thing as an "Open Source community", just individuals and orgs with different goals: some want to show off, some want additional maintainers if at all possible, some want to give away their complement, some want to establish an alternative to a competitor's product, some want to promote a language or format, some want xGPL for political reasons, yet others don't care at all
wrong answers only: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95644#c4
Acquisition is not the only way DoD operates.
I work at a Navy research center that's 3000+ employee strong and feeds all of DoD (we're a working capital fund, meaning we take customers to fund ourselves). In four years here, I've worked projects sponsored by Air Force, DHS/CBP, Navy, Marines, and two three letter agencies. There was no general involved in any of them, and I don't know any projects by coworkers that involved acquisition of technology. As for my projects, all except one still still in R&D have transitioned into production systems. All involve OSS to some degree. One was a design for a COTS multi source information fusion system meant to be deployed in the cloud and has all the open source tech there you'd expect, NiFi, Kafka, Spark, etc. My current project is a combination of us (correlation) and two contractors (separately IT and predictive models). Again, it's all the OSS and tech you'd expect to find in a real time analytics/tipping system and CI/CD pipeline/orchestration.
I bring up my experience not to say you're wrong but to show there are other avenues it comes in. There are lots of research labs, and researchers aren't about buying software from contractors.
I work at a Navy research center that's 3000+ employee strong and feeds all of DoD (we're a working capital fund, meaning we take customers to fund ourselves). In four years here, I've worked projects sponsored by Air Force, DHS/CBP, Navy, Marines, and two three letter agencies. There was no general involved in any of them, and I don't know any projects by coworkers that involved acquisition of technology. As for my projects, all except one still still in R&D have transitioned into production systems. All involve OSS to some degree. One was a design for a COTS multi source information fusion system meant to be deployed in the cloud and has all the open source tech there you'd expect, NiFi, Kafka, Spark, etc. My current project is a combination of us (correlation) and two contractors (separately IT and predictive models). Again, it's all the OSS and tech you'd expect to find in a real time analytics/tipping system and CI/CD pipeline/orchestration.
I bring up my experience not to say you're wrong but to show there are other avenues it comes in. There are lots of research labs, and researchers aren't about buying software from contractors.
>There are lots of research labs, and researchers aren't about buying software from contractors.
In these DoD research environments (I've worked in them before), researchers typically aren't about building software, either. It's expensive, even if you hire the teams. Not always, but unless your research is technology driven, it tends to be about cutting corners on software and delivering research quality MVPs (semi-working prototypes) while all focus is on the research. You can forget overhead of best practice infrastructure for development/engineering, that's wasted overhead in some of those group's eyes. The more theoretical it is, the easier it is to get away with, the more applied, the less easy it is to handwave away reality.
If you're at a heavily funded technology focused org like NRL (which the description sounds a lot like--neat stuff comes out of NRL), then this can be the case. There are, however, a lot of DoD funded research labs that are dumpster fires in terms of the software--NRL is almost the golden child compared to most of them. Again, in some such labs, a project may only have sufficient budget for 0.05 to 0.1 FTEs worth of software engineers. In such cases, 0.4 FTEs of software engineers can be considered a significant investment for a project, which is beyond laughable.
In these DoD research environments (I've worked in them before), researchers typically aren't about building software, either. It's expensive, even if you hire the teams. Not always, but unless your research is technology driven, it tends to be about cutting corners on software and delivering research quality MVPs (semi-working prototypes) while all focus is on the research. You can forget overhead of best practice infrastructure for development/engineering, that's wasted overhead in some of those group's eyes. The more theoretical it is, the easier it is to get away with, the more applied, the less easy it is to handwave away reality.
If you're at a heavily funded technology focused org like NRL (which the description sounds a lot like--neat stuff comes out of NRL), then this can be the case. There are, however, a lot of DoD funded research labs that are dumpster fires in terms of the software--NRL is almost the golden child compared to most of them. Again, in some such labs, a project may only have sufficient budget for 0.05 to 0.1 FTEs worth of software engineers. In such cases, 0.4 FTEs of software engineers can be considered a significant investment for a project, which is beyond laughable.
Out of all the comments in this thread, this one has been the most like my experience.
If dod research labs are like NNSA research labs part of why 0.4 FTEs can be considered significant investment is because overheads are quite high. My experience is that for every dollar a FTE makes their project is paying an additional 2-3 dollars in overhead. That means your $130K/yr FTE is using 50% of your $1M/yr budget.
I'm not sure what the typical cost for an FTE are in industry, but I believe they are lower than 3-4x base salary.
I'm not sure what the typical cost for an FTE are in industry, but I believe they are lower than 3-4x base salary.
I think the best future model for DoD software is to bring much more of the systems integration and product development in-house, mostly using open source, which will produce a higher quality and less expensive result for the DoD. Critical differentiated software technologies will be outsourced to boutique software companies that can offer software capabilities that are qualitatively superior to anything available in open source but can't be developed on DoD software engineering wages and/or budgets.
There are often large gaps between the mission requirements and the technical capabilities of open sources software, and research labs lack the expertise and budget to address those gaps in practice. Of course, the usual system integrators can't build this software either but they will charge the taxpayer a lot of money to fail at it.
There are often large gaps between the mission requirements and the technical capabilities of open sources software, and research labs lack the expertise and budget to address those gaps in practice. Of course, the usual system integrators can't build this software either but they will charge the taxpayer a lot of money to fail at it.
> I'm not joking or being cynical, this is how DoD acquisition works
What is that based on? Have you worked at DoD? Is there research supporting it?
DoD acquisition depends much more on civilians, not uniformed officers, and then they must be part of the President's budget and, ultimately, Congress. Officers play a minor role.
What is that based on? Have you worked at DoD? Is there research supporting it?
DoD acquisition depends much more on civilians, not uniformed officers, and then they must be part of the President's budget and, ultimately, Congress. Officers play a minor role.
Top officers select which civilians get control.
Who did and when? Can you provide any evidence. Also, the oficers often have bad relationships with the President and the National Security Advisor and staff, not to mention Congressional committees - the officers certainly wouldn't select them if they could.
How does this matter? You still have to pay these DoD contractors to implement the OSS stuff. It won't implement itself.
License fees are a rounding error compared to the cost of personnel.
License fees are a rounding error compared to the cost of personnel.
Open source software needs support. They could use the old Cygnus model and have people who are paid well to fix everything the DoD needs fixed. The general in charge of the contract could move there on retirement.
Great example of this is Michael D. Griffin. He started the Space Development Agency for commercial launchers and then retired to join the Rocket Lab he enabled.
the cynical side of me says "that's okay, the generals and such can still get cushy jobs at companies that manufacture hardware used by the DoD"
I'm sure RedHat, Amazon or Microsoft could, and would.
But they are prioritizing it! With what? Encouragement and Prioritization. It will be a huge succees /s
Real challenges to FOSS in the military (or other large organizations) would seem to be long-term stability of features, and implementation of enhancements and fixes:
A commercial vendor works to please large customers, such as the military. If the big customer needs a feature supported long term (remember that military systems commonly last decades), the vendor makes it happen. If the big customer needs a specific enhancement or fix, that is a priority for the vendor, and if it's needed ASAP, that can happen too.
FOSS communities are a bit more independent-minded (which I generally like) and some developers won't want to help the military for ideological reasons. Communication with FOSS communities is much different than communication with commercial vendors, and much less efficient.
I think DoD needs access to paid, professional developers who can provide those services for FOSS. That could be someone like Red Hat, it could be outside consultants, it could be internal developers, they could hire members of the FOSS community. Also, DoD will need to fork projects when the community is uninterested in a DoD priority.
And there's another major problem: Imagine bombs are falling, people are dying, national security priorities are being lost - and the military needs a fix or enhancement now. Submit a bug to bugzilla and wait? Email the dev list? IRC? They need someone to call, who can answer the phone and get things done.
A commercial vendor works to please large customers, such as the military. If the big customer needs a feature supported long term (remember that military systems commonly last decades), the vendor makes it happen. If the big customer needs a specific enhancement or fix, that is a priority for the vendor, and if it's needed ASAP, that can happen too.
FOSS communities are a bit more independent-minded (which I generally like) and some developers won't want to help the military for ideological reasons. Communication with FOSS communities is much different than communication with commercial vendors, and much less efficient.
I think DoD needs access to paid, professional developers who can provide those services for FOSS. That could be someone like Red Hat, it could be outside consultants, it could be internal developers, they could hire members of the FOSS community. Also, DoD will need to fork projects when the community is uninterested in a DoD priority.
And there's another major problem: Imagine bombs are falling, people are dying, national security priorities are being lost - and the military needs a fix or enhancement now. Submit a bug to bugzilla and wait? Email the dev list? IRC? They need someone to call, who can answer the phone and get things done.
I can't imagine that there's a FOSS project which directly provides mission-critical, "press this button to stop the war" functionality.
I think there are still one or two steps/precedents needed from this memo to a real pipeline of FOSS projects getting DoD funding, but these projects would be foundational things like video transports protocols, container orchestration tools, etc. (to name a few).
In addition, if there is a mission-critical fix which needs to be made to a project, this memo authorizes + encourages DoD personnel to contribute to FOSS projects!
> Employee participation in OSS projects used by DoD is often in the Government's interest, and is typically a legitimate use of government resources when the Government uses the software in question, either directly, as a component of a larger government system, or as a component of underlying government infrastructure.
> (1) Government employees may contribute to existing OSS projects (including being the primary maintainer) as part of their official duties, so long as they consult with their supervisor first to ensure a common-sense approach for contributions that preserve Operations Security (OPSEC) and further the Government's interests.
I think there are still one or two steps/precedents needed from this memo to a real pipeline of FOSS projects getting DoD funding, but these projects would be foundational things like video transports protocols, container orchestration tools, etc. (to name a few).
In addition, if there is a mission-critical fix which needs to be made to a project, this memo authorizes + encourages DoD personnel to contribute to FOSS projects!
> Employee participation in OSS projects used by DoD is often in the Government's interest, and is typically a legitimate use of government resources when the Government uses the software in question, either directly, as a component of a larger government system, or as a component of underlying government infrastructure.
> (1) Government employees may contribute to existing OSS projects (including being the primary maintainer) as part of their official duties, so long as they consult with their supervisor first to ensure a common-sense approach for contributions that preserve Operations Security (OPSEC) and further the Government's interests.
> if there is a mission-critical fix which needs to be made to a project, this memo authorizes + encourages DoD personnel to contribute to FOSS projects!
Yes, but that is rarely a means to get anything done urgently.
Yes, but that is rarely a means to get anything done urgently.
> I can't imagine that there's a FOSS project which directly provides mission-critical, "press this button to stop the war" functionality.
Of course. You have to do
sudo ./war.sh -f --command STOP
Of course. You have to do
sudo ./war.sh -f --command STOP
Here you go: https://sites.gatech.edu/falconview/
> I think DoD needs access to paid, professional developers who can provide those services for FOSS. That could be someone like Red Hat, it could be outside consultants, it could be internal developers, they could hire members of the FOSS community. Also, DoD will need to fork projects when the community is uninterested in a DoD priority.
That much is a given, and it was already true today. Even if DoD would do nothing more than contract for outsiders to build its software, the people writing the contracts need to have access to government staff who understand software (i.e. actual developers).
> And there's another major problem: Imagine bombs are falling, people are dying, national security priorities are being lost - and the military needs a fix or enhancement now. Submit a bug to bugzilla and wait? Email the dev list? IRC? They need someone to call, who can answer the phone and get things done.
I regret to inform you that the military is already used to this taking months and years. The Navy was unable to field a COVID tracking software tool in time for it to matter when the aircraft carrier USS Theodore Roosevelt got sidelined by COVID, the needed tool ended up being fielded by Defense Digital Service who operate under different authorities.
But even if we pretend the wider military could do need this, that's the internal developers you already mentioned (like DDS), or developers already on contract to provide urgent support. But that's separate from FOSS vs. proprietary. If anything, the farther we get into an "everything-as-a-Service" model, the less likely it will be that a company whose product is not open source would be able to meet those kind of urgent timeframes, because making changes willy-nilly to their -aaS product may risk their FedRAMP certification and DISA P-ATO.
That much is a given, and it was already true today. Even if DoD would do nothing more than contract for outsiders to build its software, the people writing the contracts need to have access to government staff who understand software (i.e. actual developers).
> And there's another major problem: Imagine bombs are falling, people are dying, national security priorities are being lost - and the military needs a fix or enhancement now. Submit a bug to bugzilla and wait? Email the dev list? IRC? They need someone to call, who can answer the phone and get things done.
I regret to inform you that the military is already used to this taking months and years. The Navy was unable to field a COVID tracking software tool in time for it to matter when the aircraft carrier USS Theodore Roosevelt got sidelined by COVID, the needed tool ended up being fielded by Defense Digital Service who operate under different authorities.
But even if we pretend the wider military could do need this, that's the internal developers you already mentioned (like DDS), or developers already on contract to provide urgent support. But that's separate from FOSS vs. proprietary. If anything, the farther we get into an "everything-as-a-Service" model, the less likely it will be that a company whose product is not open source would be able to meet those kind of urgent timeframes, because making changes willy-nilly to their -aaS product may risk their FedRAMP certification and DISA P-ATO.
In the section on getting money from the DOD, this is the first step:
> If you have a large project that is getting significant usage, incorporate a 501(c)(3) organization which will officially manage the project. Funding for the effort will live in this 501(c)(3)
Not too long ago, the IRS was clamping down pretty hard on the creation of new 501(c)(3)s, especially ones that were functioning (as they saw it) as the equivalent of businesses:
https://www.techdirt.com/articles/20140701/11470827745/irs-r...
This issue doesn't get much coverage today, but as far as I'm aware, this is still the status quo, with 501(c)(3) applications being subjected to additional scrutiny and denied regularly.
> If you have a large project that is getting significant usage, incorporate a 501(c)(3) organization which will officially manage the project. Funding for the effort will live in this 501(c)(3)
Not too long ago, the IRS was clamping down pretty hard on the creation of new 501(c)(3)s, especially ones that were functioning (as they saw it) as the equivalent of businesses:
https://www.techdirt.com/articles/20140701/11470827745/irs-r...
This issue doesn't get much coverage today, but as far as I'm aware, this is still the status quo, with 501(c)(3) applications being subjected to additional scrutiny and denied regularly.
There are a number of fiscal sponsors for open source out there that you can use to sidestep the need to register a 501(c)(3), for example Conservancy or SPI:
https://sfconservancy.org/ https://www.spi-inc.org/
https://sfconservancy.org/ https://www.spi-inc.org/
Sure. That's been the standard workaround for over a decade, I think. My point was that the advice being given is misleading and setting up a lot of folks for a Sisyphean task.
And it is also probably setting up obstacles for getting money from the DOD, if they aren't expecting to bill a fiscal sponsor as an intermediary.
And it is also probably setting up obstacles for getting money from the DOD, if they aren't expecting to bill a fiscal sponsor as an intermediary.
The problem I'm seeing with open source projects lately, is that they are essentially ran by companies who claim to support open source but yet almost every decision they make counters that. They are happy to have people work on their projects for free, but then you end up having to assign your work to them and they turn around and change their licensing or contributions later on where you no longer even own the open source efforts you put into the software. To them open source is just another business opportunity, but when it comes to values they could care less.
There are some open source companies that aren't like this; RedHat, Collabora, Igalia, Codethink and all the companies contributing to the Linux kernel are examples.
I work for the US EPA, speaking in my personal capacity.
We have a Github, but in a lot of ways it feels kind of like an archive. Does anyone have feedback on how it can be made more useful? We have all the authority to be good OSS community members.
Maybe if we were to prioritize like general purpose libraries instead of just our super domain specific projects?
https://github.com/USEPA
We have a Github, but in a lot of ways it feels kind of like an archive. Does anyone have feedback on how it can be made more useful? We have all the authority to be good OSS community members.
Maybe if we were to prioritize like general purpose libraries instead of just our super domain specific projects?
https://github.com/USEPA
First, the basics:
Add a description to every repository.
Make sure every repository has an informative README. There are a ton of templates and generators out there, most are pretty good (if you find or create one that is especially useful for your agency, consider promoting it internally).
Now comes the 'next level' stuff.
If a project would be useful in an academic context, perhaps include a blurb on attributing a citation for it.
There are a lot of folks out there looking for projects that will accept their contributions so they can build up some cred. Populate projects' issues with stuff you can tag as a good-first-issue, and your projects will start showing up in various aggregators (be judicious about selecting projects to do this for, you don't want issues and PRs languishing without being incorporated for lack of maintainer interest or time, that just looks worse).
For projects that really are just being thrown over the transom, mark them clearly as such and invite friendly forks.
Try to imagine who might find a particular project useful. Write that up as an example and incorporate it into the README or other documentation (eg. I imagine that anything related to stormwater runoff may be useful to civic planning departments, or perhaps environmental advocacy groups). Blog/tweet about the use case (your agency's comms team may be helpful here).
I hope these suggestions help.
Add a description to every repository.
Make sure every repository has an informative README. There are a ton of templates and generators out there, most are pretty good (if you find or create one that is especially useful for your agency, consider promoting it internally).
Now comes the 'next level' stuff.
If a project would be useful in an academic context, perhaps include a blurb on attributing a citation for it.
There are a lot of folks out there looking for projects that will accept their contributions so they can build up some cred. Populate projects' issues with stuff you can tag as a good-first-issue, and your projects will start showing up in various aggregators (be judicious about selecting projects to do this for, you don't want issues and PRs languishing without being incorporated for lack of maintainer interest or time, that just looks worse).
For projects that really are just being thrown over the transom, mark them clearly as such and invite friendly forks.
Try to imagine who might find a particular project useful. Write that up as an example and incorporate it into the README or other documentation (eg. I imagine that anything related to stormwater runoff may be useful to civic planning departments, or perhaps environmental advocacy groups). Blog/tweet about the use case (your agency's comms team may be helpful here).
I hope these suggestions help.
I'm not sure what EPA can do to make it more useful. My personal example is the USEPA/USEEIO modeling framework [1]. My agency uses a pricey and rigid proprietary modeling framework. When I suggested USEEIO as an extensible cost-effective alternative I was met with resistance that it is free and mustn't be any good and definitely not defensible in court. Classic .gov thinking. I think DoD prioritizing open-source along with the work of USDS/GSA at digital.gov needs to diffuse to Senior Execs at other agencies to shift the paradigm. Unfortunately it feels like this is a decade out. Thank you for pushing OSS in .gov and to your peers at EPA for USEEIO.
[1] https://github.com/USEPA/USEEIO
[1] https://github.com/USEPA/USEEIO
Speaking from experience, convincing management is a long-haul process.
It is needs to be repeatedly brought (but gently).
Forward links to success stories, provide counter arguments to FUD (but again, gently!)
I say gently because it's easy to create enemies by making people look bad, and that's a no-no, you have to gradually wear down the old consensus and build a new one.
It is needs to be repeatedly brought (but gently).
Forward links to success stories, provide counter arguments to FUD (but again, gently!)
I say gently because it's easy to create enemies by making people look bad, and that's a no-no, you have to gradually wear down the old consensus and build a new one.
The usual stuff; Do everything in public in the community. Make your public GitHub the place you do internal deployments from, so that you ensure everything deployed internally is public. Contribute back to your dependencies' upstream projects with bug reports/triage, patches, test cases and reviews. Contribute back to the distros you use, with bug reports, packaging updates, patches, testing etc. As well as funding US EPA developers to do this work, fund the most important projects you use through donations or support contracts or whatever else is available.
Re general purpose libraries, definitely consider splitting projects up into separate libraries, and switch the domain specific projects to using those libraries.
To a first approximation, DoD (and DoE) approximately fund Red Hat. They're fine with OSS.
[deleted]
The model is very simple, if a business uses open source, the the business has to pay an open source tax. It doesn't include OSS covered by must share source, like GNU , license.
No, it definitely isn't.
Every time a major org claims to be open to OSS it rapidly reverts to old tools and ecosystems. The new CIO can claim all he wants, it's not going to happen. DoD is run by entrenched civil servants. I've seen it happen time and time again in MDA, Army, and Navy work.
Every time a major org claims to be open to OSS it rapidly reverts to old tools and ecosystems. The new CIO can claim all he wants, it's not going to happen. DoD is run by entrenched civil servants. I've seen it happen time and time again in MDA, Army, and Navy work.
Awesome!