HackerTrans
TopNewTrendsCommentsPastAskShowJobs

MilkMp

no profile record

Submissions

Show HN: CIA World Factbook Archive (1990–2025), searchable and exportable

cia-factbook-archive.fly.dev
496 points·by MilkMp·5개월 전·99 comments

comments

MilkMp
·5개월 전·discuss
Hi there, yes I used AI to help build this website. I personally don’t have the time nor the talent to build something like this from scratch! I do have knowledge in how historical and crime data are suppose to parsed,viewed, analyzed, and presented to the world :) If someone would like to take this work and improve it, please do!
MilkMp
·5개월 전·discuss
Thanks for pointing it out! I had noticed some mapping issues (Russia's Military GDP not showing), so there is definitely room for improvement here. Just wanted to get this out there for people and create their own projects or use this one :)
MilkMp
·5개월 전·discuss
[dead]
MilkMp
·5개월 전·discuss
Thanks! I am definitely not a front-end web designer lol, and I for sure don't want to limit people's access. I will look into the standards and see how best to implement them into the website :)
MilkMp
·5개월 전·discuss
Was originally just supposed to be a data archive/download place for the parsed data.Thought a website could help! Will look into the standards
MilkMp
·5개월 전·discuss
Thanks! Will look into it
MilkMp
·5개월 전·discuss
Just the start, but I added the maps to the page. Planning on linking them to their factsheets and dashboards this week.

https://cia-factbook-archive.fly.dev/maps?page=1
MilkMp
·5개월 전·discuss
Hey there, yeah, definitely. I maintain .txt change logs for all data modifications. To be clear, no information is added or altered — the Factbook content is exactly what the CIA published. The parsing process structures the raw text into fields (removing formatting artifacts, sectioning headers, and deduplicating noise lines), but the actual data values are untouched. What I've added on top are lookup tables that map the CIA's FIPS 10-4 codes to ISO Alpha-2/3 and a unified MasterCountryID, so the different code systems can be joined and queried together.

I will add them to the github :)
MilkMp
·5개월 전·discuss
Will check out! Thank you!
MilkMp
·5개월 전·discuss
Hi there, thanks for linking this! My GitHub and website both link to and use this source! I just thought putting it in a SQL database and making the entire 1990-2025 queryable was needed since I couldn't find one anywhere :)
MilkMp
·5개월 전·discuss
Hi there! If you have anything you want added to the site, just let me know :) I can definitely try.
MilkMp
·5개월 전·discuss
Hi there, I have updated the webpage to include all countries/all years. It will give you a warning that the PDF is large. https://cia-factbook-archive.fly.dev/export
MilkMp
·5개월 전·discuss
Ohh that is a great idea! And since we already have the political field in SQL!. I will start working on some of this and update the website this week. Thank you for the awesome suggestions!
MilkMp
·5개월 전·discuss
Thanks so much!
MilkMp
·5개월 전·discuss
Found the problem, the total regex doesn't handle magnitude suffixes:

2018: total: 17,856,024 → parses as 17856024 (correct raw count) 2020: total: 18.17 million → parses as 18.17 (WRONG - drops "million") 2025: total: 39.3 million → parses as 39.3 (WRONG) So the chart jumps from ~18 million down to ~18, making it wrong. The fix is to handle "million/billion/trillion" after total.

Just deployed a new bug fix.

Thanks for bringing this to my attention!
MilkMp
·5개월 전·discuss
Found the root cause. The "World" entity (population ~8 billion) was being called alongside all individual countries, doubling the total. Thank you again!
MilkMp
·5개월 전·discuss
Will fix right now! I think I was looking at this for too long and missed some things. Thank you :)
MilkMp
·5개월 전·discuss
Hi there, I have located the root and sent out a bug fix.

Root cause: The CIA World Factbook, published by the Central Intelligence Agency, uses the U.S. Government's FIPS 10-4 country codes, which differ from the ISO 3166-1 Alpha-2 codes used by the rest of the world. Of the 281 entities in our database, 173 have different FIPS and ISO codes. Our lookups matched FIPS codes first, so when codes collided between the two systems, the wrong country was loaded. Fixed all 13 queries and 6 templates to always prefer ISO over FIPS.

Examples fixed:

Australia (ISO=AU) was loading American Samoa (FIPS=AQ, but Australia's FIPS=AS collides with American Samoa's ISO=AS) Singapore (ISO=SG) was loading Senegal (FIPS=SG) Germany (ISO=DE) was loading Gambia (FIPS=GM = Germany's FIPS, ISO=GM = Gambia) Bahamas (ISO=BS) was loading Burkina Faso (FIPS=BF = Bahamas' FIPS, ISO=BF = Burkina Faso)
MilkMp
·5개월 전·discuss
Hi there, I have located the root and sent out a bug fix.

Root cause: The CIA World Factbook, published by the Central Intelligence Agency, uses the U.S. Government's FIPS 10-4 country codes, which differ from the ISO 3166-1 Alpha-2 codes used by the rest of the world. Of the 281 entities in our database, 173 have different FIPS and ISO codes. Our lookups matched FIPS codes first, so when codes collided between the two systems, the wrong country was loaded. Fixed all 13 queries and 6 templates to always prefer ISO over FIPS.

Examples fixed:

Australia (ISO=AU) was loading American Samoa (FIPS=AQ, but Australia's FIPS=AS collides with American Samoa's ISO=AS) Singapore (ISO=SG) was loading Senegal (FIPS=SG) Germany (ISO=DE) was loading Gambia (FIPS=GM = Germany's FIPS, ISO=GM = Gambia) Bahamas (ISO=BS) was loading Burkina Faso (FIPS=BF = Bahamas' FIPS, ISO=BF = Burkina Faso)
MilkMp
·5개월 전·discuss
Hi there, I have located the root cause and will be fixing the issue:

Root cause: CIA uses FIPS codes (CanonicalCode), which differ from ISO Alpha-2 for many countries. Templates and SQL queries prioritized CanonicalCode over ISOAlpha2, so URL codes like /archive/2025/AU matched the wrong country.

Australia (AU) -> American Samoa (AS = CIA FIPS for Australia) Singapore (SG) -> Senegal (SG = CIA FIPS for Senegal) Germany (DE) -> Gambia (GM = CIA FIPS for Germany)