How ACH works: A developer perspective (2014)(engineering.gusto.com)
engineering.gusto.com
How ACH works: A developer perspective (2014)
http://engineering.gusto.com/how-ach-works-a-developer-perspective-part-1/
225 comments
My first job in 2005 was to take over an ACH processing system that was written for our company. It was all in Perl. :-P
ACH format is super weird .. tons of lines with 99999 on them as separators. And yes, we had to go through a bank to do any ACH transactions. They shipped us a router that setup an IPSec tunnel and it had a backup ISDN connection on it as well.
I wish ACH was more accessible. In many other countries, you give people your Name, Bank and Account number (Australia: BSB + Acct, NZ: it's one big number) and people can just send you money. Appears the next day, even on holidays. In the US we still have to photo checks with a phone app? WTF? I wrote a thing about our banking system here:
http://penguindreams.org/blog/the-american-banking-system-is...
ACH format is super weird .. tons of lines with 99999 on them as separators. And yes, we had to go through a bank to do any ACH transactions. They shipped us a router that setup an IPSec tunnel and it had a backup ISDN connection on it as well.
I wish ACH was more accessible. In many other countries, you give people your Name, Bank and Account number (Australia: BSB + Acct, NZ: it's one big number) and people can just send you money. Appears the next day, even on holidays. In the US we still have to photo checks with a phone app? WTF? I wrote a thing about our banking system here:
http://penguindreams.org/blog/the-american-banking-system-is...
As an American expat living in a developing country it seems very weird to me how backwards the US ACH system is. Where I live now it's the same as the countries you mentioned, anyone can send money to your bank account using just the account number. And it appears in minutes. The banks even offer you an SMS alert when founds arrive and the sender can elect to send an SMS to alert the receiver to check their account for the received funds. Everyone uses it, including many (most?) businesses.
From what I understand, the desire for instantaneous transactions is has an inverse relation with how much money you have. Poorer countries are generally way, way ahead in mobile banking, because in a rich country you are supposed to keep around a month's salary in all your accounts, making transfers less urgent.
The way I understand it, it's a coordination issue - the bigger a market is, the longer it takes for a big change to be adopted by everyone.
Plausible stories are easy to make up. Lots of countries - probably most - with higher GDP per capita than the US have lower latency banking. The US has a large proportion of people who are worse off than the equivalents in many poorer countries, too.
Plausible stories are easy to make up. Lots of countries - probably most - with higher GDP per capita than the US have lower latency banking. The US has a large proportion of people who are worse off than the equivalents in many poorer countries, too.
Your perspective describes the underlying issue much clearly than those who talk about "leapfrogging".
The newer platforms (like mobile banking) were adopted in those (poorer) markets because they were cheaper to deploy relative to the technology that preceeded them, and perhaps more importantly, they had less inertia to overcome (against entrenched interests).
The newer platforms (like mobile banking) were adopted in those (poorer) markets because they were cheaper to deploy relative to the technology that preceeded them, and perhaps more importantly, they had less inertia to overcome (against entrenched interests).
That, and how much existing infrastructure there is.
Many places leapfrogged directly to mobile phones (featurephones mostly) because there were little to no power and/or landline wiring present (and attempts at getting such infrastructure in place got disrupted by people stealing the wiring and selling it as scrap copper).
Many places leapfrogged directly to mobile phones (featurephones mostly) because there were little to no power and/or landline wiring present (and attempts at getting such infrastructure in place got disrupted by people stealing the wiring and selling it as scrap copper).
Swish in Sweden?
Swish allows to send using an arbitrary number (phone or arbitrarily assigned) rather than an actual account number.
It's also credits the recipient immediately. That's possible since all the participating banks have deposited a number of billions in a mutual fund designed to cover the losses in the case one of the banks runs out of money, and also probably by having an internal account to debit during the day.
They are tranfering money between the banks at one (or maybe a few) standard clearing runs involving the Riksbank, that works more or less like ACH. I think all of them are scheduled in the morning.
All Swedish banks are working onreplacing the payment infrastructure (the clearing. ) It's supposed to be finished in 2024 or something like that.
It's also credits the recipient immediately. That's possible since all the participating banks have deposited a number of billions in a mutual fund designed to cover the losses in the case one of the banks runs out of money, and also probably by having an internal account to debit during the day.
They are tranfering money between the banks at one (or maybe a few) standard clearing runs involving the Riksbank, that works more or less like ACH. I think all of them are scheduled in the morning.
All Swedish banks are working onreplacing the payment infrastructure (the clearing. ) It's supposed to be finished in 2024 or something like that.
Sounds about right from my experience working at Klarna.
The major US banks created clearXchange to solve this problem, which was later sold and now rebranded to Zelle. It's near-instantaneous transfers for supported banks, and only needs a phone number or email address. (I think it's quaint in comparison that you need to give out your bank account number for other systems. Holding of account numbers is exactly the problem that modern mobile wallets worked so hard to solve with tokenization.)
https://en.wikipedia.org/wiki/ClearXchange
I consider it a huge marketing failure that this is such a little known capability.
https://en.wikipedia.org/wiki/ClearXchange
I consider it a huge marketing failure that this is such a little known capability.
The difference is, however, that you can assign a bank account a number that is unique within a given banking system.
Once you move into the realm of trying to tie an identifier you have no control over (and that has no reliable verification method for) to a place where money will go, you open yourself up to a smorgasbord of potential attack vectors, confusions and edge cases.
Once you move into the realm of trying to tie an identifier you have no control over (and that has no reliable verification method for) to a place where money will go, you open yourself up to a smorgasbord of potential attack vectors, confusions and edge cases.
Bank account numbers are not unique across banks. Account number schemes aren't even uniform across banks. The only limitation is what fits within the ACH system. You need the combination of routing number (to identify the bank) and account number (to identify the individual within the bank).
Zelle associates the bank account to a email address and / or phone number. Creation of these associations is gated through the member banks. So you have all the controls you have for any other banking task.
Zelle associates the bank account to a email address and / or phone number. Creation of these associations is gated through the member banks. So you have all the controls you have for any other banking task.
You'll note of course that I said "banking system," and I said "you can assign a bank account a unique identifier." i.e. I can create a scheme where I can map a particular account to a globally unique number, with rules about who owns which ranges etc.
So if Alice provides Bob a globally unique identifier to their bank account, then that's where it's going.
If Alice gives Bob a token that is hopefully mapped to the account, then there's an additional layer that can go wrong.
Eve can compromise Alice's e-mail or phone number, and then try and convince their bank that their bank account should now be associated with that identifier. If it's a different bank, then presumably this request must be federated through the third party system. And hopefully the 'true' owner is not identified by sending an e-mail or SMS.
Or maybe Eve just creates a bunch of accounts and tries to associate them with a bunch of telephone numbers and e-mail addresses that she can compromise at will and waits for the money to roll in. This is obviously more likely if some people have more than one account, as it means that the mapping of account -> e-mail/phone no. can no longer be mandatory.
In your example it sounds like they've got some reasonable safeguards in place (like ensuring you have to de-register a mapping before you can register a new one), the only point I was trying to make is that yes, it can be done, but to be done safely it's much harder than just simply having an account number (including routing number/sort code/swift code etc.).
You lose the 1-1 mapping and it gives you yet another thing that you have to actively manage/remember how it's set up/remember to change when you change e-mail provider or your phone number changes.
So if Alice provides Bob a globally unique identifier to their bank account, then that's where it's going.
If Alice gives Bob a token that is hopefully mapped to the account, then there's an additional layer that can go wrong.
Eve can compromise Alice's e-mail or phone number, and then try and convince their bank that their bank account should now be associated with that identifier. If it's a different bank, then presumably this request must be federated through the third party system. And hopefully the 'true' owner is not identified by sending an e-mail or SMS.
Or maybe Eve just creates a bunch of accounts and tries to associate them with a bunch of telephone numbers and e-mail addresses that she can compromise at will and waits for the money to roll in. This is obviously more likely if some people have more than one account, as it means that the mapping of account -> e-mail/phone no. can no longer be mandatory.
In your example it sounds like they've got some reasonable safeguards in place (like ensuring you have to de-register a mapping before you can register a new one), the only point I was trying to make is that yes, it can be done, but to be done safely it's much harder than just simply having an account number (including routing number/sort code/swift code etc.).
You lose the 1-1 mapping and it gives you yet another thing that you have to actively manage/remember how it's set up/remember to change when you change e-mail provider or your phone number changes.
If all you need is a phone number to pay, how does it handle multiple accounts? Or can you only sign up for a single account?
It's a 1:1 association. When I changed banks and tried to sign up online, it kept giving me an error. I stopped by a branch, and my old bank had registered me at some point. The teller had to call support and remove my previous association before they could create the new one.
Also, the association is to receive, not to send. Sending is initiated by your bank, which obviously already knows your account number. You send from your bank account to an email or phone number, which Zelle then translates to the target bank account.
Also, the association is to receive, not to send. Sending is initiated by your bank, which obviously already knows your account number. You send from your bank account to an email or phone number, which Zelle then translates to the target bank account.
The weirdness is historical. The '9'*94 padding and the line lengths are basically holdovers from when the ACH format was constrained by the capacity of 9 track tapes.
In most of Europe (UK and IE are one exception still using SWIFT) you just need the IBAN (international bank account number). In a lot of cases the payment is instant, even between countries, and free.
Unlike the US the bank account number shouldn't be guarded with your life, so businesses often have it listed on their website.
Unlike the US the bank account number shouldn't be guarded with your life, so businesses often have it listed on their website.
US Bank account numbers aren't proprietary knowledge. They're written in plaintext on the bottom of every check.
And that's a problem. Here's the discussion from last year.[0]
[0]https://news.ycombinator.com/item?id=11833009
[0]https://news.ycombinator.com/item?id=11833009
In the UK we have "faster payments" and money these days typically appears in about 15 minutes or so
Faster payments was mandated by the European Payment Services Directive.
All European countries have it.
Up next: Payment Services Directive 2 - API access to bank accounts. This is going to be interesting.
All European countries have it.
Up next: Payment Services Directive 2 - API access to bank accounts. This is going to be interesting.
its still not impelemnted everywhere, in croatia the exchange between banks is happening twice a day, i believe at 12h and 17h.
alot of people here dont like to buy stuff with CC so they transfer the money with internet banking, and also get a discount because that is treated as cash, and if the webshop is with another bank you have to wait to get a confirmation they received the money, or can try to send them transaction confirmation in PDF.
sometimes its anoying when you are ordering something expenensive and have to wait a bit longer because only after your payment do they order it from the distributor.
alot of people here dont like to buy stuff with CC so they transfer the money with internet banking, and also get a discount because that is treated as cash, and if the webshop is with another bank you have to wait to get a confirmation they received the money, or can try to send them transaction confirmation in PDF.
sometimes its anoying when you are ordering something expenensive and have to wait a bit longer because only after your payment do they order it from the distributor.
Any idea why the UK is still allowed to charge so much for outgoing Euro payments? It's reduced a lot, but my UK bank still charges £9 + their poor exchange rate.
Probably because they can. Monzo have said they aren't going to add charges where they don't get charged themselves (such as EU payments currently).
So get another bank. This particular problem does not need regulatory intervention because there are much cheaper options available.
I phoned up my bank and asked them how they calculate their fees and the exchange rate on foreign transactions. Needless to say, they couldn't give me an answer.
Even less that that. Yesterday I tried sending money from Starling Bank to Monzo and it literally took a few seconds.
In Denmark its instant, as in realtime, as in you buy my user car, transfer money through your bank app, and see it instantly in my bank app.
Yeap, in the Netherlands as well. You can see the money being subtracted in almost real-time when you buy stuff in a store.
Even calling it archaic is too harsh. Granted, the batch-centric nature is not ideal (and it's hard to imagine a system with this kind of latency built in being designed today), but if you're designing a system based on batch processing then shipping files over SFTP is a pretty reasonable way to do it.
That's a fair point.
It's very easy to get stuck inside the developer bubble, where 'archaic' means 'something that was state-of-the-art four years ago (or in the web/design world, 3.5 minutes ago)'.
SFTP is a nice technology which I make quite frequent use of in my job and at home. The idea that I plug in my Yubikey, provide a passphrase to unlock the key, authenticate to my server which that's got a verifiable certificate installed, and the private key never leaves my Yubikey is about as state-of-the-art as I could ever ask for security-wise.
Granted, precisely how the SFTP site is secured isn't specified and there's plenty of ways to do that wrong, but as a technology, it's always impressed me how seamlessly it works once it's setup.
It's very easy to get stuck inside the developer bubble, where 'archaic' means 'something that was state-of-the-art four years ago (or in the web/design world, 3.5 minutes ago)'.
SFTP is a nice technology which I make quite frequent use of in my job and at home. The idea that I plug in my Yubikey, provide a passphrase to unlock the key, authenticate to my server which that's got a verifiable certificate installed, and the private key never leaves my Yubikey is about as state-of-the-art as I could ever ask for security-wise.
Granted, precisely how the SFTP site is secured isn't specified and there's plenty of ways to do that wrong, but as a technology, it's always impressed me how seamlessly it works once it's setup.
SFTP tends to be the go-to method of integrating software between organizations, especially ones that aren't used to building actual APIs. It's standardized, everyone knows how to use it, and it essentially boils down to: generate a file and upload it to a server. There's all sorts of problems with it, and it tends to be a pretty short sighted decision.
The biggest problem I've found is that there's no real feedback loop (for this class of integration, not sftp specific). Company A uploads a file -- often in the middle of the night -- and Company B processes that file some time later. Could be a minute, could be a day. Company B fails to process the file, possibly because of a change on their end, possibly because of a change on Company A's end. There's usually a bunch of back and forth, but ultimately the earliest you can determine if a fix works is a full 24 hours after the file was initially dropped. Both companies could create integration environments with shorter feedback loops to help, but ultimately its a problem of not getting an immediate response that the integration is working.
95% of my job involves writing software to integrate systems from different companies, and the integrations that involve SFTP are almost always the biggest pains. I've thought about writing an SFTP server that operated in real time, and either rejected bad files or had some other way of responding to the inputs. Never really found the time to do it, and ultimately it'd be patching a process that should have just been a well designed API from the start.
The biggest problem I've found is that there's no real feedback loop (for this class of integration, not sftp specific). Company A uploads a file -- often in the middle of the night -- and Company B processes that file some time later. Could be a minute, could be a day. Company B fails to process the file, possibly because of a change on their end, possibly because of a change on Company A's end. There's usually a bunch of back and forth, but ultimately the earliest you can determine if a fix works is a full 24 hours after the file was initially dropped. Both companies could create integration environments with shorter feedback loops to help, but ultimately its a problem of not getting an immediate response that the integration is working.
95% of my job involves writing software to integrate systems from different companies, and the integrations that involve SFTP are almost always the biggest pains. I've thought about writing an SFTP server that operated in real time, and either rejected bad files or had some other way of responding to the inputs. Never really found the time to do it, and ultimately it'd be patching a process that should have just been a well designed API from the start.
I was just going to say, working for one of the largest destributors in the world, this is still how many large corps buy stuff: by downloading a catalog through FTP and sending the order with XML. I know, shocking but not everyone is Amazon.
> I know, shocking but not everyone is Amazon.
Amazon Vendor Central only has support for EDI (at least in the UK) - as in EDIFACT files sent over FTP.
If you want to do business with them you either need to talk that or be comfortable farming out to a 3rd party (I'm 1 day into a project where I'm finding all this out, and the hell of EDI file formats).
Amazon Vendor Central only has support for EDI (at least in the UK) - as in EDIFACT files sent over FTP.
If you want to do business with them you either need to talk that or be comfortable farming out to a 3rd party (I'm 1 day into a project where I'm finding all this out, and the hell of EDI file formats).
You may want to look at MWS then. It's the same batch processed feeds and XML over https instead of sftp.
Is it sftp or ftps? They're very different things and I am fairly certain it's the latter.
Most payment vendors are on SFTP at this point. Very few still use only FTPS. But then they do silly things with SFTP, like requiring both a password and the ssh key.
In the past I've done the same things, as some have mentioned already, as a form of two-factor authentication.
There were a few reasons for this that made a lot of sense:
(1) Private keys were being stored haphazardly -- people were not protecting them with passwords and occasionally they ended up on network drives. It's possible to enforce password policies for the hosts in question, but not possible to enforce 'how you protect/handle your private key' centrally[0].
(2) The second only applied to an environment I managed in the past, but we had many Linux hosts that were AD domain joined and authentication occurred directly via AD credentials (no local account authentication). The servers were configured to allow password-only login when the user had never connected to the host before. Once the user had connected the first time, the host would receive the public key for that user and future logins would only require the key[0].
[0] ... that latter part they failed to ever get working properly. It always prompted for both. Based on my conversations with the team managing the servers, there was zero priority assigned to fixing that problem -- they were resentful of the fact that the Linux (Debian, incidentally) hosts were domain joined and allowed domain authentication and they had a poor opinion of the security of using AD user accounts for authentication (nobody would make an argument other than Micro == small, soft == not hard or some other snark like that -- we had some BOFH over there).
There were a few reasons for this that made a lot of sense:
(1) Private keys were being stored haphazardly -- people were not protecting them with passwords and occasionally they ended up on network drives. It's possible to enforce password policies for the hosts in question, but not possible to enforce 'how you protect/handle your private key' centrally[0].
(2) The second only applied to an environment I managed in the past, but we had many Linux hosts that were AD domain joined and authentication occurred directly via AD credentials (no local account authentication). The servers were configured to allow password-only login when the user had never connected to the host before. Once the user had connected the first time, the host would receive the public key for that user and future logins would only require the key[0].
[0] ... that latter part they failed to ever get working properly. It always prompted for both. Based on my conversations with the team managing the servers, there was zero priority assigned to fixing that problem -- they were resentful of the fact that the Linux (Debian, incidentally) hosts were domain joined and allowed domain authentication and they had a poor opinion of the security of using AD user accounts for authentication (nobody would make an argument other than Micro == small, soft == not hard or some other snark like that -- we had some BOFH over there).
"like requiring both a password and the ssh key."
That doesn't sound that unreasonable. Both satisfy the "something you know" and "something you have" portions of multi-factor authentication (respectively).
Considering that this is money on the line, the idea of banks actually taking security seriously is actually refreshing. Too bad they don't seem to apply those standards to their customers.
That doesn't sound that unreasonable. Both satisfy the "something you know" and "something you have" portions of multi-factor authentication (respectively).
Considering that this is money on the line, the idea of banks actually taking security seriously is actually refreshing. Too bad they don't seem to apply those standards to their customers.
If you can duplicate it, then it's something you know, not something you have. Everything you know in sum-total is one meta-passphrase, even if it's a combination of key-files and other data.
The 'something you have' would have to be a specific hardware token (E.G. that VPN tunnel device someone else mentioned that is a black box you aren't allowed to open). Tamper resistance and physical security locks to ensure that connections /must/ route through a given location would be enough.
The 'something you have' would have to be a specific hardware token (E.G. that VPN tunnel device someone else mentioned that is a black box you aren't allowed to open). Tamper resistance and physical security locks to ensure that connections /must/ route through a given location would be enough.
"If you can duplicate it, then it's something you know, not something you have."
By this logic, a typical door key is "something you know" rather than "something you have" (never mind the uselessness of memorizing one's key shape when it comes to actually unlocking a door).
Unless you're memorizing your private SSH keys (in which case I applaud your superior intellect), it's absurd to claim that said keys fall into "something you know".
By this logic, a typical door key is "something you know" rather than "something you have" (never mind the uselessness of memorizing one's key shape when it comes to actually unlocking a door).
Unless you're memorizing your private SSH keys (in which case I applaud your superior intellect), it's absurd to claim that said keys fall into "something you know".
It's a key pair, for SSH keys, which is what SFTP uses to tunnel the FTP protocol. You cannot derive the private key from the public key. So no, you cannot duplicate someone's private key unless you have access to their machine. In which case you could just install a keylogger.
Hardware tokens are vulnerable too, by your own admission. A 6 digit pin created by a token is still "something you know" even if it's only good for a couple of minutes.
Hardware tokens are vulnerable too, by your own admission. A 6 digit pin created by a token is still "something you know" even if it's only good for a couple of minutes.
[deleted]
Actually it's the former, not the latter. Several banks I've worked with used SFTP.
Secure FTP is actually a superset of SFTP, because there is also a FTPS protocol that is also a kind of "secure FTP".
Correct, you are. The later parts in the article, though, specifically call out SFTP by name. Based on the other comments and some searches I did before hitting that second part, I'm fairly certain they're actually using SFTP rather than FTPS, but having never actually touched the ACH system, I can't speak first-hand on that.
I've written an SFTP server (nodejs) that passes the files to the proper API this summer. No magic there and the user who doesn't have the resources to implement an API client is happy.
I came to ask the same question, I followed the comments link on the tfa and notice it linked back to HN 1200 days ago and that was one of the top questions.
I don't think that there's anything archaic about using SFTP to move files around!
For confirmation, SFTP is mentioned directly (at least once) in the second article of the series.
not necessarily SFTP, always FTP in some form in the US. I have seen FTPS, I have also seen banks just use FTP by default without encryption (shocking? not really).
I had an integrator request this so I stood up a nodeJS server that only implements upload, not download. This way if they leaked their own password, a malicious actor is limited to forging data, and no real data can be leaked. Because it didn't work in FileZilla, they didn't want to use it. Worked at another company that shuffled data between big name gyms & health insurance companies, it also used CSV files sent over FTP in all directions, to my dismay. CSV isn't even a well defined format, and you get all kinds of impedance mismatches with different delimiter & escaping mechanisms, character encodings, BOM, etc. Other companies will just give you a SQL user & let you go to town mining their database directly. I don't understand whats so difficult about making an API, but sometimes it seems like no one wants to do it. You can't push back too much or they will just see you as a problem & decide not to integrate with you.
Funnily, you're complaining about csv formatting, guess you never worked with ACH formatting. It's cobol fixed length files. I implemented parsing/creation for this file format at two different companies. Hired on at one company, first day, my manager said, "hey, you have experience with ACH right? we have this project...". That's when you learn to start leaving stuff off your resume.
You can do a lot worse than ACH. It's hard to read, but it's simple and pretty well-defined.
What _really_ sucks is one-off fixed-width formats that aren't well defined, or that change suddenly (oh, you thought that field would always be populated? lol no.)
What _really_ sucks is one-off fixed-width formats that aren't well defined, or that change suddenly (oh, you thought that field would always be populated? lol no.)
Yeah, ACH is surprisingly not-unpleasant, at least relative to nightmares like X12 EDI with its implicit looping constructs and billions of companion guides that supercede random parts of the base spec.
X12 EDI, been there earned the badge, both on the generating and receiving sides. 837, 834, 835 and others in health care - fun! Positional format, with situational meaning... really it is a fascinating format.
i wrote the X12 EDI stuff for a major clearwater distributor... also wrote the ACH stuff that dumped someone a file and their job was to upload it via SFTP to the bank. so many badges.
What _really really_ sucks is well-defined, reasonable, meticulously documented formats where all in-the-wild implementations stray from the spec in different ways :-/
[deleted]
Years ago I worked with a product that parsed line printer data, it was a nightmare trying to get every piece of optional data into the reports.
Previously I had no experience with ACH but my current project is essentially reverse engineering a very proprietary version of NACHA and another file that is 400 fields long. Honestly the current parser does work but I can only assume due to some black magic or other voodoo.
Every time someone complains about file formats I bring up a similar story. It's hard to believe fixed width files are still being used today. Parsing these kind of files with 45 million records, each 600+ bytes is a huge pain. I chuckle when I think about the first time I dealt with these files and thought I could open them in an editor...
> It's hard to believe fixed width files are still being used today. Parsing these kind of files with 45 million records, each 600+ bytes is a huge pain. I chuckle when I think about the first time I dealt with these files and thought I could open them in an editor...
Depending on the business-application, fixed-width formats are fantastic because they allow for genuine O(1) random-access to records without needing to precompute an index first. Writing parsers for fixed-width formats is also much simpler, for example, because it eliminates ambiguity around null-terminators vs length-prefix in text/string/array data - for resource-constrained environments it's great because you can guarantee you won't need to dynamically-allocate buffers. I feel the only real arguments against fixed-width records are concerned with wasted space - which are mitigated if your system lets you compress them somehow - because they'll typically compress very, very well.
Depending on the business-application, fixed-width formats are fantastic because they allow for genuine O(1) random-access to records without needing to precompute an index first. Writing parsers for fixed-width formats is also much simpler, for example, because it eliminates ambiguity around null-terminators vs length-prefix in text/string/array data - for resource-constrained environments it's great because you can guarantee you won't need to dynamically-allocate buffers. I feel the only real arguments against fixed-width records are concerned with wasted space - which are mitigated if your system lets you compress them somehow - because they'll typically compress very, very well.
> I feel the only real arguments against fixed-width records are concerned with wasted space
Sometimes, fields can turn out to be too short. I have received a few letters from business with my last name truncated, because somebody way back when decided eight or nine characters are definitely enough for a surname.
Sometimes, fields can turn out to be too short. I have received a few letters from business with my last name truncated, because somebody way back when decided eight or nine characters are definitely enough for a surname.
Ha! I wrote an editor for fixed length files, it lets you map to a cobol copybook, or create the layout manually. I never released it. Anyone interested?
If it works on Windows, I'd love to give it a try! Our ERP system is written in COBOL and I have to deal with fixed-width files all the time. Does it also handle REDEFINES and level 88?
Yes it handles redefines and 88s, my emails in my profile, I have a Windows build.
Ugh, reminds me of the time I had to integrate with a bunch of school software. They used csv over ftp (no auth). You just had to connect to the server and get the personal information of a district's students. Worse is you could google the URLs... I built a system that solved the problem but left shortly after it began to be integrated nationwide.
wait.. are you saying that a district had public ftp with personal student information? Pretty sure that breaks the law.
Email me at josh.ribakoff at Gmail I'd be interested in chatting. I think we worked at the same company. Or direct competitors
Sure, I'll do.
ACH needs to die. Its a very old system, and you have no clue if a transfer ever actually goes through. Only ACH rejects ever get sent back, and there isn't a definitive window in which you'll get that reject.
Wait. ACH doesn’t even tell the sender if it succeeded??!
NAK-but-no-ACK is apparently a super common pattern in enterprise (both internal and between parties.)
I have no idea why it is and why people are so resistant to adopting ACKs.
I have no idea why it is and why people are so resistant to adopting ACKs.
Why it's there in the first place is fairly excusable. It (at least) halves the number of datagrams. Why people still resist it I'm not so sure about.
But on the other hand, it's a complicated architectural decision. US military wants absolutely reliable communications? They invent TCP which relies a lot on acks. Swedish telecom company wants fast and absolutely reliable communicating systems? Cue throw hands in air with a "there's no guarantees even with socks so better to simply never assume your message arrived and instead focus on detecting anomalous behaviour." They invent Erlang where there are no acks for messages.
But on the other hand, it's a complicated architectural decision. US military wants absolutely reliable communications? They invent TCP which relies a lot on acks. Swedish telecom company wants fast and absolutely reliable communicating systems? Cue throw hands in air with a "there's no guarantees even with socks so better to simply never assume your message arrived and instead focus on detecting anomalous behaviour." They invent Erlang where there are no acks for messages.
[deleted]
Yeah there are no acks. It sucks.
The lack of acks is why most companies receiving money through ACH will "season" the transactions for a set amount of time. Most of the returns occur in the first 3 business days so most seasoning periods are just around that long. The reality is you can receive a reject up until (I believe) 60 days after the transaction date. I've seen a lot of companies be a little smarter here and reduce the seasoning times for repeat users using the same source bank account. It is a fascinating system to work with every day. Wire transfers are their own bundles of fun too.
Now I finally know why my credit-union (which serves the high-technology industry exclusively) told me that they couldn't put a hold or otherwise segregate any cheques I deposit until they've cleared... because they have no way of knowing when/if they've cleared!
It's 60 days, + some possibility of up to 2 extra days at the end for Fed Reserve holidays, and then in cases of outright fraud, occasionally banks will ignore the rules and return stuff later than that.
They already have an API, it's called FTP or SFTP or whatever.
I don't know why everyone thinks they need an API to reimplement (poorly) basic unix functionality that's been around be for years.
Especially when FTP and HTTP performance is basically comparable and will depend more on the network and storage than the protocol.
Have we gotten to the point where anything not HTTP is considered old?
Have we gotten to the point where anything not HTTP is considered old?
never mind that most "API" are JSON (or some such) delivered over (S)HTTP anyways.
But yes, anything but HTTP is "old". In large part because of corporate firewall rules that block anything but HTTP traffic (and more often than not filter even that traffic).
But yes, anything but HTTP is "old". In large part because of corporate firewall rules that block anything but HTTP traffic (and more often than not filter even that traffic).
No. I'm referring to an API to query data in realtime instead of batching the entire freaking database and then having out dated snapshot of the data
>Because it didn't work in FileZilla, they didn't want to use it
Sounds like they didn't need much if any of what you did. They wanted X and you gave them Y. Y would require them to change the way they worked. Did they need query capability? Seems like they didn't and SFTP would have suited them better.
Honestly sounds to me like you went for an overly complex solution where something simple would have done just fine.
Sounds like they didn't need much if any of what you did. They wanted X and you gave them Y. Y would require them to change the way they worked. Did they need query capability? Seems like they didn't and SFTP would have suited them better.
Honestly sounds to me like you went for an overly complex solution where something simple would have done just fine.
Our company and their company have a common customer. Neither of us need each other we want to make our customer happy. Customer would be happiest with realtime data but company B wants to send snapshots via sftp. So I setup an endpoint that speaks sftp. They even told me to "do something" to make it secure. Then they complained I used a port other than 22 and that it only worked on the command line. It sounds like the problem is the other company doesn't want to do any work / wants to send the file manually. I don't see how I went for an overly complex solution. I attempted to do exactly what they asked and they moved the target so of course I missed. In fact it's this company that is just trying to avoid doing any programming. But that comes with the territory. I respect your opinion but disagree with you
Fair enough, I didn't have the backstory.
But but rewrite all the things in node.js!! Don't worry about understanding the file format, which is well established, or what the users of the file are actually doing with it, also well established, the important thing is node.js!!!
Frankly once they hear it's node.js the users should be bowing down to worship!!!!
Frankly once they hear it's node.js the users should be bowing down to worship!!!!
Actually I wanted him to make an https API for us to query real time data. I would have used PHP to consume his API because it's easier to find developers on our end and synchronous code is easier to maintain when performance is not a concern
why use a huge codebase like node for such a simple security critical task
When all you have is a hammer...
How else do you suggest you standup a sftp that cannot leak data even if the password is compromised? I'm all ears. Node is a runtime by the way and huge in what regard? The Linux codebase is also large.
It is a system administration problem, not write a new software problem.
You:
a) think about the scope of the problem and what you are trying to achive - you are blocking reads of the files by the uid/gid of the server. It is a known and solved problem. The tool is called chmod
b) think about the surface area of the attack - it should only be the SFTP server. We have ones that are very well known. I recommend the one that comes with OpenSSH but there are others.
c) think about business requirements - user's provisioning etc. That has been a solved problem for years - for the super-complex cases LDAP is used. For the simpler ones you can use PAM.
After thinking about that you just write the needed glue.
You:
a) think about the scope of the problem and what you are trying to achive - you are blocking reads of the files by the uid/gid of the server. It is a known and solved problem. The tool is called chmod
b) think about the surface area of the attack - it should only be the SFTP server. We have ones that are very well known. I recommend the one that comes with OpenSSH but there are others.
c) think about business requirements - user's provisioning etc. That has been a solved problem for years - for the super-complex cases LDAP is used. For the simpler ones you can use PAM.
After thinking about that you just write the needed glue.
Seconding what kalleboo said, setting directory permissions has been a staple of file-sharing for decades. Setting up a node server to block read-access seems incredibly byzantine.
Good point but we also have to load balance this / have failover and provision users on the fly. I could have used messaging queues to provision PAM users on the whole cluster and this guy would have still complained because I used a non standard port. Not sure why that was a problem either but again we can't press back or question him. he would cancel the whole integration. Bottom line is our end user wanted real time data (presumably via an API)
A lot of FTP servers have provisions to essentially "ignore" the user and proxy into the directory. As far as load balancing is concerned, most FTP servers can handle very high throughput because of the simplicity of the protocol. If you really did want to load balance it, haproxy can do that just fine.
> we can't press back or question him. he would cancel the whole integration
This attitude will not serve you well. There's saying no, and then there's getting more information (requirements!?!) to then guide people to better decisions.
This attitude will not serve you well. There's saying no, and then there's getting more information (requirements!?!) to then guide people to better decisions.
what's wrong with nonstandard ports?
Can't you just host a directory with -w- permissions? "Drop box" directories have been a staple of file sharing software for decades.
You use chmod on the directory...writing any code for this kind of task would be crazy overkill.
First thing that comes to my mind is a PAM plugin that generates a new root directory for the user when they log in, named something sane under a fixed hierarchy. A recursive walk can later pick them all up and clean up.
I did a similar thing, but decided to have LIST/MLSD/NLST, but stub out GET/RETR/MFCT/MFF/MFMT to return nothing or static data.
What did you use to implement?
Wrote it myself because it was Elixir so why not. I'd be more careful doing this in C.
Reminds me of a time when I had to implement a parser for a certain kind of file to transfer money using an "API" provided by a financial institution in Asia. Yes we are using CSV too, also sent over SFTP. The file format is, unsurprisingly, ill-defined.
Wouldn't FileZilla have been happy with a fake blank directory response?
Yeah I added that but the particular node sftp implementation I used from github just simply didn't work with FileZilla. Only the command line. Don't know why.
Whenever someone says “from github”, I mentally replace it with “from some random guy on the street” to give it its appropriate intuitive feeling.
Sure I don't care who wrote it if it works and passes our code review and the license matches. Plenty of crap software comes out of big name companies and plenty of random people on the street can write good code.
If you have no other alternative and can’t take the time to write it for yourself, and need something now, sure, go nuts and take something random off Github, PyPI, npm, or whatever.
But code that does not have upstream development and support is dead code. A piece of code in production is like an Internet connection or electrical or water hookup – it must be, so to speak, “connected” upstream to whoever is providing continuous upgrades and security/bug fixes. Otherwise, it is (to continue stretching the analogy) like a mystery battery or water barrel – it could go bad in many ways at any time and you wouldn’t know it until your equipment starts to fail because of low voltage or voltage spikes, or you start to die of legionnaires’ disease. And unlike a battery or water, there are only cursory, no good and thorough, ways to test their quality, and no way to test their age as measured by the way they interact with the ever-changing outside world.
But code that does not have upstream development and support is dead code. A piece of code in production is like an Internet connection or electrical or water hookup – it must be, so to speak, “connected” upstream to whoever is providing continuous upgrades and security/bug fixes. Otherwise, it is (to continue stretching the analogy) like a mystery battery or water barrel – it could go bad in many ways at any time and you wouldn’t know it until your equipment starts to fail because of low voltage or voltage spikes, or you start to die of legionnaires’ disease. And unlike a battery or water, there are only cursory, no good and thorough, ways to test their quality, and no way to test their age as measured by the way they interact with the ever-changing outside world.
I've always been overly eager to add dependencies to projects and while I know why it may not be good I was missing a good intuitive sense of why. This is such a good way to view it. Thanks!
Here's a good NPR Planet Money episode on how transferring money works.
http://www.npr.org/sections/money/2013/10/04/229224964/episo...
Apparently, back in the days before ACH, banks met up at a parking lot in NYC every night and literally exchanged bags of paper checks.
There was a proposal build something better than ACH, but it was denied because the cost to upgrade the infrastructure would cost too much for small banks and credit unions.
http://www.npr.org/sections/money/2013/10/04/229224964/episo...
Apparently, back in the days before ACH, banks met up at a parking lot in NYC every night and literally exchanged bags of paper checks.
There was a proposal build something better than ACH, but it was denied because the cost to upgrade the infrastructure would cost too much for small banks and credit unions.
I worked on the ACH system at the Federal Reserve Bank. When you're getting multi-gigabyte files from the Social Security Service daily that have many millions of transactions in them, you appreciate the NACHA format's compactness (~100 bytes each tx). We never transmitted files on insecure protocols like FTP, though.
Every bank I've worked with uses SFTP rather than FTP.
Just to be pedantic. SFTP (file transfer subsystem of SSH) or FTPS (FTP + TLS)?
I mean, either is fine, I would just imagine that if it was FTP at some point, moving to FTPS wouldn't be unreasonable.
I mean, either is fine, I would just imagine that if it was FTP at some point, moving to FTPS wouldn't be unreasonable.
SFTP means exactly SFTP, and it's not ambiguous. FTPS is a completely different protocol.
To you, maybe. To the rest of the world, you have to check and be explicit about it.
I've worked on a similar system running on plain FTP (wasn't SFTP created just in something like 2000? So that it wouldn't have been an option back when the system was created) but it had no obvious security flaws because of this - all the FTP handled was the exchange of encrypted&signed files with encrypted&signed ACK messages, the same system might have been run over something like plain email.
Same here when I worked at a Mortgage company. SFTP everywhere.
Did they use sftp from the OpenSSH package or was it some sort of commercial variant?
Just sharing mine. My company customer consist of major international financial establishment.
As some simply can't go with certain open source project due to compliance, we settled with Tectia SSH[1]. Similar story with VPN and other security-related stuff.
Everything is SFTP with them. Even API json response lol
1. https://www.ssh.com/products/tectia-ssh
As some simply can't go with certain open source project due to compliance, we settled with Tectia SSH[1]. Similar story with VPN and other security-related stuff.
Everything is SFTP with them. Even API json response lol
1. https://www.ssh.com/products/tectia-ssh
Linux/Ruby systems used openssh sftp. Windows Server/SQL Server used some POS SQL Server SSIS sftp plugin.
A good story on ACH: http://www.npr.org/templates/transcript/transcript.php?story...
They mention that other regions' inter-bank money-transfer systems (e.g. the EU's) have been sped up to be same-day, or in some cases nearly instantaneous. The US ACH system lags behind, due to the sheer number of institutions that would be involved in a modernization effort. (There are a lot more US banks than there are UK/French/Canadian/Australian/etc. banks; I think in part because a bank that operates in 50 states is technically—and legally—50 banks, and each one maintains its own ACH infrastructure?)
They mention that other regions' inter-bank money-transfer systems (e.g. the EU's) have been sped up to be same-day, or in some cases nearly instantaneous. The US ACH system lags behind, due to the sheer number of institutions that would be involved in a modernization effort. (There are a lot more US banks than there are UK/French/Canadian/Australian/etc. banks; I think in part because a bank that operates in 50 states is technically—and legally—50 banks, and each one maintains its own ACH infrastructure?)
The number of banks in the U.S. is a legacy of President Jackson, who campaigned (and won) against rechartering the Second Bank of the United States
In general, and until very recently, Congress has abstained from directly regulating the banking industry. AFAIU, banks are subject to Federal law primarily indirectly via regulations promulgated by the Federal Reserve. But the Federal Reserve system is opt-in, and many smaller banks contract with larger banks to leverage the Federal Reserve transactional systems, and are therefore for the most part not subject to those rules.
It's all rather ironic because banking was one of the few areas of national commerce that was clearly contemplated to fall under federal regulatory powers, to some extent, during the time of the Constitution. Though opposed by Jefferson and Madison, the First National Bank was chartered in 1791. Yet now that federal powers have expanded to encompass every aspect of commerce, the legacy of the failure of the Second National Bank remains incredibly strong.
https://en.wikipedia.org/wiki/Bank_War
In general, and until very recently, Congress has abstained from directly regulating the banking industry. AFAIU, banks are subject to Federal law primarily indirectly via regulations promulgated by the Federal Reserve. But the Federal Reserve system is opt-in, and many smaller banks contract with larger banks to leverage the Federal Reserve transactional systems, and are therefore for the most part not subject to those rules.
It's all rather ironic because banking was one of the few areas of national commerce that was clearly contemplated to fall under federal regulatory powers, to some extent, during the time of the Constitution. Though opposed by Jefferson and Madison, the First National Bank was chartered in 1791. Yet now that federal powers have expanded to encompass every aspect of commerce, the legacy of the failure of the Second National Bank remains incredibly strong.
The NPR story touches on the real reason it won't be upgraded - because it would cannibalize the $30/transaction revenue banks get from wire transfers.
Seems like they wouldn't need to involve everyone. Just make it semantically similar to ACH, but for some institutions clear through the new infrastructure. Even a few banks forming a coalition would be enough to make it worthwhile.
It needs to involve everyone because the only way to get there is to force them, and forcing only some of banks is kind of unfair.
E.g. the EU got there by legal limits on transaction settlement time set by the first payment services directive, the infrastructure came into place only when the banks got the message that they have to improve the product or else (IIRC the law specified that after a couple years, customers would be entitled to compensation if their payments were "too slow"). Without the requirements, EU banks would still be happy to charge 10 eur for a 3 day payment, since that's obviously more profitable than charging cents or zero for same day payments, even if the payments would be running over fast&cheap infrastructure - case in point, UK banks, which charge extreme fees for EU EUR payments simply because they, as a non-EUR country, aren't forbidden by the EU to do so.
E.g. the EU got there by legal limits on transaction settlement time set by the first payment services directive, the infrastructure came into place only when the banks got the message that they have to improve the product or else (IIRC the law specified that after a couple years, customers would be entitled to compensation if their payments were "too slow"). Without the requirements, EU banks would still be happy to charge 10 eur for a 3 day payment, since that's obviously more profitable than charging cents or zero for same day payments, even if the payments would be running over fast&cheap infrastructure - case in point, UK banks, which charge extreme fees for EU EUR payments simply because they, as a non-EUR country, aren't forbidden by the EU to do so.
The problem is that no businesses want to spend money to fix something that isn’t broken. It’s the reason XP (or earlier!) is/are still around; “Why should we spend money updating to Windows 7 when 2000 works just fine?”
As of September 15, same-day transfer will be possible as part of phase 2 of the modernization plan:
https://www.nacha.org/rules/same-day-ach-moving-payments-fas...
https://www.nacha.org/rules/same-day-ach-moving-payments-fas...
I work in banking and it's always interesting to:
1/ See people encounter how these things work, because there's usually a sense of lost innocence about it. (If they stick around long enough they come to understand that dealing with hundreds of years of history is why glib "re-imagine everything" solutions tend to come a cropper).
2/ Continually discover that by the standards of the rest of the world, US banking is even more like banging rocks together.
1/ See people encounter how these things work, because there's usually a sense of lost innocence about it. (If they stick around long enough they come to understand that dealing with hundreds of years of history is why glib "re-imagine everything" solutions tend to come a cropper).
2/ Continually discover that by the standards of the rest of the world, US banking is even more like banging rocks together.
The UK's Faster Payments system does same day (often less than an hour) inter-bank transfers for up to £250,000.
http://www.fasterpayments.org.uk/about-us/how-faster-payment...
http://www.fasterpayments.org.uk/about-us/how-faster-payment...
Faster payments was able to work because the UK gov forced them to do it. The UK has the advantage of having a few very large banks (unlike the US that had around 15,000 different banks). The government was able to get the dozen or so banks to work together and get same day ACH working.
It's also a "push" system (and not a "pull" system like ACH) which has serious advantages -- you can send money to anyone given their sort code and account number, and those numbers do not need to be kept private.
With ACH if you have someone's routing number and account number you can pull money from their account without them needing to authorise it.
With ACH if you have someone's routing number and account number you can pull money from their account without them needing to authorise it.
In the UK there is also a 'pull' system (Direct Debit) in which you instruct your bank that a given third party can regularly take money out of your account until further notice.
It's commonplace for bill payments, subscription services, etc.
It's commonplace for bill payments, subscription services, etc.
And should be noted that not anyone can receive direct debits so there still isn't a large need for secrecy of routing numbers.
Mexico allows to transfer money between bank accounts in seconds: http://www.banxico.org.mx/sistemas-de-pago/servicios/sistema...
In 20 years FTP will still be a thing and whatever JS/Ajax/RPC/WSDL/JSON thing kids are using these days will be as dead as CORBA.
SFTP isn't that old and shares nothing with FTP other than three letters.
SFTP is the FTP protocol running through a SSL socket where as FTP is the FTP protocol running through a plain old socket.
So they do have a lot in common.
So they do have a lot in common.
You are confusing SFTP and FTPS there.
So please tell me what is the difference between FTP and SFTP.
But before we start to discuss that point lets at least agree SFTP is some sort of FTP protocol over SSL?
But before we start to discuss that point lets at least agree SFTP is some sort of FTP protocol over SSL?
It's not, SFTP is a binary protocol using SSH as transport layer. FTPS is plain old FTP over SSL/TLS. They're completely different protocols that have nothing in common except being used to transfer files.
Who knows if this link is accurate:
https://en.wikipedia.org/wiki/File_Transfer_Protocol
But it says:
SSH File Transfer Protocol
The SSH file transfer protocol (chronologically the second of the two protocols abbreviated SFTP) transfers files and has a similar command set for users, but uses the Secure Shell protocol (SSH) to transfer files. Unlike FTP, it encrypts both commands and data, preventing passwords and sensitive information from being transmitted openly over the network. It cannot interoperate with FTP software.
This suggestion this term chronologically the second of the two protocols abbreviated SFTP is duplicitous.
https://en.wikipedia.org/wiki/File_Transfer_Protocol
But it says:
SSH File Transfer Protocol
The SSH file transfer protocol (chronologically the second of the two protocols abbreviated SFTP) transfers files and has a similar command set for users, but uses the Secure Shell protocol (SSH) to transfer files. Unlike FTP, it encrypts both commands and data, preventing passwords and sensitive information from being transmitted openly over the network. It cannot interoperate with FTP software.
This suggestion this term chronologically the second of the two protocols abbreviated SFTP is duplicitous.
FTPS is a protocol (FTP) running through an SSL socket. SFTP is totally different and based on SSH.
If you look at my post I did not say SSH I said SSL.
EDIT: I think you will find FTPS is based on TSL not SSL. TSL is an alternative to SSL.
EDIT: I think you will find FTPS is based on TSL not SSL. TSL is an alternative to SSL.
Well regarding "TSL" I'm sure you mean TLS, and it's not just an alternative to SSL, it's an evolution to SSL. SSL nowadays is considered insecure and everything is using TLS.
About the SFTP vs FTPS, SFTP is a completely new protocol based on SSH. FTPS is the plain old FTP wrapped in a TLS/SSL socket.
About the SFTP vs FTPS, SFTP is a completely new protocol based on SSH. FTPS is the plain old FTP wrapped in a TLS/SSL socket.
> Well regarding "TSL" I'm sure you mean TLS
that is correct. That was my typo :(
> SFTP is a completely new protocol based on SSH.
If that is the case please explain the following scenario.
I wrote an editor that had FTP capabilities but it was based on the FTP RFC 913 specification using nothing but plain old sockets.
As a result, many user loved the FTP feature, but they asked for a version that would work over the SFTP protocol.
So I implemented an SFTP version of that editor by using OpenSLL to create a SSL socket. The editor just created a secure socket, but used the exact same FTP RFC 913 specification to talk to the server using that socket.
Ever user who asked for this change reported that it now worked perfectly with their SFTP (not TLS) server.
So how is that possible that my editor could implement the SFTP protocol with just a change to the way the socket was opened?
PS: If you don't believe me, you can trial that exact same SFTP (OpenSSL) editor from the link below and I'm pretty sure you'll find it still works with an SFTP server: http://www.zeusedit.com/download.html
that is correct. That was my typo :(
> SFTP is a completely new protocol based on SSH.
If that is the case please explain the following scenario.
I wrote an editor that had FTP capabilities but it was based on the FTP RFC 913 specification using nothing but plain old sockets.
As a result, many user loved the FTP feature, but they asked for a version that would work over the SFTP protocol.
So I implemented an SFTP version of that editor by using OpenSLL to create a SSL socket. The editor just created a secure socket, but used the exact same FTP RFC 913 specification to talk to the server using that socket.
Ever user who asked for this change reported that it now worked perfectly with their SFTP (not TLS) server.
So how is that possible that my editor could implement the SFTP protocol with just a change to the way the socket was opened?
PS: If you don't believe me, you can trial that exact same SFTP (OpenSSL) editor from the link below and I'm pretty sure you'll find it still works with an SFTP server: http://www.zeusedit.com/download.html
Maybe your customer mistook their SFTP server for an FTPS server? I'm afraid I don't have a Windows machine to test your software on, but if you're curious I encourage you to set up an SSH server like OpenSSH (which provides SFTP as well) and try with that. I'd be really surprised if your FTPS implementation managed to work with it (unless OpenSSH's SFTP implements an FTPS fallback).
[deleted]
For us and our bank it's gpg encrypted and then transferred over SFTP.
Sure, the ACH file format is sort of sucky, but it's not like it's difficult. The lack of an ACK is super awful tho. Payroll has to call in 20m after sending to verify.
Sure, the ACH file format is sort of sucky, but it's not like it's difficult. The lack of an ACK is super awful tho. Payroll has to call in 20m after sending to verify.
At least it's secure and usually FTPS...hopefully using TLS 1.2+ cryptographic protocol.
Previous discussion:
https://news.ycombinator.com/item?id=7636066
Previous discussion:
https://news.ycombinator.com/item?id=7636066
Just a clarification, NACHA uses SFTP rather than FTPS.
Which is a bit like saying, at least the 70mph mountain pass road has a guard rail.
Well, if not TLS, then what would you use? Noise?
TLS is the only widely deployed standard cryptographic protocol which has ever protected any internet communication to any degree. For its many flaws and patchwork, there is nothing even competing in the category.
TLS is the only widely deployed standard cryptographic protocol which has ever protected any internet communication to any degree. For its many flaws and patchwork, there is nothing even competing in the category.
There is ssh.
Actually sftp (ssh) tends to be a lot more popular than FTPS (TLS) because of the whole FTPS NAT catastrophe that sftp doesn't have.
Actually sftp (ssh) tends to be a lot more popular than FTPS (TLS) because of the whole FTPS NAT catastrophe that sftp doesn't have.
Yes. In my experience FTPS is a whole bucket of nope. I've never encountered a situation where we couldn't use SFTP.
The only time I've had FTPS work 'well' is when the client & server were both written by the same company (Tumbleweed) and they don't follow the RFC exactly.
The only time I've had FTPS work 'well' is when the client & server were both written by the same company (Tumbleweed) and they don't follow the RFC exactly.
It doesn't matter if the connection is secured if the data itself is not secured. Transport security is a red herring in a complex system.
If you are driving a car at 70mph around a mountain pass, even a really strong guard rail leaves the possibility that you could plummet ten thousand feet to your death. If, on the other hand, you were driving on the Bonneville salt flats at 45mph, there is much, much less danger.
Secure your data. Then paint the bike shed.
If you are driving a car at 70mph around a mountain pass, even a really strong guard rail leaves the possibility that you could plummet ten thousand feet to your death. If, on the other hand, you were driving on the Bonneville salt flats at 45mph, there is much, much less danger.
Secure your data. Then paint the bike shed.
Having spent a few years at a large energy company, I got quite used to the use of FTP servers to exchange--what else--csv files with data. And is a uploading/downloading a file to/from some ftp server really that different from POST/GETing an object to/from some REST service?
Some major news/market information provider solely made their data available to us through ftp. And used Amazon SNS to push a notification that something new is available on that ftp.
Some major news/market information provider solely made their data available to us through ftp. And used Amazon SNS to push a notification that something new is available on that ftp.
I used a SOAP service once, which had a single string-typed element. In that element went some XML, encoded as an XML string. I found out later that the original version used FTP drops, and all they did is wrap the FTP'd file as a string in the SOAP request.
So no, sometimes there's very little difference between the two. FTP is just another way to send data over a network. I mean, you could even write a custom FTP server that read the file directly into processing instead of dropping it on a file system and reading it from there in some other process.
So no, sometimes there's very little difference between the two. FTP is just another way to send data over a network. I mean, you could even write a custom FTP server that read the file directly into processing instead of dropping it on a file system and reading it from there in some other process.
I'm disappointed that they use SFTP rather than UUCP. But I guess ACH isn't quite that old...
I've implanted ACH file format and, worse, FedWire BAI2 file parsing… it's absolutely archaic. The worst part is that various partner banks will have differently erroneous variations of their implementation of the BAI2 spec… so we had to intentionally code buggy version to match the bugs they had on the other side… ridiculous.
FWIW, Swift (the company behind the interbank payment system) has been developing and pushing ISO 20012 as an XML-based long term replacement for their Swift message format, though it's not designed as a replacement for ACH.
For that, there was HBCI years ago (also XML); don't know if it's used much still.
For that, there was HBCI years ago (also XML); don't know if it's used much still.
If I remember correctly, HBCI is often used for consumer banking software to get account data from the bank.
There are several companies that provide an API on top of ACH. I work for one[1]. For high volume ACH (like a payroll company) it's usually cheaper to go through an API provider than it is to go directly through the bank. I'm not exactly sure why. Maybe because we handle technical support? We also have better reporting.
One of the challenges for banks is that there is an oligopoly on the software that runs the bank. There are 4 companies that provide the "core banking" software to most of the banks in the USA. The banks get stuck providing you with whatever services one of these four pieces of software is capable of.
[1] http://acheck21.com/api/
One of the challenges for banks is that there is an oligopoly on the software that runs the bank. There are 4 companies that provide the "core banking" software to most of the banks in the USA. The banks get stuck providing you with whatever services one of these four pieces of software is capable of.
[1] http://acheck21.com/api/
From my experience it's extremely rare even for large companies to directly deal with ODFIs.
Typically there's AT LEAST one intermediary payment processor (like Chase Paymentech) involved in the TX.
The downside to this is merchant has to register with multiple processing entities but things like "send and forget" APIs (so no need to batch things manually) - which makes it easy to combine ACH and CC payment acceptance in the same system, reporting/reconciliation, out of the box UI merchant can use to look up transactions etc etc outweigh that inconvenience.
Typically there's AT LEAST one intermediary payment processor (like Chase Paymentech) involved in the TX.
The downside to this is merchant has to register with multiple processing entities but things like "send and forget" APIs (so no need to batch things manually) - which makes it easy to combine ACH and CC payment acceptance in the same system, reporting/reconciliation, out of the box UI merchant can use to look up transactions etc etc outweigh that inconvenience.
What is the cost per transaction? I know stripe charges 85bps.
Our pricing is typically in the range of $.05-$.35 flat fee per transaction depending on various factors. The main one being your monthly transaction volume. We have a monthly minimum of $250.
We never charge basis points but places that do might make more sense for people who have low volume and low dollar amount transactions. I would say a lot of brick and mortar businesses fall into this category.
We never charge basis points but places that do might make more sense for people who have low volume and low dollar amount transactions. I would say a lot of brick and mortar businesses fall into this category.
While "part 1" of this series says "FTP" (implying plain-text/unencrypted data), "part 2" [0] and "part 3" [1] both say "SFTP". This is "more correct", in my experience, as encryption is pretty much always used nowadays.
[0]: http://engineering.gusto.com/how-ach-works-a-developer-persp...
[1]: http://engineering.gusto.com/how-ach-works-a-developer-persp...
[0]: http://engineering.gusto.com/how-ach-works-a-developer-persp...
[1]: http://engineering.gusto.com/how-ach-works-a-developer-persp...
The entire Australian energy market communicates with XML over FTP. Electricity meter data is CSV in XML over FTP.
Sometimes simple just works.
Sometimes simple just works.
CSV in XML over FTP?
That would look like
That would look like
<DATA>x,y,z</DATA>
FTP, not SFTP?
I'd increase the bill of my neighbours I don't like a bit.
The exciting thing about all these crypto currencies to me, is that banks could roll their own private "exchange" currency to do near real time transfers to any other bank / account.
> banks could roll their own private "exchange" currency to do near real time transfers to any other bank
If only US banks already had access to some sort of shared currency they could use for such transfers!
Snark aside, this just seems like a horrible use for a crypto currency. Block chains are an interesting tool to solve a lack of trust when you're willing to give up some speed and convenience. Banks have full trust in themselves, the central banks, and the clearing houses that make the financial system work; they just need to update their systems to be faster.
When your problem is that you have trust and need speed, a solution which sacrifices speed to work in situations where you don't have trust is probably not a good option.
If only US banks already had access to some sort of shared currency they could use for such transfers!
Snark aside, this just seems like a horrible use for a crypto currency. Block chains are an interesting tool to solve a lack of trust when you're willing to give up some speed and convenience. Banks have full trust in themselves, the central banks, and the clearing houses that make the financial system work; they just need to update their systems to be faster.
When your problem is that you have trust and need speed, a solution which sacrifices speed to work in situations where you don't have trust is probably not a good option.
No bank has full trust of any other entity. It's not black and white. They use ratings (either Standard and Poor's or others) to determine the risk involved in a liability. Even the U.S. Government has a certain amount of risk.
>>>No bank has full trust of any other entity. It's not black and white.
Banks basically have full trust in each other and central banks, actually (for these kind of transactions, not investments or whatever). If you're a bank and you fuck up to the degree where you don't properly handle ACH and other transactions, guess what happens: you're no longer a bank, the government swoops in and takes over, and everything gets sorted out and you go to jail.
It works pretty well.
Banks basically have full trust in each other and central banks, actually (for these kind of transactions, not investments or whatever). If you're a bank and you fuck up to the degree where you don't properly handle ACH and other transactions, guess what happens: you're no longer a bank, the government swoops in and takes over, and everything gets sorted out and you go to jail.
It works pretty well.
> No bank has full trust of any other entity. It's not black and white.
For interbank transfers, it is. That's simply how the financial system works.
> They use ratings (either Standard and Poor's or others) to determine the risk involved in a liability.
No. The reason why your $50 ACH transfer to your cousin takes so long to clear isn't that recipient bank is checking Standard and Poor's to see if Wachovia is good for it.
For interbank transfers, it is. That's simply how the financial system works.
> They use ratings (either Standard and Poor's or others) to determine the risk involved in a liability.
No. The reason why your $50 ACH transfer to your cousin takes so long to clear isn't that recipient bank is checking Standard and Poor's to see if Wachovia is good for it.
We're talking about whether transactions are legitimate and not the creditworthiness of the institutions.
Don't banks hold deposits at each others institutions?
I dont understand why people hype up cryptocurrency for this purpose. The whole reason for existence of cryptocurrency is a way to achieve consensus without having a real central authority.
What's wrong with having the bank as a central authority in their own banking system?
What's wrong with having the bank as a central authority in their own banking system?
I was referring to transfers between banks. I believe currently they go through the Federal Reserve in batches. A more distributed system could allow banks to transfer directly to/from each other in near real time.
A centralised system would also "allow banks to transfer directly to/from each other in near real time"
That wouldn't be "direct", it would be through a trusted third party... Who has banking hours, could censor the transaction, etc.
Fiat currency REQUIRES the trust of the central bank, so that's not an issue.
That shouldn't be an issue if all parties trust the Third party.
It is only an issue if there is sufficient lack of trust that the involved fear such a thing would happen.
Otherwise a block chain is overkill.
It is only an issue if there is sufficient lack of trust that the involved fear such a thing would happen.
Otherwise a block chain is overkill.
Ripple serves this exact purpose.
But why use banks at all then?
If I don't have to worry about the physical cash, there's no real reason for me to use a bank. They're a hanger on of a bygone era.
If I don't have to worry about the physical cash, there's no real reason for me to use a bank. They're a hanger on of a bygone era.
Because if you forget your bank password, you can reset it. If you forget your password to your private key, you’re SOL.
Yup, most of the modern business world works by FTPing CSV files all over the place. Usually there is security in there somewhere. Sometimes it's XML. JSON? Maybe in 2030 or so.
>At Gusto, we rely heavily on the ACH network. For example, when a company runs payroll, we’ll use the ACH network to debit the company’s account to fund their employee’s pay. Once we’ve received these funds from the company, we’ll again use the ACH network to initiate credits into each of the employee’s accounts to pay them for their hard work.
Can you use ACH to initiate a transfer between two (third) parties (i.e. you not being one of them)? If not, what are the requirements to be a broker / escrow in between them?
Can you use ACH to initiate a transfer between two (third) parties (i.e. you not being one of them)? If not, what are the requirements to be a broker / escrow in between them?
> Can you use ACH to initiate a transfer between two (third) parties (i.e. you not being one of them)?
No, the closest you could come is what you quoted. You could issue the debit & credit actions together, but you'd be taking on the risk that the debit fails and the credit succeeds, leaving you short.
> If not, what are the requirements to be a broker / escrow in between them?
The Federal Reserve is the entity that is between all inter-bank ACH transactions. Essentially US banks hold an account with the Federal Reserve. When they send ACH payments for their customers, their Fed account is debited (and the other bank is credited). When they receive ACH payments for their customers, their Fed account is credited (and the other bank is debited).
No, the closest you could come is what you quoted. You could issue the debit & credit actions together, but you'd be taking on the risk that the debit fails and the credit succeeds, leaving you short.
> If not, what are the requirements to be a broker / escrow in between them?
The Federal Reserve is the entity that is between all inter-bank ACH transactions. Essentially US banks hold an account with the Federal Reserve. When they send ACH payments for their customers, their Fed account is debited (and the other bank is credited). When they receive ACH payments for their customers, their Fed account is credited (and the other bank is debited).
Going from the UK to the US was like stepping back in time when it came to banking. I remember complaining about some aspects of UK banking - it's going to take a day for my transfer to complete!?! Now we have faster payments in the UK which complete in hours at most.
Meanwhile in the US I still had to pay my rent with a physical check because that was easier than figuring out the weird 'pay anyone' implementation my bank had.
Meanwhile in the US I still had to pay my rent with a physical check because that was easier than figuring out the weird 'pay anyone' implementation my bank had.
Your property management doesn't provide a way to pay online?
I see relative times (7:00pm, 12:01am, etc), but I notice there are no timezones. Is it implicitly Eastern time in the US?
Today my bank sent me a note saying that ACH payments for both credit and debit cards will only take 1 day to process.
They talk about this a little in part 3:
> An addendum to the ACH protocol to support same-day ACH settlement is something that is almost unanimously desired by the ACH community. The good news is that NACHA, the governing organization behind ACH, has recently announced plans to roll out a same-day ACH protocol. The challenge is coordinating the adoption of the new protocol across all participating banks. Once fully implemented, the same-day ACH system would likely cut one day from the timelines outlined here.
http://engineering.gusto.com/how-ach-works-a-developer-persp...
> An addendum to the ACH protocol to support same-day ACH settlement is something that is almost unanimously desired by the ACH community. The good news is that NACHA, the governing organization behind ACH, has recently announced plans to roll out a same-day ACH protocol. The challenge is coordinating the adoption of the new protocol across all participating banks. Once fully implemented, the same-day ACH system would likely cut one day from the timelines outlined here.
http://engineering.gusto.com/how-ach-works-a-developer-persp...
[deleted]
It is good to mention that it also a very similar system in Canada for EFT. I did implementation of that.
What does "similar" mean? Do all the major Canadian banks accept ACH files?
Oh god this is actually my job. Well, part of it.
I just moved 25% of my savings account into Ethereum yesterday... over ACH (DOH!)
Makes you pine for Bitcoin
Bitcoin takes just as long to settle and costs money per transaction. Also you can't reverse a fraudulent Bitcoin transaction.
Between the two ACH is clearly better.
Between the two ACH is clearly better.
This is why we need to provide better regulations for adapting better blockchain technologies in various industry sectors including this, banking. Also it's another away of sharing and saving infrastructure costs for bankings while providing better security than traditional banking systems like ACH. Perhaps ACH can be written in smart contract in safer way with security.
Totally more secure, less error prone, and faster than a blockchain. /s
At least payments to an invalid account number are trivially reversed, instead of disappearing into the ether...
It's still plenty archaic, but takes the headline's shock value down a small peg[1].
[0] It adds a mental pause -- a Secure ... FTP server. It hints that, possibly, it's a reference to a different aspect of the server's security (a non-technical person might refer to a server as being a "secure" server simply because it's protected by an ID and password, for instance).
[1] Based on my personal interaction with banks and software, as well as several friends who had previously been members of a few banks' IT departments, my first -- very sarcastic thought -- was "of course it works that way!"