njd · HackerTrans

9 tahun yang laludiscuss

I'm a retired data architect and C developer with 36 years of professional experience. In my opinion, this is not how you find an appropriate candidate. It would be better to ask substantive questions that are related to the thing you are making. What does this person bring to the team? How will the team receive this person? Has this person delivered something of substance in the recent past that would convince you that he/she would succeed on your project? How was this person an asset on their previous job?

njd

10 tahun yang laludiscuss

Yes, especially with early cleansing and transformation. When source semantics are dynamic, try to build recognizers rather than expect sources to obey some semantic agreement. They won't. Cast what you can recognize into an intermediate shape. I like object, property, value (i.e triple) with metadata. Don't cleanse or transform the source data. Let the data be the data. Cleansed and transformed data fall into the category of assertions. Assertions can be made by humans and software, keep metadata. Allow your applications and your analysts to overlay their own semantic meaning on the triples. Naturally, consensus understanding of source semantics is desirable, but don't want to prevent analysts from using the raw or asserted data as they see fit. Still need software to analyze triples for the bad, etc. to make assertions that data is suspect. Otherwise, your downstream programs and analysts will need to make such assertions. Given such a process, applications and analysts can write queries to ignore suspect data, undesirable assertions, etc. if they so choose.

njd

10 tahun yang laludiscuss

It is possible, but there are a lot of caveats.

For example, how do you detect when a source's semantics change? This will break any cleansing or transforms done in the stream platform. Until it gets fixed, data may be missing, wrong, or worse (e.g. corrupted) and propagated down stream.

When data is cleansed and transformed early, there is no way to go back to the raw data, unless you carry it forwards too.

Consider these sort of questions for your use case.

njd

10 tahun yang laludiscuss

Yes. But, the trie implementation would need to somehow allocate mmap file address space. Depending on supported size of the trie, may require multiple mmap files. The addressing scheme would need to consider that. If you want the trie distributed, how would you do that? Would you distribute by some function on the key, essentially resulting in N tries?

Also, if you want to support all 256 values in a byte for each byte of the string to be entered into the trie, the whole idea of using a list structure seems inefficient. On average, to find the next byte in the trie would require 128 comparison operations.

njd

10 tahun yang laludiscuss

Interesting. How would you use one of these implementations to persist a large trie beyond what physical memory can support?

njd

10 tahun yang laludiscuss

Thanks for pointing that out. Instead of the endless cascade, just cascade once off of the first link. That way, there would never be more than two references to resolve and you still get the temporal benefit.

> ln -s ../fluffy.jpg "fluffy [+cat].jpg"

> ln -s "fluffy [+cat].jpg" "fluffy [+yarn].jpg"

> ln -s "fluffy [+cat].jpg" "fluffy [-yarn].jpg"

njd

10 tahun yang laludiscuss

Instead of renaming the file, TagSpaces could create a symbolic link and preserve the original file name. All such links could be managed in a TagSpaces folder.

> ln -s ../fluffy.jpg "fluffy [cat].jpg"

Tags could be added by linking to the latest one:

> ln -s "fluffy [cat].jpg" "fluffy [cat] [yarn].jpg"

This would allow TagSpaces to have temporal knowledge as to when tags were added or removed.

If the links are cascaded as I have shown, then when the original file is moved, TagSpaces will only need to rebuild the first link. The rest will still work.

The issue of the tags causing the name to be too long can be addressed by a different tag naming strategy.

> ln -s ../fluffy.jpg "fluffy [+cat].jpg"

> ln -s "fluffy [+cat].jpg" "fluffy [+yarn].jpg"

> ln -s "fluffy [+yarn].jpg" "fluffy [-yarn].jpg"

This would require TagSpaces to have some simple logic to determine which tags were relevant at a particular point in time.

njd

10 tahun yang laludiscuss

I enjoyed reading your story about building a DBMS. In it, you mention the importance of performance. So, I'm curious as to why you chose 'go' versus 'C'. Was the choice of 'go' a convenience to get something done quicker, or no observable performance difference between 'go' and 'C' in a distributed environment, or some other reason? Thanks.

njd

10 tahun yang laludiscuss

- The challenge for an organization is to recognize that there is a significant difference between the 'data engineer' working on a vertical project and the 'data engineer' responsible for integrating data across the enterprise.

- The project 'data engineer', in today's world, most likely will be a software developer responsible for ETL, etc. The data design will be more or less up to the software developer.

- An enterprise 'data engineer' is more concerned with data that affects the enterprise. This typically involves some sort of data integration. For example, how to integrate relevant data from N projects (e.g. A,B,C .. Z) where each project has its own idea of how to represent similar concepts (e.g. person, user, customer), with different provenance, truth assertions, access rules, data retention periods, granularity of metadata (e.g. at the attribute level vs entity level), etc. The enterprise is interested in questions like 'What did we know and when did we know it?", etc. The enterprise 'data engineer' will probably levy requirements on the project 'data engineer' to meet the enterprise's needs.

njd

10 tahun yang laludiscuss

I'm developing a distributed temporal KV store in C. My motivation for developing the TKV is my interest in such questions as 'why' data changes and the effect it has on the 'system' in general. I chose to develop the TKV so that 'time' would be handled consistently regardless of source data model.

comments