This is classic Apache documentation. There are no figures in any of it. Okay, Hello, happy campers.
So I was going to try to do a presentation. On how to find out what's in the duke.
And I was doing it. And putting all these links in the slides and I realized that. Nobody care about reading slides about links.
No, paused for just a second, Steve. Okay. Just making a book end for the recording this is, just making a book end for the recording. This is, Thursday, January the eighteenth, 2024.
Our speaker tonight is Stephen Lumbark. This is the monthly meeting of the St. Louis Linux users.
Stephen is going to be talking to us tonight telling us about, Hey, Duke. And it's an introduction to Haiti.
And with that, sorry for the somewhat awkward beginning tonight. Usually we make a few an hour.
Q&A and some of our leadership here has been conducting an informal QA session here has been conducting an informal Q&A session here but now we're gonna turn it over to Stephen and a session here, but now we're gonna turn it over to Stephen and after Stephen's talk there'll be time for questions on his talk as well as other queue.
And, after Stephen's talk, there'll be time for questions on his talk as well as,
Okay.
Go, go, Stephen. Hello.
Okay, so the original slide that I started with was a line that said in the beginning there was to do and it was Google.
And that's true. I did. Show of hands that I can't see. How many people here or just scream out loud?
Have heard of the Duke and knew that Google created this thing that was for their their website and their their crawling system.
The whole gut to the original. Google ability to scrape the web clean was based on to do.
Which they no longer use, by the way. But how many people have heard of that much?
Yup.
We have hands going up. We got hands. I am going up. Okay, so.
The interesting thing is the basic idea of Hadoop. Was to have a a fully distributed system for processing all these links.
We're each node to talk about everything completely independently of all of the others. And that allowed them to schedule starting out all of you know every time they added a new node they just increased their power they didn't have to worry about balance thing or anything else they just kept coming data from more and more and more computers.
The basic idea of how that worked with a distributed file system. And several layers of processing on top of it.
Got taken up by a pack who wanted to duplicate it. The funny thing is around the time the fact they succeeded in duplicating it.
We will stop using it. So what happened next? Is sort of a new thing that created that.
It's an ecosystem of different people who came from academia, from LinkedIn.
From different companies that. That. Open source, little fix and pieces of things they were using. So that today.
A pretty common fact in, even bank. Is the HPFS system used for distributing.
All of this data among the different nodes. A product called Kafka. They came from LinkedIn.
It's used for. Turning input streams in the sequential record. I think, that designed for table scans and distributed lookup.
And a thing called, lot of African animals going on here. And in Paolo allows you to kind of empty well.
Just about anything. Which allows you to take these key store databases. And actually update them and insert them and, you know, manage them with SQL.
Once you got all the stuff running It's possible to manage because so many different layers and pieces. Of course, there's one more piece called New Hooper.
And the Zookeeper is designed to just. Well, that's because 2 people. And again, most of this is gonna be.
Because my slides are on different machines. Is where to find the documentation. And if you wanted to try using any of it or see what it looked like.
And I was also going to show people stuff on the O'reilly Learning Center who used to be the old Safari.
They've got a lot of books and a lot of clap. I are recordings of old classes and things that are, you know, in the last couple of years where they're still relevant.
They still knew how to apply these things. Oh, and the last thing was Ranger. Which is the theory tool.
And it allows you to do distributed security for all the people accessing all of these file systems at all of these computers.
Together. So, and as usual. The place to start with any of this is where? Come on, Bingo!
And there is there is an article on HDFS. The problem with And this is a consistent problem.
With all of the documents and for all of the packing. And I just asked the question HDFS the FS is file system the H you say do Yeah, distributed.
There you go. That was me my next word. HDFS, the original idea. Was he wanted to find a file system.
That you can see the file with a path. And then the the file would be started. So they an individual file would have components on a large number of nodes.
And you could find the file from a different note if you wanted to. What the You could, it was a lot faster to access things from the host.
Something was on. So it's the exact opposite of NFS. And if that's logging down because you got one server.
And it tries to make all of the hope. You to formally access the content on one server. And the caddy is the benefits that kills it.
Is trying to maintain that uniformity. HDFS throws that out the window and says, oh.
You're looking for this. It lives on that node. If you want it, I'll pull it over here.
And you can find things that ideally. You're gonna do most of your processing on the local node and find ways to start the data and start the processing.
So for example, if you store things in ATFS, around, you know, by day. And you do a round robin by date.
You try and process the deep that are on machine act using code running on.
But the HDFS was really And we the backbone of everything else, in the sense that it allows you.
Spread out the storage and it You know, it avoids the bottleneck. That a single NFS server and the and all the UDP overhead causes you.
Do that make sense to most people?
Hard to say a lot of people are stuck in what.
Yeah. Okay.
Okay, well. Backing up then originally Hadoop was created by Google. And it was a system for burning out computation.
And around the time Google decided they had a better way to do it and completely abandoned Hadoop internally.
The name got I, and they were trying to duplicate something like it. Again, the the thing the The Apache Hadoop.
Result is an ecosystem of different things that stick together. They do some really wonderful processing.
But have almost nothing to do with the original Google Mapreduce. What I'm showing you now is an ecosystem.
Of different applications that can be used or not used in all sorts of different layers. Are they original application?
Google was using it for map reduced when they're trying to take everybody's user data. And, Right.
Well, what they were doing is taking all the stuff they break off the web. And they wanted to create.
A lookup table that says, break this off the web, find everything new, find all the terms that are related to all the things and create for me a database that if you give me a list of words I'll show you where the what website to look in.
So originally for their search function. Yeah, okay. This was what created the database. For mapping.
For napping tokens on a website.
The original church. Okay. And so The name Hadoop. The basic idea of a distributed file system and feared nothing parallel processing.
Apache wanted to duplicate that is everyone thought it was the best thing for peanut butter.
And what happened was A, Google abandoned it. And B, people realize, no, it's not up there with with Christina.
Why? The basic idea, the thought. Of 5 feet parallel invest in parallel profits across the distributed file system with an arbitrary number of votes.
We're adding new homes. It was a major slowdown. For the the product that they things like NFS when you add a third number of host benefits close to a call.
So someone had to find a different way of storing this stuff.
And. That idea got labeled as the newcombe by a batty. And converted into a number of different things.
Not all of which are required. And utilize the Apache and do the ecosystem. But they fit together nicely.
And they each do. You know, in classic unique fact and you've got a whole bunch of things that each do one thing well.
And and fit together.
No, did that make sense? Starting to check somebody else ask a question here. I'm just trying to keep it.
You know, keep the, perspective on at this point. Sure. Yeah, it be quite, yeah, I can't see the question.
So if anyone sees a question, please shout it out. I'll answer it. In the Google environment.
Right. Resume that there are. Multiple. Computers.
Hosts. That are taking in. Questions. And then those multiple hosts have to have some common point to go, oh, that's not what we need for.
Okay. Google was used for the ingestion part of it. The ETL side. Oh, You go went out to the world and they dragged every website clean of content.
Okay. They then filtered out work like the if and but that are not searchable.
And. They wanted.
Hang on a second. Higher 30. They They had to find a way of taking all the new content on the internet.
They weren't going to rebuild their database every day, but they had to add all the new content and all the same content.
I think it's split into words. All the HPML stripped out of it. Script of articles of things that don't add to the searchability.
And the really third triple terms had to be stuffed up the database every single day. Does that make sense so far?
Okay. If you think of doing that on a monolithic computer, 20 years ago, where big heavy duty machines were sun, 6,400 with 4 cores.
You weren't gonna get this done on a thing. You had to use a large network of machines and you had to make it so that the entire idea would scale.
Why adding new machines without having to completely tear down the configuration. Or alternatively. To blow you down with overhead of all of the different.
Trying to talk to each other. And again, that was, you know, the biggest problem back then is NFS was the standard file system for distributed access.
And if you add more and more node to an NFS system. It it it crawls exponentially to a crawl.
Beyond a certain number is just adding one more host ad. Huge amount of overhead. And in my mind I've got a picture of a node.
Having every reference to the token. Well, what ended up happening, something like that. The idea behind that produces it was a two-state system.
And again, this is no longer used by. Cool. And you would take everything and you would.
Assign each, you would break the HTML input into individual token. And you would assign using some mapping process.
Each token.
To a node. You know, By nines on the word you put it onto a note. Whatever you do.
You try and find a balanced way of assigning each particular token gets processed on a node.
Then what you do is the reduced stage. Take the data and again reduct the idea of it. Is each of the things that I extract.
In the mapping stage can be reduced. With a shared nothing process. On the on the stage that loads it into the database, produces the loadable content.
And the idea there was that because it's feared nothing, if I add a new node, it doesn't interfere with the existing one.
One of the first things you have to do when you're trying that. You have to find a way to store all this stuff.
And you have to find a way to store all of it. Spread across 100 or thousands of computers.
In a way that you can get a file back without having to search for it. Node by node by note.
But at the same time. You don't want to have something like NFS with a central server that becomes a bottleneck to the whole process.
And so HDFS was created and it had the 2 layer file system. Where you talk to a server.
New Mac. From the vault, the, the volume is mounted across very many machines.
And then when you look for something, you go into an HCFS. Brought that that can tell basically whether or not what you're looking for.
Is on the local machine or has to be looked up across the network. One of the ideas is of the duke the ecosystem, the thing that came out of this.
Was the notion that it's easier to move process things through the data than bicycle. So.
One of the things that would happen again is you've got this distributed file system. The process is mapping.
Would take content. And map all the similar content that could be processed together onto one node of the.
A dupe file system. Where it would be available when I kicked off the process to scorp up the file system.
All the things running on any one node. Would pretty much tend to find whatever they needed on their local host.
And so most of it is local access. But if you request something and the path isn't on the local system.
The file system goes back to the data broker and it's finds it for you and it folds it in.
So you can access it as if it were one monolithic volume. But it's a hell of a lot faster to access the local part of it.
Did that make sense? Somebody else chime in here. Yeah. No, I mean.
Yeah, make sense to me. The thing I'm wondering is you say it's faster to access it on the local versus, you say it's faster to access it on the local versus the monolithic.
Right.
Virtual, theical monolithic, which makes sense. Are those the same access methods as in you always ask the local?
Yeah, it's a knock point. You're looking for you look at the map. You, it's a file system.
Right, hang on. What I was gonna ask.
You mounted it to V-node. There's a data inside of that instead of just being a physical device.
Hold on.
Hold on, let me.
You know, it's like NFS. There's a process. Underneath there. The brokers where the data is located.
Let me ask a question.
In the, most people seeing the model. You go in and it says, oh, you're looking to block X to file Y and it goes back to the NFS to her and says, please give me those blocks.
Hold on, Stephen. He's trying to, introduct a fine point to the question.
Okay.
One way audio. What I'm trying to determine is you ask the local node for a file.
Yeah.
If it's not cash locally and please forgive if I'm using the wrong terms. Does the local node proxy the request to where it is or does it do a redirect?
And you
Yes. You know, it's, what, there's, there, damn and running on this thing.
So there is one of the. How didn't want to go this deep into the details of it, but.
The basic idea is you've got one thing that the location broker. And you've got a very many things that are the data access.
And if you act for something. That is local. You get it very quickly. If you act for something that's remote.
Right.
You have to be moved on to your side and that takes longer.
Right.
So by the time you've done it's all local but asking for something to promote requires locating it.
Copying it from the remote thing over the network. And then making yours be a fixual copy of whatever it is.
Is handled by a demon. Or is that some demon?
Right, but that locating it and copying across the network. Is background as far as your As far as your user land application is concerned, something else does that for you.
As far as the application concern, black magic because you're looking at a mounted file system.
Yeah.
The trick is If I happen to know. Well, it is that we started that file system. Let's just see the trivial case.
Let's say we've got One key 6. Or towards them we got 27 servers.
And each of the servers is assigned a letter of the alphabet for every non-alchabetic symbol area.
And so what we do is we take all these tokens that of all these websites across all of the internet, we break them into tokens, we break them into words, and we take this first letter and we pick it on a
Right.
Well, if there's some that's designed the letter QI probably don't want to run the process of loading the letter M on it.
Right.
So the idea is if I know how the file system is sharded. I can do now if it turns out.
For some reason. There's a compound word that has to be loaded as 2 words. I'm not cut off!
From things that are stored on the remote system. So I don't have to build into my code, look for it here, else look for it there.
My code running on my machine looks into the V node on the file system and says, I want that.
And the file system either says, oh. Here is or oops, I gotta look at that. Leave it it back.
The central thing is I've got a distributed file system.
Which in be expanded to an enormous degree because each node has its own storage locally. And there's a data broker that knows where to find anything that isn't on the local system.
Does that much make sense?
Yes.
Okay. Once I have that file system, I can now, I can. Way that I can take my data in.
I can just distribute it among those file systems.
And if I know where I put things again, I then come back and I post-profit it.
Using the same mapping product and saying I'm doing everything. So again, in the case of Google.
They were breaking token, breaking the web content into token. And then the the data based entries for a series of token.
Would be massed under one machine with process. All the tokens that look like that.
And the file system started it the same way. So who worked with local?
Once you get, once you have the HDFS set up. The next step. You more get some data.
You know, obvious thought. So, an interesting thing happened at most of the time. The way people used to process this kind of thing.
If people are shipping you files and you get you know, Okay, network or multi gigabit, excuse me, network, seating your system, and people are seeing likely PDF thousands and thousands of thousands of dollars.
And the good old days, that network was flow enough that each time you found a file, you could have a database and you do an insert into database, arrival time, you know, size, whatever, whatever.
You had a list of all the files. You'd read the database and process. And the problem they've had at LinkedIn, people are clicking and clicking and clicking and clicking, clicking, and there is so much content going in that the databases are failing.
There's just no way to add data to the database without losing some of it in the pipeline.
So there was a product. Called contact. Which was named after. The author.
Because the guy liked the author when he took his English class from college. Okay. So, you know, it wasn't like a recursive acronym.
It wasn't. dingy.net outdoing. Sullivan was 4 letters.
It wasn't, you know, bouncing like a gazelle. It wasn't holding it like something.
Yeah.
It was, is a good author.
Hey, my watchdog system's called Fred. After my boss's dog, so.
Well, so, He does actually have a picture in it. And the basic idea of Kafka, this is a really neat thing.
This, if. If everybody takes nothing else from this. This this one tool will save you. From pipeline health.
And the basic idea of Kafka is you insert. Data into a pipe. And Kafka is very, very fast.
At acquiring new records.
And and putting them in quickly. Then what it does is it has a sequence counter. And they're 2 counter therapy.
Papu fax counter and there's the current data being retrieved, you know, the current and thirst encounter.
There's the in thirsting counter which is basically the top of the fact. And there is the next record that will be read from this queue.
Pointer. And the cute thing about that is that the The record is its atomic. You can have multiple processes.
Writing and reading from a single Kafka topic. And they will each get unique values. So the process is writing in in parallel, it is designed to very quickly suck things up.
And it puts a piece of data in a cat and it increments the pointer. And if I have 14 processes writing into it.
That atomic update of the pointer number is a record number that is currently being added. Is fast enough that you can absorb enormous amount the data.
At that point, the professors that are reading don't give a damn what the insertion pointer is.
They have a notion of a next record pointer. Which is to stop us back. And when you pull one thing out, you get the next one.
And you can query, there's some API for querying what the current pointer is and all the rest of it.
Obviously, because there's a finite amount of storage associated with this thing. If you if you inserted too many things at the top of the stack and they're not pulling them out or the bottom of the fact quickly enough.
You're going to write it through it. Okay. So people are putting stuff in. You've got all these processes pushing out.
And what they will do is they they can get back to the fact that we still need the next record.
And that that bottom end pointer will be returned and then increments. There are a few niceties that allow you to have a effectively it acted.
Configuration for it. Where when you pull the record, if you don't acknowledge getting it. The next person that act, we have that record.
So there's some. There's some nice things about making sure that you don't step on your feet.
When you're when you're processing this mode, data.
Octa then is. Oh, the problem. Of all of this click stream and all of this high bandwidth input from that thing that you know with today.
Because you can go through. And you can say, give me records as fast as I need. But going back to the picture that's on the screen here.
You notice the topic has producers and consumers. You can, because it's designed as a, as a network.
Each, another thing you can do with the topic, I forget, and that I apologize. A topic to be partitioned.
So you have producers going in. And they're putting data in and you can. Scarred these youths.
That the single topic has multiple parallel processes.
Serving part of that too. Now this is where the fan out and the parallelism comes in. Is I can step because of the way it works.
I can have caucus servers on multiple machines. Which each handle a portion of a cup.
So I can take the incoming data stream. And I can spread it among multiple machines. And have consumers that say show me the next available record in a queue and they will get it.
Ideally from the closest and. But I don't necessarily have to have only one. Processing the queue as the the new data records arrive.
Did that much make sense?
Mostly, yes.
Yeah. So, yeah, have a second, Steve. So your second then. Topics are kind of fixed or.
Topic what you end up doing with Let's say you've got, look at your bank.
And you're doing data processing for other banks. And each of these banks is going to give you, you know, X number of 1 million records.
Of transaction and accounting data every day. And you're missed in his warehouse for them so they can query it the same day and get all their customer analysis.
No, far so good. So each topic might be a different bank type of thing. Each topic would be a different data stream, would probably be a different data set.
But, but there would be, there was multiple topics for one bank. And then for that bank you'd have, well, what you have is.
It's the way that you have multiple topics. One bank might have 3 topics. They might have a business and a personal and a stock trade topic.
Okay. But as data gets written into that topic, it gets routed. Through the Kootenay cockpit server.
That is going to supply that portion. And again, the data gets sharded into different partitions of the topic.
So if I get a million records and I have 10 machines that are handling the Costco load. The input from that 1 million records will get spread across 10 machines.
Okay. And there are different ways to do it with hacking, with intervals, with round robin, whatever you want to do.
With these Kafka, the The nice thing about Kafka is looking at it from the outside.
You need a single input screen and you read what it looks to you like a single output stream. And the skirting is all blackm to come to the hood.
Hey Steve, this is.
Okay, thanks. And hey, Sheath. Yeah.
And.
This is Ed. I got a, tech. I got a technical question for you.
Hmm.
And you might have answered this, I had to step away for a second, but. So back to math reduce.
Okay. The way I understand it is that A map is an input stream. It could be a lot of records.
And what you're doing is you're mapping a function. A transform function that's the T and ETL.
Right.
Right across the stream. And every one of these functions has one input, one output. So. You know, item comes in, different item or transform comes out.
Then
Okay. I wouldn't quite finished it.
Well, again, if you, if you is this way? If you think in terms of .
The Oh, hold on. Ed wasn't quite finished yet.
Yeah, okay. So. So I use MAP all the time in my daily program, but. Okay, so, it makes perfect sense.
That, you can parallelize that, right? Because no one record, right, is dependent on any other record.
Right, that was the goal of it.
Okay. Right. Right. Exactly.
Okay. So when I use MAP and reduce in my own, my own programming, I have the output from MAP.
Which is another stream going in to reduce. And reduces a fold over that. Again, you've got this.
Usually a function that takes 2 inputs and returns one output. It's like the accumulator value and the individual item.
And the question I have about that is, let's say you parallelized this map over 10,000 machines, okay?
And they're across a distributed network. And I was watching listening to a podcast about this and they said, oh, there's this part you don't know about, which is called unshuffle.
Yeah.
Or fan in or something. Right? And so the question I have about that is these items, you know, the nodes could be spread across the network and a network could have indeterminate latencies, etc, etc, right?
So how does it know? To get them back in the same order that that map got them in in the first place.
Or doesn't that matter? Like.
It's basically you do bookkeeping with a sequence counter.
It keeps it with it.
Okay, so it's
You add a piece of data that's in order. To be perfectly honest. I have never disassembled Mapreduce to that level because I've never actually had to manage.
Yeah, I'm at. The way to do patted the way at Google had it set it up.
Yeah, cause it's.
But I'm never had to go through the. The step. Oh.
Right, but if it had if they had a little counter like I'm record one and I'm record 2,015, right?
You know, and, you know, in 2,015 came first.
Basically that you're doing something like that if you want to get them back out in the original order you have to record near the input order.
Okay.
Right and then you'd have to start it somehow Right? You'd have to sort it by those numbers.
You have to what? Yes, ultimately you have a source stuck in there, which is very expensive and the whole point of Mapreduce.
Yeah.
Was to avoid that sort. If you go back to things like Infra-matica and Adventistio.
The the original ETL model. Was you would sort everything and then do a merge. And that's how you found the new record.
And that sort was so the advantage to have an issue in Informatica. Was that you would distribute the sort over many machines.
And you'd let the orders in whatever. Take care of the fact that as more machines are included, the individual stores get faster to the point where it's more than it is because the distribution overhead is less than the saving.
Right.
Of this sort. The whole point of Mapreduce. Was really to Avoid.
And all caught. Coming to support anything. And that quiet was all one to one because then there was no the.
In the the output of the map was unordered. And it could be reduced in any order.
Oh, okay. So that was the part that I didn't understand. So basically the idea like say the plastic example with Google was you're trying to do a word count or word frequency count, right, from the HTML pages, right?
And you will say, okay, the word to do occurred 15 times in this page and, and over on this way over node over here it occurred 3 times on this page, etc. Etc.
By the time it gets to the reduced phase, right? You've got multiple. values coming in and they just get added together.
Like in a dictionary or something.
Yeah.
Hey, am I understanding that right? Like the reduce doesn't care. It doesn't, you know, it's always gonna see a dupe is one word.
Actually, you know what? Better.
They're, if you go to Wikipedia, there's an excellent description of Mapreduce.
Okay.
And it will be better than anything I can tell you at this point.
Okay, all right. Does.
Is it, when it's targeting, at some of the fine points of it? Like I said, I've never actually had to design a full, old, fact into Google, the DAOP, NATO stream with their verse of Mapreduce.
So I've never had to go through the notion of. The motion to deal with every layer of it.
Okay. Alright, well thank you.
But you need to go. The goal by getting it down to the point is independent was to avoid hurting at all costs because that's the expensive thing you can do.
And that was the idea of hacking it. If I can associate a node. In the map, the HDFS allows me to store everything locally.
I'm not cut off from values at other system. I can, I can query the entire system.
Bye, local beta can be processed very quickly on the local machine.
I'm hearing what you're saying and mentally processing it. And I'm considering what you just said as.
You can access. The data on any system. You can optimally ask or access the data. On the system.
No, let me finish, please.
No, that's what happens. Again, this is a mounted file system. It's an inode.
So from your point of view, you're looking at data. It turns out that. Unlike NFS.
Which attempts to make data access uniform across all modes. This says through it. Dated that is started locally to your system will be a hell of a lot faster.
But you're not cut off from data that did other notes.
Never mind.
Yeah. Leave it at that. If people want a real deep dive into Mapreduce, I'll come up with something later.
I was not asking about Mapreduce, but move on.
Are the distributed and the.
Yeah, it. The, the goal, the original goal of. Okay, of Google's Mapreduce, their idea.
Was to avoid. The
It was to annoy the story. And the overhead is serial loading. Was that was part of Abden A.
And informatic and all those things. At the time that were designed for loading databases.
So, going back through and rereading all of that stuff, I can't give you a better answer because it wasn't anything I memorized.
Like I said, I never manage one of those things. So I. I never had to know it at that level.
But Grant, you were saying this wasn't your point.
Right, I was.
Perhaps oversimplifying, but understanding that when data is written to HDFS, They're an affinity.
For certain types of data. So things that. You're doing that use that certain type of data.
It is an optimization for it to run on the node that has that data, but if it's running.
You would be storing what you do is you decide in advance that you think you want to store records that look like this.
On node x.
And then you process things.
On node X that you think are going to be local.
And by designing the algorithm for. Charting the data to match the algorithm for distributing the work.
You will tend to come up with.
And again, it. The details of the way that works. If people really care, I'll come up with something about it.
Forget I asked.
But I, I don't manage 8 BFS for a living.
So I can't go that much deeper into it because I've never memorized all this stuff.
Forget I asked. We're talking past each other. Move on.
Yeah.
Sorry, I'm just, well.
But, yeah, yeah, the ultimate point of HDFS is to distribute the file system along multiple nodes.
The one of the nice things about Kafka. All of these producers are pushing data in. I have rules in Kafka that sign the data.
Who a particular topic. The topic is a name that can be pulled out by a consumer. I can partition the topic.
Round robin, chunks, their various rules for like a content. So that I can have producers producing.
New random stuff and pushing it in. And consumers gets segregated data that is that they know what they're getting.
So for example, with banks, I might have like thanks producing data sets. All that stuff is the producers just see one consumer.
They push it into this port on their machine. But the The people reading it, the data set number one data set number 2 data set number 3.
Which can again be partitioned. So now what happens is I can have X number of machines accepting the data.
And that those X number of machine pushing it back out the other end. And I don't have to deal with all of the bookkeeping that it takes to shuffle the data.
For a given topic onto the correct partition onto the correct machine. And make sure that each of the consumers gets the next record that they're looking for.
That's all part of the Kafka API.
And the basic idea is that this. This piece of the, allows you to accept new data records.
At much, much higher rates. Than the old black file inserted into a database model.
So if you're looking at a situation where You're getting thousands of files committed and you can.
Yeah, profit in order. And you have to guarantee that if you think you've accepted it, you know, accepted it.
And if somebody says they're going to process it, you know they process it. It takes care of that It's not a database, but it gives you an, sort of field for.
For accepting the record and then pushing them out in the beam cost.
Did that make any sense?
Is anyone out there? No, I've put the oldest place. Everyone's thinking.
So we got this file system that distribute stuff. And we've got this BD input is mode that will distribute stuff.
Frequently using HDFS. And then what happens is these topics get distributed. The partition and the participants to purchase and reporting to the ATF.
So each of the brought it to the Kafka. Consumer. You know, the consumers can be designed, so they tend to consume data.
From the local HDFS. Again, it's easier to. Block those things where they're stored.
The entire file system can be accessed. But each participant is more quickly accessed on the third.
So again, it's designed for local storage, not for remote network storage. But it is.
The next thing you find Use a thing called
If we look up Apache to do, I apologize. Yup, block me duplicating the tab.
And.
This thing. Koodoo is a very, very high feed. Or database. You know, Berkeley, they may be.
On something more than steroids. One of the interesting things about it. You can. It's designed for OLAP.
So most relational databases. You were seeing at road. Kudu. Stores thing it's designed for.
Real time, the index update. And very, very fast record lookups. There is very limited indexing.
Basically you have to know what you're looking for type of thing. One important thing about kudu.
Is very, very different than OLAP databases. Designed to perform table again.
In a lot of cases when you're doing data warehousing. You're asking for, you know, all the records in the table at Matt Gack.
You're gonna have to pan it. There's no choice. There isn't a way to index it every way you can look it up.
So. The the kudu approach is that rather than paying the overhead of indexing everything on the way in.
You they designed something that is very very fact at scanning what there is. When a. He's Jordan K.
It is broken into what's called a tablet. Moses, decide with the yeah, the constipated database.
The trick with kudu is you can you have a tablet server. And each tablet through takes a portion of the total data.
A tablet server in an ideal world. Well, if you're dancing a query about one particular value.
One particular element that is stored in the values of the key value store. If you're really lucky, your question will be answered at a single tablet server and they do not need to communicate the between one another.
In order to aggregate through result, hand it back to you. Does that make sense for where I said it to anybody?
The statement made sense, but I'm trying to wrap my mind around that. Well, here's an example.
Let's just say. We have a kudu database. And one way to round robin the Kudu database so that the contents on each tablet server, let's say we've got.
32 third and I'm accepting millions and millions of rows a day. And I'd say, oh, I get a great way.
This is the way the textbooks tell you to do it. A, a terrible way, but.
And you, you had some by record number. You did round robin by record number, and computer number that Record number MoD number of servers and you assign it to that server in a circle.
And now let's say I do a query. And I'm querying inside of the data a particular account number.
Okay, I want to see the total amount. Let's try to collect the total count of record.
For each account number. Well, the count any one of the account numbers to get a, a of that.
I'm gonna have to, I'm gonna have to poke every single one of the tablet servers because for any one account it's probably got records for across all of them.
So now each tablet services will. I know that I don't have the complete answer. I will get my partial answer and I will forward that to a master.
Theer that integrates it and puts it back out the door. But that means there's a lot of pushing and integrating and merging at the top end in order to get the result.
If you're quite if you're Well, it's pretty easy because you know, like I can add them.
But if I'm doing something a little more complicated like. Show me the the total amount by accounts group by months that it was posted.
Again, I'm gonna have to take monks and groups and each different server is going to have a different part of the different months because they're all spread out and all these little partial results are going to go back up to the master server.
That's where everything is integrated and shipped back to the client. But there could be a lot of integrating and a lot of project.
Now let's say I know that this database is always accessed by account ID because that's its purpose.
What if I, My account ID. I haven't thought that. It's a bad life fought recently at a place that was working.
Well, now what happened? Let's say I ask you for the total on an account or a subtotal of account or a subtotal of account by day of the week or move phase or anything.
It's all going to be computed to finality on any one tablet server. Well now the master node.
When you got to see a griefin. All it does is back the results in order and hand them back to you.
It doesn't have to take pieces of things and recompute from partial total. Because every one of these knows that I have the authoritative total for this account.
I know that this entire account lives on my node. Here is your answer for this account.
Did that make sense the way I said it?
Yeah, if it happens to be the Server that has all the data. Yeah. Well, what will happen is if I've got toy if I got 20 server.
In this Tudor database and I've started to this table. By account number. And I select the total by account, you know, group by account number.
I'm going to each each node is going to get some collection of accounts. But once that note is done totaling those accounts.
On itself, it knows, wait. I can tell you for certain this is a final answer because I know that I could total this on using a column.
That it is sharded upon.
Does that make sense? Okay, I think so. Yeah, the trick is if you're dealing with these databases.
In these large OLAP systems. This is different than OLP. We're not talking about posteris in Oracle here.
If you're dealing with OLAP. You're frequently gonna get table, because you're asking, and questions about things.
There's too much data to index at all. So you're talking to a tablet, this is what 2 is designed to do.
And the second thing. Is you're gonna have to participate in that data and search it in parallel.
And again, that's for the HDFS file system comes in. Is the file system.
Is going to reside on a number of servers. And each the partitioning of the data when the data is stored.
Each record is going to be, we know where it all is.
But then when I start asking for it. The the tablet server, let's say it started by account, the tablets server starts asking for count X.
Is going to get all of the records off of its own file system. So it can be local access.
There's no network required there. It's going to do all it needs to using local storage.
And then, the result of the lift. Now if you have a join and you're joining between 2 tables.
And one of the tables is. Started by account and one of them is started by posting date. And you're going to have to merge data back at the master again.
Okay. Started by account file system. A different cable. Is gonna have is gonna maybe be broken down, you know, in different notes to be stored, not with every account.
And at that point, you're gonna have to get partial totals off of the tablet server that have to be integrated later on to get full total.
So the the lesson here is that. Distributed databases like these kudu is one Vizio is another.
It . If you start them in more or less the way you query them you will get excellent performance.
If you fly any part of the sharding so that it isn't one to one with the way you query it.
You will pay a huge cost. Now there may be times when you've got
A system that is sometimes pouring one way and sometimes query the other. And it may be used to occasionally it's queried for show me all the things that came in on this date.
Okay, your stock. That's not the way it started. You're gonna get par for the bill to accumulate.
Right, the amount of data that came in on one date. Might be a lot smaller. Then the amount of data.
For a single account when you're looking at, you know, let's say my account is, you know all of the postings from mobile oil or ex-on or a large corporate thing.
You know, you have millions of records. So. It makes more sense to live with the overhead.
Writing the query out by day, it's gonna query a small fraction of the total.
And the ones to query by account, this is gonna have to be tabled.
Right. Again, Kudu's got the neat thing. It's a. It's a, you know, they're column store stuff in it.
A bunch of different things. But the the thing to remember about kudu. Use it as a database, it was designed for incredibly fast data injection.
What are the 2 things about it? Is when you put, it's a write only database. You do not update a record in place.
So if you replace A/C value. What will happen is there's a cat in memory. And kudu always consults the in-memory cache first.
So if you get a whole bunch of new records. The new records are put into a A cat in memory.
And then an update file at the top of the table. And then the data in the table is we get thirds.
So I will always find the nearest. Record value. So let's say I've got, you know, a record that represents your, you know, your total income for the year.
And there's one copy of it. In the database somewhere. And I threw a new one. Well, now there are 2 values for the same thing.
But I know that the newer one is always going to be. On the factor story is to be in the cast.
For it to be and up in the lack. Collection of update, Asians. So I'm only, I'm always gonna get anything that is an update before I bother to look at the whole database and find any what potentially a record.
Then what happens with kudu is every so often you run a mark and suite process that takes the The updates and takes any stale data.
And market is out of use. And that eventually what you find is in almost all Cootie tables, all olap tables.
There is some time dimension. The kudu will will you'll be part that protecting by like the account ID and then you'll further park this in it, So all of the records that came in on any one day will be in any one shop of death.
And then what you say is if I'm keeping a 2 year window of data. And what I do is if this is if you eat to the partition that has got a separate file per day.
You just drop a file every so often, bang, go away. And eventually you'll find that you look at the kudu data.
And there'll be a partition. That has no data record in it. That haven't been marked as obsolete.
And at that point, that page is free to be reused in the storage. Or you'll drop it and say, okay, that.
That page of memory. Or that to be that page of gift. Is no longer relevant. Cause I don't want anything that old.
And then vanish from the queries and you keep going. So that's how you recycle the space.
But again, the. The real thing to remember about Kudu is that it's our attention.
And that it does table can very, very well. Lot of people I work with look at this. Oh my god, that's today with tables.
And you can't do that. That's not the point of 2. That the ploy of is doing the table.
So it turns a lot of the optimization. That you're used to doing in OLPP on its head.
Does that make any sense?
Hello? Is everybody else asleep? Come on, you hit the unmute button. Okay, well I did something useful you guys at least got a good map.
One thing. This has is comparison with other storage. And then, see, hang on, I think we've got somebody making noise.
Oh, good.
Bye.
I was gonna say it seems like it's Very much so pay it forward. So you get a less time.
Let me repeat the question. I'm not hearing. I mean, just before from the mic, I'm sorry.
So say that again so I can repeat it.
It seems like you're spending a little bit of overhead. To pay it forward. Kind of doing some of the processing upfront so you do less processing downstream every time you access it.
In a more economical manner.
Yeah. Well, the other thing is it. If you want to inject data very, very, very quickly.
If you look at most OLPT system, You can say update food, but you know, update.
At total equal to. Where account number equals y, right? That fact, we're gonna pull the page off the disk, write the new stuff and and push the page back down to death.
That means accessing a lot of old data. That means doing a lot of random lookups and potentially having to be the order of things.
If I think the leak from something where account number equals whatever. That involves a lot of page shuffling in order to optimize the storage.
With oil, with OLAP databases, the assumption is you're going to write. Huge, huge amount of data.
And rarely if ever. Update any of it. So most of these are based on an insert only storage mechanism.
And if I'm never going to erase anything, if I'm never going to read it up and change it and put it back.
I have to make sure that whatever you really need is always the most current person. By effectively looking at the database.
In in order of ingestion. So these are very similar. One of the. One of the designs for a relational database is a log structure database.
No, no. With the resounding, yeah. Okay. Well, so the idea here is Kafka is storing data as your writing stuff into it, Kafka has to cap it anyway.
So there's a very large, which is multiple gigabytes of in-memory cat.
For whatever is getting written. As things are written in the cast, they are immediately pushed onto the disc.
In a basically, you know, posted update. Storage.
So if I put let's say a new record that represents the total for your bank account. I'm not gonna go back and find the old record.
Or the old road, it is a table. The old row for your Thank record. I'm not going to go back and look at where it was.
I don't care. All I'm going to do is make sure that that knew it value.
Is forced into a place that. Costing cash or the in-memory patch. Where if you look for it by the candidate key.
You will find the newer value. At old value is sitting there, fat, dumb and happy. It doesn't know it's been updated.
I don't care. Cause I'm always gonna find the newest copy. Did that make sense the way I said it?
Yeah, and then that's why you need a clean up. Yeah, you need to run a periodic mark and sweep.
That sort of, you tidy things up. The normal way that you partition things in Olap Land because 90% of VoIP you store in OLAP, at least in the SAT, LIKE, the time date.
So for time or anything that sequence then. If it's time, if it's anything progressive.
You will partition by some kind of hack. Bye account number. By you know an MD 5 of the account number and MB 5 account of your name i don't care what it is plus you will additionally participate by either in guest and date.
Or pointing date. You do it by ingestion date, that's easy because I have today's table.
It was a bull with that is if I ever ever put the data. House of order, let's say there's a power problem and I put today's cotton stuff in and 2 days later I get.
Yes, yesterday's data and I put it in. The ingestion dates. Are not going to be monotonic.
With the as of date. Or the data. If anyone can picture that. So what happens then is if I try and purge my data and I expect.
That the oldest injection date is the oldest as of date. I'm going to be wrong. Because I might have loaded.
Today's data. Before I loaded yesterday's data because of a processing issue with the S of the data file.
So another way to do it is you protect in the data. By whatever account number, let's say, and additionally by posted by the as of date supposed to date.
The nice thing about that It's a bit more work. To in third because you might have been, you know, planning.
If I'm inserting things, I might ingest them in different orders, so I might not always be updating.
The Newton is the most recent participant. But in most cases I will be. But then it makes rolling the data off trivial.
Good. Because I have one participant that truly represents my oldest data. And it's fine, take all of the participants for all of you that represent data older than.
They all do go away and drop them and they manage.
And she, something, in the back of my mind. Yeah, given, let me say, I would say, I'd try the map. Friends, maybe I'll help.
I it seems like
What would we been talking about so far? Is really high volume data. But high volume. Yes.
Okay.
OLAP is on the order. Speed and wish to find old app for anybody that does or not.
Yeah, cool. There is online transaction processing, OLTP, an OLAP, online.
It Take a data warehouse. I forgot what it stands for. Somebody look it up.
Like I said, the OLAP database is with, you know, the our databases.
Did are not used for local transactions. They are used for warehousing. You's a huge amount of data.
They is loaded sequentially and frequently queried in parallel is frequently queried. By grill down.
In an OLTP environment if you're looking at your. Your account, your accounting industry, you know what it looks like.
You're going to have certain pattern. You know, you get an invoice, an invoice says line items, the line items reference the, the, you know, the product.
We've all seen you in the Parks, the Pliers database. The OLAP environments are a bit more fluid.
We walk back. Knowing that switch shop is cool.
I might have large numbers of records that don't have all the same number of fields in them. I might have data blob.
So what I got is a whole bunch of peas. And then a whole bunch of data blobs and I have to actually part the things when I load them.
Whereas in an OLPP, a transaction processing environment. You know, if you're booking flight.
You have to update the Not just the quantity on the plane, you can't recreate it. You have to say, oh, I sold that ticket.
Okay.
Look at but American Airlines have to say these are the seats on the plane. I just told that one thing that I have to know that that exact record got updated now and that it changed in the database.
Okay. Yes, and when I the question I'm acting a pretty well-determined question.
I'm not fondling the data to see who might have reserved. You know, see that we're more than a week old or something.
I've designed a database to support the queries that we need because they're preceptive queries that we need to make.
Now in an OLAP environment, that's more like, What might I know about people who bought Ford?
We're gonna need to.
I'll be on the back.
Let's then try to do political marketing. And let's say I want to say. I want to send these fliers out.
Sort of a office.
But I'm gonna give you
I'm doing basing in analysis. Is the political meanings of the people who bought. Boards showing the political leanings of the people who bought something else.
We speak.
Well, today I'm asking about Ford. Tomorrow I'm asking about parachutes. There's going to be a lot less of a pattern.
As
One
I'm dealing with another thing is order the magnitude more data. And another thing about oil TP versus OLAP is frequently OLAP data is time series.
Where are you? Managing a time theory database. That's relational is mapping. The closest thing to anything that worked for the I gave a talk about this a long time ago was both for it with its range type and its time range type.
Yes. So now I have this.
Why dealing with time ranges. In. In in relational land, Hmm. OLAP databases because they really are designed more for table scans.
Okay.
They are another important thing. You usually think of a database as pulling up a row of data. And if you look at Oracle, if you look at Oracle, if you look at Informex, Cellarxiv, that is the way the data is stored.
Okay.
So a page in portress will store adjacent column in adjacent. Does that make sense?
Yeah. Yeah. That has nothing to do with the way of a lot of OLAP work. In OLAP, you're more likely asking.
Don't be things that map these 3 J random values in these 3D random columns. You're fondling the data.
One
You're asking what would happen if I look for this in that column? A lot of them are column third.
Thank you. Okay. Sorry.
All in short. What you say is. I know that each time a new record is taken for loading into the database.
It's fields are broken into column in the database. And what the column for databases managed to do.
Is they store all the occurrences of each of those values in a column. Now that means if I have a A long if let's say I've got the state name and their state name or you know address and state names is a good one.
Or twice as secure.
Instead of storing the name Kala or Nia. 50 billion times.
A little bit of
I say I have California served in these records.
And I find the value of California and I find all of the records that contain it. That makes it very, very fat.
To find elect for me all the people who live in California. I've got the record number. What it's a hell of a lot more work to collect all the data about and still blow that you have to go to each individual column, look up each individual record, then you have to find where does you know if if you look up all the people that live in California that's easy.
He did kind of like a row number. If you look up all the data for Joe Blow, you have to say, oh, find me Joe Blow.
Oh, he's in the record. Oh. Well, now I've got him to draft.
How do I find his address? I have to find the record number. Well, now I'm searching randomly through the record number to find the record number that happens to have his treat on it.
If you see the difference. Yeah. So, OLAP databases, fuck it but added. At pulling up individual rows.
In detailed queries. They're enormously assisted. At the quickly scanning massive amount of data.
To come up with general information that's based on on inquiries about the data rather than access to individual people of their collection of data.
It's just a different focus. If you're trying to find all of the people in the theory who are likely to buy.
You know, a particular brand of cleanup fire. Bayesian analysis. They will show me all the people that buy.
You know, that bought that peanut butter, telling me their political affiliations, to me they're a direct, give me their hair color, I don't know.
Yeah, you look at all the things you know about the people that buy that peanut butter and now you say tell me all the other people who have that hairstyle on that model of car and live in the address.
And those are the people who haven't bought the peanut butter but are likely to because they look like people who already have.
And if you can see there, I'm not looking for individual records at that point. And they tell me all the people who got red hair.
Well, the column sort is ideally for that because it goes in to find the right hair to pull all their own numbers up in one.
It's a single accident to get all of the things that have any one attribute. Merging them back together into a collection of columns is a lot more work.
But that initial, that initial dive in to find me the things that looked like this. But you don't do that with a relational database.
Most relational databases are telling me the data for out 42. And I want to be all the data where I want to see some collection of column from the data, but I know what I'm looking for when I go into it.
Olap databases are hugely inefficient. At acquiring individual records as a, user to rose as a whole.
Depending on the database, it may not be sorted column. You may have things like kudos toward the key value parent.
Like the old BSC thing. So if you want to store. Inside of the key values. You make the key pretty beefy.
And the values are limited, for a lot of times they're And what you've got is a, is, is, is, is, is, is, is, is, is, is, is, is, is, is, is, is, And the actual query language for the contents of it is based on data.
The one thing that we've run out of time that I was going to show you is a thing called impala.
And Apollo is the important thing that makes Tudu usable. Because we've got working, you know, skipping from a fire hose.
We've got kudu absorbing all this stuff, once it's formatted. But if the living held a query, because he got this weird out date on format.
And you have to parse everything every time you did just a mess.
I know a little bit of metadata about what I put in there. Then I'll know that all the JSON has this collective field in it.
Well, now I can look at that as if it were a relational database.
And what Impala does is it takes some metadata. About what's in a coup or this dozen of stories.
I'm just thinking 2 noses. I've got talks about it. In power, we'll talk through lots of different flat file.
And it will take That flat file format and it will say
You want to say collect. Hair color from user info where, you know, peat up butter, brandy, and giffy.
It converts that into the data access. On the JSON, forps it out for you and handed back to you.
Good call. So you looking for the outside into Impala, you see squeal and you see columns and rows and you get your data back.
As a collection of rows. Having column. And, and, and, and, and, and you can do join.
On these key door values, you can join into the content. And it will convert it into the right set of operators that scan the thing, pull up what you need from the JSON, do the comparisons and hand you back what looks as if they were stored as column their own.
But you can get a relational model. Or quote.
On top of this key store value that will absorb the data at the rate you're asking it to.
Sounds like asking me nightmare to be shut up.
Yeah, it's not that bad. There are standardized format for all of this. I mean, you The stupid thing people do is they they will do things like They will have a kudu table.
Because again, Koot is key value or you can put anything to it. Oh, you know what?
Let's put it all in one big table. Well, know what it is, do we get it out?
Well, for foreign tell because you can't find the records you need. So you have to be careful.
Designing it. So that any one kudu table. Is storing values that have consistent metadata.
Is that, If you're willing to take that step and not everyone is. Good morning. Then impala is really just walking down based on pulling out field.
Which is very quick.
So it's. The difference is if I tried to store this as E value. Using something like Postgres or Oracle or anything.
You can't. You can't show data into a relational system.
That off to make it work.
It just, you can't, you can't break, you can't convert things. Into column and road structures and update indices and do all the rest of the page management that's required.
It keeps things nice and eaten orderly and relational.
And again, OLTP, the Postgres. Oracle, Ingrid. So, you know, all that family of relational databases.
They excel at accessing single record. And and queries related to joint thought record. That things like kudu performed badly on and you have to be careful.
Really careful. Start the data in the way that it's queries. If you knew that In your life, would be pretty good.
You can do amazing things with it. That term you using shard. Breaking it apart into little pieces to that the different pieces are on different for evening.
Here's another word for it. I like taking the 20 calms of the table and turning it 20 times on 20 different machine.
Right. That's why I get starting would be if I want to take all of my accounting data And I want to start it by account number.
So each And else I started. Bye a short spectrum, you know, a form. An integer check them on the account number.
Whatever the account number is, I get a number from one to 20. And that is me, you know, everything in.
That range goes on to this point, everything goes up. So you. A common thing to do is you'll compute a Okay, or a hat.
From the value. Then you'll take that integer modulo the number of machines, and that's the machine that that act goes on to.
And everything that happens to that value goes on that particular machine. That If you're starting is the second.
That's good. You, you will tend to get excellent performance database
Now, are these typically done in the cloud because you've got need so much flexibility with hardware and resources are Is this something you could put in Iraq with, you know, 16 decent servers in the back room?
Yeah. I mean, the machines that are made for this, Generally, you, you distribute the stuff by rack.
I'm not going to do it all. If people care about any part of what I talked about tonight.
Let's do Yeah.
I've blown the hour and very soon my wife is going to do bad things to me if we haven't eaten there yet.
Well, I, I would like to survive the night. So, yeah. This is if anyone is interested in what NATO warehousing looks like today.
Starting and that's why I tried to do this by going into Wikipedia and showing you these things.
Start by looking up Apache. On Wikipedia. The other place we didn't get a chance to look at it in, how many people here have subscription to the O'reilly learning?
FARI books online
I do.
I, too rich for my blood.
I think standard. Lucky I got one of the original subscriptions to it. And the key thing was it wasn't doing very well.
And then Riley bought it and, and we branded it slightly. And when they did that, they said anyone who gets an unlimited subscription.
All of it. And back then it was 170 something bucks the year, which was indeed.
They will never raise the price.
And now that unlimited thing is like, 600 bucks.
Why you can get a free account on O'reilly. We shall give you access to digital copies of 10 books.
And where they get you, you know, it's like the guy that you see the first toque and then maybe you want to do just a little bit of stuff.
You can highlight 10 books, but then your eleventh book, you get a ditch one of the books you highlighted.
No! And we buy the description and you go from 20 books to 50 books to whatever. They have really good online classes.
They have recordings of old classes. They have. Some of these elections, some of them there's really good lectures.
It's easy. It's a guy who wrote SQL. We said that we shouldn't use it.
We should use quell. He was the original person who implemented a working relational database as maybe 2 at IBM.
And there's a really neat lecture. On, well and relational theory. Where he shows currently how screwed up SQL is.
But partly and an important part of it is he fill it. Rule that if you follow them They're not difficult.
But with a few mathematical identities, you can produce SQL that is mathematically, provably correct.
And this boy, there's a book, it's too well in relation to theory. And the Ds are full of examples of SQL that look perfectly reasonable and is total.
It is garment.
So it's a great lecture. The only problem you got is CJ D. Is a mathematician.
You write like a math professor. He talked like a math professor. He looks like a math professor. So I was just having a capinated beverage for whatever you sort nearby when you started.
But it is very, very, very much worth completing if you deal with SQL in general.
With Safari books online, there's a great series of books from O'reilly called Up and Running.
Oh, it's up and running, Kafka up and running, whatever. And they show you enough of how this thing works and enough of how to tune it and enough of how to configure it and enough the pattern when stole it and enough and had a room you can use it to get the back out of it.
That you understand how to make it. Run well enough to use it. It's not. The deep dive 800 page every detail of everything everywhere.
It aims to give you a good feeling for how something ought to work and what it does and it's for both.
And the places where you'll stop your time. All right, Terry, the books. And there are for each of these topics you will find highs the zookeeper.
There are animal books and packet publishing has night ones. And good lecture, PhD KP publishing.
That night stuff.
For all of these topics. Well, it's relevant because it's what everyone's doing.
And if you go, if you want to work in IT today and if you want to make money. There are lots of people who don't really understand how to implement it.
And if you take a little bit of time to actually understand what sharding is and why it works and how it's used.
And how to match queries to shards and things like that. It's not lack of science.
But it's, good day.
So that is the kind of sort of lecture, a dive from the fact that. You wanna fly.
IN. Oh, by the way, could you, Yeah, I think so. It. It might help if you send us the slides, we can post them.
And then, if I, I never created them. Oh, I thought you said your other one time I was supposed to spend creating them I was banging my head out trying to get the, well, there was an intro slide that got you into starting into where Wikipedia and all the rest of the stuff was and what the click.
Okay, but I realized all I was doing was making slides of Wikipedia and showing the stuff and making slides of the Apache documentation.
Apache projects are manual, monumentally badly documented. All they say is this is what is it's like you know code was absolutely no comments they don't show you why they don't see the foibles they don't show example all it is is Fairly fuzzy, this is what it is.
If you want to know how to use any of this goats and books on law and go to O'reilly learning, packet publishing, wherever your favorite place is and there will be good books on that.
Cool. I guess so. How about a call for questions, I guess?
Anybody else got some, Final follow up questions here, receiving.
So where does systems like Let us show up in this. Nice.
Did you guys Steve? No. Where does this, where do systems like Redis show up in this, Right.
I think red is 5.
Hockey's time. I look it up, but I believe redness is a column. For a database that is used for high volume retrieval.
Yeah, you see it used an awful lot in like caching.
Alright. Alright, I use it for our salary workers too. To and coordinate what tasks get ran by.
And I was looking into.
Using it also for said workers to cache.
Credentials and other sensitive items but
All I know is it's really fast and I told me to put a really long password.
Yeah. Okay.
And it also comes in a nice container version that I've seen used on some of the applications we put out in production.
Yeah, very quick, very.
Yeah, one of the things we find with that,
Okay.
When you're using containerized, then you've got one extra layer. Of virtual, layer.
And between the container and the physical storage.
And if you try to contain a ride, a lot of these overlap prompt. The the virtual network ports and the, but all that virtual.
As a layer of processing on top of it. Yeah, it becomes very expensive. It's the same reason that mainframes are still manufactured by IBM.
Mainframes are enormously factor. Than Linux than any type of. Yeah, virtual memory. Is the period end of story.
Because they have direct memory access, they have directions, access to everything. You ran one job on the thing.
Even an LPAR is basically a physical So you think about every everyone says, oh, memory is faster than it is.
Yeah, but memory is slow as held for every time you look up to a virtual lookup, you'd have to do a page map and you have to go into the page map and find the page map for the physical map.
They have to take, There's all this layers and crap. It don't even happen on a mainframe because it's the Derks and Poo.
Get it.
So mainframe chips use fundamentally different types. Of tackling and plan and and brand the internal branding logic.
Then the ones that are designed from the 3, the 3, family in later. They were designed to support Munich and virtual memory.
Both to remember the wonderful thing. By if It runs at less than half the speed of memory access. And I'm entry.
Good point. And that there is a great book and I'll get the name of it.
It's for architects of these systems. And it started to look into the physical layer. I'm one of the last people I know that actually understands this machinery down to the hardware.
And I could look at things like these. There we say, oh, just put it in Docker.
No, not if you want performance. That virtual, that I can still be where the virtual, and eats the weight of your performance.
Sure. It's easier to manage. But there's nothing you can't solve with an extra layer of abstraction if you have the time.
What happens in a lot of these situation is here. You're tipping from a fire hose if you're reading all of the data that your clients are pushing into you, you can't affect that volume.
It is what it is. And you can't, your FLA says you can't drop record.
You have to have law data in yet in performance that is at least equal to your input data.
Got it.
Alright.
We still get a bunch of it.
Oh, when you have I saw like that you have to the budget for many frames.
Yeah, did you hear that Steve? Hmm. You be seeing SLAs like that take the budget of a mainframe.
Well, and that's the point.
The the mainframe. That's that's why ticketing and fire it calculations a lot of our stuff is still got our mainframes and it's 9 30 guys my time.
I've got to get out of here. Steve, we really, if we were in a bar I would go to midnight with my wife, get hungry and kill me very shortly.
She's a doctor. She knows from my jugular head. All right, folks, Steve, we love it.
Sorry for the production problems. Yeah, no, that's my thing for just zoom not being happy, but Hopefully by the next time we do this, always figured out how to use Zoom.
I like there's actually the problem was that there was a an accident. No, well,
Oh no, the wife is struck.
Go to dinner.
Okay. Go to dinner, Steve, before the wife gets mad at us. Yeah, well, you're the next people should come after anyway.
Have a wonderful evening. Thanks for all the camaraderie and I will hit you guys later.
Thank you.
They will. Thank you very much. Steve Lombard, the principal at Warcars Computing.
Can I see you? And the secondary. Have a good one.
Alright, we're off. Alright, thank you all very much for coming this evening. Once again, we apologize for the technical difficulty at the start.
That our speaker kind of had to step around. To recover doing his presentation here this evening. So. Anybody have any, Either comments or questions related to tonight's talk, though we don't have our speech speaker anymore.
And if not, you have any general Q&A that you just want to chat here amongst the group before we call it an even
I'll throw out there. What Steve was talking about is very much so and in demand. Line of computing and services.
But I feel like it's very similar to the web scale problem. Of It's orders of magnitude.
Very true.
Fewer people need that type of service. Then need. Basic databases. I don't need a tractor trailer.
To haul 3 bags of gravel.
I don't need an ingest system. Okay, the temperature. Off of my scattered around my house.
He's very
I just, I just, I just needed not to kill the flash on my raspberry pi.
Right.
Yeah. Okay.
So the example I use is big data and big data solutions are the systems that you use when all of the other things can't handle it.
When it satisfies those big V, the 5 V's of big data. And basically you know you have big data when You're having problems with like say your SQL database keeping up and you can't chart it in scale it up anymore and it's a hard bird.
I'm guessing that's. Between one and a hundred one in a thousand.
Yeah. No matter what management says.
I still have to figure out how to.
My Django project. Hi, available.
A royal pain. I got a question. Okay.
Hi. To me is one of those things that it's. If you need it, yes, great, but the level of complex you go through from Basically a single point of failure to not having a single point of failure.
Is very high.
What's
Yeah, I'm thinking it's I'm thinking with mine, let's see if I can get it into a Kubernetes where I'll automatically recover.
Don't be happy. Okay, question for you guys though.
Well, in.
Gotcha.
External SSL certificates. With. Sure, that's incorrect. Where do you guys use to manage those or is that?
Just manually going through the process.
Well, Ludson Crip is pretty straightforward. Run, with the command line initially and then.
You run it once a month to check for renewal.
That's one way. Cert bought from Let's Encrypt is one of Multiple dozen clients.
For let's encrypt. I'm using the acme.sh.
Client or let's encrypt. And I have mine.
Polling bi-weekly, I think, but I manually do a piece because I haven't.
Too lazy to do something. What's . Thanks
Well, what I'm trying to do is I have a whole bunch of services and stuff sitting around my.
Internal network and when I rebuilt my router I everything over to a domain that I own.
So I was thinking, okay, let's get a wildcard certificate and start shoving it into places like.
Yeah.
This is gonna be a real pain to do every 3 months. I was like, if there was some way I can say, okay, here's the admin password to the router and for the as and for these applications.
Good.
We, and then distribute it.
Yeah, I see. An issue with wild card CERT. Actually multiple, but the biggest one I see.
Is it's usually a singular item. That is security sensitive. That is distributed, read, duplicated.
2 multiple devices. So if any one of those devices is compromised, the certificate on everything. Is now compromised.
Yes.
The and you have touched on with Let's encrypt and let's encrypt.
Does the 90 days. There are other certificate authorities that use the ACME protocol. That do a different number of days.
The next issue I run into is if you don't. Do a wild card. You're doing specific host slash service names.
Now those names are in a. Sir certificate transparency report. Thus internal names.
Are explicitly leaked. Contrary to security best practice, but I understand completely what certificate transparency reports are for.
But it makes using things like Let's Encrypt. Problematic for a hostname you want to keep private.
Even if you can't get to it. It's name is published.
One of the other problems I have particularly with wildcards is goes back to alpaca. Attack multiple years ago.
Wherein if I use the same certificate. For SMTP and HTTP and pop 3.
Somebody could theoretically or not quite so theoretically. Interfere as in monkey in the middle network traffic that is going to port.
9 9 5. And redirect it. 2 4 4 3. And depending on which protocols are in play.
There can you can end up with some weird interactions and vulnerabilities like how does an HTTP server respond if you give it pop 3 commands or IMAP commands or SMTP commands or pick some other combination.
Like, how does an SMTB server respond if you give it HTTP commands, etc?
When you're reusing certificates. Alpaca. Becomes a potential thing. Maybe you don't care.
But it is something to be aware of.
What I have done for what you're doing I have a private Ca in my house and I have my route Ca.
Installed on my devices.
Alright.
But you, you really don't need it at the house. You needed more importantly when you're exchanging data with somebody outside.
Right.
I want it to be able to access. The drag on my server or the lom on my.
Sun server or talk to my printer or talk to the web interface to backup PC or when I talked by Vmware server.
I want traffic to be encrypted. Because let's be honest, we are. In and partially passed as in the ship has sailed.
Or opportunistic encryption as in if we can encrypt it it should be encrypted while on the wire.
So. So yeah, I have TLS encryption. From the office on my second floor to.
System in my basement over a wired network.
Okay.
Mostly because I play with things. I wanna practice what I preach. And I have real heartburn.
With both rotating certificates. Frequently, which lets encrypt once. And I have even more heartburn.
3 alarm fire quite heart burn. With my internal server names. Published. The other thing that I can do with an internal Ca that I cannot do with let's encrypt.
Is I can get a certificate for private IPs. You cannot as do that with global certificate authorities as in the certificate authority slash browser forum expressly forbids.
Providing TLS certificates for things like example. Dot netcom org or whatever not example or RFC.
Yeah.
19 IP addresses. I can easily do that. With my perfect, personal private Ca.
Interesting. Really? And
Huh? Name drop somebody that you might recognize. There is a TLS Mastery book.
I tech reviewed it and the author is Michael W. Lucas.
Oh, yeah, he's an old friend.
Yeah.
Let's see.
He does not like private CAs for scalability. Reasons. Mostly open SSL comes with a Ca.
He refers to it and other people do as well. Proof of concept. Kind of a wrapper that leaves a lot to be desired.
But I can do everything I want to do. With the Open SSL command. There are many, many, many modules.
Sub-commands as part of Open SSL. It is amazing. About the only thing I have found that is more complex and Orders magnitude more annoying to use and this might get some grounds from the room.
Mega Ca line.
I can tell from the lack of groans nobody has had the displeasure.
No, not really.
I have never used a tool as obtuse. As mega Ca line. Let's just say that I would rather type XML or LDAP queries than Mega CLI.
Well, what's it useful?
Is that for like a bias control
Yes, it is a command line utility to interface with monitor, manage, configure. Mega raid.
Cards, different versions of it. Over time. Newer ones are worse than older ones because more features been added.
Yes.
Oh, you mean, oh, you mean LSI tool? Oh, okay. Well, yeah, I've been using it for 20 years.
Not very often.
I'm sorry. Yeah. Okay.
Got the recovery time is so great
You say that, but I worked on a system earlier today that I had to go through a process to recover.
Access to the web interface. And the. 2 hot spare discs were configured as dedicated to one of 2 pools.
And a disc in the other pool had died. Hotspur hadn't kicked in because it was dedicated.
I had to reconfigure and it we are probably 8 to 10 h into an estimated 25 h.
Rebuild and it is 24 TB drives.
Wow. Yeah.
At work. I getting a little off topic from the question, but I. Toated a comment on mastodon earlier today.
Tell me your, Nick's veteran without telling me or Nick's veteran. I'll go first.
Hi, I tracked down. Web interface. Could not access the sub web interface.
For the raid card. Was getting a weird error with no V and C. Which is an HTML 5 to VNC gateway.
The VNC was in turn talking to XV and C. Which is a V and C to X 11 gateway for a virtual display server.
For a fat application, a fat x 11. GUI application. To manage the raid card.
That application was failing. Because and different and that applications written in Java. Because a different Java application living on local host had stopped running.
Had to get into the box and peel layers off the onion. And sniff local host as the first application tried to talk to it.
Find where it was failing, look on another unit to see what's listening on the port that nothing was listening on.
Back trace through and it scripts to find what starts once I identify the process name. Find what starts that process through like a chain of 3 sets of in its scripts.
Go back to the first system, find it. Run the in its script status. And serves his stop.
Try starting it. Try running status. Hey, it's there. Go all the way back, put the layers back on the onion and try to bring up the web page.
Hey, your raids critical. You should do something about this. I know that's why I wanted to get into this page.
Yeah.
So tell me if you're a veteran Nix admin without telling me you're a veteran Nick's admin.
Sorry, TLS. Did anybody else have? Comments.
Or questions.
Oh, I'll do just an announcement in terms of, filler and you all can think of more questions or announce error.
Comments. The announcement is, is the time of year where we let people to our board of directors.
The election is next month. It basically goes on. For all of the meetings, all of our meetings next month.
And so, Stan probably knows the exact date to the last meeting. It's probably the third or fourth week of February.
Maybe to the steer count meeting at the end of February anyway whatever anyone who attended 2 of our meetings and that means the general meeting.
The Linux log meeting, St. Louis log meeting, the, new users. Log meeting or slack.
Hmm.
The St. Louis area computer club. Or the steering committee or I'm there's probably something else that calls into that category but if you return if you were at 2 meetings in the last calendar year you could vote nor election.
And so there's at least 2 seats on the board of directors. I don't recall off the top of my head who the current occupants are.
But, if you wanna make sure your name doesn't appear on the ballot, submit $20 to Stan who is the one drawing up the rivers into the ballot.
And that's your only hope of not getting elected. With that announcement, are there any more comments questions?
Oh!
This is Stan. I'd, I'd like to point out that the zoom. Space to text.
That creates the closed captioning. At this point is worse than I've ever seen it before.
Cool.
It seems like the last 2 or 3 meetings that we've done with Zoom is gotten steadily worse.
And it is really bad. So. If you were somebody that can't hear worth a darn, Close captioning is not gonna really help you that much.
Hello.
Zoom sucks.
Really sorry to hear that. Thank you for finding it out.
All.
Oh.
Yeah, but at least at least it's the sucky tool that we know.
Yes, as far as accounting the balance, Okay, we'll count anything at the last minute and we count them at the Steering committee meeting in March.
So right up to that last minute if you sit. That's why we have that you put the date that you do your ballot.
Also, if you want to do a secret ballot, you have to print it out and mail it to us in time to get there.
So we can count it. Otherwise, you could submit it electronically and we could figure out who it came from.
But not that we really care. You just like to have people vote.
All toss out there. I had sent a message to discuss asking people what they were interested in knowing.
Yeah.
About encrypted email. I have lots of opinions and I'm happy to talk about things if people want to not necessarily here and now but basically Send your questions to me, please.
Thank you.
This is Stan. I didn't know you could encrypt mail.
Oh yes, it's a royal pain in your end.
It can be. It can also be so simple I spend. 30 min on it a year. And do it on my phone, on my tablet on my computer, and multiple other places.
Well, when somebody else designs the system.
Yeah, Yeah, I know about GPG or. Yeah, PGP, mail or whatever, but never really.
Thought that anybody Yeah, could use it efficiently.
It. The way.
Maybe one on one it might be useful, but I mean in large groups. Could it be used? Question mark.
Our entire company uses it.
Okay.
They're way doing groups.
But I use it one on one. Or one on few. And it duplicates encryption.
Interesting.
Real 30 s type thing is. An ephemeral key is generated and used to encrypt.
And each recipient. That is participating. That ephemeral key is encrypted with their public key.
So they can decrypt the ephemeral and use the ephemeral to decrypt the message.
Very high level. But you can send it to multiple people encrypted.
10 min warning.
Okay. 10 min, your clock's off, I show 49.
Not 50.
We don't have to quit at 9 o'clock, but by tradition we announced it. It's getting close to 9 o'clock because It's getting close to my bedtime.
Shoot.
Alright, question for you guys. You wanna review one of. Me, my build of. Yeah, the harvester system that I'm working on.
Sure. Yeah. Yeah, sure. Yeah.
Let's say, I don't think it's ready for our prime time, but it's it.
Good is from what I've heard from other people is seems to be at least decent enough for home.
So what what exactly is the purpose?
Harvesters, basically a She say controlled. Hi, hypervisor for All 3. Kubernetes.
Okay.
Well, the whole goal or was is that rancher is the management portal, whereas harvesters the code sitting on the per metal.
Oh, okay.
So instead of using rancher with cloud provider or Vmware. User with harvester.
Especially since you know Vmware got consumed by evil incarnate chipmaker.
Yeah, I can't believe that happened.
Like, people are gonna be jumping ship.
Thank you.
Have it.
Yeah, so many things about Vmware last I looked management on top of base hypervisor. There's so many other things in that ecosystem.
I don't know how many other people play at that level.
Bear hypervisor? Absolutely. There's a whole bunch of options.
Hi, I figured since I'm gonna jump ship off of Vmware. My home lab, I might as well try this one out.
Plus I have some. Interesting workloads that I will be probably Trying to
How well does the console Interface to the virtual machines work. That was one of the main things.
Vmware remote console. One of the main things that pushed me to 3 ESXI.
Well. Okay.
And Harvester? I don't know. Have even plugged in the systems you had to But if you guys wanna talk, then I will.
Pick notes as I deploy.
Yeah.
Sure. Well, Grant, I can say I've been using BERT Manager for 10 years.
And it works great connect with SSH and give you a spice console.
Yeah, I used Virt Manager. That was my preferred for a long time. And then. Early, 2020.
Ended up working from home with my last employer. And had a MacBook. And found that VMRC is.
On the platforms I needed. Linux Mac, Windows. And I've not seen, manager. Outside of Linux or maybe mixes in general.
And I also do a healthy amount with Virtualbox. But it's the interacting with the council.
For each of the guests is different in KVM is different in virtual box is different in Vmware.
And
Yeah. Yeah, in fact, that's one of the questions I was gonna ask. I have it.
I was looking for it. I couldn't find a way to do snapshot with work manager.
Have you ever seen one?
I want to say the last time I looked at it. And I don't like hot snapshots.
I prefer to shut down the machine. So at that point, I could either do an LVM snapshot if I was using an LV as the backing for the virtual disc.
Or I could copy the file or ZFSE. I could do file system level copies.
But yeah, and that's kind of what I'm talking about of some of the integrated features that Vmware provides.
That I have not seen as good. In other hypervisors. Or like virtual desktop infrastructure or you can go to a web interface.
Through HTML 5 and it does translation to connect to the back end VM. Or management automatic moving across nodes in a cluster.
Those are more. Big business enterprise features that are. Very much in demand that I find Maybe it's my ignorance, but I find very lacking.
In the other hypervisors I've looked at.
Yeah.
I know Virtualbox. Supports snapshots cold and hot. And you can actually run Virtualbox.
I don't want to say headless but more as a service on a system. And you can connect to it remotely.
I think you can get the Bebox as a fat client to talk to it. There's also PHP Virtual Box.
I had Reasonable success with that, but it was. VM, VNC.
To the council basically, which is back to what's the console like.
And console makes more. Sense or important. When you're playing with retro operating systems. Not so much when you're just running quasi-production systems that you.
Log in to when crap power failed. Need to enter a password for an FS check type thing.
Okay.
Alright, anybody else?
Look in, Gary.
I guess this is it. This was the, January, well Thursday, January the eighteenth, 2024.
Monthly meeting of the St. Louis Linux users group St. Louis Lug. The speaker tonight was Stephen Lumbark.
His topic was an introduction to Haydu. And thank you all very much for coming this evening and thank you for participating.
We've had a lot of good questions and comments. And we hope to see you all either at the See, there's a Slack meeting next week.
Stan? 2 weeks, I think, 2 weeks. And next week.
New log is next week.
And next new log is next week. In for beginners workshop.
Okay, check our calendar. But right now I'll close the book end here for the recording.
So good night