Impostor syndrome is a psychological pattern in which an individual doubts their skills, accomplishments, or knowledge and fears being exposed as a fraud, despite evidence of their competence. It’s common among high achievers, but anyone can experience it. Symptoms include feeling like you’ve “just been lucky,” downplaying your achievements and constantly comparing yourself to others.

My observations in years of hiring

As team lead I have interviewed hundreds of engineers. For junior and senior roles. Over time, I noticed that even when I hired top engineers, they often struggled right from the start. Why? Would you ask. Answer is simple. Many of high achievers come from smaller companies or teams that didn’t have a lot of high skilled talent. They begin to feel overwhelmed. These people used to be go-to people at their previous job and had everything under control. This changed now and their bubble burst a bit. The realization, that they are not the ones smartest can be tough. They tend to self doubt and struggle to find their place where they would feel they are adding value.

Overcoming it effectively

Let’s have a look at some of the ways to overcome this situation, since the longer it lasts, the longer you will be struggling with self doubt, bad self image and will in general be unhappy at work. Keep in mind these are guidelines and not exact answers to the issue. Main thing we want to achieve is, to get you to believe, that you actually are valuable part of the team and company. Unless you truly believe it, you can’t move forward from it.

• Recognise it
  It’s important to acknowledge that it’s happening to you and understanding that this is just a temporary feeling. It is not long lived and as you become more comfortable in new environment, it will begin to fade. Understand you were hired because a senior hiring engineer or team lead saw a potential in you. Trust the process! It’s going to be all right.

• Finding your strength
  Focus on finding that thing that you are best at and can really shine if you are given opportunity to show your skills. It’s important to get rid of the feeling that you should know it all, and that you should be best at everything. In large companies it does not work. There is just too much code, infrastructure and technologies used, for you to be able to do that.

• Let your skills shine

  Take time for solving the task that you were given. Research, study, learn. Try to do absolutely best solution that you can think of and write reasoning why you chose something the way you did. This will give you confidence that it’s not a rash decision and that you put thought into it. It will also be ground for senior engineer to look at, read it, and understand your thought process. It will generally lead to much better responses than if you rush and prioritize speed over quality.

• Don’t fear asking for feedback

  Many times when you are in self doubt position, it’s best to ask others what do they think of your solution or code that you produced. Get constructive feedback and take it as an opportunity to learn. Choose senior people for such code reviews, or questions about correctness of your approach.

• Set realistic goals

  Perfectionism often fuels impostor syndrome. Strive for excellence but understand that mistakes and learning curves are normal. It will take time to get there and won’t happen over night.

• Learn, learn, learn

  If you notice that you are “behind” with knowledge, you can catch up faster, if you do a research on technologies that you think you are lacking knowledge of. Ask your team lead to suggest you good resources you can look up, to learn things in depth. It’s really a boost to self confidence if you get recognition by passing an official certification. So, try to do some certificate. Not because you need it for a job, but because of yourself and self confidence.

Final words

In general, the more positive feedback you will get, the more comfortable you will feel. Sense of achievement most effectively helps with overcoming impostor syndrome. Most of overachievers are prone to downplaying their achievements and therefore it takes longer for them to really and truely believe they are valuable part of the team and company.

Keep that in mind and I hope this post helped you at least understand the situation. Good luck and don’t worry too much about it. In time things will get better!

Traditional and crypto exchanges often times obscure the location of their infrastructure. They use Content Delivery Networks to mask their true locations. CDN also helps them protect against DDoS attacks, and mitigate other security threats. Keeping the details of their servers and IP addresses private is in their best interest, and such information is typically shared only with selected market makers that they work with. This basically means you, as a retail customer, can not get to this information.

However, there is a way to pinpoint the exact region of an exchange's infrastructure, if you know an exchange’s cloud provider. (90% of crypto exchanges for example are hosted on them) In this article, we’ll explore one such crypto exchange and try to figure out region in which to deploy your trading or arbitrage bots, ensuring the lowest possible latency to the crypto exchange.

Methodology

First of all, before we begin, we have to think a little bit how to tackle this problem and here is how my train of thought went:

• Find out what cloud provider is crypto exchange using. Most of them post this publicly and it’s not really a secret.

• We need to get response from non heavily cached endpoint in order to get initial latency to the exchange. It would be best to choose un-authed endpoint, so we can geo query it.

• Geo query the endpoint, get lowest latency node and find a region on a cloud provider, that this node is closest to.

• We can go further and benchmark from which availability zone we get lowest time on limit order endpoint response. Usually matching engine runs in one availability zone with hot standby in another. Unfortunately if exchange is doing loadbalancing that introduces additional network hops, knowing exact AZ of matching engine doesn’t matter anymore.

Cross AZ network hop in AWS usually adds 1ms and can vary up to 2ms of additional latency.

Tracing AWS region of an exchange

Step 1

Looking at API reference of the exchange that we will try to pinpoint shows that server time endpoint is accessible here:

GET /api/v3/time

Step 2

Lets open up: https://www.site24x7.com/tools/check-website-availability.html or any similar website where you can check response times of a website from different parts of the world and fire it against our server time url:

We can clearly see that the total response time seems to be fastest in Singapore, which coresponds to the AWS region ap-southeast-1.

Step 3

Benchmark limit order endpoint and response times from each availability zone in that AWS region. Unfortunately particular exchange didn’t show much better performance from single AZ, but we could pinpoint that connection times from apse1-az2 are a little faster in average of ~1-2ms.

Summary

Knowing in what region exchange hosts its infrastructure is mostly important for automated trading. An example of those that are latency sensitive would be market makers. Beyond region, pinpointing actual AZ usually does not result in much better performance in speed of setting limit orders. Some exchanges do use smart routing on the ingress though and route requests directly to the matching engine, so testing limit order endpoint from each AZ still makes sense to at least try, maybe you get lucky.

"Sorry Server" is a term often used informally in web development. It typically refers to a server that serves a fallback page in order to handle errors gracefully and inform users when something goes wrong with the service. In most cases the content served is static in order to be as lightweight on the resources and fast to serve as possible.

Common use cases

• Rate limiting

  In this case you would show your clients a sorry page when you are experiencing higher load or notice a larger queue in your service and would like to rate limit the intake of the requests.

• Maintenance Pages

  One of standard use cases is also using sorry server to inform clients that currently a maintenance is happening.

• Graceful Degradation:

  When your service is experiencing increased load and cannot keep up with the intake of the traffic, you would use sorry server to only be shown on certain parts of your system. Meaning, it would limit the functionality, but wouldn’t out right cut everyone out.

• Error handling

  When your servers detect an error, let there be code issue or another resource issue, they could instead of showing an error, rather forward the user to the sorry server.

Lessons from production

There are takeaways from production where sorry server saved us:

• SPOF service inplace upgrades

  Sometimes, you have to patch or upgrade a SPOF service, that doesn’t have additional server that you can fallback to, during upgrade. Running commands like apt upgrade or do-release-upgrade can lead to very unexpected behaviour of the server. For example, a PHP webserver for example could unload module for PHP in Apache during the upgrade and restart the httpd server. This leads to exposing raw PHP files of all of the clients hosting websites on that server for a period of time. This is something that happened lately with ubuntu server. Perfect example, where you should put sorry server up and make sure that for that period of time, nothing connects to the server being upgraded.

• Inplace updating your code

  Let’s say you are hosting your code on some webhosting server provider. Developer would usually just upload files via FTP or rsync. Unfortunately, that is not immutable deploy. Meaning, for the time in between when you start syncing your new files over the old ones and time they finish transferring, there can come to unprecedented errors, even wrong execution of code, because clients are all the time executing whatever is synced at the time. This is something we saw a lot, when I was working in advertising company and decided to go with immutable squashfs images of code, that would just be remounted (there was no Docker at the time). This kind of deployment of new code is also a case for a sorry server, to put it in front of your server while you are updating files and avoid the problems.

• Handling of all types of responses

  As most of crypto exchanges, we serve some html websites and on certain endpoints we also offer API access that is returned as JSON data. This means, putting a standard sorry server that just returns html output, does not cut it. We have to think about our clients. What happens with the code that connects to API service and tries to JSON load the input, but instead of JSON, gets html? That leads to awful errors. This is why we modify endpoints that we serve on our sorry serve, swapping from html output to JSON on appropriate paths and returning valid JSON with error message that explains that we are in maintenance mode.

• CDN implementation

  For actual exchange infrastructure, we have our own sorry servers of course, but for certain third party software that we are running, having CDN serve a maintenance page is just fine. That is also an option for all of you, that don’t actually have several servers and loadbalancers, to utilize.

• Serve static content

  A very tricky thing happened to us back in the day, when we were trying to be very smart about our maintenance sorry server page and actually served dynamic content on it. We tried to do it lightweight, but it was also connecting to database. The real problem happened, that we didn’t think about all edge cases. One of those would be, providing favicon.ico file on it, because certain browsers like FireFox require it. That led to awful spam to the database, trying to get that file and ended up executing dynamic content. This doubled the requests that we would have to handle, one for favicon and one for website and of course that led to certain degradation of the database performance. So, keep these maintenance pages static without any logic on it.

Footnote

Incorporating a "Sorry Server" isn't just a safety net, it's a strategy to enhance resilience and user satisfaction in production environments. In case you are not using it, hopefully above examples from production convinced you to at least have a look at it and maybe it’s really easy to add it into your procedure when doing maintenance.

In our fast pace environment of crypto exchange, write speed to database plays important role. For example, every trade, transaction or even non executed order needs to be written somewhere and for some of these, we use AWS Aurora. If you worked on the field for some time, you know, there is never ending issue with developers requesting an index to this or that column in some table. Adding an index in MySQL does not go without penalty to write performance. Reason is, that because for every database insert, an index needs to be updated too. This means there is more work to do and therefore this impacts write performance.

This is nothing new, but what you might be wondering is, how much does it actually impact performance and which types of columns in MySQL are worse than the other and for how much?

That is the real question.

Battle plan

In order to answer some of these questions, I prepared a small benchmark on db.c5.large Aurora cluster. Nothing special about it, everything default, so in short, ACID compliant database.

Main thing that we want to do is:

1. Create a table with multiple different types of columns

2. Pre-fill few million rows of random data

3. Do a SysBench insert benchmark, to see how everything performs without indexes

4. Add an index on a column

5. Repeat benchmark, to see how much that impacted the write performance

6. Rinse-repeat step 4 and 5 until all of the columns have index

Of course having index on every column makes absolutely no sense from practical standpoint, but we are not trying to benchmark reads. Writes are what we are interested in and how types of indexes impact them.

Actual setup

1. Table

   Note, few other types exist in MySQL, but from perspective of adding an index, it makes no sense to put index on for example “TEXT” type or something similar…

CREATE TABLE benchmark_table (
    id INT AUTO_INCREMENT PRIMARY KEY,  -- Integer
    name VARCHAR(255),                  -- Variable-length string
    birthdate DATE,                     -- Date
    salary DECIMAL(10, 2),              -- Decimal for financial values
    is_active BOOLEAN,                  -- Boolean (True/False)
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, -- Timestamp with auto-update
    rating FLOAT,                       -- Floating-point number
    visit_count BIGINT                  -- Large integer for counters
);

2. SysBench parameters

sysbench \
  --db-driver=mysql \
  --mysql-host=xxx \
  --mysql-user=xxx \
  --mysql-password=xxx \
  --mysql-db=benchmark \
  --report-interval=3 \
  --time=120 \
  --threads=150 \
  ./insert.lua run

3. Insert Lua script

function get_conn()
   drv = sysbench.sql.driver()
   return drv:connect()
end

function thread_init(thread_id)
  con = get_conn()
end

function random_string(length)
   local chars = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"
   local str = ""
   for i = 1, length do
      local rand_index = math.random(#chars)
      str = str .. chars:sub(rand_index, rand_index)
   end
   return str
end

function random_date(start_year, end_year)
   local year = math.random(start_year, end_year)
   local month = math.random(1, 12)
   local day = math.random(1, 28) -- To avoid invalid dates
   return string.format("%04d-%02d-%02d", year, month, day)
end

function event(thread_id)
   local name = random_string(8)  -- Generate a random 8-character name
   local birthdate = random_date(1970, 2000)  -- Generate a random birthdate between 1970 and 2000
   local salary = math.random(30000, 100000) + math.random()  -- Generate a random salary with decimals
   local is_active = (math.random() > 0.5) and "TRUE" or "FALSE"
   local rating = string.format("%.1f", math.random(10, 50) / 10)  -- Random rating between 1.0 and 5.0
   local visit_count = math.random(1, 500)  -- Random visit count

   local query = string.format("INSERT INTO benchmark_table (name, birthdate, salary, is_active, created_at, rating, visit_count) VALUES ('%s', '%s', %.2f, %s, NOW(), %s, %d)",
                               name, birthdate, salary, is_active, rating, visit_count)

   con:query(query)
end

function thread_done(thread_id)
  con:disconnect()
end

4. Indexes were added in this order:

1. CREATE INDEX idx_name ON benchmark_table (name);
2. CREATE INDEX idx_birthdate ON benchmark_table (birthdate);
3. CREATE INDEX idx_salary ON benchmark_table (salary);
4. CREATE INDEX idx_is_active ON benchmark_table (is_active);
5. CREATE INDEX idx_created_at ON benchmark_table (created_at);
6. CREATE INDEX idx_rating ON benchmark_table (rating);
7. CREATE INDEX idx_visit_count ON benchmark_table (visit_count);

Results

Keep in mind indexes are compounded. So, Visit_count_qps for example consists of all other columns having index, plus a visit_count.

Queries per second - higher is better

Latency (ms) - less is better:

Since above graphs might be a bit noisy and hard to read, I also prepared drill-down of degradation of performance per column type added:

And visual graph of above table:

Summary

Here it is, magic answer to your questions on how much degradation in writes can you expect per index type. As we can see some types are a little worse than the others, but in general, they all add certain penalty. Boolean being one of the best indexes and varchar being one of the worst column types you could choose to add index on. Keep in mind that with MySQL, adding an index also increases table/disk size, and by the rule of thumb, if you have more than 30% of additional disk claimed by indexes, you are probably abusing indexes too much and should rethink your strategy.

Just like monitoring and logging, tracing and profiling are also important in general observability of your running applications. Here come Application Performance Monitoring, or in short APM. Some of the leaders in this field are certaily AppDynamics, New Relic and Dynatrace.

Cost problems

As time passed, these APMs became very popular and many companies turned to them, when they needed to optimize or otherwise debug their applications. These proprietary APMs actually make sense when you need to monitor few servers, but running them on thousands of servers, things become very expensive…

Open Source Alternatives

I recently found a few open source APMs that aren’t that known, so I decided to add them alongside some of the de-facto standard ones and I’ll let you decide if you want to try them out.

1. Jaeger Tracing - https://www.jaegertracing.io

   Jaeger is an open-source application performance monitoring (APM) tool originally developed by Uber and later contributed to the Cloud Native Computing Foundation (CNCF). Jaeger is a distributed tracing system designed to troubleshoot microservices-based distributed systems. As its fast adoption by many companies, Jaeger is a solid option to evaluate if you're looking for an APM solution to replace some proprietary one.

2. Coroot - https://github.com/coroot/coroot

Coroot is a monitoring platform for tracking performance of microservices and distributed applications, particularly in cloud-native environments. It prioritizes service-level objectives (SLOs), enabling teams to monitor application health from a service-oriented perspective, focusing on user experience and reliability instead of just basic metrics. Using eBPF, Coroot automatically collects detailed telemetry data, including metrics, logs, and traces. Integration with kubernetes is especially awesome and I was pretty impressed by it.

3. SkyWalking - https://skywalking.apache.org

Application performance monitor tool for distributed systems, especially designed for microservices, cloud native and container-based (Kubernetes) architectures. Leveraging eBPF technology, Kubernetes Network Monitoring delivers granular insights into network traffic, topology, and TCP/HTTP metrics.

4. Uptrace - https://uptrace.dev

Uptrace is another open source APM that supports distributed tracing, metrics, and logs. You can use it to monitor applications and troubleshoot issues. Uptrace comes with an intuitive query builder, rich dashboards, alerting rules, notifications, and integrations for most languages and frameworks.

5. Signoz - https://signoz.io

SigNoz is an open-source observability tool powered by OpenTelemetry. You can use SigNoz to monitor your applications and infrastructure and get insights about it. It supports a wide variety of telemetry signals like logs, metrics, traces, and exceptions. You can set alerts and get notified in your preferred channel about any issue or some critical metric crossing a threshold. You can then use SigNoz as an analytics tool to debug the issue using its different features.

My takeaway

I’ve tried several options, and while some are well-suited for specific use cases, I still lean towards a few favorites. In my opinion, if you're looking for a well-rounded APM with an intuitive UI, Jaeger is an excellent choice.

Please keep in mind that the stories in the Battlefront Chronicles may not all be my own. Some of them come from other engineers working in the field, and their contributors may choose to remain anonymous.

Advertising company

Back in the day I was working at an advertising company, making sure our servers were delivering ads as fast as possible, to as many clients as possible. It was at the time second biggest advertising company in the central europe and we were doing roughly 40-60k requests per second on our frontend servers.

Infrastructure

There were quite a lot of supporting servers and services, but ~20 main virtual servers were responsible for delivering ads to the clients. They were all running on hosts that had special kernels, which were patched to run containerized applications. The technology was called vServer. Pretty similar to Docker, but very high performant. Along with these ones, we also ran few of virtualized frontend serving instances on kvm and XenServer. Keep in mind, these were very early days and money was scarce, so we had to do with what we got, often using very old hardware or reused/refurbished servers that we bought from other companies, which were replacing their own infrastructure with newer hardware.

Incident

One day, CTO approached me with very interesting finding. Looking at the charts, we figured out there was lower click rate of our ads. Looking further, it seemed to have started happening gradually, but in last 2 weeks or so. Advertising is very chaotic environment and a lot of events affect it a lot. For example, when Soccer World Cup was happening, we had no clue what is going on, because we were serving 30% of the traffic for an hour or so, but then got hit by a sudden enormous spike. We later figured out, it was a half time of the game. So, these kind of events drastically affect serving, but not the internal click rate and ad showing etc., so this was indeed a weird situation.

Investigation

As sysadmin in charge back at the time, I went and dug into the problem further and figured out that only 1 out of 20 servers has lower click rate. Very odd! Why only this server? Metrics were fine, everything was fine. But then, I started to think a bit and tried to understand what is actually going on. So… to put it simple, when the ad is served, we send an event of “ad served” back to the servers and when it’s seen, we send “ad opened” and so on. Logically, “ad served” and “ad opened” usually happen quite fast upon serve, but “ad clicked” really depends on a person, if they click an ad or not. Looking at code I figured additional information that helped debugging the situation. Our code was actually doing additional check, that if “ad click” came through way late, like 10-15 seconds late, that would mean for us, we need to discard it as non-valid click event. Digging even further, that is exactly what was happening on this offending server. This started ringing the bells. It has something to do with time!

The culprit

After the initial investigation, I checked server time of the offending server and then compared it with other servers. Bingo. A difference! This server, seemed to have difference of server time of about 10 seconds!

So what. You would say…

But no, this affects our “ad click” event. It will compare timestamp when ad was served, which was served from another server, that had clock in sync and then “ad" click” came to this server, that had difference in clock of about 10 seconds, which automatically flagged this click event as discarded.

Mind blown.

Reasoning

Okay, so why does this even happen? I mean, we already have ntpd running on all of the hosts! Investingating a bit more, I figured out there was a difference between the rest of the ad serving servers and this one that was faulty. This one, was actually running in full virtualization! Oh, noes.

Unfortunately full virtualization can have effects on the clock itself, since it exposes it through a virtual clock. Due to this, a clock can drift significantly.

The fix

I immediately installed ntpd on full virtualized servers. I also figured out, there were other fully virtualized VM’s, that were drifting over the months of running. As soon as ntpd synced on them, this solved the problem. Ad discards were no longer happening and we had to go and asses the damage.

Aftermath

We were actually discarding all ad clicks on 1/20th of the servers, which meant a lot of clicks in two week span. The loss in revenue wasn’t that substantial though and that was actually an issue. If it was greater loss, we would have noticed it much earlier and would go investigate. But, as it was only 1/20th of the traffic being discarded, that was still in margin of error in reporting, considering ad serving is quite chaotic environment as mentioned before. What we could do after that, was set alerts on these discards, just in case something like this happens again, this should give us fast enough alarm that there is something wrong.

Final thought

This incident was a lesson to myself that I should always have metrics for time drift from now on. No matter where I am working at and what I am doing, servers should have ntpd or chrony, since server time is something that applications rely heavily on. Any clock drift pops up in a very weird manner and it takes a lot of time to debug and pinpoint the exact issue.

When dealing with large-scale databases, we often encounter the following challenges during schema migrations:

1. Downtime and Service Disruption - Schema changes many times require taking the database offline, resulting in downtime for applications depending on the database.

2. Data Integrity Issues - Data consistency can be problematic during a migration, especially when altering large tables.

3. Performance Degradation - Adding indexes, altering large tables, adding columns etc. can cause performance bottlenecks, slow queries, and increased load on the database during migration itself.

4. Locking Issues - Most of schema changes result in table locks, blocking writes to the database.

Renaming approach

Common approach when dealing with these kind of problems is, to take advantage of MySQL’s online operations and work a solution around that. One of those instantanious operations would be for example rename operation and here’s how we utilize rename to be able to do schema migration with zero downtime:

Step 1: Copy schema of Table1 and create empty Table1_tmp

Step 2: Write down and drop foreign keys from Table1_tmp

Reason we do this, is because foreing keys checks pose a lot of additional processing for the database when it’s inserting data. If we are to sync the data into this new table, we want to keep processing at minimum, to have as fast sync as possible.

Step 3: Apply schema migration on Table1_tmp

Make sure that if you are adding column, you specify the default value that will be used while inserting data from Table1 to Table1_tmp for existing data.

Step 4: Start syncing data from Table1 to Table1_tmp

During the sync, Table1 can be in use. This means, we are free to use database, but we need to make sure we have binlog enabled, so that write queries can be replicated on the new Table1_tmp and we won’t have it out of sync after filling of data from Table1 finishes.

Step 5: Once we are in sync, just rename tables

Rename is instantanious process, so what we need to do is rename Table1 to Table1_old for example and right after that, rename Table1_tmp to Table1.

Step 6: Add the dropped foreign keys

Available tooling

To help us with the process, we already have few open source tools available today. One of them would be gh-ost (https://github.com/github/gh-ost), but there are others, like pt-online-schema-change (https://docs.percona.com/percona-toolkit/pt-online-schema-change.html).

Gh-ost snippets

To save you some time, here are few snippets we usually run when doing the migrations using gh-ost, so you can modify them to your liking and test things yourself.

Running gh-ost

To run gh-ost you will need to install it on a server that has access to your database. Replace all ${VARS} with appropriate details from your database and modify the “ADD INDEX …” with your schema change statement that you wish to perform over the table.

gh-ost \
--assume-rbr \
--user="${DB_USER}" \
--password="${DB_PASS}" \
--host="${DB_HOST}" \
--allow-on-master \
--database="${DB_NAME}" \
--table="${DB_TABLE}" \
--verbose \
--alter="\
ADD INDEX table_name (column_one,column_two,datetime)," \
--initially-drop-ghost-table \
--skip-foreign-key-checks \
--max-load=Threads_running=30 \
--chunk-size=50 \
--cut-over=default \
--exact-rowcount \
--concurrent-rowcount \
--serve-socket-file=/tmp/ghost.sock \
--panic-flag-file=/tmp/ghost.panic.flag \
--postpone-cut-over-flag-file=/tmp/ghost.postpone.flag \
--execute

This command will create copy of schema of your ${DB_TABLE} table, run the --alter commands on the newly created table and start syncing data from main table to this new one.

Needless to say, you should run this command in screen or something similar, that you can detach from, since it will probably take long time.

For further information on flags that you can/should choose, have a look at the cheatsheet.

Checking status

To check status of the sync, there is a convenient socket available.

echo status | nc -U /tmp/ghost.sock

# Migrating `mydb`.`mytable`; Ghost table is `mydb`.`_mytable_gho`
# Migrating ip-172-27-0-49:3306; inspecting ip-172-27-0-72:3306; executing on ghost
# Migration started at Wed Jan 09 16:25:10 +0000 2024
# chunk-size: 500; max-lag-millis: 1500ms; dml-batch-size: 10; max-load: Threads_running=30; critical-load: ; nice-ratio: 1.500000
# throttle-additional-flag-file: /tmp/ghost.throttle
# postpone-cut-over-flag-file: /tmp/ghost.postpone.flag [set]
# panic-flag-file: /tmp/ghost.panic.flag
# Serving on unix socket: /tmp/ghost.test.sock
Copy: 22404000/81210224 27.6%; Applied: 4052; Backlog: 0/1000; Time: 2h21m25s(total), 2h21m24s(copy); streamer: mysql-bin-changelog.003876:31236708; State: migrating; ETA: 6h11m11s

Throttling

If you see impact on production due to this process, you can set niceness and reduce burden on the database:

echo "nice-ratio=1.5" | nc -U /tmp/gh-ost.sock

Perform rename

Once your tables are in sync, you are free to perform cut off. To do that, gh-ost constantly checks specific file and if you remove it, the running process will perform cut off and exit, finishing the migration process.

rm /tmp/ghost.postpone.flag

More commands…

If you are interested in more interactive commands, you can check here:

https://github.com/github/gh-ost/blob/master/doc/interactive-commands.md

Final thoughts

Although pt-online-schema-change is a powerful tool, we prefer using gh-ost for the job. Its additional features, which are essential for managing large databases, make it invaluable. Of course you are free to test both and choose one of your liking.

To put it simple, when you want to provide a context that is too large to be “copy-pasted” into your context prompt, you need RAG.

Or, as NVIDIA says:

“Retrieval-augmented generation (RAG) is a technique for enhancing the accuracy and reliability of generative AI models with facts fetched from external sources.”

Here's how it works:

1. The system first searches a large dataset (like documents, web pages, or code repositories) to retrieve the most relevant information based on a your question.

2. After retrieving the relevant data, the system uses a generative model to generate a response that incorporates the retrieved information.

This way we can produce more accurate and contextually relevant answers by affecting its responses in real, retrieved data rather than relying on pre-existing knowledge.

Enough talk, let’s roll!

Let’s have a look at example of how my AI GitHub Code Analyzer written in Python does it’s thing.

Prerequirements

We will need ollama running on the same machine you want to run the analyzer. More information on how to set it up in my other article: https://bitnirmata.com/p/how-to-run-your-own-chatgpt

Installation

Obviously, you should clone the repository and install dependencies:

git clone https://github.com/krmeljalen/ai-github-code-analyzer.git
cd ai-github-code-analyzer
pipenv install

Configuration

All kinda important configuration is in config.yaml file:

############
# Defaults #
############

repo: "krmeljalen/kdeploy"
selected_model: "llama3.1:8b"
ollama_endpoint: "http://localhost:11434"

###############
# Llama-Index #
###############

chat_mode: "compact"

#####################
# Advanced Settings #
#####################

num_thread: 12
system_prompt: "You are a sophisticated virtual assistant designed to assist users in comprehensively understanding and extracting insights from a wide range of documents at their disposal. Your expertise lies in tackling complex inquiries and providing insightful analyses based on the information contained within these documents."
embedding_model: "BAAI/bge-large-en-v1.5"
top_k: 3

Parameters you probably should change to fit your needs:

repo - It’s basically repository url. This string is used this way https://github.com/{repo}

selected_model - needs to be the same as the model that you downloaded and loaded into your ollama

num_thread - if you are using CPU and don’t have GPU like my poor laptop for example, it can speed up things, so modify this to number of cores you have

top_k - increase this if you are getting bad answers, to make it think a little more before blabbering nonsense

Running

Simple enough, just run these two commands in the folder you ran pipenv install:

pipenv shell
python3 main.py

It should look like this:

You are free to ask it questions now and when you are done just type: EOF

End of the line

Thanks for your attention, now go parse some public repos and ask AI what kind of security vulnerabilities did it find in the code.

Please keep in mind that the stories in the Battlefront Chronicles may not all be my own. Some of them come from other engineers working in the field, and their contributors may choose to remain anonymous.

Webhosting

It was way back in the day when I was only 16 years old and was running a small server to host websites on it. To mention, there wasn’t many webhosting companies at the time. The business grew, so I expanded business and actually created a webhosting company under my family business name. I started to accept more and more customers. Some of them migrated from other web hosting providers, because they got hacked there, or were using too many resources. But that was totally fine in my eyes. I always valued this in myself, that helping those in need is honorable. And if they are desperate, I am here to accept them and help them in any way I can.

Webserver

As I am coming from a bit of a black hat-ish background, I knew how to prevent majority of web attacks, even if code was very insecure. Apache’s module mod_security helped me tremendously. I was digging deep into kernel hardening with custom kernel grsec patches and similar stuff. Majority of the webpages were running PHP so I was running suPHP that actually allowed me to limit resources that each customer was using. I never was into cPanel and such proprietary solutions. No. I rather wrote my own control panel. I integrated all required services and tied them together. From ProFTPd, MySQL, Apache web server and mail server that was running Postfix, to disk quotas, cpu and memory limits with cgroups. It was perfect.

Fear of attack

Only thing I was worried about was being DDOS-ed. (Back in the time, internet was very chaotic environment.) So I decided to take it up a notch and bought co-location for my server with, at the time, enormous network pipe of 1Gbit! I was safe. There wasn’t many DDOS attacks that this kind of pipe couldn’t handle and even so, my ISP was very helpful, if I needed to block some traffic before it actually hit my server. So, that part was neatly covered.

The Nightmare

As engineers, we may think we got things covered, feel comfortable that we really nailed this one and nothing can go wrong… there is always another thing that we didn’t think about.

One day, my customers started calling on my mother’s company phone, that their websites that are being hosted on my server are not available. Of course, I got chills in my neck. What could it be, I felt I need to act immediately. When I reflect on it now, I feel I was being a bit silly. Even if the website dedicated to some pet dog wasn’t working for 5mins, its not the end of the world. But at the time, it did feel like that to me. I thought I failed miserably and need to fix this right a way.

I barely could log in to the server through SSH. I noticed it was getting hammered with traffic. CPU was going wild and Load was terrible. But what kind of traffic is this that is causing this? Immediately I started iptraf to pinpoint where the traffic is coming from. It was thousands of different IPs, connecting to… SMTP server! What the hell? Is someone sending a lot of emails to one of my customers? Is this legit traffic or not? I started to dig into it and could pinpoint that ther was only one affected email address that was getting mail bombed. I contacted customer and asked if I can review some of these emails coming in, because I felt something is off and I can’t pinpoint without looking at the content, to say if its legit or not. Customer agreed and I reviewed his incoming mail. It was all rubbish text. A lot of it. Randomly generated. Not a single thing, besides “To:” field that had email of my client was same in any of them. I thought, why is this even hitting my mail server, I have SpamAssassin and I do check in Postfix if there is valid Reverse DNS on IP, before even accepting any mail. How is this coming through? I reviewed some of the offeding IPs and checked Reverse DNSes, MX records and SPF records. They were all fine! They were totally legit emails that are sending enormous amounts of rubbish spam and totally overwhelming my server.

The situation seemed miserable. How can I stop this. I can either just block all SMTP traffic, but then every customer will get affected. I don’t want that. I also don’t know when this attack will stop and they cannot be without emails for all that time. There was no light at the end of the tunnel. But then I said, okay, lets chill out and think. What can we do? And then it hit me. Mail servers won’t quit sending email if service is unavailable, so blocking was out of question. They will just delay it for a while and retry after retry period expires. What will happen if I change the MX record for the domain name of this affected email address…. to 127.0.0.1 or better said: localhost? That will actually make mail servers to resolve domain name to their own mail server and… start spamming themselves. This will also utilize their own spam prevention systems, that if email account doesnt exist on their system, they will discard it. And so decision was made.

Fight back

I called affected customer and explained the situation. I had to break it to him that his email is screwed anyway, but in order to get the attackers to stop, is to let these mailservers DDOS themselves to oblivion and they will hopefully stop after some time. He agreed and I swapped MX record of that domain to localhost. Quite soon, my server’s load went back to normal. Attack was stopping, as TTLs for this MX record was expiring all over the world and my server was back online.

Victory is mine!

Feeling of victory overwhelmed me. I got them good! I managed to throw their own rubbish back at them. Customers started to report everything is awesome and I could finally take a breather. I decided to leave this domain MX pointed to localhost for 24 hours, before I return it back. I was a bit worried, attack would start to happen again, after I reset it to correct IP, but no. Attackers learned their lesson and just stopped with the attack. Luckily, they never tried something like this again, but then again, they probably lost a lot of their zombie nodes, since server admins figured out something fishy is going on, with their mail servers DDOSing themselves and started to remove malicious code doing it.

Task shadowing is a concept similar to job shadowing but more focused on specific tasks or processes rather than an entire job role. In a production environment, especially when dealing with critical tasks, task shadowing can be a valuable practice to ensure accuracy, quality, and knowledge transfer.

Considerations

Understanding when and how to utilize task shadowing is crucial. Let’s break it down into a few key points to consider when implementing task shadowing:

• Identify Critical Tasks: Determine which tasks in the production environment are critical and would benefit from task shadowing.

• Select Appropriate Engineers: Choose individuals who will perform the task and those who will shadow, ensuring the latter have the necessary foundational knowledge. Preferably the one shadowing should be a Senior Engineer that has the know how.

• Establish a Shadowing Plan: Plan the task shadowing session, including when it will take place and what will be covered in the session. If physical presence of the person shadowing is not possible, we usually utilize screen sharing software like Teams or Zoom. Person executing tasks on production environment is sharing a screen and usually waits for confirmation of the person shadowing.

• Engage in Active Observation: The person shadowing should actively observe, point out the mistakes and double check what is being executed.

Benefits

We observed number of benefits utilizing such process. Let’s have a look at some of them:

• Training: Task shadowing can be an effective way to train less experienced team members on critical tasks. Observer can add additional comments to the one executing commands, so they are more confident in what they are doing.

• Documenting: During task shadowing, the observer updates the steps involved in the task in case there is any change needed in the procedure for future reference.

• Double-checking: In a production environment, critical tasks often require high accuracy. Task shadowing allows for a second pair of eyes to oversee the process, reducing the likelihood of errors. We, as engineers are very prone to see other people’s errors and shadowing is leaning on it heavily.

• Real-time feedback: The shadowing individual can provide or receive immediate feedback on the process, identifying potential issues or areas for improvement.

• Minimizing risk: During the execution of particularly sensitive tasks (e.g., deploying a major update or handling a high-risk operation), having someone shadow can help ensure that all steps are followed correctly, minimizing chance of mistake happening.

• Contingency Planning: Often times person shadowing will give feedback about possible issues we can encounter executing a specific task, along with remediation procedure for it, ensuring that the person executing task is aware of what to do in case something goes wrong.

• Building Confidence: For someone who will eventually take over the task, shadowing provides hands-on experience that builds confidence. It relieves stress and basically removes unnecessary fear that could build up otherwise.

• Cross training: Sometimes we are using shadowing as method of ensuring that more people are capable of performing critical tasks if the primary person is unavailable.

• Feedback: Last but not least, person shadowing can give valuable feedback to the one executing a task after the session. Observer can give a positive feedback on well performed task, or points out areas that the person executing crucial task should work on, to do better next time.

Summary

By integrating task shadowing into our critical production activities, we really enhanced the reliability and efficiency of our operations as well as lowered stress of engineers that are involved with executing these tasks. This is especially important in a sector like crypto and finance, where millions of dollars are on line for every minute of downtime. If your organisation is not doing task shadowing, you should strongly consider it.

Our work is highly specialized, focusing on optimizing network performance to ensure exchanges can handle the maximum number of requests without delay. To give you an idea, our median response time on certain endpoints is approximately 3-4ms. Enhancing performance beyond this level is extremely challenging. While our hot path applications are written in Go and we might achieve slight improvements by transitioning to Rust, we don’t see that as a viable option. Such small potential improvement does not justify the work and effort needed to achieve it.

Further optimization

At this point, we start looking into other kind of improvements that we can squeeze out of our infrastructure. One of them would be, using cpu and network optimized instances, using placement groups or cluster placement groups etc.

Arguably you should not put whole infrastructure in placement group that is latency optimized, since that brings additional risks, because instances are possibly located on the same rack in the datacenter. That being said, if something happened to that rack, a large portion of infrastructure would be at risk, not just few instances. It’s not really acceptable risk for us.

What we can actually do, without affecting overall risk, is improve general network performance inside VPC by utilizing Jumbo frames. But, Jumbo frames are used to improve throughput when dealing with large files, not the latency! Arguably that is true, but let’s have a look at the approach.

Hint: There is an overhead of handling a large amount of small packets vs. handling small amount of large packets.

Jumbo frames on AWS

In general AWS supports 9001 MTU Jumbo frames. But how do we get to the 9001? Seems rather weird number. We would assume 9000 would be sensible in the first place. The answer lays in the AWS hardware setup. AWS hardware actually supports 9216 MTU, but AWS, to be able to encapsulate all the traffic inside a tunnel uses 215 bytes per frame. So (9216 - 215) brings us to 9001 and this is the portion that is left for us to use.

Jumbo frames are basically supported on latest generation instances1, but keep in mind, AWS Traffic is limited to a maximum MTU of 1500 in the following cases:

• Traffic over an internet gateway

• Traffic over an inter-region VPC peering connection

• Traffic over VPN connections

• Traffic outside of a given AWS Region

For our case, this is totally fine. We need Jumbo frames support on cross AZ and intra-AZ.

Proof of Concept

To prove the theory that overhead of processing large number of packets has its effect on overall latency, we will be using the largest packet that we can find on our hot path infrastructure, and that would be orderbook call. Fully fledged order books for coin pairs are usually up to 500kB of size, so we will take this as data size we want to test on.

Enabling Jumbo frames

To enable Jumbo frames on Amazon Linux 2023 upon boot up:

echo "MTU=9001" | sudo tee -a /etc/sysconfig/network-scripts/ifcfg-eth0
echo "request subnet-mask, broadcast-address, time-offset, routers, domain-name, domain-search, domain-name-servers, host-name, nis-domain, nis-servers, ntp-servers;" | sudo tee -a /etc/dhcp/dhclient.conf

Setting it manually for testing purposes:

ip link set dev ens5 mtu 9001
ip link show ens5

Tracepath

Once we’ve set MTU on instances that we wish to test, to be able to verify if our path is actually supporting that MTU, we need tracepath:

sudo yum install iputils

Testing if our server to client path supports our desired MTU:

[root@mtu1 ec2-user]# tracepath 10.0.1.18
 1?: [LOCALHOST]                      pmtu 9001
 1:  10.0.1.18                                             0.166ms reached
 1:  10.0.1.18                                             0.127ms reached
     Resume: pmtu 9001 hops 1 back 2
[root@mtu1 ec2-user]#

Benchmark

References in code:
mtu1 - 10.0.1.89 - eu-west-1a - Server that is running listening server to accept data
mtu2 - 10.0.2.55 - eu-west-1b - Server that is running tcpdump script and client that will send 500kB of data
local - My own laptop that will trigger benchmark automation procedure remotely

I’ll be running server.py on mtu1 server. It will listen to connection, accept data, write some info about the transfer and shutdown:

server.py:

import socket

# Server configuration
HOST = '10.0.1.89'  # Localhost
PORT = 65432        # Port to listen on (non-privileged ports are > 1023)

# Create a TCP/IP socket
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as server_socket:
    # Bind the socket to the address and port
    server_socket.bind((HOST, PORT))

    # Listen for incoming connections
    server_socket.listen(1)
    print(f"Server listening on {HOST}:{PORT}")

    # Wait for a connection
    conn, addr = server_socket.accept()
    with conn:
        print(f"Connected by {addr}")
        total_data = b''

        # Keep receiving data until the connection is closed
        while True:
            data = conn.recv(1000000)
            if not data:
                break
            total_data += data
            print(f"Received {len(data)} bytes")

        print(f"Total data received: {len(total_data)} bytes")
        print("Data received successfully.")

Client code is run on mtu2 instance that is in another AZ than mtu1. It basically crafts string of 512kB and sends it to mtu1 server. Nothing fancy.

client.py:

import socket

def start_client(host='10.0.1.89', port=65432):
    # Create a TCP/IP socket
    client_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

    # Connect to the server
    client_socket.connect((host, port))

    try:
        # Create a 500 KB message (512000 bytes)
        message = b'a' * 500 * 1024

        # Send the data
        client_socket.sendall(message)
        print(f"Sent {len(message)} bytes of data to the server.")

    finally:
        # Clean up the connection
        client_socket.close()

if __name__ == "__main__":
    start_client()

We will run tcpdump script in automation script called analyzer besides the client on the mtu2 server, so that we can verify the whole path, from the send of SYN packet, to the FIN packet received, marking our connection closed. We will subtract the times in this script and get total microseconds needed for connection and transfer to finish. This will be used to determine how changing MTU affects the latency.

tcpdump script:

#!/bin/bash
echo "Starting analysis"

# Get current MTU
MTU=$(ip addr show | grep ens5 | grep mtu | awk -F" " {'print $5'})
echo "MTU: $MTU"

# Run for 5 seconds. Enough for us to run one execution and everything to be written down
timeout 4 tcpdump -i ens5 'port 65432' > dump.log 2>/dev/null

# Gather start and stop numbers
START_T=$(cat dump.log | grep "\[S\]" | awk -F" " {'print $1'} | awk -F "." {'print $2'})
END_T=$(cat dump.log | grep "\[F\.\]" | awk -F" " {'print $1'} | awk -F "." {'print $2'})
NUM_PACKETS=$(wc -l dump.log|awk -F" " {'print $1'})

# Cast to string cause sometimes we can have 092340 number
START=$(expr $START_T + 0)
END=$(expr $END_T + 0)
DIFF=$(($END - $START))
echo "Time: $(($END - $START)) μs"

# Graph section:
# If we have argument, use that as index otherwise just put in timestamp
if [ $# -eq 0 ]; then
    INDEX=$(date +%s)
else
    INDEX=$1
fi

TIMESTAMP=$(date +%s)
# Log to csv
echo "$INDEX,$MTU,$NUM_PACKETS,$DIFF" >> data.csv

To put everything together, we will run automation script on our own local computer that will execute everything in correct order. It’s all quick and dirty since this is PoC to show you how things behave.

automation script:

#!/bin/bash
# Note:
# mtu1 = server
# mtu2 = client

# Stop if theres an error
set -eu -o pipefail

# We will test these MTUs
MTUS=("1500" "9001")
for MTU in "${MTUS[@]}"
do
    # Set MTU on both servers, then proceed with testing
    echo "Setting MTU to $MTU"
    ssh mtu1 sudo ip link set dev ens5 mtu $MTU
    ssh mtu2 sudo ip link set dev ens5 mtu $MTU

    # Run test 40 times and gather results. They will be available in data.csv as per analyze script code.
    for NUM in {1..40}
    do
        echo "Iteration ${NUM}: ${MTU} MTU"
        ssh mtu1 sudo "python3 server.py" &
        sleep 1
        ssh mtu2 sudo "./analyze $NUM" &
        sleep 1
        ssh mtu2 sudo "python3 client.py"
        sleep 4
    done
done

I think 40 times is good enough to get at least some insight if there is difference between the two MTUs and transfer over network.

Benchmark results

We can notice a rough estimation of 500 microseconds latency improvement sending 500kB of data over the wire. Difference in number of packets needed to send this data is noticable. In 9001 MTU the data was sent in total of ~88 packets logged by tcpdump, while the 1500 MTU needed ~401 packets to send this data. We also notice that cross AZ latencies vary. They are not consistent and they take 1ms more from time to time. If I ran this long enough we would see some spikes where latency cross AZ can spike much more than 1ms though, in range of 4ms or so. This is something we raised with AWS in the past, because this kind of fluctuation does affect our matching engine performance if we use it cross AZ, but unfortunately there was no follow up on it.

Final thoughts

The grand total of improvement on orderbook API call that in start needed 3ms and shaving off 0.5 millisecond is 16,67% and if we count the max numbers at 4ms, then its 12.5% improvement. Some may not think this is a lot, but when dealing with so low numbers and going into microseconds realm, this does make a difference, for very low effort needed to achieve it.

AWS Bedrock is a managed service by Amazon Web Services that lets developers build, train, and deploy machine learning models. It offers pre-trained models and scalable infrastructure, simplifying the integration of AI into applications.

Problem

AWS Bedrock, as it’s awesome and pretty cheap way of accessing variety of LLMs without having to run server on your own, which can be pretty pricey, it lacks decent web UI. I was searching around to find something useful, but just couldn’t find it.

What I did find though is a project called: Bedrock Access Gateway, which is basically OpenAI-compatible RESTful API for Amazon Bedrock.

Putting it all together

Bedrock Access Gateway

Let’s get our Bedrock Access Gateway first to get it up and running, lets not use default Dockerfile, but Dockerfile_ecs:

git clone https://github.com/aws-samples/bedrock-access-gateway.git
cd bedrock-access-gateway/src/
rm Dockerfile
mv Dockerfile_ecs Dockerfile
docker build -t bedrock-proxy .

We will need AWS access so let’s enter proper access credentials by running:

aws configure

Now that we have our credentials set, let’s bring up our bedrock-proxy and pass it credentials to get access to AWS Bedrock:

docker run --rm -v $HOME/.aws/config:/root/.aws/config:ro -d -p 8081:80 bedrock-proxy       

OpenWebUI

Now lets bring OpenWebUI up and running by executing:

docker run --rm -d -p 8080:8080 -v ollama:/root/.ollama -v open-webui:/app/backend/data --name open-webui ghcr.io/open-webui/open-webui:ollama

Configuring

Lets spin up our http://localhost:8080 in web browser, register and login to the OpenWebUI. Click on your profile icon on top right and choose: Admin Panel → Settings → Connections

Disable Ollama API and edit the OpenAI API host to match your docker local ip and path /api/v1 as seen on the screenshot. Default password for Bedrock Access Gateway is: bedrock

Here we go, lets test it, as you can see on top we see bunch of available models on AWS Bedrock:

I chose anthropic.claude-3-sonnet-20240229-v1:0, but you can choose any other.

Voila!

Troubleshooting

If you get an error like this, you probably didnt subscribe to the model in AWS model access.

To verify it, you can check logs of bedrock-proxy container and you will see:

    | fastapi.exceptions.HTTPException: 500: An error occurred (AccessDeniedException) when calling the ConverseStream operation: You don't have access to the model with the specified model ID.

To enable it, just go to web console and under AWS Bedrock service, subscribe and add the model you wish to use. More info here.

Using public AI services involves several concerns that users and organisations need to be aware of to ensure security, privacy, and compliance, including:

1. Data privacy

2. Security risks

3. Compliance and legal issues

4. Intellectual property

5. Cost and Vendor Lock-in

Certainly, there are additional concerns, but these highlight some of the key challenges. Running your own Large Language Model (LLM) can help you overcome many of these issues.

Considerations

There are several ways to run your own LLM-powered AI service. You can choose to use CPU processing for your AI assistant or swap to GPU processing. I did test Distributed Llama, which uses multiple servers to enhance response times, but it didn't meet my expectations. Therefore, in this article, I will focus on setting up a server and utilising GPU processing.

Setup

Step 1: Prepare server

You will need a GPU server, and have SSH access to it. I will use AWS GPU instance, but you might consider some other provider that is a bit cheaper.

If you insist on AWS, few cheapest GPU instances on AWS would be:

• g3s.xlarge - NVIDIA Tesla M60 (~$0.3591/h Spot price)

• g4dn.xlarge - NVIDIA T4 Tensor Core (~$0.1919/h Spot price)

• p2.xlarge - NVIDIA Tesla K80 (~$0.3355/h Spot price)

Choose at least 40GB of storage, depending on the LLM you will be using.

Be sure to choose AMI image that has your NVIDIA drivers installed, otherwise you will need to install them on your own. Instructions for installing the drivers on your own are here.

Step 2: Setup Llama 3

Once you have your instance up and running, lets install Llama 3 and test how it works. 1

curl -fsSL https://ollama.com/install.sh | sh

Result should look like this:

[root@llama ec2-user]# curl -fsSL https://ollama.com/install.sh | sh
>>> Downloading ollama...
#######################################################################
>>> Installing ollama to /usr/bin...
>>> Creating ollama user...
>>> Adding ollama user to video group...
>>> Adding current user to ollama group...
>>> Creating ollama systemd service...
>>> Enabling and starting ollama service...
Created symlink from /etc/systemd/system/default.target.wants/ollama.service to /etc/systemd/system/ollama.service.
>>> NVIDIA GPU installed.

Step 3: Test Llama

ollama run llama3

My test looked like this:

In above test, llama 3 answered really fast, unlike if we would be using CPU instead of GPU.

Step 4: Setup OpenWebUI

Let’s run OpenWebUI in docker and not go with pip version, since that one is in beta.

Prepare prerequirements:

yum -y install docker docker-runtime-nvidia libnvidia-container-tools
systemctl enable docker
systemctl start docker

Run the OpenWebUI docker:

docker run -d -p 8080:8080 --gpus=all -v ollama:/root/.ollama -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:ollama

Verify that docker container is up and Healthy:

[root@llama ~]# docker ps -a
CONTAINER ID   IMAGE                                  COMMAND           CREATED              STATUS                    PORTS                                       NAMES
d9a8c31d4ad2   ghcr.io/open-webui/open-webui:ollama   "bash start.sh"   About a minute ago   Up 56 seconds (healthy)   0.0.0.0:8080->8080/tcp, :::8080->8080/tcp   open-webui

Step 5: Test OpenWebUI

OpenWebUI exposes port 8080 as we can see above. Let’s connect to it via web browser and register:

Lets fix the model now. On top right click your profile icon and go to Admin panel → Settings → Models and enter one of the models. I chose llama3.1:8b. To verify all other models click here.

Once download process finishes, you can open new chat, choose your model on top left and ask your question.

RDS audit logs are a crucial component of monitoring and maintaining the security and compliance of a database. These logs capture all activities and events within the RDS instance, providing a detailed record of who accessed the database, when they accessed it, and what actions they performed.

CloudWatch Log exports

By default, RDS audit log exports are sent to CloudWatch Logs. However, for databases with high activity, this can result in significant costs reaching tens of thousands of dollars per month. As a result, it is important to explore alternative solutions to manage these costs effectively.

Solutions

There are two solutions to this:

1. Download logs via RDS API and store them manually

   This is the cheapest method, but requires manual handling and combining of these logs, since logs are rotated by RDS quite frequently. You need to make sure you store the logs to alternative location, before they expire. Fortunately calls to RDS API are free and you will not be charged for them, but you will be charged for data transfer. Keep in mind API calls do have rate limits, so you will have to work around that.

2. Change CloudWatch log group to infrequent access

   Since these logs are not meant to be searched all the time, changing log group to infrequent access might just do it. Its much simpler solution that only requires recreation of log group and setting it to infrequent access.

In this article, we will explore the second option as an alternative to the first, which requires coding a custom solution from scratch.

Procedure

Backup

RDS uses predefined paths for exporting logs to CloudWatch. These paths are static and cannot be changed:

• /aws/rds/cluster/<cluster_name>/error

• /aws/rds/cluster/<cluster_name>/slowquery

• /aws/rds/cluster/<cluster_name>/audit

• /aws/rds/cluster/<cluster_name>/general

Because of this reason, there is a small drawback that you will have to delete them. They cannot be renamed, so if you want to keep logs, you can export them to S3 beforehand by clicking on log group and under action you can choose:

Changing log group to Infrequent Access

1. Disable current CloudWatch Exports, so your current log group gets released. Unless you do this, log groups will be constantly recreated by RDS.

2. Delete log groups:

   • /aws/rds/cluster/<cluster_name>/error

   • /aws/rds/cluster/<cluster_name>/slowquery

   • /aws/rds/cluster/<cluster_name>/audit

   • /aws/rds/cluster/<cluster_name>/general

3. Recreate all log groups with exact same name, but with Infrequent Access log class

   

4. Enable log exports on your cluster again

   Log groups that you created will be reused, meaning they will be using new Infrequent Access log class.

   

Summary

By switching log exports to infrequent access, you can reduce your storage costs by approximately 50%. This reduction also applies to API calls, which are the primary cost generator when publishing to CloudWatch logs.

Reference pricing table:

Latest prices for your region: https://aws.amazon.com/cloudwatch/pricing/

Finance sector in general, as you can imagine, is very paranoid about credentials storing and rightfully so. To mitigate this issue and only use encrypted store for credentials, using AWS Secrets Manager is one way to tackle the problem. It has been de facto standard for storing credentials for many companies, including ours.

Problems

But here come the problems. So far using secrets manager has resulted in few issues to say the least. Calling Secret Manager API directly from your code was the way it was suppose to be used and if you have few secrets, that might even float the boat. But as soon as you have 100s of credentials and a huge infrastructure, you run into several issues:

• Slow boot of an application - Calling API to load these secrets and so on, it just precious time to the boot up of application and we hate that.

• API rate limits - Even if high by AWS specification, you need to multiply amount of credentials by the amount of servers and this number all of the sudden gets pretty high.

• Costs - Imagine continuous deploys and autoscaled infrastructure where servers are all the time going up and down and being deployed and so on… these calls get costly after a while.

• Secrets Manager not being available - Low probability, but still… Ouch. You better not restart anything and immediately stop all autoscaling processes… :)

AWS did introduce bulk loading of secrets, which kinda helped with the situation, but still doesn’t solve all of the problems.

Secret Manager Agent

One way to deal with the situation is running some kind of an agent next to your application and here comes AWS Secrets Manager Agent. You can find it here: https://github.com/aws/aws-secretsmanager-agent

What it does is, it basically caches secrets for application to have them available immediately. This solves majority of the above listed problems.

To list at least few features that are significant improvement vs directly calling Secrets Manager API:

• Caches secrets (d0h!)

• In-memory storage of these secrets - thats good, cause it doesn’t store them on disk and they are immediatly available

• Ability to expire secrets after a while and re-retrieve them from Secrets Manager

• Interface to communicate with Agent instead of Secret Manager directly

For most of AWS Cloud infrastructures this is good enough. Unfortunately not for all. There are several issues where Agent still falls short:

• Doesn’t do cache invalidations - It’s a pull method upon TTL expiry. It doesn’t know that a secret changed and it should go re-fetch it and expose new value to the app.

• Secrets values in memory are not encrypted - Scrubbing memory and trying to protect credentials from memory dump attacks is hard.

• It fetches secret by secret, doesn’t do bulk load.

• Restart of Agent will wipe out all cached secrets and it will need to re-fetch them - Arguably this is good thing, since this means secrets are for sure refreshed and this way you can force cache invalidation.

Summary

Unfortunately this Agent came out a little late for us, to be using it, because we already implemented our own, called Secrets Caching Service or SCS in short. For now we will stick to our implementation, since SCS actually supports push method, grouping of secrets (pushing only a group of secrets in bulk to a SCS that is running on server) and utilizing Redis pub/sub functionality to notify appropriate servers that they should pull new secrets (because one of them changed in Secret Manager for example).

Disregarding the above, I still think majority of companies that are using AWS Secret Manager, should use Secret Manager Agent from now on. It’s just a no-brainer and a great improvement in secrets management in general. Great work AWS, I hope there will be additional features coming to it soon…

I work at a crypto exchange, where latency and througput of trading hot path is utmost important. We try to use every trick that we can come up with in AWS Cloud in order to get even more performance out of our systems. Saying that, load balancing is obviously very important. AZ aware connections, proximity of servers inside AZ, network hops from service to service, direct memory access, optimizing TCP stack, so that data fits in one packet… it all matters. In this post we will focus on our benchmark process regarding internal load balancing.

Loadbalancing approaches

Idea is not to delve too much into details of each, but high overview is still due.

AWS Application load balancer

• Provided by AWS

• Operates at Layer 7 (HTTP/HTTPS) and supports features like path-based routing, host-based routing, and HTTP/2

• Automatically scales to handle changes in application traffic

AWS Network load balancer

• Provided by AWS

• Operates at Layer 4 (TCP/UDP) and provides extreme performance

• Handles millions of requests per second while maintaining ultra-low latencies

• Provides a static IP address per Availability Zone

• Useful for load balancing traffic to databases, media streaming, and other use cases that require ultra-low latencies

IPVS load balancer

• Not provided by AWS as service and usually used in conjunction with kubernetes

• IPVS is a load balancing solution built into the Linux kernel

• It implements transport-layer load balancing, handling TCP and UDP traffic

• IPVS can direct requests to a cluster of real servers and make the services appear as virtual services on a single IP address

iptables load balancer

• Not provided by AWS as service and usually used in conjunction with kubernetes

• iptables load balancing is implemented directly in the Linux kernel, providing very high performance

• The nat table in iptables is used to perform Network Address Translation (NAT). This allows iptables to redirect incoming traffic to different backend servers

Proxy load balancer

• Not provided by AWS as service

• HAProxy, NGINX Plus, Traefik etc…

• Distributes incoming traffic across multiple backend servers based on predefined algorithms (e.g., round-robin, least connections)

• In summary proxy load balancer combines the benefits of a reverse proxy (security, caching, SSL termination) with the load balancing capabilities (traffic distribution, health checks, scalability) to provide a comprehensive solution for optimizing the delivery of web applications.

DNS based loadbalancing

• Not provided by AWS as service

• Consul, there might be more, but the priciple is same.

• DNS load balancer provides active health checking of services

• If a service fails its health check, it is removed from the service discovery results

• This enables load balancing to avoid unhealthy instances

Purpose

I aim to determine which type of load balancer is best suited for throughput in terms of requests per second with as little, to no loss comparing to actual direct client / server throughput.

Rules

1. HTTP traffic

Since there are multiple approaches of talking service to service, majority of clients that actually use these services in finance usually talk HTTP or FIX protocol. For intra service communication gRPC would probably be more suitable, but for the sake of giving all loadbalancers a chance in the race, I chose HTTP to be the one that we are going to benchmark. No keep-alive connections!

2. Static file serving

Purpose is to benchmark load balancers, not webserver performance itself. Static file serving (an image) is as simple as it gets for a webserver to serve without any interruption. Image gets cached and we get minimum delay serving it from memory upon receiving request. Decision was made, to serve small, only 4kb image file which more or less reflects size of common API service response.

Sweet spot

A baseline benchmark from one server to another, without any load balancer, has been established as our target requests per second. We aim to achieve this target, or get as close to it as possible, while introducing various load balancers between the client server and the webserver.

Setup

General

• VPC

• 1 AZ (no cross AZ traffic allowed)

• c5.large instance for client and server

• Amazon Linux 2023

• No additional optimizations of TCP stack in kernel apart from necessary changes needed for specific loadbalancer to actually work

• Apache benchmark tool

• 200 concurrent connections and 1 million requests, repeated at least 10 times, taking average requests per second as a metric that we are observing

AWS Application Load Balancer

Loadbalancer with HTTP port 80 forwarding to target group with webserver. Additional AZ subnet was added to ALB, since it requires at least 2 AZs to be able to run, but no instances exist in that second AZ.

AWS Network Load Balancer

TCP loadbalancer, forwarding port 80 to target group.

IPVS

First we installed ipvsadm, loaded kernel module and set kernel params to be able to run IPVS. Whole setup is basically the same as kube-proxy does it, when choosing IPVS as proxy mode.

yum -y install ipvsadm

modprobe ip_vs

sysctl -w net.ipv4.ip_forward=1
sysctl -w net.ipv4.ip_nonlocal_bind=1

systemctl start ipvsadm

After that we created a virtual IP and added our server as backend to it:

ipvsadm -A -t 10.0.1.50:80 -s rr
ipvsadm -a -t 10.0.1.50:80 -r 10.0.1.100:80 -m

At this point we can run benchmark on our vip 10.0.1.50.

iptables

I made minimal setup. Intentionally I didn’t add all additional stuff that usually kubernetes does by marking chains and all shebang that would result in even more impact on the actual benchmark, cause of additional rule processing. In the end it comes down to this:

iptables -F
iptables -t nat -F
iptables -t nat -X

# Forward traffic from port 80 to 192.168.10.211:80
iptables -t nat -A PREROUTING -p tcp --dport 80 -j DNAT --to-destination 192.168.10.211:80
iptables -t nat -A OUTPUT -p tcp -o lo --dport 80 -j DNAT --to-destination 192.168.10.211:80
iptables -t nat -A POSTROUTING -p tcp -d 192.168.10.211 --dport 80 -j MASQUERADE

What this does is very similar or even simplified version that kube-proxy does when you choose to run iptables as proxy mode.

Proxy server

Even if it seemed as non-sence to even try this one, since running additional webserver and amount of processing needed to run it, just to forward traffic to another backend seems very resource intense and a very poor solution for load balancer. I still decided to give it a chance and chose HAProxy as it’s suppose to be most suitable for these cases general.

Here is the excerpt of the config for HAproxy:

frontend myfrontend
  mode http
  bind :80
  default_backend web_servers

backend web_servers
  mode http
  balance roundrobin
  server webserver 192.168.10.211:80 check

Benchmark results

Forenote: For obvious reason it makes no sense to be benchmarking DNS load balancing solution, since that kind of load balancing already happens on the DNS resolution stage, before we even start establishing connection to the server and therefore is treated as a direct client to server connection.

Observation

AWS Application Loadbalancer

We can immediately notice how slow start is in effect here. It starts with about 30% of final throughput. It took loadbalancer roughly 10 minutes to scale up on AWS side to be performing at par.

AWS Network Loadbalancer

As advertised by AWS, very fast to react and at times even faster than direct connection to webserver. One could say, that doesn’t even make sense! But it actually does, you just don’t see whats going on in the background since it’s hidden in the “Cloud”. The fact is, servers are scattered around in datacenter and there is X switches between them. I would assume AWS reserves portion of the switches and routes network so that their services get best latency from servers. this doesn’t mean server to server is same well optimised! Meaning, RTT from client to server might take longer than RTT from client to network loadbalancer to server. Interesting, although as seen from the graph, difference is very minimal.

IPVS

Best from non AWS native load balancers. As it came up strong, it still fell short in the long run vs Application Load Balancer. Kernel module as soon as it was loaded did have an effect on the overall network performance. It added additional processing needed to be done and that is seen from the graph. For kubernetes, if you don’t want to be paying for network load balancers, because you might have a lot of microservices and what not, IPVS is best option to go with.

iptables

Second to last. Processing of iptables is burdening the system and if you add even more rules that usually kubernetes comes with, you will have a cumbersome infrastructure. It might do it for non latency sensitive and low troughput stuff, but I don’t see why would anyone use it, if the change to IPVS in kube-proxy proxy mode is pretty trivial.

Proxy

For obvious reasons last. Processing needed to forward traffic is just too much for server to be dealing with additionally, to stay competitive. Usually HAProxy is something you would use in AWS to overcome other limitations of native AWS load balancers. Example of that would be like a active-standby setup. Where only one server can be answering, but you still want to have a highly available setup in case it goes down.

Summary

We can conclude best choice for internal load balancer would be AWS Network Load Balancer. If you don’t have a lot of microservices and are in need of huge number of load balancers, this is your preferred choice. It can become costly though in the long run as your infrastructure grows and you need more and more of them…

IPVS is best “free” loadbalancer, if you go with kubernetes and kube-proxy to manage it, otherwise it can become quite a burden to manage.

General rule of thumb is, if you want to even further optimise the impact of loadbalancer on overall performance and throughput, is to use keep alive or connection pooling. Not all programming languages support connection pooling, but HTTP keep-alive is pretty straight forward improvement. From my testing, enabling keep-alive pretty much doubled the performance in requests per second.

Intro to kdeploy

kdeploy is small tool written in python that utilizes kubernetes and docker libs to build & publish images to your docker repository and deploy them to kubernetes cluster. Aim is to have your development environment very simple, fast and with little to no tinkering to get your small dockerized app up and running.

Problem with development env

Coding on your own laptop and trying your code out should be fun and easy. Majority of developers don’t want to deal with publishing, building and doing pipelines for their small app they developed in their spare time. Preparing scripts that will deploy an app, copying them around from project to project is cumbersome. Let’s say I coded small flask app and I just want it up and running as soon as possible, without figuring out kubernetes deployment, services and fixing the secrets for docker repo etc. I just want to specify what is name of my app, it should run in pod and it needs exposed port 80 as service, thats it. Here comes kdeploy that gives you just that.

Overview of kdeploy

So let’s quickly overview what kdeploy needs as requirement to run and what it does.

Requirements

• python - obviously

• kubernetes cluster configured - you should have your kubernetes cluster configured and be able to run kubectl on it, to manipulate it.

• docker repository configured - you should have your docker credentials setup and be able to login to your docker repository that you want to store your app to.

Process

Once you have kdeploy installed locally, you should have manifest.yaml in your folder and just run: kdeploy

What it kdeploy does is:

1. Finds Dockerfile in folder you ran kdeploy

2. Reads manifest.yaml file to make sure what is the name of this app and how do you want it deployed (Either as Pod, CronJob or maybe a Deployment with replicas…). It will also figure out if you need a service if you provided exposed port etc.

3. Builds docker image locally and tags it with app name

4. Pushes image to your docker repository with appropriate tag and version

5. Checks if you have secret set for kubernetes to be able to pull this docker image

6. Generates kubernetes files for deploying the app

7. Deploys app to your kubernetes

Demo

Here is simple example how would you deploy a very simple html serving app:

Try it yourself

You are welcome to follow instructions for installation and usage on my github: https://github.com/krmeljalen/kdeploy

Final words

I basically swapped all my personal projects to kdeploy. I used to have bunch of bash scripts that did similar things and I just got to a point where I wanted to change something in that deploy process and would need to update bunch of repos for it. kdeploy solved this issue for me. There will probably be more updates to kdeploy as I might need more features, but for majority of my projects, kdeploy already covers them. There are just few projects that need specific kubernetes files, but for those, kdeploy also supports path to kubernetes yaml specifying. This means, even if I want to have this one project very special and need to setup some persistent volumes and whatnot, I still can use kdeploy for it.