RSS feed [root] /design /weblog



title search:


Mon Mar 14 15:40:26 AEDT 2022


(google search) (amazon search)
download zip of files only

Sat May 21 23:43:53 AEST 2022 From /weblog/design/examples


Experience of using AWS Redshift -[..]on-events-a-day-the-shift-to-redshi.html

腾讯信息流内容理解算法工程全流程实践 -

(google search) (amazon search)

Sat May 21 23:06:44 AEST 2022 From /weblog/design/examples


Open Sourcing Twitter’s Algorithm Part 1: How Twitter Works -

(google search) (amazon search)

Fri Apr 22 09:34:15 AEST 2022 From /weblog/design/examples


DB Locking in Reservation Systems -[..]king-in-reservation-systems-3b3d574c7676

(google search) (amazon search)

Sat Apr 09 22:18:32 AEST 2022 From /weblog/design/distribute


Partitioning GitHub’s relational databases to handle scale -[..]ning-githubs-relational-databases-scale/

A Distributed System Interview Question: How to Avoid Read Inconsistency During a Transaction -[..]stency-during-a-transaction-6ac3b4a8f92a

(google search) (amazon search)

Mon Mar 14 15:40:26 AEDT 2022 From /weblog/design


Nothing specific, mostly code clean up, refactoring and simplification, the performance boost was a surprise. <- This is a good one -[..]12/07/disruptor-v3-faster-hopefully.html

Is there anything we can do about this when designing algorithms and data-structures? Yes there is a lot we can do. If we perform chunks of work on data that is co-located, and we stride around memory in a predictable fashion, then our algorithms can be many times faster. For example rather than using bucket and chain hash tables, like in the JDK, we can employ hash tables using open-addressing with linear-probing. Rather than using linked-lists or trees with single items in each node, we can store an array of many items in each node. -[..]emory-access-patterns-are-important.html

Skip lists are used instead of b-trees because b-trees don’t scale. -[..]ast-mvcc-inmem-lockfree-codegen-and.html

Beware about the performance issue of static initalization -[..]-cause-problems-with-multi-threading-how

Experiment show why arraylist is better in most cases -

How to design low latency application in java -[..]5/chronicle-and-low-latency-in-java.html[..]ciples-of-high-performance-programs.html[..]principles-of-high-performance-programs/

Suggestion about how to determine number of thread pool -[..]rmine-Web-Applications-Thread-Poll-Size/

Beware the performance penalty of logging -

Keep thing dynamic -[..]ciples-of-high-performance-programs.html[..]st-rule-of-performance-optimisation.html[..]u/~rcs/research/interactive_latency.html[..]aling-data-platforms.html?SSLoginOk=true

Discuss about developing low latency financial application -

Discussion of object pooling -[..]-but-forgotten-trick-object-pooling.html[..]m_performance_part_1_object_pooling.html

Efficiency - the amount of work you need to do.
Performance - how fast you can do that work
Efficiency - governed by your algorithm
Performance - governed by your data structures.[..]e-difference-between-efficiency-and.html

Turning off power save mode on the CPU reduced brought the max latency from 11 msec down to 8 msec.
Guaranteeing threads will always have CPU resources using CPU isolation and thread affinity brought the maximum latency down to 14 microseconds.[..]eduler-jitter-using-cpu-isolation-a.html[..]

About design for performance for webapi -[..]y-decision-as-a-service-design-patterns/

check list -[..]ery-software-development-team-should-ask

Beware if system utilization over 80% -[..]/20/service-latency-and-utilisation.html

scalable-io-events-vs-multithreading-based -[..]lable-io-events-vs-multithreading-based/

How to find out bottleneck -[..]entile-latencies-in-Chronicle-Queue.html

Compiler Performance and LLVM -

Know Thy Complexities! -[..]oW9D2qyCTt3qMftzYxsdz7KJs8LkSeRPWLzguwpA

10 大高性能开发宝石 -

How to detect and fix IO related performance issue -

RangeBitmap produces a RoaringBitmap of the indexes which satisfy a predicate, and can take RoaringBitmap parameters as inputs to skip over rows already filtered out. The Streams API code used before is translated into RangeBitmap API calls: -

(google search) (amazon search)

Tue Feb 15 10:37:05 AEDT 2022 From /weblog/design


How to write safer concurrency code -

reentrant and thread safe functions -[..]/31/reentrant-and-thread-safe-functions/

Libraries / toolkits for multicore process -[..]intableArticle.jhtml?articleID=212900103

Introduction -[..]QQSNDLRSKHSCJUNN2JVN?articleID=212903586[..]CQSNDLRSKHSCJUNN2JVN?articleID=213001517

Collections of links -[..]rallel-Or-Get-Left-Behind.html&Itemid=29

Briefing of difference modeling of threading system -[..]intableArticle.jhtml?articleID=215900465[..]inners-guide-to-multithreaded-libraries/[..]haloms_blog/2010/08/concurrency-101.html

Saving the Failwhale: The Art of Concurrency (Page last updated December 2012, Added 2012-12-26, Author Dhanji R. Prasanna, Publisher informit). Tips:

1) Contention is unavoidable - some resources are just slower, and you must wait for them. The secrets to good concurrency are 1) ensuring that these slower resources are rarely used, and 2) during such waiting periods, giving the faster tiers other work to do so that they continue to be utilized well.
2) Overuse of synchronization constructs such as locks and mutexes leads to systems that perform poorly under load.
3) ConcurrentHashMap is an efficient thread-safe map while HashMap is not thread-safe.
4) ConcurrentHashMap doesn't do away with locks, it still uses them but it uses more than the single global lock, so that threads gain some measure of concurrency. It uses separate locks for partitions, so that multiple threads writing to the map are likely to access different partitions, using separate locks and therefore process their data simultaneously. This technique is known as lock-striping. Efficient striping uses a number of locks proportional to the number of CPU cores in a system.
5) The asynchronous processing model smooths resource spikes by adding requests to a queue which is serviced by a pool of workers - spikes in requests make the queue grow rather than overloading the workers. (The ExecutorService is essentially a thread pool accompanied by a task queue.)

Discussion of using difference model for concurrency -[..]cks-application-architecture-pros-a.html

Concurrency vs Parallelism -[..]/2013/12/concurrency-vs-parallelism.html

Compare between Actors, CSP, Disruptor and raw Threads -[..]omparision-of-different-concurrency.html

Few coding tips that should be useful for most languages -[..]erformance-scalability-and-liveness.html[..]2015/09/java-concurrency-essentials.html

Service Design
Do one thing, do it well
No shared operational state
Bound your queues
Name custom thread pools and register an UncaughtExceptionHandler
Prefer immutable data objects over mutable state[..]-to-25-billion-notifications-during.html

On Parallelism and Concurrency -[..]lism-what-is-the-difference-bdf01069b081

Solving Common Concurrency Problems -[..]

(google search) (amazon search)

Sun Feb 06 00:10:09 AEDT 2022 From /weblog/design/pattern


Lenses in Java -

(google search) (amazon search)

Wed Jan 26 22:53:53 AEDT 2022 From /weblog/design/examples


Alright folks, gather round and let me tell you the story of (almost) the biggest engineering disaster I’ve ever had the misfortune of being involved in. It’s a tale of politics, architecture and the sunk cost fallacy [I’m drinking an Aberlour Cask Strength Single Malt Scotch] -

Uber Architecture and System Design -[..]hitecture-and-system-design-e8ac26690dfc

Designing Uber -

(google search) (amazon search)

Thu Jan 20 00:44:27 AEDT 2022 From /weblog/design/interview


I love the tool that Linus using -[..]23/stiff-asks-great-programmers-answers/ and here is the abstract -

Ted interview with Linus -

Not really interview, just a discuss of what programmer difference from other -[..]in-on-codewhat-makes-you-able-to-program

Vote for most-influential-programmers -[..]02/most-influential-programmers-results/

Few links about Knuth -

Not really an interview, just with few pointers of few articles -

Robin Milner, creater of ML -[..]nology/2010/apr/01/robin-milner-obituary

Things Every Programmer Should Know -[..]things-every-programmer-should-know.html

egnyte -[..]ons-learned-in-building-and-scaling.html

Bram Moolenaar: "Vim is a very important part of my life." -

(google search) (amazon search)

Wed Jan 19 11:24:47 AEDT 2022 From /weblog/design/distribute


In one sentence, here's why: humans are notoriously bad at keeping "self" distinct from "other". Egomania, projection (transference), and enmeshment are well-known symptoms of this problem. OK, so I hear you saying, "yeah, but what does this have to do with programming?" It certainly seems absurd to suggest that if we are bad at something we know the most about (our "selves"), how could we possibly say that we have a good approach for the programming analogues - objects, modules, etc. -

Argue why space base design is better than n-tier design -[..]0The%20End%20of%20Tier-based%20Computing

Some key research of google for distributed computation -

Someone think we are not yet (per Oct 2007) have good language support for distibuted computing -

A blog contain a lot distributed computing information -

How Wikipedia manage their site -

Google tutorial for Design Distributed System -

The Hadoop Distributed File System: Architecture and Design -[..]-a-list-of-distributed-key-value-stores/

Applying experience from CPU design for distributed solution -[..]o/post/26909672264/on-distributed-memory

Distributed systems for fun and profit -

Monitor and design -[..]buted-mission-critical-applications.html

Uber case study -[..]les-their-real-time-market-platform.html

Load balancer design -

Some issues of distributing logic to difference systems -[..]t-f-up-your-microservices-in-production/[..]lised-solution-to-distributed-consensus/

Two Phase Commit -[..]istributed-systems/two-phase-commit.html

(google search) (amazon search)

Wed Jan 19 11:23:06 AEDT 2022 From /weblog/design/examples


(google search) (amazon search)

Sat Jan 15 22:28:58 AEDT 2022 From /weblog/design


What's in a Good Error Message? -

(google search) (amazon search)

Wed Jan 05 00:59:20 AEDT 2022 From /weblog/design


System wide unique nanosecond timestamps -[..]m-wide-unique-nanosecond-timestamps.html[..]22/01/distributed-unique-time-stamp.html

(google search) (amazon search)

Tue Jan 04 15:32:08 AEDT 2022 From /weblog/design/examples


ZeroMQ: The Design of Messaging Middleware -[..]&siteSectionName=architecture-and-design

When NOT to use a message queue -

现代IM系统中的消息系统架构 - 架构篇 -

(google search) (amazon search)

Sat Jan 01 23:44:21 AEDT 2022 From /weblog/design


Date handling look simple but easy to fail in trap. Even worst is this is easy to have bad data which hard to fix. Here is an example -

One tip for testing application with time dependence, treat it as random -[..]

Explanation of issues of Joda-time, basically it still haven't map the human view of time close enough than machine view of time -

Screencast of how to driven a fluent Date API -[..]-apis-in-java-episode-1-comparing-dates/

解读数据架构的 2021:大数据 1.0 体系基本建成,但头上仍有几朵乌云 -

(google search) (amazon search)

Sun Nov 28 23:45:20 AEDT 2021 From /weblog/design/examples


Life of a Netflix Partner Engineer — The case of the extra 40 ms -[..]eer-the-case-of-extra-40-ms-b4c2dd278513

A look under the hood of the most successful streaming service on the planet -

(google search) (amazon search)

Fri Nov 26 10:26:25 AEDT 2021 From /weblog/design


Top Ten Software Architecture Mistakes -

Discussion about the balance of upfront design -[..]4/25/the-scatology-of-agile-architecture

There is no conflict in agile and architecture -[..]s-no-conflict-in-agile-and-architecture/

Experience of build instead of buy -[..]uy-one-companys-decision-to-go-it-alone/

Problem of too many layer -[..]/10/the-madness-of-layered-architecture/[..]are-architect-here-is-your-reading-list/

Sometime it is worth to build to throw -

Business Transactions
External Dependencies
Caching Strategy
Garbage Collection
Application Topology[..]s-to-capture-in-enterprise-applications/

invent a solution the majority of whose components depend on no others with those few components whose task it is to compose those other components being left largely to coordination roles. -[..]ting-started-with-big-data-architecture/

Twitter -[..]tter-handles-3000-images-per-second.html

Suning, more on HTTP -[..]oduct-details-system-architecture-design

Decision -

Some suggestion of making system simple, well, but the suggestion itself it is that simple -[..]ural-patterns-in-a-nutshell-a0b47a1e9013

Explaining common different architecture patterns -[..]l-patterns-you-need-to-know-a1f5ea7e4e3d

Software Architecture Patterns -[..]ecture-patterns-5-mins-read-e9e3c8eb47d2

(google search) (amazon search)

Mon Nov 22 00:07:33 AEDT 2021 From /weblog/design/examples


Crashproofing the Original NoSQL Key-Value Store -

The history of Berkeley DB -

(google search) (amazon search)

Thu Sep 30 11:22:17 AEST 2021 From /weblog/design/pattern


Non-Software Examples of Software Design Patterns -[..]vers/nonSoftwareExample/patexamples.html[..]hain-of-Responsibility-Pattern-in-C.aspx

Patterns to replace switch -

Design pattern in JDK -

Pattern examples in java -

(google search) (amazon search)

Tue Aug 24 00:12:35 AEST 2021 From /weblog/design/examples


10 Things You Didn’t Know About jOOQ -

(google search) (amazon search)

Sun Jul 25 00:13:07 AEST 2021 From /weblog/design/pattern


Saga Orchestration for Microservices Using the Outbox Pattern -

(google search) (amazon search)

Mon Jun 21 23:31:01 AEST 2021 From /weblog/design/pattern

Best pattern

Discussion about which pattern is most important -[..]page/abien?entry=most_important_patterns

Pattern Language Titles -

10 Design Patterns every Software Architect and Software Engineer must know -[..]oftware-architect-must-know-b33237bc01c2

(google search) (amazon search)

Sat Jun 19 23:49:45 AEST 2021 From /weblog/design


Tired with duplicating Try-Catch for resource acquisition? Can try to create proxy method -[..]clarative-synchronization-with-java.html

Discussion of naming of if block -

Factors for choosing a strategic technology stack, runtime performance is not a factor for him, probably good idea -[..]1/04/factors-for-choosing-strategic.html

Don't think , stay focus -

Using class name as variable name is a bad thing? I doubt about it, but interesting idea -

The problem of using generic name -

Beware of where memory allocated, sometime there is no off side for lazy initalization -[..]ou-dare-to-change-hashmap-implementation

Some suggestion of how to code in order to prevent unreproducible bugs -[..]4/02/minimizing-unreproducible-bugs.html

More careful to check null -[..]et-paranoid-java-programming-techniques/

The fewer conditions you test for, the better your code “tastes”. -[..]ding-requirement-99749f37684a#.1glkhe7np

Think twice before running a loop -

Example of how to remove mutation status -

The Best Algorithm No One Knows About, linear randomization -[..]0/the-best-algorithm-no-one-knows-about/

Small change can make big difference -[..]ars-with-a-single-character-code-change/

Eager is Easy, Lazy is Labyrinthine -[..]s-easy-lazy-is-labyrinthine-b12605f13048

How to organize your code? -

When you have to -[..]r-tutorial-from-basics-to-best-practices

(google search) (amazon search)