RSS feed [root] /design /weblog




login:

password:

title search:




 


Sun Sep 10 16:25:08 HKT 2017

design



(google search) (amazon search)
second
download zip of files only

Sun Oct 16 01:52:15 HKT 2016 From /weblog/design/interview

yahoo


How yahoo use Hadoop for machine learning - https://www.infoq.com/articles/peter-cnudde-yahoo-big-data

(google search) (amazon search)


Tue Sep 13 00:10:46 HKT 2016 From /weblog/design

latency


http://highscalability.com[..]here-and-it-costs-you-sales-how-crush-it

*Do not use locks in the main transaction flow because they cause context switches, and therefore latency and unpredictable jitter.
*Never have more threads that need to run than you have cores available.
*Set affinity of threads to cores, or at least sockets, to avoid cache pollution by avoiding migration. This is particularly important when on a server class machine having multiple sockets because of the NUMA effect.
*Ensure uncontested access to any resource respecting the Single Writer Principle so that the likes of biased locking can be your friend.
*Keep call stacks reasonably small. Still more work to do here. If you are crazy enough to use Spring, then check out your call stacks to see what I mean! The garbage collector has to walk them finding reachable objects.
*Do not use finalizers.
*Keep garbage generation to modest levels. This applies to most JVMs but is likely not an issue for Zing.
*Ensure no disk IO on the main flow.
*Do a proper warm-up before beginning to measure.
*Do all the appropriate OS tunings for low-latency systems that are way beyond this blog. For example turn off C-States power management in the BIOS and watch out for RHEL 6 as it turns it back on without telling you!
*Macro-benchmarking is much more valuable than micro-benchmarking.
*Amazing results are achieved by truly agile companies, staffed by talented individuals, who are empowered to make things happen. Make sh*t happen is more important than following a process.

http://highscalability.com[..]ughput-by-32x-and-reduce-latency-by.html

http://bravenewgeek.com[..]rything-you-know-about-latency-is-wrong/

How to monitor - https://plumbr.eu[..]siness-value-from-performance-monitoring

How to measure latency - https://vanilla-java.github.io[..]/07/20/Latency-for-a-set-Throughput.html

Various cause of latency, and the solution - https://www.informatica.com[..]Topics-in-High-Performance-Messaging.htm

(google search) (amazon search)


Fri Sep 09 16:00:46 HKT 2016 From /weblog/design/examples

patching


How QQ deliver hotfix at Android - http://www.infoq.com/cn/articles/Android-hot-fix

(google search) (amazon search)



Wed Aug 17 00:06:59 HKT 2016 From /weblog/design/examples

linkedin


How linkedin handle logs - https://engineering.linkedin.com[..]ould-know-about-real-time-datas-unifying

(google search) (amazon search)


Thu Jul 21 19:29:09 HKT 2016 From /weblog/design/distribute

scalability


There are two key primary ways of scaling web applications which is in practice today.
1) “Vertical Scalability” - Adding resource within the same logical unit to increase capacity. An example of this would be to add CPUs to an existing server, or expanding storage by adding hard drive on an existing RAID/SAN storage.
2) “Horizontal Scalability” - Adding multiple logical units of resources and making them work as a single unit. Most clustering solutions, distributed file systems, load-balancers help you with horizontal scalability.

Scalability can be further sub-classified based on the “scalability factor”.
1) If the scalability factor stays constant as you scale. This is called “linear scalability“.
2) But chances are that some components may not scale as well as others. A scalability factor below 1.0 is called “sub-linear scalability“.
3) Though rare, its possible to get better performance (scalability factor) just by adding more components (i/o across multiple disk spindles in a RAID gets better with more spindles). This is called “supra-linear scalability“.
4) If the application is not designed for scalability, its possible that things can actually get worse as it scales. This is called “negative scalability“.

http://www.royans.net/arch/2007/09/22/what-is-scalability/

Report of building web application with 55k pageload with rail - http://shanti.railsblog.com[..]mongrels-handled-a-550k-pageview-digging

XMPP a IM protocol about scalability - http://www.process-one.net[..]icle/the_aol_xmpp_scalability_challenge/

Presentation and resources of making you website more scalable - http://www.scribd.com[..]9/Real-World-Web-Performance-Scalability http://www.theserverside.com[..]lications&asrc=EM_NLN_3990118&uid=703565 http://www.theserverside.com[..]ionsPart2&asrc=EM_NLN_3990119&uid=703565

Brian Zimmer, architect at travel startup Yapta, highlights some worst practices jeopardizing the growth and scalability of a system:
* The Golden Hammer. Forcing a particular technology to work in ways it was not intended is sometimes counter-productive. Using a database to store key-value pairs is one example. Another example is using threads to program for concurrency.
* Resource Abuse. Manage the availability of shared resources because when they fail, by definition, their failure is experienced pervasively rather than in isolation. For example, connection management to the database through a thread pool.
* Big Ball of Mud. Failure to manage dependencies inhibits agility and scalability.
* Everything or Something. In both code and application dependency management, the worst practice is not understanding the relationships and formulating a model to facilitate their management. Failure to enforce diligent control is a contributing scalability inhibiter.
* Forgetting to check the time. To properly scale a system it is imperative to manage the time alloted for requests to be handled.
* Hero Pattern. One popular solution to the operation issue is a Hero who can and often will manage the bulk of the operational needs. For a large system of many components this approach does not scale, yet it is one of the most frequently-deployed solutions.
* Not automating. A system too dependent on human intervention, frequently the result of having a Hero, is dangerously exposed to issues of reproducibility and hit-by-a-bus syndrome.
* Monitoring. Monitoring, like testing, is often one of the first items sacrificed when time is tight.

http://highscalability.com/scalability-worst-practices

Useful Corporate Blogs that Talk About Scalability - http://highscalability.com[..]l-corporate-blogs-talk-about-scalability

Overview of mapreduce and how it compare with other distributed programming model -http://natishalom.typepad.com[..]0/is-mapreduce-going-to-main-stream.html

Paper of data store at amazon http://www.allthingsdistributed.com/2007/10/amazons_dynamo.html

Discuss how haven't sync can cause performance issue - http://www.theserverside.com[..]lications&asrc=EM_NLN_6273194&uid=703565 http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6423457

Discussion about Cloud Based Memory Architectures - http://highscalability.com[..]ased-memory-architectures-next-big-thing

http://highscalability.com[..]alability-and-performance-best-practices

Interview with google engineer - http://www.zdnet.co.uk[..]gle-at-scale-everything-breaks-40093061/

Surprisingly youtube is blocking - http://highscalability.com[..]e-scalability-lessons-in-30-minutes.html

If we are seeing a sustained arrival rate of requests, greater than our system is capable of processing, then something has to give. Having the entire system degrade is not the ideal service we want to give our customers. A better approach would be to process transactions at our systems maximum possible throughput rate, while maintaining a good response time, and rejecting requests above this arrival rate. - http://mechanical-sympathy.blogspot.com.au[..]apply-back-pressure-when-overloaded.html

How twitter scaling - http://www.infoq.com/presentations/Twitter-Analytics

How Reddit scaling - http://www.infoq.com/presentations/scaling-reddit

How Hotjar scaling - https://www.hotjar.com[..]-while-scaling-hotjars-tech-architecture

How infiniteDB prevent locking and IO - http://highscalability.com[..]-scalable-relational-database-manag.html

http://highscalability.com[..]ard-way-about-scaling-a-million-use.html http://martin.kleppmann.com[..]2014/03/26/six-things-about-scaling.html

The experiences of various big companies, about network issues - http://aphyr.com/posts/288-the-network-is-reliable

Stackoverflow, scale without cloud - http://highscalability.com[..]nth-25-servers-and-i.html?SSLoginOk=true

(google search) (amazon search)




Mon May 09 10:41:01 HKT 2016 From /weblog/design/distribute

distribute


In one sentence, here's why: humans are notoriously bad at keeping "self" distinct from "other". Egomania, projection (transference), and enmeshment are well-known symptoms of this problem. OK, so I hear you saying, "yeah, but what does this have to do with programming?" It certainly seems absurd to suggest that if we are bad at something we know the most about (our "selves"), how could we possibly say that we have a good approach for the programming analogues - objects, modules, etc. - http://www.artima.com/weblogs/viewpost.jsp?thread=46706

Argue why space base design is better than n-tier design - http://www.google.com[..]0The%20End%20of%20Tier-based%20Computing

Some key research of google for distributed computation - http://www.infoq.com/news/2007/06/google-scalability

Someone think we are not yet (per Oct 2007) have good language support for distibuted computing - http://kasparov.skife.org/blog/2007/10/11/

A blog contain a lot distributed computing information - http://www.highscalability.com/

How Wikipedia manage their site - http://dammit.lt/uc/workbook2007.pdf

Google tutorial for Design Distributed System - http://code.google.com/edu/parallel/dsd-tutorial.html

http://en.wikipedia.org/wiki/Distributed_hash_table

The Hadoop Distributed File System: Architecture and Design - http://hadoop.apache.org/core/docs/r0.18.0/hdfs_design.html

http://www.metabrew.com[..]-a-list-of-distributed-key-value-stores/

Applying experience from CPU design for distributed solution - http://blog.paralleluniverse.co[..]o/post/26909672264/on-distributed-memory

Distributed systems for fun and profit - http://book.mixu.net/distsys/single-page.html

Monitor and design - http://highscalability.com[..]buted-mission-critical-applications.html

Uber case study - http://highscalability.com[..]les-their-real-time-market-platform.html

Load balancer design - http://www.thegeekstuff.com/2016/01/load-balancer-intro/

Some issues of distributing logic to difference systems - http://blog.takipi.com[..]t-f-up-your-microservices-in-production/

(google search) (amazon search)


Tue Feb 16 11:41:20 HKT 2016 From /weblog/design/interview

Collections


http://www.javacodegeeks.com[..]7/top-97-things-every-programmer-or.html

I love the tool that Linus using - http://sztywny.titaniumhosting.com[..]23/stiff-asks-great-programmers-answers/ and here is the abstract - http://www.ultrasaurus.com/sarahblog/archives/000322.html

Not really interview, just a discuss of what programmer difference from other - http://thecodist.com[..]in-on-codewhat-makes-you-able-to-program

Vote for most-influential-programmers - http://blog.assembleron.com[..]02/most-influential-programmers-results/

Few links about Knuth - http://www.codinghorror.com/blog/archives/001034.html

Not really an interview, just with few pointers of few articles - http://www.codinghorror.com/blog/archives/000987.html

Robin Milner, creater of ML - http://www.guardian.co.uk[..]nology/2010/apr/01/robin-milner-obituary

Things Every Programmer Should Know - http://www.javacodegeeks.com[..]things-every-programmer-should-know.html

egnyte - http://highscalability.com[..]ons-learned-in-building-and-scaling.html

(google search) (amazon search)



Wed Dec 09 11:20:40 HKT 2015 From /weblog/design

availability


https://www.pgrs.net/2013/02/21/high-availability-at-braintree/

https://medium.com[..]y-available-code-7eaf3d7aae00#.cbae171gy

(google search) (amazon search)


Thu Dec 03 10:33:50 HKT 2015 From /weblog/design/interview

UncleBob


http://www.infoq.com/news/2015/11/uncle-bob-oath-programmer

(google search) (amazon search)


Sun Nov 22 21:31:00 HKT 2015 From /weblog/design

API design guideline


This is a message from a management blog, but I think the arguement is also apply API design. In fact, I think most critical difference of good and bad API is knowing which small detail is important and which is not - http://www.goodproductmanager.com[..]er.com/2007/11/08/sweat-the-small-stuff/

Design tips:
http://www.artima.com/weblogs/viewpost.jsp?thread=142428
http://openide.netbeans.org/tutorial/api-design.html
http://www.cincomsmalltalk.com[..]gView?showComments=true&entry=3258158706
http://today.java.net[..]its-of-highly-profitable-developers.html
http://www.infoq.com/news/2007/08/why-api-design-matters
http://www.infoq.com/presentations/effective-api-design
http://neuroning.com/2006/11/19/on-api-design-guidelines

About Compatibility issues
http://wiki.eclipse.org/Evolving_Java-based_APIs

A blog of using interfaces-vs-abstract-classes
http://hoskinator.blogspot.com[..]6/04/interfaces-vs-abstract-classes.html

XOM design overview - http://www.xom.nu/designprinciples.xhtml#d0e309

You need to identify the business value but not pick any tools/design just because it is cool - http://myarch.com/what-is-good-soa

A lot of links here - http://kasparov.skife.org/blog-live/src/api-design-refs.writeback http://discuss.joelonsoftware.com/default.asp?design.4.527465

An introduction of good OSS project to study their source - http://techkriti.wordpress.com[..]com/2007/06/28/learning-from-the-source/

CCCCDPIPE - http://blog.objectmentor.com/articles/2007/08/02/which-came-first

Discussion of why getting feedback quick is important - http://blog.objectmentor.com[..]u-dont-know-until-you-take-the-next-step

Someone saying that routines is the greatest invention in CS, I agree - http://www.codinghorror.com/blog/archives/001129.html

New way of modeling system then layering, The Onion Architecture - http://jeffreypalermo.com/blog/the-onion-architecture-part-1/

SOLID - http://www.lostechies.com[..]of-the-month-march-solid-principles.aspx http://dotnet.dzone.com/news/solid-software-works <-- very good picture! http://www.blackwasp.co.uk/SOLIDPrinciples.aspx

Idempotent, Orthogonality, Immutability - http://www.codinghorror.com/blog/archives/001244.html

Some example of how to driven good API - http://jdegoes.squarespace.com[..]al/2009/5/11/good-api-design-part-3.html

It's not enough to write tests for an API you develop, you have to write unit tests for code that uses your API. When you do, you learn first-hand the hurdles that your users will have to overcome when they try to test their code independently. http://butunclebob.com[..]MichaelFeathers.TheGoldenRuleOfApiDesign

Active interface vs. Passive interface, for me this is another way to under "Tell, don't ask" - http://code-o-matic.blogspot.com[..]-on-api-design-call-stack-as-source.html

The Principles of Good Programming - http://www.artima.com/weblogs/viewpost.jsp?thread=331531

http://net.tutsplus.com[..]software-principles-you-must-understand/

Difference choice of how API evolute with change - http://blog.jooq.org[..]sive-api-evolution-with-java-interfaces/

JUnit's evolving structure. - http://edmundkirwan.com/general/junit.html

API performance contract - http://queue.acm.org/detail.cfm?ref=rss&id=2576968

Consistent naming conventions
Standard terminology
Uniform error responses
Attention to detail (that’s the love part for APIs)
REST APIs that work with only 2 resources and use 4 HTTP verbs (oh, there is so much more to talk about here…)
Avoid API design by way of method-driven approach – that leads you down a slippery slope
- See more at: http://blogs.ancestry.com[..]are-like-parenting/#sthash.Ux1LHMNU.dpuf

http://blog.jooq.org[..]ake-this-mistake-when-developing-an-spi/

(google search) (amazon search)


Fri Nov 13 00:07:00 HKT 2015 From /weblog/design

memory


It won't happened at java, but for non-GC language like C++, crash at constructor will cause memory leak: http://blogs.asman-it.com.au[..]oders-have-it-easy-crashing-constructors

Discussion of if there are way to recover OOME - http://dobbscodetalk.com[..]-a-Non-Recoverable-Error-.html&Itemid=29

http://blog.maxindelicato.com[..]ghly-available-inmemory-shard-index.html

Discussion about using memory as store - http://martinfowler.com/bliki/MemoryImage.html

When using direct memory can be faster, when you have a large amounts of data and/or you have to perform some IO with that data. Tick data is a good example - http://vanillajava.blogspot.com.au[..]n-using-direct-memory-can-be-faster.html

http://www.memorymanagement.org/index.html

(google search) (amazon search)


Sat Nov 07 10:18:03 HKT 2015 From /weblog/design/interview

instagram



Do the simple thing first.
Do fewer things better.
Upfront work but can pay huge dividends.
Don’t reinvent the wheel.
Nothing lasts forever.

http://highscalability.com[..]-from-5-years-of-building-instagram.html

(google search) (amazon search)


Wed Sep 30 15:29:04 HKT 2015 From /weblog/design/pattern

tutorial


Collection of patterns from objectmentor - http://bradapp.blogspot.com[..]07/agile-scm-principles-from-ood-to.html

Robert Martin tutorial - http://www.java.net/pub/au/90

Links and abstract of few principle - http://www.surfscranton.com/Architecture/JavaPages.htm

Tutorial about common anti-pattern - http://giorgiosironi.blogspot.com/2009/11/anti-patterns.html

Development of Further Patterns of Enterprise Application Architecture - http://martinfowler.com/eaaDev/

Expanation and C++/Java samples - http://www.vincehuston.org/dp/

Java tutorial - http://www.javacodegeeks.com/2015/09/java-design-patterns.html


(google search) (amazon search)


Thu Sep 10 14:18:31 HKT 2015 From /weblog/design/interview

Martin Odersky


One of the challenges we were facing is we wanted to be both functional and object-oriented. We had very early on the notion that immutable objects would become very, very important. Nowadays everybody talks about immutable objects, because people think they are a key part of the solution to the concurrency problems caused by multi-core computers. Everybody says, no matter what you do, you need to try to have as much of your code using immutable objects as possible. In Scala, we did that very early on. Five or six years ago, we started to think very hard about immutable objects. It actually turns out that a lot of the object-oriented field up to then identified objects with mutability. For them, mutable state and objects were one and the same: mutable state was an essential ingredient of objects. We had to, in essence, ween objects off of that notion, and there were some things we had to do to make that happen.

http://www.artima.com/scalazine/articles/goals_of_scala.html

(google search) (amazon search)


Sat Aug 15 23:06:06 HKT 2015 From /weblog/design

game


Introduction to AI of game - http://aigamedev.com/theory/online-adaptation-game-opponent

How to create angry bird - http://www.wildbunny.co.uk[..]11/06/07/how-to-make-angry-birds-part-2/

How to program breast in game - http://www.inside.com.tw[..]15/07/30/how-video-game-breasts-are-made

http://www.inside.com.tw[..]eveloper-nintendo-president-satoru-iwata

(google search) (amazon search)


Wed May 27 10:17:34 HKT 2015 From /weblog/design

YAGNI


When XP says "go for the simplest possible design", what does that mean?
1. Simplest possible design to IMPLEMENT?
1. Simplest possible design to UNDERSTAND LATER?

An thread discuss how far should we go. Seen most people agree what should be done is "Simplest possible design to TEST and IMPLEMENT"

http://groups.yahoo.com/group/extremeprogramming/message/90466

It apply on feature, not on quality - http://martinfowler.com/bliki/Yagni.html

(google search) (amazon search)


Wed Dec 31 19:11:17 HKT 2014 From /weblog/design/concurrency

issues


Case study of a parallel slowdown, and the analysis of how to find it out - http://webtide.intalio.com[..]2/avoiding-parallel-slowdown-in-jetty-9/

Discuss about the racing condition when multiple accessing java.util.HashMap - http://mailinator.blogspot.com.au[..]au/2009/06/beautiful-race-condition.html

(google search) (amazon search)


Mon Dec 22 12:04:04 HKT 2014 From /weblog/design/distribute

performance


Basically, cache as much as you can, limit the bandwidth as much as you can - http://horicky.blogspot.com[..]2009/08/skinny-straw-in-cloud-shake.html

http://www.edwardcapriolo.com[..]ry/cassandra_compression_is_like_getting

Compression usually very useful - http://abdullin.com[..]observations-on-big-data-for-retail.html

Discussion of design of Aeron, a new messaging system - http://highscalability.com[..]eally-need-another-messaging-system.html

How BBG use Hadoop, and tune it - http://highscalability.com[..]2/17/the-big-problem-is-medium-data.html

(google search) (amazon search)


Thu Dec 11 18:21:23 HKT 2014 From /weblog/design/pattern

notification


Sometime it is better to notify user about failing to validate input than throw - http://martinfowler.com[..]ticles/replaceThrowWithNotification.html

(google search) (amazon search)


Fri Nov 14 11:35:30 HKT 2014 From /weblog/design

functional


Having functional program in java? http://codemonkeyism.com/functional-programming/

Functional programming in old java, with eclipse template - http://www.javacodegeeks.com[..]functional-programming-with-map-and.html

Functional Programming For The Rest of Us - http://www.defmacro.org/ramblings/fp.html?

collection-pipeline - http://martinfowler.com/articles/collection-pipeline/

(google search) (amazon search)


Thu Sep 25 17:42:08 HKT 2014 From /weblog/design

datetime


Look like Java standard library already handle the 400 years leap day. However, there are still a lot to take care. - http://discuss.joelonsoftware.com/default.asp?design.4.326089

P.S.: an article about java timezone handling - http://www.javaworld.com/javaworld/jw-10-2003/jw-1003-time_p.html

Basic about computer time - http://blogs.windwardreports.com[..]ry-developer-should-know-about-time.html

Difference of time function provided by IPP - http://software.intel.com[..]g-function-for-measuring-ipp-api-timing/

Detail explanation of nano second and millisecond in java at windows platform - https://blogs.oracle.com[..]olmes/entry/inside_the_hotspot_vm_clocks

How to model thing that change with time - http://www.martinfowler.com/ap2/timeNarrative.html

Timezone handling issues of most application - http://martinfowler.com/bliki/TimeZoneUncertainty.html


(google search) (amazon search)