RSS feed [root] /design /distribute /weblog




login:

password:

title search:




 


Tue Aug 08 18:09:43 HKT 2017

distribute



(google search) (amazon search)
second
download zip of files only

Tue Dec 12 10:41:54 HKT 2017 From /weblog/design/distribute

scalability


There are two key primary ways of scaling web applications which is in practice today.
1) “Vertical Scalability” - Adding resource within the same logical unit to increase capacity. An example of this would be to add CPUs to an existing server, or expanding storage by adding hard drive on an existing RAID/SAN storage.
2) “Horizontal Scalability” - Adding multiple logical units of resources and making them work as a single unit. Most clustering solutions, distributed file systems, load-balancers help you with horizontal scalability.

Scalability can be further sub-classified based on the “scalability factor”.
1) If the scalability factor stays constant as you scale. This is called “linear scalability“.
2) But chances are that some components may not scale as well as others. A scalability factor below 1.0 is called “sub-linear scalability“.
3) Though rare, its possible to get better performance (scalability factor) just by adding more components (i/o across multiple disk spindles in a RAID gets better with more spindles). This is called “supra-linear scalability“.
4) If the application is not designed for scalability, its possible that things can actually get worse as it scales. This is called “negative scalability“.

http://www.royans.net/arch/2007/09/22/what-is-scalability/

Report of building web application with 55k pageload with rail - http://shanti.railsblog.com[..]mongrels-handled-a-550k-pageview-digging

XMPP a IM protocol about scalability - http://www.process-one.net[..]icle/the_aol_xmpp_scalability_challenge/

Presentation and resources of making you website more scalable - http://www.scribd.com[..]9/Real-World-Web-Performance-Scalability http://www.theserverside.com[..]lications&asrc=EM_NLN_3990118&uid=703565 http://www.theserverside.com[..]ionsPart2&asrc=EM_NLN_3990119&uid=703565

Brian Zimmer, architect at travel startup Yapta, highlights some worst practices jeopardizing the growth and scalability of a system:
* The Golden Hammer. Forcing a particular technology to work in ways it was not intended is sometimes counter-productive. Using a database to store key-value pairs is one example. Another example is using threads to program for concurrency.
* Resource Abuse. Manage the availability of shared resources because when they fail, by definition, their failure is experienced pervasively rather than in isolation. For example, connection management to the database through a thread pool.
* Big Ball of Mud. Failure to manage dependencies inhibits agility and scalability.
* Everything or Something. In both code and application dependency management, the worst practice is not understanding the relationships and formulating a model to facilitate their management. Failure to enforce diligent control is a contributing scalability inhibiter.
* Forgetting to check the time. To properly scale a system it is imperative to manage the time alloted for requests to be handled.
* Hero Pattern. One popular solution to the operation issue is a Hero who can and often will manage the bulk of the operational needs. For a large system of many components this approach does not scale, yet it is one of the most frequently-deployed solutions.
* Not automating. A system too dependent on human intervention, frequently the result of having a Hero, is dangerously exposed to issues of reproducibility and hit-by-a-bus syndrome.
* Monitoring. Monitoring, like testing, is often one of the first items sacrificed when time is tight.

http://highscalability.com/scalability-worst-practices

Useful Corporate Blogs that Talk About Scalability - http://highscalability.com[..]l-corporate-blogs-talk-about-scalability

Overview of mapreduce and how it compare with other distributed programming model -http://natishalom.typepad.com[..]0/is-mapreduce-going-to-main-stream.html

Paper of data store at amazon http://www.allthingsdistributed.com/2007/10/amazons_dynamo.html

Discuss how haven't sync can cause performance issue - http://www.theserverside.com[..]lications&asrc=EM_NLN_6273194&uid=703565 http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6423457

Discussion about Cloud Based Memory Architectures - http://highscalability.com[..]ased-memory-architectures-next-big-thing

http://highscalability.com[..]alability-and-performance-best-practices

Interview with google engineer - http://www.zdnet.co.uk[..]gle-at-scale-everything-breaks-40093061/

Surprisingly youtube is blocking - http://highscalability.com[..]e-scalability-lessons-in-30-minutes.html

If we are seeing a sustained arrival rate of requests, greater than our system is capable of processing, then something has to give. Having the entire system degrade is not the ideal service we want to give our customers. A better approach would be to process transactions at our systems maximum possible throughput rate, while maintaining a good response time, and rejecting requests above this arrival rate. - http://mechanical-sympathy.blogspot.com.au[..]apply-back-pressure-when-overloaded.html

How twitter scaling - http://www.infoq.com/presentations/Twitter-Analytics

How Reddit scaling - http://www.infoq.com/presentations/scaling-reddit

How Hotjar scaling - https://www.hotjar.com[..]-while-scaling-hotjars-tech-architecture

How infiniteDB prevent locking and IO - http://highscalability.com[..]-scalable-relational-database-manag.html

http://highscalability.com[..]ard-way-about-scaling-a-million-use.html http://martin.kleppmann.com[..]2014/03/26/six-things-about-scaling.html

The experiences of various big companies, about network issues - http://aphyr.com/posts/288-the-network-is-reliable

Stackoverflow, scale without cloud - http://highscalability.com[..]nth-25-servers-and-i.html?SSLoginOk=true

How netflix scale - http://highscalability.com[..]ix-what-happens-when-you-press-play.html

(google search) (amazon search)


Tue Aug 08 18:12:08 HKT 2017 From /weblog/design/distribute

event


Event Bus Implementation(s) - https://hackernoon.com/event-bus-implementation-s-d2854a9fafd5 http://www.infoq.com[..]2017/08/Summary-event-bus-implementation

(google search) (amazon search)


Mon May 09 10:41:01 HKT 2016 From /weblog/design/distribute

distribute


In one sentence, here's why: humans are notoriously bad at keeping "self" distinct from "other". Egomania, projection (transference), and enmeshment are well-known symptoms of this problem. OK, so I hear you saying, "yeah, but what does this have to do with programming?" It certainly seems absurd to suggest that if we are bad at something we know the most about (our "selves"), how could we possibly say that we have a good approach for the programming analogues - objects, modules, etc. - http://www.artima.com/weblogs/viewpost.jsp?thread=46706

Argue why space base design is better than n-tier design - http://www.google.com[..]0The%20End%20of%20Tier-based%20Computing

Some key research of google for distributed computation - http://www.infoq.com/news/2007/06/google-scalability

Someone think we are not yet (per Oct 2007) have good language support for distibuted computing - http://kasparov.skife.org/blog/2007/10/11/

A blog contain a lot distributed computing information - http://www.highscalability.com/

How Wikipedia manage their site - http://dammit.lt/uc/workbook2007.pdf

Google tutorial for Design Distributed System - http://code.google.com/edu/parallel/dsd-tutorial.html

http://en.wikipedia.org/wiki/Distributed_hash_table

The Hadoop Distributed File System: Architecture and Design - http://hadoop.apache.org/core/docs/r0.18.0/hdfs_design.html

http://www.metabrew.com[..]-a-list-of-distributed-key-value-stores/

Applying experience from CPU design for distributed solution - http://blog.paralleluniverse.co[..]o/post/26909672264/on-distributed-memory

Distributed systems for fun and profit - http://book.mixu.net/distsys/single-page.html

Monitor and design - http://highscalability.com[..]buted-mission-critical-applications.html

Uber case study - http://highscalability.com[..]les-their-real-time-market-platform.html

Load balancer design - http://www.thegeekstuff.com/2016/01/load-balancer-intro/

Some issues of distributing logic to difference systems - http://blog.takipi.com[..]t-f-up-your-microservices-in-production/

(google search) (amazon search)



Mon Dec 22 12:04:04 HKT 2014 From /weblog/design/distribute

performance


Basically, cache as much as you can, limit the bandwidth as much as you can - http://horicky.blogspot.com[..]2009/08/skinny-straw-in-cloud-shake.html

http://www.edwardcapriolo.com[..]ry/cassandra_compression_is_like_getting

Compression usually very useful - http://abdullin.com[..]observations-on-big-data-for-retail.html

Discussion of design of Aeron, a new messaging system - http://highscalability.com[..]eally-need-another-messaging-system.html

How BBG use Hadoop, and tune it - http://highscalability.com[..]2/17/the-big-problem-is-medium-data.html

(google search) (amazon search)


Mon Apr 14 11:43:47 HKT 2014 From /weblog/design/distribute

mapreduce


MapReduce patterns

Basic MapReduce Patterns
Counting and Summing
Collating
Filtering (“Grepping”), Parsing, and Validation
Distributed Task Execution
Sorting
Not-So-Basic MapReduce Patterns
Iterative Message Passing (Graph Processing)
Distinct Values (Unique Items Counting)
Cross-Correlation
Relational MapReduce Patterns
Selection
Projection
Union
Intersection
Difference
GroupBy and Aggregation
Joining

- http://highlyscalable.wordpress.com[..]press.com/2012/02/01/mapreduce-patterns/

Showing that map reduce can support real time transaction processing - http://googleblog.blogspot.com[..]09/12/relevance-meets-real-time-web.html

Using map-reduce in cloud - http://horicky.blogspot.com/2010/02/cloud-mapreduce-tricks.html

Papers of using mapreduce - http://atbrox.com[..]thms-in-academic-papers-may-2010-update/ http://code.google.com/edu/parallel/mapreduce-tutorial.html

mapreduce experiment - http://www.macs.hw.ac.uk/~rs46/multicore_challenge1/

Pattern and anti-pattern - http://developer.yahoo.com[..]/2010/08/apache_hadoop_best_practices_a/ http://blog.dynatrace.com[..]bout-the-performance-of-map-reduce-jobs/

http://techalpine.com/what-are-the-hadoop-mapreduce-concepts/

(google search) (amazon search)


Fri Apr 11 16:26:11 HKT 2014 From /weblog/design/distribute

cloud


1. Use Cloud for Scaling
2. Use Cloud for Multi-tenancy
3. Use Cloud for Batch processing
4. Use Cloud for Storage
5. Use Cloud for Communication

http://horicky.blogspot.com/2009/11/cloud-computing-patterns.html

http://horicky.blogspot.com/2009/11/nosql-patterns.html

Database in cloud - http://drdobbs.com[..]int?articleId=218900502&siteSectionName=

An overview of the Hadoop/MapReduce/HBase framework and its current applications in bioinformatics - http://www.biomedcentral.com/1471-2105/11/S12/S1

The architecture that survived when amazon outage - http://www.infoq.com/news/2011/04/twilio-cloud-architecture

Introduction of few tools for cloud development - http://blog.codecentric.de[..]/01/best-development-tools-in-the-cloud/ http://www.javacodegeeks.com[..]/07/developing-and-testing-in-cloud.html

Google Finds: Centralized Control, Distributed Data Architectures Work Better than Fully Decentralized Architectures - http://highscalability.com[..]control-distributed-data-architectu.html

(google search) (amazon search)


Wed Mar 05 16:42:02 HKT 2014 From /weblog/design/distribute

whatapp


http://highscalability.com[..]ture-facebook-bought-for-19-billion.html

(google search) (amazon search)


Mon Dec 23 18:06:30 HKT 2013 From /weblog/design/distribute

Idempotence


Sample chater of REST book , which contain a nice discussion of why Idempotence is important - http://www.infoq.com[..]dson-ruby-restful-ws/en/resources/04.pdf

New Acid:
* A – Associative
* C – Commutative
* I – Idempotent
* D - Distributed

http://www.eaipatterns.com/ramblings/68_acid.html

Idempotency patterns - http://jonathan-oliver.blogspot.com[..]ot.com/2010/04/idempotency-patterns.html

Detailed discussion about how to design Idempotence - http://queue.acm.org/detail.cfm?id=2187821

Discuss about clock in distributed environment - http://basho.com[..]e-bad-or-welcome-to-distributed-systems/

(google search) (amazon search)


Fri Dec 20 12:17:18 HKT 2013 From /weblog/design/distribute

overload


Solution #1: Have More Resources than You'll Ever Need
Solution #2: Disable Features During High Loads
Solution #3: Auto Scaling
Solution #4: Use Message Queues

http://blog.iron.io[..]spikability-applications-ability-to.html

http://highscalability.com[..]izing-and-capacity-planning-assumin.html

(google search) (amazon search)


Tue Oct 29 17:54:03 HKT 2013 From /weblog/design/distribute

feed


http://highscalability.com[..]for-scaling-your-high-traffic-feeds.html

(google search) (amazon search)


Fri Oct 18 14:22:38 HKT 2013 From /weblog/design/distribute

nosql


Compare SQL ( RDBMS ) and noSQL ( object base {distributed??} ) data management - http://queue.acm.org/detail.cfm?id=1961297&ref=fullrss

The definition - http://martinfowler.com/bliki/NosqlDefinition.html http://martinfowler.com/articles/nosqlKeyPoints.html

Is Cassandra really that good? - http://www.datastax.com/dev/blog/2012-in-review-performance

http://martinfowler.com/articles/bigData/

NoSQL Options Compared - http://www.drdobbs.com[..]cleId=240151198&siteSectionName=database

A 3 Step Guide to Getting Started with NoSQL - http://jyops.blogspot.com.au[..]-step-guide-to-getting-started-with.html

SAFER THAN RDBMSs – BUILT FOR DISASTER AVOIDANCE - The http://www.datastax.com[..]/10/why-nosql-can-be-safer-than-an-rdbms

(google search) (amazon search)


Thu Oct 17 18:16:02 HKT 2013 From /weblog/design/distribute

realtime


Few real time map reduce framework - http://www.infoq.com/news/2011/09/twitter-storm-real-time-hadoop

How facebook and twitter doing real-time analytic - http://natishalom.typepad.com[..]ers-approach-to-real-time-analytics.html

(google search) (amazon search)


Tue Oct 02 20:34:57 HKT 2012 From /weblog/design/distribute

spanner


Google spanner

https://cloudant.com/blog/cloudant-labs-on-google-spanner/
http://nosql.mypopescu.com[..]ogle-spanners?e57078b0?317bb970?b3b99980

(google search) (amazon search)


Mon Apr 16 22:38:34 HKT 2012 From /weblog/design/distribute

storage


How facbook manage photo storage - http://www.facebook.com/note.php?note_id=76191543919&ref=mf

Promotion letter for oracle coherence, but still a good reading - http://highlyscalable.wordpress.com[..]e-of-high-performance-ecommerce-backend/

(google search) (amazon search)


Tue Feb 08 00:38:32 HKT 2011 From /weblog/design/distribute

GridGain


Implement ping-pong play between two nodes on the cloud using GridGain Distributed Actors - http://gridgaintech.wordpress.com[..]11/01/26/distributed-actors-in-gridgain/

(google search) (amazon search)


Fri Nov 27 15:56:17 HKT 2009 From /weblog/design/distribute

eventual-consistency


A short example to show how eventual-consistency work - http://sbtourist.blogspot.com[..]/11/eventual-consistency-by-example.html

(google search) (amazon search)